Special Publication 500-268

Source Code Security Analysis Tool 

Functional Specification Version 1.0 

Paul E. Black 

Michael Kass 

Michael Koo 

Software Diagnostics and Conformance Testing Division 

Information Technology Laboratory 

National Institute of Standards and Technology 

Gaithersburg, MD 20899 

May 2007 

U.S. Department of Commerce 

Carlos M. Gutierrez, Secretary 

Technology Administration 

Robert Cresanti, Under Secretary of Commerce for Technology 

National Institute of Standards and Technology 

William Jeffrey, Director
NIST SP 500-268 May 2007 Page 2 of 14 

Abstract: 

Software assurance tools are a fundamental resource for providing an assurance argument for 

today’s software applications throughout the software development lifecycle. Some tools analyze 

software requirements, design models, source code, or executable code to help determine if an 

application is secure. This document specifies the behavior of one class of software assurance 

tool: the source code security analyzer. Because many software security weaknesses today are 

introduced at the implementation phase, using a source code security analyzer should help 

assure that software doesn’t have many security vulnerabilities. This specification defines a 

minimum capability to help software professionals understand how a tool will meet their software 

security assurance needs. 

Keywords: 

Homeland security; software assurance tools; source code analysis; vulnerability. 

Errata to this version: 

None 

Any commercial product mentioned is for information only. It does not imply recommendation or 

endorsement by NIST nor does it imply that the products mentioned are necessarily the best available for 

the purpose. 
NIST SP 500-268 May 2007 Page 3 of 14 

Table of Contents 

1 Introduction .............................................................................................................................................4

1.1 Purpose ...........................................................................................................................................4

1.2 Scope ..............................................................................................................................................4

1.3 Audience..........................................................................................................................................5

1.4 Technical Background.....................................................................................................................5

1.5 Glossary of Terms ...........................................................................................................................7

2 Functional Requirements........................................................................................................................8

2.1 High Level View...............................................................................................................................8

2.2 Requirements for Mandatory Features ...........................................................................................8

2.3 Requirements for Optional Features ...............................................................................................8

3 References..............................................................................................................................................9

Annex A Source Code Weaknesses .......................................................................................................10

Annex B Code Complexity Variations......................................................................................................13
NIST SP 500-268 May 2007 Page 4 of 14 

1 Introduction 

 

1.1 Purpose 

The National Institute of Standards and Technology (NIST) is working with the U.S. Department of 

Homeland Security National Cybersecurity Division to improve the state of the practice in software 

assurance. Through the development of tool functional specifications, test suites and tool metrics, the 

NIST Software Assurance Metrics and Tool Evaluation, or SAMATE, project aims to better quantify the 

state of the art for different classes of software security assurance tools. 

Source code security analysis tools scan a textual (human readable) version of source files that comprise 

a portion or all of an application program. These files may contain inadvertent or deliberate weaknesses 

that could lead to security vulnerability in the executable version of the application program. This 

document specifies a set of functional feature requirements for a source code security analysis tool or set 

of tools, including a list of common code weaknesses that account for many of today’s vulnerabilities. 

This specification, together with the corresponding test plan and test suite, serves as a guide to 

measuring the capability of source code security analysis tools against this set of weaknesses. There are 

many useful tools that do not attempt to identify all of the weaknesses listed in this specification. The goal 

of this specification is not to prescribe the features and functions that all source code security analysis 

tools must have. The goal is to identify code weaknesses that significantly affect the security of software 

applications today and provide a user of such tools with a way to measure if, and how well a tool, or 

combination of tools, identifies these particular weaknesses. 

We plan for the next version of this specification to include more weaknesses, explicitly apply to tools that 

only identify a subset of weaknesses, have more precise and substantiated metrics, and other 

enhancements. 

Use of a tool or toolkit that complies with this specification does not guarantee the code will be free of 

weaknesses. It does however provide a tool user with knowledge that their tool solution covers some of 

the most prevalent and highly exploitable security weaknesses. 

1.2 Scope 

This specification is limited to software tools that examine source code files for security weaknesses and 

potential vulnerabilities. Tools that scan other artifacts, like requirements, bytecode or binary code, and 

tools that dynamically execute code are outside the scope. Annex A of this document, Source Code 

Weaknesses, specifically addresses C, C++, and Java source code. 

We start with these three because they are the languages in which most of today’s vulnerabilities have 

been identified and on which most source code security analysis tools focus. These weaknesses may 

exist in other languages as well. 

As an aside, there are languages that are, by design, more suitable for secure programming. Such 

languages entirely preclude many common weaknesses and minimize or expose others. Choosing such 

languages mitigates many security risks. 

This document specifies minimum functionality only. Critical production tools should have capabilities far 

beyond those indicated here. Many important attributes, like compatibility with integrated development 

environments or IDEs and ease of use, are not addressed. 

The misuse or proper use of a tool is outside the scope of this specification. 

The issues and challenges in engineering secure systems and their software are outside the scope of this 

specification. 
NIST SP 500-268 May 2007 Page 5 of 14 

1.3 Audience 

The target audience for this specification is users and evaluators of source code security analysis tools. It 

may also be useful to source code security analysis and software assurance researchers, and developers 

of source code security analysis tools. 

1.4 Technical Background 

This section gives some technical background, defines terms we use in this specification, explains how 

concepts designated by those terms are related, and details some challenges in source code analysis for 

security assurance. 

The Role of Source Code Analysis in Software Assurance

No amount of analysis and patching can imbue software with high levels of security or quality or 

correctness or other important properties. Such properties must be designed in and built in. Good choice 

of language, platform, and discipline are worth orders of magnitude more than reactive efforts. 

Nevertheless testing or examination of code has benefits in some situations. 

For instance, to determine how different methods or processes affect the quality of the resultant code, the 

code can be examined. If the origin of code has limited visibility, testing or static analysis are the only 

ways to gain higher assurance. Existing, legacy code must be examined to assess its quality and 

determine what, if any, remediation is needed. 

Testing, or dynamic analysis, has the advantage of examining the behavior of software in operation. In 

contrast, only static analysis can be expected to find malicious trapdoors. Analysis of binary or executable 

code, including bytecode, avoids assumptions about compilation or source code semantics. Only the 

binary may be available for libraries or purchased software. However, source code security analysis can 

give developers feedback on better practices. 

Remediation is often done in source code. Analysis of higher-level constructs, such as models, designs, 

use cases, or requirements documents, is possible, too. However, these higher-level artifacts often lack 

rigor and rarely reflect all the critical detail in source code implementations. Thus static analysis of source 

code is a reasonable place to work for higher software assurance. 

Terms Used in This Specification

Often, different terms are used to refer to the same concept in software assurance and security literature. 

Different authors may use the same term to refer to different concepts. For the purposes of this 

document, the following terms and definitions apply. To begin, any event that is a violation of a particular 

system's explicit (or implicit) security policy is a security failure, or simply, failure. For example, if an 

unauthorized person gains "root" or "admin" privileges, security has failed. Again if unauthorized people 

can read Social Security numbers from your web site, security has failed. 

A vulnerability is a property of system security requirements, design, implementation, or operation that 

could be accidentally triggered or intentionally exploited and result in a security failure. (After [SP800-27]) 

In our model the source of any failure is a latent vulnerability. In other words, if there is a failure, there 

must have been a vulnerability. A vulnerability is the result of one or more weaknesses in requirements, 

design, implementation, or operation. 

In the unauthorized privileges example above, the combination of the two weaknesses of allowing weak 

passwords and of not locking out an account after repeated password mismatches allow the vulnerability. 

This vulnerability can be exploited by a brute force attack to cause the failure of an unauthorized person 

gaining elevated privileges. An SQL injection vulnerability might be exploited several different ways 

to produce different failures, such as dropping a table or revealing all its contents. If spyware can steal a 

user's password, it is a vulnerability. But it may be hard to attribute the vulnerability to a few lines of code 

that can be "fixed." Spyware typically exploits system weaknesses, which require changes at the system 

level. 

Sometimes a weakness can never result in a failure, in which case it is not exploitable and not a 

vulnerability. Such a weakness might be masked by another part of the software or might only cause a 
NIST SP 500-268 May 2007 Page 6 of 14 

failure in combination with another weakness. Thus we use the term "weakness" instead of "flaw" or 

"defect." 

For several reasons no tool can correctly determine in every conceivable case whether or not a piece of 

code has a vulnerability. First, a weakness may result in a vulnerability in one environment, but not in 

another. Second, Rice proved that no algorithm can correctly decide in every case whether or not a piece 

of code has a property, such as a weakness. Third, practical analysis algorithms have limits because of 

performance and intellectual investment. Some vulnerabilities can only be identified if a tool performs 

inter-file, inter-procedural, or flow-sensitive analysis of the code. Each different code complexity, such as 

fixed or variable loops, memory indexing nested within indexing, local vs. global scope, and others listed 

in Annex B, may require additional analytical capabilities. Deliberate obfuscation with convoluted code 

structures makes the analysis even harder. Fourth, a tool may not have "rules" to find all known 

vulnerabilities. Worse, new exploits are being invented and new vulnerabilities recognized all the time. 

Since no tool can be omniscient, a tool may be written to be cautious and report questionable constructs. 

Some of those reports may turn out to be false alarms or false positives. To reduce wasting users’ time on 

false alarms, a tool may be written to only report constructs that are (almost) certainly vulnerabilities. In 

this case it may miss some vulnerabilities. A missed vulnerability is called a false negative. A tool may do 

a more detailed analysis, which is computationally intensive, to reduce both false alarms and missed 

vulnerabilities. The ideal is a tool that reports all real vulnerabilities (no false negatives) with no false 

alarms. Although this is impossible even in theory, tools may use a combination of approaches to balance 

performance, false alarms, and missed vulnerabilities. Since a failure only takes one vulnerability, the 

requirements have a tone of catching all weaknesses. Practical considerations require the false positive 

rate [Fleiss] to be acceptably low for the domain. 

A tool may grade weaknesses according to severity, potential for exploit, certainty that they are 

vulnerabilities, etc. Ultimately people must analyze the tool’s report and the code then decide 

• which reported items are not true vulnerabilities, 

• which items are acceptable risks and will not be mitigated, and 

• which items to mitigate, and how to mitigate them. 

To save analysis time in later runs, some tools allow the user to suppress weakness instances so they 

are not reported again. 
NIST SP 500-268 May 2007 Page 7 of 14 

1.5 Glossary of Terms 

This glossary provides descriptions for terms used in this document. 

Name Description 

false negative When a tool does not report a weakness where one is present. If 

the tool does not claim to identify a certain class of weakness, not 

reporting a weakness of that class is not a false negative. 

false positive When a tool reports a weakness where no weakness is present. 

false positive rate The number of false positives divided by the sum of the number of 

false positives and the number of true positives. 

flow-sensitive analysis Analysis of a computer program that takes into account the flow of 

control. 

inter-file analysis Analysis of code residing in different files that have procedural, 

data, or other interdependencies. 

inter-procedural analysis Analysis between calling and called procedures within a computer 

program. 

security failure Any event that is a violation of a particular system's explicit or 

implicit security policy. 

security vulnerability A property of system requirements, design, implementation, or 

operation that could be accidentally triggered or intentionally 

exploited and result in a security failure. 

source code A series of statements written in a human-readable computer 

programming language. 

tainted input Input data that has not been examined or sanitized prior to use by 

an application. 

true positive When a tool reports a weakness where one is present. 

weakness A piece of code that may lead to a vulnerability. 

weakness suppression system A feature that permits the user to flag a line of code not to be 

reported by the tool in subsequent scans. 
NIST SP 500-268 May 2007 Page 8 of 14 

2 Functional Requirements 

2.1 High Level View 

Informally, what does a source code security analysis tool or tool set do? At a minimum the tool(s) should: 

• Identify a select set of classes of software security weaknesses in source code. 

• Report the security weaknesses that it identifies, what kind of weakness each one is, and where each 

one is located. 

• Not have many false positives. 

Optionally a tool should: 

• Produce a report compatible with other tools. 

• Allow weaknesses to be suppressed by the user. 

• Use standard names for weakness classes. 

 

2.2 Requirements for Mandatory Features 

To meet a minimum capability, a source code security analysis tool or set of tools must be able to 

accomplish the tasks described below. The tool(s) shall: 

SCSA-RM-1: Identify all of the classes of weaknesses listed in Annex A. 

SCSA-RM-2: Textually report any weaknesses that it identifies. 

SCSA-RM-3: For any identified weaknesses in classes listed in Annex A, report the class using a 

semantically equivalent name. 

SCSA-RM-4: Report the location of any weaknesses by providing the directory path, file name and line 

number. 

SCSA-RM-5: Identify weaknesses despite the presence of the coding complexities listed in Annex B. 

SCSA-RM-6: Have an acceptably low false positive rate. 

2.3 Requirements for Optional Features 

The following requirements apply to optional tool features. If the tool supports an optional feature, then 

the requirement for that feature applies, and the tool can be tested against it. A specific tool might 

optionally provide none, some, or all of the features described by these requirements. Optionally, the 

tool(s) shall: 

SCSA-RO-1: Produce an XML-formatted report. 

SCSA-RO-2: Not report a weakness instance that has been suppressed. 

SCSA-RO-3: Use the Common Weakness Enumeration [CWE] name of the weakness class it reports. 
NIST SP 500-268 May 2007 Page 9 of 14 

3 References 

[CVE] Common Vulnerability and Exposures, the MITRE Corporation, http://cve.mitre.org/

[CWE] Common Weakness Enumeration, the MITRE Corporation, http://cwe.mitre.org/

[Fleiss] Fleiss, J. L. (1981). Statistical Methods for Rates and Proportions, 2nd ed., John Wiley and Sons, 

New York, pp 4-8.

 [Kratkiewicz] Kratkiewicz, K. (2005). Evaluating Static Analysis Tools for Detecting 

Buffer Overflows in C Code, Master's Thesis, Harvard University, Cambridge, 

MA, 285 pages. http://www.ll.mit.edu/IST/pubs/KratkiewiczThesis.pdf

[SP800-27] Engineering Principles for Information Technology Security (A Baseline for Achieving 

Security), NIST SP 800-27, Revision A, June 2004. http://csrc.nist.gov/publications/nistpubs/

[XML] Extensible Markup Language, World Wide Web Consortium (W3C), http://www.w3.org/XML/ 
NIST SP 500-268 May 2007 Page 10 of 14 

Annex A Source Code Weaknesses 

The classes of source code weaknesses listed in this table represent a “base set” of code weaknesses. 

Criteria for selection of weaknesses include: 

• Found in existing code today – The weaknesses are found in existing software applications. 

• Recognized by tools today - Tools today are able to identify these weaknesses in source code 

and identify their associated file names and line numbers. 

• Likelihood of exploit is medium to high – The weakness is fairly easy for a malicious user to 

recognize and to exploit. 

For each weakness, this table contains the CWE name and identifier number, a short description, the 

relevant language(s), and the code complexities that apply. For other relevant information, such as 

likelihood of exploitation, instances in the common vulnerability and exposures list [CVE], background, 

consequences, and remediation, please see [CWE]. 

Name CWE ID Description Language(s) Relevant Complexities 

Input Validation 

Basic XSS 80 Unfiltered input is passed 

to a web application that in 

turn passes that data back 

to another client in the 

form of a malicious script. 

C,C++, Java taint, scope, address alias level, 

container, local control flow, loop 

structure, buffer address type 

Resource 

Injection 

99 Unfiltered input is used in 

an argument to a resource 

operation function. 

C, C++, Java taint, scope, address alias level, 

container, local control flow, loop 

structure, buffer address type 

OS Command 

Injection 

78 Unfiltered input is used in 

an argument to a system 

operation execution 

function. 

C, C++, Java taint, scope, address alias level, 

container, local control flow, loop 

structure, buffer address type 

SQL Injection 89 Unfiltered input is used in 

an argument to a SQL 

command calling function.

C, C++, Java taint, scope, address alias level, 

container, local control flow, loop 

structure, buffer address type 

Range Errors 

Stack overflow 121 Input is used in an 

argument to the creation 

or copying of blocks of 

data beyond the fixed 

memory boundary of a 

buffer on the stack. 

C, C++ All 

Heap overflow 122 Input is used in an 

argument to the creation 

or copying of blocks of 

data beyond the fixed 

memory boundary of a 

buffer in the heap portion 

of memory. 

C, C++ All 
NIST SP 500-268 May 2007 Page 11 of 14 

Format string 

vulnerability 

134 Unfiltered input is used in 

a string used to format 

data in the printf() style of 

C/C++ functions. 

C, C++ taint, scope, address alias level, 

container, local control flow, loop 

structure, buffer address type 

Improper Null 

Termination 

170 The software does not 

properly terminate a string 

or array with a null 

character or equivalent 

terminator 

C, C++ taint, scope, address alias level, 

container, local control flow, loop 

structure, buffer address type 

API Abuse 

Heap Inspection 244 Using realloc() to resize 

buffers that store sensitive 

information can leave the 

sensitive information 

exposed to attack because 

it is not removed from 

memory. 

C, C++ taint, scope, address alias level, 

container, local control flow, loop 

structure, buffer address type 

Often Misused: 

String 

Management 

251 Some string manipulation 

functions can be exploited 

through their input to 

produce buffer overflows. 

C, C++ taint, scope, address alias level, 

container, local control flow, loop 

structure, buffer address type 

Security Features 

Hard-Coded 

Password 

259 Hard-coded data is passed 

as an argument to a login 

function. 

C/C++, Java scope, address alias level, 

container, local control flow, loop 

structure, buffer address type 

Time and State 

Time-of-check 

Time-of-use race 

condition 

367 Between the time in which 

a given resource (or its 

reference) is checked, and 

the time that resource is 

used, a change occurs in 

the resource to invalidate 

the results of the check. 

C, C++, Java asynchronous 

Unchecked Error 

Condition 

391 No action is taken after an 

error or exception 

condition occurs. 

C, C++, Java none 

Code Quality 

Memory leak 401 Memory is allocated, but is 

not released after it has 

been used. 

C, C++ scope, address alias level, 

container, local control flow, loop 

structure 

Unrestricted 

Critical Resource 

Lock 

412 A resource is “deadlocked” 

by obtaining an exclusive 

lock or mutex, or modifying 

the permissions of a 

shared resource. 

C, C++, Java asynchronous 
NIST SP 500-268 May 2007 Page 12 of 14 

Double Free 415 An attempt is made to free 

memory using an address 

that has previously been 

used in a free () function 

call.

C, C++ scope, address alias level, 

container, local control flow, loop 

structure, buffer address type 

Use After Free 416 An attempt is made to 

access the same memory 

address previously 

released by a call to the 

free() function.

C, C++ scope, address alias level, 

container, local control flow, loop 

structure, buffer address type 

Uninitialized 

variable 

457 A variable is created 

without assigning it a 

value. It is subsequently 

referenced in the program, 

causing potential 

undefined behavior or 

denial or service.

C, C++ scope, address alias level, 

container, local control flow, loop 

structure 

Unintentional 

pointer scaling 

468 Improper mixing of pointer 

types in an expression 

may result in references to 

memory beyond that 

intended by the program. 

C, C++ data type 

Null Dereference 476 A pointer with a value of 

NULL is used as though it 

pointed to a valid memory 

area. 

C, C++ taint, scope, address alias level, 

container, local control flow, loop 

structure 

Encapsulation 

Leftover Debug 

Code 

489 Debug code can create 

unintended entry points in 

an application. 

C, C++, Java none 
NIST SP 500-268 May 2007 Page 13 of 14 

Annex B Code Complexity Variations 

To locate and identify source code weaknesses listed in Annex A, a source code security analysis tool 

must be able to find those weaknesses within the complex coding structures relevant to that weakness. A 

general list of these types of structures, adapted from [Kratkiewicz], is provided below. Some of the 

complexities are language specific (e.g. the use of pointers in C, C++), however, most exist in C, C++ and 

Java. Equivalent constructs in other languages will be added, as tools for those languages are addressed 

in this specification. 

Complexity Description Enumeration 

address alias level level of “indirection” of buffer alias 

using variable(s) containing the 

address 

1 or 2 

array address complexity level of complexity of the address 

value of an array buffer 

constant, variable, linear expression, 

nonlinear expression, function return 

value, array content value 

array index complexity level of complexity of the index 

value of an array buffer using 

variable assignment 

constant, variable, linear expression, 

nonlinear expression, function return 

value, array content value 

array length/limit complexity level of complexity of the index of 

an array buffer’s length or limit 

value 

constant, variable, linear expression, 

nonlinear expression, function return 

value, array content value 

asynchronous asynchronous coding construct threads, forked process, signal handler 

buffer address type method used to address buffer pointer, array index 

container containing data structure array, struct, union, array of structs, 

array of unions 

data type type of data read or written character,integer,floating point,wide 

character,pointer,unsigned 

character,unsigned integer 

index alias level level of buffer index alias 

indirection 

1 or 2 

local control flow type of control flow around 

weakness 

if,switch,cond,goto/label,setjmp,longjmp, 

function pointer, recursion 

loop complexity component of loop that is complex initialization, test, increment 

loop iteration type of loop iteration/termination fixed, indefinite 

loop structure type of loop construct in which 

weakness is embedded 

standard for,standard do while, standard 

while, non standard for, non standard do 

while, non standard while 

memory access type of memory access related to 

weakness 

read, write 

memory location type of memory location related to 

weakness 

heap, stack, data region, BSS, shared 

memory 
NIST SP 500-268 May 2007 Page 14 of 14 

scope scope of control flow related to 

weakness 

local, within-file/inter-procedural, withinfile/global, inter-file/inter-procedural, 

inter-file/global, inter-class 

taint type of tainting to input data argc/argv, environment variables, file or 

stdin, socket, process environment