# Presentation: 021009

## Techniques for Validating the Security Quality of Infrastructure Software

- John D. McGregor
- johnmc@cs.clemson.edu

## Outline

- Motivation
- Proposed strategy
- Detailed actions
- Conclusion

## Theme – Life Cycle Threats

- There is not a means for automated testing of large software, both static and mobile code, to detect, identify malicious code, sleeper codes, and exploitable vulnerabilities and to determine and understand the potential impact on the life-cycle of the codes. Current testing approaches are largely manual rather than automated.
**CSIIR Workshop Themes document**

## A Recommended Strategy

- Software security vulnerabilities are often caused by defective specification, design, and implementation.
- ...require that software be designed with security at the very heart of the design process...
- Establish a security verification and validation program to evaluate different software development processes and practices for effectiveness in producing  secure software.
- Certify those processes demonstrated to be effective for producing secure software.
-      Security Across the Software Development Lifecycle Task Force

## Recommended Practices

**Statistical testing** - Usage based testing permits valid statistical estimation of quality with respect to all the executions not tested and tends to find any remaining high-failure-rate defects early.
**Production testing** - Two strategies are testing security functionality with standard functional testing techniques, and risk-based security testing based on attack patterns and threat models. A good security test plan (with traceability back to requirements) uses both strategies.
**Process models** - organizations can use the goals and attributes defined in process models as high-level guides for defining and improving their management and engineering processes in the ways they feel are most appropriate for them.
-     Security Across the Software Development Lifecycle Task Force

## SEI 

- The security of a software-intensive system is directly related to the quality of its software1.
- Over 90% of software security incidents are caused by attackers exploiting known software defects.
- Analysis of 45 e-business applications showed that 70% of security defects were design defects.
- Experienced and capable software engineers inject, on average, one defect every nine lines of code.
- A one million line of code systems typically contains 1,000-5,000 defects when shipped.
- 1 http://www.sei.cmu.edu/tsp/tsp-security.html

## A Final Source

- “One of the key things that developers can do to help secure their systems is to write code that can withstand attack and use security features properly.”
- Defend Your Code with Top Ten Security Tips Every Developer Must Know
- 8 out of 10 tips are directly programming issues
- _[http://msdn.microsoft.com/security/securecode/default.aspx?pull=/msdnmag/issues/02/09/securitytips/default.aspx](http://msdn.microsoft.com/security/securecode/default.aspx?pull=/msdnmag/issues/02/09/securitytips/default.aspx)_

## Basic Assumptions

- Testing early is cheaper than testing later.
- System test of a system with j modules and k experiments (iterations or different implementations for a given interface) results in the following cost of testing

- Where ct is the cost per test.
- Module-level tests with j modules and k experiments results in

- Baldwin and Clark. Design Rules The Power of Modularity, volume 1, MIT Press, 2000.

## Basic assumptions - 2

|  | Design | Design Inspection | Code | Development Test | Acceptance | Production |
| --- | --- | --- | --- | --- | --- | --- |
| IBM | 1 |  | 1.5 | 60 |  | 100 |
| Boehm | 1 |  |  | 15 to 40 | 30 to 70 | 40 to 1000 |
| Remus | 1 |  | 20 | 82 |  |  |
| Ackerman |  | 1 |  | 2 to 10 |  |  |
| Russell |  | 1 |  | 2 to 4 |  | 30 |

- Though estimates vary, the cost of removing defect increases dramatically later in the life cycle.

## Premise

- Our premise is that poorly written software will contain more vulnerabilities than well written software where the security quality attribute is a design driver.
- Current views of security often take a defensive approach. Some of the security infrastructure even adds to the security risk due to the complexity it adds to the product.
- We propose an offensive approach in which security is a key design driver and a priority throughout the development process.

## Context

- A chain of quality should be threaded through the entire process so that validation is most effective and efficient.

## A Proposed Strategy

- Develop method engineering tactics and guidelines that enhance the security quality of the software through improved processes.
- Structure architecture evaluation techniques to focus on security by searching for static security patterns.
- Discover and capture test patterns that correspond to dynamic security patterns.
- Develop focused test techniques to effectively explore security test patterns while reducing the test suite size.
- Create a defect model for security that can be used to predict types and number of security vulnerabilities in scientific codes.

## Action – Develop method engineering techniques

- Method engineers create custom-made processes to help a project achieve specific goals.
- The goal of being “secure” needs to be operationalized so that these engineers can assemble methods in ways that enhance the security of the software product built using their processes.
- This task would involve extending the Software Process Engineering MetaModel1 (SPEM) standard to define security-specific constructs. The model would be automated using existing tools such as MetaEdit+.
- 1 Software Process Engineering Metamodel, version 1.1, Object Management Group (OMG), 2005.

## Action – Develop method engineering techniques - 2

- The security-oriented method fragments would be suitable to integrate into processes defined in the context of the SPEM.
- Process audits could evaluate the strength of the security aspect of the process, once it is explicitly embedded in the process, just as other qualities are validated.
- Deliverables: A process definition guide that would show how to design security-centric development process fragments, an assembly guide, and an example process.

## Action – Create architecture security analysis

- Architects have techniques that they apply to an architecture in order to improve the behavior of the architecture with respect to classes of security threats1.
- Assuring system survivability requires showing that the system architecture is adequately resilient to likely patterns of attack2.
- One approach to architecture design is to identify quality attributes for the architecture and make design decisions that enhance the desirable qualities and degrade the less desirable ones.
- Architecture evaluation techniques such as the architecture trade-off analysis method (ATAM) can be used to focus architecture evaluations on the security quality through security-specific scenarios.
- One aspect of this task would be to develop a set of architecture-level security scenarios that would guide the evaluation of an architecture for the security quality.
- 1 Security and Survivability Reasoning Frameworks and Architectural Design Tactics. CMU/SEI-2004-TN-022
- 2 Architectural Refinement for the Design of Survivable Systems** **CMU/SEI-2001-TN-008

## Action – Create architecture security analysis - 2

- This evaluation can be automated if the architects use a formal architecture description language (ADL).
- Our current implementation uses the Architectural Analysis and Design Language (AADL) from the Society of Automotive Engineers (SAE) and Eclipse plug-ins.
- A second part of the task could be to identify standard mechanisms for representing security in an architecture description so that security patterns could be automatically recognized.
- Deliverables: A set of validation scenarios and specific architecture tactics for recognizing security vulnerabilities at the architecture level

## Current implementation as Eclipse plug-in for AADL

## Action – Create architecture security analysis - 3

- This analysis could also be conducted on pre/post-test basis
- Use an architecture re-construction technique such as OAR1 to extract the “as built” architecture from the implementation.
- Compare to the “as designed” architecture to determine if changes have occurred
**1**** **Options Analysis for Reengineering (OAR): A Method for Mining Legacy Assets CMU/SEI-2001-TN-013

## Action – Discover test patterns for security

- Test patterns are solutions to testing problems in context. They often correspond to design patterns.
- Test patterns are language independent
- Content of a test pattern1
- Problem - Description of pattern to be tested
- Context - Special testing conditions
- Forces - What types of faults are we looking for?
- Solution - Test case selection strategy that tests the interactions among the components that implement the pattern
- Example – A sample implementation
- 1 John McGregor and David Sykes Practical Guide to Testing Object-Oriented Software, Addison-Wesley, 2001.

## Action – Discover test patterns for security - 2

- Problem: The synchronous communication between two objects is modified to be asynchronous by adding a callback object. How should this be tested?

- Client

- Server

- Client

- Server

- Callback

## Action – Discover test patterns for security - 3

- Context
- An intermediate object forwards messages
- Forces
- Possible to intermingle successive messages/responses
- Temporal considerations must be added to the test case
- Solution
- Construct tests that exercise the callback in a variety of states
- Construct multiple clients, submit multiple requests through multiple callback objects
- Build test cases that submit a second  message prior to receiving the response from the first message

## Action – Discover test patterns for security - 4

- In this task, security vulnerabilities and standard designs to correct them would be studied.
- The standard designs would lead to a catalog of specific test patterns.
- Users of the test patterns could implement them in whatever language was in use.
- Deliverables: test pattern catalog

## Task force identified security patterns

- 1. Make the Client Invisible
- 2. Target Programs That Write to Privileged OS Resources
- 3. Use a User-Supplied Configuration File to Run Commands That Elevate Privilege
- 4. Make Use of Configuration File Search Paths
- 5. Direct Access to Executable Files
- 6. Embedding Scripts within Scripts
- 7. Leverage Executable Code in Non-executable Files
- 8. Argument Injection
- 9. Command Delimiters
- 10. Multiple Parsers and Double Escapes

- 11. User-Supplied Variable Passed to File System Calls
- 12. Postfix NULL Terminator
- 13. Postfix, Null Terminate, and Backslash
- 14. Relative Path Traversal
- 15. Client-Controlled Environment Variables
- 16. User-Supplied Global Variables (DEBUG=1, PHP Globals, and So Forth)
- 17. Session ID, Resource ID, and Blind Trust
- 18. Analog In-Band Switching Signals (aka “Blue Boxing”)
- 19. Attack Pattern Fragment: Manipulating Terminal Devices
- 20. Simple Script Injection

- 49 security patterns have been identified. Here are a few.

- Security Across the Software Development Lifecycle Task Force

## Action – Develop focused test techniques - 1

- Operational profiles establish a relative frequency of operations for a particular type of user or application.
- Test cases are chosen with the same relative frequency to mimic actual use or to mimic attack scenarios.
- This narrows the range of values to cover in test.

| User | Operation | Relative frequency |
| --- | --- | --- |
| 
User1 | Op1 | .2 |
|  | Op2 | .6 |
|  | Op3 | .2 |
| 
User2 | Op1 | .4 |
|  | Op2 | .3 |
|  | Op3 | .3 |

## Action – Develop focused test techniques - 2

- Testing all possible values even once, much less all possible combinations of values, is usually impossible.
- Combinatorial test designs can systematically sample test values to ensure the maximum coverage with minimum test cases.
- Orthogonal array testing is one combinatorial approach.

## OATS tool - 1

- Three variables, each with three possible values, would require 27 test cases for all combinations coverage.

| Database driver | JDBC1.0 | JDBC2.0 | ODBC1.0 |
| --- | --- | --- | --- |
| browser | Netscape | IE5.0 | Firefox |
| # of web servers | 5 | 50 | 200 |

## OATS tool - 2

- OATS reduces this to 9 test cases, testing pair-wise but remains approximately 96% as effective as “all combinations” coverage at finding defects.

| 1 | JDBC1.0 | Netscape | 200 |
| --- | --- | --- | --- |
| 2 | JDBC1.0 | IE5.0 | 50 |
| 3 | JDBC1.0 | FoxFire | 5 |
| 4 | JDBC2.0 | Netscape | 50 |
| 5 | JDBC2.0 | IE5.0 | 5 |
| 6 | JDBC2.0 | FoxFire | 200 |
| 7 | ODBC1.0 | Netscape | 5 |
| 8 | ODBC1.0 | IE5.0 | 200 |
| 9 | ODBC1.0 | FoxFire | 50 |

## Action – Develop focused test techniques - 3

- Pair-wise value combinations can reduce the size of the test suite dramatically
- These techniques have been applied in domains such as telecommunications.
- In this task we would develop an industrial strength tool that allows the design of combinatorial test suites.

## Action – Develop focused test techniques - 4

- This technique is used in conjunction with traditional functional and structural strategies for test case selection.
- Functional tests ensure that all specified functions are correct and structural tests ensure that all feasible paths are valid.
- The result is confidence that the software does everything it is supposed to and nothing that it Is not supposed to.
- Combinatorial techniques increase the coverage of large software products without significant increases in test effort.
- This approach allows for risk-based test selections.
- Deliverables: software tool that uses combinatorial techniques and risk-based techniques to define effective test cases.

## Action – Develop a security defect model for scientific codes

- If you just found the 73rd defect in your 50,000 LOC program, do you feel good about it?
- With a validated defect model you would at least know how to feel.
- You would know approximately how many defects to expect and what types of defects to look for.

## Action – Develop a security defect model for scientific codes - 2

- Using the relative frequency and estimated size, estimates can be made of the number of defects and the test case selection process be more precisely directed.

| Security defect type | Relative frequency
per KLOC |
| --- | --- |
| Static queue sizes | 60% |
| HTTP cookies | 30% |
| Argument injection | 10% |
|  |  |
| Total | 100% |

## Action – Develop a security defect model for scientific codes - 3

- In this task we would develop a defect model framework for scientific codes using historic data.
- The framework would be packaged with a specialization method for tailoring the framework to a specific development context.
- This would allow us to provide statistical estimates of the number of defects remaining in a software product and to provide a confidence interval on statistics such as reliability.
- Deliverables: defect model

## Conclusion

- No single action or algorithm will produce secure software. There must be a chain of quality activities.
- The outlined strategy places security quality gates at several places in the software product development life cycle.

## Conclusion - 2

- Implementing any part of this strategy will improve the security of the software products produced.
- Develop method engineering tactics and guidelines that enhance the security quality of the software through improved processes.
- Structure architecture evaluation techniques to focus on security by searching for static security patterns.
- Discover and capture test patterns that correspond to dynamic security patterns.
- Instrument combinatorial test techniques to effectively explore security test patterns while reducing the test suite size.
- Create a defect model for security that can be used to predict types and number of security vulnerabilities in a given product.