# Presentation: 224079

## Technology Infusion of CodeSonar into the Space Network Ground Segment (RII07) 

- Software Assurance Symposium
- Technical Summary

## Problem Context (1)

- Ongoing maintenance by the sustaining engineering group has limited personnel and 		test resources but must maintain 			99.9% proficiency and 97.0% availability of systems
- Major new capabilities fielded for the Space Network are often contracted to entities outside of the sustaining engineering group and historically have higher than required defect rates

**Notes:**
Two recently fielded systems have undergone significant reengineering efforts post delivery to reduce defect rates to acceptable levels.  These are the targets of this study.  Features and fixes are added by sustaining engineering but new systems are developed elsewhere.

(sch – loss) / sch = prof
(alltime – downtime) / alltime = avail

## Problem Context (2)

- CSCI A and CSCI B account for 42% of the previous eight months software data loss

- Percentages reflect the portion of losses attributed to software with respect to the total loss within the control of the Space Network.  For example, if the annual average loss attributable to the Space Network was 500 out of 130k total hours (99.996% proficiency), then the portion caused by software would be 500 x 0.28 = 140 hours.

- Software has historically accounted for an annual average of 28% of the Space Network loss of availability and proficiency (low of 11% and high of 57% annually)

**Notes:**
Annual average in 2006 and 2007 was under 20% for software, which can be attributed to a combination of software process improvements and aging hardware 
Absolute loss values for software have been lower in the last two years than the historical average which can be partially attributed to process improvements but also may be due to hardware reaching the right side of the “bathtub” curve.

## Problem Context (3)

- CSCI A and CSCI B have discrepancies an order of magnitude larger than other CSCIs
- CSCI A was deployed several years ago
- CSCI B was deployed eight months ago
- These data represent the past eight months

**Notes:**
DRs are kept that record problems as well as amount of data loss.  Availability loss is only counted if both systems in a redundant pair are out of service.  For example, a DR will be written if a failover or reboot must be performed but no loss number will be attributed if service is available.

DR = discrepancy report

## Problem Context (4)

- CSCI A represented approximately 15% of all scheduled support time over the past eight months but accounted for 37% of all loss
- CSCI B has not had schedule data long
- 	enough to trend schedule time versus loss but 	had delayed deployment due to problems 	found during system and acceptance testing
- The remainder of the Space Network systems accounted for 58% of the data loss while providing at least 75% of scheduled support

**Notes:**
Note that the ratio of problems is inverse: 37% of total loss in only 15% support versus 58% total loss in 75% support.

## Analysis Results (1)

- Running CodeSonar and performing preliminary review of the results averaged 3.5 minutes per finding (approx. 20 hours total)
- An additional 40 hours is estimated
- 	to analyze the 37 findings deemed too complex for the initial review
- Using CodeSonar’s tools to suppress
- 	known non-problems, delta tool runs will not repeat findings marked as non-problems, further reducing the time needed for review

**Notes:**
Search and filtering makes future runs of the tool much more streamlined.

## Analysis Results (2)

- Of the 330 total findings, 70 were associated with the Standard Template Library (STL) and 109 had duplicate sources to other findings (fixing one either fixes the duplicate problems or provides a straightforward method to fix the duplicate problems)

**Notes:**
TP here are those findings deemed worth fixing or more investigation.  Complexity implies code may at least need re-factored.  (D) implies definite true positives [U + R] and (P) potential [U + R + N].

U = urgent (causes data loss or processor reset)
R = less than urgent but still problematic
N = not investigated (but will be)

## Analysis Results (3)

- Of the remaining 151 findings, 29 are findings that could potentially be problems but the coding style and standards used prevent the problem
- An addition 29 are definite false positives, which were determined to not be problems upon review

## Analysis Results (4)

- Of the final 93 findings, 37 require more review because of the complexity of the code in where the potential defect exists, 5 are considered urgent (will cause data corruption or processor reset) and 37 are considered routine (worth correcting as time is available)

**Notes:**
Complex items are those dealing with “deep” data flows (data passed between many functions), potential issues with socket states, multiple nested conditions that could cause an uninitialized variable to be used.  Many of the complex values exist in infrequent corner cases that are likely to have very rare impacts on operations but ones that could be very difficult to uncover.

## Findings Comparisons (1)

- 	Data has just recently started being collected on the amount of time required to investigate DRs and provide a complete fix, so no trends can yet be provided regarding time and dollars expended on defects

## Findings Comparisons (2)

- The average data loss per software discrepancy is from 2003 to 2007 was 0.331 hours
- The average data loss per software discrepancy over the past eight months is 0.579 which may be attributable to additional users of CSCI A in addition to fielding CSCI B
- A conservative estimate of savings of data loss by correcting the 5 urgent findings is 2.895 hours of data loss, which is 11.5% of the estimated annual software data loss for 2008

**Notes:**
This is conservative because difficult to resolve problems cause larger loss times.

## Findings Comparisons (3)

- Jet Propulsion Laboratory (JPL) performed a case study that found 1.2 fielded defects / LKSLOC in ground software systems they developed1
- Capers Jones found that CMMI Level 5 organizations deliver 1.05 defects / KLOC2
- Of the 12 CSCs analyzed in this study, all fall below both these defect rates except two with respect to the true positive findings

*Credits:**	**1 – Spagnuolo, J. and Powell ,J. Defect Measurement and Analysis of JPL Ground Software: A Case Study. 2004.*
*	**2 – Jones, C. Software Assessments, Benchmarks, and Best Practices. 2000.*

**Notes:**
TP / KSLOC versus 1.05 defects / KLOC versus 1.2 defects / LKSLOC are not exact comparisons but give the best available

Tool findings cannot account for all latent defects in the CSCIs but are a basis of comparison

## Findings Comparisons (4)

- The Coverity project to use static analysis technology on Open Source software produced an average of 0.605 findings / KSLOC over 96 projects using a limited set of the most easy to understand defects
- CodeSonar produced an average of 0.730 findings that were confirmed to be valid, which implies that improvements should be made in the overall quality of CSCI A and CSCI B

**Notes:**
Coverity versus CodeSonar findings are not necessarily equal, of course.
Open Source software often can be lived without but CSCI A and CSCI B are mission critical software

## Acceptance Measures (1)

- 	Measures are meant to aid in determining the impact of the technology on the project both in the expenditure of effort and the technical results of the technology

*Average time to correct defect*

*Average data loss per defect*

**Notes:**
These are all example data...nothing real here.

Average time to correct starting to be collected

## Acceptance Measures (2)

- After relatively small initial investment for the initial tool run (compared to anecdotal comparisons to discrepancy investigation time), an estimated 10 minutes per KSLOC developed of extra time is anticipated based on preliminary use of the tool (without formal training)

## Acceptance Measures (3)

- Integration of tool into the build process may also provide further savings by preventing developers from having to configure and operate the tool separately

## Conclusions (1)

- Preliminary results show the tool to be easy to use and incorporate into engineering processes
- Preliminary findings give significant potential improvements in proficiency and availability on the part of software

## Conclusions (2)

- As time-to-fix data becomes available a better cost trade can be made on person hours saved versus tool cost
- Selective factors may be necessary to determine where best to apply CodeSonar to balance cost and benefits