May 2011 

NASA/CR–2011-217150

Concept Development for Software Health 

Management

Jung Riecks, Walter Storm, and Mark Hollingsworth 

Lockheed Martin Aeronautics Company, Fort Worth, Texas 
NASA STI Program . . . in Profile

Since its founding, NASA has been dedicated to 

the advancement of aeronautics and space science. 

The NASA scientific and technical information (STI) 

program plays a key part in helping NASA maintain 

this important role. 

 The NASA STI program operates under the 

auspices of the Agency Chief Information Officer. It 

collects, organizes, provides for archiving, and 

disseminates NASA’s STI. The NASA STI program 

provides access to the NASA Aeronautics and Space 

Database and its public interface, the NASA Technical 

Report Server, thus providing one of the largest 

collections of aeronautical and space science STI in 

the world. Results are published in both non-NASA 

channels and by NASA in the NASA STI Report 

Series, which includes the following report types: 

TECHNICAL PUBLICATION. Reports of 

completed research or a major significant phase 

of research that present the results of NASA 

programs and include extensive data or 

theoretical analysis. Includes compilations of 

significant scientific and technical data and 

information deemed to be of continuing 

reference value. NASA counterpart of peerreviewed formal professional papers, but having 

less stringent limitations on manuscript length 

and extent of graphic presentations. 

TECHNICAL MEMORANDUM. Scientific 

and technical findings that are preliminary or of 

specialized interest, e.g., quick release reports, 

working papers, and bibliographies that contain 

minimal annotation. Does not contain extensive 

analysis. 

CONTRACTOR REPORT. Scientific and 

technical findings by NASA-sponsored 

contractors and grantees. 

CONFERENCE PUBLICATION. Collected 

papers from scientific and technical 

conferences, symposia, seminars, or other 

meetings sponsored or co-sponsored by NASA. 

SPECIAL PUBLICATION. Scientific, 

technical, or historical information from NASA 

programs, projects, and missions, often 

concerned with subjects having substantial 

public interest. 

TECHNICAL TRANSLATION. Englishlanguage translations of foreign scientific and 

technical material pertinent to NASA’s mission. 

 Specialized services also include creating custom 

thesauri, building customized databases, and 

organizing and publishing research results. 

 For more information about the NASA STI 

program, see the following: 

Access the NASA STI program home page at 

http://www.sti.nasa.gov

E-mail your question via the Internet to 

help@sti.nasa.gov

Fax your question to the NASA STI Help Desk 

at 443-757-5803 

Phone the NASA STI Help Desk at 

443-757-5802 

Write to: 

 NASA STI Help Desk 

 NASA Center for AeroSpace Information 

 7115 Standard Drive 

 Hanover, MD 21076-1320
National Aeronautics and 

Space Administration 

Langley Research Center Prepared for Langley Research Center 

Hampton, Virginia 23681-2199 under Contract NNL06AA08B 

 

May 2011 

NASA/CR–2011-217150

Concept Development for Software Health 

Management

Jung Riecks, Walter Storm, and Mark Hollingsworth 

Lockheed Martin Aeronautics Company, Forth Worth, Texas 
Available from: 

NASA Center for AeroSpace Information 

7115 Standard Drive 

Hanover, MD 21076-1320 

443-757-5802 

The use of trademarks or names of manufacturers in this report is for accurate reporting and does not 

constitute an official endorsement, either expressed or implied, of such products or manufacturers by the 

National Aeronautics and Space Administration. 
CONTENTS

Foreword.......................................................................................................................................................................7

Introduction...................................................................................................................................................................8

Approach...................................................................................................................................................................8

PreparingtheData.................................................................................................................................................8

ClassificationDetails..............................................................................................................................................9

CreatingtheBaseline.............................................................................................................................................9

CreatingtheFailureTaxonomy..............................................................................................................................9

AnalysisResults...........................................................................................................................................................10

FailureClasses..........................................................................................................................................................10

Algorithm.............................................................................................................................................................10

BusInterface........................................................................................................................................................11

ConfigurationManagement(CM)........................................................................................................................11

CompilerError.....................................................................................................................................................12

DataDefinition.....................................................................................................................................................12

DataHandling......................................................................................................................................................12

Documentation....................................................................................................................................................13

Hardware.............................................................................................................................................................13

InputOutput(I/O)System...................................................................................................................................13

Implementation...................................................................................................................................................14

InterprocessCommunication.............................................................................................................................14

Performance........................................................................................................................................................14

SelfTest...............................................................................................................................................................15

SystemIntegration...............................................................................................................................................15

Tools....................................................................................................................................................................16

User/Pilot.............................................................................................................................................................16

ErrorAnalysis...........................................................................................................................................................17

Background..........................................................................................................................................................17
Page |6

TheRiskPriorityNumber.....................................................................................................................................17

DetailedClassAnalysis.........................................................................................................................................19

RPNComponentAnalysis....................................................................................................................................20

BusInterfaceErrorClassProfile..........................................................................................................................23

ConfigurationManagementErrorClassProfile...................................................................................................24

DataDefinitionErrorClassProfile.......................................................................................................................24

DataHandlingErrorClassProfile.........................................................................................................................25

InterProcessCommunicationErrorClassProfile................................................................................................25

Input/OutputSystemErrorClassProfile.............................................................................................................26

SelfTestErrorProfile...........................................................................................................................................26

SystemIntegrationErrorClassProfile.................................................................................................................27

RootFailureCauseandEffectRelationshipAnalysis...............................................................................................27

Background..........................................................................................................................................................27

GroundRules.......................................................................................................................................................28

OverviewofRootFailureCauseandEffectRelationshipChart...........................................................................28

DocumentationandExternalProblemsCategory...............................................................................................29

RequirementsCategory.......................................................................................................................................30

ConfigurationManagementCategory.................................................................................................................31

AlgorithmCategory..............................................................................................................................................31

SystemIntegration/CommunicationCategory..................................................................................................34

SelfTestCategory................................................................................................................................................36

ApplicationofDataAnalysisResultstoEvaluatingFutureTechnologies............................................................36

References...................................................................................................................................................................39
Page |7

FOREWORD

LockheedMartinCorporation,actingthroughitsLockheedMartinAeronauticsCompany(LMAero)operatingunit,

haspreparedthisdocumentfortheNationalAeronauticsandSpaceAdministration’s(NASA)LangleyResearch

CenterundercontractNNL06AA08B,deliveryordernumber:NNL07AB06T.Theworkdocumentedhereinwas

performedfromOctober,2008throughJuly,2009.

ContributorsincludedJungRiecks,WalterStorm,andMarkHollingsworth.Additionalsupportwasprovidedby:

ClaudiaMarshall,DanHarbour,DianeNixon,andTomSchech.

Page |8

INTRODUCTION

ThisreportdocumentstheworkperformedbyLockheedMartinAeronautics(LMAero)underNASAcontract

NNL06AA08B,deliveryorderNNL07AB06T.TheConceptDevelopmentforSoftwareHealthManagement(CD

SHM)programwasaNASAfundedeffortsponsoredbytheIntegratedVehicleHealthManagementProject,oneof

thefourpillarsoftheNASAAviationSafetyProgram.TheCDSHMprogramfocusedondefiningastructured

approachtosoftwarehealthmanagement(SHM)throughthedevelopmentofacomprehensivefailuretaxonomy

thatisusedtocharacterizethefundamentalfailuremodesofsafetycriticalsoftware.

ToenablethedetectionandmitigationofsoftwareerrorsthroughSHM,ourapproachistotreatsoftwareas

anothersystemdevicethatexhibitsfailuremodesaccordingtoacanonicalfailurereferenceoflegacyand

emergingsafetycriticalsoftware.ManySHMconceptsstemfromfailuremodesandeffectsanalysis(FMEA)of

softwareinamannersimilartothatusedforhardware,howeverthefailuremodesforsoftwarearenotwell

known,andthetechniquesforapplyingasoftwareFMEAduringsystemdesignarenotwidelypublished[1],[2].

Ourgoalwastoaddresstheseshortcomingsbyquantifyingthescope,magnitudeandtypesoffundamental

softwareerrorsthatmanifestthemselvesthroughoutthedevelopmentofadvancedflightcriticalsoftware.We

developedourapproachintwophases:1)thecreationofataxonomyforfundamentalsoftwareanomaliesbased

ondatafromvariousadvanced,flightcriticalsoftwaredevelopmentprograms;and2)thedevelopmentof

integratedriskmodels,mitigationschemes,designconsiderationsandpatternsbasedonfundamentalfailuredata.

Thefollowingsectionsdocumenttheprocessandresultsofthestudy.

APPROACH

PREPARINGTHEDATA

Thesourceofourstudywasthedevelopmentofflightcriticalsoftwaresystemsfromacombinationofseveral

recent,advanceddevelopmentandproductionprograms.Thebackgroundinformationrequiredforthe

investigationandanalysiswasgatheredfromacrossvariousdatabasesystemsandnormalizedtoacommon

database.Weusedtheresultingdatabaseasthesourceforourerrorclassificationandtaxonomydevelopment.

Theanalysisofthedatabasewasperformedmanually,asseveralsubjectmatterexpertsreadthroughand

classifiedeachanomalyreportasatypeoffundamentalfailure.Thefailuretypesweredevelopedafterseveral

passesthroughthedata,wheretherootcausesweredistilledtobasicphrasesortermsthatadequatelydescribe

andclassifytheirnature.Onlythosetermswhichadequatelydescribedatleast0.1%ofallthecasesstudiedwere

consideredaneligibletermforthefundamentalfailuretype.Page |9

CLASSIFICATIONDETAILS

Asitturnsout,alloftherawdatasourcesforthisanalysisare(moreorless)freeformtext.Fromthis,itwas

quicklyevidentthattheonlywaytoproduceacomprehensivetaxonomywastoreadeachaccountindividually.

Weheldmanymeetingswithourprogramcontactstostudythecurrentanomalyreportstructures.Inthecurrent

anomalyreportstructure,thereisamultitudeofinformation;howeverthereisnoeasywaytooutlinethecause

classificationorrootcauseindetail.Nonetheless,weidentifiedareasthatstillgaveussomeadvantages.Usingthe

currentreportingsystem,wewereabletoidentifytheanomalyfound,thephaseinwhichitwasintroducedandits

severity.Thisinformationisthefoundationofourstudyandthebasisforourrecommendations.

CREATINGTHEBASELINE

Thefirststepincreatingthebaselinedatasetinvolvedeliminatingalloftheunnecessaryinformationfromtheraw

reports,andboilingthemdowntothefundamentalsymptoms,phases,severities,androotcauses.Thesteps

involvedinthedataeliminationprocesswere:

1. Deletealltheblanksections

2. Deleteunimportantsectionsforthisproject.(i.e.UserID,date,...etc)

3. Delete‘cancelled’or‘analysis’instatus

4. Delete‘external’,‘duplicate’,‘notaproblem’,‘suspended’infinalresolution

5. Delete‘No’inconfirmedproblem

6. Deleteallthedatawhichisnotasoftwarerelatedprobleminproblemproduct

Afterthispurging,theresultantdatabasewasthebaselinefortheproject.

CREATINGTHEFAILURETAXONOMY

Therearefourdifferentsectionsfromtheanomalyreportsthatwereceivefromanygivenprogram.These

sectionsarethe:AnomalyBehavior;ExpectedBehavior;RootCauseandCorrectiveActionTask.Allofthese

sectionshaveadescriptionfieldthatisfreeformattextwhichcontainsalimitof2,000characters.Fromthefour

sectionsabove,wecreatesectionsthatarenamed:Anomaly;CauseClassificationandRootFailure.

1. The“Anomaly”containsaveryshortdescriptionoftheproblembehavior.The“anomaly”comesfromthe

“AnomalyBehavior”and“ExpectedBehavior”sectionsfromtheoriginalreport.

2. The“CauseClassification”istheclassificationandabstractionofthefailure.The“CauseClassification”

informationcomesfromthe“rootcause”and“correctiveactiontask”sectionoftheanomalyreports.

3. The“RootFailure”isthetaxonomyoffailures.The“RootFailure”informationalsocomesfromthe“Root

Cause”and“CorrectiveActionTask”sectionoftheanomalyreports.Page |10

SincewedonothaveanoutlineoftheCauseClassificationandRootFailure,wefirststartedwithasamplegroup

ofanomalyreportstoattempttoidentifyapatternofCauseClassificationandRootFailure.Whilewewere

workingonthissamplegroup,werealizedthattheanomalyreportsarenotalargeenoughsamplegroupto

discernapatternofcauseclassificationandrootfailure.Wedecidedthatweneededtoreviewalloftheanomaly

reportstocreatetheinitialoutlineofCauseclassificationandRootFailure.Theanomalyreportdatacontainsall

thelifecycleoftheprogram.Afterexaminingseveralhundredanomalyreports,westartedtoseesomepatterns.

ThepatternsenabledustokeepasmuchdetailaspossiblewithrespecttotheCauseClassificationandRoot

Failurewhilestillallowingenoughentriestobestatisticallysignificant.Thisanalysiswasthenrefinedintothefinal

taxonomydescribedinthefollowingsection.

ANALYSISRESULTS

Ourtaxonomyconsistsof16failureclassesand114fundamentalfailuretypes.Inordertodefineaspecificfailure

type,thetypemustprovidestatisticalsignificanceforthetermbyadequatelydefiningatleast0.1%ofallanomaly

reportsstudied.Eachclassandthefundamentaltypesderivedfromthemaredescribedinthefollowingsections.

FAILURECLASSES

ALGORITHM

TheAlgorithmfailureclassdefinesafamilyof31softwareerrorsthatrepresent,ingeneralterms,fundamental

errorsinthesoftwaredesign.Forexample,errorssuchasinvalidassumptionsabouttheenvironmentinwhichthe

systemoperatesmaybeconsideredAlgorithmerrors.

AlgorithmFailureClass

FailureTypeDefinition

compoundlogicincorrectcompoundlogic(i.e.and,or,nand,nor...)

datatransfer/messageincorrectalgorithmofdatatransferring(refresh)

deadcodeleftovercodeformpastcausesaproblem

decisionlogicincorrectdecisionlogic(i.e.ifthenelse,casestatements,beginend,mode

transition,wrongexecutionsequence....)

designlogicofalgorithmisincorrect

engineeringunitincorrectengineeringunitisusedincalculation

equation/calculationincorrectequationorcalculation

failuredetectionincorrectfailuredetectionalgorithm

failureisolationincorrectfailureisolationalgorithm

failuremanagementincorrectfailuremanagementlogic(failurereporting)

failurereportingincorrectfailurereportingortriggerlogictogeneratefailurereport

incorrectsignalincorrectsignalisusedincalculation

initializationlogicincorrectinitializationalgorithm

initializationofvaluesincorrectinitializationvalues

invertedlogicinvertedtrueorfalselogicPage |11

AlgorithmFailureClass(Cont'd)

FailureTypeDefinition

missinginitializationmissinginitializationfunction

missinglimitermissinglimiterinthecalculation

prototypemissingprototype

rangeincorrectorunnecessaryrangeincalculationorcondition

relationaloperatorincorrectrelationaloperator(i.e.>,<,>=,<=...)

resetlogicincorrectresetalgorithm

resettimingincorrectresettiming

responsetodetectedfailureconditionincorrectreposetodetectedfailurecondition

samplingtimeincorrectsamplingtime

settingvalue/variableincorrectalgorithmtosettingvaluesorvariables

syntaxsyntaxerror

testmodelingincorrecttestmodelingproduceincorrectvaluesforthetest

thresholdincorrectthreshold

timingincorrectdelay

typotypoinalgorithmcausesdisconnectbetweensignals

validitychecktimingmissingorincorrectorinappropriatetimingofvaliditycheck

BUSINTERFACE

TheBusInterfaceclassdefinesacollectionoferrortypesthatrepresentdatasourceandbustranslationerrors.

Thisisarelativelyfocusedclasswiththefollowing4errortypes.

BusInterfaceFailureClass

FailureTypeDefinition

bitpositionincorrectbitposition

businitializationfailurebusinitializationfailure

datasourceincorrectdatasourceisconnectedtobusinterface

missingsignalmissingasignalinbusinterface

CONFIGURATIONMANAGEMENT(CM)

Althoughoftenreferredtointhecontextofprocessandtools,problemswithinCMmanifestthemselvesasreal

problemsinflightcriticalsoftwaresystems.Throughthisstudy,weidentifiedthefollowing6CMfailuretypes.

ConfigurationManagementFailureClass

FailureTypeDefinition

approvaldelaycorrectversionofSWwasnotapproved.

implementationdelay

incorrectversionofsoftwareusingincorrectversionofSW

missingCRimplementationmissingCRimplementation

outdatedrequirementdidnotupdaterequirementtomatchaSWchange

requirementincorporationdelaydidnotupdateSWtomatcharequirementchangePage |12

COMPILERERROR

TheCompilerErrorisageneralclassoferrorthatiscreatedbythetoolsinthesoftwarebuildchain.Thatis,an

errorinanyspecifictoolusedintheprocessoftranslatingsourcecodeintoexecutablecodeisconsidereda

CompilerError.Inthisstudy,theonlytypeofcompilererroridentifiedwasthegenerationofincorrectassembly

code—mostlikelybecausethetoolsusedtobuildtheflightcriticalsystemsinthestudyarematureandhavebeen

prequalified.Infact,whendevelopingflightcriticalsystemsusingmaturesoftwaredevelopmentenvironments,

compilererrorsaccountforlessthan0.5%ofallsoftwareerrors.

CompilerErrorFailureClass

FailureTypeDefinition

IncorrectAssemblyCodeIncorrectAssemblyCode

DATADEFINITION

Incorrectrepresentationofdatastructuresinmemory,dataoffsetsandroworderingareallexamplesofData

Definitionerrors.Duringthisstudy,weidentifiedthefollowing6distinctdatadefinitionerrortypes:

DataDefinitionFailureClass

FailureTypeDefinition

datastructureincorrectdatastructure

datatypeincorrectdefinitionofdatatype

enumerationincorrectenumeration

lookuptabledataincorrectlookuptabledata

offsetincorrectdataoffsetforI/Oorbuslistormemorymappedmessage

sizeincorrectbitorbytesize

DATAHANDLING

ADataHandlingerrorisaclassofsoftwareerrorthatinvolvesillegal,undefinedorincorrectuseofadataelement

orvariable.DataHandlingerrorsdifferfromDataDefinitionerrorsinthattheydonotmanifestthemselvesatthe

moduleinterface,anddonotnecessarilyinvolveincorrectstructuredefinitions.Wehaveidentifiedthefollowing

14typesofDataHandlingerrors:

DataHandlingFailureClass

FailureTypeDefinition

biasmissingorincorrectbias

bitconversionincorrecthandlingof16bitand32bitconversions

breakpointincorrectbreakpoint

byte/bitorderincorrectbyteorbitorder(i.e.endianness,byteswap,LSBandMSBreversed)

indexingimproperindexingintoarraysortablePage |13

DataHandlingFailureClass(Cont'd)

FailureTypeDefinition

inputfaulttoleranceincorrecttolerancetodetectinputfault

logicincorrectdatahandlinglogic

maskingdatamaskingdatawithincorrectvaluesornotmaskingdatawhichweareexpecting

tobemasked

memoryaddressusingincorrectmemoryaddress

mnemonicsincorrectmnemonicsinhashtable

scalingfactorusingincorrectscalingfactor

transitionlogicincorrecttransitionlogic

variableincorrectvariablesorvariabletypetoaccessdata

variablescopeincorrectvariabletype(global,local)

DOCUMENTATION

TheDocumentationErrorisageneralclassthatdefineserrorsinthedocumentation(requirements,design

documents,flowcharts,statecharts,architecturediagrams,etc.)thatleadtosoftwareanomaliesdownstreamin

theprocess.Therewerenoemergentpatternsfromthisstudytodefinespecificdocumentationerrortypeswith

anystatisticallysignificantbasis,eventhough11%ofallerrorswereofthistype.Fortunately,Documentation

errors—havingahighphasecontainmentratio—areoftendetectedduringthedevelopmentphaseinwhichthey

arecreated,ortheverynextphaseintheprocess.Wediscussthesignificanceofthisinmoredetaillater1

.

HARDWARE

HardwareErrorsaredefinedasaclassoferrorthatelucidatedeficienciesorflawsinthephysicalsystemsupon

whichthesoftwarehasdirectorindirectinfluence.Thisstudydefines1typeofhardwareerror:

HardwareFailureClass

FailureTypeDefinition

unexpectedbehaviorHardwaredeficiencymitigatedbySoftware

INPUTOUTPUT(I/O)SYSTEM

I/OSystemErrorsrepresentaclassoferrorsthatareresidentinmodulesorsubsystemswhichareresponsiblefor

providingdatato(andgettingdatafrom)othermodulesorsubsystemswithinthearchitecture.Althoughthisclass

oferrorisnotthemostprevalent,I/OSystemerrorshavethehighestaverageseverityofalltheerrorclasses.

Again,thesignificanceofthiswillbediscussedlaterinthereport2

.Werecognize4distinctI/OSystemerrortypes.



1

SeeErrorAnalysis–RankingsbyOccurrence.2

SeeErrorAnalysis–RankingsbySeverity.
Page |14

I/OSystemFailureClass

FailureTypeDefinition

datalistincorrectdatalist

I/OsynchronizationCoordinationofI/Otiming,lists,etc.

orderofdatastructureincorrectorderofdatastructure

signalassignmentmissingorincorrectsignalassignment

IMPLEMENTATION

AnImplementationErrorisdefinedasageneralclassoferrorthroughwhicharequirementorsoftwarechange

requestwasimplementedincorrectlyinthesourcecode.Thisstudydidnotrevealanysignificantordistinct

implementationerrortypes,andallimplementationerrorsaccountforlessthan1%ofallanomalyreportsstudied.

INTERPROCESSCOMMUNICATION

Wedefine,ingeneral,InterprocessCommunication Errorsasincorrecthandshakingbetweenprocessesorparallel

modules.Thisincludescoordinationofresources,failuremanagementandoveralltimingissues.Thisstudy

revealed9distinctinterprocesscommunicationerrortypes.

InterprocessCommunicationFailureClass

FailureTypeDefinition

decisionlogicincorrectdecisionlogic(i.e.ifthenelse,casestatements,beginend,modetransition,

wrongexecutionsequence....)

engineeringunitmismatchengineeringunitmismatch

failuremanagementincorrectfailuremanagementlogic

I/OsynchronizationI/Oisnotsynchronizedininterchanneldatabox

initializationlogicincorrectinitializationlogic

logicincorrectlogicofinterprocesscommunication

resettimingincorrectresettiming

samplingtimeincorrectsamplingtime

timingincorrectdelay

PERFORMANCE

TheclassoferrorsconsideredunderthetermPerformancedefinesthoseerrorswhichviolateeitherrealtime

requirementsorprocessorutilizationthresholds.Duringourstudy,wewereabletostatisticallysubstantiatethe

followingperformanceerrortype:

PerformanceFailureClass

FailureTypeDefinition

ExceedProcessorUtilizationTargetExceedProcessorUtilizationTargetPage |15

SELFTEST

Aspartofthedevelopmentprocessforflightcriticalsystems,itisnecessarytoincorporateintothesystema

sufficientsuiteofpreflightteststhatverifythesuitabilityofthesystemrelativetothemissionitisaboutto

perform.Thistestsequence;oftenreferredtoasSelfTestorbuiltintest,isdesignedtoprovideago/nogo

decisionrelativetopredeterminedfitnessconditions.However,errorsintheSelfTestitselfmayyielderroneous

results.Suchistheclassoferrordefinedbythiscategory,fromwhichweidentifythefollowing8distincttypes:

SelfTestFailureClass

FailureTypeDefinition

impropertestconditionrunningtestwithimpropercondition

designincorrecttestdesign

inadequaterequirementrequirementisnotspecificenoughtotest

testtimingincorrecttesttiming

timemanagementinefficientuseoftime

valueoflocationlocationcontainsincorrectvaluesintestpattern

valuesfortestincorrectvaluesorreferencefortest

missingresetfunctionmissingresetfunctionintestprocedure(foreithernecessaryorworkaround)

SYSTEMINTEGRATION

SystemIntegrationdefinesaclassoferrorsthatarisewhenmajorsystemcomponentscometogetherorinteract

withmoderatedependency.Sucherrorsmaybeobviousrightatsystempowerup,whileothersmaynotbe

identifieduntilthesystemissubjecttouniqueorunforeseencircumstances.Basedonthisstudy,System

Integrationerrorshavethemostderivedtypesofalltheerrorclasses.Weidentified24ofthem.

SystemIntegrationFailureClass

FailureTypeDefinition

channelsynchronizationchannelsarenotsynchronized

conflictingrequirementconflictingrequirement

changerequest(CR)incorrectCRwaswritten,approvedandincorporated.

datasourceincorrectdatasourceisconnectedtobusinterface

engineeringunitmismatchsignalsfromtwodifferentsystemsdidnotagreeonunits(i.e.radian,degree)

ICDandSWmismatchICDandSWarenotmatching

inconsistentinterfaceorderinconsistentindex(order)ofI/Obetweensystems

incorrectrequirementincorrectrequirement

interfaceincorrectinterface

manualincorrectmanual(flightmanual)Page |16

SystemIntegrationFailureClass(Cont'd)

FailureTypeDefinition

memoryuseusingincorrectkindofmemory(i.e.useCPUcheckRAMinsteadofinternalRAM)

missingdatamissingdatainatableofdesigndocument

missingdatapumpmissingdataindatapumplist

missingheaderfilemissedincludeheaderfileinthemaincode

missingsignalsinICDmissingsignalsinICD

missingSWupdatehardwarechangedbutSWdidnotchange

missingtestpointsymbolismissingfortestsymboltable

norequirementthereisnorequirementforanissuessoitneededtobecreated

parameterincorrectparameter

parameterorderparameterorder

ratesynchronizationratesynchronization

requirementnotclearnotenoughguidelinestounderstandrequirement

testpointnamesymbolnameofsignalandsignalincodearenotthesame

unnecessaryrequirementunnecessaryrequirementneededtobedeleted

TOOLS

Unfortunately,toolsalsointroduceerrorsintosoftwaresystems.Throughourstudy,weidentifiedthefollowing2

ToolErrortypes:

ToolFailureClass

FailureTypeDefinition

Algorithmtoolsgeneratesincorrectsignalorvalues

inputdatamissingorincorrectinputdatasotoolgeneratejunkcode

USER/PILOT

Anyerrorsassociatedwiththeoperationofthesystempurelyfromtheperspectiveoftheuserorpilot,under

normaloperatingconditions,fallundertheUser/Pilotclass.Thatis,errorsidentifiedthroughspecificflighttestsor

failureconditions—perhapsemployingapilotoruser—arenotconsideredUser/Piloterrors.Throughthisstudy,

therewerenoinstanceswhereanyactiononbehalfoftheuserorpilotcausedasoftwarefailurethatwasnot

properlymatchedtoanothererrorclass.Allqualificationsconsidered;weidentifiedthefollowingtypeof

User/Piloterrortype:

User/PilotFailureClass

FailureTypeDefinition

preferenceresultsthatarenotnecessarilyincorrectorunsafebutpilotswanttochangesotheyfeelmore

comfortableorlowCooperHarperratingsPage |17

ERRORANALYSIS

Onceweidentifiedthepropertaxonomy,wewereabletoperformsomeusefulanalysisontheresultantdata.

Thissectiondescribesouranalysisandthecorrespondingresults.

BACKGROUND

Similartomanyriskmanagementapproaches3,ourapproachconsiderstheprimarydriversofprobabilityand

severity.Wealsoaddathirddimension—thelikelihoodofdetection.Althoughsimilarinnametowhatonemay

encounterinafailuremodeandeffectsanalysisworksheet4

,thisparametermeasureshowlongagiventypeof

softwareerrorislikelytoremainpresentinthesystembeforeitisfound.Thatis,itisameasureofthedelta

betweenthephaseinwhichanerrorisdetectedandthephaseinwhichtherootcauseanalysisdetermineditwas

likelyinjected.

Theprimarydifferencebetweenouranalysisandotherriskassessmentsisthatourresultsarebasedondataand

eventsthatalreadyexistandhavetranspiredratherthanestimatingaprobabilityofoccurrenceandaseverity.

Wethenusetheentirecollectionofdatatomakepredictiveinferencesandsuggestionsforsolutionsthatcan

mitigatehighriskareasthroughsoftwarehealthmanagement.

THERISKPRIORITYNUMBER

TheRiskPriorityNumber(RPN)isafundamentalmeasureofriskassociatedwitheachfailuretype.Itisa

parameter,normalizedtoavaluebetween0and1000,whichclearlyindicatestherelativeriskpriorityofelements

withinthetaxonomy.Itiscalculatedas:

      	

Where:

 
 



 
 

	 
 
 ! " # 
$%&! "

CALCULATINGRELATIVEFREQUENCY

Therelativefrequencyofaclassiscalculatedbythesumofallanomaliesunderthatclassdividedbythenumberof

anomalyreportsinthemostfrequentclass.Itisrepresentedasanormalizednumberbetween010.



3

i.e.quantitativeorprobabilisticriskassessment

4

Seehttp://en.wikipedia.org/wiki/Failure_mode_and_effects_analysisforanexample.
Page |18

CALCULATINGRELATIVESEVERITY

Theseveritytermiscalculatedbynormalizingtheanomalyseveritycodesagainstaweightedscale.Eachanomaly

reportweanalyzedhadanassociatedseveritycoderangingfrom15,whereseverities1&2directlyaffectsafetyof

flight.Toaccuratelyrepresentthisseparation,wenormalizedtheseveritycodeasanumberbetween1and10

accordingtothefollowingtable:

CALCULATINGTHEDETECTIONPARAMETER

ThefinalparameteroftheRPNrepresentshowlongasoftwareerrorremainedwithinthesystemsincetheerror

wasfirstintroduced.Thatis,itisanindicatorofhowlikelyacertainclassoferrorwillgoundetectedbythe

establishedverificationandvalidation(V&V)process.

Tocreatetheparameter,weanalyzedeachanomalyreportandcalculatedtheweighteddeltaphasefactordirectly

fromthetablebelow.Forexample,ifananomalywasdetectedduringIntegrationandTest,andtherootcauseof

theerrorwasfoundtobeanerrorintheRequirementsofthatmodule,thenthedeltaphasevalueis8.

PRESCRIPTIONSOFTHERPNMODEL

Ingeneral,anyelementwithanRPNgreaterthan100can

beconsideredhighrisk.Althoughthiscutoffisopento

conjecture,theupperendoftheRPNspectrumsurely

deservesattention.Forinstance,thetopmostelement—

algorithmdesign—canemergeasanentirefieldofstudyin

itsownright.Thetabletotherightshowselementsfrom

theentiretaxonomywhoseRPNisgreaterthan100.Page |19

DETAILEDCLASSANALYSIS

Thefollowingsectionspresentadetailedanalysisofeacherrorclass.TheanalysisshowstheRPNforeachspecific

errortypeofthetaxonomyaswellasthetype’srelativedistributionprofilewithintheclass.Thefollowingtableis

asummaryofthoseerrorclasseswhichhavealimitednumberoftypes.

ErrorClassErrorTypeRPN

DocumentationDocumentationerror262

Implementationrequirementimplementationerror46

ToolsAlgorithm30

CompilerErrorIncorrectAssemblyCode29

PilotPreference12

Hardwareunexpectedbehavior8

PerformanceExceedProcessorUtilizationTarget7

Toolsinputdata1

Arolluptheindividualerrortypesrevealssomenotable

observationsabouttheindividualerrorclasses

themselves.Perhapsthemostnotableofwhichisthatthe

topthreeerrorclasses—Algorithm,DataHandlingand

SystemIntegration—accountforover70%ofallsoftware

errors,asillustratedinthegraphshowninFigure10,at

right.

Notonlyarethetopthreeclassesthemostfrequent;with

RPNvaluesbetween100and1000,theyarealsointhe

highriskcategory,asseeninFigure11below.

Figure2–ClassLevelErrorProfile

Figure1ClassLevelAnalysis
Page |20

RPNCOMPONENTANALYSIS

Atthispoint,wediscusstheindividualparametersofRPNforthefailureclassanalysis.Themostdominant

discriminatorforRPNanalysisistheoccurrenceparameter.Thereissomedistinctdifferentiationbetween

severityanddetectionaswell,butnotnearlyasdrasticasoccurrence.Thefollowingsectionspresenttheresults

ofeachRPNparameterindividually.

OCCURRENCEPARAMETER

Theoccurrenceparameteristhemostdiscriminatingfactorofallthefailureclasses.Figure3,above,showsthe

breakdownbyfailureclass.NotethatthereareseveraldisplacementsfromtherawRPNbreakdown.Thisis

because,althoughsomeerrorsaremorefrequentthanothers,theymaynotbeassevereorashardtodetect—

whichjustifiesthefailureanalysisacrossthethreefundamentaldimensionsofoccurrence,severity,andlikelihood

ofdetection.

Figure3–OccurrenceDimensionPage |21

SEVERITYPARAMETER

Theseveritydimension,illustratedinFigure4above,showsthatthedominantfailureclassisI/Osystem.Thatis,

mosterrorsinthisclassarelikelytoaffectsafetyofflight—resultingingroundedaircraftorspecificoperating

limits.

DETECTIONPARAMETER

Figure4–SeverityDimension

Figure5–DetectionDimensionPage |22

Thedetectionparameteralsoofferssomeusefulinsightintothenatureoftheerrors.Figure5showsthat

hardwareandusererrorsexistlongestinthedevelopmentcycle,whileimplementation,tools,anddocumentation

errortypesaredetectedratherquickly.

ALGORITHMERRORCLASSPROFILE

ConsideringtheAlgorithmfailureclass,overallalgorithmdesignhasthehighestRPNandalsoaccountsfor22%of

allalgorithmerrors.Decisionlogicanddatatransfer/messagingcomponentscomeinnext;wherethetopthree

combinedaccountfornearlyhalfofallthealgorithmerrors.

SomeexamplesofanAlgorithmerrormaybe:incorrectpoweruporinitializationroutinesafteraresetthatcause

failuremonitorstotripinanothermodule;goodchannelaverageselectionalgorithmsthatinadvertentlyinclude

thebadsignalinthecalculation;orperhapsasetoflimitvaluesthatarenotusedwhendifferentloadingorair

vehicleconfigurationsareselectedfromanothersubsystem.Inhindsight,thesetypesoferrorsmayseemobvious

andmayleadonetobelievemoreunittestingisrequired.Therealityis,however,thatthesetypesoferrorsmay

besoembeddedinthealgorithmthatunittestswouldnotexercisetheunforeseenstatesproperly.Considerthe

caseofthelimitervalueswitchingalgorithm.Aunittestmayverifythatthesetoflimitsisproperlyswitched

underallconditionsthroughwhicharequestmaybemade.Butifthelogicinthealgorithmisdesignedtonever

maketheproperrequest,thelimitsetisneverswitched.

Figure6AlgorithmErrorProfilePage |23

Thisreportisnotintendedtoprovidephilosophicaloranecdotaljustificationofthedatapresented;howeverthis

particularcaseisconsideredatlengthin[3].Essentially,properalgorithmdesignrequiresintimateknowledgeof

theenvironmentinwhichthesoftwareistooperateaswellassufficientdomainknowledgetoconsiderpurposeful

orinadvertentchangestothatenvironment.Thisstudyrevealsthegravityofthiserrorclassandrecommendsthat

technologiesbedevelopedtoaddressit.

BUSINTERFACEERRORCLASSPROFILE

ThebusinterfaceerrorswestudiedallhaveanRPNlowerthan100,butgreaterthan10.Basedontheentireset

ofdatarepresentedinthisstudy,RPNvaluesbetween10and100couldbeconsideredmediumrisk,whereRPN

valueslowerthan10representlowriskitems.Thedistributionoferrorreportsclassifiedasinterfaceerrortypes

arefairlyevenlydistributedacrossthespecifictypeswithintheclass,asidentifiedbythecumulativepercentage

lineinred.



Figure7–BusInterfaceErrorProfilePage |24

CONFIGURATIONMANAGEMENTERRORCLASSPROFILE

AllCMerrorsareinthemediumriskRPNrange.Manyoftheseerrorscanbeaddressedbyexistingprocesses.

DATADEFINITIONERRORCLASSPROFILE

Datadefinitionerrorsarealsomediumriskerrorsandcanbeaddressedearlierbymoredetaileddataand

interfacemodels.

Figure8 – ConfigurationManagementErrorProfile

Figure9–DataDefinitionErrorProfilePage |25

DATAHANDLINGERRORCLASSPROFILE

Thetwohighriskerrortypesforthedatahandlingerrorclassare:scalingfactorandmemoryaddress.Thisis

essentiallytheinterfacebetweensubsystemsandcanbeaddressedwithmoredetailedinterfacemodelingand

designverificationtechniques.

INTERPROCESSCOMMUNICATIONERRORCLASSPROFILE

IPCerrorsaregenerallylowrisk.Timingandsynchronizationerrorscanpracticallybecaughtonlyinalab

environment,althoughformalanalysisanddesignverificationcanaddressseveraloftheothers.

Figure10–DataHandlingErrorProfile

Figure11–IPCErrorProfilePage |26

INPUT/OUTPUTSYSTEMERRORCLASSPROFILE

I/Oerrorsaregenerallydifficulttofindduringdevelopmentandexistforasignificanttimeintheproductlifecycle.

Moredetailedandrealisticmodelingcouldaddresstheseissues,butwouldrequireadetailedcostbenefitanalysis

todeterminebreakevenpointsformitigatingtherisk.

SELFTESTERRORPROFILE

Selftesterrorsareofmarginalconcernandcouldbeaddressedthroughprocessandtechnique.

Figure12–I/OSystemErrorProfile

Figure13–SelfTestErrorProfilePage |27

SYSTEMINTEGRATIONERRORCLASSPROFILE

Thesystemintegrationclasscontainsmanyspecificfailuretypes.Thisobservationinitselfshowsthatasignificant

amountoferrors,ingeneral,areofthisclass.Althoughsoftwaremayworkwellinindividualmodulesorunittest

levels,itiswhenthemodulesareintegratedwithalargersystemthatalloftheenvironmentalassumptionsand

erroneousinvariantsbegintosurface.Thiserrorclassrequiresanentirededicatedstudy,astherootoftheerrors

lieintheoriginalrequirementsandspecificationsthatneededinterpretation.

ROOTFAILURECAUSEANDEFFECTRELATIONSHIPANALYSIS

HavingcalculatedtheRPNfortheFundamentalfailuretypes,wemovedourfocusfromindividualriskassessment

toexaminingtherelationshipsbetweenthefundamentalfailuretypes.Wemadechartstoshowtherelationships.

Thissectiondescribestherootfailurecauseandeffectrelationshipchartsandouranalysisonit.

BACKGROUND

Whenwewereworkingonthefailuretypetaxonomy,werealizedthatsomeofthefailuretypeshavecauseand

effectrelationships.Forinstance,thefailuretypesof“algorithm:initializationofvalues”,“algorithm:timing”,and

“algorithm:initializationlogic”wouldallberelatedinthefailuresofinitializingcorrectlytostartanewmode

duringamodetransition.Thishasshownupinconcreteexampleswhereaprocessswitchedintoanewmode

Figure14–SystemIntegrationErrorProfilePage |28

beforeanotherprocessgeneratinginputshadswitchedtothenewmode.Inthiscase,theanalysisengineers

wouldrecordthedefectinoneofthethreefailuretypesbutitisamistaketoconsiderthatfailuretypeinisolation

fromtheothertwo.Weconstructeddiagramsindicatingthefailuretypesthatweshouldconsidertogether.We

connectedrelatedfailuretypesbyarrows.Thedirectionofthearrowsisfromthebroaderscopedfailuretypeto

themorespecificfailuretype.Thenwepulledtogethertheconnectedpartsintologicalgroupingscenteredon

thelargestofthe17failureclasses.Severalofthe17failureclassesendedupsplitbetweenlogicalgroupings.

GROUNDRULES

1. Therelationshipswerenotnecessarilydirectcauseeffectrelationships,butwereratheralogical

correlationbetweenthetwo.

2. Anerrororconfusioninoneareamighttendtoimplyanerrororconfusionintherelatedarea.

3. Eachfailuretypeappearsonlyonceinthediagrams.Wesplitthediagramssothatnorelationshipswere

lost.Onlytherequirementsclassappearsinmultiplediagramstoindicatewheretherequirementscome

intothosediagrams.

4. Wecolorcodedthe114failuretypestoindicatetheirRPNpercentileamongthefailuretypesby:

Red=5%HighestRPNfailuretypes

Orange=Next10%RPNfailuretypes

Yellow=Next15%RPNfailuretypes

Blue=Next20%RPNfailuretypes

Green=RemainingLowest50%RPNfailuretypes

Inthisreportwecallthesethe“RPNpercentilegroups”.Theredandorangeblocksarethe“highRPN”

failuretypes.Theyellowandblueblocksarethe“mediumRPN”failuretypes.

OVERVIEWOFROOTFAILURECAUSEANDEFFECTRELATIONSHIPCHART

Weorganizedthe114failuretypesintorelateditemsandformedsevenlogicalgroups.Thesevenlogicalgroups

areRequirement,ConfigurationManagement(CM),ExternalProblems,Documentation,Algorithm,System

Integration/Communication,andSelfTest.Page |29

Figure15showsthetoplevelorganizationofthesesevengroups.The“Requirements”categoryisatthecenter

becauseitaffectsvirtuallyalloftheothercategories.“ExternalProblems”categorydoesnotconsistexclusivelyof

softwareproblemsbuttheyareproblems

thatrequiresoftwaremodificationto

overcomethem. The“Algorithm”

categoryisthelargestandcontainsa

concentrationofhighRPNfailuretypes.

“SystemIntegration/Communication”is

alsoalargecategorywithsomehighRPN

failuretypes.The“SelfTest”category

hasnohighRPNfailuretypes.

“Documentation”wasalargecategory

onlybecausewedidnotsubdivideit.

Weleftthe“ConfigurationManagement”

categoryasastandaloneitembecauseitinvolveseverystepinthesoftwaredevelopmentprocess.Wecanlook

atthe“ConfigurationManagement”categoryasaprocessproblemthatrunsparallelwithothercategoriesof

problems.Foritssmallsize,ithasalargenumberofmediumRPNfailuretypes.

HereisthenumberofdifferentRPNpercentilegroupsineachcategory:

Requirements:2orange,1yellow,1green

CM:2yellow,4blue,2green

Externalproblems:1blue,3green

Documentation:1red

Algorithm:3red,9orange,6yellow,8blue,20green

SystemIntegration/Communication:1red,1orange,7yellow,8blue,17green

SelfTest:1yellow,2blue,13green

DOCUMENTATIONANDEXTERNALPROBLEMSCATEGORY

Figure16showstheDocumentationcategory.Documentationerrorsareinthetop5%RPNduetotherateof

occurrence.Thesefailuresaccountedforover11%ofthetotalfailures.Theseverityscorewasaverageandthe

detectionscorewaslow(meaningtheywereeasyto

detectandwereremovedquickly).Wedidnotanalyze

orsubdividethisfailuretypecategory.Wedidnottry

toanalyzetherelationshipsbetweenthesefailuresand

others.Wedidnottrytodetermineifotherfailures

influencedthedocumentationerrorsorviceversa.

Theremightbesomeconnectionbetweenthem.

Figure15–RelatedRootFailureCategories

Figure16–DocumentationCategoryPage |30

Figure17showsthe

ExternalProblems

category.Itisa

“CatchAll”category

forasmallnumber

ofproblems.The

rootcausesofthese

failuresareall

externaltothecore

software

development

processofthe

applicationcode.Theyareprimarilyareduetorequirementsfortheapplicationsoftwaretomitigateunexpected

failuresinotherareas.Exceptfor“CompilerError:IncorrectAssemblyCode”,allthesefailuretypesareinthelow

RPNrange(green).The“CompilerError:IncorrectAssemblyCode”hasunremarkableseverityanddetection

scores.The“Pilot:preference”failuretypeisduetotestpilotsnotagreeingorchangingtheirpreference.Ithasa

lowseverityscorebutarelativelyhighdetectionscore.Noneofthesefailuretypeshasahighoccurrencerate,but

theirdetectionscoresarehigh.The“systemintegration:manual”referstoerrorsintheflightmanual.Thisfailure

typehasanespeciallyhighdetectionscorealthoughitsseverityscoreislow.

REQUIREMENTSCATEGORY

Figure18showsthe

Requirementscategory.

Theseareallsystem

integrationproblems.

Requirementsrarely

conflictandareusually

clearenough.Theyare

morelikelytobe

missingorincorrect.

TherearetwohighRPN

failuretypes.TheRPN

differencesoftheRequirementscategoryaremostlyduetotherateofoccurrence.Therearenoclear

relationshipsbetweenthesefailuretypesorwithanyotherfailuretypes.

Figure17–ExternalProblemsCategory

Figure18–RequirementsCategory
Page |31

CONFIGURATIONMANAGEMENTCATEGORY

Figure19showstheConfigurationManagementcategory.MostofthesefailuresarerelatedtoChangeRequest

(CR)processdelaysandtheirimpactonsystemintegration.Thiscategoryhastwoyellowfailureblocksandseveral

blue blocks.Itisasignificantfailurecategory.TheRPNdifferencesoftheConfigurationManagementcategoryare

mostlyduetotherate

ofoccurrence.Thisis

thefirstcategorywith

relationshipsbetween

failuretypes.Several

“systemintegration”

failuretypesappearin

thisdiagrambecauseof

theirrelationshipswith

the“configuration

management”failuretypes.Thetwoyellowblocks,“CM:implementationdelay”and“CM:missingCR

implementation”aregroupedtogetherwiththegreen“CM:requirementincorporationdelay”tocollectthe

problemswithdelaysinalreadyapprovedchanges.Thiscollectionrelatestoseveral“systemintegration”failure

types,allhavingtodowithincompatiblesoftwareorinterfaces.The“systemintegration:missingSWupdate”

failuretypecanbecausedbythe“CM:implementationdelay”,or“CM:missingCRimplementation”failuretypes.

Thesamerelationshipistrueforthe“systemintegration:inconsistentinterfaceorder”and“systemintegration:

ICDandSWmismatch”failuretypes.Thegreen“CM:approvaldelay”isgreenbecauseitdoesnotoccuroften,but

itsseverityscoreishigh.Itcancontributetothe“CM:incorrectversionofsoftware”failuretype,whichisblue.

ALGORITHMCATEGORY

Figure20illustratestheAlgorithmcategory.Thisisasignificantandinterrelatedcategoryoffailuretypes.Itshows

therelationshipbetweenalgorithmdesign,interprocesscommunication,andrequirementscategory.Itisthe

mostsignificantcollectionofrelatedfailuretypes.ItincludesthetoptwoRPNrankedfailuretypes,“algorithm:

design”and“algorithm:decisionlogic”.The“algorithm:design”failuretypealoneaccountsforover10%ofallthe

rootfailuresinthestudy.Thenexthighestis“algorithm:decisionlogic”,whichaccountsforover5%ofalltheroot

failuresinthestudy.Thefinalredrootfailuretypeinthediagramis“algorithm:failuremanagement”.Thistype

involvesthelogicofsignalredundancy,selection,andverification.Itaccountsforabout3%alltherootfailures.

Thedesignsinthatsystemshouldnotrequireagreatdealofmodificationinthenormaldesignloop.Another

noticeablepartoftheAlgorithmdiagramisthethreerelatedorangefailuresof“algorithm:initializationlogic”,

“algorithm:timing”,and“algorithm:initializationofvalues”.Togethertheseareover4%ofalltherootfailures.

Thisfailuretypeincludesproblemsintimingofinitializationswhenmodeschangeandtheinputsarenotcorrect

Figure19–ConfigurationManagementCategory
Page |32

forthenewmode.Inaddition,statevariablesmaynothavebeenresetcorrectlywhennewmodestartedrunning.

Severalofthefailuretypesgrouptogether.Intheupperleftofthediagramisasetofthreesignaldefinition

problems,“datadefinition:lookuptabledata”,“algorithm:incorrectunit”,and“algorithm:incorrectsignal”.

Theseareproblemswhichareinteriortothealgorithmbuttheycanbeinfluencedbythe“systemintegration”

faulttypesof“systemintegration:missingdata”or“systemintegration:engineeringunitmismatch”.Thissetof

failuretypescancause“algorithm:equation/calculation”failuretypes.Anothersignificantcollectionoffailure

typesdealswiththerangeprocessingofsignals.Itconsistsofthe“algorithm:range”,“algorithm:threshold”,and

“algorithm:missinglimits”failuretypes.Thissetalsocaninfluencethe“algorithm:equation/calculation”failure

type.Onesetoffailureswhichisunrelatedtootherfailuresisthesetofrandom“mutation”typefailures,

“algorithm:syntax”,and“algorithm:typo”.Usuallythecompilerdetectsthesetypesoferrorsimmediatelybutthe

onesthatslipthroughcanbeverydifficulttodetect.Itisdifficultforthecompilertodetectavariablenametypo

thatendsupmatchingthewrong,butotherwisevalid,variable.Itisalsodifficultforcompilerstospotthe“if(A=

B)”vs.“if(A==B)”problemunlessthefirstoneisspecificallydisallowed.Thesefailurescangoundetectedfora

longtime.Wehavealsoincluded“algorithm:deadcode”inthissetalthoughitmayhaverelationshipstoCM

failuretypeswhichwehavenotestablishedyet.The“algorithm:resettiming”failuretypeisgreen.Ithasalow

occurrenceratebutahighseverityscore.Itisinfluencedbythe“algorithm:resetlogic”failuretype,whichis

orangeduetoahighoccurrencerate.The“algorithm:resettiming”failuretypeissecondarytothe“algorithm:

resetlogic”failuretype.Thereisasignificantsetofdiscretelogicproblemsconsistingof(listedinorderof

decreasingRPN)“algorithm:decisionlogic”,“algorithm:invertedlogic”,“algorithm:relationaloperator”,and

algorithm:compoundlogic”.The“algorithm:decisionlogic”failuretypeisredduetoitshighrateofoccurrence.

Itmayincludesomefailuresthatbelongintheothermorespecificlogiccategoriesifweexaminedthemfurther.

Thesefailuresarelargelyselfinitiatedduetothecomplexityofthelogicanddonothaverelationshipstoother

failuretypes.Theyarestructural/discretelogicdefectsthatmaybedetectedifformalmethodscanbeapplied.

Towardtherightofthediagramareseveralfailuremanagement/failurereconfigurationblocks.Manyofthese

arehavesignificantRPNvalues.Theentirecollectionis“algorithm:failuredetection”,algorithm:failure

reporting”,“algorithm:failuremanagement”,“algorithm:failureisolation”,“algorithm:responsetodetected

failurecondition”,“interprocesscommunication:failuremanagement”,“datahandling:inputfaulttolerance”,and

“businterface:businitializationfailure”.AtthelowerleftofthediagramisalargecollectionoflowRPN

green/blueblocksdealingprimarilywithinterprocesscommunicationtimingproblems.Thered“algorithm:

design”blockhasalreadybeendiscussed.Page |33

Figure20–AlgorithmCategory
Page |34

SYSTEMINTEGRATION/COMMUNICATIONCATEGORY

Figure21showstheSystemIntegration/CommunicationCategory.Itincludesasignificantnumberof

high/mediumRPNfailuretypesandincludesmanyrelationships.

Figure21–SystemIntegration/CommunicationCategory
Page |35

ThehighRPNrootfailureshereare“algorithm:datatransfer/message”,“datahandling:scalingfactor”,and“data

handling:memoryaddress”,whichaccountforabout4%,4%,and3%ofthealltherootfailures,respectively.

ThesedatadictionaryinterfaceproblemscanbedealtwithusingsystemengineeringtoolssuchasSysMLorAADL.

Thetoolsshouldbesystemwide.Parttaskinterfacecontrolsdonothavethesamebenefitsunlesstheyare

coordinated.The“datahandling:scalefactor”failuretypepointstothedifficultyoftrackingfixedpointscaling

correctlythroughalltheengineeringunits,hardwareinterfaces,etc.Theengineeringdisciplinesusedifferent

unitswhentheyaddressfixedpointscalingandbias.ElectricaldiagramswillhaveVolts,current,andother

engineeringunits.Softwareengineerswantleastsignificantbit(LSB)values,fullrangemax/min,etc.Andallare

furthercomplicatedbybiases,bothphysicalandcomputational,alongtheway.Possiblyengineersneedatoolto

helpwithfixedpointrange,bias,scale,engineeringunits/LSB,etc.Severalsystemintegration/communication

blockshavealreadyappearedinotherdiagramswheretheyhadsignificantrelationshipswiththeblocksthere.

Wedividedthediagramssothatnorelationshipswerebroken.Alltheblockshereconnecttothemaindiagram.

Thered“algorithm:datatransfer/message”failurescanbecausedbythesetof“datahandling:logic”and“data

handling:transitionlogic”.Theycan,inturn,cause“algorithm:validitycheck”failures.Intheupper,centerofthe

diagramisacollectionofmissinginterfaceitems,“systemintegration:missingsignalsinICD”,“businterface:

missingsignal”,and“systemintegration:missingdatapump”.Theseareallgreenblocksandarenotvery

significant.Theycanbecausedbythe“I/Osystem:datalist”failuretypewhichisyellowduetoahighseverity

score.Intheirturn,theycancontributetothe“datahandling:indexing”failuretype,whichisyellowduetoahigh

occurrencerate.Thisreflectsproblemscausedbyshiftingdatawhenasignalismissing.Inthebottomleftofthe

diagramisacollectionofmediumRPNdatadefinitionfailuretypes.Theyare“datadefinition”offset,size,data

type,anddatastructure.Thefinallargecollectionoffailuretypesisthedatahandlingcollectiontothebottom

rightofthediagram.Thesearedatadictionaryissues.The“datahandling:scalingfactor”and“datahandling:

memoryaddress”failuretypesarethemostsignificantbyfar.Theyhavebeendiscussedabove.

Page |36

SELFTESTCATEGORY

Figure22showstheSelfTestCategory.TherearenohighRPNrootfailureshereandonlythreemediumRPN

failuretypes.Themostseriousrootfailureistheyellow“outdatedrequirement”rootfailurewhichaccountsfor

slightlyover1%ofalltherootfailures.Therearetwobluefailuretypes,“selftest:valuesfortest”and“tools:

algorithm”.Thesereflecttheproblemofgenerating“truthdata”fromthetoolsforuseintheselftest.Alltherest

oftheblocksaregreen.Atthetop,centerofthediagramareacollectionoftopleveldesignproblems.Theyare

“selftestprocedure:missingresetfunction”,“selftest:testtiming”,“selftest:timemanagement”,and

“performance:exceedprocessorutilizationtarget”.Atthecenter,rightaretwogreenblocksthatreflecttheneed

toincludetestpointsinthecodeformonitoringortestvalueinsertion.Theyarethe“systemintegration:missing

testpoint”,andthe“systemintegration:testpointname”failuretypes.Atthebottom,leftofthediagramaretwo

requirementsissues:outdatedandunnecessary.Atthebottomrightofthediagramareseveralissueswith

modelingandgeneratingvalidtruthdata.

APPLICATIONOFDATAANALYSISRESULTSTOEVALUATINGFUTURETECHNOLOGIES

Thedataanalysisresultscanbeusedtoanalyzetheimpactofthetechnologies,forexample,possiblyapplying

formalmethodstothealgorithms.Lookingatfigure20,thealgorithmrelateddefectsareamixtureofdiscrete

logicerrorslike“algorithm:decisionlogic”andfloatingpointcalculationerrorslike“algorithm:design”.An

applicationofformalmethodscouldbeusedtoidentifyandremovediscretelogicdefectsintheearly

developmentstages.Infigure20,formalmethodswouldreducethenumberoferrorsin“algorithm:decision

Figure22–SelfTestCategory
Page |37

logic”,“algorithm:failuremanagement”,“algorithm:initializationlogic”.

AnadjustmentcouldbemadeintheOccurrenceorDetectionnumbers

forthoseentriesintheRPNcalculations.UndertheSystemIntegration

/Communicationsection,thecollectionofdatahandlingfailurespoints

tothepossiblebenefitofanautomateddatadictionarydrivingthe

interfacegenerationtools.Additionally,evidencepointstothebenefits

ofhavingmodelbaseddesigntoolsthatencompasstheentiresystem.In

particular,requirementsfailuretypesmaybereducedbyusingsystemleveldesigntoolslikeSysMLorAADL.

ConflictingorimpreciserequirementswouldbespottedbyFormalMethodswhereitcouldbeapplied.Ingeneral

figure20,showsthatthedatadictionaryinformationisaproblem(size,location,address,bitorder,etc).

However,itisveryhardtofindasingletechnologythatcoverstheentireproblemspace.

However,itisbelievedwithhighconfidencethatasignificantnumberofsoftwareproblemscanbereducedbefore

enteringthenextphaseoftheprogrambyidentifyingthecorrectcombinationoftechnologytocovertheproblem

space.

Hereisoneexampleofhowthedataanalysisresultscanbeusedtoidentifypossiblecombinationsoftechnologies

forsoftwarehealthmanagement:

1.CreateMatrixofevaluationoftechnologieswitheachrootfailure.

A. Selecttechnologies/methodsthatyouwanttoexamine.

B. PrepareatablethatcontainsinformationoftheRPNandwhichfactoristhemostandtheleast

dominatingfactoroftheRPN.(ColorCodeinexample.Orange=themostdominantfactor,Yellow=2nd

dominantfactor,andGreen=theleastdominantfactor)

C. EvaluatealltheTechnologies/Methodschosenwithrespecttotheoccurrence,severity,detectionofeach

rootfailure.(Figure23illustratesthisprocess)

2.EvaluateeachTechnology/Methodsbyaffectabilitywithrespecttothemostandleastdominantfactorofthe

RPN.(Figure24istheexampleofthisprocess)

Figure23–RelatedRootFailureCategories

Figure24–RelatedRootFailureCategoriesPage |38

3.FromStep2,comeupwithdifferentcombinationofTechnologies/Methodstouseandevaluatethem.From

Table2,wecandrawconclusionsthat“method1”isthemosteffectiveforSoftwareHealthmanagementmethod.

However,itdoesnotcoveralltheissues.Figure23providessomeadditionalexampletablesthatshowhowmany

problemsthatcanbecoveredwithdifferentcombinationsofTechnologies/Methods.

Individualsthataredevelopingmethodsortoolsforsoftwarehealthmanagementandusingcurrentlyavailable

methodsortoolscanbenefitfromthiskindofpractice.

FortheDeveloperofmethodsortoolsforsoftwarehealthmanagement,thispracticecanbetheirassessment,and

itwillhelpusersidentifywhatkindofmethodstheyaregoingtousefortheirproject.

Figure25–CombiningTechnologiesandMethodsPage |39

Herearesomesoftwaredevelopmenttechnologieswhichareofinterestintheliteratureandresearch:

AutomatedVerificationManagement

FormalRequirementsSpecifications

RequirementsandTraceabilityAnalysis

FormalMethods

ComputerAidedSystemEngineering

V&VRunTimeDesign

RigorousAnalysisforTestReduction

RequirementsandDesignAbstraction

Probabilistic/StatisticalTest

TestingMetrics

Itwouldbevaluabletoexaminesomeofthesetechnologieswiththenewinformationobtainedfromthisstudy.

Selectionoftheemergingtechnologiestobeevaluatedshouldbeguidedbythe“lessonslearned”inresearch

effortssuchasVVIACS(Validation&VerificationofIntelligentandAdaptiveControlSystems),CerTAFCSCPI

(CertificationTechniquesforAdvancedFlightCriticalSystems–ChallengeProblemIntegration),andMCAR(Mixed

CriticalityArchitectureRequirements). SeveraltechnologiesincludingAutoCode,AutoTest,RapidPrototyping,

SystemModelBased,andSimulationBasedDesignarematureenoughtoalreadybeestablishedwithrecognized

benefits.

Futureresearchshouldincludeanalysisofsomeadditionalprogramstoreflectalargervarietyofsoftware

developmentprocesses.

REFERENCES

[1]Goddard,P.L.,“SoftwareFMEATechniques”,ProceedingsoftheAnnualReliabilityandMaintainability

Symposium,January2000.

[2]Goddard,P.L.,“ValidatingtheSafetyofEmbeddedRealTimeControlSystemsusingFMEA”,Proceedingsofthe

AnnualReliabilityandMaintainabilitySymposium,January1993.

[3]Jackson,D.,Thomas,M.,andMillett,L.,Eds.SoftwareforDependableSystems:SufficientEvidence?National

ResearchCouncil.NationalAcademiesPress,2007.REPORT DOCUMENTATION PAGE Form Approved

OMB No. 0704-0188

2. REPORT TYPE 

Contractor Report

 4. TITLE AND SUBTITLE

Concept Development for Software Health Management 

5a. CONTRACT NUMBER

NNL06AA08B

 6. AUTHOR(S)

Riecks, Jung; Storm, Walter; Hollingsworth, Mark

 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

NASA Langley Research Center

Hampton, VA 23681-2199 

 9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)

National Aeronautics and Space Administration

Washington, DC 20546-0001

 8. PERFORMING ORGANIZATION

 REPORT NUMBER

10. SPONSOR/MONITOR'S ACRONYM(S)

NASA

13. SUPPLEMENTARY NOTES

Langley Technical Monitor: Eric G. Cooper

12. DISTRIBUTION/AVAILABILITY STATEMENT

Unclassified - Unlimited

Subject Category 06

Availability: NASA CASI (443) 757-5802

19a. NAME OF RESPONSIBLE PERSON

STI Help Desk (email: help@sti.nasa.gov)

14. ABSTRACT

This report documents the work performed by Lockheed Martin Aeronautics (LM Aero) under NASA contract

NNL06AA08B, delivery order NNL07AB06T. The Concept Development for Software Health Management (CDSHM)

program was a NASA funded effort sponsored by the Integrated Vehicle Health Management Project, one of

the four pillars of the NASA Aviation Safety Program. The CD-SHM program focused on defining a structured

approach to software health management (SHM) through the development of a comprehensive failure taxonomy

that is used to characterize the fundamental failure modes of safety-critical software.

15. SUBJECT TERMS

Software, failure, health management, safety-critical, taxonomy

18. NUMBER

 OF 

 PAGES

40

19b. TELEPHONE NUMBER (Include area code)

(443) 757-5802

a. REPORT

U

c. THIS PAGE

U

b. ABSTRACT

U

17. LIMITATION OF 

 ABSTRACT

UU

Prescribed by ANSI Std. Z39.18

Standard Form 298 (Rev. 8-98)

3. DATES COVERED (From - To)

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

5d. PROJECT NUMBER

5e. TASK NUMBER

NNL07AB06T

5f. WORK UNIT NUMBER

645846.02.07.07

11. SPONSOR/MONITOR'S REPORT

 NUMBER(S)

NASA/CR-2011-217150

16. SECURITY CLASSIFICATION OF:

The public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, 

gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this 

collection of information, including suggestions for reducing this burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and 

Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person 

shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number.

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.

1. REPORT DATE (DD-MM-YYYY)

01- 05 - 2011