# Presentation: 243844

## Four Mega-Challenges Facing The Commercial “Software Quality Movement”: Definition, Composition,  Certification, and Commercialization and Return-On-Investment

**Four Mega-Challenges Facing The Commercial “Software Quality Movement”:**** ****Definition, Composition,  Certification, and Commercialization and Return-On-Investment**

- Jeffrey M. Voas
- Chief Scientist

## Definitional Problem

## Quality Issues

- What is Software Quality?
- What are the ingredients in Quality?
- Are the ingredients measurable?
- Are all ingredients of equal importance?
- Can we create a mathematical equation for Quality?
- What are the economic and technical tradeoffs between the ingredients?

## Ask the CIO

- 	A:  What do you mean by that?

-    Q: Are you interested in quality software?

-    Q: Are you interested in software quality?

- 	A:  Yes.

## What is Software Quality?

-    Subjective term that produces confusion among _most _software engineering professionals

## IEEE

- “*Totality of features of a software product that bears on its ability to satisfy given needs*.” *[Source: IEEE-STD-729]*
- “*Composite characteristics of software that determine the degree to which the software in use will meet the expectations of the customer*.”* **[Source: IEEE-STD-729]*

## Three High-Level Attributes

- Quality Software

## High-Level Attributes

_**Problem:**_**  Intuitive, but not formal**

- Quality Software

## Lower-Level Attributes

**Lower-Level Attributes**

- Reliable/
- accurate

- Secure/
- private

- Timeliness

- reliability

- security

- performance

- availability

- privacy

- fault tolerance

- fault tolerance

- confidentiality

- intrusion tolerance

- testability

- Non-functional attributes (“ilities”)

- Quality Software

- Functional attributes

##    Software Quality (or Quality Software) must be viewed/defined as some combination of: (1)  the degree to which the functional requirements are met, as well as, (2) the degree to which the non-functional requirements are met.

**Position Statement**

- reliability

- security

- performance

- availability

- privacy

- fault tolerance

- fault tolerance

- confidentiality

- intrusion tolerance

- testability

- Non-functional attributes (“ilities”)

**+**

- Functional attributes

##    Software Quality then is some combination of the following functional and non-functional attributes:

		Reliability [R], Performance [P], Safety [Sa]
		Fault Tolerance [F], Security [Se], Availability [A]
		Testability [T], and Maintainability [M]

- 		Reliability [R], Performance [P], Safety [Sa]
- 		Fault Tolerance [F], Security [Se], Availability [A]
- 		Testability [T], and Maintainability [M]

**Position Statement**

##    Software Quality can also be viewed as some combination of the previous attributes PLUS:

		Scalability, Usability, Sustainability, Survivability,
		Interoperability, Extensibility, Reusability,    	Readability, etc.

- 		Scalability, Usability, Sustainability, Survivability,
- 		Interoperability, Extensibility, Reusability,    	Readability, etc.

**However ....**

##    Q = aR + bP + cF + dSa + eSe + fA + gT + hM

   where a, b, c, d, e, f, g, and h are units of quantitative or qualitative measures of a particular attribute.

-    where *a, b, c, d, e, f, g,* and *h* are units of quantitative _or qualitative_ measures of a particular attribute.

**Eight in an Equation?**

## Key Problems

**     **

- The equation cannot be linear, since the units of measure for each attribute cannot be standardized (the apples and oranges problem).
*d*Sa = *Q* – (*a*R + *b*P + *c*F + *e*Se + *f*A + *g*T + *h*M)
- Most “ilities” are not quantifiably measurable.
- Reliability, Availability, and Performance are measurable (via testing).

## For Example ...Maintainability

**     **

- Size, defect density, amount of testing, T, R, cohesion, coupling, documentation, complexity, depth of inheritance, number of objects, testing infrastructure, mean-time-to-repair, experience of maintenance personnel as well as their domain knowledge, existence of impact analysis tools, etc., all impact M.
- Q: So how can you assign a single numerical score for M?

## Security

**     **

- The level of security of an information system is a function of the partially unknown threat space, that changes by the minute.
- Q: So how can you assign a single numerical score for Se?
- A: You can assess, for a bounded set of anticipated threats, how the system will respond to those, e.g., 100 known threats, 50 mitigated.
- A: Or you could measure the percentage of patches that are installed based on the number that need to be, and then test to make sure those installed work.  Such information could also be used to give security “a score” but once again, that is only a score based on known threats and available patches.

## The “Culprit” Phenomenon

**The “Culprit” Phenomenon**

## So Where Does That Leave Us?

**Without a Numerical Quality Equation, But With a Way to Discuss the Attributes of Quality Software,and Therefore With a Means for Industry to Define Quality Goals**

**So Where Does That Leave Us?**

## Composition Problem

## Composition Issues

- _The Goal_:  Make software engineering into a real engineering discipline.
- The composition problem is a _behavioral prediction_ problem.

## Two Software Components

- 

- 

**Two Software Components**

## With Attributes

-  has the following properties:
-   (*a*R, *b*P, *c*F, *d*Sa, *e*Se, * f*A, *g*T, *h*M)
-  has the following properties:
* **(i*R, *j*P, *k*F, *l*Sa, *m*Se,* n*A, *o*T, *p*M)

- 

- 

**With Attributes**

## What Have You Got?

- Then *f*(  ) will inherit some level of Quality from the individual components.  Is that level of quality an integer? Probability? An n-tuple of values? Color coded (green red yellow)?
- Key Point:  The Composite Quality must represent something from which _predictions of future behavior_ can be made.

- 

- 

**What Have You Got?**

## Key Problems

**     **

- It is hard enough to know, with any _precise_ accuracy, what the composite reliability score will be as a result of the *a* and *i* values (let alone for the non-functional attributes).
- But an even greater challenge exists here.  For example, the security mechanisms in component  could thwart the performance that is built into component .
- Attributes are only reasonable to talk about within the context of a system, i.e., it is not reasonable to talk about them and attempt to measure them as standalone component properties.  Their eventual target environments must weighed into their individual assessments.

## Environment

**Environment**

- Reliable/
- accurate

- Secure/
- private

- Timeliness

- reliability

- security

- performance

- availability

- privacy

- fault tolerance

- fault tolerance

- confidentiality

- intrusion tolerance

- testability

- Non-functional attributes (“ilities”)

- Quality

***Operational environment***

## So Where Does That Leave Us?

**In Search of a Calculus or Calculi for Predicting How a Composite System Will Behave in the Future in a Specific Environment**

**So Where Does That Leave Us?**

## Product Certification and 
Software Engineering Standards
To Aide the Composition Problem

Standardized Parts?

- Software Engineering Standards
- To Aide the Composition Problem
- Standardized Parts?

##  What is a Standard?

**Ideally, it is a line in the sand from which a certificate of ****compliance**** can be written.**

** ****What is a Standard?**

## Pros

**Any bar or hurdle is better than no  bar or hurdle**

## Cons

**Possibly the developers would have done ****more**** to improve quality but now feel they have a license to do ****less****.**

## Premise for SW Product Certification

**Commercially built software should be tagged ****with some guarantee (or **_**at least**_** a “warm fuzzy”) as to how good the software should be.**
**Problem: Software Of Unknown Pedigree (****SOUP****)**
**Goal of Product Certification: ****SO****(****K****nown)****P**

## Three Schools of Thought

**Processes**

**Products**

**People**

**All SE standards **
**incorporate**
**one or more **
**of these **
**perspectives**

**Three Schools of Thought**

**Notes:**
1. Please explain what this slide is trying to convey.

## 1. Process: Clean Pipes, Dirty Water?

**1. Process: Clean Pipes, Dirty Water?**

**Certifying that you know how to do things correctly does not mean that you do them correctly!**

## 2. People

**2. People**

## 3. Product: The Software Itself

**3. Product: The Software Itself**

-     Spectrum of possibilities as to what a certificate proclaiming that some “quantified” level of quality has been built in  could state --- it  could say anything in the range between “Nothing” (*e.g.*, “here is a piece of software”, etc.) to “This software will always work perfectly under all conditions” (i.e., a 100% guarantee of perfection).

**0% confidence**

**100% confidence**

## But Problems Exist With Standards

- Vague:  *Develop software that only does "good" things*
- Common sense "dos" and "don'ts"   - Very watered done by voting time
- Disclaimers by publishing organizations
- Profitable to organization that publishes them
- Used only if mandated
- Return-on-investment is unknown
- Thwart intellectual creativity
- "Protectionist" legislation
- Paperwork
- 2167A: ~400 English words per Ada code statement
- "Old news" before being ratified
- Relating one to another is very hard
- Hundreds in existence
- Cannot be easily tested for compliance
- Mis-certifications are possible

## Different interpretations

Lack of fairness during certification judgment 

So much legacy code exists that complies with no standards and therefore get excluded in heterogeneous systems, making it’s impact to the system unknown.

- Lack of fairness during certification judgment
- So much legacy code exists that complies with no standards and therefore get excluded in heterogeneous systems, making it’s impact to the system unknown.

- Suppose you have the following logical expression:
- (A and B) or (B and C) or (A and C)
- where A, B, and C are Boolean variables
- To meet verification requirements for Level A software in RTCA DO178-B, you need to know the number of *conditions* in this statement
- 	_Condition_: A Boolean expression containing no 	Boolean operations
- How many conditions are there?  3, 4, 6, or 9

**Example of “Standards”  Confusion**

- [Source: “Challenges in Software Aspects of Aerospace Systems”, K. Hayhurst & C.M. Holloway, Presented
-  at the 26th Software Engineering Workshop, Greenbelt, MD, November 28, 2001]

**The FAA Says ...**

- Distribution of Responses from 39 FAA Certification Authorities

**35.9**

**17.9**

**41.0**

**5.1**

- [Source: “Challenges in Software Aspects of Aerospace Systems”, K. Hayhurst & C.M. Holloway, Presented
-  at the 26th Software Engineering Workshop, Greenbelt, MD, November 28, 2001]

## 6

**And the Answer is ...**

- [Source: “Challenges in Software Aspects of Aerospace Systems”, K. Hayhurst & C.M. Holloway,
- Presented at the 26th Software Engineering Workshop, Greenbelt, MD, November 28, 2001]

**Notes:**
1. What does the acronym MTTF stand for?  Will your audience know it?

- (A and B) or (B and C) or (A and C) has 6 conditions
- The full definition for condition is not contained in the glossary entry for that term
- Part of the definition is given in the entry for decision
- 	_Decision_: A Boolean expression composed of conditions and zero or more Boolean operators.  A decision without a Boolean operator is a condition.  If a condition appears more than once in a decision, each occurrence is a distinct condition.

**Explanation**

- [Source: “Challenges in Software Aspects of Aerospace Systems”, K. Hayhurst & C.M. Holloway, Presented
-  at the 26th Software Engineering Workshop, Greenbelt, MD, November 28, 2001]

## So Where Does That Leave Us?

**In Need of More Precise, Less Vague, and Repeatable Processes, for Grading The Quality of Software**

**So Where Does That Leave Us?**

## Commercialization and 
ROI Issues

- ROI Issues

## Commercialization Issues

**     **

- Proven technology? (empirical vs. anecdotal)
- Prototypes?  Are they Maintainable/Extensible or Trashware?
- Scalable?  Theoretical or Practical?  Maturity?
- Automated? Is it a solution or standalone?
- What languages/architectures does it support?  Fad/Lifetime?
- Difficult to learn? Ease of use? Time-to-market enabler or disabler?
- Client base:  commercial or government?  Number of site?
- Evolutionary (leap frog-able) vs. Revolutionary?

**     **

- Compatible with existing technology (*e.g*., Microsoft)?
- Point of Origination: University? Small business?, Large Corporation?, Government?
- Competing foreign technologies?
- Process, People, and Product - oriented?
- Which attribute(s) does it address?  Measurement or design? If design, how much of that attribute can it offer?
- SOUP or SOKP?
- Does this technology self-certify Quality after use?

## Final Thoughts

## 1. Quality is a Recipe

**     **

- As with food, ingredients of different types (liquids, powders, vegetables, meats, etc.) can all be mixed together.
- What food tastes like is a function of the ingredients and their proportions.
- Quality Software can be viewed/defined in a similar manner.
- And certain ingredients overpower others.

- Software engineering standards are completely necessary despite their limitations. They usually are a “good rule of thumb”, but not an absolute process for achieving perfection.
- Virtually any standard beats development chaos
- What is Missing in SE Standards?
- Not technology!
- How to Implement,
- How to gain regulatory approval given uncertain knowledge as to how judgment will be rendered,
- Fairness in the certification processes,
- And ROI (most are anecdotes, not statistical studies)

**2. Standards Beat Chaos**

- Until the non-functional attributes of software components can be graded, and assumptions about the target environments of those components can be nailed down (HUGE PROBLEM), *a priori* certification of the quality of any software component is suspect.
- Research is needed into how to compose both functional and non-functional attributes.

**3. Only Product Certification Can Address The Composition Problem**

## 4. Attributes Need to Be Pre-Defined

**     **

- Requirements should prescribe at some level of granularity as to what the weights are for various “ilities”, as well as how much of each “ility” is desired.
- But HOW?
- Ignoring the non-functional attributes is not an option for high assurance and trustworthy systems!  Make an attempt to discuss them with the client even if quantification is not possible.  Just get the issue on the table!

## 5. Weighting is Important

*w**1*R 		 *w**2*P  		*w**3*F 		 *w**4*Sa
*w**5*Se 		*w**6*A  		*w**7*T 		*w**8*M
- in order to not _over-design_ any attribute into the system.
- For example, for an e-commerce application, *w**4* would probably equal 0.0 and *w**7 *would also be less than something like *w**2*

**5. Weighting is Important**

**     **

- How much will you spend for increased reliability
- knowing that doing so will take needed, financial
- resources away from security or performance or ...?

**6. Tradeoffs**

- Security vs. Performance
- Fault tolerance vs. Testability
- Fault tolerance vs. Performance
- etc.

## Counterintuitive Realities

**     **

- 100% safety and 0% reliability
- 100% reliability and 0% safety
- 0% functionality/reliability and 100% security
- 100% availability and 0% reliability
- 100% availability and 0% performance
- 0% performance and 100% safety

**     **

- .... is alive and well.  There are still many interesting and
- fascinating software engineering research challenges
- to pursue that not only will benefit NASA, but
- industry at-large.

**7. The Software Quality Movement ...**

## 21351 Ridgetop CircleDulles,  VA  20166 USA www.cigital.com

**21351 Ridgetop Circle****Dulles,  VA  20166 USA ****www.cigital.com**

- Jeffrey Voas
- phone: 703.404.9293
- e-mail: voas@cigital.com