Certification Authorities Software Team (CAST)

Position Paper

CAST-7

####### Open Problem Report (OPR) Management for Certification

Completed August, 2001

####### Open Problem Report (OPR) Management for Certification

# 

# **1. SCOPE**

> This paper discusses issues related to open problem report management
> and expands on DO-178B/ED-12B, section 11.20j, for software status
> reporting. Many software development teams do not distinguish between
> problem reports and change requests (e.g., they use a tool to address
> both at the same time). This discussion paper focuses only on problem
> reports, as addressed in ED-12B/DO-178B guidelines.

## **1.1 Background**

> ED-12B/DO-178B is an assurance standard; therefore, software developed
> to be compliant with ED-12B/DO-178B objectives has a better assurance
> of behaving as intended (i.e., specified) and having minimal
> unintended behaviour (i.e., deviation from specification or
> unspecified behaviour). In ED-12B/DO-178B, software specification is
> contained in the High-Level Requirements (HLR).
>
> However, most software baselines have known problems (referred to as
> Open Problem Reports or OPR's), which affect strict compliance to
> specification. (Note: These problems can come from a lack in the
> specification or HLR themselves).
>
> There can be many types of OPR's on a particular software baseline.
> Such OPR's can identify deviation from the requirements for the
> baseline. Such software may not behave as intended and can even show
> some unintended/adverse side effects. This could make it difficult or
> impossible to comply with JAR/FAR xx.1301 and xx.1309.
>
> This paper gives some recommendations for applicants to deal with
> OPR's and still meet JAR/FAR objectives and to satisfy the objectives
> of ED-12B/DO-178B.

## **1.2 Objectives**

> This paper has two objectives:

1.  To clarify the role of the aircraft manufacturer and the equipment
    supplier in the assessment of limitations of a piece of equipment
    embedding software because of known problems at the time of
    certification.

2.  To help the certification authority in its assessment of the piece
    of equipment by requiring inclusion of OPR's in the Software
    Accomplishment Summary.

# 

# **2. GLOSSARY**

> Text in *italic* is extracted from ED12B/DO178B glossary.
>
> **[Error]{.underline}**: *With respect to software, a mistake in
> requirements, design or code.*
>
> **[Defect]{.underline}**: an "error" in executable code. Under
> specific conditions, the SOFTWARE is not performing as needed.
>
> **[Fault:]{.underline}** *A manifestation of an error in software. A
> fault, if it occurs, may cause a failure.*
>
> **[Failure]{.underline}**: *The inability of a system or system
> component to perform a required function within specified limits. A
> failure may be produced when a fault is encountered.*
>
> **[Failure condition:]{.underline}** *The effect on the aircraft and
> its occupants both direct and consequential, caused or contributed to
> by one or more failures, considering relevant adverse operational and
> environmental conditions. A failure condition is classified according
> to the severity of its effect as defined in FAA AC 25.1309-1A or JAA
> AMJ 25.1309.*
>
> The usual chain of events is Error  Defect  Fault  Failure 
> Failure condition.
>
> The term **defect** is added to highlight the difference between
> errors in written requirements and error in the executable code.

# **3. OPR Management Principles**

## 

## **3.1 Principles** 

> The two principles below establish the foundation of this position
> paper.

1.  Possible effects of OPR's should be analyzed to specify operational
    limitations and to ensure safety of the system (i.e., compliance to
    JAR/FAR).

2.  If not addressed, OPR's may lead to deterioration of software
    quality (i.e., safe behaviour) when people are faced with software
    modification. This is typical of complex systems.

## 

## **3.2 Recommendations** 

> From these two principles, the following recommendations can be
> derived:

1.  All OPR's should be analysed so that deviation from the
    specification (HLR) can be determined. An OPR with multiple root
    causes (i.e., with multiple errors/defects implicated) may be split
    into different related OPR's in order to conduct the safety effect
    analysis.\
    An OPR should be evaluated again in the new baseline in order to
    claim credit from previous acceptance.

2.  All OPR's should be included in the Software Accomplishment Summary
    (see ED12B/DO178B, Section 11.20).

3.  All deviations from HLR should be analyzed to determine the
    potential affect on aircraft safety, operations, and/or crew
    workload.

4.  Even if the equipment manufacturer has the sufficient knowledge to
    explain the functional effect on the equipment/item for an OPR, only
    the aircraft manufacturer can assess or confirm the potential
    operational effect.

# 

# **4. Recommendations on OPR assessment**

## 

## **4.1 Types of OPR's**

> This section classifies OPR's into different types. The different
> types determine the expectation of certification authorities on OPR
> assessment.
>
> An OPR logged when developing a software build with a methodology
> compliant to ED12B/DO178B can be classified depending on the origin of
> the problem report as:

1.  Known deviation from methodology (plans), which can lower the
    assurance for the software but not necessarily induce errors or
    defects into the software.

2.  A failed test case due to a incorrect test case or an actual
    software error.

3.  Error in a requirement or development standard

> Note: If the software is not behaving properly; this is the evidence
> of a defect. This kind of error can be raised through any kind of
> execution of the software. It is then considered a type 2 error.

## **4.2 Guidelines on the analysis of an OPR**

> This section details the recommendations to evaluate the PR depending
> on its origin.

### 

### **4.2.1 Plans/methodology (type 1)**

> In this case the assurance has been compromised. The equipment
> supplier should show that the possibility for a software defect to be
> introduced or not detected is mitigated by another activity or a new
> specific activity in order to restore equivalent assurance.
>
> This should be stated in the Software Accomplishment Summary for
> coordination with the certification authority.
>
> *Note that for a problem on a process itself DO-248B/ED-94B FAQ #17*
> *applies and provides further explanations.*

### 

### **4.2.2 Failed test (type 2)**

> Tests include all possible execution of the software. This includes
> flights (either by aircraft manufacturer or airline) so that an ISP
> (In Service Problem) can lead to a software OPR.

#### *4.2.2.1 Error in the test*

> If the test is incorrect, the functionality (i.e., software
> requirement) may be verified by another way (including analysis, bench
> testing, flight-testing) or by correcting the test. Any possibility of
> inconsistencies in the requirements should be investigated and
> analysed for possible effects.

#### *4.2.2.2 Error in the requirement*

> This is a type 3 error, see 4.2.3 below.

#### *4.2.2.3 Defect evidence (error in the executable code)*

> The root cause of the software failure should be determined (a root
> cause analysis should be conducted in order to determine software
> defects).
>
> Software defect effects should be analysed to determine all potential
> software failures.
>
> Software failure should be given to the Safety Assessment Process in
> order to assess software compliance to safety/regulatory requirements.

### **4.2.3 Deviation from standards (design, coding...) or requirement documents (type 3)**

> In this case, the possible software defects can be induced directly
> from the design.
>
> Software deviation should be analysed for all potential impact on the
> software.
>
> Impacts on software should be given to the Safety Assessment Process
> in order to assess software compliance to safety/regulatory
> requirements.
>
> Such an OPR should be corrected as soon as possible (and at least when
> modifying the applicable Configuration Item). The Configuration Item
> affected should be traceable to the OPR. (See ED-12B/DO-178B Section
> 7.)

**5. Proposed presentation of OPR's in the Software Accomplishment
Summary**

The minimal problem report information that should be included in the
Software Accomplishment Summary is defined in DO-178B/ED-12B as:

1.  Identification of the OPR (configuration management number)

2.  Short description of the OPR (one or two sentences), including
    system/safety effect

3.  Mitigation for certification acceptability (justification, why this
    OPR can be left open)

4.  OPR interrelationships (if needed)

Below is an example of a table that could be included in the Software
Accomplishment Summary.

+-------+--------+----+------------+-----------+---------+-----+----+
| >     | > OPR  | >  | > D        | >         | >       | >   | >  |
|  Date | >      |  T | escription |  Severity |  Justif | OPR | St |
|       | Number | it |            |           | ication | >   | at |
|       |        | le |            |           |         | rel | us |
|       |        |    |            |           |         | ati |    |
|       |        |    |            |           |         | ons |    |
|       |        |    |            |           |         | hip |    |
+-------+--------+----+------------+-----------+---------+-----+----+
| >     | > C    |    | > Possibly | > Categ   | > Why   | >   | >  |
|  Date | onfigu |    | >          | orization | > the   |  Ot |  N |
| >     | ration |    |  including | > to be   | > eq    | her | ew |
|  when | > mana |    | > other    | > defined | uipment | >   | /O |
| > the | gement |    | > OPR's    | >         | > could |  OP | ld |
| > OPR | >      |    | > related  | depending | > be    | R's | >  |
| > was | Number |    | > to this  | > on the  | > e     | > r |  ( |
| > o   |        |    | > one      | > system  | mbedded | ela | ve |
| pened |        |    |            | > effect  | > with  | ted | rs |
|       |        |    |            |           | > this  |     | io |
|       |        |    |            |           | > OPR   |     | n) |
+-------+--------+----+------------+-----------+---------+-----+----+