# Presentation: 371582

## NASDA IV&V ...........

***July 29th, 2003 @NASA IV&V Facility***
- Naoki Ishihama, NASDA
- Hideki Nomoto, JAMSS/MIT
- Haruka Nakao, JAMSS

## Contents

- Introduction of NASDA IV&V approaches
- Project IV&V activities’ status and findings
- Satellites IV&V (SELENE, WINDS)
- JEM Pre-Shipping IV&V
- JEM IV&V
- HTV IV&V
- Centrifuge IV&V
- H-IIA Safety Range Ground system IV&V
- New Methodology Research
- Software QA activity and Process Improvement

## NASDA IV&V Functions

- Project IV&V (Assessment ReportProject Evaluation)
- Development Independent Assessment
- Shipping Validation, Pre-Launch Review
- Non-conformance Investigation/Assessment
- Research of IV&V methodology and tool

- Criteria
- Handbook
- Tools

- IV&V Planning

- Software Process Improvement

- Software Quality Assurance

*1. Introduction of NASDA IV&V approaches*

**Notes:**
計画から立案、
どういう種類のIVVがあるのか? 3つのIVVがある。
また、IVV活動においては、プロセス改善側との情報交換を相互に行う。

At first, we made a criteria , handbook and tools.
Using these things, we make planning of IVV activity. On this planning, we execute project IVV ( assess reports, and evaluate the project ).
As this presentation Project IVV, we executed  “development independent assessment” , “shipping validation pre-launch review” , “Non-conformance Investigation/Assessment”.
And iterate these flow.
At other side, we execute software process improvement, feedback the result of this improvement to Project IVV. The other way of this relationship, 
Feedback the result of Project IVV to software process improvement.

## IV&V approaches

*1. Introduction of NASDA IV&V approaches*

**Notes:**
This map is made by NASDA IVV guide line.
This is a time flow. (from top level (system analysis) ...requirements analysis .....
Upper side is process assessment. Down side is product assessment. 
As product assessment,
At this each phase( system analysis, requirements analysis....)  we made a SpecTRM model and check completeness and consistency of this model and check the equivalency between each phase.
And we make a operational model using the visio (visio is microsoft draw soft), translate to SpecTRM model, and now we try to check the equivalency between operation model and requirement model.
And we do the reachability analysis using SpecTRM model and Hazard model. 
At the coding phase, we check the code using tools. By this code checking, we get the code structure, memory leak, error handling information, et all. And we get a flow chart by reversing source codes, check the consistency between source code and document. This is reverse engineering.
As process assessment, 
If system use re-use software, we validate which part is re-used, analyze the verification coverage, and review the test result.  

## Satellite IV&V Activity- WINDS, SELENE -

*2. **Project IV&V activities’ status and findings*

**SELENE**

**WINDS**

**Notes:**
about satellite, we tried to do IVV of WINDS, SELENE attitude control software requirement.

WINDS is the Wideband InterNetworking engineering test and Demonstration Satellite.
SELENE(SELenological and ENgineering Explorer), first large lunar probe in Japan, developed in the first ISAS/NASDA joint lunar program will be launched by H-IIA launch vehicle. 
Both satellite will be launched in 2005 Years.

## Satellite IV&V

- _Project Characteristic_
- Basically non 2FT system (Fail safe, 1 fail operative)
- One third of software/firmware are re-use.
- High level specification defines detailed control theory and logic.
- High speed calculation has become to be required.

*2. **Project IV&V activities’ status and findings*

**Notes:**
When we do IVV activity, it is necessary to perform not all cases but to perform activity united with the project characteristic and situation and development phase.

In case of satellite, we has some project characteristic.
Satellite system is basically non 2FT system. 
One third of satellite software / firmware are re-use.
High level specification defines detailed control theory and logic (satellite attitude control software requirement )
High speed calculation has become to be required to control satellite.

## Modeling of Satellite Attitude Control Software Requirement using UPPAAL

- Real-time Application
- Hard to deal with time constraints
- “What” and “How” should we do?
- We used UPPAAL.
- UPPAAL:
- Model networks of extended timed automata with typical data
- Verify an invariant and reachability of the model

*2. **Project IV&V activities’ status and findings*

**Notes:**
Attitude control software requirement is written by data flow chart. 
It is difficult to use SpecTRM tool for attitude control software requirement.
So, to analyze this document, we used UPPAAL 
It is easy to make modeling by using UPPAAL. Because UPPAAL model is data flow diagram.

On designing of a real-time application, it often becomes hard to deal with time constraints.  
For time critical system, this is a very important point. about this case, system stop by a little deviation and cannot be rebooted, so that we lost the system. Or a serious damage is given to a system.
This is a big problem.
So, as first trying, to find these problem at early phase, we used UPPAAL, which can model networks of extended timed automata with typical data such as boolean variables and integer variables. UPPAAL can also verify an invariant and reachability of the model.

## Modeling of Satellite Attitude Control Software Requirement using UPPAAL

***Modeling of Satellite Attitude Control Software Requirement using UPPAAL***

***Demo #1 !***

*2. **Project IV&V activities’ status and findings*

**Notes:**
This is a main routine. When the state is here, main routine call a subroutine.
A value is divided by 3, and when remainder is ret = 1, it passes along this.
In this demo, It is setting up so that a remainder may be ret = 1.
But, there is a random function in this demo. 
This routine run as the process different from the main.
Random function set the data arg1 = 5.
So that, remainder ret data = 2.
System designer designed arg1 data is set to only 13.
But by the timing running the random function, arg1 data is set to 5.
The system which have a non-deterministic data is very dangerous.
It is very dangerous for system.
So, using the UPPAAL, we can find the timing problem of time critical system.

## Pre-shipping Verification for Japan Experiment Module (JEM) Software

*2. **Project IV&V activities’ status and findings*

## Pre-shipping Verification for JEM Software

- _Project Characteristic_
- All Development Activity had been done.
- Reverse Engineering
- Monstrous and complex on-board system relatively (DB)
- Need tool support
- Long term development and operation
- Need to keep the design rationale and records

*2. **Project IV&V activities’ status and findings*

**Notes:**
About Pre-......
We thought, “what could we do at JEM IVV activity? “what is JEM characteristic?”
So that,  we got some information.

JEM all development activity had been done. Therefore using the on-board source code and product document (software specification document), we need to reverse engineering.
Monstrous and complex on-board system. DB system is very monster and complex. So that, we need to support using tool.
long term development and operation. So we need to keep the design rationale and records.

## Equivalency Checking Tool of source code (ECT)

- Equivalency Checking Tool of source code (ECT) is a reverse engineering tool that generates a flow diagram from a source code.
- Modifying GCC(GNU Compiler) 2.95.3 and DOT included in GRAPHVIZ tool set

*2. **Project IV&V activities’ status and findings*

**Notes:**
Now, by Using the UML tool we can develop the system, therefore we can get the C++/ JAVA スケルトンsource code.
And C++, JAVA source code can reverse source code.
But C language can not use UML TOOL.
Spacecraft on-board software can be made by C/ Assembler Language. So we can not use UML tool to on-board software.
On-board source code is made by a software detailed design specification.
This specification is almost made by flow diagram of each software components.
How can we analysis the relationship between specification and source code?
How can we check a equivalency between specification and source code?
Then we made a reverse engineering tool ECT.
ECT is Equivalency Checking Tool of source code.
ECT is a reverse engineering tool that generates a flow diagram from on-board source code (C Language).
It is made up by modifying GCC(GNU Compiler) 2.95.3 and by the tool called “dot” included in “graphviz” tool set.

## Equivalency Checking Tool (ECT)

***Equivalency Checking Tool (ECT)***

***Outline of tool procedure***

*2. **Project IV&V activities’ status and findings*

**Notes:**
This is the outline of our tool procedure.
First of all, the preprocessor of GCC is applied to a source code.
Then, the parse of GCC is applied to a source code. The output of parser is modified so as to generate a medium code. The flow diagram is finally generated by “dot”. “dot” is a language to use the drawing the map. Graphviz is open source graph drawing software.

## Equivalency Checking Tool (ECT)

- void start_waiting_jobs ()
- {
-   struct child *job;
-   if (waiting_jobs == 0)
-     return;
-   do
-     {
-       /* Check for recently deceased descendants.  */
-       reap_children (0, 0);
-       /* Take a job off the waiting list.  */
-       job = waiting_jobs;
-       waiting_jobs = job->next;
-       /* Try to start that job.  We break out of the loop as soon
-        as start_waiting_job puts one back on the waiting list.  */
-     }
-   while (start_waiting_job (job) && waiting_jobs != 0);
-   return;
- }

***Source Code***

***Flow Diagram***

- Reverse Engineering

***Equivalency Checking Tool (ECT)***

**Notes:**
This source code is a part of GNU make 3.79 source code. It is a source of a function named “start_waiting_jobs”.
Using the ECT, reverse a source code of “start_waiting_jobs”.
So, this flow diagram is the final generated flow diagram of function “start_wait_jobs”.

## Equivalency Checking Tool (ECT)

***Equivalency Checking Tool (ECT)***

***Demo #2 !***

**Notes:**
This source code is a part of GNU make 3.79 source code. It is a source of a function named “start_waiting_jobs”.
Using the ECT, reverse a source code of “start_waiting_jobs”.
So, this flow diagram is the final generated flow diagram of function “start_wait_jobs”.

## Equivalency Checking Tool (ECT)

-    label=”job.c:1583: void start waiting jobs()”;
-    node [shape=box];
-    edge [weight=100];
-    start [shape=ellipse, height=0.01, label=”START”];
-    subgraph if 1586{
-       node [shape=box];
-       edge [weight=100];
-       Na1586 [shape=diamond, label=”1586: if (waiting jobs==0)\n”...
-       start -> Na1586;
-       Nb1587 [shape=ellipse, label=”1587: return;”, fontsize=14];
-       Na1586 -> Nb1587 [taillabel=”Y”];
-    }
-   .......
- }

***Medium Code***

***Equivalency Checking Tool (ECT)***

*2. **Project IV&V activities’ status and findings*

**Notes:**
This is a medium code of “start_wait_jobs” which is for tool of “dot”.

## Lesson and Learned

- Flow diagram and additional information can be clearly compared with detailed design specification to determine the equivalency and differentials.
- realize misunderstanding and miscommunication between programmer and system designer

*2. **Project IV&V activities’ status and findings*

**Notes:**
The generated flow diagram and additional information such as interface definition can be clearly compared with detailed design specification to determine the equivalency and differentials.
Equivalency checking is one of technique enough to realize misunderstanding and miscommunication between programmer and system designer.

## Light weight code checking tool: Splint

- Comment (warning) is different with compiler.
- Most security attacks exploit instances
- Many of these flaws could be detected and eliminated before software is deployed
- Not been integrated into the software development
- We used the Splint.
- Lightweight static analysis to detect common security vulnerabilities
- Coding mistakes (to avoid specific cord pattern which caused the past non-conformance)

*2. **Project IV&V activities’ status and findings*

**Notes:**
When we compile C/C++ software, we use GCC compiler and other compiler.
GCC compiler is most famous compiler is C/C++ programming.
Depending on system, we use the different compiler.
If we use different compiler at same source code, perhaps we can get different comment. Actually, there is unclear information in source code .

Most security attacks exploit instances of well-known classes of implementation flaws. Many of these flaws could be detected and eliminated before software is deployed. These problems continue to be present with disturbing frequency, not because they are not sufficiently understood by the security community, but because techniques for preventing them have not been integrated into the software development process. 

So, at first trying , we used the lint tool, which is splint, that uses lightweight static analysis of C program to detect common security vulnerabilities (including buffer overflows and format string vulnerabilities) and can be readily extended to detect new vulnerabilities and checks coding mistakes. With minimal effort, Splint can be used as a better lint. If additional effort is invested adding annotations to programs, Splint can perform stronger checking than can be done by any standard lint.

## Benefit of Lint tool

***GCC compiler can’t find this area.***

## Light weight code checking tool: Splint

- void attitude_jobs ()
- {
- ......
- ......
-   switch (id_flg)  {
-        case 1:
-              control();
-      ......
-      case 2:
-              stop_attitude();
-      ......
-      case 3:
-              restart_attitude();
-  ......
-   ......
- }

***GCC:            No warning!!!***

**But**
**case 4  ??? **
**case 5  ??? **
**...****.. **
**default  ???**

_***undefined***_
***Who am I?***
***Where am I?***
***What should I do?***

*2. **Project IV&V activities’ status and findings*

**Notes:**
For example, at switch case, even if we does not define "default case", GCC compiler does not give warning. Because that is grammatical. 
We wrote the program with thinking all case. switch case A.... case B...
If system producer think, at this case, we have only three case. 
Routine attitude_jobs id_flg is case 1, 2 and 3. we can not find other case.
So that we does not define other case without three conditions.
when the system have case x at id_flg, software is thinking.... huum..
"Who am I?" "Where am I?" "What should I do?"
We think software has capability to become panic.
Splint can find that case which is software become to be unknown state.

## Light weight code checking tool: Splint

- void attitude_jobs ()
- {
- ......
- ......
-   switch (id_flg)  {
-        case 1:
-              control();
-      ......
-      case 2:
-              stop_attitude();
-      ......
-      case 3:
-              restart_attitude();
-  ......
-   ......
- }

***GCC:            No warning!!!***

**But**
**case 4  ??? **
**case 5  ??? **
**...****.. **
**default  ???**

_***undefined***_
***Who am I?***
***Where am I?***
***What should I do?***

***Demo #3 !***

*2. **Project IV&V activities’ status and findings*

**Notes:**
For example, at switch case, even if we does not define "default case", GCC compiler does not give warning. Because that is grammatical. 
We wrote the program with thinking all case. switch case A.... case B...
If system producer think, at this case, we have only three case. 
Routine attitude_jobs id_flg is case 1, 2 and 3. we can not find other case.
So that we does not define other case without three conditions.
when the system have case x at id_flg, software is thinking.... huum..
"Who am I?" "Where am I?" "What should I do?"
We think software has capability to become panic.
Splint can find that case which is software become to be unknown state.

## Japan Experiment Module (JEM) IV&V activity Our first target of IV&V efforts:Wrap-up of these 5 years

*2. **Project IV&V activities’ status and findings*

## The biggest issue of JEM Software

-  Complexity of Automated FDIR
*Over 150 procedures which could conflict each other when executed simultaneously.*

- if (MTL flow rate < min)
-   stop pump-a
-   close valve-a
-   set pump-b 3500rpm
-   open cross-over
-   set pump-b 7000rpm

- if (MTL temp > max)
-   stop pump-a
-   close valve-a
-   shutdown payload racks
-   set pump-b 3500rpm
-   open cross-over
-   set pump-b 7000rpm

- if (NASA_H-EX_M is error)
-   shutdown payload racks
-   set pump-a 3500rpm
-   open H-EX bypass valve
-   open cross-over
-   set pump-a 6000rpm

***The biggest issue of JEM Software***

*2. **Project IV&V activities’ status and findings*

## Background

- if (MTL flow rate < min)
-   stop pump-a
-   close valve-a
-   set pump-b 3500rpm
-   open cross-over
-   set pump-b 7000rpm

- if (MTL flow rate < min)
-   mask LTL flow rate check
-   mask MTL temperature check
-   if (current mode != 1WCL)
-     if (NASA H-EX-MTL != error)
-       if (current flow rate > 3500)
-          stop low priority procs
-          wait 15 sec
-          send message "MTL-stop procedure
-                 going on. Do not bother me."
-          stop pump-a
-       else
-         ......

- @ PDR

- @ CDR

***Background***

*2. **Project IV&V activities’ status and findings*

## Initial JEM IV&V approach (~ 1999)

* **Difficulties of reproducing high fidelity execution environment (especially for fluid dynamics on zero G)*
* **Cost of execution was enormous for good coverage of possible TIMING*

- Corrected JEM IV&V approach (2000~)
-  Build state-machine model for static analysis
* **Focus more on the software architecture*
* **Focus more on the development of analysis algorithm*
* **Provide top-down overhaul option (It gives cheap way in the end in many cases)*

***Initial JEM IV&V approach (~ 1999)***

*2. **Project IV&V activities’ status and findings*

## Examples (Thermal control FDIR redesign)

-  Complete matching survey for FDIR
-   procedures coupling
* **Simply extract critical commands and their execution conditions to build a model *
* **Check conflicts by simple matching algorithm for every possible coupling of the commands*
* **Detected over 800 conflicts for the first round*
* **Eliminated in the second round*

**Overhaul of the architecture performed for :**
** ****Parallel execution control mechanism**
** ****Scratch building the sequence and grouping**

**Iteration Cycle**

***Examples (Thermal control FDIR redesign)***

*2. **Project IV&V activities’ status and findings*

## Examples (File Transfer Protocol)

* **Simply build conventional state transition matrix from the design documents *
* **Apply normalization techniques for the table which is quite popular in the relational database design*
* **Check incomplete combinations of conditions*
* **Detected over 20 incomplete conditions*
* **Eliminated in the second round*

-  Solution was given by simply populating the table rather than spending long time with whacking a mole of test non-conformances.

***Examples (File Transfer Protocol)***

*2. **Project IV&V activities’ status and findings*

## Lessons we learned

* **Static analysis eliminates the difficulty of timing issue*
* **Static model reveals architectural solutions more explicitly than dynamic analysis *
* **Static model inspires us to be creative and fun*
** ****Think basic**
* **For the post-CDR systems, everyone's focus is distributed to different details and could fall into hell of complexity. Keep ourselves harmonized by looking back to the architecture.*
* **Don't rely on the super fancy-sexy analysis from the preliminary risk assessment stage. They are expensive.*

***Lessons we learned***

*2. **Project IV&V activities’ status and findings*

## H-IIA transfer vehicle (HTV)IV&V activity

*2. **Project IV&V activities’ status and findings*

## Finding

- FDIR methodology validation using formal model revealed significant numbers of undefined failure modes (especially in quintuple voting mechanism among dissimilar navigation sensors).
**Root cause of the design flaw:**
**FDIR design originally started from bottom-up approach for each combination of failures**
**The numbers of combinations inflated as the software was adapted to additional devices and more safety/mission constraints**

*2. **Project IV&V activities’ status and findings*

## Issue

**Quintuple voting mechanism shall handle 2****10**** of **
**combinations of failures**

| Comparison item | 1st Case | 2nd Case | 3rd Case | ・・・ | 210th Case |
| --- | --- | --- | --- | --- | --- |
| SensorA/SensorB | 〇 | 〇 | 〇 | ・・・ | × |
| SensorA/SensorC | 〇 | 〇 | 〇 | ・・・ | × |
| SensorA/SensorD | 〇 | 〇 | × | ・・・ | × |
| SensorA/SensorE | 〇 | × | × | ・・・ | × |
| ・・・ | ・・・ | ・・・ | ・・・ | ・・・ | × |
| SensorD/SensorE | 〇 | 〇 | × | ・・・ | × |

- Unrealistic to have huge table to search (resource constraints)
- → ★**Top-down study for a dedicated law to realize a feasible**
**solution was required.**

*2. **Project IV&V activities’ status and findings*

## Solution

- _“__**Graph algorithm**__” to detect Failed Sensor _
- Rule:  “Identify failure modes by identifying closed triangle(s) in the chart”
- 5 categories of failures
- Single failure
- Two failures
- More than three failures
- Unidentifiable failure
- Incredible failure

*2. **Project IV&V activities’ status and findings*

## Merit

_**Using “Graph algorithm”**__ _
- Simple and fast(Grouping of patterns , no large table search needed)
- Mathematically complete
- Reusable for any kinds of similar voting logic

*2. **Project IV&V activities’ status and findings*

## Centrifuge Rotor (CR) and Life Science Grove-box (LSG) IV&V activity

*2. **Project IV&V activities’ status and findings*

## System Overview

-    Centrifuge is the experiment module for the life-science
-    _Centrifuge Rotor (CR) :_
-    CR is rotational equipment for gravitational experiment
-    The target S/W : Rotor Control Unit (RCU),  (C-RIC), Laptop
-    Hazardous Condition related with software is
- Loss of spin control		***Catastrophic***
- Contact of rotating equipment	***Catastrophic***
- Rotor Dynamics hazard		***Catastrophic***
-    _Life-Science Grove-box (LSG):_
-    LSG provides an enclosed environment for biological specimens and chemicals
-    Hazardous Condition related with software is
- Specimen death		***Critical***
- Lost of experiment data		***Critical***

*2. **Project IV&V activities’ status and findings*

## IV&V Process overview

- Formal Model

- Natural Language
- Document

**Development**
**Process**

**PRR**

**SRR**

- Analysis

- ・Inconsistency
- ・Completeness
- ・Reachability

- ・Inconsistency
- ・Completeness
- ・Traceability

***1st Step***

***2nd Step***

*2. **Project IV&V activities’ status and findings*

## The 1st step -  Modeling

- AA Mode
- The CR shall (REQ-01001) transfer mode from _**every mode without Initialization Mode or B Mode**_ to AA Mode by command.

***Natural language definition***

- AA Mode (REQ-01001)
-       ‘CR is Stanby_Mode’ and ‘AA_Mode_Command is ON’
- Or ‘CR is AA_Mode’ and ‘AA_Mode_Command is ON’
- Or ‘CR is AB_Mode’ and ‘AA_Mode_Command is ON’
- =>	CR is AA_Mode

***Natural language based Model***

- CR = AA_Mode

***Formal Model (Matrix)***

| AA_Mode_Command is ON | T   T   T |
| --- | --- |
| CR is Standby_Mode  | T   *    * |
| CR is AA_Mode | *   T    * |
| CR is AB_Mode | *    *   T |

_**Improvements:**_
**We can analyze it formally **
**with automated tools**
**(Static Analysis) **

- _Concerns:_
- The ambiguous documentation and
- inconsistency often cause
- the former accident and incident.

## The 2nd step  -  Analysis

- ★Consistency Analysis on documentation
- 	★Completeness Analysis on documentation
- 	→ DEMO #4
- ★Traceability Analysis
- 	→ DEMO #5
- ★Hazard Reachability Analysis
- 	→ DEMO #6

*2. **Project IV&V activities’ status and findings*

## Natural Language Modeling and Analysis Tools (Demo #4)

- Tools: (run with gtk on multiplatform)
- Function
- Natural Language Modeling
- Model Checking
- Consistency and Completeness Analysis
- Component Windows
- Editor Window
- Model Viewer
- Analysis Result Viewer

*2. **Project IV&V activities’ status and findings*

**Notes:**
Demo #1
This tool mainly has two functions and three component windows (viewers).
First function is an editor for modeling of specification, e.g. software requirement, preliminary design specification, using the natural language. 
Second function is model checkers for the consistency and completeness analysis of the model. 

## Editor Window  (modeling window)

- chapter

- section

- AA Mode (REQ-01001)
-         ‘CR is Stanby_Mode’
- And  ‘AA_Mode_Command is ON’
- Or    ‘CR is AA_Mode’
- and  ‘AA_Mode_Command is ON’
- Or    ‘CR is AB_Mode’
- and  ‘AA_Mode_Command is ON’
- =>	CR is AA_Mode

- Rule word

- State

***Editor Window  (modeling window)***

*2. **Project IV&V activities’ status and findings*

**Notes:**
This is a editor window.
First function is an editor for modeling of specification, e.g. software requirement, preliminary design specification, using the natural language. 
This is chapter field, section field.
This is the condition for becoming AA mode.
### Ex を用いて説明
This tool can be modeling at Japanese and English language. 

## Modeling Viewer

***Modeling Viewer***

***Analysis Result Viewer***

*2. **Project IV&V activities’ status and findings*

**Notes:**
A system state is not written in the same chapter or section of document. 
The state is written in the place separate from each other.
about the modeling viewer, this tool can package parameter of same state, so that analysis people can understand easily!
This is a formal model matrix.

About analysis, we have two method. 
One is consistency analysis, the other is completeness analysis.

About analysis Result Viewer,
This tool is taught to me the warning point which is document’s chapter and section number.
If modeling miss, we must change the model of editor windows.

At this point (document number.....) , this system is inconsistency.

## About Consistency Analysis

***About Consistency Analysis***

- Last State

- Next State

- Condition B

- Analysis which check that there is not two or more states transitions (Ex: Next State include the condition B and condition C....) when system condition transit from last state to next state.

- Condition A

- Condition C

*2. **Project IV&V activities’ status and findings*

**Notes:**
About consistency analysis,
Consistency analysis is checking that there is not two or more states transitions.
At event, system state change state. (we define From last state to Next state).
In some case, next state include some conditions.( condition B, condition C..)
Essentially system define at 1 to 1. But system development people makes mistakes. So that there is not consistency on this system. This is the consistency analysis.

## About completeness analysis

- Analysis which check that the state corresponding to the defined conditions surely exists
- The state of a system will be set to Unknown Mode when the defined conditions do not belong to all the state.

|  | A | B | C | D | E |
| --- | --- | --- | --- | --- | --- |
| α | T | T | T | F | F |
| β | T | T | F | T | F |
| γ | T | F | * | T | * |

_**Mode**_

_**Parameter**_

- T:True
- F:False

*2. **Project IV&V activities’ status and findings*

**Notes:**
About completeness analysis
Completeness analysis means that analysis which check that the state corresponding to the defined conditions surely exists.
When the define conditions do not belong to all the sate, the state of a system will be set to unknown mode. So that system is panic!!
For example,
The system consist of A, B,C,D,E Mode, and each mode is expressed using parameter α-γ.
Mode A is expressed at alpha = Ture......
* (Don’t care means True or False)

## For example of completeness analysis:

**β**

**γ**

**A**

**β**

**γ**

**B**

**γ**

**γ**

**C**

**C**

**D**

**?**

**E**

**E**

**Yes = “T”**

**No = “F”**

**Yes**

**Yes**

**Yes**

**Yes**

**Yes**

**No**

**No**

**No**

**No**

**No**

**No**

**Yes**

**TTT**

**TTF**

**TFT**

**TFF**

**FTT**

**FTF**

**FFT**

**FFF**

**αβγ**

**Mode**

|  | A | B | C | D | E |
| --- | --- | --- | --- | --- | --- |
| α | T | T | T | F | F |
| β | T | T | F | T | F |
| γ | T | F | * | T | * |

_**Mode**_

***For example of completeness analysis******:***

**Unknown Mode**

*2. **Project IV&V activities’ status and findings*

**Notes:**
There is 8 pattern in this parameter. If alpha = F, beta = F, gamma = F....  This pattern does not belong to Mode A – E. This pattern is unknown mode!!

## DEMO #5TRAD(Traceability Diagnosis) Tool

- Traceability of functions described in the Document at different level (e.g. requirements and specifications) must be confirmed
-             →  Evaluate automatically by TRAD tool
- PREPARATION: Translate functions to Formal model “Natural-Language Model”
-          Example
-           [CSM-01514]
-                CR_Mode in Mode Rotation_Mode
-                Operational_Mode_Transfer_CMD is ON (from Laptop)
-            -> CR_Mode in Mode Operational_Mode

*2. **Project IV&V activities’ status and findings*

## DEMO #5 TRAD(Traceability Diagnosis) Tool

- EXECUTION:
-     (1) Conversion Formal Model of each document to UML Sequence Diagram in every function
-     (2) Evaluate similarity of two Diagrams → Traceabirity
-                    Example
- RESULT:
-     - Evaluate Traceability accurately in a very short time
-     - Found miss-traced functions based on Traceability

*2. **Project IV&V activities’ status and findings*

## DEMO #6 Reachability Analysis Environment (RAE)

- Preparation:
- <1> Formal Modeling of software specification
- <2> Formal Modeling of the hazard
- Objective of the integration:
- ★Generate software fault tree from system hazard
- ★Search for single failure point
- ★Search for state variables which eliminate hazard
- ★List up initial conditions to lead to the hazards

*2. **Project IV&V activities’ status and findings*

## H-IIA Flight Safety Control System (SIRIUS) IV&V activity

*2. **Project IV&V activities’ status and findings*

## System Overview

- “SIRIUS” is the ground range safety control system for H-IIA rocket

- SIRIUS provides
- -Flight Path
- -Area affected by explosion
- -Attitude

- Inputs from
- -Radar
- -Telemeter
- -etc...

- Outputs for
- ground safety
- decision

- SIRIUS requirements
- ★ Hard real-time (msec Synchronized)
- ★ Non-Stop (of dozens of computers)
- ★    Built on top of commercial OS

*2. **Project IV&V activities’ status and findings*

## Analysis Approach

- Hard Real Time on top of commercial OS
- Extensive analysis of code implementation for:
- Timing control parameters (Detailed validation of unit test results)
- Signal handling mechanism (Graphical sequence modeling)
- Global variable handling (Automated tool to exhaustive search for potential conflicts)
- Frequently updated for specific requirements of each launch
- Line by line comparison for all code from the previous version *at the launch site*

*2. **Project IV&V activities’ status and findings*

## Findings

- _Findings_
- Design flaw of process synchronization
- Fluctuation of CPU load due to commercial OS system service
- Lack of exception handling
- Task priority inversion
- Global variable conflicts
- System configuration management problem

*2. **Project IV&V activities’ status and findings*

## Lessons Learned 

- #1 Hard-real-time & multi-CPU/multi-processing upon non-real-time OS : Surprisingly hard challenge kept the moral of the development team excellent.
- #2 On-site IV&V is essential because huge amount of updates were conducted at the last minute change.
- #3 On-site ***real-time*** IV&V (repeating the check for daily updates) contributed to the good synergy effect between design team and IV&V team.

*2. **Project IV&V activities’ status and findings*

## New Methodology Research - IV&V Future Study -H-IIA transfer vehicle (HTV)

*3. **New Methodology Research*

## 2-stage quantitative risk assessment flow

***2-stage quantitative risk assessment flow***

**Analysis**

**1st Stage (Pre-PDR phase): **
**What software function need to be focused on for analysis?**

**2nd Stage (CDR phase): **
**What are the critical functions **
**which need to be tested intensively?**

**Test**

**2**

**3**

*3. **New Methodology Research*

## How to identify critical software function ?

- Two stages of risk assessment:
- ★Preliminary stage sensitivity analysis
*                        **(Conventional sensitivity analysis)*
- Purpose: In-depth analysis targeting
- Parameters: Mission operation parameters (Orbital elements or other dominant-role numbers)
- ★Post-design-stage sensitivity analysis
*                        **(Newly developed methodology)*
- Purpose: Efficient testing (You cannot test everything!)
- Parameters: Sensitivity of non-scalar state variables (calculated from state machine model)

*3. **New Methodology Research*

## 1st Stage: What analysis need to be done? Dominant Control Parameter Sensitivity Analysis

- Delta-V [m/s]

- Error Ratio[%] = Error Correction Delta-V / Whole Delta-V

- Commanding Event

- Change of Delta-V  = Delta-V [m/s] / Time duration to next maneuver [s]

- Change of Delta-V

***1st Stage: What analysis need to be done? ******Dominant Control Parameter Sensitivity Analysis***

## 2nd Stage: Sensitivity analysis of State Variables

- Software Sensitivity by Input             high

- Software Function (FDIR)

**Earth Pointing Attitude Check**

**Gyro Status  Check**

**Rendezvous Sensor Failure Detection**

**Gyro Failure Detection**

**GPS Receiver Failure Detection**

**Propulsion System Failure Detection**

**Accelerometer Failure Detection**

**Switch to Redundant Propulsion system**

**Switch to Redundant I/O Computer**

**Abort Control Computer Selection**

**Abort Decision**

***Isolation and Recovery***

***Fault Detection***

*3. **New Methodology Research*

## Sensitivity Contour Map

***Sensitivity Contour Map***

-     Point = Criticality on Flight Operation ×
-   					  Sensitivity of state transition

**Pre-launch**

**Rendezvous**

**Proximity**

**Attached**

**Departure**

**Valid ****Check**

**Failure Detection**

**Redundant Switch**

**Abort**

- Flight Phase

- Software Function

-  Highest

- Second
- highest

- Moderate

## Management’s aspect

- NASDA established new software QA standard in 2001, but....
- Started problem investigation widely of software development activities and processes.
- Current quality activities
- Sub-contractor or vender control
- Nonconformance cause investigation especially for common causes or contributing factors.
- Improvement of activity as software acquisition point of view
- Software Process Improvement

*4. **Software QA activity and Process Improvement*

**Notes:**
NASDA established new software STD in 2001, but all most dead STD....
Therefore we started problem investigation widely of software development activities and processes.
 that is current quality activities, sub-contractor or vender control, nonconformance cause investigation  especially for common causes or contributing factors, improvement of activity as software acquisition point of view.

## NASDA Software Process Improvement 

- Process
- Model study

- ISO15504, CMM/CMMI, NASDA-STD

- Gap
- Assessment

- Interview(Bottom-Up), Process Analysis(Top-Down),
- Nonconformance/Accident Report Analysis(Background)

- Process
- Improvement

- Domain (e.g. satellite, rocket, ground, ...) Specific Processes
- (NASDA, contractors) which are integrated and clearly defined

- Tools and
- Measurements

- Modifying or newly introducing tools,
- Measuring and accessing processes

- Space Domain
- Specific Model

**Notes:**
About NASDA software process improvement.
Now using ISO15504, CMM/CMMI, NASDA-STA document, we make a space domain specific model.
We do interview and process analysis, we measure a gap assessment between space domain specific model and that result.
As this gap is base line, space domain should  integrate and clearly define domain specific processes.
After this phase, we modify tools or introduce new tools, we measure processes....
Iterate the flow.
Now, we are thinking what to do software process improvement like this.
And we just started this process.

Thank you.