# Presentation: 660290

## CxSI Validation from “The Way We Were” to “Where We (think) We Are”

- CxSI Validation from “The Way We Were” to “Where We (think) We Are”

- Objectives
- Convey processes and plans used by CxSI IV&V
- Share best practices and lessons learned
- Provide a context for discussion about modeling and validation
- Agenda
- March 29 Brown Bag Recap, CxP Requirements
- The CxSI SRM
- Validation Plans and Status

- _[christina.d.moats@nasa.gov](mailto:christina.d.moats@nasa.gov)_
- _[cxsi_team@ivv.nasa.gov](mailto:cxsi_team@ivv.nasa.gov)_

## March 29 Brown Bag Recap (Initial Validation Plans)

CxSI Initial Framework for Validation (Ready to Use version presented 7/16/07)

Best Practices/Lessons Learned

**CxSI Initial Framework for Validation (Ready to Use version presented 7/16/07)**
**Best Practices/Lessons Learned**

## Current Efforts follow plans developed in March 07

*Three tasks will complete by July 27, 2007:*
* **Develop Behavior Model (SRM),*
* **Analysis Preparation*
* **Validation Methods*

*Validation analysis will occur from July - Sept 2007*

***Note: this process will be expanded in future slides***

## CxSI Behavior Model Generation and L2 Validation Process based on this Model

**CxSI Behavior Model Generation and L2 Validation Process based on this Model**

*Feb-July,*
*2007*

*Aug-Sept,*
*2007*

***Note: this process will be expanded in future slides***

## Constellation Requirements

**Constellation Requirements**

_**Requirements Level Definitions**_
- Level 1: Need, Goals and Objectives (NGO)
- Level 2: CxP Architecture Requirements (CARD)
- Level 3: System Requirement Documents (SRDs)
- Level 4: Element

**CxSI Requirements Validation Objectives:**
- Validate Level 2 Requirements  Section 3.2 of the CARD
- Verify Trace from Level 2 to Level 3  Section 3.2 to 3.7 of CARD
- Verify Section 3.7 of CARD flows to Sections 3.2 of appropriate SRDs
***SRD Section 3.2 is analysis starting point for other CxP IV&V teams***

**SRDs:**
- Section 3.2: System Requirements
- Section 3.7: Element Requirements (allocation of system requirements)

- etc

*Section 3.7 of the Parent Document Is the same as Section 3.2 of the Child*

- CxP Requirements Artifacts
- (note: CxP Requirements Mgmt in Cradle Database)

## Detailed plans based on framework support integrated planning and execution

**Detailed plans based on framework support integrated planning and execution**

***Part A***

***Part B***

***Part C***

|  | Activity | Product |
| --- | --- | --- |
| Part A | Develop Behavior Model | Deliverable 1: SRM |
| Part B | Analysis Preparation |  |
| Part C | Validation | Deliverable 2: Validation Report, associated Issues |

## Framework and Planning: Best Practices/Lessons Learned

- Best Practices
- CxSI spent time benchmarking modeling as part of development and extrapolating what we learned to an overall IV&V framework
- This framework has had good longevity for our team
- Because our work was “the first time” for a key NASA initiative, we planned and performed as a group
- Socialization of ideas facilitated innovation
- Socialization of products resulted in high rigor
- Initiative was embraced by the team
- A good thing to do was to understand how we were going to use the model for validation so we knew when we were “done enough”

## The CxSI SRM

Dependability Criteria

SRM Readiness to support Requirements Validation
Requirements Assessment
Peer Review

SRM: Best Practices/Lessons Learned

**Dependability Criteria**
**SRM Readiness to support Requirements Validation**
**Requirements Assessment**
**Peer Review**
**SRM: Best Practices/Lessons Learned**

***How do we validate our model?***

## Constellation Behavior Model Hierarchy

**Constellation Behavior Model Hierarchy**

**CxSI SRM addresses four primary goals: ISS, Moon Sortie, Moon Base, Mars**

## Dependability Criteria

**Dependability Criteria**

- Dependability criteria:
_**Availability**_: The probability that a system is operating correctly and is ready to perform its desired functions.
_**Consistency:**_ The property that invariants will always hold true in the system.
_**Correctness**_: A characteristic of a system that precisely exhibits predictable behavior at all times as defined by the system specifications.
_**Reliability**_: The property that a system can operate continuously without experiencing a failure.
_**Robustness**_: A characteristic of a system that is failure and fault tolerant.
_**Safety**_: The property of avoiding a catastrophic outcome given a system fails to operate correctly.
_**Recoverability**_: The ease for which a failed system can be restored to operational use.

- We use a process similar to the development of the BMK in Caffall thesis to assure completeness of the model
- BMK development analyzed assertions to assure they addressed the seven dependability criteria (section 8H-Analysis of Prototype)
- CxSI used these criteria against the model instead of against assertions

## Sample supporting analysis to ensure Dependability in our model

- CxSI Dependability analysis required supporting analysis
- anytime/constant services analysis
- operational environment
- hazard analysis
- End to end use case views further increased model dependability
- Safety
- Consistency
- Recoverability
- Reliability

*Hazard analysis performed supports model dependability (robustness). *

*Anytime failure/continuous services analysis as a separate analysis (availability and robustness)*

## Sample end-to-end analysis for Dependability in our model

**Sample end-to-end analysis for Dependability in our model**

**Level 2 MSS Steps**
**Assertions**
**Extension Conditions**
**Extension Handling Steps** *(with frequency of occurrence)*
**Action verbs for each Actor** *(with frequency of occurrence)*
**Objects** that get acted on *(with frequency of occurrence)*

**Use Cases**

**Parsed MSS Steps (with extensions)**

**End-to-End Summaries facilitate availability, **
**consistency, reliability, safety, recoverability analysis**

**End-to End Filter criteria**

## Necessary Model Elaboration against L2 Requirements

**Necessary Model Elaboration against L2 Requirements**

*Assessment resulted in some, (but not many), elaborations to our Level 1 use cases and our Level 2 Launch model*

- An assessment of the L2 requirements was made to determine need for elaboration of the model
- CxSI emphasis to have consistent level of model abstraction
- Level 1: independent of architecture elements, design reference missions
- Level 2: Driven by architecture and associated behavior provided in system ops-con
- Particularly, elaboration needs for Moon base and Mars necessary to determine need for Level 2 elaboration of future system goals

## Review of SRM

**Review of SRM**

- Peer Review
- In addition to our internal CxSI reviews, peer reviews are held when we complete major portions of our model, further increasing model confidence
- The peer review includes a combination of domain and modeling experts from the facility to review our products
- Feedback has been great and has resulted in better model and processes 
- Project Review: CxSI is targeting early August for CxP SE&I review

***A PITS database is setup to track***
***fixes we need to incorporate***

***Formal schedule used for review***

- _RIDs State Machine_

## SRM: Best Practices, Lessons Learned

- Best Practices:
- Using rigorous peer review (internal and external) led to confidence in our model
- Taking all comments seriously took a good amount of time (6 weeks allocated)
- Dependability criteria and how they were used for BMK provided a framework for model validation
- We have a repeatable process (four months to generate the Lunar Sortie model, six weeks to generate the ISS model)
- Generating our validation plans in parallel with the model allowed us to know when we were at the “good enough” stage
- Lessons learned:
- Modeling was even more iterative than any of us thought. It was worth the time for us to get on the same page as a team.
- We had to learn to not sweat the small stuff
- Terminology discussions got old – we created an “open issues” field that helped
- After the fact, we are performing a rigorous mapping of OpsCon to our model. This work could be performed in parallel with model development
- As we better understand how to use our model, more updates are likely (they are expected and planned for)

## Validation Plans

Validation Generation Process and Status

Validation Analysis Tasks
SRM (Behavior, Interface)
Requirements Trace
Requirements Evaluation
Other (Safety, Performance, Design)

**Validation Generation Process and Status**
**Validation Analysis Tasks**
**SRM (Behavior, Interface)**
**Requirements Trace**
**Requirements Evaluation**
**Other (Safety, Performance, Design)**

***How do we validate the CxP Level 2 Requirements?***

## Generation of CxSI Validation Plans

- In preparation for validation analysis, we performed a classification of CxP Level 2 requirements into the following categories:
- _Behavior_ – requirements that are driven from the behavior of the system,
- _Interface_ – requirements that are driven from interface interactions, or necessary for interfacing across the systems
- _Performance_ – requirements that indicate thresholds of goodness
- _Design Implementation_ - trade based requirements ... could drive lower level behavior, but at Level 2, are driven from the design
- _Safety_
- The validation plans are tailored to these categories
- e.g. SRM is used to validate the behavior and interface requirements but the Loss of Mission requirement stems from trades and has a performance aspect
- Our process to generate CxSI validation plans involved:
- Initial Draft
- “blue sky” view of what needed to be done for requirements validation
- Reconciliation of this view with Facility WBS
- Team Peer Review (CxSI Team and JWST-Vorndran)
- (update) Infuse CxP processes into our assessment (both as information and as a “standard”)

## CxSI Validation Analysis Tasks are binned into four Categories

**CxSI Validation Analysis Tasks are binned into four Categories**

**CxSI Validation Analysis Tasks**

**1. SRM Related Tasks *********** for behavior  and interface Requirements ***
- C1.  Validate Behavior Requirements and Interface Interactions against SRM
- C2.  Interface Requirements (trades, and trace related activities)

**2.  Requirements Trace Related Tasks *********** for all requirements***
- C3.  Level 1 to Level 2 trace (verify project trace)
- C4.  Level 2 to Level 3 trace (verify project trace)
- C5.  Artifact consistency – verify CARD Section 3.7 is the same as associated SRD Section 3.2

**3. Requirements Evaluation Tasks *********** for all requirements***
- C6.  Requirements Evaluation – per requirement
- C7.  Requirements Evaluation – per mission goal
- C8.  Compare mission goal requirements for completeness

**4. Other Tasks**
- C9.  Safety Requirements *** for safety requirements*
- C10. Design Implementation *** for design/trade related requirements*
- C11. Performance Requirements *** for performance related requirements*

***Note: this framework will be expanded in future slides. Task ID’s are***
***used in planning, e.g. C1, C2, etc***

## CxSI IV&V Analysis – Utilized multiple tasks, tools, methods to increase assurance of coverage

## SRM Related Validation Tasks

**SRM Related Validation Tasks**

- SRM used to validate behavior requirements and interface interactions. Issues from analysis require CxSI assurance model is correct

_*Specific Plans:*_
*Evaluate attributes of system behavior extracted from behavior model to ensure complete coverage of requirements at a consistent level of abstraction. Perform a detailed correlation between the SRM and behavior requirements from the CARD. This trace must occur in a two way manner:*
*1) that the model represents all behavior requirements in the CARD, and that the CARD Requirements are correct (note: if SRM needs to be updated, CxSI process will be to find reference for the change from somewhere else besides the requirements or reference behavior as an assumption)*
*2) that all behaviors (given level of abstraction) are represented in the requirements*
*In performing the correlation, all fields of the UC/AD should be evaluated for completeness in the requirements.*
*In terms of a more detailed representation of behavior in the IV&V model than the CARD requirements, IV&V should either*
*a) write an issue to capture the missing behavior even though it may be in a lower level document, or*
*b) go into the lower level requirements to determine if the behavior is captured appropriately, and evaluate if a (lower level) TIM should be generated which recommends moving a requirement from Level 3/4 to Level 2.*

## Validation Data Capture Products

**Validation Data Capture Products**

- Updates to both the model and the CARD are expected
- Validation results will correspond to a snapshot of these products
- Results must be captured in a way that supports assessing change impacts and re-validation

- CxSI IV&V SRM

- CARD Requirements

| Model Element | CARD Requirement |
| --- | --- |
| XXX | AAA |
| YYY | BBB |
| ZZZ | CCC |

## SRM Behavior Validation Analysis Data Capture

- CxSI Team is preparing for validation of CARD Requirements
- Involves creating a correlation map between model elements and requirements
- Capabilities for capturing validation results needed in the near term (CARD validation task scheduled for August and September 07)
- CxSI IV&V models developed thus far in Microsoft Office products
- Recognize that selection of a common tool set (Together) requires some amount of augmentation for IV&V processes
- CxSI IV&V model will be migrated to selected tool for long-term maintenance
- Near-term solution: Access Database
- Correlation data must be captured in a way that supports validation analysis views (many-to-many relationship)
- Ideally, this database, and associated lessons learned in our analysis, can  provide insight into the IV&V tailoring of the Together tool
*Note: The "how" of this task is tbd, there have been multiple approaches brought up in CxSI discussions. The convergence on a single method will occur after this task is begun (as compared to other tasks where the method is known prior to beginning analysis)*

## Requirements to Model Correlation – Access Database (note: still under construction)

**CARD Requirements**

**Link Capability**

**CxSI Model (shown**
**is a Use Case)**

**Step 1 – Perform Correlation:**
- Database contains the requirements and the model
- Analyst links requirements to the elements of the model

**Step 2 – Perform Validation Analysis:**
- Database reports enhance analysis by showing created links
- Analyst performs correctness and completeness analysis between requirements and model
- Discrepancies analyzed and addressed (TIMs)

**Database construction:**
- Analysis reports going through upgrade to include requirement/model text and means to capture analysis
- Widows and Orphans (requirements or model) reports
- Merge/replace with Facility tool when appropriate

**The CxSI access database facilitates our model validation analysis**

**Output Report: **
**Correlation**
**of the CxSI model to**
**CARD Level 2 **
**requirement**

## Requirements Trace Validation Tasks

**Requirements Trace Validation Tasks**

- Same as trace work performed under prior WBS 3.1
- CxSI will perform trace from Level 12 and Level 23
-   At level 3, additional consistency check necessary across artifacts

- Specific Plans:
*For provided trace, use parent --> child trace as primary artifact: *
*1) ensure appropriate links exist; *
*2) ensure that the linked items at the lower level constitute a correct and complete decomposition of the upper level requirements; and *
*3) review widows/orphans as a potential source of issues.*
*4) ensure that child-parent and parent-child traces show equivalent information*
*In analysis setup,*
*Reconcile results with Civil Servant trace performed.*
*Ensure rationale explanations do not contradict the trace or are necessary to complete trace*

## Requirements Evaluation Related Validation Tasks

**Requirements Evaluation Related Validation Tasks**

- Same as requirements evaluation work performed under prior WBS 3.2
- CxSI will perform evaluations in three areas:
- On a requirement by requirement basis (correctness)
- Within a mission goal (correctness)
- Across mission goals (consistency)

*Specific Plans:*
*per requirement: Is this requirement : atomic (single thought), precise (unambiguous), testable. E.g. a good requirement (so if you were doing a design of it, you'd know what was expected) - and if not; why (see Req't Eval Worksheet) <-- note that the specific criteria may evolve*
*	**Additionally, check that the requirement and rationale are consistent (this is a lower level issue). Ensure that information in the rationale should not be in the requirement (this would be a higher level completeness issue)*
*evaluate requirements by mission goal (ISS, lunar, Mars) to ensure that multiple interpretations can't occur across suites of requirements or that contradictions don't exist within requirements. This check is to ensure consistency within a mission. *
*Correlate related requirements for each mission against each other to ensure consistent  level of depth across the four missions.: ISS, moon sortie/base and Mars*

## Other (Safety, Design, Performance) Validation Tasks

*Specific Plans:*
***Safety****: Shuttle and ISS safety requirements flow from some Program standard documents, which are probably related to NASA standards. CxP should have something analogous (if they don't, we should use shuttle/ISS standards). The analysis performed for the Level 2 requirements involved adherence to Constellation safety standards. Safety requirements are evaluated against the identified aspects/criteria (Caffall) to assess the requirements in terms of system-software safety: Fault Avoidance, Fault Warning, Fault Correction, Fault Tolerance,  Fail Operational, Fail Safe *
*Note that some of the safety requirements are references to other documents/standards. As part of this activity, a decomposition of the referenced documents is likely necessary for verification. Also, some of the safety requirements will overlap and be also mapped into performance, behavior or design requirements*
***Design Implementation Requirements****: Analyze design implementation requirements to ensure they are appropriate constraints on system behavior at this level for subsequent decomposition. *
*Provide rationale for why design based requirements are appropriate, e.g. traced to a trade study. If requirements can't be traced to a trade study, then the requirement is an assumption by IV&V and will be carried as such.*
***Performance Requirements.**** For each performance requirement, analyze to ensure the requirement is appropriate in defining the system goal. Appropriateness is viewed in two ways:*
*1) The applicability of the requirement to meet the overall performance objective (performed via analysis)*
*2) The reasonableness of this requirement from a technology readiness point of view (for CxSI safety performance, extrapolate these *
*requirements from Shuttle/ISS safety)*

- Performed under prior WBS 3.2
- Combination of:
- Tracing to standards/policies
- Identifying underlying trade or rationale
- Assessing technology readiness
- Domain Understanding

## Where We Are Now

- Finalizing report with all analysis results
- C1, C8:   	Missing requirements
- C6, C10:  	Process problems manifest as technical problems
- C1, C2:  	SRM ≠ Requirement level of detail
- C10:  	O&M  Requirements bin
- C3, C4, C5:  	Requirement traceability/consistency
- C6:  		Requirement clarity vs. requirement rationale vs. requirement 		verification
- Always need to incorporate Project’s plans/standards into the IV&V analysis
- Next
- Continue C1 with L3 requirements
- Update all analysis with future CARD/SRD releases
- More detailed scrutiny on the interfaces
- Use Validation results to focus future efforts

## Conclusion

- Charts are available on the Validation Workshop website
- Contact the CxSI team if you have questions/comments:
- _[Christina.D.Moats@nasa.gov](mailto:Christina.D.Moats@nasa.gov)_
- _[cxsi_team@ivv.nasa.gov](mailto:Cxsi_team@ivv.nasa.gov)_

## Appendix 1 – SRM Development (Part A),including Dependability Checks on SRM

## Slide 30

| A. SRM Development |  |  | Needs Input From: | Feeds into : |
| --- | --- | --- | --- | --- |
| SRM | A1 | A1. Generate SRM through Level 2 Use Cases and Activity Diagrams- do not use requirements in the generation of the model- Assess CARD requirements to help set the correct abstraction level of the model (note: analysis indicates going to L2 UC and AD is sufficient for CARD validation) | OpsCon - PrimaryCARD - only to assess correct level of abstraction | A2, A3, A4 |
| SRM | A2 | A2. SRM Dependabilityevaluate SRM against the following dependability criteria (Team together will develop definitions of these criteria and what they mean to us in review of this model. The documentation requirements are generated as part of this exercise – see tab "DependabilityCrit-ModelEval for starting point): Availability: The probability that a system is operating correctly and is ready to perform its desired functions.Consistency: The property that invariants will always hold true in the system.Correctness: A characteristic of a system that precisely exhibits predictable behavior at all times as defined by the system specifications.Reliability: The property that a system can operate continuously without experiencing a failure.Robustness: A characteristic of a system that is failure and fault tolerant.Safety: The property of avoiding a catastrophic outcome given a system fails to operate correctly.Recoverability: The ease for which a failed system can be restored to operational use. |  -  A1 -  IPHA List (Hazard Analysis which feeds into Robustness) -  SME Experience (Anytime Analysis which feeds into Robustness) -  OpsCon (Abort/Contingency section which feeds into Robustness) | A5 |
| SRM | A3 | A3. For each mission goal, compare Use Cases and Activity diagram modeled behavior for consistent abstraction across the CxP mission types (ISS, moon, Mars)Understand the differences in abstraction levels and either fix  the model or explain why a difference in abstraction is appropriate as a note in the affected Use Cases | A1 | A5 |
| SRM | A4 | A.4 Operational Environment - – identify aspects of the operational environment that have an impact on the system, and document them in a list/tabular format. For each operational environment factor, identify behavioral or other characteristics of the system that are required for successful performance of mission goals. Map behavioral responses to the applicable SRM elements. For CxSI, this task has been completed, and the byproduct of this work fed into the anytime analysis and its impact to the model | A1 | A5 |
| SRM | A5 | SRM Ready to Support Validation | A1, A2, A3, A4 | C1, C2, C10, C11 |

## Slide 31

| Dependability Definitions (Caffall Thesis) | CxSI Activity  |
| --- | --- |
| Availability: The probability that a system is operating correctly and is ready to perform its desired functions. | Review Availability through an explicit look at our preconditions and preparation (ops and crew) steps. Result of this look could be additional preconditions or preparation MSS steps or assertions or some other model augmentation to ensure that you don't leave system in a bad situation no ready for next step. The anytime/constant services analysis performed supports this objective. General observation - our level of abstraction is sufficiently high that readiness for working is almost a given |
|  Consistency: The property that invariants will always hold true in the system.  | Look at our assertions and extensions and ensure that we are consistent across the model for similar behavior. Result of this analysis would be additional assertions and/or extensions and/or other model augmentation.The second check would be to analyze each invariant (assertion) to ensure it holds true for full model behavior, not only the situation for which it was generated |
| Correctness: A characteristic of a system that precisely exhibits predictable behavior at all times as defined by the system specifications.  | Model has been validated against the ops-con which should address this concern. Our internal reviews address this, two additional tasks identified: 1) Fully reference specifics of use cases against OpsCon. Holes in this analysis should be documented in assumptions.2) Verify changes from updated OpsCon  |
| Reliability: The property that a system can operate continuously without experiencing a failure.  | 1) End-to-end review extensions and assertions for “must work” behavior. 2) Add constant service, anytime, hazard analysis as a reference item to the model (analysis already completed)  |
| Robustness: A characteristic of a system that is failure and fault tolerant.  | 1) Failures have been explicitly addressed as part of our internal peer review with compensating behavior addressed in extensions or assertions.2) Anytime analysis is complete and already been traced to the model.3) Complete hazard analysis and incorporate results into model4) Additional task (new): perform trace from Section 6 Ops Con - Contingency and off-nominal operations to our model. Address discrepancies - specifically where ops-con has more detail than our model." |
| Safety: The property of avoiding a catastrophic outcome given a system fails to operate correctly.  | End-to-end check of each MSS step to ensure minimal guarantee is satisfied. Add extensions/assertions as necessary |
| Recoverability: The ease for which a failed system can be restored to operational use.  | Verify that each path through the system meets minimum success guarantee and allows a path for recovery (minimal is a safe state of the system) |

## Appendix 2 – Validation Plans

## Slide 33

| C. Requirements Validation  |  |  | Input: | Benefit |
| --- | --- | --- | --- | --- |
| Validation - SRM | C1 | C1. For Behavior Requirements: For Behavior Requirements: Evaluate attributes of system behavior extracted from behavior model to ensure complete coverage of requirements at a consistent level of abstraction (need analysis results from steps A4 and B4). Perform a detailed correlation between the SRM and behavior requirements from the CARD. This trace must occur in a two way manner:1) that the model represents all behavior requirements in the CARD, and that the CARD Requirements are correct (note: if SRM needs to be updated, CxSI process will be to find reference for the change from somewhere else besides the requirements or reference behavior as an assumption)2) that all behaviors (given level of abstraction) are represented in the requirementsIn performing the correlation, all fields of the UC/AD should be evaluated for completeness in the requirements.In terms of a more detailed representation of behavior in the IV&V model than the CARD requirements, IV&V should eithera) write an issue to capture the missing behavior even though it may be in a lower level document, orb) go into the lower level requirements to determine if the behavior is captured appropriately, and evaluate if a (lower level) TIM should be generated which recommends moving a requirement from Level 3/4 to Level 2.Note: The "how" of this task is tbd, there have been multiple approaches brought up in CxSI discussions. The convergence on a single method will occur after this task is begun (as compared to other tasks where the method is know prior to beginning analysis) | A5B4 | Validation of behavior requirements (correctness, completeness). This step helps to ensure that the program completely covers behavior requirements at a consistent level of abstractionThis analysis contributes to answering the key validation questions:- Does system do what it is supposed to do?- Does system not do what it is not supposed to do?- Does system work correctly under adverse conditions? |

## Slide 34

| C. Requirements Validation  |  |  | Input: | Benefit |
| --- | --- | --- | --- | --- |
| Validation - SRM | C2 | C2. For Interface Requirements: Evaluate the SRM for interactions that need to occur between/amongst Constellation Systems. Correlate these behaviors to the requirements to ensure that the CARD requirements provides appropriate standards and behavior requirements for interfaces.We may need to do more here, as a minimum, we may need to flesh out the other interface requirements through the NGO trace. Note that many of the communication requirements are referenced in CxP 70022-01, Constellation Program Command, Control, Communication, and Information (C2I) Interoperability Standards Book, Volume 1: Interoperability Specification. As part of this activity, a decomposition of the Interoperability Specification is likely necessary. (need analysis results from step B4 and Task A) | A5B4 | Verification of interface requirements and sufficiency for decomposition (correctness, completeness)This analysis contributes to answering the key validation questions:- Does system do what it is supposed to do?- Does system not do what it is not supposed to do? |
| Validation - Trace | C3 | C3. For all requirements (Level 1 to Level 2): All: Verify downwards trace of all requirements For provided trace, use parent --> child trace as primary artifact: 1) ensure appropriate links exist; 2) ensure that the linked items at the lower level constitute a correct and complete decomposition of the upper level requirements; and 3) review widows/orphans as a potential source of issues.4) ensure that child-parent and parent-child traces show equivalent informationIn analysis setup,Reconcile results with Civil Servant trace performed.Ensure rationale explanations do not contradict the trace or are necessary to complete trace | B2 | Correctness, consistency, completeness of requirements (minimal step for validation which will occur across all requirements) to ensure requirements are completely flowed from L1 to L2This analysis contributes to answering the key validation questions:- Does system do what it is supposed to do?- Does system not do what it is not supposed to do?- Does system work correctly under adverse conditions? |

## Slide 35

| C. Requirements Validation  |  |  | Input: | Benefit |
| --- | --- | --- | --- | --- |
| Validation - Trace | C4 | C4. For all requirements (Level 2 to Level 3): All: Verify downwards trace of all requirements For provided trace, use parent --> child trace as primary artifact: 1) ensure appropriate links exist; 2) ensure that the linked items at the lower level constitute a correct and complete decomposition of the upper level requirements; and 3) review widows/orphans as a potential source of issues.4) ensure that child-parent and parent-child traces show equivalent informationIn analysis setup,Reconcile results with Civil Servant trace performed.Ensure rationale explanations do not contradict the trace or are necessary to complete trace | B2 | Correctness, consistency, completeness of requirements (minimal step for validation which will occur across all requirements) to ensure requirements are completely flowed from L1 to L2This analysis contributes to answering the key validation questions:- Does system do what it is supposed to do?- Does system not do what it is not supposed to do?- Does system work correctly under adverse conditions? |
| Validation - Trace | C5 | C5. For Level 3 Requirements: Ensure that the CARD 3.7 requirements have correctly been flowed to appropriate SRD Sections 3.2. Included in this comparison are both the requirements and associated rationale | B2B6 | Correctness to ensure requirements are completely flowed from CARD to SRDsThis analysis contributes to answering the key validation questions:- Does system do what it is supposed to do?- Does system not do what it is not supposed to do?- Does system work correctly under adverse conditions? |
| Validation - Eval | C6 | C6. per requirement: Is this requirement : atomic (single thought), precise (unambiguous), testable. E.g. a good requirement (so if you were doing a design of it, you'd know what was expected) - and if not; why (see Req't Eval Worksheet) (Need CARD requirements from analysis B2 for this task) <-- note that the specific criteria may evolveAdditionally, check that the requirement and rationale are consistent (this is a lower level issue). Ensure that information in the rationale should not be in the requirement (this would be a higher level completeness issue)see Req't Eval Map to old WBS for start of analysis spreadsheet | B2 | Ensure requirements are unambiguous (by requirement)This analysis contributes to answering the key validation questions:- Does system do what it is supposed to do? |

| C. Requirements Validation  |  |  | Input | Benefit |
| --- | --- | --- | --- | --- |
| Validation - Eval | C7 | C7. evaluate requirements by mission goal (ISS, lunar, mars) to ensure that multiple interpretations can't occur across suites of requirements or that contradictions don't exist within requirements. This check is to ensure consistency within a mission. (Need analysis results from steps B3) | B3 | Ensure requirements are unambiguous (across mission goal)This analysis contributes to answering the key validation questions:- Does system do what it is supposed to do? |
| Validation - Eval | C8 | C8. Correlate related requirements for each mission against each other to ensure consistent  level of depth across the four missions.: ISS, moon sortie/base and Mars  (Need analysis B3 to perform this) | B3 | Ensure requirements are consistent and at the same level of abstraction across the mission areasThis analysis contributes to answering the key validation questions:- Does system do what it is supposed to do?- Does system not do what it is not supposed to do?- Does system work correctly under adverse conditions? |
| Validation - Other | C9 | C9. Safety: Shuttle and ISS safety requirements flow from some Program standard documents, which are probable related to NASA standards. CxP should have something analogous (if they don't, we should use shuttle/ISS standards). The analysis performed for the Level 2 requirements involved adherence to Constellation safety standards. Safety requirements are evaluated against the identified aspects/criteria (Caffall) to assess the requirements in terms of system-software safety (note: terms defined in CxSI IPEP and Caffall Thesis).       Fault Avoidance, 
     Fault Warning,  
     Fault Correction, 
     Fault Tolerance,      Fail Operational , 
     Fail Safe  | B4 | Completeness and consistency - additional validation check using SME understanding. This is appropriate for the safety requirements since the safe return of astronauts is the minimal success guarantee of the Constellation modelThis analysis contributes to answering the key validation questions:- Does system do what it is supposed to do?- Does system not do what it is not supposed to do?- Does system work correctly under adverse conditions? |
|   |   | Note that some of the safety requirements reference other documents/standards. As part of this activity, a decomposition of the referenced documents is likely necessary for verification.Also, some of the safety requirements will overlap and be also mapped into performance, behavior or design requirements |   |   |

## Slide 37

| C. Requirements Validation  |  |  | Inputs: | Benefit |
| --- | --- | --- | --- | --- |
| Validation - Other | C10 | C10. Design Implementation Requirements: Analyze design implementation requirements to ensure they are appropriate constraints on system behavior at this level for subsequent decomposition. Need analysis B4 to perform thisProvide rationale for why design based requirements are appropriate, e.g. traced to a trade study. If requirements can't be traced to a trade study, then the requirement is an assumption by IV&V and will be carried as such. | A5B4 | Compliance to WBS (correctness element of WBS 1.2)This analysis contributes to answering the key validation questions:- Does system do what it is supposed to do? |
| Validation - Other | C11 | C11. Performance Requirements. For each performance requirement, analyze to ensure the requirement is appropriate in defining the system goal. Appropriateness is viewed in two ways:1) The applicability of the requirement to meet the overall performance objective (performed via analysis)2) The reasonableness of this requirement from a technology readiness point of view (for CxSI safety performance, extrapolate these requirements from Shuttle/ISS safety) | A5B4 | Correctness of the requirements. Correctness is based on SME analysis as to the reasonableness of a performance requirement in meeting the system goals, and from a technological standpointThis analysis contributes to answering the key validation questions:- Does system do what it is supposed to do? |