# Presentation: 334306

## A Method for Verification and Validation Certificate Management in Eclipse

- Mark Sherriff and Laurie Williams
- North Carolina State University
- SoftCeMent ‘05
- November 8, 2005

## Agenda

- Background
- Motivation and Hypothesis
- Parametric Modeling in Software Engineering
- Research Methodology
- Building the Model
- Verification & Validation Certificate Management
- Parametric model for estimating defect density
- Tool Support
- Limitations
- Current Research Efforts
- Questions

- Slide  of 17

## Motivation

- Software Reliability
- Often not estimated until development is complete
- Actual reliability not known until system is shipped to customers
- Corrective action is more expensive later in the process
- If defect density could be estimated in-process...
- Steps could be taken to address issues early
- More economical, could improve development effort

- Slide  of 17

## Hypothesis

- Defect density estimation can be based upon the history of verification and validation techniques that have been performed on the project.

- Slide  of 17

## Background - STREW

**S**oftware **T**esting **R**eliability **E**arly **W**arning
- Java version – Nagappan; Haskell version - Sherriff.
- Uses a suite of metrics gathered on static code to provide a reliability estimate
- The prediction model is calibrated to an organization using a regression equation formed from previous metric values
- Why use STREW?  Because...
- Operational profiles are expensive to create and maintain
- An effective automated testing suite could return no failures
- We want this method to be able to be used in-process to affordably guide corrective action
- Parametric modeling has been an effective tool to predict software projects
- Works for testing and static metrics; what if we add other V&V information to the model?

- Slide  of 17

## Parametric Modeling

- Method by which dependant variables are related to one or more independent variables with regards to previous data
- In Software Engineering...
- Purpose is to provide an estimated answer to a software development question earlier in the development lifecycle
- Famous SE parametric models: COCOMO 81 and COCOMO II

- Slide  of 17

## Building the Model

- Sherriff, M., Boehm, B., Williams, L., and Nagappan, N.  “An Empirical Process for Building and Validating Software
- Engineering Parametric Models.”  Submitted to Empirical Software Engineering Journal.

- Slide  of 17

## Literature Review and Expert Opinion

- Verification and validation efforts used to improve system reliability
- But...
- Often this effort is not recorded or managed in a systematic way
- Lack of proper documentation can hinder development and duplicate effort
- Could be useful information in both project management and system maintenance
- Software certificates
- Software certification demonstrates the reliability ... of software systems in such a way that it can be checked by an independent authority.
- Programatica for Haskell (OGI/OHSU)

- Slide  of 17

## Parametric Model and Method

**D**efect **E**stimation with **V**&V **C**ertificates on **P**rogramming (DevCOP)
- Two main components
- Software Certificate Management System
- DevCOP Certificates and Eclipse Plugin
- Parametric Model for defect density estimation
- Provide guidance to developers on current V&V efforts

- Slide  of 17

## Formulate a-priori model

- DevCOP Certificate
- Includes:
- Basic identifying information of creator
- Basic identifying information for code being certified
- Record / Evidence of V&V technique
- Hash of certified source code
- Certificates follow the code base and can be referenced at any time

- Slide  of 17

## Certificate Management

- Types of certificates
- Informal
- Includes all “manual” techniques, such as pair programming and code inspections
- Automated Static Analysis
- Includes all techniques that can be run on uncompiled code, such as automated static analysis
- Dynamic
- Includes all techniques that are run at runtime, such as automated testing suites
- Formal
- Includes all formal methods, such as proofs
- Each provides a different type of evidence/assurance of reliability

- Slide  of 17

## Gather Initial Data

- DevCOP Software Certificate Management System
- Eclipse Plugin v. 1.2
- A method for enabling developers to record certificates without excessive overhead
- Active Certificates
- Version Log:
- v. 1.0: record basic informal certificates to study pair programming vs. code inspections
- v. 1.1: added Active Certificate functionality
- v. 1.1.2: added support for Eclipse rename refactoring
- v. 1.2: currently adding coverage statistics

- Slide  of 17

## DevCOP Eclipse Plugin Demo

- Available: http://agile.csc.ncsu.edu/mssherri/devcop/

- Slide  of 17

## Limitations

- Granularity of Certificates
- Method level, not class or line of code
- Composition of Certificates
- Not much is known about how one V&V technique adds to another
- Defect Severity
- All defects are treated equally
- Certificate rubric specificity
- Making relative effectiveness scale is hard

- Slide  of 17

## Validation

- Base Model with Customization
- Take model and calibrate it to projects...
- ... from different companies / development teams
- ... of different scopes
- ... with different programming languages
- ... with different V&V styles
- Show that model can be effective estimator in each scenario

- Slide  of 17

## Potential Side Effects

- Retrospective Causal Analysis
- Once certificates are recorded on a project and bugs are reported, developers can use certificate and defect information to evaluate the efficacy of their V&V practices.
- Building certificate information in with compiled code base
- If certificate information could travel with compiled code, it could be referenced at runtime so that other systems could evaluate whether it wants to work with that system.

- Slide  of 17

## Thank you!

- Questions?  Queries?  Quandaries?
- Contact Information:
- 	Mark Sherriff
- 	mark.sherriff@ncsu.edu
- 	http://www4.ncsu.edu/~mssherri/
- 	http://research.csc.ncsu.edu/softwareengineering/realsearch/
- DevCOP Plugin: http://agile.csc.ncsu.edu/mssherri/devcop/

- Slide  of 17