# Presentation: 070771

## Understanding Simulators Using Tree-Query Language

- Tim Menzies1, Ying Hu1, James D. Kipers2
- 1SE, ECE, UBC, Canada
- 2Com Sci, Miami, Oxford, Ohio, USA
- _[tim@menzies.com](mailto:tim@menzies.com)_, _[huying](mailto:huying_ca@yahoo.com)__[_ca@yahoo.com](mailto:huying_ca@yahoo.com)_, _[kiperjd@muohio.edu](mailto:kiperjd@muohio.edu)_

- New idea: Treatment learners

## Business motivation

- IV&V practitioners** **seek:
- Cheapest questions from IV&V to the projects
- Conversations take time
- Data collection = $$$
- Least changes proposed by IV&V to the projects
- Organizational change is hard
- Fewest monitors required for the IV&V effort
- Monitoring = on-going data collection = $$$
- Method:
- Knowledge farming
- Treatment learning

## Knowledge Farming	

- If  data is plentiful:
- Then data mining (e.g. Khoshgoftaar &  Allen)
- Else, farm it:
- Plant a seed
- Quickly build (or borrow) a model
- Grow the data
- Monte Carlo (ish) simulations
- Harvest
- Summarize
- Happy surprise:
- When we explore a large space of “maybes”...
- ...Stable conclusions exist.

## Planting some seeds

- ARRT
- The CMM2 model-
- 350 lines of Prolog
- Unsure of influence weights?
- Jiggle!
- Unsure which of N to do?
- At runtime, pick at random!
- The COCOMO-R risk mitigation model
- Risk of development time over-run
- Unsure of internal tuning parameters?
- Pick randomly from tunings in the literature
- Unsure of inputs (e.g. SLOC)?
- Pick randomly from possible ranges.
- Generate sketches of local intuitions
- Qualitative models, balanced score cards, fault trees,...
- Unsure if X or not X? Add them both in!

**stableRequirements   if ****    effectiveReviews @ 0.3****    and  requirementsUsed @ 0.3****    ....****    and (workProductsIdentified @ 0.3****             or ...****             or  softwareTracking @ 0.3).**

## Harvest:    improve the scores

- Harvest:    improve the scores

-    sunny, 	    85, 	   86, 	false, 	none (21=$2)
-    sunny, 	    80, 	   90, 	true, 	none
-    sunny, 	    72, 	   95, 	false, 	none rain, 	    65, 	  70, 	true,          none
-       rain, 	    71, 	   96, 	true, 	none
*      *rain, 	    70, 	   96, 	false,    	some (22=$4)
-       rain, 	    68, 	   80, 	false,  	some rain, 	    75, 	   80, 	false, 	some
-      sunny,      69, 	   70, 	false, 	lots    (23=$8)
-      sunny,      75, 	   70, 	true, 	lots
-   overcast,     83, 	   88, 	false, 	lots
-   overcast,     64, 	   65, 	true, 	lots
-   overcast,     72, 	   90, 	true, 	lots
-   overcast,     81, 	  75, 	false, 	lots

*Humidity= 90.. 96%*

*Outlook=*
*overcast*

*   *

*Delta(outlook.overcast)=10=*
*        **((8-2)*(4-0))+((8-4)*(4-0))**                   (4+0+0)*

## Harvesting from data: golf

**Least monitor:**
**watch the humidity- alert if rising over 90%**

*If humidity=90..97 *
*Then lots of  golf 1/(4)= 25%*

**BTW- deltaf >> tree query ****(faster, simpler).**

## Harvesting from seed1: ARRT

- Harvesting from seed1: ARRT

- ARRT: manual risk balancing
- TAR2+ARRT:
- 88 possible actions (28 8  billion*billion*billion combinations)
- Random combinations  <benefit,cost>
- Seek actions high benefit, low cost

***P697=Y***
*** ******P919=Y***
*** ******P753=Y***
***P692=N***
*** ***

***Untreated:***

***Treated (found automatically in 88 secs):***

**worst                           best**

**worst                           best**

***do!***

***don’t!***

## Harvesting from seed2: CMM-2 

**Same**** **
**Not 2**** **
**Lower cost of requirements**** used (via B?)**

**(A) Using ultra-lightweight formal methods such as proposed by Leveson et.al.**
**(B) sharing  requirements documents around the development team in some searchable hypertext format**
**(C) Build test stubs**

**Lower cost of formal reviews at milestones  ****(via A?)**
**Do ****periodic**** ****software**** ****reviews**
**Lower cost of u****nit****  testing (via C?)**

- Warning: general conclusions may not apply to specific projects (as we shall see)

## Harvesting from seed3: COCOMO-R

- Harvesting from seed3: COCOMO-R

## Harvesting from seed4: a qualitative model

- Qualitative constraint model of a circuit :
- 9 switches, 9 bulbs, 3 batteries
- Many unknowns:
- E.g. bulbs may be blown/ok
- Score each run by # shining light bulbs (max=9)
- Run I: Find top treatment T learnt from  run I. T must be:
- Acceptable to users
- And Possible
- Constrain run I+1 with T
- 1 question culls 90% of options

## Discussion

- For coarse-grain decision making
- Don’t fret what you don’t know
- Simulate across the space of don’t knows
- Actions = top ranked treatment(s)
- Monitors = lowest ranked treatment(s)
- To ask the fewest questions:
- Explore issues in order of treatment ranking
- Generality:
- Average case analysis: use with care for safety critical sub-systems
- Works in many domains
- Assumes low-hanging fruit
- Elsewhere: theoretically, empirically, many maybes mean (mostly) the same thing

## Questions or comments?

- (Some random other slides follow)

## Software	

- The TAR2 treatment learner
- Visual C++/Windows
- Fast:
- Thousands of examples in a few seconds
- Free:
- If you tell us what you are doing with it
- Simple:
- Easy to use (good documentation)
- But we’re happy to help
- Dr. Tim Menzies, professor, UBC, tim@menzies.com
- Ms. Ying Hu, masters student, UBC

## Generality

- TAR2 works when
- domains generates a small number of outstandingly
- large deltaf values