# Presentation: 579683

## caArray 2.0 Lessons Learned Meeting

- February 15, 2008

## Goals of this meeting

- Recognize our accomplishments
- Identify what did not go well
- Start discussing how we can improve!

## Setting expectations

- Today’s session will provide input into where we can improve processes.  We will not make decisions today.
- Keep the discussion professional

## Why do it?

**I don't think it matters how you conduct the postmortem, as long as you do it.** Most shops are far too busy rushing ahead to the next project to spend any time thinking about how they could improve and refine their software development process. And then they wonder why their new project suffers from all the same problems as their previous project
- http://www.codinghorror.com/blog/archives/000736.html

## Project accomplishments

- Project duration – 9 months
- Inception started: April, 2007
- Product delivered: February 5, 2008
- Components delivered
- Web application
- caBIG programmatic interfaces
- Documentation
- Installation package
- Central installation

## Process accomplishments

- Implemented an entirely new build and deploy process
- Maintained fully documented use cases
- Communication with the community throughout the project
- Evolved into a closely working team

## Best measure of success is customer feedback

_**From the caArray_Users listserv, 6:17pm yesterday**_
- .....HOWEVER, I understand that this is our institutional preference -- _**and since caArray is such an attractive product I may make an exception.**_
- thank you,
- -andy
- Andrew McMurry
- Sr. Research Software Developer
- Harvard Medical School Center for BioMedical Informatics
- Children's Hospital Informatics Program @ Harvard/MIT HST
- _https://cbmi.med.harvard.edu/spin_
- _http://chip.org/ihl_
_**From the caArray_Users listserv, 3:48pm yesterday**_
- We are setting up a caArray installation ourselves at The Jackson Laboratory.  While I've run into a few stumbling blocks as an early adopter of caArray 2.0, _**I have to say that the new installer the caArray team provides is pretty slick**_**.** It does the jboss installations, and deployment to the jboss server for you.
- Dave
- Dave Walton
- Manager Scientific Computing
- Computational Sciences
- The Jackson Laboratory
- Bar Harbor, Maine 04609
- dave.walton@jax.org

_**From the caArray_Users listserv, 2/6/08**_
- The installers are very nice, and are very refreshing after having battled with the much older versions of caArray 1.x.
_**Our instance is up at**_
- _[http://caarray2.lbl.gov:8080/caarray](http://caarray2.lbl.gov:8080/caarray)_
_**and everything seems to be working nicely.**_
- Thanks. Great job,
- -g
- (Gerald Fonenay, LBNL)

## Areas needing improvement

- Prioritizing efforts on high risk areas
- Ensuring appropriate test coverage across the product
- Even better customer engagement
- Prioritizing documentation
- Both internal and external

## Team presentations

- Development - Bill
- IQA - Tom
- Curation – Xiaopeng
- Documentation – Jill
- Systems – Sriram
- DBAs – Andrea
- Application Support - Don

## Development

## Dev - What Went Right

- Good cross-team-member communication; Dev interacted with QA and Systems to solve problems
- Use cases & UI mock-ups guided development well
- Dev team working as a single unit; developers picking each other’s work when necessary
- AHP build integration greatly reduced time to get builds out to various tiers.
- If needed, we could create a new build in the morning and move it up to production by the afternoon
- Code peer review process has been very successful.
- Access to DEV and QA tiers was critical to success (ie, performance testing / logs / etc).  big thanks to systems for this.

## Dev - What went wrong/do better...

- We had enough environments for testing, though it would be better if each tier had the same or very similar configuration.
- Better management of the SVN repository – restructure so the large test files, environment zip files and documents do not get checkout for every build
- Improve quality of bug reports.  Lots of dev cycles were spent to understand what was being reported.  Many were not bugs, or duplicates of existing behavior.
- Improve unit testing. We needed more robust unit tests to verify the functional behavior of the business layers

## Dev - What went wrong/do better...

- We need to investigate having debugging, profiling, and monitoring (JMX) hooks on as many of the tiers as possible - ideally on DEV, QA, and STAGE. This will help diagnose tricky bugs.
- We had several issues with NCI technologies (csm / grid / sdk) that should get fixed in the core products
- Need to incorporate remote API and Grid testing in the CI build process. Several times the build was successful but later found out commits had broken the remote API and/or grid functionality.

## Dev - What went wrong/do better

- Installation/Deployment should have had more testing *without* a network connection so that we didn't discover problems late in the development cycle (e.g. database and SMTP connections)
- Hosting target environments outside the NCI network to improve testing (emulate the cancer centers experiences better)
- Run installer creation as part of a nightly build - since it's tightly integrated with the caarray build.xml
- Capability of "scorching" target environments down to the operating system and automatically reinstalling components (e.g. database, web container, etc.)

## Dev - How can we do better

- Work with systems to get builds done in their “build window” rather than building “just in time”
- Create builds before they are “due”
- Get more user feedback early in the process
- Everyone needs to read the use cases before starting their task.
-   Dev before developing
-   QA before writing a test case
-   PM to understand all requirements
- Improve Performance testing.
- Work with large data sets in realistic situations.
- Found major issues way too late in the game
- More developer-level documentation about all the pieces of the project (grid, remote API, load tests, etc) .

## Independent QA

## caArray 2.0 Lessons Learned: the IQA View

- What went right
- Compared to caArray 1.6, the defect workflow process was much improved, and the process just got better as time went on.
- We all made all our deadlines.
- Compared to caArray 1.6, the installation process is much less cumbersome
- RUP was a great workflow organizing system.
- The artifacts were much more numerous and much more useful.
- The distribution of the workload throughout the time period was quite uniform. The workload never got onerous (at least not for IQA).
- The contribution of DEV resources to testing (Bill and Rashmi et al) was a big help.
- The quick response of DEV to open tickets was great.
- DEV and ABD worked well with QA on resolving deployment issues.
- Even though QA team members were initially unfamiliar with some of the technologies, with DEV and ABD’s help they quickly got up to speed.

## caArray 2.0 Lessons Learned: the IQA View

- What went wrong
- The QA team effort was not well organized, with unclear division of labor: several times effort was duplicated.
- Dev team often delivered large numbers of partially complete functions, rather than a smaller number of complete (but more testable) functions.
- Test coverage was barely adequate.
- Fixed defects weren’t being closed promptly, although sometimes it was unclear whether the defect was in fact ready to be closed.
- Some defects submitted by QA team were not given appropriate weight by DEV team.
- There was inconsistency in application behavior/performance between environment hardware (dev laptop vs. QA VMware vs. production vs. LSD).
- QA on occasion ignored the artifacts/documentation, which was duplicated between Word documents and GForge trackers.
- Use cases didn’t always have clearly documented scenarios.
- QA was never able to formulate scientifically useful scenarios.
- Data validation efforts were insufficient.
- The test data files were scattered all over the place, even though there was an effort to consolidate it.
- Data provenance determination was attempted but was deficient.
- Deployment process  standard SOP (DevQASTAGEPROD) was often ignored, or had workarounds.

**Notes:**
DEV

## caArray 2.0 Lessons Learned: the IQA View

- How can we do better?
- Need clear criteria for what constitutes a legitimate defect.
- More, regular QA reports to product management team
- A ‘QA coordinator’ that can help assign tasks and monitor progress
- Delivery of complete functional components rather than partially completed components.
- GForge tracker entries need to follow a standard format (this improved with time).
- The hard cases (gigantic array designs, huge zip files, obscure formats) must be tested earlier.
- QA needs to be able to evaluate how well caArray cooperates with other caBIG applications (caTissue, caGWAS, caGRID, caBIO, caDSR, geWorkbench, GenePattern).
- Sending large datasets over the enterprise Grid (node to node) needs to be able to be simulated.
- QA needs to provide more documentation of their tests and test results. The defect tracker items had good detail of both the tests and the results, but for tests that never resulted in bugs, little was captured (good news is no news).
- Deployment and initial usage documentation (e.g. tutorials) needs to be evaluated by QA.
- There need to be test cases that simulate what actual end users might actually do.

## Curation

## Documentation

## Documentation: What went right

- Fairly good communication at the end
- Reaching out to me with my logistical challenges
- Brent’s patience

## Documentation: What went wrong

- Unreasonably short doc development time
- GUI changing until the very end (lack of reasonable code freeze)
- Poor communication (see last slide)
- Slow reviews of documents
- Few reviewers
- Lack of SMEs other than Brent

## Documentation: To improve the process

- Plan a reasonable time frame (at least 3 months)
- Provide better communication about:
- Feature changes
- Dev doc changes
- New builds
- Availability of test data
- Document deadlines
- Observe a reasonable code freeze date
- Assign a SME other than head honchos
- Faster doc review turnaround

## Systems and DBAs

## Systems

- What Worked!
- Successful auto deployments (build & deploy)
- Use of 4 tiers effectively
- Migration of caArray2.x – Linux Platform
- Increased communication between teams
- Scrums were helpful!
- Effective work-around/solutions developed immediately – wherever feasible

## Systems

- What didn’t?
- Understanding of application requirements
- Auto-deployment pilot process – Interaction between different groups (early stages)
- Not having DEV and QA servers behind BIG-IP
- Split communications
- Hardware constraints
- Approach issues in neutral gear
- Consistent code promotion through the tiers – differences in build process

## Systems

- How can we improve?
- We are a TEAM with common GOALS
- More test plans to detect issues early-on – Reduce crunch times
- Positive communication with focus on issues
- Provide test cases to simulate issues
- Knowledge of app design requirements
- Collaborate to generate documents
- Form working groups to address critical issues – Reduce duplication (probably?)

## Database

- What Worked!
- Successful auto deployments
- Moving Prod database to Staging server
- What didn’t?
- Performance issues
- SAN
- Queries
- Sub-select statements
- How can we improve?
- Keep communication open especially during loading
- Analyze vs Optimizing the database (schedule)
- Test plans with large data sets across the tiers (at least QA)

## Application Support