# Presentation: 860255

## NGOP Status and Plans

- Jim Fromm
- Marc Mengel
- Jack Schmidt
- May 2, 2006

## Today’s talk...

- Current Status
- Farms/CMS/General Server split
- Recent Enhancements
- Performance Tuning
- Configuration File cleanup
- CMS Enhancements.
- Future Enhancements

## Current Status: Farms/CMS/General Server Split

- Goals:
- Relieve bottlenecks by splitting out the servers
- Reduce configuration upgrade times
- Provide groups with independence
- Simplify the General server by consolidating the two machines into one.

## Current Status: Farms/CMS/General Server Split 

- Bottlenecks
- Farms and CMS Server hangs have been non-existent since split.
- General Server has experienced occasional hangs, but to a lesser degree (still two systems).
***This goal has been successfully met.***

## Current Status: Farms/CMS/General Server Split 

- Reduction of configuration upgrade times
- Prior to the split, it took 2+ hours to perform a system configuration upgrade when things went well.
- Farms/CMS
- Takes less than 20 minutes to perform a configuration upgrade
- Less monitored elements per server
- One status engine allowed for the removal of Warshall’s algorithm for finding the transitive closure of a graph.

## Current Status: Farms/CMS/General Server Split

- General Server
- Configuration upgrade time reduced to less than 30 minutes
- Recent parser optimizations will likely cut configuration upgrade  times to 1⁄4 .
***This goal has been successfully met***.

## Current Status: Farms/CMS/General Server Split 

- Server Independence
- Both CMS and Farms are up to speed with doing their own configurations.
- Upgrades are performed only when they need them.
- CMS (Gary Stiehr) has taken the initiative to add several features.
- Both groups have taken advantage of the splitting of the cluster.
***This goal has been successfully met.***

## Current Status: Farms/CMS/General Server Split

- General Server Consolidation
- Not complete: still using two servers.
- Doesn’t have the urgency as the other items, and has been easy to put on the backburner.
- Need to make this a priority.

## Recent Enhancements 	

- Performance Tuning
- Preprocessor speedup.
- Marc Mengel implemented a change that improved performance of the XML preprocessor.
- NGOP preprocessor expands If_xxx/For_xxx tags
- Was using 90% CPU on startup.
- This was a known python performance issue.
***Stunning improvements on configuration upgrade times!***

## Recent Enhancements

- Configuration File Cleanup
- New "grand unified" XML Document Type Description _[http://www.fnal.gov/docs/products/ngop/ngop_unified.dtd](http://www.fnal.gov/docs/products/ngop/ngop_unified.dtd)__ _
- XML editor friendly
- Works well with Merlin XML editor.

## Merlin Screenshot

## Recent Enhancements	

- CMS
- No Downtimes: Modified to allow multiple status engines roles to be defined for one set of definitions.  This allows re-configuration on one while the other remains active, eliminating downtimes due to configuration upgrades.
- Used the SE API to create GUI that only shows “bad” things.
- Developed a generic plug-in agent that allows for a standard way of defining agents in the CMS system.

## Future Enhancements

- Dynamic Configuration Upgrades
- By far the most difficult enhancement to implement.
- CMS needs have been addressed with the multiple status engine solution.
- With reduction of configuration upgrade times coupled with the CMS workaround, this requirement becomes a very low priority.

## Future Enhancements(Cont)

- CMS specific requested enhancements:
- Marking Monitored Elements down across clusters.
- Accelerate alarms based on time (i.e. yellow becomes red after 8 hours)
- Verify scalability to CMS planned growth.
- Documentation upgrade
- General
- Improvement of logging subsystem
- Research UDP protocol issues
- Dropped packet issue seems under control with recent network tunings
- May need to do this anyway to address CMS requirements for scalability.
- Web/Swatch agents need DELAY/GAP parameters
- “Anti” rules for Swatch agent

## Future Enhancements(Cont)

- Wish List
- Real dynamic configuration
- SNMP agent
- Email watcher

## Summary	

- Split of farms and CMS has been successful:
- Quicker reconfigs result in less downtime.
- Splitting load has reduced NGOP hangs.
- CMS and Farms groups are managing things on their own timetable.
- Need to consolidate General server to one machine
- New release is needed:
- New CMS requests
- Investigate potential scalability issues.
- Improved logging
- New and improved agents.
- Revamp documentation and website.
- Develop maintainable metrics

## Information

- Main Site:
- _[http://www-isd.fnal.gov/ngop/ngop.html](http://www-isd.fnal.gov/ngop/ngop.html)_
- Documentation:
- Users Guide-
- _[http://www-isd.fnal.gov/ngop/current/ngop_ug.htm](http://www-isd.fnal.gov/ngop/current/ngop_ug.htm)_
- Admin Guide-
- _[http://www-sd.fnal.gov/ngop/current/ngop_admin_guide.htm](http://www-sd.fnal.gov/ngop/current/ngop_admin_guide.htm)__ _