Issues:
1. If validate perms (validating the parent obj etc) fails, the action done by http_post_collection is not undone. In the case of instance-ip/floating ip, we may endup leaking a route.
2. Subnet allocator zookeeper is not deleted on delete of subnet or on update of virtual network
3. Retry of delete node doesn’t pass recursive flag
4. Confusing stevedore error due to parameter mismatch

Fix:
1. Move all validation before starting to make state changes. (http_post, http_put)
    2. For all state modification, append a cleanup routine and walk the clean up routine in failure path
    3. Added “Failure/Cleanup” hook for all required classes.
    4.  Fix the parameter list for stevedore extension method
    5. Fix subnet delete to pass correct subnet name(which should include virtual network fq name)

Sanity report: http://10.204.216.50/Docs/logs/0000_2014_03_25_17_23_01/test_report.html
Three issues identified.
1. Crash due to state missing
    Route is matched by the match condition and state is added before adding to WorkQueue.
    Route Entry gets deleted and delete request is added to Work Queue
    Route entry gets reused and Add request is now added to WorkQueue and subsequently the route entry gets deleted.
    When the work queue is processed with above 3 events, we can delete the MatchsTate with route still present in the WorkQueue.
2. Crash due to early unregister of DBClient even with DBState pending.
    When service chain config is deleted, the Unregister of the condition is done after Walk is completed. This event can occur before all MatchSTates are removed from the route. Due to this RemoveMAtchState function can possibly get called after the client is unregistered.
3. Concurrency issue in setting/accessing “stopped” flag of service chain object. It was set in the WalkDone function(in db:DBTable task) and flag is checked in Match function(Called from Notify in db:DBTable task). Since these two can be running in parallel (From different partition task), we can enqueue requests/set-state even after stopping the service chain. Which was the root cause calling unregister with MatchState set.

Solution:
To overcome the above issues,  I have made following fixes
1. Maintain a ref count on MatchState. Refcount is taken when the route is added to the work queue. RemoveMatchSTate is only called when refcount goes to zero. IF the route entry gets reused, recycle the state by resetting the delete flag
2. Track the number of MatchSTate added on a MatchCondition. UnregisterCondition is called only after all MatchStates are deleted.
3. Removed the "stopped" flag as the fix handles entries after the service chain is stopped.
4. added test cases which recreates the crashes. (One unregister with state pending and Missing Match state while handling add request).
1. If the auth token is not specified, call to fetch auth token from keystone (_authenticate()) is done in try catch. In case of exception, give an error message “unable to authenticate”
2. If auth token is specified, and authentication fails, no attempt should be made to fetch token again. Instead return error.
3. Authentication protocol is made configurable(key stone access may be secured(https) or unsecured(http))
Tweak test to exercise code in SchedulingGroup::~SchedulingGroup.
Add more tests for RTargetRoute.
Address agent review comments - arp module
Address agent review comments - pkt
Add tests for ShowBgp[Instance|Neighbor|Peering]Config.
Adding task exclusion between stats_collector and PktFlowResponder to ensure synchronized access
Merge pull request #537 from Juniper/psethi_bugs

Adding task exclusion between stats_collector and PktFlowResponder
Merge pull request #535 from Juniper/nsheth_test_coverage

Ongoing work to improve UT coverage for BGP code.
Merge pull request #511 from Juniper/2397

Fix for 2397
Merge pull request #513 from Juniper/auth_token_vnc_api

fix for https://bugs.launchpad.net/opencontrail/+bug/1297579
Initialization files for api-server and schema-transformer
Add test for ShowBgpServer.
Add test for ShowXmppServer.
Fix formatting in bgp_sandesh.cc.
Add tests for BgpPeering update.
Adding datafile for scaled config
1) consolidate all flowseries select processing where raw data is requested 2) Flowseries UT