Issues:
1. If validate perms (validating the parent obj etc) fails, the action done by http_post_collection is not undone. In the case of instance-ip/floating ip, we may endup leaking a route.
2. Subnet allocator zookeeper is not deleted on delete of subnet or on update of virtual network
3. Retry of delete node doesn’t pass recursive flag
4. Confusing stevedore error due to parameter mismatch

Fix:
1. Move all validation before starting to make state changes. (http_post, http_put)
    2. For all state modification, append a cleanup routine and walk the clean up routine in failure path
    3. Added “Failure/Cleanup” hook for all required classes.
    4.  Fix the parameter list for stevedore extension method
    5. Fix subnet delete to pass correct subnet name(which should include virtual network fq name)

Sanity report: http://10.204.216.50/Docs/logs/0000_2014_03_25_17_23_01/test_report.html
Three issues identified.
1. Crash due to state missing
    Route is matched by the match condition and state is added before adding to WorkQueue.
    Route Entry gets deleted and delete request is added to Work Queue
    Route entry gets reused and Add request is now added to WorkQueue and subsequently the route entry gets deleted.
    When the work queue is processed with above 3 events, we can delete the MatchsTate with route still present in the WorkQueue.
2. Crash due to early unregister of DBClient even with DBState pending.
    When service chain config is deleted, the Unregister of the condition is done after Walk is completed. This event can occur before all MatchSTates are removed from the route. Due to this RemoveMAtchState function can possibly get called after the client is unregistered.
3. Concurrency issue in setting/accessing “stopped” flag of service chain object. It was set in the WalkDone function(in db:DBTable task) and flag is checked in Match function(Called from Notify in db:DBTable task). Since these two can be running in parallel (From different partition task), we can enqueue requests/set-state even after stopping the service chain. Which was the root cause calling unregister with MatchState set.

Solution:
To overcome the above issues,  I have made following fixes
1. Maintain a ref count on MatchState. Refcount is taken when the route is added to the work queue. RemoveMatchSTate is only called when refcount goes to zero. IF the route entry gets reused, recycle the state by resetting the delete flag
2. Track the number of MatchSTate added on a MatchCondition. UnregisterCondition is called only after all MatchStates are deleted.
3. Removed the "stopped" flag as the fix handles entries after the service chain is stopped.
4. added test cases which recreates the crashes. (One unregister with state pending and Missing Match state while handling add request).
1. If the auth token is not specified, call to fetch auth token from keystone (_authenticate()) is done in try catch. In case of exception, give an error message “unable to authenticate”
2. If auth token is specified, and authentication fails, no attempt should be made to fetch token again. Instead return error.
3. Authentication protocol is made configurable(key stone access may be secured(https) or unsecured(http))
Bug 2367: UVE must be sent in a separate loop because connections could be altered as a result of modifying the remote network
Add an option to specify rabbitmq server
Fixes for config systemless test - making it run without rabbitmq, and without requiring the presence of /var/log/contrail
DHCP Snoop is enabled for a port in fabric-vrf without IP address
Maintain DHCP Snoop table in interface table.
Use DHCP Snoop table to add IP address of interface
Support renew of DHCP from fabric port
Add test code for DHCP Snooping
Merge remote-tracking branch 'github/master' into anishm_fixcfg
Merge pull request #525 from praveenkv/master

DHCP Snoop is enabled for a port in fabric-vrf without IP address
Making exception handling for /var/log/contrail more specifig
Merge pull request #524 from Juniper/anishm_fixcfg

Fixes for config systemless test - making it run without rabbitmq, and w...
Merge pull request #510 from Juniper/ashoksr_ut

-Define Object Factory for agent
Merge pull request #519 from Juniper/sb

Bug 2367: UVE must be sent in a separate loop because connections could ...
Tweak test to exercise code in SchedulingGroup::~SchedulingGroup.
Add more tests for RTargetRoute.
When create node is request throws exception, the node could still be created in zookeeper database. So subsequent request to create the node would return NodeExists error. To handle this error, validate the node content against the data passed to created node. IF the create_node is requested without any data, allocate unique(uuid) and use it for create node. Sanity result: http://10.204.216.50/Docs/logs/0000_2014_03_29_17_33_26/test_report.html
Fix spelling error (vnswad)
Merge pull request #531 from sbadia/master

Fix spelling error (vnswad)
Fix for bug 2713 - Make ifmap_state_machine_test run faster
Change the timer from seconds to milliseconds and other code that needs to
change because of this change. Change the testcases appropriately and also
change the testcases so that they pass with high gtest_repeat values.
Add test file