Record and log reason for dhcp agent resync

A dhcp resync can be triggered at a number of points, but the actual
resync is done asynchronously by a helper thread.  This means by the
time the resync happens, it's hard to establish what actually caused
it.

There have been a number of problems in production systems that cause
excessive resyncs.  One is a ipv6/dnsmasq issue (rhbz#1077487) and
another is db corruption with duplicate entries [1].  The resync
triggers a whole lot of logs itself, so it becomes very unclear how to
establish any causality.

This change keeps track of what triggered the resync with some helpful
information.  We add a schedule_resync() function to to replace a
explicit set of needs_resync which tracks why it was called.

[1] The logs will contain output like "DBDuplicateEntry
(IntegrityError) (1062, "Duplicate entry
'6d799c6a-7a09-4c1e-bb63-7d30fd052c8a-d3e3ac5b-9962-428a-a9f8-6b2' for
key 'PRIMARY'") ..." in this case

Closes-Bug: #1294445
Change-Id: I9b1c6202f5a6bbad8589a8b64b2a38c9d9edb43b
remove token from notifier middleware

notifier middleware is capturing token and sending it to MQ. this
is not advisable so we should filter it out.

Closes-Bug: #1321080
Change-Id: Ia1bfa1bd24989681db1d2f385defc12e69a01f8d
Add NVP advanced service check before deleting a router

Since NVP advanced services(LB/FW/VPN) are all routed inserted,
it is necessary to check service in use before deleting a router.
    *If there is one service inserted into the router, it would raise
RouterInUseByXXService error when deleting the router.

Change-Id: I9844f53918cad17cafde0783fe6b69cff246791d
Close-Bug: #1295494
NSX: Fix request_id in api_client to increment

Previously, the NSX request_id in the api_client would always be
0 because the current request id was stored in the class which is
always initialized on each request. This patch fixes that by storing
the request_id as a class variable.

Change-Id: I9ff4d3eb09899fe6d14b462dddf9daddbf77873b
Closes-bug: 1325100
Neutron does not follow the RFC 3442 spec for DHCP

When setting a gateway and additional host routes in neutron subnet, the
gateway is only sent to clients via the router dhcp option, dhcp clients
conforming to rfc3442 will ignore router option if
classless-static-routes are available. This patch ensures setting both
the router option and the classless-static-routes including the gateway

Change-Id: Ia00b9385025020f848872309ae42ddac08528f53
Closes-Bug: #1317935
Big Switch: fix capabilities retrieval code

References the raw REST response in the capabilities
parsing code so json.loads doesn't get an object
that may already be decoded and fail.

Closes-Bug: #1326173
Change-Id: Ia3179b7499f35a6ab2e9ce1631ab15ed27d64647
Improve module-level mocks in midonet tests

Remove unnecessary module-level mocks and replace
them with test-specific mocks so they don't persist
through the entire test suite.

Partial-Bug: #1316401
Change-Id: I49039749bb700ec0a88639f4772701ba3b920352
add engine parameter for offline migrations

Offline migration required config file containing connection string.
In that case only engine from URL was used. With this patch engine can be
passed from command line (or config file) along with plugins which sql
script will be generated accordingly.

DocImpact

Change-Id: Ib667a71960b833fd981f97fe7d6b1856084ef5c8
Closes-Bug: #1304326
Pass object to policy when finding fields to strip

During the evaluation of fields to strip in responses to list
operations, pass also the first object in the list to the
policy engine.
This will avoid errors in policy evaluation if during
an upgrade from icehouse policy.json was not updated to remove
attribute-level policies dependent on resource values.

Closes-Bug: #1323715

Change-Id: Iaa6ed3bbf4a07ce0b614a4197cabdfa1cb36d427
Add flat type driver unittests

Partial-Bug: #1269127

Change-Id: Ib56c5823ddb927d8269cfcdaa8df8b0f78dc5a6e
Remove run-time version checking for openvswitch features

The current method of checking openvswitch and kernel versions for
specific feature support is brittle, distro-specific and unsupportable.

This patch removes the runtime version checks and implements a test
script which allows testing specific neutron features or can test
that all features required by a specific configuration are available.

For example, to test VXLAN support in openvswitch, either:

  neutron-sanity-check --ovs_vxlan

or pass in the deployed configuration files with AGENT/tunnel_types
containing 'vxlan', like:

  neutron-sanity-check --config-file /etc/neutron/plugins/ml2.conf.ini

Deployment tools can then test that the required features exist without
relying on version numbers and without incurring a penalty every time
the agent is started.

Implements: blueprint remove-openvswitch-version-check
DocImpact
Change-Id: I5e822d97b9bece9242ba5ac8901aa0ca4f090cfe
Allow neutron-sanity-check to check OVS patch port support

This patch allows to check OVS patch port support using:

  neutron-sanity-check --ovs_patch

or pass in the deployed configuration files with a non-empty
AGENT/tunnel_types (indeed patch ports are used to interconnect
br-int and br-tun):

  neutron-sanity-check --config-file ml2.conf.ini

DocImpact
Related-Bug: #1285335
Change-Id: Ic7bfabd1f4b85bfa490dd06c70eb5682ee42a681
Merge "Record and log reason for dhcp agent resync"
Merge "Remove run-time version checking for openvswitch features"
Merge "Allow neutron-sanity-check to check OVS patch port support"
Merge "Neutron does not follow the RFC 3442 spec for DHCP"
Merge "Pass object to policy when finding fields to strip"
Merge "Improve module-level mocks in midonet tests"
Merge "remove token from notifier middleware"
Merge "Add flat type driver unittests"