Record and log reason for dhcp agent resync

A dhcp resync can be triggered at a number of points, but the actual
resync is done asynchronously by a helper thread.  This means by the
time the resync happens, it's hard to establish what actually caused
it.

There have been a number of problems in production systems that cause
excessive resyncs.  One is a ipv6/dnsmasq issue (rhbz#1077487) and
another is db corruption with duplicate entries [1].  The resync
triggers a whole lot of logs itself, so it becomes very unclear how to
establish any causality.

This change keeps track of what triggered the resync with some helpful
information.  We add a schedule_resync() function to to replace a
explicit set of needs_resync which tracks why it was called.

[1] The logs will contain output like "DBDuplicateEntry
(IntegrityError) (1062, "Duplicate entry
'6d799c6a-7a09-4c1e-bb63-7d30fd052c8a-d3e3ac5b-9962-428a-a9f8-6b2' for
key 'PRIMARY'") ..." in this case

Closes-Bug: #1294445
Change-Id: I9b1c6202f5a6bbad8589a8b64b2a38c9d9edb43b
remove token from notifier middleware

notifier middleware is capturing token and sending it to MQ. this
is not advisable so we should filter it out.

Closes-Bug: #1321080
Change-Id: Ia1bfa1bd24989681db1d2f385defc12e69a01f8d
Add NVP advanced service check before deleting a router

Since NVP advanced services(LB/FW/VPN) are all routed inserted,
it is necessary to check service in use before deleting a router.
    *If there is one service inserted into the router, it would raise
RouterInUseByXXService error when deleting the router.

Change-Id: I9844f53918cad17cafde0783fe6b69cff246791d
Close-Bug: #1295494
Improve readability of MagicMock use in RYU test

Replaces an unnecessarily complex setup of a magicmock
in an RYU test module.

Change-Id: If2e99ff76114d896a6480a884a531f901198caa5
NSX: Fix request_id in api_client to increment

Previously, the NSX request_id in the api_client would always be
0 because the current request id was stored in the class which is
always initialized on each request. This patch fixes that by storing
the request_id as a class variable.

Change-Id: I9ff4d3eb09899fe6d14b462dddf9daddbf77873b
Closes-bug: 1325100
NSX: fix tenant_id passed as security_profile_id

Previously we were passing the tenant_id as the security_profile_id
to NSX so these ids would be switched around in the system backend.
This does not affect any operations as this is just extra metadata
in nsx to help an operator debug.

Change-Id: I1afdcc3621fef8fdd7d4eb3c57bff749a71d0d33
Closes-bug: 1325118
NSX: fix invalid docstring

Updated the text of docstring.

Change-Id: Ic82fe4719c8b04d01cfb5ebb69411f2f35e7f646
Radware LBaaS driver is able to flip to a secondary backend node

Change-Id: Ifbfef493d5339f61dcf58dddcc8e3830aaf06bf1
Closes-Bug: #1324131
Neutron does not follow the RFC 3442 spec for DHCP

When setting a gateway and additional host routes in neutron subnet, the
gateway is only sent to clients via the router dhcp option, dhcp clients
conforming to rfc3442 will ignore router option if
classless-static-routes are available. This patch ensures setting both
the router option and the classless-static-routes including the gateway

Change-Id: Ia00b9385025020f848872309ae42ddac08528f53
Closes-Bug: #1317935
BSN: Set hash header to empty instead of False

Sets the consistency hash header to empty instead
of False since 'False' is handled like a string on
the backend and requires special-casing to detect.

Closes-Bug: #1325771
Change-Id: Iee1651574c01a32e78167a9bbed4e0433abbdec2
Log firewall status on delete in case of status inconsistency

This should help to troubleshoot related gate failres.

Change-Id: I92d4d886cafed5faeb2a6d0b224f4b5dc8522100
Related-Bug: #1314313
NSX: bump http_timeout to 30 seconds

We've seen that sometimes NSX takes longer then 10 seconds to return
in production setups under high bursts. Setting this value to 30 seconds
seems more appropriate and solves the issue at load.

Suggestion by: Jacob Cherkas

Change-Id: If604d5d52d15cc1165e19d20b38774dfecec275f
Closes-bug: 1325055
Improve iptables_manager _modify_rules() method

As the number of ports per default security group increases, the
number of iptables entries on the Compute Node grows.  Because of
this, there is a gradual increase in the time taken to apply chains
and rules.

Currently we are using list comprehensions to find if a new chain or
rule matches an existing one.  Instead, walk through the list in
reverse to find a matching entry.

Added a new method, _find_last_entry(), to return the entry we are
searching for.

Change-Id: I3585479ffa00be556b8b21dc9dbd6b36ad37f4de
Closes-Bug: #1302272
Related-Bug: #1253993
Big Switch: fix capabilities retrieval code

References the raw REST response in the capabilities
parsing code so json.loads doesn't get an object
that may already be decoded and fail.

Closes-Bug: #1326173
Change-Id: Ia3179b7499f35a6ab2e9ce1631ab15ed27d64647
Improve module-level mocks in midonet tests

Remove unnecessary module-level mocks and replace
them with test-specific mocks so they don't persist
through the entire test suite.

Partial-Bug: #1316401
Change-Id: I49039749bb700ec0a88639f4772701ba3b920352
add engine parameter for offline migrations

Offline migration required config file containing connection string.
In that case only engine from URL was used. With this patch engine can be
passed from command line (or config file) along with plugins which sql
script will be generated accordingly.

DocImpact

Change-Id: Ib667a71960b833fd981f97fe7d6b1856084ef5c8
Closes-Bug: #1304326
Refactor some router-related methods

This is done in preparation for the distributed
router functionality. Breaking down and factoring
out some functionality helps the DVR work to come
as a more loosely coupled addition to the L3
centralized case. This also ensures that the two
code bases are kept separately to minimize chance
of regression, and simplify code coverage effort.

Partially-implements: blueprint neutron-ovs-dvr

Change-Id: Ide1bc193666ce03a3ae26b1869bde8093ed453ef
Allow L3 base to handle extensions on router creation

By changing the boolean flag, API extensions made to
the router model can be handled correctly: this means
that on router creation, the response body will
contain all the extension attributes being part of
the resource. Prior to this fix, it was only on GETs
or PUTs, leaving the user at loss as to whether
the flag was actually being processed.

Closes-bug: #1325608
Supports-blueprint: neutron-ovs-dvr

Change-Id: I6f913c8417676a789177e00f30eb5875e7aaa3ae
Pass object to policy when finding fields to strip

During the evaluation of fields to strip in responses to list
operations, pass also the first object in the list to the
policy engine.
This will avoid errors in policy evaluation if during
an upgrade from icehouse policy.json was not updated to remove
attribute-level policies dependent on resource values.

Closes-Bug: #1323715

Change-Id: Iaa6ed3bbf4a07ce0b614a4197cabdfa1cb36d427
Add flat type driver unittests

Partial-Bug: #1269127

Change-Id: Ib56c5823ddb927d8269cfcdaa8df8b0f78dc5a6e