Improve ovs_lib bridge management

Add the ability to add and remove bridges, check for bridge
existence, and lookup the bridge associated with a port.

This change is in support of functional testing for an ovsdb
monitor.

Partial-Bug: #1177973

Change-Id: I419923d8d77983997cd347fcf063b0bc367c0bbc
(cherry picked from commit 010bd1f392e67a6fcd276593b8c79acfe41d1cc7)
Add the option to minimize ovs l2 polling

This change adds the ability to monitor the local ovsdb for
interface changes so that the l2 agent can avoid unnecessary
polling.  Minimal changes are made to the agent so the risk
of breakage should be low.  Future efforts to make the agent
entirely event-based may be able to use OvsdbMonitor as a
starting point.

By default polling minimization is not done, and can only be
enabled by setting 'minimize_polling = True' in the ovs
section of the l2 agent's config file.

Closes-Bug: #1177973

Change-Id: I26c035b48a74df2148696869c5a9affae5ab3d27
(cherry picked from commit cb0df591a9508e863ad5d5d71190eca349dc551f)
Cisco nexus plugin fails to untrunk vlan if other hosts using vlan

Without this fix, if two or more compute hosts have
instances which are
sharing a given VLAN on a Nexus switch, and then
all instances on one of the hosts which are using that
VLAN are terminated, while instances which are using
that VLAN on other hosts remain active, then
the VLAN is not being untrunked from the
corresponding interface on the Nexus switch as
expected.

This fix changes the VLAN removal logic from:
----If this the last instance using this VLAN on this switch:
--------untrunk the vlan from the switch interface
--------delete the VLAN from the switch
To:
----If this the last instance using this VLAN on this switch interface:
--------untrunk the vlan from the switch interface
--------If this the last instance using this VLAN on this switch:
------------delete the VLAN from the switch

Note that this bug also exists in the Cisco ML2
mechanism driver, but the code which implements
this is being redesigned, so it will be addressed for
the ML2 separately.

Closes-Bug: #1246080

Change-Id: Icb1f95d1db4baa56c0f6fd68ce6342bbff27641d
(cherry picked from commit a0a462f0303f68d885ff344898437e310c64188c)
Backport of https://review.openstack.org/#/c/108272/3 from Juno tip.

It has yet to be fully accepted, but I don't forsee any major code changes
and this should address our issue with the dhcpnak replies after a restart.

Partial-rally-bug: DE494
Simplify using external routers and metadata

The dhcp agent only pushes out the metadata
static route when the subnet is isolated, and it
determines that by checking if the gateway_ip is
not set. This makes it tricky to use external
routers and metadata from dhcp at the same time.

This patch changes how the dhcp agent determines
that the subnet is isolated. It now considers it
isolated if there is no Neutron router on it.
This makes it straightforward to use an external
router on a provider network and get the
metadata from the dhcp namespace.

Change-Id: I0e29a2f058564c267176dab26da00f6ef579808b
Closes-Bug: 1236783
(cherry picked from c73b54e50b62c489f04432bdbc5bee678b18226e)
Ensure get_pid_to_kill works with rootwrap script

To ensure that correct process is killed when using a rootwrap
script, we must recursively list the children of our top-level
process and kill the last one.

This solves the same issue as 61e9268bd4b696860eeb5a6ce59ae6bf42b1f0c8
but is a different implementation due to not being able to add the
psutil dependency to an existing release.

Change-Id: Ic9d0512a3426c7b85ad88a71ee7062ba2c304322
Add configurable ovsdb monitor respawn interval

If minimize_polling=True and the ovsdb-client process died, the
default respawn interval of 0 would mean that we'd start polling
again. This patch adds the option ovsdb_monitor_respawn_interval and
defaults it to 30 seconds.

Change-Id: I12c1e05ec4d6d2bd4d84024a91116e2ac3974868
Close-Bug: 1243867
(cherry picked from commit 14950e492ad4423aa6f8719f7178b0477294ba5a)
Update send_arp_for_ha default in l3_agent.ini

The default for send_arp_for_ha was 3 and is now 0.
This patch changes the comment in l3_agent.ini to
show the new default.

Change-Id: I59a807105d71d1118fd9ae586a6c9a07995f81cd
Closes-Bug: #1292293
(cherry picked from commit 4a6e77e316cf50761b1ee07be63e490f29519711)
Reduce debug logging output

Reduce logging output from amqp/qpid to greatly increase the
signal-to-noise ratio of debug logs.  Also alphabetize the list.

Cherrypick of oslo update [1].  Introduced into neutron master branch
with other changes in [2]

[1] https://github.com/openstack/oslo-incubator/commit/fb8075a1
[2] https://github.com/openstack/neutron/commit/1506f26b

Change-Id: I46a5f37afd92d174d8844cb73c13ccf701baa312
Fix error message typo

 * Fix error message typo in "_network_admin_state"
   function where "Network Admin State Validation Falied"
   should be changed to "Network Admin State Validation Failed"

Change-Id: I767e93c300250b4422e3980f799862ceb976c951
Closes-Bug: #1276409
(cherry picked from commit 844ae433c17ee3956a8c83ed693ca44c8e3c38de)
Fixes the Hyper-V agent individual ports metrics

Replaces aggregated metric values with separated values for each port.

Co-Authored-By: Adrian Vladu <avladu@cloudbasesolutions.com>

(cherry picked from commit 4559a5f381082dba30bb535b69deeb09135da680)

Change-Id: Ie946dff984ef53f014c6c57f8d1d5bb9c6e7596d
Closes-Bug: #1289007
Include cisco plugin in migration plugins with ovs

Currently we have many migration files with missing cisco
plugin in migrate_plugin when ovs is included.This
causes missing tables when cisco plugin is enabled
and migration is run. This fix should automatically
include the cisco plugin if ovs is detected in the
migrate_plugins.

Change-Id: I4dedfbafe9b431e85255d5427766e22eed09ee5e
Closes-Bug: #1292114
(cherry-picked from f23d081b26071e1b309f49e5a5ab0cdc1a739e9d )
Add request timeout handling for Mellanox Neutron Agent

Add request timeout handling for messages sent to eswitch Daemon.
Using configurable number of retries and increasing waiting interval
between retries resend the message.
If request timeout persists, eswitch daemon is not reachable.
In such case, exit the agent process.

Closes-Bug: #1228827

Change-Id: If1290bedafe7a0dd8a61cbe328510075edeb1e5b
(cherry picked from commit 793c76344715790531d39436508b188b536e2ff5)
Call _destroy_metadata_proxy from _destroy_router_namespaces

Refactor _spawn/destroy_metadata_proxy so that it can be called
with only the namespace and the router_id.

Change-Id: Id1c33b22c7c3bd35c54a7c9ad419831bfed8746b
Closes-Bug: #1252856
(cherry picked from commit 07d597079781967f5a149f1812ddca3897fa49d9)
Related-Bug: #1175695
Optionally delete namespaces when they are no longer needed

Adds a configuration option to tell the network agents to delete
namespaces when they are no longer in use.  The option defaults to
False so that the agent will not attempt to delete namespaces in
environments where this is not safe.

This has been working well in deployments where iproute2 has been
patched with commit 58a3e8270fe72f8ed92687d3a3132c2a708582dd or it is
new enough to include it without being patched.

Change-Id: Ice5242c6f0446d16aaaa7ee353d674310297ef72
Closes-Bug: #1250596
Related-Bug: #1052535
(cherry picked from commit 7336f3bd27d138b3d11d601f977a1e3df2a44b3e)
Related-Bug: #1175695
Simplify ip allocation/recycling to relieve db pressure

I found that multiple calls to delete_port can pile up on the
_recycle_ip operation.  This patch simplifies this operation.  It
reduces the _recycle_ip operation to a single row delete in the ip
allocations table and doesn't touch the availability table.

To acheive the recycling of ips in a pool, this code runs a more
complex operation of rebuilding the availability table when it is
exhausted.  Only one API process will perform this more expensive
operation and others waiting for allocation will immediately benefit.
The amortized cost of this operation is much less than the cumulative
cost of running the more expensive _recycle_ip operation for every
port delete.

IP allocation behaves a bit differently with this patch.  Instead of
giving out the first IP available in a pool, the entire pool will be
allocated before wrapping around and recycling ip addresses that have
been released.  This is a desirable feature as it puts ip addresses in
a sort of quarantine after they are released.  It is easier to
distinguish newly allocated ips from old ones.

Change-Id: Ia55b66128de9986e075b0f87acc401d211cd91d3
Closes-Bug: #1252506
Closes-Bug: #1257815
(cherry picked from commit 89f25623a4345b5044a58cff36e7fd103c6b88e8)
Refactor to remove _recycle_ip

Since _recycle_ip is now just a pass-through to _delete_ip_allocation
it can be removed.

Change-Id: Ifba3da902de599f748038a33ef3bd98ff77c22b9
Closes-Bug: #1269501
(cherry picked from commit 212c755c34c195148c668b553ccd9f371df63420)
Stop removing ip allocations on port delete

The _delete_port() method was manually removing related
IPAllocation instances despite the existence of a perfectly
good cascade deletion relationship in the model.  This patch
puts an end to that nonsense and the potential for deadlock that
it represented.

Closes-bug: #1288379
Related-Bug: #1283522

Change-Id: Ib31550fa9000fc75768a327cb6cc1c419e06568f
(cherry picked from commit 2648aa3561d23e1215e0cc6f446253e5df56c8f6)
Set wsgi logger to use __name__

Although underneath this is the eventlet.wsgi.server server, Other
OpenStack projects, don't use that logger instead they use there own.

This will also mean the wsgi server will stop logging on just the
warning level.  We want wsgi server to at least log on info level
so it will log when a request is processed and what the return code is.

Change-Id: I96fd73b84b4550f85ad96030fea815376901813a
(cherry picked from commit 83ce89f471d887ec9e3aa48c26ffe0d88a1957b1)
tests/unit: Initialize core plugin in TestL3GwModeMixin

TestL3GwModeMixin can fail randomly because it doesn't initialize
core_plugin and can be run random core plugin depending on execution
order of tests. It also fails with core plugin uninitialized when it
is run without other tests.

This patch refactors the setup code of core plugin and apply it to the
related tests.
This patch reveled the same bug of test_metaplugin.py which is also
fixed by this patch.

Closes-bug: #1273259

Conflicts:
	neutron/tests/base.py
	neutron/tests/unit/test_extension_ext_gw_mode.py

Change-Id: I3c1d4d8b3d69262b89c7747daa8267bf2c8e7f6b
(cherry picked from commit 9e247277e2d916fa705715e018ef5f3f221f0efa)