LBaaS: fix handling pending create/update members and health monitors

When agent requests loadbalancer logical config from server,
server returns only active pool members and health_monitors.
Need to make server return also members and monitors which are in pending states.

Also a small refactoring moving ACTIVE_PENDING set to common place

Change-Id: I8e10004f199f982b055da18ea7a0e5e4d11fa7fb
Closes-Bug: #1259965
[ML2] l2-pop MD handle multi create/delete ports

If more than one port is added or removed simultaneously, port db entry
have status BUILD or DOWN and pass to ACTIVE when agent have finish to
configured it.
l2-pop mechanism driver use events port pass to ACTIVE or DOWN to send
fdb entries. In case of port is the first or the last network port on
an agent, the flooding entry need to be add or removed.

This patch fix the method to determine how many ports are active on a
agent by adding filter on status port to be ACTIVE.

Closes-bug: #1263881
Change-Id: I9c1f8bd69dee37bc01a5d42327aa5f737998c5aa
LBaaS: handle NotFound exceptions in update_status callback

LBaaS agent may send update_status requests to server on objects
which were already deleted from db: this is due to creating and
deleting objects with a high rate (like tempest API tests do).
As a result errors and stacktraces appear in server and agent logs.
The proposed solution is to catch NotFound exceptions and print a warning.

Change-Id: I4446b678893d0bda578ad7ccdf3f109cb1c91b4d
Closes-Bug: #1260682
Use information from the dnsmasq hosts file to call dhcp_release

Certain situations can cause the DHCP agent's local cache to get out
of sync with the leases held internally by dnsmasq.  This method of
detecting when to call dhcp_release is idempotent and not dependent on
the cache.  It is more robust.

Change-Id: I4eafd9cfb94a77a2f0229f89de5483dad23725cf
Closes-Bug: #1250644
Fix race condition in delete_port method. Fix update_port method

Port can be gone between
  l3plugin.prevent_l3_port_deletion(context, id)
and port query. Need to handle it properly.
ALso need to handle non-existing port in update_port properly.

Closes-Bug: #1266537

Change-Id: Ic203f21db277f83c604a8757a8c421b3fb9ae709
NVP: Add LOG.exception to see why router was not created

Add missing LOG.exception so that it exposes what error occurred.

Change-Id: I9194795678a6be59eb1d2555dfa99ca7a035c418
Closes-bug: #1269152
Fix the migration adding a UC to agents table

The migration script mistakenly assumes that all core
plugins use agents extension, which is not true (e.g.
plumgrid and bigswitch don't).

Apply this migration script only for plugins that are
stated in the original migration script adding agents
table (511471cc46b_agent_ext_model_supp.py).

Related-Bug: #1254246

Change-Id: I7915ef8d183782eb5d46ac47f45014aa9e9640fb
Remove release_lease from the DHCP driver interface

Neither the midonet nor the linux dhcp driver needs a call to
release_lease.  Removing this from the driver API simplifies more code
in the DHCP agent.

Change-Id: Ib144b2bf3720b1b999205ace1a7d1ffe5ef3b167
Closes-Bug: #1269505
Expunge session contents between plugin requests

In the NVP plugin, metadata processing performs several plugin operations
with the same context (and db session). The first operation might leave
persisted objects in the session instance which then conflict with objects
created in the second operation.

Closes-Bug: #1266482

Change-Id: Ic9e2b952013610647aa222f1a8b6e33504b54b37
Report proper error message in PLUMgrid Plugin

Change-Id: Ifc1bb55f6b025bba77cf9858ed392dbf170075a7
Closes-Bug: #1268460
Signed-off-by: Fawad Khaliq <fawad@plumgrid.com>
Use an independent iptables lock per namespace

Since iptables is independent from namespace to namespace, it makes
sense to use an independent lock per namespace.  This improvement is
aimed at improving the parallel performance in the L3 agent.

Partially implements blueprint: neutron-tempest-parallel

Change-Id: I15e9c9da9a7c15981757a09bc744501722d62db2
Minimize the cost of checking for api worker exit

A recent change to oslo allows the configuration of the interval
that ProcessLauncher waits between checks of child exit.  The
default interval of 0.01s resulted in the neutron service consuming
unnecessary cpu cycles checking whether api workers had exited (5%
cpu on idle in a VM).  This patch extends the interval to 1s to
minimize the cost of the checks.

Change-Id: I0407ccb2db65cd3839586faff15e70dbc35f005e
Closes-bug: #1095346
Rename nvp_cluster for VMware NSX plugin

Rename the module into nsx_cluster and the class into NSXCluster,
and update references accordingly.

Also, rename utility method create_nvp_cluster to create_nsx_cluster
and move it into nsx_utils.

Partial-implements blueprint: nicira-plugin-renaming

Change-Id: I1a943cc64730ea67f464c3804370dad46411f8a2
Remove garbage in vim header

Introduced by commit 9d01010e8826eafc48044fc4018944a34bc96ba4

Change-Id: I97c7789fa13e61d16466cddac49fc41d1f917e80
Reduce severity of log messages in validation methods

I noticed this while reviewing Ic2c87174.  When I read through log
files, I don't want to see errors like this that come from validating
bad user input.  Info severity is more appropriate.

Change-Id: Ib8a4dd08570923c6cade6447b52bb73d20558258
Closes-Bug: #1272565
Simplify ip allocation/recycling to relieve db pressure

I found that multiple calls to delete_port can pile up on the
_recycle_ip operation.  This patch simplifies this operation.  It
reduces the _recycle_ip operation to a single row delete in the ip
allocations table and doesn't touch the availability table.

To acheive the recycling of ips in a pool, this code runs a more
complex operation of rebuilding the availability table when it is
exhausted.  Only one API process will perform this more expensive
operation and others waiting for allocation will immediately benefit.
The amortized cost of this operation is much less than the cumulative
cost of running the more expensive _recycle_ip operation for every
port delete.

IP allocation behaves a bit differently with this patch.  Instead of
giving out the first IP available in a pool, the entire pool will be
allocated before wrapping around and recycling ip addresses that have
been released.  This is a desirable feature as it puts ip addresses in
a sort of quarantine after they are released.  It is easier to
distinguish newly allocated ips from old ones.

Change-Id: Ia55b66128de9986e075b0f87acc401d211cd91d3
Closes-Bug: #1252506
Closes-Bug: #1257815
Merge "Fix the migration adding a UC to agents table"
Tweak version nvp/nsx version validation logic for router operations

This patch improves how the nsx/nvp controller version is validated
prior to some router operations. This is done by greatly simplifying
the boolean condition.

Missing unit tests are also added for increased coverage.

Closes-bug: 1274361

Change-Id: I1b21444fd86b6e6a0c72bb5a43f1707934dd70e4
Change metadata-agent to spawn multiple workers

There is currently only one metadata-agent per network node,
which could be handling connections from hundreds or thousands
of metadata-namespace-proxy processes.

This change addes a new "metadata_workers = XX" to the ini file
to support creating more workers to help improve performance.

Change-Id: Ib9ebcfc543a83982dd93db79c7dc631283fd3bfa
Partial-bug: #1274536
Sync with commit-id: 9d529dd324d234d7aeaa3e6b4d3ab961f177e2ed

(Pulling from the gate, neutron fail rates are currently too high
to have neutron jobs in the queue.)

Related-Bug: #1270212
Change-Id: I94965b395cfb34ed1576b84f21aa48edf991d629