Delete DHCP port without DHCP server on a net node

A DHCP-network was deleted from one host using neutron
dhcp-agent-network-remove and then added to another host
using neutron dhcp-agent-network-add command. While the
dhcp-agent-network-remove command was in progress, the
host crashed.  As a result, the removal of the DHCP-network
was partially done. The network was disassociated from
the agent in mysql. However, the agent never made the
release_dhcp_port RPC call to delete the port -- even
after the agent restarted. The end result is that there
are two DHCP ports for the same network. One of these
is found on the host that is no longer hosting the
dhcp-server.

This fix make the DHCP agent invoke the release_dhcp_port
RPC call on a stale network whose dnsmasq process is not
running (not active). Before this change, the RPC call is
made on a stale network only when the dnsmasq process is
running.

Closes-Bug: #1244860
Change-Id: Ie0bafdac698810b5455550c306c6a75ddf91d9bb
Restore ability to run functional tests with run_tests.sh

Change f20b1d42 which split the running of functional tests from
the running of unit tests caused running:
  ./run_tests.sh neutron.tests.functional
to fail as the OS_TEST_PATH variable will be defaulted to
neutron.tests.unit which means that the functional tests will not
be discovered in the above case.

This patch trys to detect an argument passed in the form of a
neutron.tests string and correctly sets the OS_TEST_PATH variable
based on it.

Change-Id: I8ccdc7f10d3c8478281aeaf8d02175d0eeb8d6e9
Add physical_network to binding:vif_details dictionary

During port binding binding:vif_details dictionary is
updated with physical_network to be used by mlnx nova vif driver.
binding:profile update will be removed once nova side fix is merged
to keep consistent solution.

Change-Id: If9525929a1546e8d05f7923570d30dfc2eed8898
Closes-Bug: 1304872
Record and log reason for dhcp agent resync

A dhcp resync can be triggered at a number of points, but the actual
resync is done asynchronously by a helper thread.  This means by the
time the resync happens, it's hard to establish what actually caused
it.

There have been a number of problems in production systems that cause
excessive resyncs.  One is a ipv6/dnsmasq issue (rhbz#1077487) and
another is db corruption with duplicate entries [1].  The resync
triggers a whole lot of logs itself, so it becomes very unclear how to
establish any causality.

This change keeps track of what triggered the resync with some helpful
information.  We add a schedule_resync() function to to replace a
explicit set of needs_resync which tracks why it was called.

[1] The logs will contain output like "DBDuplicateEntry
(IntegrityError) (1062, "Duplicate entry
'6d799c6a-7a09-4c1e-bb63-7d30fd052c8a-d3e3ac5b-9962-428a-a9f8-6b2' for
key 'PRIMARY'") ..." in this case

Closes-Bug: #1294445
Change-Id: I9b1c6202f5a6bbad8589a8b64b2a38c9d9edb43b
ofagent: Add a missing push_vlan action

Fix the flow for _provision_local_vlan_inbound_for_tunnel
to push-vlan explicitly.  While the old code happened to
work with older versions of OVS, it was spec-wise incorrect because
it failed to meet the prerequisite of the following set-field.
The latest version of OVS correctly rejects such a flow.

Closes-Bug: #1308927
Change-Id: I66221eec0cb4083d178d7d5651360ee1874e3d1b
Big Switch: Don't use MagicMocks unnecessarily

Replace the mocks purely being used to block external
calls with functions that just return and don't record
calls.

Closes-Bug: #1315574
Change-Id: I79533135b5cb456d941ddffa62f5365970146a7e
Fix some typos in neutron/db and IBM SDN-VE plugin

For example, in neutron/db/agents_db.py, db_plugin_base_v2 should be changed to db_base_plugin_v2.
There is no bug or blueprint related to this fix.

Change-Id: I0d348d92c9c32f172d1e505cf660aed4d064a440
Add 'secret' property for 'connection' option

Neutron cli registers 'connection' option without
'secret=True' property. This commit makes sure
'connection' option has 'secret' property when
it is registered.

Change-Id: I0b0a5aca44a8eeee1c12513f30b9559f1d008442
Closes-Bug: #1319651
Reorg table ml2_port_bindings when db migration

For DB2, there are restrictions on table access
after ALTER TABLE statements containing
REORG-recommended operations.
Table will be placed in "reorg pending" state
after some continuous alter table operations.
Need run reorg table to make the table available.
More info refer to:
http://publib.boulder.ibm.com/infocenter/db2luw/v9/index.jsp
?topic=%2Fcom.ibm.db2.udb.admin.doc%2Fdoc%2Fc0023297.htm

Change-Id: I7fdd61bb4d636d7a3260c0b23152f1f3083cdcfd
Fixes: bug #1318498
OFAgent: Avoid processing ports which are not yet ready

Port the following patch to OFAgent.
commit: 2702baed390d094b0eac07d0ae167ed236868d00
https://review.openstack.org/#/c/65838/

Closes-Bug: 1307745

Change-Id: Ie1fe0944fb541ba45ffb5b03747d5da289cede51
Log iptables rules when they fail to apply

Log the set of rules that causes iptables-restore
to fail in the Linux agent iptables manager.

If a specific rule is identified as the cause, only
that rule and a few surrounding it will be logged to
reduce the output.

Closes-Bug: #1306399
Change-Id: I8e94c1faae75760e439c5abe3d9b723548398105
Fix non-existent 'assert' calls to mocks

Fixes calls to non-existent 'assert' methods
called on mock objects.

Closes-Bug: #1320774
Change-Id: I7eab28fc9b23cbb39d5507ff3e8004a493c90bed
Allowed Addresspairs: Removing check for overlap with fixed ips

Current code does not allow assigning a fixed ip to a port when that ip
overlaps with one of the addresses in the allowed-addresspairs list.
This is an unnecessary check as the overlap does not have any negative
effect. Further, such a check actually makes it hard to use this
API. For example, if a fixed IP 10.10.1.1 exists on a port and we
want to allow addresses in 10.10.1.0/24 cidr on that port, then one
has to configure a list of 8 cidrs ([10.10.1.0/32, 10.10.1.2/31,
10.10.1.4/30, ..., 10.10.1.128/25]) on the allowed-addresspairs.
In addition to the above reasons, the current code also does not
check for the overlaps in all cases.

This patch summarily removes this overlap check.

Closes-Bug: #1321864
Change-Id: I5498c4a72b31267644da10a54a9860c1fc3bb250
remove token from notifier middleware

notifier middleware is capturing token and sending it to MQ. this
is not advisable so we should filter it out.

Closes-Bug: #1321080
Change-Id: Ia1bfa1bd24989681db1d2f385defc12e69a01f8d
Remove all mostly untranslated PO files

We now only import PO files that are at least 75 % translated,
so we can delete now all other PO files. The patch only
removes the mostly untranslated files. Once a file becomes mostly
translated, the bot will import it again.

This removes all languages besides en_US.

Change-Id: If1700477e01b8ccf515c140597f47caf2b81c803
Closes-Bug: #1317794
Cisco APIC ML2 mechanism driver, part 2

    This set of changes introduces a mechanism driver for the
    Cisco APIC. This is the second and final part of a 2 part commit.
    Please see the blueprint for more information.

    The review is submitted in two parts:
    - Part 1 (Posted earlier, required for Part 2)
        o APIC REST Client
        o APIC data model and migration script
        o APIC configurations
    - Part 2 (this commit)
        o APIC mechanism driver
        o APIC manager

    Partially implements: blueprint ml2-cisco-apic-mechanism-driver

Change-Id: I5ed3ac133146635083e2d0093057b43b64f347fe
Set onlink routes for all subnets on an external network

The addition of the on-link routes gives us some freedom to allocate a
router's IP address from any one of multiple subnets on one external
network.  Different routers can get their IPs from different subnets and
they still have direct on-link connectivity to each other.  For example,
one router with its primary IP from 10.0.0.0/24 and another from
192.168.0.0/24 can communicate directly.  It is important that each
router has on-link routes to *all* of the subnets.

Any router can host floating ips from any of the subnets regardless of
which subnet the primary IP address comes from.

This is an alternative to the "Multiple floating IP pools" section in
the administration guide.  It is a simpler alternative that avoids
having to create multiple external networks.  It is also more flexible
because routers will no longer be restricted to getting floating IPs
from the pool to which they happen to be connected.

DocImpact
Document the procedure for adding subnets to the external network.
Potentially remove the existing procedure for "Multiple floating IP
pools" from the docs.

Change-Id: I2c283f5be0cbb6b5d350cafc1b636c300b796a7b
Closes-Bug: #1312467
OFAgent: Fixing lost vlan ids on interfaces

Port the following patch to OFAgent.
commit: 60cb0911712ad11688b4d09e5c01ac39c49f5aea
https://review.openstack.org/#/c/66375/

Closes-Bug: #1309286

Change-Id: I04f99ef7928cd07959b29eb64b5744df397979b8
Extraroute extension support for nuage plugin

Nuage's VSP supports adding static-route to L3 Domain
which fits nicely with extraroute extension supported
by openstack's neutron. This set of change enables that.

Change-Id: Icd4d95b9077056c33c7509fe45966ff2a04cd923
Implements: blueprint extraroute-ext-support-for-nuage-plugin
Use list copy for events in nova notifier

There is no need to call range() and len() and .pop() for such a
simple task as list copy.

Change-Id: I4d25e0887ba0a7a5c7f49d195e90a7d2818caac2