Reduce security group db calls to neutron server

Within ovs agent daemon loop, prepare_devices_filter will impose heavy workloads
to neutron server in order to retrieve the security groups message to apply
firewall rules. If agent is configured to use Noopfirewall driver or security
groups are disabled, there is no need for loading the rules from server and
refreshing the firewalls. This will reduce the number of db calls and improve
performance for neutron server in this case.

Change-Id: Id244aab3cac37fc6ed3dc05cbee91cdf9e34d9cc
Closes-Bug: #1365806
(cherry picked from commit 524981cce05a9b365036c0a1e9810036936d3d5b)
Improve performance of security group DB query

The _select_ips_for_remote_group method was joining the
IP allocation, port, allowed address pair, and security group tables
together in a single query. Additionally, it was loading all of
the port columns and using none of them. This resulted in a
very expensive query with no benefit.

This patch eliminates the unnecessary use of the port table by joining
the IP allocation table directly to the security groups and allowed
address pairs tables. In local testing of the method, this sped it up
by an order of magnitude.

Closes-Bug: #1373851
Change-Id: I12899413004838d2d22b691f1e2f3b18f7ec2c27
(cherry picked from commit 04df85b6e5a098f8f55bb82f04d9769763beb487)
Optimize query in _select_dhcp_ips_for_network_ids

Only query the DB for relevant columns instead of
all of the port columns.

Partial-Bug: #1373851
Change-Id: I32cd4a0bc6799ce77cea13188676308e3e641d19
(cherry picked from commit 8d430a7f2e903dda06d8d75d6abcd63423c4c0a1)
Only fetch port_id from SG binding table

Change a query to only retrieve the port_id instead of
every column from the row of security group binding info.

Partial-Bug: #1373851
Change-Id: I0fba9c9623898ee52590207ebbb728503bb59a5b
(cherry picked from commit 6acadab5eb8b7b627e097a638d8486bef59a7f30)
_update_router_db: don't hold open transactions

This patch prevents the L3 _update_router_db method from
starting a transaction before calling the gateway interface
removal functions. With these port changes now occuring
outside of the L3 DB transaction, a failure to update the
router DB information will not rollback the port deletion
operation.

The 'VPN in use' check had to be moved inside of the DB deletion
transaction now that there isn't an enclosing transaction to undo
the delete when an 'in use' error is raised.

===Details===

The router update db method starts a transaction and calls
the gateway update method with the transaction held open.
This becomes a problem when the update results in an
interface removal which uses a port table lock.

Because the delete_port caller is still holding open a
transaction, other sessions are blocked from getting an
SQL lock on the same tables when delete_port starts
performing RPC notifications, external controller calls,
etc. During those external calls, eventlet will
yield and another thread may try to get a lock on the
port table, causing the infamous mysql/eventlet deadlock.

This separation of L2/L3 transactions is similiar to change
I3ae7bb269df9b9dcef94f48f13f1bde1e4106a80 in nature. Even
though there is a loss in the atomic behavior of the interface
removal operation, it was arguably incorrect to begin with.
The restoration of port DB records during a rollback after some
other failure doesn't undo the backend operations (e.g. REST calls)
that happened during the original deletion. So, having a delete
rollback without corresponding 'create_port' calls to the backend
causes a loss in consistency.

Conflicts:

	neutron/db/l3_db.py

Closes-Bug: #1377241
Change-Id: I5fdb6b24bf2fb80ac5e36a742aa7056db72c8c7d
(cherry picked from commit f23f2ecee68ba4abd12139bbb91b77ba9410f581)
Batch ports from security groups RPC handler

The security groups RPC handler calls get_port_from_device
individually for each device in a list it receives. Each
one of these results in a separate SQL query for the security
groups and port details. This becomes very inefficient as the
number of devices on a single node increases.

This patch adds logic to the RPC handler to see if the core
plugin has a method to lookup all of the device IDs at once.
If so, it uses that method, otherwise it continues as normal.

The ML2 plugin is modified to include the batch function, which
uses one SQL query regardless of the number of devices.

Closes-Bug: #1374556
Change-Id: I15d19c22e8c44577db190309b6636a3251a9c66a
(cherry picked from commit abc16ebfcf8fd1fbdb4ef68590140d4d355b0a7c)
Set vif_details to reflect enable_security_group

While plugging vif, VIFDriver in Nova follows "ovs_hybrid_plug" and
"port_filter" in "binding:vif_detail" which is passed from Neutron, but
those are always true.  This patch make ML2 OVS mech driver set those
param depends on enable_security_group flag.  It enables users to avoid
ovs_hybrid plugging.

This patch also fixes the same issue in the following plugins/drivers:
  * NEC Plugin
  * BigSwitch Plugin
  * Ryu Plugin
  * ML2 Plugin - OFAgent Mech Driver

Closes-Bug: #1336624
Change-Id: I2b7fb526a6f1b730ad65289307b24fd28b996e1b
(cherry picked from commit e73f8da072cb41559ecee7f29f864a10db475444)
Allow to add router interface to IPv6 SLAAC network

This patch will allow an IPv6 subnet configured with SLAAC
(or dhcpv6-stateless) to be attached to a router interface.

Closes-Bug: #1382076
Change-Id: If0c48a7287a828eef4a0f0b0859d4f898d2937bd
(cherry picked from commit 95accb535017f1384b38b12eb78f0e58287d8e08)
Merge "Set vif_details to reflect enable_security_group" into stable/juno
Fix L3 HA network creation to allow user to create router

Update HA Network creation to use an admin context to allow Neutron
to create the tenant-less network required for the HA router when
it does not yet exist and is being created by a non-admin user.

Neutron creates these resources without a tenant so users cannot see
or modify the HA network, ports, etc.  Port creation and association
already use elivated admin contexts to allow their function when
an user attempts to create a HA L3 router.

Conflicts:
        neutron/tests/unit/db/test_l3_ha_db.py

Change-Id: I36166158a0970b8d08d6702054b11a43fb684281
Closes-Bug: #1388716
(cherry picked from commit cc9bc24229f1d79dc99303db1affc03c030c011e)
Remove openvswitch core plugin entry point

The openvswitch core plugin has been removed but not its associated
"neutron.core_plugins" entry point. This change removes it from
setup.cfg.

Change-Id: I79f7c334cdeb0f4d0d68743734f69c0ec8523467
Related-Bug: #1323729
Closes-Bug: #1391326
(cherry picked from commit f36c29903afb713670f9e7c3f9530e117dfc6bcb)
rootwrap config files reference deleted quantum binaries

dhcp and l3 rootwrap filter files contain reference to the deleted
binary quantum-ns-metadata-proxy. This change removes these obsolete
filters.

Change-Id: Iad4772752d74a1c0535144b5faf9a1f8ae89f6a3
Closes-Bug: #1391256
(cherry picked from commit 226c4f6d809de3d6ed622a1f90b06e34506696d6)
Fix KeyError in dhcp_rpc when plugin.port_update raise exception

KeyError exception is seen because of following reasons

* DhcpRpcCallback._port_action() is called by two functions
   -  DhcpRpcCallback.create_dchp_port()
   -  DhcpRpcCallback.update_dhcp_port()

* When create_dhcp_port() function calls _port_action(), the
  function argument 'port' will have the body as
  {'port': {'network_id': foo_network_id, 'fixed_ips': [..] ...}

* When update_dhcp_port() function calls _port_action(), the
  function argument 'port' will have the body as
  {'id': port_id, 'port': {{'port': {'network_id': foo_network_id,
			            'fixed_ips': [..] ...}}

* If an exception occurs when _port_action() calls plugin.create_port(),
  network id is accessed as
  net_id = port['port']['network_id']

* If an exception occurs when _port_action() calls plugin.update_port(),
  network id is accessed as
  net_id = port['port']['network_id']
  which is causing the KeyError. network_id should have been accessed as
  net_id = port['port']['port']['network_id']

This patch fixes the issue by making the _port_action() take the
same port body. update_dhcp_port() insteading of passing the port_id
and port information in a single argument, it now adds port_id
in the port body itself.

Change-Id: I70b92fa20b421b05ca2053a9a57f62db726f7625
Closes-bug: #1378508
(cherry picked from commit 7ea605df3ac71dc568194bcd5eaf1c115008e1ee)
fix event_send for re-assign floating ip

Neutron can associate a floating ip to a new port
without disassociate from original instance port.
This situation will send network changed event only
for new instance port, and that event object contains
the new instance's id.

In this case nova will update new instance's info
but not original one's in nova's database table
instance_info_caches. For nova can get new instance's
id from the above event. So in table instance_info_caches,
both original instance and new instance will have the
same floating ip in their records. And this make it
possible that, in most situation, after your re-assign
floating ip, run "nova list" will return incorrect info,
multiple instances have a same floating ip, and this may
confuse users.

Nova will sync data in table instance_info_caches, but it
may take dozens of seconds.
The new added code will send network changed event for the
original instance, and this will make nova update instance_
_info_caches table in a few seconds.

Change-Id: If3ee11535f649fc51bf1a52806008c1c5c0e73b6
Closes-Bug: 1381886
(cherry picked from commit add8944d4dba2a69e8f7da47f120d5dc63952f77)
Merge "Fix L3 HA network creation to allow user to create router" into stable/juno
Merge "rootwrap config files reference deleted quantum binaries" into stable/juno
Merge "Fix KeyError in dhcp_rpc when plugin.port_update raise exception" into stable/juno
Merge "fix event_send for re-assign floating ip" into stable/juno
Merge "_update_router_db: don't hold open transactions" into stable/juno
Merge "Improve performance of security group DB query" into stable/juno