BSN: Optimistic locking strategy for consistency

Summary:
  Adds an optimistic locking strategy for the Big Switch
  server manager so multiple Neutron servers wanting to
  communicate with the backend do not receive the consistency
  hash for use simultaneously.

  The bsn-rest-call semaphore is removed because serialization
  is now provided by the new locking scheme.

  A new DB engine is added because the consistency hashes
  need a life-cycle with rollbacks and other DB operations
  than cannot impact or be impacted by database operations
  happening on the regular Neutron objects.

  Unit tests are included for each of the new branches
  introduced.

Problem Statement:
  Requests to the Big Switch controllers must contain the
  consistency hash value received from the previous update.
  Otherwise, an inconsistency error will be triggered which
  will force a synchronization. Essentially, a new backend
  call must be prevented from reading from the consistency
  hash table in the DB until the previous call has updated
  the table with the hash from the server response.

  This can be addressed by a semaphore around the rest_call
  function for the single server use case and by a table lock
  on the consistency table for multiple Neutron servers.
  However, both solutions are inadequate because a single
  Neutron server does not scale and a table lock is not
  supported by common SQL HA deployments (e.g. Galera).

  This issue was previously addressed by deploying servers
  in an active-standby configuration. However, that only
  prevented the problem for HTTP API calls. All Neutron
  servers would respond to RPC messages, some of which would
  result in a port update and possible backend call which
  would trigger a conflict if it happened at the same time
  as a backend call from another server. These unnecessary
  syncs are unsustainable as the topology increases beyond
  ~3k VMs.

  Any solution needs to be back-portable to Icehouse so new
  database tables, new requirements, etc. are all out of the
  question.

Solution:
  This patch stores the lock for the consistency hash as a part
  of the DB record. The guaruntees the database offers around
  atomic insertion and constrained atomic updates offer the
  primitives necessary to ensure that only one process/thread
  can lock the record at once.

  The read_for_update method is modified to not return the hash
  in the database until an identifier is inserted into the
  current record or added as a new record. By using an UPDATE
  query with a WHERE clause restricting to the current state,
  only one of many concurrent callers to the DB will successfully
  update the rows. If a caller sees that it didn't update any
  rows, it will start the process over of trying to get the
  lock.

  If a caller observes that the same ID has the lock for
  more than 60 seconds, it will assume the holder has
  died and will attempt to take the lock. This is also done
  in a concurrency-safe UPDATE call since there may be many
  other callers may attempt to do the same thing. If it
  fails and the lock was taken by someone else, the process
  will start over.

  Some pseudo-code resembling the logic:
    read_current_lock
    if no_record:
      insert_lock
      sleep_and_retry if constraint_violation else return
    if current_is_locked and not timer_exceeded:
      sleep_and_retry
    if update_record_with_lock:
      return
    else:
      sleep_and_retry

Conflicts:
	neutron/tests/unit/bigswitch/test_servermanager.py

Closes-Bug: #1374261
Change-Id: Ifa5a7c9749952bc2785a9bf3fed69ad55bf21acc
(cherry picked from commit cdaa502f899d4c90c5e40ccd745f1d92bfd1127b)
Updated from global requirements

Change-Id: I4959b7d76e9a6e33e1ac6eab88675b8218e5a353
Alter execute_alembic_command() to not assume all commands

Alembic 0.7.1 is adding new diff directives add_fk and remove_fk.
The logic in execute_alembic_command() appears to assume all potential
commands are present in METHODS.   This patch adjusts it so that
it checks METHODS first, allowing the tests in neutron.tests.unit.db.test_migration
to pass.  However, if the design here requires that all potential
Alembic commands are handled, then this would not be the fix,
though I recommend that the potential list of Alembic directives
not be hardcoded.

Change-Id: Ib08685be252e1fdb0f10853f8fc9a7e19ac54a94
Closes-Bug: #1397796
(cherry picked from commit 6a9d1121472028f3583e577b62a7e576fff5806f)
Merge "Updated from global requirements" into stable/juno
Fix hostname validation for nameservers

Fixes the hostname validation to align with the RFC's demands[1].
This was done by replacing the full regex with a function that
broke the FQDN into individual components that were easier to
reason about with regular expressions.

Also added several test cases for domains so if someone wants
to convert it back to pure regex there will be better test vectors.

1. RFC 1123 says an all-digit hostname is allowed in section 2.1. It
   says that this more liberal syntax MUST be supported.

Conflicts:
	neutron/tests/unit/test_attributes.py

Closes-Bug: #1396932
Change-Id: I003cf14d95070707e43e40d55da62e11a28dfa4e
(cherry picked from commit 35662d07628452d14306f5197871ad64f6396ff3)
Updated from global requirements

Change-Id: Ieded1c4bdc6fd977fcb3131707028616ee184bd9
Bump stable/juno next version to 2014.2.2

Change-Id: If016d6f492c2b23d6ce3e77731cd1d9de94a9b18
Teach DHCP Agent about DVR router interfaces

When DVR is enabled and enable_isolated_metadata=True,
the DHCP agent should only inject a metadata host route
when there is no port with the gateway IP address configured
on the subnet.  Add a check for DEVICE_OWNER_DVR_INTERFACE
when we look at each port's device_owner field, otherwise
it will always add this route to the opts file when DVR
is enabled.

Change-Id: I3ff3bb85105b8215b36535983016d8c0ff3d8cb7
Closes-bug: #1377307
(cherry picked from commit 29250949012e9c0a60b0ddb56ddbf18d7b68106b)
Fix enable_metadata_network flag

The following patch: 9569b2fe broke the desired functionality of
the enable_metadata_network flag, by not allowing the metadata
proxy to be spawn for 'metadata networks', which are used for
accessing the metadata service when the logical router is not
implemented through the l3 agent.

This patch enables spawning of the metadata proxy for metadata
networks when the appropriate flag is set to True.

The patch also adds rather pedant unit test coverage for the
should_enable_metadata method which previously had no unit test.

Change-Id: I8dca1fce9fbc83e75ba7e4ce948531427bf7e88b
Closes-bug: 1394020
(cherry picked from commit c45842af38da322b93d1200451a4a254abfcaed1)
Merge "BSN: Optimistic locking strategy for consistency" into stable/juno
Updated from global requirements

Change-Id: I22831dbe79fe30e6932d4df068c7309c025dc8c5
Auto allocate gateway_ip even for SLAAC subnets

For a SLAAC subnet that is created without specifying the gateway_ip,
Neutron currently allocates (If0c48a7287a828eef4a0f0b0859d4f898d2937bd)
the gateway_ip at a later stage (i.e., neutron router_interface_add).
In order to keep the API consistent between IPv4 and IPv6, it is
recommended to allocate the gateway_ip during subnet_create stage itself.

Closes-Bug: #1402407
Closes-Bug: #1394112
Partial-Bug: #1377985
Change-Id: I965232930502c21b605fe360bb138bb6ea73d2b0
(cherry picked from commit 66dcb8b935a00e7f7566802d662ebb1f265eab1f)
Updated from global requirements

Change-Id: I0271e5b25934ade786852125b33f13a2b548c3f2
Merge "Auto allocate gateway_ip even for SLAAC subnets" into stable/juno
Fix a race condition adding a security group rule

setup_port_filters() needs to grab self.devices_to_refilter
before it calls prepare_devices_filter(), else it could skip
processing a device if an RPC arrives while it's processing
new devices.  That device will now be handled the next time
it's called.

Bug introduced in commit 3046c4ae22b1

(Cherry-picked from ee0e41d09d6ffc3cb7bf8406c8e8b009dfb92e17)
Change-Id: Ib2f460cc095bbea8f9c767dcb9b4d4b66f1a7811
Closes-Bug: 1393925
Fix race condition on processing DVR floating IPs

Fip namespace and agent gateway port can be shared by multiple dvr routers.
This change uses a set as the control variable for these shared resources
and ensures that Test and Set operation on the control variable are
performed atomically so that race conditions do not occur among
multiple threads processing floating IPs.
Limitation: The scope of this change is limited to addressing the race
condition described in the bug report. It may not address other issues
such as pre-existing issue with handling of DVR floatingips on agent
restart.

closes-bug: #1381238

cherry-picked from 9902400039018d77aa3034147cfb24ca4b2353f6
Change-Id: I6dc2b7bad6e8ddbaa86c1f7a1e2028aeacc3afef
Delete FIP namespace when last VM is deleted

On a compute node when the last VM with a floating IP association
is deleted, the L3 agent did not delete the fip namespace. However
the api server has already deleted the fip agent external gateway
port from the database.

This problem is happening on DVRs because the deletion of a VM port,
in addition to a floating IP disassociation, may also result in the
removal of the external gateway port binding AND the removal of the
fip agent external gateway port.

When the L3 agent is handling a routers_updated notification, it is
not processing floating ip address updates when the router has both
a floating ip disassociated and a external gateway port deleted.
This patch corrects this problem.

cherry-picked from e3b949c3bc08808e3df15215bc30d6610f3a4bd3
Closes-bug: #1377156
Change-Id: I86bdef7c9d988cb9d87c88adde55548d459f29a5
Moves the HA resource creations outside of transaction

Currently the HA resources are created in the
_create_router_db which includes calls to
the plugin and generates RPC calls. Even if the
resource creations are outside of any transaction
from the _create_router_db point of view, this
method is  called in a transaction from the
create_router method.
This patch moves the resource creations to the
create_router method outside the transaction.
The failures are handled as previously with
a try/expect.

Conflicts:
	neutron/tests/unit/db/test_l3_ha_db.py

Change-Id: If8fcfd012f8e992175e49bbefb2ae667881a620a
Closes-bug: #1374461
(cherry picked from commit ccd650732729451aa8e5ce3401f9570c70c4f066)
return the dict of port when no sec-group involved

Commit abc16ebf made the get_sg_ids_grouped_by_port function not return
entries for ports without security groups. This causes the agent to not remove
previously created security groups for that port since the port is not
returned in the security_group_info_for_devices data.

This change fixes that by always including a list of security groups for each
port, even if that list is empty.

Change-Id: I9616708462a8b6f3d46ebd76db5cf8cb2826f4ad
Closes-Bug: #1398312
(cherry picked from commit 997e70751c74004983b1e4079b084431ed0c7d27)
Merge "Moves the HA resource creations outside of transaction" into stable/juno