Remove ignored do_request timeout argument

The timeout to the underlying request was always self._request_timeout,
regardless of the value of the do_request(timeout) arg provided.  Both
callers always pass timeout=self._request_timeout anyway, so just remove
this unused/misleading argument.

Change-Id: I911ee986caef8fcf9b97ef622a03856d4351bee0
Predictable field and filter ordering

This fixes the fields and filters units tests that break with a
randomized PYTHONHASHSEED (see the bug report).

The RESOURCE_ATTRIBUTE_MAP is stored as a dict leading to an
unpredictable output order. Values in kvp strings are being stored as
sets underpinned by dicts when converted, leading to unpredictable
ordering of values when read.

Discovered with PYTHONHASHSEED = 2455351445 on these tests:
test_api_v2.APIv2TestCase.test_fields
test_api_v2.APIv2TestCase.test_fields_multiple
test_api_v2.FiltersTestCase.test_attr_info_with_convert_list_to
test_api_v2.APIv2TestCase.test_filters_with_fields
test_api_v2.APIv2TestCase.test_fields_multiple_with_empty

There are 3 parts to this fix:
1. Update the APIv2TestCase _do_field_list function to construct
field list in the same order as the controller constructs its list.
2. Ensure the APIv2TestCase _get_collection_kwargs maintains order
throughout.
3. Use new assertOrderedEqual function to sort values before assertion
in test_attr_info_with_convert_list_to

Change-Id: I547cfa80cf83b0340b459279df9283443562326b
Partial-bug: #1348818
Fix 404 error fetching metadata when using DVR

The metadata agent was unable to find networks
attached to the DVR router because it was only
filtering ports for 'centralized' routers.

To fix the issue, this patch expands the search
filters to include DVR router interfaces during
the network lookup operation.

The extra filter cause no evident performance
loss while serving the request; a different
approach would require to pass the router type
around to narrow down the search filter, but it
sounds like an overkill.

Closes-bug: #1353271

Change-Id: Iefbefa1ff300adad48ab9fc472d5eb1913fbe488
Imported Translations from Transifex

Change-Id: I71dc194a5ced63b8549f8bffb82d22774d3da74b
VPNaaS: Cisco fix validation for GW IP

The validation to check that the router has a GW IP
for the Cisco service driver was trying to directly
get the gw_port information, which is not available, as
the validator only has a dict of the vpn_service info.

Modified validator to get the router object (not a
dict representation), using the vpn_service's router_id
and then check the attributes for the router directly.

Change-Id: I48286cda23d9737ddd90251dc0d1db1c310e7784
Closes-Bug: 1356127
Fix PortNotFound exception during sync_routers

This trace is observed when an L3 agent invokes
sync_routers right about the same time a port
interface is removed from a router.

Related-bug: #1355409

Change-Id: I825b25080cbf054462318fc01248692b9e0e4ecb
Fix KeyError during sync_routers

Method sync_routers is used by the L3 agent to query
routers it knows about. Routers and GW ports lists
are populated in two different times, which means that
they can be interleaved by a delete request which
results in gateway ports being missing in one of the
two data structures.

This patch takes care of the race condition.

Closes-bug: #1355409

Change-Id: Id3a6fe145058f690e107bfe7023980ede61cff90
Remove SELECT FOR UPDATE use in ML2 tunnel driver add_endpoint

SELECT FOR UPDATE expression, which is triggered with the use of the
SQLAlchemy Query object's with_lockmode('update') method, is
detrimental to performance and scalability of the database
performance code in Neutron due to the lock contention it produces.

SELECT FOR UPDATE can be entirely avoided in add_endpoint methods
with the use of single-shot SELECT and INSERT expressions and the
correction of VxlanEndpoint primary key: indeed previously it was not
possible to create multiple endpoints with the same ip, now the model
primary key constraint ensures it.

Change-Id: Id69fbc15c8f51b4b275cd742312e6ff6802d8c0f
Partial-Bug: #1330562
Delete DVR namespaces on node after removing last VM

After removing the last VM using a distributed router,
the router's namespaces are still present on the VM host
The problem is that the neutron API server sent the router
remove notification to the L3 agent using the name of the
host running the L3 agent instead of the agent's uuid. This
caused an error when sending the notification. So the L3
agent never had the chance to cleanup the namespace.
This problem is fixed here.

Afterwards, it was found that the notification was still not
sent. The reason is that the router/L3-agent binding has
already been deleted before the routine that sends the
router removed notification was called. The notifier routine
errored out when it tried to delete the same router/L3 agent
binding. This problem is fixed in this patch by removing the
binding removal step from the DVR scheduler.

Change-Id: I6323d7ff438bb6c31e4a794bd3da96bf132fdc85
Closes-Bug: 1353165
Merge "Fix 404 error fetching metadata when using DVR"
Merge "Imported Translations from Transifex"
Fix interface add for dvr with gateway

when an interface is added after router gateway set, external
connectivity using snat fails. Instead of just adding the snat port for
the new subnet, method internal_network_added(..) incorrectly re-adds
all the snat ports with wrong cidr.

Change-Id: I7bfe266288670fba0c90990bf350f43ef7829bad
Closes-bug: #1355087
Merge "Fix PortNotFound exception during sync_routers"
Merge "Fix KeyError during sync_routers"
Merge "Remove ignored do_request timeout argument"
Merge "VPNaaS: Cisco fix validation for GW IP"
Merge "Delete DVR namespaces on node after removing last VM"
Merge "Fix interface add for dvr with gateway"
Merge "Remove SELECT FOR UPDATE use in ML2 tunnel driver add_endpoint"
Use jsonutils instead of stdlib json

jsonutils provides multiple benefits in comparison to pure stdlib json
(like using simplejson on Python 2.6).

Similar patch was already merged before [1], but since it lacked hacking
rule to enforce jsonutils usage, new occurrences of stdlib json module
usage were introduced.

This patch switches all the code to using jsonutils and adds a hacking
rule to enforce the rule.

The hacking rule requires that jsonutils module does not mimic as 'json'
thru using import renames, so the code was updated not to rename the
module when doing import.

The hacking rule was shamelessly copied from the corresponding nova
review [2].

[1]: https://review.openstack.org/#/c/99760/
[2]: https://review.openstack.org/111296/

Change-Id: Ie7a5bb76445e15cde9fbf9ff3d2101a014637b37