Fix session's InvalidRequestError because of nested rollback

This patch addresses the issue by removing the extra nesting
(which is effectively redundant).

The longer story about this patch is the following:

Race conditions may cause DBDuplicateEntry exceptions that
require a transaction to be rollbacked back, and yet make
the whole operation succeed. A classic example is what has
been solved in commit fbc6b99. If the rollback is done in a
nested transaction, the above mentioned exception is raised.

To address the problem, we could use savepoints by means of
sqlalchemy's begin_nested(); Even though this approach is
preferable, it causes quite a bit of changes in the unit
tests (because of sqlite); it may also require that certain
DBMS, or certain DB backend configurations, support savepoints.

In the end, the simpler approach was chosen.

Closes-bug: #1354072

Change-Id: Ic9393319e55f71d681124bd3052e939724ebab6b
Predictable field and filter ordering

This fixes the fields and filters units tests that break with a
randomized PYTHONHASHSEED (see the bug report).

The RESOURCE_ATTRIBUTE_MAP is stored as a dict leading to an
unpredictable output order. Values in kvp strings are being stored as
sets underpinned by dicts when converted, leading to unpredictable
ordering of values when read.

Discovered with PYTHONHASHSEED = 2455351445 on these tests:
test_api_v2.APIv2TestCase.test_fields
test_api_v2.APIv2TestCase.test_fields_multiple
test_api_v2.FiltersTestCase.test_attr_info_with_convert_list_to
test_api_v2.APIv2TestCase.test_filters_with_fields
test_api_v2.APIv2TestCase.test_fields_multiple_with_empty

There are 3 parts to this fix:
1. Update the APIv2TestCase _do_field_list function to construct
field list in the same order as the controller constructs its list.
2. Ensure the APIv2TestCase _get_collection_kwargs maintains order
throughout.
3. Use new assertOrderedEqual function to sort values before assertion
in test_attr_info_with_convert_list_to

Change-Id: I547cfa80cf83b0340b459279df9283443562326b
Partial-bug: #1348818
Remove SELECT FOR UPDATE use in ML2 tunnel driver add_endpoint

SELECT FOR UPDATE expression, which is triggered with the use of the
SQLAlchemy Query object's with_lockmode('update') method, is
detrimental to performance and scalability of the database
performance code in Neutron due to the lock contention it produces.

SELECT FOR UPDATE can be entirely avoided in add_endpoint methods
with the use of single-shot SELECT and INSERT expressions and the
correction of VxlanEndpoint primary key: indeed previously it was not
possible to create multiple endpoints with the same ip, now the model
primary key constraint ensures it.

Change-Id: Id69fbc15c8f51b4b275cd742312e6ff6802d8c0f
Partial-Bug: #1330562
ML2: Make get_device_details report mac address as well

This will be used by ofagent to route packets to local VMs.

Partially-implements: blueprint ofagent-merge-bridges
Change-Id: Icaa26448a370dd75d09e95ed439df5b486a862cf
l2pop: get_agent_ports: Don't yield (None, {})

There's no point to yield None lvm with empty entries.

Closes-Bug: #1355759
Related: blueprint ofagent-l2pop
Change-Id: I42aeb48a062d35038116978d70c8dac4139a5583
Merge "Remove SELECT FOR UPDATE use in ML2 tunnel driver add_endpoint"
Use jsonutils instead of stdlib json

jsonutils provides multiple benefits in comparison to pure stdlib json
(like using simplejson on Python 2.6).

Similar patch was already merged before [1], but since it lacked hacking
rule to enforce jsonutils usage, new occurrences of stdlib json module
usage were introduced.

This patch switches all the code to using jsonutils and adds a hacking
rule to enforce the rule.

The hacking rule requires that jsonutils module does not mimic as 'json'
thru using import renames, so the code was updated not to rename the
module when doing import.

The hacking rule was shamelessly copied from the corresponding nova
review [2].

[1]: https://review.openstack.org/#/c/99760/
[2]: https://review.openstack.org/111296/

Change-Id: Ie7a5bb76445e15cde9fbf9ff3d2101a014637b37
Merge "Use jsonutils instead of stdlib json"
Merge "Predictable field and filter ordering"
Merge "Fix session's InvalidRequestError because of nested rollback"
Merge "ML2: Make get_device_details report mac address as well"
Merge "l2pop: get_agent_ports: Don't yield (None, {})"