Synchronized with oslo db and db.sqlalchemy

In most cases Oslo-incubator contains the most relevant code.
Furthermore in certain cases exists bugs which can be fixed by
synchronization with Oslo. Thus regular sync is required.

                        Changes from oslo-incubator
==============================================================================
8b2b0b7 Use hacking import_exceptions for gettextutils._
9bc593e Add docstring for exception handlers of session
855644a Removal of _REPOSITORY global variable.
ea6caf9 Remove string.lowercase usage
a33989e Remove eventlet tpool from common db.api
e40903b Database hook enabling traditional mode at MySQL
f2115a0 Replace xrange in for loop with range
c802fa6 SQLAlchemy error patterns improved
1c1f199 Remove unused import
6d0a6c3 Correct invalid docstrings
fb0e86a Use six.moves.configparser instead of ConfigParser
6603e8f Remove sqlalchemy-migrate 0.7.3 patching
97d8cf4 Remove lazy loading of database backend
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Synced from Oslo-Incubator commit: 517c4cc65969f93b4b758a568725a1625abfd24d

Change-Id: Ida6b38b8e41c73c67170e4ec43943e661f014629
Remove unused test function

The appconfig function in keystone.tests.TestCase wasn't used.
This change removes it.

Change-Id: I5377b7a3a42df1d111736f9cce0e2f74fa5931e2
Remove unnecessary line in test_auth

TestCase sets up the kvs identity backend already, by using
test_overrides.conf, so this line doesn't do anything and is going
to lead to confusion if test_overrides.conf changes to a different
backend.

Change-Id: I3f62e70a0dd7a27b675a29a56bfbccffa1b9be50
Flush tokens in batches with DB2

When there is a very large number of expired tokens to be flushed,
keystone-manage token_flush will fail with DB2 because it fills
the transaction log. This fix causes tokens to be flushed with DB2 in
batches of 100 at a time to prevent the transaction log from
filling up.

Closes-Bug: #1257323
Change-Id: Ic57d8795a995462fc277ec9655eca9b460ffcbee
races cause 404 when removing user from project

When removing a user from a project, the assignment manager retrieves a
list of all roles the user has on the project, then removes each role.
Each (user, role, project) tuple is removed with a separate call into
the driver. If, before a particular role has been removed, that role is
deleted by another request calling into the manager (i.e., via
delete_role), the call into the driver by the user removal request will
raise a RoleNotFound exception and the request will return an HTTP 404
error. Furthermore, any roles in the list after the exceptional role
will not be deleted.
This patch add try-except around the
driver.remove_role_from_user_and_project calls to catch the RoleNotFound
exceptions, so that roles after the exceptional role will be deleted as
expected, and a 404 is not returned.

Closes-Bug: #1257390

Change-Id: Ic48d869711d0275d402616c550fc0e7444b0a0d0
Don't need session.flush in context managed by session

Within the scope of a single method, keeping all the reads and writes within
the context managed by a single session. In this way, the session's __exit__
handler will take care of calling flush() and commit() for you.
If using this approach, you should not explicitly call flush() or commit().
See http://docs.sqlalchemy.org/en/rel_0_7/orm/session.html#committing

Closes-Bug: #1258044
Change-Id: I1125c72712203f1594ef245761dbfc25ed31eab2
Sync oslo-incubator rpc	module

Sync oslo-incubator rpc module with oslo-incubator commit
f8f6fd46b3b9264c21c0ee5547a4e8236cb257ac

Updates consist in using six.iteritem and fixing some typos.

Remove securemessage.py, which in not used anywhere, and dropped from
oslo-incubator since commit f88d59ab8a456e7d2d5a22fc5591491d2f85bdf4

$ python update.py --nodeps --base keystone --dest-dir ../keystone/ --modules rpc
$ git rm keystone/openstack/common/rpc/securemessage.py

Change-Id: I890cf1ce89ce5946522f7891be71a3e58ecc1af3
Change ListOpt default value from str or None to list

In keystone/common/config.py, some ListOpt config object was inited
with a string or None default value, it's incorrect. This patch modifies
their default value to list objects.

Change-Id: I2aec4f49d94e131d45f7dc9bfeb63b8d819d0bb1
Closes-Bug: #1259735
Resolve oauth dependency after paste pipeline is loaded

When trying to authenticate with the oauth plugin, it is
possible that the dependency is not yet loaded, and as a result
the user will receive an error message: "Oauth is disabled"

This patch will ensure the dependency is loaded after the pipeline

Fixes: bug #1264803

Change-Id: I37484bf844896b1a1b560923a8afaf99e91635e4
Remove netifaces requirement

netifaces is not required to run the tests, so remove it from
the requirements.

Related-Bug: #1266513
Change-Id: Ifb3b262f47d629670b06c670353dbe798af4dc03
Tests use cleanUp rather than tearDown

The tests were putting clean-up code in tearDown. Using tearDown
causes problems with developing and testing because if there's an
exception raised in setUp then tearDown is not called and this
typically affects other tests. For example, you'll get several
tests failing because one test's setUp failed and now the other
tests can't load fixtures.

Also, this should make it easier to switch to using fixtures since
fixtures use cleanUp.

Change-Id: I0d0d14ec34ebbeb9dbf0fe2fe3dd304b868bcf93
Cleanup test_associate_project_endpoint_extension

This moves the addCleanup closer to where it's required and
takes care of a TODO. Turns out that cleanup up self.conf_files
is required.

Change-Id: I4229b950fe3dc7e575f921f2b63c4462ed459108
Fix external auth (REMOTE_USER) plugin support

According to the WSGI specification "REMOTE_USER should be the string
username of the user, nothing more" [1], therefore no modifications
should be made to the REMOTE_USER variable and it should be fully
considered as the username. Otherwise the expected semantics of the
REMOTE_USER variable change, and an site administrator could get
undesirable side-effects.

[1] http://wsgi.readthedocs.org/en/latest/specifications/simple_authentication.html#specification

Moreover, it is important to have a consistent behaviour regarding
external authentication in V2 (not domain aware), V3 with default
domain and V3 with domain (see Bug #1253484) so that we produce similar
results with the three methods.

This change aims to solve this issues by removing the split of the
REMOTE_USER variable by "@" at all:

- In external.DefaultDomain, we cannot split REMOTE_USER by "@". This split
  will cause errors for remote users containing an "@" (not only
  emails, but also X.509 subjects, etc). The external.DefaultDomain plugin
  considers the REMOTE_USER variable as the username, and the configured
  default domain as the domain

- In external.Domain we should not split also the REMOTE_USER by "@". A
  new environment variable (REMOTE_DOMAIN) is introduced, so that any
  external plugin can pass down the right domain for the user. The
  external.Domain plugin considers the REMOTE_USER as the username, the
  REMOTE_DOMAIN as the domain if it is present, otherwise it takes the
  configured default domain.

- Two legacy plugins are also provided with the same behaviour as the
  Havana shipped ones. This plugins should not be used and are provided
  for compatibility reasons (see Bug #1254619)

Closes-Bug: #1254619
Closes-Bug: #1211233
Closes-Bug: #1253484

DocImpact: This change breaks backwards compatibility in favour of
security (see bug #1254619), therefore an upgrade not is needed. It is
needed to document the new plugins and state clearly the semantics of
the REMOTE_USER and REMOTE_DOMAIN variable for the WSGI filters. The
default external authentication plugin has been changed from
exernal.ExternalDefault to external.Default.

Change-Id: I1b2521a526fa976146dfe2fcf4d4c1851416d8ae
Merge "races cause 404 when removing user from project"
Merge "Remove unused test function"
Merge "Remove netifaces requirement"
Merge "Change ListOpt default value from str or None to list"
Merge "Tests use cleanUp rather than tearDown"
Merge "Resolve oauth dependency after paste pipeline is loaded"
Merge "Fix external auth (REMOTE_USER) plugin support"