Fix doc build errors in db.sqlalchemy

There was some invalid rst in the db.sqlalchemy docstrings which
causes warnings to be generated during the doc build and also
causes the docs to be rendered incorrectly.

Change-Id: I5c14b714b3ab0186150e8f3e481bac4c0bb58c25
Handle ibm_db_sa DBDuplicateEntry integrity errors

This patch adds duplicate entry integrity error handling for the
ibm_db_sa driver to the common sqlalchemy db session.

Closes-Bug: #1227321
Part of nova blueprint: db2-database

Change-Id: I2f0d3d660373e4f647615f121fed0da9442f8cd2
Introduce a method to set any MySQL session SQL mode

Rather than implement a method just for the special case of setting
MySQL TRADITIONAL mode, implement one that can set any supported
server mode, and reimplement the old method as a wrapper around that.

In addition, check the actual effective SQL mode
(as per SHOW VARIABLES LIKE 'sql_mode'), and log a warning if it's
deemed unsafe.

Co-author: Ben Nemec <openstack@nemebean.com>

Change-Id: I90139b915b2ff169878fd93625e5320d39094ed3
Introduce mysql_sql_mode option, remove old warning

Introduce a config option, mysql_sql_mode, that allows the user
to set the desired SQL mode for MySQL sessions.

For MySQL sessions, set the SQL mode session variable
accordingly.

For compatibility purposes, retain the mysql_traditional_mode
argument in method signatures, but only use it to override the
config option as if it were set to 'TRADITIONAL'.

Change-Id: Ia527e850671ab9542db5864617384cee21e91bf6
Closes-bug: 1271706
Make TRADITIONAL the default SQL mode

Rather than log a warning when TRADITIONAL mode is not enabled,
just make it the default. If this causes INSERTs or UPDATEs from
applications to break, it just exposes that they are inserting
broken data to begin with.

Change-Id: I84c1781454ed5f46c7a135c96bafc575f313f9bc
Drop special case for MySQL traditional mode, update unit tests

Since there are no users of the special-case TRADITIONAL mode set,
it's OK to drop those references and use only the more flexible
one tuned via a config option.

Update the TRADITIONAL mode unit test, and also add one for
STRICT_ALL_TABLES mode.

Change-Id: Ie9df3aaa5f48d992b61a9ea480d1e17519f16422
db: move all options into database group

This moves all the options in the database group so it's easier to iterate
over them once and for all in order to pass them to the method creating
the engine.

Change-Id: If3117f01d746f12a07cf4c3b4c4fc1a1ce8b679d
Add from_config() method to EngineFacade

Add a helper that allows to create an EngineFacade instance given an
oslo.config instance. This helper is completely optional, but it was
requested by a few people syncing the latest oslo.db code to target
projects.

Note, that oslo.config is mocked in tests, so oslo.db still has no
runtime dependency on oslo.config.

Blueprint: oslo-db-lib

Change-Id: Icab476386b859be0b0e984a94cbfcf3d581ab6d2
Python3: define a __next__() method for ModelBase

In Python 3, __next__() has replaced next().

Blueprint: make-python3-compatible
Change-Id: I96a7a4602be92cf78d4926bc4afd66bda9789395
Prevent incorrect usage of _wrap_db_error()

_wrap_db_error() has an implicit contract that it can be applied only
to methods of SQLAlchemy Session subclasses. This seems to be a good
task for an assert statement (as it's something we want to check
during development and debugging, but not something that an end user
can trigger, which would require a general runtime check).

Change-Id: I0a15bf4a90b0aea5fa715027d13d8af816f9e72d
Get mysql_sql_mode parameter from config

Patch Ia527e850671ab9542db5864617384cee21e91bf6 added mysql_sql_mode
option to config, so we should get it's value from CONF object, not
from function argument.

Closes-Bug: #1288445

Change-Id: I4be81718399393d817185abe36f4cc4d3da15000
Remove requirements.txt from .gitignore

There is no reason to keep requirements.txt ignored by get, so we should
remove it from .gitignore

Change-Id: I30852f8281d608c5f7c48217c21c3099b9b0fd0e
Make db utils importable without migrate

One of functions defined in openstack.common.db.sqlalchemy.utils uses
UniqueConstraint class imported from migrate (sqlalchemy-migrate). It
actually makes sense to move import of migrate directly into the
function to make utils module importable in projects which use
alembic instead of sqlalchemy-migrate (e.g. neutron and ironic).

Change-Id: I382f8c14c69050c8f2ea0a0ba6bf2f1eef84e14d
Adapt DB provisioning code for CI requirements

With recent changes made to our CI MySQL openstack_citest user now
has enough priviliges to create and drop databases. But the code,
that we use for provisioning of databases must be updated, so that
it's not mandatory for openstack_citest user to be also able to
do GRANT/ALTER ROLE queries. Actually this simplifies provisioning
code a bit and allows to reuse the same queries for PostgreSQL and
MySQL.

This will be used for opportunistic db test cases as well as for
running of DbTestCase tests on different backends.

Change-Id: Ia917f647182d908dbb2a86659ad8b4e49c492881
Move test_insert_from_select unit test from nova to oslo

Nova had it's own copy of InsertFromSelect so change Iefa5b431 is
removing that copy from Nova to use the version in Oslo. The only
difference is Nova had an additional unit test for InsertFromSelect so
we can move that to Oslo.

Partial-Bug: #1280055

Change-Id: I457acf335c4877c398c443b0cfde55bb997dfe6d
Add lockutils fixture to OpportunisticTestCase

We're getting hit hard with db race problems right now and the
proper fix can't go in until infra has all of their regions
updated.  Using the lock fixture to serialize all of the db tests
should help with the problem until infra is ready.

Change-Id: If883832b0eba08f1508a247310b8eebd67b27971
Partial-Bug: #1288916
Fix excessive logging from db.sqlalchemy.session

A message about the MySQL server mode was logged at the info level
every time a connection was checked out. This happens often, and
logging every time is unnecessary.

Here's an example of the message when using the default
configuration:

 MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,...

The fix is to send the message once per engine creation. Also,
this message is now logged the debug level since there's a warning
logged if the mode isn't a "safe" one already.

Closes-Bug: #1288443
Co-Authored-By: Ben Nemec <bnemec@redhat.com>

Change-Id: I13b24d04c28e69a8534aaa8d3a4463ca1d65570f
Set sql_mode callback on connect instead of checkout

There's no need to set the sql_mode on every checkout of a
connection - once when the connection is created is sufficient.

Change-Id: If34db31d9646c9a75d591e556a7abcc0bf8a7ab1
Closes-Bug: #1288444
Do not use the 'extend' method on a dict_items object

This method does not exist: it is meant to be used on a list. In Python 3,
keys() return a dict_items object, instead of a list in Python 2, so an error
happens. Fix this by converting the dict_items object  to a list before calling
extend().

This is enough to fix tests/unit/db/sqlalchemy/test_models.py in Python 3, so
we can enable this test.

Change-Id: Ic09af6c084e89e19ddc60058270890dbe052c85e
Python3: pass bytes as 'contents' to create_tempfiles()

os.write() expect bytes, so make sure that's what we're passing.

Change-Id: I98eae905e1d2b3802b7f5326c40f8b66aba21519