Metaswitch/calico-neutron — pull request #4 by neiljerram (2014-11-21 11:29:43+00:00)
Title: Use a distinct Neutron DB context on each green thread

Without this, we were frequently seeing exceptions like:

Mechanism driver 'calico' failed in create_port_postcommit
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/managers.py", line 168, in _call_on_drivers
getattr(driver.obj, method_name)(context)
File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/mech_calico.py", line 462, in create_port_postcommit
constants.PORT_STATUS_ACTIVE)
File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/plugin.py", line 779, in update_port_status
updated = True
File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/usr/lib/python2.7/contextlib.py", line 121, in nested
if exit(*exc):
File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 439, in __exit__
self._assert_active(deactive_ok=True, prepared_ok=True)
File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 203, in _assert_active
raise sa_exc.ResourceClosedError(closed_msg)
ResourceClosedError: This transaction is closed

and:

Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/mech_calico.py", line 321, in _ensure_socket_to_felix_peer
'start_flag': True})
File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 180, in create_or_update_agent
return self._create_or_update_agent(context, agent)
File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 174, in _create_or_update_agent
greenthread.sleep(0)
File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 439, in __exit__
self._assert_active(deactive_ok=True, prepared_ok=True)
File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 203, in _assert_active
raise sa_exc.ResourceClosedError(closed_msg)
ResourceClosedError: This transaction is closed

We believe these were caused by using the same DB context object on
multiple green threads. It seems likely that:

- one thread can start a DB transaction, and yield after sending a
request to the SQL server

- another thread can start an unrelated DB transaction, but using the
same context object

- things then get confused.

This fix, therefore, implements a more rational design for the
creation and use of DB contexts:

- Each green thread that the Calico driver spawns gets its own DB
context.

- When Calico driver code is called via hooks from outside, it uses
the DB context that those hooks pass in.