Repository: translate/pootle

Issue:
Stats: RQ fails jobs if no jobs were executed during more than MySQL `wait_timeout` time
We get `OperationalError: (2006, 'MySQL server has gone away')` every time we touch (do something that causes stats refreshing) our staging server after long break. 
To reproduce:
- set MySQL `wait_timeout` variable to 10 seconds (or any value you are able to wait);
- start `rqworker`;
- add or change a translation for any unit;
- check that stats were refreshed;
- wait a little bit more than 10 seconds;
- add or change a translation for any unit;
- stats were not refreshed, last RQ job failed.



Discussion:
As described in [Django ticket 15119](https://code.djangoproject.com/ticket/15119), we can see that Django was making a ping to validate if the connection was alive before executing every query. This behavior was fixed in [commit 282b2f4](https://github.com/django/django/commit/282b2f40cd0aa09815001e178f60c9e71667d847).

Django developers gave one short answer for all questions like this in https://code.djangoproject.com/ticket/21597#comment:29

Therefore `rqworker` process should validate connection itself for each new job.
(Note: if we close a connection then Django will create a new one).

I’d suggest to use Django per request approach for jobs and call `django.db.close_old_connections()` before and after every job.

Potentially opening a new connection each time can slow down the work if we have many small jobs. Is there a way to close a connection only when the rq queue is empty?

Sometimes i still have this problem, but i resolve it restarting the rqworker :/

@iafan yes, it's possible to check if rq queue is empty. But we need to call `django.db.close_old_connections()` BEFORE a job. And if the job is not last queue won't be empty.

Another idea: to wrap a job within `try..except` and call `django.db.close_old_connections()` if `DatabaseError` is caught and then retry the job. 

> Another idea: to wrap a job within try..except and call django.db.close_old_connections() if DatabaseError is caught and then retry the job.

Looks good to me.

> Another idea: to wrap a job within try..except and call django.db.close_old_connections() if DatabaseError is caught and then retry the job.
> 
> Looks good to me.

This has the potential of masking other DB errors which are unrelated to this issue. I don't believe we want to go down that route.

Going back to the first approach proposed by @ta2-1, and if IIUC, using [`close_old_connections()`](https://github.com/django/django/blob/8f42934f86c1c6dff9fb24fd6fadb9d72b89d7b6/django/db/__init__.py#L103-L110) before and after jobs are processed looks safe to me; in the end [connections will only be closed if they timed out or an unrecoverable error has happened](https://github.com/django/django/blob/8f42934f86c1c6dff9fb24fd6fadb9d72b89d7b6/django/db/backends/__init__.py#L475-L498). New connections will only be (implicitly) triggered on the next ORM query, therefore avoiding _server has gone away_ issues. This is what Django's own request/response cycle does.

It's also probably worth for deployments to check the [`CONN_MAX_AGE` setting for persistent connections](https://docs.djangoproject.com/en/1.7/ref/databases/#persistent-connections) (not to be confused with [connection pooling](https://groups.google.com/forum/#!topic/django-developers/NwY9CHM4xpU)), which defaults to 0 due to backwards compatibility.

@julen, I'm afraid Django doesn't check if a connection timed out  in [close_old_connections()](https://github.com/django/django/blob/8f42934f86c1c6dff9fb24fd6fadb9d72b89d7b6/django/db/__init__.py#L103-L110). Therefore we can use this approach if we set `CONN_MAX_AGE` to a value less than MySQL `wait_timeout` only.

Doesn't a timeout qualify for _unusable connection_, then? Refs. [`close_if_unusable_or_obsolete()`](https://github.com/django/django/blob/8f42934f86c1c6dff9fb24fd6fadb9d72b89d7b6/django/db/backends/__init__.py#L487-L494) (also: [`errors_occurred`](https://github.com/django/django/blob/8f42934f86c1c6dff9fb24fd6fadb9d72b89d7b6/django/db/utils.py#L90-L93), [`is_usable()` from MySQL](https://github.com/django/django/blob/8f42934f86c1c6dff9fb24fd6fadb9d72b89d7b6/django/db/backends/mysql/base.py#L558-L564)).

I'd like to think so. But `error_occured` attribute can be set after an error has happened. It's enough one error RQ jobs fail. You can check and reproduce it if `CONN_MAX_AGE` is `None` or greater than MySQL `wait_timeout`.

I'm not sure I understand your comment regarding RQ; can you reword it please?

This also affects my installation.

> Doesn't a timeout qualify for unusable connection, then?

Do you mean MySQL `wait_timeout`?

 [`close_if_unusable_or_obsolete()`](https://github.com/django/django/blob/8f42934f86c1c6dff9fb24fd6fadb9d72b89d7b6/django/db/backends/__init__.py#L487-L494) will not close your connection if you set `CONN_MAX_AGE` to `None` or to a value greater than MySQL `wait_timeout`.

You are right that [`errors_occurred`](https://github.com/django/django/blob/8f42934f86c1c6dff9fb24fd6fadb9d72b89d7b6/django/db/utils.py#L90-L93) is set by  [`is_usable()` from MySQL](https://github.com/django/django/blob/8f42934f86c1c6dff9fb24fd6fadb9d72b89d7b6/django/db/backends/mysql/base.py#L558-L564) but  [`close_if_unusable_or_obsolete()`](https://github.com/django/django/blob/8f42934f86c1c6dff9fb24fd6fadb9d72b89d7b6/django/db/backends/__init__.py#L487-L494) doesn't call it. And as we see Django has gone from pinging before every request.

It will work if we use the first approach and leave `CONN_MAX_AGE = 0` by default. After we can change  `CONN_MAX_AGE` value to specific value for our DB and server in custom settings.

For clarity I'll describe a solution that we can go forward with (after me and @julen discussed it).

We can wrap jobs with `close_if_unusable_or_obsolete()` to imitate Django's request/response cycle (see [connection management](https://docs.djangoproject.com/en/1.7/ref/databases/#connection-management)).

Django (within rqworker process) opens a connection to the DB when it first makes a DB query. It keeps this connection open and reuses it in jobs. Django closes the connection once it exceeds the maximum age defined by `CONN_MAX_AGE` before and after any job.   

We need to warn people that setting `CONN_MAX_AGE` can have negative side-effects and we can't use persistent connections where `CONN_MAX_AGE = None` if DB  terminates idle connections after some time `X`. `CONN_MAX_AGE` should be set to a not `None` value lower than `X`.

Default value: `CONN_MAX_AGE = 0`. Thus Django will close the connection after every job is executed and create new one for a next job. I tested if it's expensive to recreate connections on `refresh_stats` for the whole EN database and there was no valuable difference in timing.

Nice summary @ta2-1. Also worth mentioning that any value `X` for `CONN_MAX_AGE` lower than the value of MySQL's `wait_timeout` setting will be fine.


Resolving patch:
diff --git a/pootle/core/mixins/treeitem.py b/pootle/core/mixins/treeitem.py
--- a/pootle/core/mixins/treeitem.py
+++ b/pootle/core/mixins/treeitem.py
@@ -681,7 +681,14 @@ def update_cache_job(instance):
     job = get_current_job()
     job_wrapper = JobWrapper(job.id, job.connection)
     keys, decrement = job_wrapper.get_job_params()
+
+    # close unusable and obsolete connections before and after the job
+    # Note: setting CONN_MAX_AGE parameter can have negative side-effects
+    # CONN_MAX_AGE value should be lower than DB wait_timeout
+    connection.close_if_unusable_or_obsolete()
     instance._update_cache_job(keys, decrement)
+    connection.close_if_unusable_or_obsolete()
+
     job_wrapper.clear_job_params()
 
 


Test patch: