Repository: Flexget/Flexget

Issue:
DB_Analyze + Database is locked
#### Log:

```
2016-08-26 12:17 INFO     db_analyze    task    Running ANALYZE on database to improve performance.
2016-08-26 12:19 WARNING  apscheduler.scheduler                 Execution of job "* (trigger: interval[1:00:00], next run at: 2016-08-26 12:19:02 EEST)" skipped: maximum number of running instances reached (1)
Exception in thread APScheduler:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/xxx/apps/flexget/local/lib/python2.7/site-packages/apscheduler/schedulers/blocking.py", line 30, in _main_loop
    wait_seconds = self._process_jobs()
  File "/home/xxx/apps/flexget/local/lib/python2.7/site-packages/apscheduler/schedulers/base.py", line 960, in _process_jobs
    jobstore.update_job(job)
  File "/home/xxx/apps/flexget/local/lib/python2.7/site-packages/apscheduler/jobstores/sqlalchemy.py", line 101, in update_job
    result = self.engine.execute(update)
  File "/home/xxx/apps/flexget/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1991, in execute
    return connection.execute(statement, *multiparams, **params)
  File "/home/xxx/apps/flexget/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "/home/xxx/apps/flexget/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/xxx/apps/flexget/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/home/xxx/apps/flexget/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "/home/xxx/apps/flexget/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
    exc_info
  File "/home/xxx/apps/flexget/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/xxx/apps/flexget/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/home/xxx/apps/flexget/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) database is locked [SQL: u'UPDATE apscheduler_jobs SET next_run_time=?, job_state=? WHERE apscheduler_jobs.id = ?'] [parameters: (1472206742.394899, <read-only buffer for 0x7fd6b4014be0, size -1, offset 0 at 0x7fd6a85a5ef0>, '4b7429f83440f1967e4c5e82c357e52d4911ee88')]
```

APScheduler crashing at regularly irregular intervals
APScheduler keeps crashing with the following message. It doesn't do it every run, or even every day. But at least once a week I need to manually restart the flexget daemon to get everything running again.


#### Log:
```
Exception in thread APScheduler:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/apscheduler/schedulers/blocking.py", line 30, in _main_loop
    wait_seconds = self._process_jobs()
  File "/usr/local/lib/python2.7/dist-packages/apscheduler/schedulers/base.py", line 979, in _process_jobs
    jobstore.update_job(job)
  File "/usr/local/lib/python2.7/dist-packages/apscheduler/jobstores/sqlalchemy.py", line 101, in update_job
    result = self.engine.execute(update)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2064, in execute
    return connection.execute(statement, *multiparams, **params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 945, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1209, in _execute_context
    self._root._commit_impl(autocommit=True)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 726, in _commit_impl
    self.connection._reset_agent is self.__transaction:
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 351, in connection
    self._handle_dbapi_exception(e, None, None, None, None)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1405, in _handle_dbapi_exception
    util.reraise(*exc_info)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 349, in connection
    return self._revalidate_connection()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 429, in _revalidate_connection
    raise exc.ResourceClosedError("This Connection is closed")
ResourceClosedError: This Connection is closed
```

### Additional information:

- FlexGet version: 2.11.4
- Python version: 2.7.12
- Installation method: pip
- Using daemon (yes/no): yes
- OS and version: Ubuntu 16.04.3 LTS



Discussion:
Run with `--debug-db-sessions` if I remember the name correctly 

Hi, nearly a year ago I had the [database locked issue](https://github.com/Flexget/Flexget/issues/703) as well, it just disappeared by itself for a time, but came back soon after, just was too lazy to follow up on this.
I recently installed the OS from scratch on my DS 215j, this time I used a cleaner installing method for flexget by using opkg instead of the package manager. The database locked error however is still there. I attached the whole log starting today 10am with `--debug-db-sessions` parameter. Within August I had 411 crash files with database locked errors.

[crash_report.2016.09.24.120017673523.txt](https://github.com/Flexget/Flexget/files/491112/crash_report.2016.09.24.120017673523.txt)
[crash_report.2016.09.24.123035310988.txt](https://github.com/Flexget/Flexget/files/491111/crash_report.2016.09.24.123035310988.txt)
[crash_report.2016.09.24.133017777554.txt](https://github.com/Flexget/Flexget/files/491113/crash_report.2016.09.24.133017777554.txt)
[crash_report.2016.09.24.140035551689.txt](https://github.com/Flexget/Flexget/files/491110/crash_report.2016.09.24.140035551689.txt)
[crash_report.2016.09.24.143035321351.txt](https://github.com/Flexget/Flexget/files/491115/crash_report.2016.09.24.143035321351.txt)
[flexget.txt](https://github.com/Flexget/Flexget/files/491114/flexget.txt)

i've been seeing the same issue on my freshly installed ubuntu 16.04 server. no idea how to reproduce it cause it seems to be happening randomly. btw, my sqlite db was copied over from my old flexget installation on an ubuntu 12.04 server. maybe that has something to do with it?

I don't think that an old database makes the issue persist, I tried it with a new one multiple times. I also just tried switching from python 2.7 to 3.5, but it didn't help.

#1000 is related

```
2018-03-13 22:27 INFO     manager       foo Running database cleanup.
2018-03-13 22:27 INFO     db_analyze    foo Running ANALYZE on database to improve performance.
Exception in thread APScheduler:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/apscheduler/schedulers/blocking.py", line 30, in _main_loop
    wait_seconds = self._process_jobs()
  File "/usr/local/lib/python2.7/dist-packages/apscheduler/schedulers/base.py", line 979, in _process_jobs
    jobstore.update_job(job)
  File "/usr/local/lib/python2.7/dist-packages/apscheduler/jobstores/sqlalchemy.py", line 107, in update_job
    result = self.engine.execute(update)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2064, in execute
    return connection.execute(statement, *multiparams, **params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 945, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
    context)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1402, in _handle_dbapi_exception
    exc_info
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
    context)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 470, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) database is locked [SQL: u'UPDATE apscheduler_jobs SET next_run_time=?, job_state=? WHERE apschedule
r_jobs.id = ?'] [parameters: (1520973863.953532, <read-only buffer for 0x6ba78c050a00, size -1, offset 0 at 0x6ba782a22d70>, '2b8a9516624eabf6de4
d0f84aab06939d493e2d0')]
```

After this error Flexget just stops doing anything. I'd prefer a crash over a weird freeze.

I actually had to add a healthcheck to my FG config that pokes a webservice on each run just to see if it has frozen or not. The other option would be to look at the log mtime and see if it gets updated.
If someone can reproduce this consistently, it'd be a huge help. I have been unable to.
This is 100% a race condition of some sort, but the regularity of its irregularity is baffling.

There really is no pattern to it, at least one that I can determine.

Resolving patch:
diff --git a/flexget/_version.py b/flexget/_version.py
--- a/flexget/_version.py
+++ b/flexget/_version.py
@@ -7,4 +7,4 @@
 The jenkins release job will automatically strip the .dev for release,
 and update the version again for continued development.
 """
-__version__ = '2.14.26.dev'
+__version__ = '2.15.0.dev'
diff --git a/flexget/plugins/daemon/scheduler.py b/flexget/plugins/daemon/scheduler.py
--- a/flexget/plugins/daemon/scheduler.py
+++ b/flexget/plugins/daemon/scheduler.py
@@ -3,6 +3,7 @@
 
 import hashlib
 import logging
+import os
 
 import pytz
 import tzlocal
@@ -121,7 +122,13 @@ def setup_scheduler(manager):
     global scheduler
     if logging.getLogger().getEffectiveLevel() > logging.DEBUG:
         logging.getLogger('apscheduler').setLevel(logging.WARNING)
-    jobstores = {'default': SQLAlchemyJobStore(engine=manager.engine, metadata=Base.metadata)}
+    # Since APScheduler runs in a separate thread, slower devices can sometimes get a DB lock, so use a separate db
+    # for the jobs to avoid this
+    db_filename = os.path.join(manager.config_base, 'db-%s-jobs.sqlite' % manager.config_name)
+    # in case running on windows, needs double \\
+    db_filename = db_filename.replace('\\', '\\\\')
+    database_uri = 'sqlite:///%s' % db_filename
+    jobstores = {'default': SQLAlchemyJobStore(url=database_uri)}
     # If job was meant to run within last day while daemon was shutdown, run it once when continuing
     job_defaults = {'coalesce': True, 'misfire_grace_time': 60 * 60 * 24}
     try:


Test patch: