Issue: ResultTimeout when using SqliteQueueDatabase in forked process
Author: theotherp
State: closed
Created: 2016-10-05 17:32:34

I tried your new SqliteQueueDatabase which appears to be awesome and exactly what I needed to get rid of some ugly _database is locked_ errors. Unfortunately I can't get it to work inside a forked process. I use [os.fork()](https://github.com/theotherp/nzbhydra/blob/master/nzbhydra.py#L38) to run my program as a daemon process. The next time I try to access to the database I get:

```
Traceback (most recent call last):
  File "/home/user/app/nzbhydra/libs/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/user/app/nzbhydra/libs/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/user/app/nzbhydra/nzbhydra/web.py", line 338, in wrapped_function
    return f(*args, **kwargs)
  File "/home/user/app/nzbhydra/libs/webargs/core.py", line 368, in wrapper
    return func(*new_args, **kwargs)
  File "/home/user/app/nzbhydra/nzbhydra/web.py", line 667, in internalapi_search
    searchResult = startSearch(search_request)
  File "/home/user/app/nzbhydra/nzbhydra/web.py", line 631, in startSearch
    results = search.search(search_request)
  File "/home/user/app/nzbhydra/nzbhydra/search.py", line 225, in search
    oldSearchResultsCount = SearchResult.select().where(SearchResult.firstFound < (datetime.date.today() - datetime.timedelta(days=keepFor))).count()
  File "/home/user/app/nzbhydra/libs/peewee.py", line 3086, in count
    return self.aggregate(convert=False) or 0
  File "/home/user/app/nzbhydra/libs/peewee.py", line 3079, in aggregate
    return self._aggregate(aggregation).scalar(convert=convert)
  File "/home/user/app/nzbhydra/libs/peewee.py", line 2850, in scalar
    row = self._execute().fetchone()
  File "/home/user/app/nzbhydra/libs/playhouse/sqliteq.py", line 92, in fetchone
    self._wait()
  File "/home/user/app/nzbhydra/libs/playhouse/sqliteq.py", line 51, in _wait
    raise ResultTimeout('results not ready, timed out.')
```

I realize this use case / problem is somewhat obscure but would be happy for any help. Thank you.

Edit: I'm currently trying to write a script that reproduces this but, of course, haven't been able so far.


From the SQLite FAQ:

> Under Unix, you should not carry an open SQLite database across a fork() system call into the child process.


Thanks. But what if I initialize the database in the child process, not the parent process?


Can you show more code, particularly:
- How are you instantiating the database?
- What database things happen before fork?
- What database things happen after fork?


Are you calling `db.start()` in the child process? Are you doing anything like creating tables before the fork? Are any connections (or implicit connections) created before the fork?


You're right, I was not explicitly calling `db.start()` but instead using `autostart=True`. I thought that when initializing the database in the child process that would suffice. But apparently the SQLite instance is opened in the parent process but the database connection only made in the child process or something? You know that better than I do, but after disabling autostart and explicitly calling `db.start()` in the child process everything works fine.

Thanks so much! I'd be happy to send you a tip or a postcard or what not


> But apparently the SQLite instance is opened in the parent process but the database connection only made in the child process or something?

No, that's not correct. I've started adding documentation for this new feature, which explains how all of this works:

http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#sqliteq

If, after reading, you still have questions about how this works feel free to respond on this ticket.


> Thanks so much! I'd be happy to send you a tip or a postcard or what not

Thanks for the offer, the best thanks you can give me is to continue creating issues. In this case there wasn't a bug, technically, but I'm sure other people will have the same question and hopefully find our discussion.


Alright, thanks again and I'll definitely open tickets when I have questions or found a bug that's actually one ;-)