Repository: qutebrowser/qutebrowser

Issue:
New completion: "Database is locked" errors
Around once a week I get a report with something similar to this:

```
20:42:34 DEBUG    sql        sql:run:99 Running SQL query: "SELECT EXISTS(SELECT * FROM History WHERE url = :val)"
20:42:34 DEBUG    sql        sql:run:102 query bindings: {':val': 'https://wiki.archlinux.org/index.php/ArchWiki:Privacy_policy'}
20:42:34 DEBUG    sql        debug:__exit__:264 Historycontains took 0.000158 seconds.
20:42:34 DEBUG    sql        sql:run:99 Running SQL query: "SELECT EXISTS(SELECT * FROM History WHERE url = :val)"
20:42:34 DEBUG    sql        sql:run:102 query bindings: {':val': 'https://wiki.archlinux.org/index.php/ArchWiki:About'}
20:42:34 DEBUG    sql        debug:__exit__:264 Historycontains took 0.000111 seconds.
20:42:34 DEBUG    sql        sql:run:99 Running SQL query: "SELECT EXISTS(SELECT * FROM History WHERE url = :val)"
20:42:34 DEBUG    sql        sql:run:102 query bindings: {':val': 'https://wiki.archlinux.org/index.php/ArchWiki:General_disclaimer'}
20:42:34 DEBUG    sql        debug:__exit__:264 Historycontains took 9.4e-05 seconds.
20:42:34 DEBUG    sql        sql:run:99 Running SQL query: "SELECT EXISTS(SELECT * FROM History WHERE url = :val)"
20:42:34 DEBUG    sql        sql:run:102 query bindings: {':val': 'https://wiki.archlinux.org/index.php?title=Tor&mobileaction=toggle_view_mobile'}
20:42:34 DEBUG    sql        debug:__exit__:264 Historycontains took 0.000114 seconds.
20:42:34 DEBUG    sql        sql:__init__:80 Preparing SQL query: "INSERT INTO History (url, title, atime, redirect) values(:url, :title, :atime, :redirect)"
20:42:34 DEBUG    sql        sql:run:99 Running SQL query: "INSERT INTO History (url, title, atime, redirect) values(:url, :title, :atime, :redirect)"
20:42:34 DEBUG    sql        sql:run:102 query bindings: {':atime': 1501461754, ':redirect': False, ':title': 'Tor - ArchWiki', ':url': 'https://wiki.archlinux.org/index.php/Tor#Usage'}
20:42:34 ERROR    misc       crashsignal:exception_hook:210 Uncaught exception
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/qutebrowser/browser/history.py", line 144, in add_from_tab
    self.add_url(url, title)
  File "/usr/lib/python3.6/site-packages/qutebrowser/browser/history.py", line 166, in add_url
    'redirect': redirect})
  File "/usr/lib/python3.6/site-packages/qutebrowser/misc/sql.py", line 214, in insert
    q.run(**values)
  File "/usr/lib/python3.6/site-packages/qutebrowser/misc/sql.py", line 105, in run
    self.lastQuery(), self.lastError().text()))
qutebrowser.misc.sql.SqlException: Failed to exec query "INSERT INTO History (url, title, atime, redirect) values(:url, :title, :atime, :redirect)": "database is locked Unable to fetch row"
```

Reports:

- https://crashes.qutebrowser.org/view/366bb784
- https://crashes.qutebrowser.org/view/ec6b1124
- https://crashes.qutebrowser.org/view/6625b09c
- https://crashes.qutebrowser.org/view/3fedf0f3
- https://crashes.qutebrowser.org/view/9c7bd0f3
- https://crashes.qutebrowser.org/view/7dd8737b
- https://crashes.qutebrowser.org/view/14e98b96
- https://crashes.qutebrowser.org/view/5c58ce61
- https://crashes.qutebrowser.org/view/1f24788f
- https://crashes.qutebrowser.org/view/41a3a643
- https://crashes.qutebrowser.org/view/9dd6f9df
- https://crashes.qutebrowser.org/view/1f997d9a
- https://crashes.qutebrowser.org/view/73e77cae
- https://crashes.qutebrowser.org/view/cbc9fe19

@rcorre Not sure how to solve this. Should we just retry if we get such an error? Or just log it and ignore it?


Discussion:
Hmm, for https://crashes.qutebrowser.org/view/6625b09c it's right during launch:

```
10:44:07 DEBUG    init       app:_process_args:209 Init finished after 0.763425s
10:44:07 DEBUG    misc       standarddir:_writable_location:143 writable location for AppLocalDataLocation: /home/daniel/.local/share/qutebrowser
10:44:07 DEBUG    init       app:init:163 Init done!
10:44:07 DEBUG    misc       mainwindow:_update_overlay_geometry:285 new geometry for <qutebrowser.mainwindow.messageview.MessageView object at 0x7f72a5039f78>: PyQt5.QtCore.QRect(0, 785, 1361, 1)
10:44:07 DEBUG    misc       app:on_focus_object_changed:793 Focus object changed: <qutebrowser.browser.webkit.webview.WebView tab_id=0 url=''>
10:44:07 DEBUG    webview    tabbedbrowser:on_title_changed:561 Changing title for idx 0 to 'https://start.duckduckgo.com/'
10:44:07 DEBUG    signals    signalfilter:_filter_signals:86 emitting: cur_url_changed(PyQt5.QtCore.QUrl('https://start.duckduckgo.com/')) (tab 0)
10:44:07 DEBUG    webview    tabbedbrowser:on_title_changed:553 Ignoring title change to ''.
10:44:07 DEBUG    webview    tabbedbrowser:on_title_changed:561 Changing title for idx 0 to 'DuckDuckGo'
10:44:07 DEBUG    save       savemanager:mark_dirty:72 Marking cookies as dirty.
10:44:07 DEBUG    save       savemanager:mark_dirty:72 Marking cookies as dirty.
10:44:07 DEBUG    save       savemanager:mark_dirty:72 Marking cookies as dirty.
10:44:07 DEBUG    save       savemanager:mark_dirty:72 Marking cookies as dirty.
10:44:07 DEBUG    webview    webpage:acceptNavigationRequest:478 navigation request: url https://start.duckduckgo.com/post2.html, type NavigationTypeOther, target ClickTarget.normal override None
10:44:07 DEBUG    sql        sql:__init__:80 Preparing SQL query: "INSERT INTO History (url, title, atime, redirect) values(:url, :title, :atime, :redirect)"
10:44:07 DEBUG    sql        sql:run:99 Running SQL query: "INSERT INTO History (url, title, atime, redirect) values(:url, :title, :atime, :redirect)"
10:44:07 DEBUG    sql        sql:run:102 query bindings: {':atime': 1503650647, ':redirect': False, ':title': 'DuckDuckGo', ':url': 'https://start.duckduckgo.com/'}
10:44:12 ERROR    misc       crashsignal:exception_hook:210 Uncaught exception
```

And with a comment which says:

> closing tab. qutebrowser has a habit of freezing on this computer. when killing process, qutebrowser is unable to relaunch without reboot

That might be a slightly different issue actually.
Related thread: https://dba.stackexchange.com/questions/45368/how-do-i-prevent-sqlite-database-locks

Could we use the [write-ahead logging](https://www.sqlite.org/wal.html)? It's available since sqlite 3.7.0.

The last report sounds like the database was left in a locked state when the process was killed. No idea if it will be in a reasonable state when you unlock it...
Looking at [Repology](https://repology.org/metapackage/sqlite3/versions), pretty much everything seems to have sqlite 3.7.0 or newer available. I'm not sure whether it'd help though, the page above says "However, since there is only one WAL file, there can only be one writer at a time." - here, looking at the log, I'd guess two threads tried to write at the same time.

I'm not even sure how this is happening - I think everything (adding history and the completion) should happen in the main thread. Maybe Qt/sqlite does the operations in the background through, and one is still running while the next one already followed?
The DBA thread lists this as a possible cause of SQLITE_LOCKED error: 

> Trying to write to a table while a SELECT is active on that same table.

That's what it looks like from the first log and the assumption that all SQL things happen in the same thread. Maybe the conflicts are due to interleaved processing of callbacks coming from different tabs loaded in the background?

I too have no idea if WAL would help, I'm more into PostgreSQL rather than SQLite, but it sounds like it should solve this problem. There would still be only one writer, but it would not conflict with the reads.
It's actually also possible to check if a select query is active: http://doc.qt.io/qt-5/qsqlquery.html#isActive
Maybe calling `finish()` would be enough?
I suppose we could try to detect when the completion is closed (maybe there is already a signal?) and call `finish`. Currently the completion model is only replaced when you change completion types, so if the user closes and reopens `url` completion, it will be the exact same `SqlCategory` instance. Maybe the `SELECT` query is remaining `active` that whole time.

I haven't been able to repro though, and I'm using sql completion for my day-to-day browsing, so I'm sure I browse around after opening url completion all the time.

Another potential culprit is `qute://history`, is it possible to tell if that tab is open in the crash reports?
I don't think `qute://history` is to blame, and I'm not sure whether the completion is to blame either. From what it looks, it's only the history insertion (on page load) and checking the history (to colorize visited links) conflicting. All the reports seem to be on QtWebKit-NG. I've never seen it happen either, but I use QtWebEngine.
Right, there is no equivalent of [WebHistoryInterface.historyContains](https://github.com/qutebrowser/qutebrowser/blob/master/qutebrowser/browser/webkit/webkithistory.py#L46-L57) for webengine (which might also be a problem for #2941). I doubt that it's the particular type of query that would cause conflicts, it's just that the `historyContains` calls happen fast enough.
I tried to reproduce this with `--backend webkit` and a `--basedir` on a very slow location (a remote directory mounted with sshfs), but still no crash. I even tried to open multiple URLs at once from the command line, so there were many selects via `historyContains` and inserts when each tab finished.

By the way, I think that the `historyContains` part might be actually slower than the original non-SQL version on some systems. The original function worked only on data in memory (an OrderedDict if I remember correctly), but with SQL each query leads to some disk reads - there can be hundreds or thousands per page load. So if the disk is very slow (such as my experiment with the sshfs mount), the browser will become unusable. Maybe that's what lead to the "qutebrowser has a habit of freezing on this computer" comment.
Yeah, they definitely might be a bit slower. There's some aggressive caching though, since WebKit somehow asks for the same links over and over again even on a single page load:

https://github.com/qutebrowser/qutebrowser/blob/2598fd8c5d981e3cb0fa771989d89ca2dd288861/qutebrowser/browser/webkit/webkithistory.py#L46-L47

I've been toying with just not persisting history for the QWebHistoryInterface, but then found out how annoying it actually is to not have stuff you read/visited already marked that way :wink:
Makes me wonder whether this is related to http://beets.io/blog/sqlite-nightmare.html

FWIW, all reports were on Archlinux/Manjaro so far, and at least the [PKGBUILD](https://git.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/sqlite) doesn't mention `HAVE_USLEEP=1` - not sure if it should?
sqlite's configure script contains:
```
#########
# Figure out whether or not we have these functions
#
for ac_func in fdatasync gmtime_r isnan localtime_r localtime_s malloc_usable_size strchrnul usleep utime pread pread64 pwrite
pwrite64
do :
  as_ac_var=`$as_echo "ac_cv_func_$ac_func" | $as_tr_sh`
ac_fn_c_check_func "$LINENO" "$ac_func" "$as_ac_var"
if eval test \"x\$"$as_ac_var"\" = x"yes"; then :
  cat >>confdefs.h <<_ACEOF
#define `$as_echo "HAVE_$ac_func" | $as_tr_cpp` 1
_ACEOF

fi
done
```
And when I tried to compile the package, I saw
```
...
checking for fdatasync... yes
checking for gmtime_r... yes
checking for isnan... yes
checking for localtime_r... yes
checking for localtime_s... no
checking for malloc_usable_size... yes
checking for strchrnul... yes
checking for usleep... yes
checking for utime... yes
checking for pread... yes
checking for pread64... yes
checking for pwrite... yes
...
```
So `#define HAVE_USLEEP 1` indeed ends up in the generated `config.h` file. At least on my machine, obviously...
I managed to get qutebrowser in a state where it crashes with this every time I start it. Unfortunately I'm at work so I can't really dig in right now, but I sent you 3 crash reports :grinning: 
I'm pretty sure it started after I force-quit qutebrowser while a page was loading (probably during the `INSERT` into `History`). The database is now stuck locked even with no qutebrowser instance running. I can't even `INSERT` with the `sqlite3` CLI.
rcorre's reports:

- https://crashes.qutebrowser.org/view/314d218a
- https://crashes.qutebrowser.org/view/b7ae354a
- https://crashes.qutebrowser.org/view/fe8bf9fe
This should still be fixed properly in some way, but now at least those aren't hard crashes anymore.
Whoops, looks like my workaround for this didn't work correctly - 8607ddce249535ed7f72207bf986ad8585b401d2 should hopefully help.

Report: https://crashes.qutebrowser.org/view/8dfe97be

> - Had two tabs open
> - Second tab would freeze when I tried to either :open, :open-t, 'd'
> - The Qutebrowser would freeze completely
> - Killed it in the terminal with 'kill -9'
> - Reponed Qutebrowser fine but the two tabs where still there
> - Made a new window, and :quit from there
Is there a way to 100% reproduce this? I might have a hunch on what might cause this.

Basically, `QSqlQueryModel`s only fetch up to 256 rows and sometimes [1] they fail to release the database lock.

Having a sure way to reproduce this would allow me to check if that's the case or the problem lies somewhere else.

Also, sqlite itself knows which queries are holding a lock, and can report this info with the native C API, but Qt doesn't expose such methods AFAIK.

[1] I actually don't know the exact conditions that trigger this
No, I don't know a way to reproduce it. It seems to happen randomly while qutebrowser is running. Apparently it's also a problem that the database stays locked (i.e. is already locked on startup) after doing something like `pkill -9 qutebrowser` sometimes.

Resolving patch:
diff --git a/qutebrowser/misc/sql.py b/qutebrowser/misc/sql.py
--- a/qutebrowser/misc/sql.py
+++ b/qutebrowser/misc/sql.py
@@ -115,6 +115,11 @@ def init(db_path):
         raise SqliteError("Failed to open sqlite database at {}: {}"
                           .format(db_path, error.text()), error)
 
+    # Enable write-ahead-logging and reduce disk write frequency
+    # see https://sqlite.org/pragma.html and issues #2930 and #3507
+    Query("PRAGMA journal_mode=WAL").run()
+    Query("PRAGMA synchronous=NORMAL").run()
+
 
 def close():
     """Close the SQL connection."""


Test patch: