Package: browserexport
browserexportSupported BrowsersInstallUsagesaveinspect/mergeSerializing to JSONShell CompletionUsage with HPILibrary UsageComparisons with promnesiaContributingDevelopmentTestingThis:locates and backs up browser history by copying the underlying database files to some directory you specifycan identify and parse the resulting database files into some common schema:Visit:
  url: the url
  dt: datetime (when you went to this page)
  metadata:
    title: the <title> for this page
    description: the <meta description> tag from this page
    preview_image: 'main image' for this page, often opengraph/favicon
    duration: how long you were on this pagemetadatais dependent on the data available in the browser (e.g. firefox has preview images, chrome has duration, but not vice versa)Supported BrowsersThis currently supports:FirefoxWaterfoxFloorpLibrewolfFirefox Android (pre-2020 schema and currentFenix)ChromeChromiumBraveVivaldiOperaArcEdge(andDev Channel)SafariPalemoonThis can probably extract visits from other Firefox/Chromium-based browsers, but it doesn't know how to locate them tosavethemInstallpython3 -m pip install --user browserexportRequirespython3.7+UsagesaveUsage: browserexport save [OPTIONS]

  Backs up a current browser database file

Options:
  -b, --browser
      [chrome | firefox | opera | safari | brave | waterfox |
      librewolf | floorp | chromium | vivaldi | palemoon | arc |
      edge | edgedev]
                                  Browser name to backup history for
  --pattern TEXT                  Pattern for the resulting timestamped filename, should include an
                                  str.format replacement placeholder for the date [default:
                                  browser_name-{}.extension]
  -p, --profile TEXT              Use to pick the correct profile to back up. If unspecified, will assume a
                                  single profile  [default: *]
  --path FILE                     Specify a direct path to a database to back up
  -t, --to DIRECTORY              Directory to store backup to. Pass '-' to print database to STDOUT
                                  [required]
  -h, --help                      Show this message and exit.Must specify one of--browser, or--pathAfter your browser history reaches a certain size, browsers typically remove old history over time, so I'd recommend backing up your history periodically, like:$browserexportsave-bfirefox--to~/data/browsing
$browserexportsave-bchrome--to~/data/browsing
$browserexportsave-bsafari--to~/data/browsingThat copies the sqlite databases which contains your history--tosome backup directory.If a browser you want to backup is Firefox/Chrome-like (so this would be able to parse it), but this doesn't support locating it yet, you can directly back it up with the--pathflag:$browserexportsave--path~/.somebrowser/profile/places.sqlite\--to~/data/browsingThe--patternargument can be used to change the resulting filename for the browser, e.g.--pattern 'places-{}.sqlite'or--pattern "$(uname)-{}.sqlite". The{}is replaced by the browser name.Feel free to create an issue/contribute abrowserfile to locate the browser if this doesn't support some browser you use.Can pass the--debugflag to showsqlite_backuplogs$ browserexport --debug save -b firefox --to .
[D 220202 10:10:22 common:87] Glob /home/sean/.mozilla/firefox with */places.sqlite (non recursive) matched [PosixPath('/home/sean/.mozilla/firefox/ew9cqpqe.dev-edition-default/places.sqlite')]
[I 220202 10:10:22 save:18] backing up /home/sean/.mozilla/firefox/ew9cqpqe.dev-edition-default/places.sqlite to /home/sean/Repos/browserexport/firefox-20220202181022.sqlite
[D 220202 10:10:22 core:110] Source database files: '['/tmp/tmpcn6gpj1v/places.sqlite', '/tmp/tmpcn6gpj1v/places.sqlite-wal']'
[D 220202 10:10:22 core:111] Temporary Destination database files: '['/tmp/tmpcn6gpj1v/places.sqlite', '/tmp/tmpcn6gpj1v/places.sqlite-wal']'
[D 220202 10:10:22 core:64] Copied from '/home/sean/.mozilla/firefox/ew9cqpqe.dev-edition-default/places.sqlite' to '/tmp/tmpcn6gpj1v/places.sqlite' successfully; copied without file changing: True
[D 220202 10:10:22 core:64] Copied from '/home/sean/.mozilla/firefox/ew9cqpqe.dev-edition-default/places.sqlite-wal' to '/tmp/tmpcn6gpj1v/places.sqlite-wal' successfully; copied without file changing: True
[D 220202 10:10:22 core:230] Running backup, from '/tmp/tmpcn6gpj1v/places.sqlite' to '/home/sean/Repos/browserexport/firefox-20220202181022.sqlite'
[D 220202 10:10:22 save:14] Copied 1840 of 1840 database pages...
[D 220202 10:10:22 core:246] Executing 'wal_checkpoint(TRUNCATE)' on destination '/home/sean/Repos/browserexport/firefox-20220202181022.sqlite'For Firefox AndroidFenix, the database has to be manually backed up (probably from a rooted phone usingtermux) fromdata/data/org.mozilla.fenix/files/places.sqlite.inspect/mergeThese work very similarly,inspectis for a single database,mergeis for multiple databases.Usage: browserexport merge [OPTIONS] SQLITE_DB...

  Extracts visits from multiple sqlite databases

  Provide multiple sqlite databases as positional arguments, e.g.:
  browserexport merge ~/data/firefox/*.sqlite

  Drops you into a REPL to access the data

  Pass '-' to read from STDIN

Options:
  -s, --stream  Stream JSON objects instead of printing a JSON list
  -j, --json    Print result to STDOUT as JSON
  -h, --help    Show this message and exit.As an example:browserexport --debug merge ~/data/firefox/* ~/data/chrome/*
[D 210417 21:12:18 merge:38] merging information from 24 sources...
[D 210417 21:12:18 parse:19] Reading visits from /home/sean/data/firefox/places-20200828223058.sqlite...
[D 210417 21:12:18 common:40] Chrome: Running detector query 'SELECT * FROM keyword_search_terms'
[D 210417 21:12:18 common:40] Firefox: Running detector query 'SELECT * FROM moz_meta'
[D 210417 21:12:18 parse:22] Detected as Firefox
[D 210417 21:12:19 parse:19] Reading visits from /home/sean/data/firefox/places-20201010031025.sqlite...
[D 210417 21:12:19 common:40] Chrome: Running detector query 'SELECT * FROM keyword_search_terms'
....
[D 210417 21:12:48 common:40] Firefox: Running detector query 'SELECT * FROM moz_meta'
[D 210417 21:12:48 common:40] Safari: Running detector query 'SELECT * FROM history_tombstones'
[D 210417 21:12:48 parse:22] Detected as Safari
[D 210417 21:12:48 merge:51] Summary: removed 3001879 duplicates...
[D 210417 21:12:48 merge:52] Summary: returning 334490 visit entries...

Use vis to interact with the data

[1] ...You can also read from STDIN, so this can be used in conjunction withsave, to merge databases you've backed up and combine your current browser history:browserexportsave-bfirefox-t-|browserexportmerge--json--stream-~/data/browsing/*>all.jsonlOr, to just print the demo for your current browser history:$browserexportsave-bfirefox-t-|browserexportinspect-
Demo:Yourmostcommonsites....[('github.com',21033),...Or, useprocess substitutionto save multiple dbs in parallel and then merge them:$browserexportmerge<(browserexportsave-bfirefox-t-)<(browserexportsave-bchrome-t-)Logs are hidden by default. To show the debug logs setexport BROWSEREXPORT_LOGS=10(useslogging levels) or pass the--debugflag.JSONTo dump all that info to JSON:$browserexportmerge--json~/data/browsing/*.sqlite>./history.json
du-hhistory.json
67Mhistory.jsonOr, to create a quick searchable interface, usingjqandfzf:browserexport merge -j --stream ~/data/browsing/*.sqlite | jq '"\(.url)|\(.metadata.description)"' | awk '!seen[$0]++' | fzfMerged files likehistory.jsoncan also be used as inputs files themselves, this reads those by mapping the JSON onto theVisitschema directly.In addition to.jsonfiles, this can parse.jsonl(JSON lines) files, which are files which contain newline delimited JSON objects. This allows you to parse JSON objects one at a time, instead of loading the entire file into memory. The.jsonlfile can be generated with the--streamflag:browserexport merge --stream --json ~/data/browsing/*.sqlite > ./history.jsonlAdditionally, this can parse compressed JSON/JSONL files (usingkompress):.xz,.zip,.lz4,.zstd,.zst,.tar.gz,.gzFor example, you could do:browserexportmerge--stream--json~/data/browsing/*.sqlite|gzip--best>./history.jsonl.gz# test parsing the compressed filebrowserexport--debuginspect./history.jsonl.gzIf you don't care about keeping the raw databases for any other auxiliary info like form, bookmark data, orfrom_visitinfo and just want the URL, visit date and metadata, you could usemergeto periodically merge the bulky.sqlitefiles into a gzipped JSONL dump to reduce storage space, and improve parsing speed:# backup databasesrsync-Pavh~/data/browsing~/.cache/browsing# merge all sqlite databases into a single compressed, jsonl filebrowserexport--debugmerge--json--stream~/data/browsing/*>'/tmp/browsing.jsonl'gzip'/tmp/browsing.jsonl'# test reading gzipped filebrowserexport--debuginspect'/tmp/browsing.jsonl.gz'# remove all old datafilesrm~/data/browsing/*# move merged data to database directorymv/tmp/browsing.jsonl.gz~/data/browsingI do this every couple months with a scripthere, and then sync my old databases to a harddrive for more long-term storageShell CompletionThis usesclick, which supportsshell completionforbash,zshandfish. To generate the completion on startup, put one of the following in your shell init file (.bashrc/.zshrcetc)eval"$(_BROWSEREXPORT_COMPLETE=bash_sourcebrowserexport)"# basheval"$(_BROWSEREXPORT_COMPLETE=zsh_sourcebrowserexport)"# zsh_BROWSEREXPORT_COMPLETE=fish_sourcebrowserexport|source# fishInstead ofevaling, you could of course save the generated completion to a file and/or lazy load it in your shell config, seebash completion docs,zsh functions,fish completion docs. For example forzshthat might look like:mkdir-p~/.config/zsh/functions/_BROWSEREXPORT_COMPLETE=zsh_sourcebrowserexport>~/.config/zsh/functions/_browserexport# in your ~/.zshrc# update fpath to include the directory you saved the completion file tofpath=(~/.config/zsh/functions$fpath)autoload-Uzcompinit&&compinitHPIIf you want to cache the merged results, this has amodule in HPIwhich handles locating/caching and querying the results. Seesetupandmodule setup.That usescachewto automatically cache the merged results, recomputing whenever you backup new databasesAs a few examples:✅OK:my.browser.all
✅-stats:{'history':{'count':1091091,'last':datetime.datetime(2023,2,11,1,12,37,302883,tzinfo=datetime.timezone.utc)}}✅OK:my.browser.export
✅-stats:{'history':{'count':1090850,'last':datetime.datetime(2023,2,11,4,34,12,985488,tzinfo=datetime.timezone.utc)}}✅OK:my.browser.active_browser
✅-stats:{'history':{'count':270363,'last':datetime.datetime(2023,2,11,22,26,24,887722,tzinfo=datetime.timezone.utc)}}# supports arbitrary queries, e.g. how many visits did I have in January 2020?$hpiquerymy.browser.all--order-typedatetime--after'2022-01-01 00:00:00'--before'2022-01-31 23:59:59'|jqlength50432# how many github URLs in the past month$hpiquerymy.browser.all--recent4w-s|jq.url|grep'github.com'-c16357Library UsageTo save databases:frombrowserexport.saveimportbackup_historybackup_history("firefox","~/data/backups")# or, pass a Browser implementationfrombrowserexport.browsers.allimportFirefoxbackup_history(Firefox,"~/data/backups")To merge/read visits from databases:frombrowserexport.mergeimportread_and_mergeread_and_merge(["/path/to/database","/path/to/second/database","..."])You can also usesqlite_backupto copy your current browser history into a sqlite connection in memory, as asqlite3.Connectionfrombrowserexport.browsers.allimportFirefoxfrombrowserexport.parseimportread_visitsfromsqlite_backupimportsqlite_backupdb_in_memory=sqlite_backup(Firefox.locate_database())visits=list(read_visits(db_in_memory))# to merge those with other saved filesfrombrowserexport.mergeimportmerge_visits,read_and_mergemerged=list(merge_visits([visits,read_and_merge(["/path/to/another/database.sqlite","..."]),]))If this doesn't support a browser and you wish to quickly extend without maintaining a fork (or contributing back to this repo), you can pass aBrowserimplementation (seebrowsers/all.pyandbrowsers/common.pyfor more info) tobrowserexport.parse.read_visitsor programmatically override/add your own browsers as part of thebrowserexport.browsersnamespace packageComparisons with PromnesiaA lot of the initial queries/ideas here were taken frompromnesiaand thebrowser_history.pyscript, but creating a package here allows its to be more extendible, e.g. allowing you to override/locate additional databases.TLDR on promnesia: lets you explore your browsing history in context: where you encountered it, in chat, on Twitter, on Reddit, or just in one of the text files on your computer. This is unlike most modern browsers, where you can only see when you visited the link.Sincepromnesia #375,browserexportis used inpromnesiain thebrowser.pyfile (to read any of the supported databases here from disk), seesetupandthe browser source quickstartin the instructions for moreContributingClone the repository and [optionally] create avirtual environmentto do your work in.gitclonehttps://github.com/seanbreckenridge/browserexportcd./browserexport# create a virtual environment to prevent possible package dependency conflictspython-mvirtualenv.venv# python3 -m pip install virtualenv if missingsource.venv/bin/activateDevelopmentTo install, run:python3-mpipinstall'.[testing]'If running in a virtual environment,pipwill automatically install dependencies into your virtual environment. If runningbrowserexporthappens to use the globally installedbrowserexportinstead, you can usepython3 -m browserexportto ensure its using the version in your virtual environment.After making changes to the code, reinstall by runningpip install ., and then test withbrowserexportorpython3 -m browserexportTestingWhile developing, you can run tests with:pytest
flake8./browserexport
mypy./browserexport# to autoformat codepython3-mpipinstallblack
findbrowserexporttests-name'*.py'-execpython3-mblack{}+