Package: ffexport
ffexportThis backs up firefox history and parses the resulting history (sqlite) files.Primary function here is to export/interact with my firefox history. Functionality for Chrome are vestigal and I've left them there in case someone wants to mess with it. I recommend you take a look atpromnesiaif you want immediate support for that.Seeherefor how firefox stores its history.Installpip3 install ffexportRequirespython3.6+UsageUsage: ffexport [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  inspect  Extracts history/site metadata from one sqlite database.
  merge    Extracts history/site metadata from multiple sqlite databases.
  save     Backs up the current firefox sqlite history file.Theinspectandmergecommands also accept a--jsonflag, which dumps the result to STDOUT as JSON. Dates are serialized to epoch time.Logs are hidden by default. To show the debug logs setexport FFEXPORT_LOGS=10(useslogging levels)saveUsage: ffexport save [OPTIONS]

  Backs up the current firefox sqlite history file.

Options:
  --browser [firefox|chrome]  Provide either 'firefox' or 'chrome' [defaults
                              to firefox]
  --profile TEXT              Use to pick the correct profile to back up. If
                              unspecified, will assume a single profile
  --to PATH                   Directory to store backup to  [required]Since firefox (and browsers in general) seem to remove old history seemingly randomly, I'd recommend running the following periodically:$ ffexport save --to ~/data/firefox
[D 200828 15:30:58 save_hist:67] backing up /home/sean/.mozilla/firefox/jfkdfwx.dev-edition-default/places.sqlite to /home/sean/data/firefox/places-20200828223058.sqlite
[D 200828 15:30:58 save_hist:71] done!That atomically copies the firefox sqlite database which contains your history--tosome backup directory.inspectUsage: ffexport inspect [OPTIONS] SQLITE_DB

  Extracts history/site metadata from one sqlite database.

  Provide a firefox history sqlite databases as the first argument. Drops
  you into a REPL to access the data.

Options:
  --json  Print result to STDOUT as JSONAs an example:ffexportinspect~/data/firefox/places-20200828223058.sqlite[D21022504:38:27parse_db:77]Readingindividualvisitsfrom/home/sean/data/firefox/places-20200828223058.sqlite...[D21022504:38:28parse_db:96]Readingsitedatafrom/home/sean/data/firefox/places-20200828223058.sqlite...Demo:Yourmostcommonsites....[('github.com',13775),('www.youtube.com',8108),('duckduckgo.com',8054),('www.google.com',6542),('discord.com',6134),('sean.fish',3264),('stackoverflow.com',2528),('gitlab.com',1608)]Usemvisormsitetoaccessrawvisits/sitedata,visforthemergeddataIn[1]:....That drops you into a REPL with access to the history from that database (visandmvis/msite)mergeSimilar toinspect, but accepts multiple database backups, removing any duplicates and dropping you into a REPLUsage: ffexport merge [OPTIONS] SQLITE_DB...

  Extracts history/site metadata from multiple sqlite databases.

  Provide multiple sqlite databases as positional arguments, e.g.: ffexport
  merge ~/data/firefox/dbs/*.sqlite

  Provides a similar interface to inspect; drops you into a REPL to access
  the data.

Options:
  --include-live              In addition to any provided databases, copy
                              current (firefox) history to /tmp and merge it
                              as well
  --json                      Print result to STDOUT as JSON(also accepts the--browserand--profileflags like thesavecommand, provide those if you have multiple profiles and are using the--include-liveflag.Example:ffexportmerge--include-live~/data/firefox/*.sqlite[D21022504:39:43save_hist:71]backingup/home/sean/.mozilla/firefox/lsinsptf.dev-edition-default/places.sqliteto/tmp/tmpv2ct3t0g/places-20210225123943.sqlite[D21022504:39:43save_hist:75]done![D21022504:39:43merge_db:48]merginginformationfrom10databases...[D21022504:39:43parse_db:142]Readingvisitsfrom/home/sean/data/firefox/places-20200828223058.sqlite...[D21022504:39:44parse_db:142]Readingvisitsfrom/home/sean/data/firefox/places-20201010031025.sqlite...[D21022504:39:45parse_db:142]Readingvisitsfrom/home/sean/data/firefox/places-20201031031121.sqlite...[D21022504:39:46parse_db:142]Readingvisitsfrom/home/sean/data/firefox/places-20201121031143.sqlite...[D21022504:39:47parse_db:142]Readingvisitsfrom/home/sean/data/firefox/places-20201212031157.sqlite...[D21022504:39:49parse_db:142]Readingvisitsfrom/home/sean/data/firefox/places-20201226031225.sqlite...[D21022504:39:50parse_db:142]Readingvisitsfrom/home/sean/data/firefox/places-20210109031324.sqlite...[D21022504:39:52parse_db:142]Readingvisitsfrom/home/sean/data/firefox/places-20210208052811.sqlite...[D21022504:39:53parse_db:142]Readingvisitsfrom/home/sean/data/firefox/places-20210222065542.sqlite...[D21022504:39:55parse_db:142]Readingvisitsfrom/tmp/tmpv2ct3t0g/places-20210225123943.sqlite...[D21022504:39:57merge_db:60]Summary:removed1,597,629duplicates...[D21022504:39:57merge_db:61]Summary:returning235,681visitentries...Usemerged_vistoaccessmergeddatafromalldatabasesTo dump all that info to json:$ffexportmerge--include-live--json~/data/firefox/*.sqlite>./history.json
$du-hhistory.json
67Mhistory.jsonLibrary UsageCan also import and provide files from python elsewhere. The two major functions you'd use areffexport.read_visits(which reads/parses one database) andffexport.read_and_merge(which parses multiple).>>>importffexport,glob>>>visits=list(ffexport.read_and_merge(*glob.glob('data/firefox/*.sqlite')))# note the splat, read_and_merge accepts variadic arguments>>>visits[10000]Visit(url="https://github.com/python-mario/mario",visit_date=datetime.datetime(2020,6,24,2,23,32,482000,tzinfo=<UTC>),visit_type=1,title="python-mario/mario: Powerful Python pipelines for your shell",description="Powerful Python pipelines for your shell . Contribute to python-mario/mario development by creating an account on GitHub.",preview_image="https://repository-images.githubusercontent.com/185277224/2ce27080-b915-11e9-8abc-088ab263dbd9",)For another example, see myHPIintegration.NotesSeeherefor what thevisit_typeenum means.I considered usingcachewbut because of the volume of the data, it ends up being slower than reading directly from the sqlite database exports. Both thevisitsandsitedatafunctions arecachewcompliant though, you'd just have to wrap it yourself. Seeherefor more info.save_hist.py/initial structure is modified fromkarlicoss/promnesiaTestinggitclonehttps://github.com/seanbreckenridge/ffexportcd./ffexport
pipinstall'.[testing]'mypy./ffexport
pytest