Package: htmls-to-datasette
htmls-to-datasetteHtmls-to-datasette is a tool to index HTML files into aSqlitedatabase so they can be searched and
visualized at a later time. This can be useful for web archival/web clipping purposes.The database created is designed to be served onDatasetteand to allow to read the indexed
files through it.This tool was created to serve my own work flow that is:Have a browser withSingleFileextension installed.When there is an interesting blog post or article save a full web page into one HTML using SingleFile.The created.htmlfile on the downloads folder is moved to a common repository (via cron job).This common repository is synched to my main server (I useSyncthingfor this).On my personal server all the new HTML files are moved to the serving folder and this indexer is called to populate
the search database.Datasette with an specific configuration will allow searching on these files and reading them online.The indexing tool can insert the HTML contents on the database itself, to be served from there, or not. In this second
case the files will be served from the location they were indexed from.SetupStandard installpipinstallhtmls-to-datasetteAnd you can start running the command, use--helpto see specific commands help.htmls-to-datasette--help
htmls-to-datasetteindex--helpDevelopment installThis project usesPoetryto make it easier to setup the appropriate dependencies to run.Installation steps forPoetrycan be checked on theirwebsitebut for
most of the cases this command line would work:curl-sSLhttps://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py|python-Note that you should exercise caution when running something directly from Internet.Install dependencies:poetryinstallRunYou can usepoetry runin front of htmls-to-datassette so it is using the virtual environment that you just created
before.poetryrunhtmls-to-datassette[options]Build an installable packagepoetrybuild# The resoult will be in dist directoryI usepipxfor installing packages on isolated environments. You can install this package
from thedist/directory in whichever way you prefer or you caninstall pipx.The installation with pipx would be similar to:pipxinstalldist/htmls-to-datasette-0.1.2.tar.gzUsagehtmls-to-datasette index [OPTIONS] [INPUT_DIRS]...will create a database named `htmlstore.db' (by default).ExampleGet into the server directory:cdserverBecause this example requires Datasette to run you would have to get them using poetry:poetryinitNow index the example file usinghtmls-to-datasette:htmls-to-datasetteindexinputAll files contained ininput(.htmland.html) will be indexed and a full text search index created. Whenever
there are new files to be indexed this command can be run in the same way.And now run the Datasette server:poetryrundatasetteservehtmlstore.db-mmetadata-files.json--plugins-dir=pluginsYou'll see the address to send your browser to on the screen. There is also a shortcut to make it easier to perform a
full text search. Should be reachable athttp://127.0.0.1:8001/htmlstore/searchjust fill the query on the 'q' parameter
and you will search over the indexed HTMLs. Click on the HTML file name will load its contents.For this to work the server will require the files to be on their location (relative in this case). So if theinputfolder is moved away or not accesible the files would be searchable but the contents will not be available.There is an additional example that stores these files onto the Sqlite database itself. This has its advantages as
everything needed for serving and searching the content will be contained in one file.# You should be on the server directoryrmhtmlstore.db# Remove the previous example's databasehtmls-to-datasetteindexinput--store-binary# Index files and store its contents# Now run Datasette, note that now we need to use a different metadata as the contents needed to be served# in a different way (from the DB itself).poetryrundatasetteservehtmlstore.db-mmetadata-binary.json--plugins-dir=pluginsTODOClear content when extracting files.Better documentation.