Package: dogsheep-beta
dogsheep-betaBuild a search index across content from multiple SQLite database tables and run faceted searches against it using DatasetteExampleA live example of this plugin is running athttps://datasette.io/-/beta- configured usingthis YAML file.Read more about how this example works inBuilding a search engine for datasette.io.InstallationInstall this tool like so:$ pip install dogsheep-betaUsageRun the indexer using thedogsheep-betacommand-line tool:$ dogsheep-beta index dogsheep.db config.ymlTheconfig.ymlfile contains details of the databases and document types that should be indexed:twitter.db:tweets:sql:|-selecttweets.id as key,'Tweet by @' || users.screen_name as title,tweets.created_at as timestamp,tweets.full_text as search_1from tweets join users on tweets.user = users.idusers:sql:|-selectid as key,name || ' @' || screen_name as title,created_at as timestamp,description as search_1from usersThis will create asearch_indextable in thedogsheep.dbdatabase populated by data from those SQL queries.By default the search index that this tool creates will be configured for Porter stemming. This means that searches for words likerunwill match documents containingrunsorrunning.If you don't want to use Porter stemming, use the--tokenize noneoption:$ dogsheep-beta index dogsheep.db config.yml --tokenize noneYou can pass other SQLite tokenize argumenst here, seethe SQLite FTS tokenizers documentation.ColumnsThe columns that can be returned by our query are:key- a unique (within that type) primary keytitle- the title for the itemtimestamp- an ISO8601 timestamp, e.g.2020-09-02T21:00:21search_1- a larger chunk of text to be included in the search indexcategory- an integer category ID, see belowis_public- an integer (0 or 1, defaults to 0 if not set) specifying if this is public or notPublic records are things like your public tweets, blog posts and GitHub commits.CategoriesIndexed items can be assigned a category. Categories are integers that correspond to records in thecategoriestable, which defaults to containing the following:idname1created2saved3receivedcreatedis for items that have been created by the Dogsheep instance owner.savedis for items that they have saved, liked or favourited.receivedis for items that have been specifically sent to them by other people - incoming emails or direct messages for example.Datasette pluginRundatasette install dogsheep-beta(or usepip install dogsheep-betain the same environment as Datasette) to install the Dogsheep Beta Datasette plugin.Once installed, a custom search interface will be made available at/-/beta. You can use this interface to execute searches.The Datasette plugin has some configuration options. You can set these by adding the following to yourmetadata.jsonconfiguration file:{"plugins":{"dogsheep-beta":{"database":"beta","config_file":"dogsheep-beta.yml","template_debug":true}}}The configuration settings for the plugin are:database- the database file that contains your search index. If the file isbeta.dbyou should setdatabasetobeta.config_file- the YAML file containing your Dogsheep Beta configuration.template_debug- set this totrueto enable debugging output if errors occur in your custom templates, see below.Custom results displayEach indexed item type can define custom display HTML as part of theconfig.ymlfile. It can do this using adisplaykey containing a fragment of Jinja template, and optionally adisplay_sqlkey with extra SQL to execute to fetch the data to display.Here's how to define a custom display template for a tweet:twitter.db:tweets:sql:|-selecttweets.id as key,'Tweet by @' || users.screen_name as title,tweets.created_at as timestamp,tweets.full_text as search_1from tweets join users on tweets.user = users.iddisplay:|-<p>{{ title }} - tweeted at {{ timestamp }}</p><blockquote>{{ search_1 }}</blockquote>This example reuses the value that were stored in thesearch_indextable when the indexing query was run.To load in extra values to display in the template, use adisplay_sqlquery like this:twitter.db:tweets:sql:|-selecttweets.id as key,'Tweet by @' || users.screen_name as title,tweets.created_at as timestamp,tweets.full_text as search_1from tweets join users on tweets.user = users.iddisplay_sql:|-selectusers.screen_name,tweets.full_text,tweets.created_atfromtweets join users on tweets.user = users.idwheretweets.id = :keydisplay:|-<p>{{ display.screen_name }} - tweeted at {{ display.created_at }}</p><blockquote>{{ display.full_text }}</blockquote>Thedisplay_sqlquery will be executed for every search result, passing the key value from thesearch_indextable as the:keyparameter and the user's search term as the:qparameter.This performs well becausemany small queries are efficient in SQLite.If an error occurs while rendering one of your templates the search results page will return a 500 error. You can use thetemplate_debugconfiguration setting described above to instead output debugging information for the search results item that experienced the error.Displaying mapsThis plugin will eventually include a number of useful shortcuts for rendering interesting content.The first available shortcut is for displaying maps. Make your custom content output something like this:<divdata-map-latitude="{{ display.latitude }}"data-map-longitude="{{ display.longitude }}"style="display: none; float: right; width: 250px; height: 200px; background-color: #ccc;"></div>JavaScript on the page will look for any elements withdata-map-latitudeanddata-map-longitudeand, if it finds any, will load Leaflet and convert those elements into maps centered on that location. The default zoom level will be 12, or you can set adata-map-zoomattribute to customize this.DevelopmentTo set up this plugin locally, first checkout the code. Then create a new virtual environment:cd dogsheep-beta
python3 -mvenv venv
source venv/bin/activateOr if you are usingpipenv:pipenv shellNow install the dependencies and tests:pip install -e '.[test]'To run the tests:pytest