Package: datasette-scraper
datasette-scraperdatasette-scraperis a Datasette plugin to manage small-ish (~100K pages) crawl and extract jobs.Opinionated yet extensibleSome useful tasks are possible out-of-the-box, or write your own pluggy hooks to go furtherLeans heavily into SQLiteIntrospect your crawls via ops tables exposed in DatasetteBuilt on robust librariesDatasetteas a hostselectolaxfor HTML parsinghttpxfor HTTP requestspluggyfor extensibilityzstandardfor efficiently compressing HTTP responsesNot for adversarial crawling. Want to crawl a site that blocks bots? You're on your own.InstallationInstall this plugin in the same environment as Datasette.datasette install datasette-scraperUsageConfiguredatasette-scraperviametadata.json. You need to enable the plugin
on a per-database level.To enable it in themy-databasedatabase, write something like this:{
  "databases": {
    "my-database": {
      "plugins": {
        "datasette-scraper": {
        }
      }
    }
  }
}The next time you start datasette, the plugin will create several tables in
the specified database. Go to thedss_crawltable to define a crawl.A 10-minute end-to-end walkthrough video is available:Usage notesdatasette-scraperrequires a database in which to track its operational data,
and a database in which to store scraped data. They can be the same database.Both databases will be put into WAL mode.The ops database'suser_versionpragma will be used to track schema versions.Architecturedatasette-scraperhandles the core bookkeeping for scraping--keeping track of
URLs to be scraped, rate-limiting requests to origins, persisting data into the DB.
It relies on plugins to do almost all the interesting work. For example, fetching
the actual pages, following redirects, navigating sitemaps, extracting data.The tool comes with plugins for common use cases. Some users may want to author
their ownafter_fetch_urlorextract_from_responseimplementations to do custom
processing.Overviewflowchart LR
direction TB

subgraph init
  A(user starts crawl) --> B[get_seed_urls]
end

subgraph crawl [for each URL to crawl]
  before_fetch_url --> fetch_cached_url --> fetch_url --> after_fetch_url
  fetch_cached_url --> after_fetch_url
end

subgraph discover [for each URL crawled]
  discover_urls --> canonicalize_url --> canonicalize_url
  canonicalize_url --> x[queue URL to crawl]
  extract_from_response
end

init --> crawl --> discoverPlugin hooksMost plugins will only implement a few of these hooks.connis a read/writesqlite3.Connectionto the databaseconfigis the crawl's configget_seed_urls(config)Returns a list of strings representing seed URLs to be fetched.They will be considered to have depth of 0, i.e. seeds.before_fetch_url(conn, config, job_id, url, depth, request_headers)request_headersis a dict, you can modify it to control what gets sent in the request.Returns:truthy to indicate this URL should not be crawled (for example, crawl max page limit)falsy to express no opinionNotebefore_fetch_urlvscanonicalize_urlYou can also use thecanonicalize_urlhook to reject URLs prior to them entering
the crawl queue.A URL rejected bycanonicalize_urlwill not result in an entry in thedss_crawl_queueanddss_crawl_queue_historytables.Which one you use is a matter of taste, in general, if youneverwant the URL,
reject it at canonicalization time.fetch_cached_url(conn, config, url, depth, request_headers)Fetch a previously-cached HTTP response. The system will not have checked that
there was rate limit available before calling this.Returns:None, to indicate not handleda response object, which is a dict with:fetched_at- an ISO 8601 time like2022-12-26 01:23:45.00headers- the response headers, eg[['content-type', 'text/html']]status_code- the respones code, eg200text- the response bodyOnce any plugin has returned a truthy value, no other plugin'sfetch_urlhook will be invoked.fetch_url(conn, config, url, request_headers)Fetch an HTTP response from the live server. The system will have checked that there
was rate limit available before calling this.Same return type and behaviour asfetch_cached_url.after_fetch_url(conn, config, url, request_headers, response, fresh, fetch_duration)Do something with a fetched URL.discover_urls(config, url, response)Returns a list of URLs to crawl.The URLs can be either strings, in which case they'll get enqueued as depth + 1, or tuple of URL and depth. This can be useful for paginated index pages, where you'd like to crawl to a max depth of, say, 2, but treat all the index pages as being at depth 1.canonicalize_url(config, from_url, to_url, to_url_depth)Returns:Falseto filter URLan URL to be crawled insteadNoneorTrueto no-opThe URL to be crawled can be a string, or a tuple of string and depth.This hook is useful for:blocking URLs that we never wantcanonicalizing URLs, for example, by omitting query parametersrestricting crawls to same originresetting depth for paginationextract_from_response(config, url, response)Returns an object of rows-to-be-inserted-or-upserted:{
  "dbname": {  // can be omitted, in which case, current DB will be used
    "users": [
      {
        "id!": "cldellow@gmail.com",  // ! indicates pkey, compound OK
        "name": "Colin",
      },
      {
        "id!": "santa@northpole.com",
        "name": "Santa Claus",
      }
    ],
    "places": [
      {
        "id@": "santa@northpole.com",
        "__delete": true
      },
      {
        "id@": "cldellow@gmail.com",
        "city": "Kitchener",
      },
      {
        "id@": "cldellow@gmail.com",
        "city": "Dawson Creek"
      }
    ]
  }
}Column names can have sigils at the end:!says the column is part of the pkey; there can be at most 1 row with this value@says the column should be indexed; there can be multiple rows with this valueColumns with sigils must be known at table creation time. Although you can have
multiple columns with sigils, you cannot mix!and@sigils in the same table.Any missing tables or columns will be created. Columns will haveANYdata type.
Columns will be nullable unless they have the!sigil.You can indicate that a row should be deleted by emitting__deletekey in your object.datasette-scrapermay commit your changes to the database in batches in order to
reduce write transactions and improve throughput. It may also elide
DELETE/INSERT statements entirely if it determines that the state of the database
would be unchanged.If you'd like to control the schema more carefully, please create the table manually.Metadata hooksThese hooks don't affect operation of the scrapes. They provide metadata to
help validate a user's configuration and show UI to configure a crawl.config_schema()Returns aConfigSchemaoption that defines how this plugin is configured.Configuration is done viaJSON schema. UI is done viaJSON Forms.Look at the existing plugins to learn how to use this hook.The schema is optional; if omitted, you will need to configure the plug in
out of band.config_default_value()ReturnsNoneto indicate that new crawls should not use this plugin by default.Otherwise, returns a reasonable default value that conforms to the schema inconfig_schema()DevelopmentTo set up this plugin locally, first checkout the code. Then create a new virtual environment:cd datasette-scraper
python3 -m venv venv
source venv/bin/activateNow install the dependencies and test dependencies:pip install -e '.[test]'To run the tests:pytest