Package: udata-hydra
udata-hydra 🦀udata-hydrais an async metadata crawler fordata.gouv.fr.URLs are crawled viaaiohttp, catalog and crawled metadata are stored in aPostgreSQLdatabase.CLICreate database structureInstall udata-hydra dependencies and cli.poetry installpoetry run udata-hydra migrateLoad (UPSERT) latest catalog version from data.gouv.frudata-hydra load-catalogCrawlerudata-hydra-crawlIt will crawl (forever) the catalog according to config set inconfig.py.BATCH_SIZEURLs are queued at each loop run.The crawler will start with URLs never checked and then proceed with URLs crawled beforeSINCEinterval. It will then wait until something changes (catalog or time).There's a by-domain backoff mecanism. The crawler will wait when, for a given domain in a given batch,BACKOFF_NB_REQis exceeded in a period ofBACKOFF_PERIODseconds. It will retry until the backoff is lifted.If an URL matches one of theEXCLUDED_PATTERNS, it will never be checked.WorkerA job queuing system is used to process long-running tasks. Launch the worker with the following command:poetry run rq worker -c udata_hydra.workerAPIRunpoetry install
poetry run adev runserver udata_hydra/app.pyGet latest checkWorks with?url={url}and?resource_id={resource_id}.$ curl -s "http://localhost:8000/api/checks/latest/?url=http://opendata-sig.saintdenis.re/datasets/661e19974bcc48849bbff7c9637c5c28_1.csv" | json_pp
{
   "status" : 200,
   "catalog_id" : 64148,
   "deleted" : false,
   "error" : null,
   "created_at" : "2021-02-06T12:19:08.203055",
   "response_time" : 0.830198049545288,
   "url" : "http://opendata-sig.saintdenis.re/datasets/661e19974bcc48849bbff7c9637c5c28_1.csv",
   "domain" : "opendata-sig.saintdenis.re",
   "timeout" : false,
   "id" : 114750,
   "dataset_id" : "5c34944606e3e73d4a551889",
   "resource_id" : "b3678c59-5b35-43ad-9379-fce29e5b56fe",
   "headers" : {
      "content-disposition" : "attachment; filename=\"xn--Dlimitation_des_cantons-bcc.csv\"",
      "server" : "openresty",
      "x-amz-meta-cachetime" : "191",
      "last-modified" : "Wed, 29 Apr 2020 02:19:04 GMT",
      "content-encoding" : "gzip",
      "content-type" : "text/csv",
      "cache-control" : "must-revalidate",
      "etag" : "\"20415964703d9ccc4815d7126aa3a6d8\"",
      "content-length" : "207",
      "date" : "Sat, 06 Feb 2021 12:19:08 GMT",
      "x-amz-meta-contentlastmodified" : "2018-11-19T09:38:28.490Z",
      "connection" : "keep-alive",
      "vary" : "Accept-Encoding"
   }
}Get all checks for an URL or resourceWorks with?url={url}and?resource_id={resource_id}.$ curl -s "http://localhost:8000/api/checks/all/?url=http://www.drees.sante.gouv.fr/IMG/xls/er864.xls" | json_pp
[
   {
      "domain" : "www.drees.sante.gouv.fr",
      "dataset_id" : "53d6eadba3a72954d9dd62f5",
      "timeout" : false,
      "deleted" : false,
      "response_time" : null,
      "error" : "Cannot connect to host www.drees.sante.gouv.fr:443 ssl:True [SSLCertVerificationError: (1, \"[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'www.drees.sante.gouv.fr'. (_ssl.c:1122)\")]",
      "catalog_id" : 232112,
      "url" : "http://www.drees.sante.gouv.fr/IMG/xls/er864.xls",
      "headers" : {},
      "id" : 165107,
      "created_at" : "2021-02-06T14:32:47.675854",
      "resource_id" : "93dfd449-9d26-4bb0-a6a9-ee49b1b8a4d7",
      "status" : null
   },
   {
      "timeout" : false,
      "deleted" : false,
      "response_time" : null,
      "error" : "Cannot connect to host www.drees.sante.gouv.fr:443 ssl:True [SSLCertVerificationError: (1, \"[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'www.drees.sante.gouv.fr'. (_ssl.c:1122)\")]",
      "domain" : "www.drees.sante.gouv.fr",
      "dataset_id" : "53d6eadba3a72954d9dd62f5",
      "created_at" : "2020-12-24T17:06:58.158125",
      "resource_id" : "93dfd449-9d26-4bb0-a6a9-ee49b1b8a4d7",
      "status" : null,
      "catalog_id" : 232112,
      "url" : "http://www.drees.sante.gouv.fr/IMG/xls/er864.xls",
      "headers" : {},
      "id" : 65092
   }
]Get crawling status$ curl -s "http://localhost:8000/api/status/crawler/" | json_pp
{
   "fresh_checks_percentage" : 0.4,
   "pending_checks" : 142153,
   "total" : 142687,
   "fresh_checks" : 534,
   "checks_percentage" : 0.4
}Get worker status$ curl -s "http://localhost:8000/api/status/worker/" | json_pp
{
   "queued" : {
      "default" : 0,
      "high" : 825,
      "low" : 655
   }
}Get crawling stats$ curl -s "http://localhost:8000/api/stats/" | json_pp
{
   "status" : [
      {
         "count" : 525,
         "percentage" : 98.3,
         "label" : "ok"
      },
      {
         "label" : "error",
         "percentage" : 1.3,
         "count" : 7
      },
      {
         "label" : "timeout",
         "percentage" : 0.4,
         "count" : 2
      }
   ],
   "status_codes" : [
      {
         "code" : 200,
         "count" : 413,
         "percentage" : 78.7
      },
      {
         "code" : 501,
         "percentage" : 12.4,
         "count" : 65
      },
      {
         "percentage" : 6.1,
         "count" : 32,
         "code" : 404
      },
      {
         "code" : 500,
         "percentage" : 2.7,
         "count" : 14
      },
      {
         "code" : 502,
         "count" : 1,
         "percentage" : 0.2
      }
   ]
}Using Webhook integration** Set the config values**Create aconfig.tomlwhere your service and commands are launched, or specify a path to a TOML file via theHYDRA_SETTINGSenvironment variable.config.tomlor equivalent will override values fromudata_hydra/config_default.toml, lookup there for values that can/need to be defined.UDATA_URI="https://dev.local:7000/api/2"UDATA_URI_API_KEY="example.api.key"SENTRY_DSN="https://{my-sentry-dsn}"The webhook integration sends HTTP messages toudatawhen resources are analyzed or checked to fill resources extras.Regarding analysis, there is a phase called "change detection". It will try to guess if a resource has been modified based on different criterions:harvest modified date in catalogcontent-length and last-modified headerschecksum comparison over timeThe payload should look something like:{"analysis:filesize":91661,"analysis:mime-type":"application/zip","analysis:checksum":"bef1de04601dedaf2d127418759b16915ba083be","analysis:last-modified-at":"2022-11-27T23:00:54.762000","analysis:last-modified-detection":"harvest-resource-metadata",}Developmentdocker-composeMultiple docker-compose files are provided:a minimaldocker-compose.ymlwith PostgreSQLdocker-compose.broker.ymladds a Redis brokerdocker-compose.test.ymllaunches a test DB, needed to run testsNB: you can launch compose from multiple files like this:docker-compose -f docker-compose.yml -f docker-compose.test.yml upLogging & DebuggingThe log level can be adjusted using the environment variable LOG_LEVEL.
For example, to set the log level toDEBUGwhen initializing the database, useLOG_LEVEL="DEBUG" udata-hydra init_db.Writing a migrationAdd a file namedmigrations/{YYYYMMDD}_{from}_up_{to}.sqland write the SQL you need to perform migration.fromshould be the revision from before (egrev1),tothe revision you're aiming at (egrev2)Modify the latest revision (egrev2) inmigrations/_LATEST_REVISIONudata-hydra migratewill use the info from_LATEST_REVISIONto upgrade torev2. You can also specifyudata-hydra migrate --revision rev2Deployment3 services need to be deployed for the full stack to run:workerapi / appcrawlerRefer to each section to learn how to launch them. The only differences from dev to prod are:useHYDRA_SETTINGSenv var to point to your customconfig.tomluseHYDRA_APP_SOCKET_PATHto configure where aiohttp should listen to areverse proxy connection (eg nginx)and useudata-hydra-appto launch the app server