Package: databird
databirdPeriodically retrieve data from different sources.Thedatabirdpackage only provides a framework to plan and run the tasks needed to keep a local data-file-store up do date with various remote sources.
The remote sources can be anything (e.g. FTP Server, ECMWF, HTTP Api, SQL database, ...), as long as there is adatabird-driveravailable for the specific source.UsageDatabird is configured with configuration files and invoked by$ databird retrieve -c /etc/databird/databird.conf

# or (as the above is the default)
$ databird retrieveYou can store the configuration files anywhere and for example run the above command periodically as cron job.Also, some rq workers are required:$ rq worker databirdThis will start one worker. You should use a supervisor to start multiple workers.ConfigurationThe following example configuration defines a repository, which is populated with daily GNSS data fromftp://cddis.nasa.gov/gnss/data/daily/.The main configuration file (usuallydatabird.conf) could look like that:general:
  root: /data/repos # root path for data repositories
  num-workers: 16   # max number of async workers
  include: "databird.conf.d/*.conf"  # include config filesGenerally you can configure anything in any file, as all configuration files are merged to one configuration tree. Theincludeoption is an exception, as it can only be declared in the top config file.Then indatabird.conf.d/cddis.confyou can configure a profile and a repository:profiles:
  nasa_cddis:
    driver: standard.FtpDriver
    configuration:
      host: cddis.nasa.gov
      user: anonymous
      password: ""
      tls: False

repositories:
  nasa_gnss:
    description: Data from NASAs Archive of Space Geodesy Data
    profile: nasa_cddis
    period: 1 day
    delay: 2 days
    start: 2019-01-01
    targets:
      status: "{time:%Y}/cddis_gnss_{iso_date}.status"
    configuration:
      user: anonymous  # this could override 'user' from profile
      root: "/gnss/data/daily"
      patterns:
        status: "{time:%Y}/{time:%j}/{time:%y%j}.status"When calling databird with this configuration the following is achieved:A repository in the folder/data/repos/nasa_gnss/is createdFor every day, a file like2019/nasa_gnss_2019-01-20.statusis expectedIf that file is missing, retrieve it fromftp://cddis.nasa.gov/gnss/data/daily/2019/020/19020.statusIf there are many files missing, the data is retrieved asynchronouslyThis example used thestandard.FTPDriver.MonitoringUsedatabird webmonitor [PORT]to start the web interface.Since databird uses RQ for managing jobs, you also check the options atRQ/docs/monitoring.DriversAnyone can write drivers (see below). Currently, the following drivers are available:Included:standard.FilesystemDriver: Retrieve data from the local filesystemstandard.CommandDriver: Run an arbitrary shell commandstandard.FtpDriver: Retrieve data from an FTP serverClimate:climate.EcmwfDriver: Retrieve data from the European Centre for Medium-Range Weather Forecasts (ECMWF) via their APIclimate.C3SDriver: Retrieve data from the Copernicus Climate Change Service (C3S) via their APIclimate.GesDiscDriver: Retrieve data from the NASA EarthData GES DISC service.DevelopmentCreate a Python environment and activate it$python3-mvenv.&&sourcebin/activateInstall the development environment:(databird)$pipinstall-rrequirements-dev.txtWriting a new driverDrivers are published in a namespace packagedatabird-drivers. Everyone can develop drivers and share them.Installdatabirdand run mr.bob to create a new driver package:(databird) $ cd $HOME/projects
(databird) $ python -m mrbob.cli databird.blueprints:driverAfter answering some questions, a new directorydatabird-driver-<chosen_name>is created.
Lets asume<chosen_name> = foo, then your driver is usually implemented indatabird/drivers/foo/foo.pyin a class namedFooDriver().
Until more documentation is available, you have to look at the code to figure out how to write a driver.Other people will be able to use it withdriver: foo.FooDriver.Tell me if you wrote a new driver, so I can include it in the list.