Package: ldp-archive-mirror
This tools allows you to get a local copy of all your cold stored archives on Logs Data Platform:[2018-12-14 17:20:00,200 INFO LDP Mirror] Create local DB if not exists
[2018-12-14 17:20:00,204 INFO LDP Mirror] Looking for new archives of stream e9397556-31d1-4d4d-b2bd-e5367b522cc8
[2018-12-14 17:20:05,190 INFO LDP Mirror] Service ldp-jb-52560 found for stream e9397556-31d1-4d4d-b2bd-e5367b522cc8
[2018-12-14 17:20:05,579 INFO LDP Mirror] Archive 9fb75957-2cde-435e-bdd9-6dfd33663f2c added to cache
[2018-12-14 17:20:05,850 INFO LDP Mirror] Archive 11d8630a-7b38-4fa4-9d7c-dfd17b0b00f6 added to cache
...
[2018-12-14 17:20:08,392 INFO LDP Mirror] Directory mirror/e9397556-31d1-4d4d-b2bd-e5367b522cc8 created
[2018-12-14 17:20:08,535 INFO LDP Mirror] Archive e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-10.zst missing, download scheduled
[2018-12-14 17:20:08,635 INFO LDP Mirror] Archive e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-11.zst missing, download scheduled
...
[2018-12-14 17:20:09,535 INFO LDP Mirror] Will retry to download e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-10.zst after 596 seconds
[2018-12-14 17:20:09,745 INFO LDP Mirror] Will retry to download e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-11.zst after 598 seconds
...
[2018-12-14 17:20:10,927 INFO LDP Mirror] Archive e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-10.zst downloaded
[2018-12-14 17:20:11,655 INFO LDP Mirror] Sha256 OK on e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-10.zst
[2018-12-14 17:20:10,927 INFO LDP Mirror] Archive e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-11.zst downloaded
[2018-12-14 17:20:11,655 INFO LDP Mirror] Sha256 OK on e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-11.zstQuickstartFirst, installldp-archive-mirrorusingpip:pip3 install -U ldp-archive-mirrorThen you can use the binaryldp-mirror:usage: ldp-mirror [-h] [--app-key KEY] [--app-secret SECRET]
              [--consumer-key KEY] [--ovh-region REGION] [--db DIR]
              [--mirror DIR] [--ldp-host HOST] [--ldp-token TOKEN]
              [--chunk-size CHUNK] [--gpg-passphrase SECRET]
              STREAM_ID [STREAM_ID ...]

LDP archive Mirror CLI - 0.2.0

positional arguments:
  STREAM_ID            LDP Stream UUIDs

optional arguments:
  -h, --help              show this help message and exit
  --app-key KEY           OVH application key (default: dcd57be8c9dc53ff)
  --app-secret SECRET     OVH application secret (default: d37f35c27e60be58746e81e3351a84db)
  --consumer-key SECRET   OVH consumer key (default: 819fb70c64f91f797daf0ed3990e5ff0)
  --ovh-region REGION     OVH region (default: ovh-eu)
  --db DIR                Where to place the local sqlite database (default: /data/db)
  --mirror DIR            Where to place your archives (default: /data/mirror)
  --ldp-host HOST         If set, push logs of the current application to given LDP hostname
  --ldp-token TOKEN       If set, push logs of the current application to associated LDP stream token
  --chunk-size CHUNK      Download chunk size in bytes (default: 16384)
  --gpg-passphrase SECRET PGP private key passphrase (default: None)Setup1. Create an applicationTo interact with the APIs, the application needs to identify itself using anapplication_keyand anapplication_secret. To get them, you need
to register your application. Depending the API you plan to use, visit:OVH EuropeOVH USOVH North-AmericaSo you Start EuropeSo you Start North AmericaKimsufi EuropeKimsufi North AmericaYou will get 2 keys,Application KeyandApplication Secret.On the restriction step, we invite you to set the following access rules:curl -XPOST -H"X-Ovh-Application: OVH_API_AK" -H "Content-type: application/json" \
https://eu.api.ovh.com/1.0/auth/credential  -d '{
    "accessRules": [
        {
            "method": "GET",
            "path": "/dbaas/logs"
        },
        {
            "method": "GET",
            "path": "/dbaas/logs/*/encryptionKey/*"
        },
        {
            "method": "GET",
            "path": "/dbaas/logs/*/output/graylog/stream"
        },
        {
            "method": "GET",
            "path": "/dbaas/logs/*/output/graylog/stream/*/archive*"
        },
        {
            "method": "POST",
            "path": "/dbaas/logs/*/output/graylog/stream/*/archive/*/url"
        },
        {
            "method": "GET",
            "path": "/dbaas/logs/*/output/graylog/stream/*/archive/*/encryptionKey"
        }
    ],
    "redirection":"https://www.mywebsite.com/"
}'ReplaceOVH_API_AKby yourApplication Key.In the curl response, you will have a validation URLvalidationUrland aConsumer KeyconsumerKey.
Please follow the linkvalidationUrland connect your OVH account (use unlimited lifespan).
Once the user has been authenticated, it will be automatically redirected to the URL you entered when the token was created
(https://www.mywebsite.com/in the previous example).Once created, you will obtain anapplication key (OVH_API_AK), anapplication
secret (OVH_API_AS)and aconsumer key (OVH_API_CK)2. Environment variablesDefault cli values can be set using environment:Cli parameterNameAbout–app-keyOVH_API_AKOVH application key–app-secretOVH_API_ASOVH application secret–consumer-keyOVH_API_CKOVH customer key–ovh-regionOVH_API_REGIONOVH api location. Default:ovh-eu–dbDB_DIRECTORYWhere to place the local sqlite database. Default:db–mirrorMIRROR_DIRECTORYWhere to place your archives. Default:mirror–ldp-hostLDP_HOSTIf set, push logs of the current application to given LDP hostname–ldp-tokenLDP_TOKENIf set, push logs of the current application to associated LDP stream token–chunk-sizeCHUNK_SIZEDownload chunk size in bytes–gpg-passphraseGPG_PASSPHRASEPGP private key passphrase3. LaunchOnce all the mandatory environment variables set, launch the process like this:$ ldp-mirror STREAM_ID [STREAM_ID ...]This will:looks for the LDP service associated with the givenSTREAM_ID(s)populate a local cache with all the archives found on the APIrequest for each of them a temporary download urldownload the files when unseal time is reacheddecrypt the archives if they are encryptedask every hour the api if a new archive is availableCreate docker image from sourcesAs this application is supposed to be kept alive indefinitely, launching it from a Docker daemon looks obvious.To build the image form the sources, uses the givenMakefile:$ git clone https://github.com/ovh/ldp-archive-mirror
$ cd ldp-archive-mirror
$ make build-dockerAnd to run it:$ mkdir -p ~/ldp-archive-mirror/db ~/ldp-archive-mirror/mirror
$ docker run -d \
    --name ldp-archive-mirror \
    --user $(id -u):$(id -g) \
    -v ${HOME}/ldp-archive-mirror/mirror/:/data/mirror \
    -v ${HOME}/ldp-archive-mirror/db:/data/db \
    -v ${HOME}/.gnupg:/data/gnupg \
    -e OVH_API_AK=MY_OVH_AK \
    -e OVH_API_AS=MY_OVH_AS \
    -e OVH_API_CK=MY_OVH_CK \
    -e GPG_PASSPHRASE=MY_GPG_PRIVATE_KEY_PASSPHRASE \
    ldp-archive-mirror --ldp-host graX.logs.ovh.com MY_LDP_STREAM_ID_1 MY_LDP_STREAM_ID_2ReplacegraX.logs.ovh.comby your LDP cluster address,MY_LDP_STREAM_ID_1/MY_LDP_STREAM_ID_2/… by your LDP stream id.RequirementsPython >= 3.6Project LinksPyPI:https://pypi.python.org/pypi/ldp-archive-mirrorIssues:https://github.com/ovh/ldp-archive-mirror/issuesLicenseOVH SAS