Package: apibackuper
apibackuper is a command line tool to archive/backup API calls.
It’s goal to download all data behind REST API and to archive it to local storage.
This tool designed to backup API data, so simple as possible.Contents1History2Main features3Installation3.1Linux3.2Windows, etc.3.3Python version4Quickstart5Config options5.1settings5.2project5.3params5.4data5.5follow5.6files5.7storage6Usage6.1Examples7Advanced8History8.11.0.7 (2021-11-4)8.21.0.6 (2021-11-1)8.31.0.5 (2021-05-31)8.41.0.4 (2021-05-31)8.51.0.3 (2020-10-28)8.61.0.2 (2020-09-20)8.71.0.1 (2020-08-14)1HistoryThis tool was developed optimize backup/archival procedures for Russian government information from E-Budget portal budget.gov.ru and
some other government IT systems too. Examples of tool usage could be found in “examples” directory2Main featuresAny GET/POST iterative API supportedAllows to estimate time required to backup APIStores data inside ZIP containerSupports export of backup data as JSON lines fileDocumentationTest coverage3Installation3.1LinuxMost Linux distributions provide a package that can be installed using the
system package manager, for example:# Debian, Ubuntu, etc.$aptinstallapibackuper# Fedora$dnfinstallapibackuper# CentOS, RHEL, ...$yuminstallapibackuper# Arch Linux$pacman-Sapibackuper3.2Windows, etc.A universal installation method (that works on Windows, Mac OS X, Linux, вЂ¦,
and always provides the latest version) is to use pip:# Make sure we have an up-to-date version of pip and setuptools:$pipinstall--upgradepipsetuptools$pipinstall--upgradeapibackuper(Ifpipinstallation fails for some reason, you can tryeasy_install apibackuperas a fallback.)3.3Python versionPython version 3.6 or greater is required.4QuickstartThis example is about backup of Russian certificate authorities.
List of them published at e-trust.gosuslugi.ru and available via undocumented API.$apibackupercreateetrust$cdetrustEdit apibackuper.cfg as:[settings]initialized=Truename=etrust[project]description=E-TrustUClisturl=https://e-trust.gosuslugi.ru/app/scc/portal/api/v1/portal/ca/listhttp_mode=POSTwork_modes=full,incremental,updateiterate_by=page[params]page_size_param=recordsOnPagepage_size_limit=100page_number_param=page[data]total_number_key=totaldata_key=dataitem_key=РеестровыйНомерchange_key=СтатусАккредитации.ДействуетС[storage]storage_type=zipAdd file params.json with parameters used with POST requests{"page":1,"orderBy":"id","ascending":false,"recordsOnPage":100,"searchString":null,"cities":null,"software":null,"cryptToolClasses":null,"statuses":null}Execute command “estimate” to see how long data will be collected and how much space needed$apibackuperestimatefullOutput:Totalrecords:502Recordsperrequest:100Totalrequests:6Averagerecordsize32277.96bytesEstimatedsize(jsonlines)16.20MBAvgrequesttime,seconds66.9260Estimatedallrequeststime,seconds402.8947Execute command “run” to collect the data. Result stored in “storage.zip”$apibackuperrunfullExports data from storage and saves as jsonl file called “etrust.jsonl”$apibackuperexportjsonletrust.jsonl5Config optionsExample config file[settings]initialized=Truename=<name>splitter=.[project]description=<description>url=<url>http_mode=<GETorPOST>work_modes=<combinationoffull,incremental,update>iterate_by=<pageorskip>[params]page_size_param=<pagesizeparam>page_size_limit=<pagesizelimit>page_number_param=<pagenumber>count_skip_param=<keytoiterateinskipmode>[data]total_number_key=<totalnumberkey>data_key=<datakey>item_key=<itemkey>change_key=<changekey>[follow]follow_mode=<typeoffollowmode>follow_pattern=<urlprefixtofollowlinks>follow_data_key=<followdataitemkey>follow_param=<followparam>follow_item_key=<followitemkey>[files]fetch_mode=<filefetchmode>root_url=<filerooturl>keys=<keyswithfiledata>storage_mode=<filestoragemode>[storage]storage_type=zipcompression=True5.1settingsname - short name of the projectsplitter - value of field splitter. Needed for rare cases when ‘.’ is part of field name. For example for OData requests and‘@odata.count’ field5.2projectdescription - text that explains what for is this projecturl - API endpoint urlhttp_mode - one of HTTP modes: GET or POSTwork_modes - type of operations: full - archive everything, incremental - add new records only, update - collect changed data onlyiterate_by - type of iteration of records. By ‘page’ - default, page by page or by ‘skip’ if skip value provided5.3paramspage_size_param - parameter with page sizepage_size_limit - limit of records provided by APIpage_number_param = parameter with page numbercount_skip_param - parameter for ‘skip’ type of iteration5.4datatotal_number_key - key in data with total number of recordsdata_key - key in data with list of recordsitem_key - key in data with unique identifier of the record. Could be group of keys separated with commachange_key - key in data that indicates that record changed. Could be group of keys separated with comma5.5followfollow_mode - mode to follow objects. Could be ‘url’ or ‘item’. If mode is ‘url’ than follow_pattern not usedfollow_pattern - url pattern / url prefix for followed objects. Only for mode ‘item’’follow_data_key - if object/objects are inside array, key of this arrayfollow_param - parameter used in ‘item’ modefollow_item_key - item key5.6filesfetch_mode - file fetch mode. Could be ‘prefix’ or ‘id’. Prefixroot_url - root url / prefix  for fileskeys - list of keys with urls/file id’s to search for files to savestorage_mode - a way how files stored in storage/files.zip. By default ‘filepath’ and files storaged same way as they presented in url5.7storagestorage_type - type of local storage. ‘zip’ is local zip file is default onecompression - if True than compressed ZIP file used, less space used, more CPU time processing data6UsageSynopsis:$apibackuper[flags][command]inputfileSee alsoapibackuper--help.6.1ExamplesCreate project “budgettofk”:$apibackupercreatebudgettofkEstimate execution time for ‘budgettofk’ project. Should be called in project dir or project dir provided via -p parameter:$apibackuperestimatefull-pbudgettofkOutputTotalrecords:12282Recordsperrequest:500Totalrequests:25Averagerecordsize1293.60bytesEstimatedsize(jsonlines)15.89MBAvgrequesttime,seconds1.8015Estimatedallrequeststime,seconds46.0536Run project. Should be called in project dir or project dir provided via -p parameter$apibackuperrunfullExport data from project. Should be called in project dir or project dir provided via -p parameter$apibackuperexportjsonlhhemployers.jsonl-phhemployersFollows each object of downloaded data and does requests for each objects
.. code-block:: bash$ apibackuper follow continueDownloads all files associated with API objects
.. code-block:: bash$ apibackuper getfiles7AdvancedTBD8History8.11.0.7 (2021-11-4)Fixed “continue” mode. Now supports continue not only for “follow” command but for “run” command too. Use “apibackuper run continue” if it was stopped by error or user input.8.21.0.6 (2021-11-1)Added “default_delay”, ‘retry_delay’ and “retry_count” to manage error handlingIf get HTTP status 500 or 503 starts retrying latest request till HTTP status 200 or retry_count ends8.31.0.5 (2021-05-31)Minor fixes8.41.0.4 (2021-05-31)Added “start_page” in case if start_page is not 1 (could be 0 sometimes)Added support of data returned as JSON array, not JSON dict and data_key not providedAdded initial code to implement Frictionless Data packaging8.51.0.3 (2020-10-28)Added several new optionsAdded aria2 download support for files downloading8.61.0.2 (2020-09-20)Using permanent storage dir “storage” instead of temporary “temp” dirAdded logic to do requests to get addition info on retrieved objects, command “follow”Added logic to retrieve files linked with retrieved objects, command “getfiles”8.71.0.1 (2020-08-14)First public release on PyPI and updated github code