Package: oecd-toolbox
OECD ToolboxA library of abstract classes that can serve as a skeleton for writing downloaders and converters for DbNomics style fetchers.
Additionally it contains utility/helper functions to handle common operations or transformations in both the downloading and conversion process.InstallationTo install the package - proceed the usual way:pipinstalloecd_toolboxor if you have it installed already, upgrade:pipinstalloecd_toolbox--upgradeBuild the projectTo build the project, after changes, make sure the version number insetup.cfgis updated.
Then issue the following command:python-mbuildPublish the project on pypi.orgWARNING!!! make sure no confidential data is stored in the published packageThe package is published on pypi.org. Manually manage the available variants of the packagehere.
Access details are stored in the Practices teams access details store.To push the distributions that are available in the toolbox'sdistfolder use twine with the commandline:twineuploaddist/*DataCapture conversionsNEW in version 0.3.27 of the toolbox - the package includes a converter to generate DataCapture style csvs. To use this new converter copy into your fetcher project the 'tests\datacapture-postprocessor.py' file or just its contents (below).importasyncioimportsysfromoecd_toolboximportcsv_writersaslbcdefmain():cnvtr=lbc.DataCaptureConverter()cnvtr.init_arguments_and_logging()asyncio.run(cnvtr.convert_resources(cnvtr.prepare_resources(),cnvtr.process_single_resource))if__name__=="__main__":sys.exit(main())In order to run the conversion from jsonl files use the following powershell command assuming that you have already created a folder <projectname-datacapture-data> to recieve your csv files:pythondatacapture-postprocessor.py<path-to\projectname-json-data><path-to\projectname-datacapture-data>--forceA similar command could be added to the postprocessor in the continouous integration pipeline. The usual behaviour modifiers [--only --except --limit] are available.NEW in version 0.4.0 beyond the basic converter two new converter flavours are available:DataCaptureConverterWithRegexcan filter series from a resource, so that the conversion is lighter both in used resources and resulting file (filtering is based on an SDMX webservice like syntax - dots for dimension separators, '+' to connect eligible dimension members, empty position allows all members) The above sample code adapted:defmain():filterset=[('sts_trtu_m','.TOVT+TOVV.G46+G47..I15.'),('prc_hicp_ctrb','.I15+INX_A_AVG+RCH_A+RCH_A_AVG+RCH_M..CP00+CP01+CP02+CP03+CP04+CP041+CP043+CP044+CP045')]cnvtr=lbc.DataCaptureConverterWithRegex()cnvtr.init_arguments_and_logging()asyncio.run(cnvtr.convert_resources(cnvtr.prepare_resources(filterset),cnvtr.process_single_resource))DataCaptureConverterWithRegexAndAggregatorcan handle frequency aggregations, it is sufficient to provide a target frequency from the list ['M','Q','A'] and an aggregator function (pandas-style)defmain():filterset=[("MMSD008A",None,"M",pd.DataFrame.mean),("MMSD402A",None,"A",pd.DataFrame.median)]cnvtr=lbc.DataCaptureConverterWithRegexAndAggregator()cnvtr.init_arguments_and_logging()asyncio.run(cnvtr.convert_resources(cnvtr.prepare_resources(filterset),cnvtr.process_single_resource))Common behaviour modifiersAll fetcher components that adopt the toolbox inherit some behaviour modifiers from the underlying toolbox. These command line arguments can be used to modify how the programs iterate throughresources:removing the excluded ones if the--excludeoption is used; provide a space separated list of resource IDskeeping only some of them if the--onlyoption is used; provide a space separated list of resource IDsprocessing a limited number of resources if the--limitoption is used; provide an integer after  the argumentBy default resources that were already processed with aSUCCESSorFAILUREstatus will not be processed again.
If the option--retry-failedis used, resources with FAILURE status will be retried.
If the option--forceis used, process all resources. !!! This is often needed if the status log is not cleared after each execution.The basic behaviour of the iterator will callprocess_resource(resource), logging messages, allowing
to track the processing progress. If an exception is raised during the execution ofprocess_resource:log the error and process the next resource, or re-raise if--fail-fastoption is usedcallresource.delete()if--delete-on-erroroption is used