Package: datapackage-pipelines-fiscal
datapackage-pipelines-fiscalExtension for datapackage-pipelines used for loading Fiscal Data Packages into:S3 (or compatible) storage, in a denormalized forma database in normalized form.Metadata will be stored in an elasticsearch instance (if available) viaos-package-registryAbabbagemodel will also be generated and written to the datapackage for querying the database using its APIThis extension works with a custom source spec and a set of processors. The generator will convert the source spec into a set of inter-dependent pipelines, which when run in order will perform data processing and loading to selected endpoints (based on environment variables).This extension is used byos-conductorandos-data-importers.Environment variablesDPP_DB_ENGINE- connection string for an SQL database to dump data intoELASTICSEARCH_ADDRESS[OPTIONAL] - connection string for an elasticsearch instance (used for package registry updating)S3_BUCKET_NAME[OPTIONAL] - S3 bucket for uploading data. If not provided, local ZIP files will be created instead.AWS_ACCESS_KEY_ID- S3 credentials (required if S3 bucket was specified)AWS_SECRET_ACCESS_KEY- S3 credentials (required if S3 bucket was specified)DependenciesIn order to fully run the fiscal datapackage flow you need to haveos-typesinstalled, using npm:$ npm install -g os-typesThis external node.js utility is used to perform fiscal modelling for the processed datapackage.fiscal.source-spec.yamlEach source-spec contains information regarding a single Fiscal Data Package.Top level properties are:titleTitle, or Display name, of the data packagedataset-name[OPTIONAL]A slug to be used as the data package's name.If not provided, a slugified version of the title will be used.resource-name[OPTIONAL]A slug to be used as the main resource's name in the final data package.If not provided, the dataset name will be used.owner-idThe id of the owner of this datapackage.This identifier is used to generate various paths and storage names.sourcesContains a non-empty list of data sources for the fiscal data package.Each data source has these properties:url: The location of the dataname: [OPTIONAL] A name for this source (will later be used as an intermediate resource name)Othertabulatorparameters can also be added as properties here, e.g.sheet,encoding,compressionetc.fieldsContains a non-empty list of fields for the fiscal data package.Each field definition has these properties:header: Thenameof the field in the resulting resourcetitle[OPTIONAL]: The display name of the field in the resulting resourcecolumnType: TheColumnTypeof the fieldoptions: Extra options to be added to the field, e.g. json-table-schema properties such asdecimalCharetc.measures[OPTIONAL]Extra information for measure normalization processing.
(Measure normalization is the process of reducing the number of measures to one while multipltying the number of rows and adding extra columns to contain values for identifying the original measure).Contains the following sub-properties:currency: The currency code of the output measure columntitle[OPTIONAL]: The title for the output measure columnmapping: Unpivoting map.The unpivoting map is a map from a measure's name to its unpivoting data."Unpivoting data" is a map from an extra column's name to a valueExample:measures:currency:GTQmapping:APPROVED:PHASE_ID:"0"PHASE:InicialRELEASED:PHASE_ID:"1"PHASE:VigenteCOMMITTED:PHASE_ID:"2"PHASE:Comprometidocurrencies[OPTIONAL]: List of currency codes to convert to ('USD' by default).
See next section for detailscurrency-conversion[OPTIONAL]Instructions for adding an extra column or columns with measure values in another currency.date_measure[OPTIONAL]: Column name from which a date can be extracted.
If not provided, a guess will be made according to theColumnType.title[OPTOINAL]: Title for the currency-converted measure columns.datapackage-url[OPTIONAL]Contains the URL for a source datapackage from which this data came from.
If provided, metadata for this datapackage will be loaded from this URL.deduplicate[OPTIONAL]Iftrue, then the source data will be processed to remove duplicate rows (i.e. rows which have the same values in the primary key). Measure values for these rows will be summed in order to generate a single output row.postprocessing[OPTIONAL]A list of extra processors (and parameters) that will be applied to the data.
Format is as in anypipeline-spec.yamlsuppress-os[OPTIONAL, default isFalse]IfFalse, an OpenSpending compatible datapackage is created on the datastore. This basic datapackage ensures a basic FDP is available for editing with OpenSpending. Packages created withos-conductoralready create this artefact, so would usesuppress-os: True, to prevent another being created unnecessarily.keep-artifacts[OPTIONAL, default isFalse]By default, pipeline artifacts (temporary directories and files creating during pipeline execution) will be removed after all pipelines have successfully been run. To keep the artifact, set this option toTrue.Generated Pipelines./denormalized_flowLoads external metadataCollects all data from all sourcesCombines different sources onto one unified streamDoes measure normalizationDoes currency conversionDoes row deduplicationDoes extra processing stepsOutputs:Denormalized data (local file)List of fiscal years in a separate resource (local file)Updates os package registry (if configured)./finalize_datapackage_flow_splitter(depends on./denormalized_flow)Loads denormalized packageWrites separate per-year filtered copies of the data./finalize_datapackage_flow(depends on./finalize_datapackage_flow_splitter)Loads all resources from thesplitterpipeline as well as the full denormalized datasetOutputs:Stores results in S3 bucketZip file with the datapackage (in case an S3 bucket is not configured)Updates os package registry (if configured)./dimension_flow_{hierarchy}(depends on./denormalized_flow)Loads denormalized dataPicks onlyhierarchycolumnsAdd auto-incrementing id columnRemove duplicatesOutputs:Normalized hierarchy data (local file)./normalized_flow(depends on./denormalized_flowand all./dimension_flow_{hierarchy})Loads denormalized data as fact tableLoads all normalized hierarchy dataCreates babbage modelReplaces all hierarchy columns in fact table with corresponding ids from normalized hierarchy tablesOutputs:Normalized fact table (local file)Updates os package registry (if configured)./dumper_flow_{hierarchy}(depends on corresponding./dimension_flow_{hierarchy})Loads normalized hierarchy dataFixes nulls in primary key (replacing them with empty strings)OutputsSaves data as a single table in an SQL database./dumper_flow(depends on./normalized_flow)Loads normalized fact table dataFixes nulls in primary key (replacing them with empty strings)OutputsSaves data as a single table in an SQL database./dumper_flow_update_status(depends on./dumper_flow)OutputsUpdates os package registry (if configured) that the package was loaded successfullyContributingPlease read the contribution guideline:How to Contribute