Package: tap-bls
tap-blsThis is aSingertap that produces JSON-formatted data
following theSinger
spec.
If you are reading this description on PyPi and anything looks strange please refer tothe version found on GitHubThis tap pulls raw data from theBureau of Labor Statistics (BLS) APIas follows:Provides a sample list of available BLS data series that you can add to your catalog - pick from the list or add your own.Runs the API to fetch the data series from the BLS and outputs the data to StdOut based on theSinger.ioformat.Optionally updates the tap'sSTATE.jsonfile (which is not standard, as theTARGET.jsontypically does this, but this is an option you can set inCONFIG.json)Copyright © 2020 StitchPyPi package repo:https://pypi.org/project/tap-bls/pip install tap-blsVersion 0.1.7 released 01/30/2022Extract BLS (Bureau of Labor Statistics) data using Singer.ioThe BLS providesan API for pulling data from their records, andSinger.iois a common framework for building data flows.Installation quickguiderequirements: Python 3.5.3 & modules os, pytz, sys, json, datetime, backoff, getopt, requests, and SingerCreate a virtual environment, such aspython3 -m venv ~/.virtualenvs/tap-blsand activate it withsource ~/.virtualenvs/tap-bls/bin/activateSet the local version of Python to 3.5.3:pyenv local 3.5.3Installwheelwithpip install --upgrade pip wheelInstall the tap in your venv usingpip install tap-blsexit the virtual environment withdeactivatemake a copy ofsample_config.json(asconfig.json) andseries.json(asseries.json) from the root of the repo into your preferred configuration folder (for example I use~/tap-bls-config)edit theconfig.jsonfile - the main thing you want to change is the API key ("api-key": in the json file) and put in your BLS API key. You can even leave this blank if you just want to get started.run the tap once in 'Discovery mode' to build yourcatalog.jsonfile - your command will looksomethinglike~/.virtualenvs/tap-bls/bin/tap-bls --config ~/tap-bls-config/config.json --discover > ~/tap-bls-config/catalog.jsonYou can now run the tap in standard mode - if you just want to test, run it 'unpiped' with a command such as~/.virtualenvs/tap-bls/bin/tap-bls --config ~/tap-bls-config/config.json --catalog ~/tap-bls-config/catalog.jsonbut if you havetap-csvinstalled you can make pretty outputs using~/.virtualenvs/tap-bls/bin/tap-bls --config ~/tap-bls-config/config.json --catalog ~/tap-bls-config/catalog.json | ~/.virtualenvs/target-csv/bin/target-csvnote: when creating the catalog, the schemas will get created following the normal 'schema' process for Singer, adding the definition files in a 'schema' folder within the tap files ( for example in~\.virtualenvs\tap-bls\lib\python3.6\site-packages\tap_bls\schemas). To change the series that you want to pull you will need to delete this folder, edit theseries.jsonfile, then recreate the catalog by running the discovery mode in step 7 above.You can use a--statefile if you like. This tap provides the option to update the State from the tap, rather than the target. If you want info on Singerstatefilescheck out the docs.So why is this tap cool?The BLS is the most reliable source of economic data for the USA when it comes to things like unemployment rates, the cost of labor, etc. It also includes Consumer Price Indices, Inflation, Workplace injuries and a bunch of other useful stuff.A list of topics can be found hereand the most popular data series (a.k.a. "The BLS Greatest Hits!") can be foundhere.So say you wanted to know the trend for unemployment during the COVID-19 pandemic of 2020 you could simplyquery the APIand see that it rapidly rose from from 3.5% at the end of 2019 to a high of14.7%in April 2020, and back down to 6% by March 2021.You need to select the BLS series you want to ingestThe volume of data available can quickly get overwhelming. Just one topic -Producer Price Indexes - has 318 distinct data series. Others are so complex, they provide entire excel sheets full of time series references, as is the case for theAmerican Time Use study. An explanation ofhow the BLS structures Series IDs for each topic can be found here.A good starting point when looking for available data series isDatabases, Tables & Calculators by Subjector thedata series by topic.With this in mind, the tap provides a framework you can use to ingest BLS data using Singer, but the catalog file would be overwhelming if we attempted to provide every available data series, even if these were maked asunselectedin the catalog.js. So the tap comes out-of-the-box with a dozen different series taken from some of the most popular ones, but you can configure it for the data series you want to pull in by editing ./series.json.Grab a keyYou can access BLS data without registering a key. If you do not provide an API key, you will be restricted in the volume of data you can pull. So go to theBLS registration pageand grab a key. This said, even an authenticated user has limits.data source API (and limitations thereof)Our data source isversion 2 of the BLS APIwhich provides a mechanism for grabbing JSON historical timeseries data, along with optional calculations and averages.The API has some 'fair use' limitations outlinedhere- namely 50 series per query, 500 daily queries, 50 requests per 10 seconds etc.The BLS Public API utilizes two HTTP request-response mechanisms to retrieve data: GET and POST. GET requests data from a specified source. POST submits data to a specified resource to be processed. The BLS Public Data API uses GET to request a single piece of information and POST for all other requests.The BLS have imposed a maximum of 20 years in the query (with the API key - and 10 years without). Bear this in mind as longer time queries will simply be cut off.Python is the language of choice for Singer.io taps, we are going to stick with that andsample code is provided here. The BLS provide alternatives in most popular languages.endpointsHTTP Type: POST
URL (JSON):https://api.bls.gov/publicAPI/v2/timeseries/data/Header: Content-Type= application/jsonquery parametersThe BLS API allows us to query multiple series in a single call, using distinct series IDs. . Registered users can include up to 50 series IDs, each separated with a comma, in the body of a request. This said, for the ease of execution we are going to call each series one at a time. Sure, this eats into our 500 daily queries, but after all this data does not chage often (monthly at most).config.jsonThis tap requires a config file although none of the parameters arerequired. This said, it will accept the following parameters. We recommend getting and adding your BLS API key to get the most out of the integration:{
  "user-id": "your.name@emailprovider.com",
  "api-key": "your-bls-issued-api-key-goes-here",
  "startyear": "2019",
  "endyear": "2020",
  "calculations": "true",
  "annualaverage": "false",
  "aspects": "false",
  "disable_collection": "true",
  "update_state": "false",
  "series_list_file_location": "<absolute or relative path>/series.json"
}user-idis optional. The BLS specifies it but then it's not passed in the API call ¯\(ツ)/¯api-keyis your BLS issued API keystart yearis the year you want your data extract to start. Not the limits: you can pull up to 20 years in one go, and most data seris start at 2000, so you do the math... If left blank it will default to 2000. [ should be a year as a string - i.e. in quote marks ]endyearis when you want the series to end. If left blank it will default to the current year. [ should be a year as a string - i.e. in quote marks ]aspectsis an option you will find in the API documentation but setting this to 'true' has caused issues (in my experience) whereby some data points are no longer provided in the returning payload.series_list_file_locationis an optional absolute or relative path to the series.json. If not provided (and if the schemas directory is empty), the tap will look for a file namedseries.jsonin the same location as theconfig.jsonfileThe next three parameters are explained in more detailon the BLS websiteParameterdescriptionvalues acceptedcalculationsprovides 1,3,6 and 12 month changes in the data in both net and percentage format. If selected, these 6 additional datapoints will be included in separate columns.will accept "true" or "false"annualaverageIf selected, an annual data series will include aM13datapoint with the annual average value.will accept "true" or "false"aspectsReturns data series aspect data in the format[{'name': 'Standard Error', 'value': '-', 'footnotes': [{'code': 'A', 'text': 'Dashes indicate data not available.'}]}]. Not many BLS series include this.will accept "true" or "false"disable_collectionshould theoretically prevent Singer from collecting additional anonymous data on your runs, which are used to help improve Singer. You can set to "true" if you like, although it appears the additional data is collected either way ¯\(ツ)/¯update_stateis an uncharacteristic feature for a Singer tap. Thetargetshould updateSTATEonce it has established that the data has been loaded to the endpoint, but this flag allows you to instruct thetapto updateSTATE.jsonat the end of the run. So typically you would set this to 'false'.tap --config CONFIG [--state STATE] [--catalog CATALOG]STATE and CATALOG are optional arguments both pointing to their own JSON file. If you do not specify astate.jsonfile the tap will generate one in the same folder at theconfig.jsonfile. tap-bls will useSTATE.jsonto remember information from the previous invocation such as the point where it left off, namely the year of the most recent data point.Schema fields.MonthlyQuarterlySemi-AnnualAnnualFormatNoteannualaverageY*Y*Y*Nfloat, or NullIf set in config as"annualaverage": "true"aspectsY*Y*Y*Y*Array of json dicts in the format [{'name': 'Standard Error', 'value': '0.1', 'footnotes': [{}]}] - if blank, presented as empty array :[]If set in config as"aspects": "true"net_change_1Y*Y*Y*Y*floatIf set in config as"calculations": "true"net_change_3Y*Y*Y*Y*floatIf set in config as"calculations": "true"net_change_6Y*Y*Y*Y*floatIf set in config as"calculations": "true"net_change_12Y*Y*Y*Y*floatIf set in config as"calculations": "true"pct_change_1Y*Y*Y*Y*floatIf set in config as"calculations": "true"pct_change_3Y*Y*Y*Y*floatIf set in config as"calculations": "true"pct_change_6Y*Y*Y*Y*floatIf set in config as"calculations": "true"pct_change_12Y*Y*Y*Y*floatIf set in config as"calculations": "true"record__footnotesY*Y*Y*Y*textPotentially returns multiple footnotes, although extremely rare.record__full_periodYYYYDateTimeComplete date for the datapoint.record__monthYNNNintegerMonth (1-12)record__periodYYYYtextFormat examples: "M11","Q2","S02","A01"record__quarterNYNNtextrecord__SeriesIDYYYYtextThe series Idrecord__time_extractedYYYYDateTime"Complete date plus hours, minutes, seconds and a decimal fraction of a second"record__valueYYYYfloatrecord__yearYYYYintegerschemaYYYYtextThe applied schema to this series (same as series id)streamYYYYtexttime_extractedYYYYDateTime"Complete date plus hours, minutes, seconds and a decimal fraction of a second"typeYYYYtextRECORDfrequencyYYYYtextSet to 'M','Q','S' or 'A' for monthly, quarterly or annual series.(*) Note - the value will be included in the schema, but that does not guarantee that the API call we return a value. Sometimes the data series siply does not include data for this item.Autogenerating the SCHEMAsTypically, the fileCATALOG.json(generated during discovery) is used to filter which streams should be synced from all the possible streams available in the /schemas/ folder.tap-bls behaves a bit differently because of the enormous number of potential data series you migth pull from the BLS. Therefore it works as follows:If no.jsonschema files are to be found in thetap_bls/schemas/folder, the tap will generate them for you based on theseries.jsonfile. The tap will look for theseries.jsonfile in the same folder as yourconfig.jsonfile. This allows you to rapidly select which BLS data series you want to work with. A sample 'series.json' file is found in the root of this repo with a bunch of the most popular data seris included, so you can just copy that to your main config folder.Once you have a set of schema files created (manually or using the automated approach above) you can generate the Singercatalog.jsonfile using the tap's --discover mode using a command such as~/.virtualenvs/tap-bls/bin/tap-bls --config ~/tap-bls-config/config.json --discover > catalog.json(your set up may use a different folder than 'tap-bls-config' - that is up to you.)A note on series frequencyAs far as I can figure out, the BLS provides data series in the following frequencies:Monthly - the most commonBi-monthly - very uncommon ( See CUURS12BSA0 )Quarterly - fairly commonSemi-Annually (6 months) - uncommon, seeCUURS12BSAFAnnual - fairly commonBi-monthly and semi-annual appears to be most prevalent in data series related to consumer price index reports.Note that where available (and where set to 'true' inconfig.json), annual averages are should asM13in the dataset.Tap structureThere is no predefined file structure for building a Singer tap, but here is a logical structure for this particular set-up.paginationerror codesBLS API sourcecodehttps://github.com/OliverSherouse/blsOpen questions and to-dosTEST SERIES:CES0000000001 - Monthly series without annual averageWPUFD49104 - Monthly series with annual averagePRS85006092 - Quarterly series with annual averageMPU4910012 - annual seriesCUURS12BSA0 - bi-monthly series - essentially monthly, but with blank values every other month.https://api.bls.gov/publicAPI/v2/timeseries/data/LNS14000000?startyear=1940&endyear=1960Series starts mid-windowInvestigate why PRS85006092 returns Q5 data which does not match annual averages.