Package: econuy
OverviewThis project simplifies gathering and processing of Uruguayan economic statistics. Data is retrieved from (mostly) government sources, processed into a familiar tabular format, tagged with useful metadata and can be transformed in several ways (converting to dollars, calculating rolling averages, resampling to other frequencies, etc.).Ifthis screenshotgives you anxiety, this package should be of interest.A webapp with a limited but interactive version of econuy is available atecon.uy. Check out therepoas well.The most basic econuy workflow goes like this:fromeconuy.coreimportPipelinep=Pipeline()p.get("labor_rates")InstallationPyPI:pipinstalleconuyGit:gitclonehttps://github.com/rxavier/econuy.gitcdeconuy
pythonsetup.pyinstallUsageFull API documentation available at RTDThePipeline()classThis is the recommended entry point for the package. It allows setting up the common behavior for downloads, and holds the current working dataset.fromeconuy.coreimportPipelinep=Pipeline(location="your_directory")ThePipeline.get()methodRetrieves datasets (generally downloads them, unless thedownloadattribute isFalseand the requested dataset exists at thelocation) and loads them into thedatasetattribute as a Pandas DataFrame.ThePipeline.available_datasets()method returns adictwith the available options.fromeconuy.coreimportPipelinefromsqlalchemyimportcreate_engineeng=create_engine("dialect+driver://user:pwd@host:port/database")p=Pipeline(location=eng)p.get("industrial_production")Which also shows that econuy supports SQLAlchemyEngineorConnectionobjects.Note that every time a dataset is retrieved,PipelinewillCheck if a previous version exists atlocation. If it does, it will read it and combine it with the new data (unlessdownload=False, in which case only existing data will be retrieved)Save the dataset tolocation, unless thealways_saveattribute is set toFalseor no new data is available.Data can be written and read to and from CSV or Excel files (controlled by theread_fmtandsave_fmtattributes) or SQL (automatically determined fromlocation).Dataset metadataMetadata for each dataset is held in Pandas MultiIndexes with the following:Indicator nameTopic or areaFrequencyCurrencyInflation adjustmentUnitSeasonal adjustmentType (stock or flow)Cumulative periodsWhen writing, metadata can be included as dataset headers (Pandas MultiIndex columns), placed on another sheet if writing to Excel, or dropped. This is controlled byread_headerandsave_header.Pipeline transformation methodsPipelineobjects with a valid dataset can access 6 transformation methods that modify the held dataset.resample()- resample data to a different frequency, taking into account whether data is of stock or flow type.chg_diff()- calculate percent changes or differences for same period last year, last period or at annual rate.decompose()- seasonally decompose series into trend or seasonally adjusted components.convert()- convert to US dollars, constant prices or percent of GDP.rebase()- set a period or window as 100, scale rest accordinglyrolling()- calculate rolling windows, either average or sum.fromeconuy.coreimportPipelinep=Pipeline()p.get("balance_nfps")p.convert(flavor="usd")p.resample(rule="A-DEC",operation="sum")Saving the current datasetWhilePipeline.get()will generally save the retrieved dataset tolocation, transformation methods won't automatically write data.However,Pipeline.save()can be used, which will overwrite the file on disk (or SQL table) with the contents indataset.TheSession()classLike aPipeline, except it can hold several datasets.Thedatasetsattribute is adictof name-DataFrame pairs. Additionally,Session.get()accepts a sequence of strings representing several datasets.Transformation and saving methods support aselectparameter that determines which held datasets are considered.fromeconuy.sessionimportSessions=Session(location="your/directory")s.get(["cpi","nxr_monthly"])s.get("commodity_index")s.rolling(window=12,operation="mean",select=["nxr_monthly","commodity_index"])Session.get_bulk()makes it easy to get several datasets in one line.fromeconuy.sessionimportSessions=Session()s.get_bulk("all")fromeconuy.sessionimportSessions=Session()s.get_bulk("fiscal_accounts")Session.concat()combines selected datasets into a single DataFrame with a common frequency, and adds it as a new key-pair indatasets.External binaries and librariesunrar librariesThepatoolpackage is used in order to access data provided in.rarformat. This package requires that you have theunrarbinaries in your system, which in most cases you should already have. You can can get them fromhereif you don't.Selenium webdriversSome retrieval functions need Selenium to be configured in order to scrape data. These functions include adriverparameter in which a Selenium Webdriver can be passed, or they will attempt to configure a Chrome webdriver, even downloading the chromedriver binary if needed. This still requires an existing Chrome installation.Caveats and plansCaveatsThis project is heavily based on getting data from online sources that could change without notice, causing methods that download data to fail. While I try to stay on my toes and fix these quickly, it helps if you create an issue when you find one of these (or even submit a fix!).PlansImplement a CLI.Provide methods to make keeping an updated database easy.Session.get_bulk()mostly covers this.Visualization.(I have decided that visualization should be up to the end-user. However, thewebappis available for this purpose).Translations for dataset descriptions and metadata.