Package: owid-catalog
owid-catalogA Pythonic API for working with OWID's data catalog.Status: experimental, APIs likely to changeOverviewOur World in Data is building a new data catalog, with the goal of our datasets being reproducible and transparent to the general public. That project is ouretl, which going forward will contain the recipes for all the datasets we republish.This library allows you to query our data catalog programmatically, and get back data in the form of Pandas data frames, perfect for data pipelines or Jupyter notebook explorations.graph TB

etl -->|reads| walden[upstream datasets]
etl -->|generates| s3[data catalog]
catalog[owid-catalog-py] -->|queries| s3We would love feedback on how we can make this library and overall data catalog better. Feel free to send us an email atinfo@ourworldindata.org, or start adiscussionon Github.QuickstartInstall withpip install owid-catalog. Then you can begin exploring the experimental data catalog:fromowidimportcatalog# look for Covid-19 data, return a data frame of matchescatalog.find('covid')# load Covid-19 data from the Our World in Data namespace as a data framedf=catalog.find('covid',namespace='owid').load()# load data from other than the default `garden` channellung_cancer_tables=catalog.find('lung_cancer_deaths_per_100000_men',channels=['open_numbers'])df=lung_cancer_tables.iloc[0].load()DevelopmentYou need Python 3.8+,poetryandmakeinstalled. Clone the repo, then you can simply run:# run all unit tests and CI checks
make test

# watch for changes, then run all checks
make watchData typesCatalogA catalog is an arbitrarily deep folder structure containing datasets inside. It can be local on disk, or remote.Load the remote catalog# find the default OWID catalog and fetch the catalog index over HTTPScat=RemoteCatalog()# get a list of matching tables in different datasetsmatches=cat.find('population')# fetch a data frame for a specific match over HTTPSt=cat.find_one('population',namespace='gapminder')# load other channels than `garden`cat=RemoteCatalog(channels=('garden','meadow','open_numbers'))DatasetsA dataset is a folder of tables containing metadata about the overall collection.Metadata about the dataset lives inindex.jsonAll tables in the folder must share a common format (CSV or Feather)Create a new dataset# make a folder and an empty index.json fileds=Dataset.create('/tmp/my_data')# choose CSV instead of feather for filesds=Dataset.create('/tmp/my_data',format='csv')Add a table to a dataset# serialize a table using the table's name and the dataset's default format (feather)# (e.g. /tmp/my_data/my_table.feather)ds.add(table)Remove a table from a datasetds.remove('table_name')Access a table# load a table including metadata into memoryt=ds['my_table']List tables# the length is the number of datasets discovered on diskassertlen(ds)>0# iterate over the tables discovered on diskfortableinds:do_something(table)Add metadata# you need to manually save your changesds.title="Very Important Dataset"ds.description="This dataset is a composite of blah blah blah..."ds.save()Copy a dataset# copying a dataset copies all its files to a new locationds_new=ds.copy('/tmp/new_data_path')# copying a dataset is identical to copying its folder, so this works tooshutil.copytree('/tmp/old_data','/tmp/new_data_path')ds_new=Dataset('/tmp/new_data_path')TablesTables are essentially pandas DataFrames but with metadata. All operations on them occur in-memory, except for loading from and saving to disk. On disk, they are represented by tabular file (feather or CSV) and a JSON metadata file.Columns ofTablehave attributeVariableMeta, including their type, description, and unit. Be carful when manipulating them, not all operations are currently supported. Supported are: adding a column, renaming columns. Not supported: direct assignment tot.columns = ...or to index namest.columns.index = ....Make a new table# same API as DataFramest=Table({'gdp':[1,2,3],'country':['AU','SE','CH']}).set_index('country')Add metadata about the whole tablet.title='Very important data'Add metadata about a fieldt.gdp.description='GDP measured in 2011 international $'t.sources=[Source(title='World Bank',url='https://www.worldbank.org/en/home')]Add metadata about all fields at once# sources and licenses are actually stored a the field levelt.sources=[Source(title='World Bank',url='https://www.worldbank.org/en/home')]t.licenses=[License('CC-BY-SA-4.0',url='https://creativecommons.org/licenses/by-nc/4.0/')]Save a table to disk# save to /tmp/my_table.feather + /tmp/my_table.meta.jsont.to_feather('/tmp/my_table.feather')# save to /tmp/my_table.csv + /tmp/my_table.meta.jsont.to_csv('/tmp/my_table.csv')Load a table from diskThese work like normal pandas DataFrames, but if there is also amy_table.meta.jsonfile, then metadata will also get read. Otherwise it will be assumed that the data has no metadata:t=Table.read_feather('/tmp/my_table.feather')t=Table.read_csv('/tmp/my_table.csv')Changelogdevv0.3.8Switch from isort & black & fake8 to ruffv0.3.8Pin dataclasses-json==0.5.8 to fix error with python3.9v0.3.7Fix bugs.Improve metadata propagation.Improve metadata YAML file handling, to have common definitions.RemoveDatasetMeta.origins.v0.3.6Fixed tons of bugsprocessing.pymodule with pandas-like functions that propagate metadataSupport for Dynamic YAML filesSupport for R2 alongside S3v0.3.5Removecatalog.frames; useowid-repackpackage insteadRelax dependency constraintsAdd optionalchannelargument toDatasetMetaStop supporting metadata in Parquet format, load JSON sidecar insteadFix errors when creating new Table columnsv0.3.4Bumppyarrowdependency to enable Python 3.11 supportv0.3.3Add more arguments toTable.__init__that are often used in ETLAddDataset.update_metadatafunction for updating metadata from YAML filePython 3.11 support via update ofpyarrowdependencyv0.3.2Fix a bug inCatalog.__getitem__()Replacemypytype checker bypyrightv0.3.1Sort imports withisortChange black line length to 120AddgrapherchannelSupport path-based indexing into catalogsv0.3.0UpdateOWID_CATALOG_VERSIONto 3Support multiple formats per tableSupport reading and writingparquetfiles with embedded metadataOptionalrepackargument when adding tables to datasetUnderscore|Getversionfield fromDatasetMetainitResolve collisions ofunderscore_tablefunctionConvertversiontostrand load jsondimensionsv0.2.9Allow multiple channels incatalog.findfunctionv0.2.8UpdateOWID_CATALOG_VERSIONto 2v0.2.7Split datasets into channels (garden,meadow,open_numbers, ...) and make garden default oneAdd.find_latestmethod to Catalogv0.2.6Add flagis_publicfor public/private datasetsEnforce snake_case for table, dataset and variable short namesAdd fieldspublished_byandpublished_atto SourceAdded a list of supported and unsupported operations on columnsUpdatedpyarrowv0.2.5Fix ability to load remote CSV tablesv0.2.4Update the default catalog URL to use a CDNv0.2.3Fix methods for finding and loading data from aLocalCatalogv0.2.2Repack frames to compact dtypes onTable.to_feather()v0.2.1Fix key typo used in version checkv0.2.0Copy dataset metadata into tables, to make tables more traceableAdd API versioning, and a requirement to update if your version of this library is too oldv0.1.1Add support for Python 3.8v0.1.0Initial release, including searching and fetching data from a remote catalog