Package: biotrade
Thebiotradepackage analyses international trade of bio-based products. It focuses on
the agriculture and forestry sectors, from primary production to secondary products
transformation. It loads bilateral trade data from UN Comtrade, production and trade
data from FAOSTAT and socio-economic indicators from the World Bank.DocumentationThe documentation is available at:https://bioeconomy.gitlab.io/forobs/biotrade/InstallationBase installationInstall the biotrade package from the python package index with pip:python -m pip install biotradeUpgrade the package to the latest version with:python -m pip install --upgrade biotradeTo install the latest development version, use the--upgradeparameter and install
from the main branch of the gitlab repository:python -m pip install --upgrade --force-reinstall https://gitlab.com/bioeconomy/forobs/biotrade/-/archive/main/biotrade-main.tar.gzInstallation for contributorsIf you plan to contribute to the development of the biotrade package, clone the biotrade
repository athttps://gitlab.com/bioeconomy/forobs/biotrade/. You need to tell python
where the package is located by adding it to your PYTHONPATH. You can do this by
changing the environment variables or by adding the following line to your shell
configuration file such as.bash_aliases:export PYTHONPATH="$HOME/repos/biotrade/":$PYTHONPATHSpecify where you want to store the data by adding the following environment variable:export BIOTRADE_DATA="$HOME/repos/biotrade_data/"Dependencies are listed in theinstall_requiresargument ofsetup.py.UsageThe biotrade package can download data from FAOSTAT and UN Comtrade and store it inside
a database. By default it will use an SQLite database stored locally in the folder
defined by the environment variableBIOTRADE_DATA. Alternatively, a PostGRESQL
database can be used if a connection string is defined in the environment variableBIOTRADE_DATABASE_URL, for example by adding the following to your .bash_aliases or
.bash_rc:export BIOTRADE_DATABASE_URL="postgresql://user@localhost/biotrade"Note that database queries are abstracted withSQL
Alchemywhich is what makes it possible to use different
database engines. SQLite is convenient for data analysis on laptops. PostGreSQL can be
used on servers.FAOSTATFaostat provides agriculture and forestry data on their websitehttps://www.fao.org/faostat/en/#data/The biotrade package has afaostat.pumpobject that loads a selection of datasets. The
list of downloaded datasets is visible infaostat.pump.datasets. Column names and
product descriptions are reformatted to snake case in a way that is convenient for
analysis. The data is then stored into an SQLite database (or PostgreSQL). The following
commands download and transfer the given datasets to the database:>>> from biotrade.faostat import faostat
>>> faostat.pump.update(["crop_production", "forestry_production"])
>>> # Loading trade data can take a long time on slow connections
>>> faostat.pump.update(["crop_trade", "forestry_trade"])
>>> faostat.pump.update(["food_balance"])
>>> faostat.pump.update(["land_use", "land_cover", "forest_land"])List available datasets and metadata links:>>> faostat.pump.datasets
>>> faostat.pump.metadata_linkOnce the data has been loaded into the database, you can query it. For example select
crop production data for 2 countries>>> from biotrade.faostat import faostat
>>> db = faostat.db_sqlite
>>> cp2 = db.select(table="crop_production",
>>>                 reporter=["Portugal", "Estonia"])Select forestry trade flows data reported by all countries, with
Austria as a partner country:>>> ft_aut = db.select(table="forestry_trade",
>>>                    partner=["Austria"])Select crop trade flows reported by the Netherlands where Brazil was a
partner>>> ct_nel_bra = db.select(table="crop_trade",
>>>                        reporter="Netherlands",
>>>                        partner="Brazil")Select the mirror flows reported by Brazil, where the Netherlands was a partner>>> ct_bra_bel = db.select(table="crop_trade",
>>>                        reporter="Brazil",
>>>                        partner="Netherlands")Select land use and land cover data>>> lu = faostat.db.select("land_use")
>>> lc = faostat.db.select("land_cover")ComtradeSee the documentation of the various methods. As an example  here is how to download
trade data from the Comtrade API and return a data frame, for debugging purposes:>>> from biotrade.comtrade import comtrade
>>> # Other sawnwood
>>> swd99 = comtrade.pump.download(cc = "440799")
>>> # Soy
>>> soy = comtrade.pump.download(cc = "120190")Display information on column names used for renaming
and dropping less important columns:>>> comtrade.column_namesGet the list of products from the Comtrade API>>> hs = comtrade.pump.get_parameter_list("classificationHS.json")Get the list of reporter and partner countries>>> comtrade.pump.get_parameter_list("reporterAreas.json")
>>> comtrade.pump.get_parameter_list("partnerAreas.json")Metadata and configuration dataRelease datesFAOSTAT release dates are available at :https://fenixservices.fao.org/faostat/static/releasecalendar/Default.aspxVariable definitions and harmonizationColumn names and product descriptions are reformatted to snake case in a way that is
convenient for analysis. See example below.Variables are defined and compared between the data sources in a notebook calleddefinitions_and_harmonizationVariable names are harmonized between the different sources using a mapping table
defined inbiotrade/config_data/column_names.csvSee for example how theproduct_codecolumn is calledPRODUCT_NCin Eurostat Comext,commodity_codeorcmdcodein UN Comtrade anditem_codein FAOSTAT.snake_caseis the preferred way of naming files and variables in the code. This
follows the Rtidyverse style guide for object
namesand the pythonPEP
8style guide
for function and variable names.To illustrate the advantage of using snake case for data exploration, compare the use of
column names with space which have to be quoted. Python>>> df["Product Code"]
>>> df.product_codeRR> df["Product Code"]
R> df$`Product Code`
R> df$product_codeConfiguration dataThebiotradepackage stores a small amount of configuration data such as country and
product mapping tables and conversion coefficients in thebiotrade/config_datafolder.LicenceThis software is licenced under the MIT licence.
See theLICENCE.mdfile.Similar projectsThe R packages FAOSTAT and WDITestsThis package uses pytest for unit testing. Run the test suite withpytestRun pytest with code coveragecd python_project_dir
coverage run --source=. -m pytestFollowed bycoverage htmlTo generate a report.
These tests are run as part of the Continuous Integration.AcknowledgementsThe authors would like to acknowledge ideas and feedback received from the following
persons: Giovanni Bausano, Noemi Cazzaniga, Mirco Migliavacca, Roberto Pilli, Lucas
Sinclair.