Package: opentoolflux
ContentsContentsOverviewHow the flux estimation worksInstallationBasic usageRun an exampleUse the built-in documentationConfigure OpenToolFlux using a configuration fileImport data from source filesOther ways to get a databaseShow info about the databaseEstimate gas fluxesPlot resultsflux-fits: flux estimation diagnosticsflux-time-series: fluxes over timeNotes on dimensions and unitsThe database fileOn data types and database sizeSource data file formatNotes on timestampsConfiguration optionsOverviewOpenToolFlux is a software to estimate gas fluxes from soil using data from automatic chambers.The software is used through a command-line interface (CLI) which allows the user to:Import data from one or more source data files into a database file.Filter the database based on time period, alarm status values, and other criteria.Identify segments of the database corresponding to closures of chambers.Estimate gas fluxes from the concentration time series during closure, and export these to atidy datafile.Generate diagrams for diagnostics.How the flux estimation worksThe software analyzes time series of gas concentrations in gas collected from multiple automated chambers connected to a single gas analyzer. The multiple chambers sequentially close (e.g. in 20-minute intervals) and gas is continuously pumped into the analyzer from the currently closed chamber. The profile of the concentration change during each chamber closure is used to estimate the flux from the soil under that chamber.A detailed explanation of the flux estimation and its underlying assumptions is to be published in a paper by Galea et al. When published, a reference to the paper will be added here.InstallationOpenToolFlux is written in Python and works with Python 3.8+.This installation guide assumes that you already have Python 3.8+ and that you know how to use a terminal and install Python packages usingpip. If you are unsure about any of these points, you might find it helpful to follow the instructions given by any of these sources:Instructions from RealPython oninstalling PythonandusingpipInstructions fromPython.org on getting started with PythonInstructions from PyPA onusing pip and virtual environmentsInstructionsPython for Data Analysis, 3rd EditionusingminicondaHow to install:Optionally, create and activate a virtual environment.Install usingpip install opentoolfluxVerify that the installation was successful by running the commandopentoolflux --help. If the installation has succeeded, this will show a list of the available commands.If installation according to these instructions fail, please submit a bug reportusing the issue tracker.Basic usageThe command-line interface consists of a commandopentoolfluxwith subcommands such asopentoolflux importandopentoolflux fluxes. Here is a quick introduction to basic usage of the command-line interface.Run an exampleYou might find it instructive to get started using an example. An example of a configuration file, input data, and corresponding output data can be downloaded from here:https://github.com/rasmuse/flux-estimation-example.Use the built-in documentationThe command-line interface has built-in documentation which is accessed by calling commands such as:opentoolflux --help
opentoolflux import --helpWhen adding--helpto a command, the command will do nothing except print an information message.We recommend exploring the software by reading the built-in help and experimenting with commands on some test data. OpenToolFlux will never change or delete your source data files, so you can safely play around. (And in any case, you do have a backup of your important research data,right?)Configure OpenToolFlux using a configuration fileMost configuration is made in a configuration file written inTOML language. All the configuration options are listed and explainedat the end of this document.A small number of configuration options can be made on the command line. These are listed and explainedin the built-in documentation.OpenToolFlux by default looks for a configuration file namedopentoolflux.tomlin the working directory. This default can be overridden using the--configflag as follows:opentoolflux --config my_config.toml [command]Import data from source filesWhen the configuration file is in place, import data from source files using the following command:opentoolflux importThis will create a new database, or add data to an existing one, located atopentoolflux/database.feather. The data files to read are specified in theimportsection of theconfig file. Read more below aboutthe source data file format.Note:opentoolfluxwill never change or remove the source data files, so you can safely try commands to see what happens. If you want to start over from zero, simply remove thedatabase.featherfile and runopentoolflux importagain.Other ways to get a databaseIt is also possible to copy theopentoolflux/database.featherfile between computers. To "export" the database, simply copy thedatabase.featherfile and save it on a USB stick or network drive, or even send it by email if it's not too large.You can also create the database file in any other way you like, following thetechnical specification of the database below.Show info about the databaseTo show some basic info about the database file, run the following command:opentoolflux infoEstimate gas fluxesWhen thedatabase.featherfile is in place, estimate gas fluxes using the following command:opentoolflux fluxesThis will do several things:Optionally, filter the database as specified in thefilterssection of theconfig file.Split the remaining data into segments by chamber as specified in themeasurementssection of theconfig file.Ignore any segments that are too short or too long, following themeasurementssection of theconfig file. Unexpectedly short segments can be created, for example, if the equipment is shut down or restarted.For each of the remaining measurements, estimate the flux as specified in thefluxessection of theconfig file. (Read more abouthow the flux estimation worksbelow.)Save all the results to a fileopentoolflux/fluxes.csv.Plot resultsThe command groupopentoolflux plotcan be used to generate the following figures:flux-fits: flux estimation diagnosticsTo visualize the gas flux estimation and identify potential problems, run the following command:opentoolflux plot flux-fitsThis command estimates gas fluxes followingthe same steps as theopentoolflux fluxescommand, but instead of a results table it outputs one figure for each measurement in the folderopentoolflux/plots/flux-fits. Each figure shows the gas concentration(s) over time during a chamber closure, and the curve that has been fit to estimate the gas flux(es). Thet0_delayandt0_marginparameters are also shown in this figure.flux-time-series: fluxes over timeTo generate figures showing the fluxes over time in each chamber, run:opentoolflux plot flux-time-serieThis command estimates gas fluxes followingthe same steps as theopentoolflux fluxescommandand then creates a figure for each chamber, showing the flux estimates of all gases over time.Notes on dimensions and unitsThe software expects volumetric concentration data and produces volumetric flux estimates, i.e., fluxes in dimensions of length / time (= volume / area / time). The units can be, e.g., ppmv, ppbv, or simply fractions (i.e., m3/m3). The "prefix" implicit in the unit (e.g., 10^-6 for ppmv) also affects the flux estimates. For example, if concentration indata are in ppmv, and all other parameters given in SI units, then the flux estimates are given in 10^-6 m/s (micrometers per second).As explained inour paper on the estimation procedure, the flux estimates also depend on the chambers' footprint area A, volume V, and the sample flow rate Q. The configuration file (see below) requires specification of A, V, and Q. Technically, the only requirement is that Q/V has unit second, butwe strongly recommend using SI units for A, V, and Q (i.e., specify A in m2, V in m3, and Q in m3/s). See the example below.The output flux estimates will have the units of [c * V / A] per second, where c is the concentration. E.g., if concentrations are in ppmv, V in m3 and A in m2, then flux estimates are in 10^-6 m/s (micrometer per second).The database fileOpenToolFlux uses a database file which is just a table stored as aFeather file. The default file path to the database isdatabase.featherstored in the output directory.The database has one row per sample and normally contains the following columns:__TIMESTAMP__: a timestamp of the sample, inUTC. This column is used as primary key in the database, so the timestamps must be unique. The table must be sorted by timestamp in ascending order. The__TIMESTAMP__column is the only mandatory column in the database (although a database with only timestamps is not really useful).One column identifying the current chamber. With the Picarro data, this is a number, but any data type will work.One column giving sample concentration of each gas to analyze, e.g.,CO2,CH4_dryand/orN2O_dry.Optionally, additional columns used to filter out samples. For exampleALARM_STATUSin the case of Picarro data files.The commandopentoolflux import(see above) can be used to create the database from Picarro (or other) data files. It is also possible to create the database in any way (e.g., using a custom Python or R script). The database file can also be copied between folders or computers without any problem.On data types and database sizeThe database can contain columns of different data types: unsigned (nonnegative) integers (uint), signed integers (int), floating-point numbers (float), booleans (bool), and strings (str). The numeric datatypes (uint,int, andfloat) come in different precisions/sizes:uint8: Integers 0 to 255uint16: Integers 0 to 65,535uint32: Integers 0 to 4,294,967,295uint64: Integers 0 to 18,446,744,073,709,551,615int8: Integers -128 to 127int16: Integers -32,768 to 32,767int32: Integers -2,147,483,648 to 2,147,483,647int64: Integers -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807float16: 16-bitfloating point(a.k.a. "half precision")float32: 32-bitfloating point(a.k.a. "single precision")float64: 64-bitfloating point(a.k.a. "double precision")Optimizing only for data types capacity and precision, it makes sense to always chooseuint64,int64andfloat64. However, for large databases (long time series), the database can grow fairly large, and it may make sense to choose a more restrictive data type.For example, if the chambers are encoded as integers 1-12, or even 1-100, it is more than enough to use anint8oruint8, which takes only 1/8 of the space compared to anint64.For thefloatdata types, the choice is not as obvious because there is a loss of precision going fromfloat64tofloat32orfloat16. But practically speaking, even afloat16in many cases will be sufficient. In technical terms, afloat16significand has 3-4 significant decimal digits.Therefore, for example, when we work with Picarro data on N2O concentrations in ppmv (roughly 0.3 ppmv N2O), afloat16can encode the difference beteween 0.300 ppmv and 0.301 ppmv without problem (a difference of 1 ppbv). This precision is much better than thesecond-to-second noise in the Picarro concentration data. This example shows that for many purposes, usingfloat16instead offloat64for gas concentrations will practically make very little difference for results. (By the way, the Picarro software converts all gas concentrations tofloat64before doing the flux estimate calculation, so only the value stored in the database is limited by thefloat16encoding.)If in doubt aboutfloatdata types, we suggest to usefloat32which hasat least 6 decimal digits of precision in the significandand thus should be far more precise than practically speaking any gas analyzer out there.How much space can be saved by choosing smaller data types? As an example, consider a database with the following data columns:Timestamp (always 64 bits in a Feather file)Chamber number (uint8oruint64)Alarm status (int8orint64)Five gas concentrations: N2O, NO2, CH4, CO2, H2O (float16orfloat64)With the smaller data types, each row will take 64 + 2 * 8 + 5 * 16 = 160 bits = 20 bytes.With the larger data types, each row will take 64 + 2 * 64 + 3 * 64 = 384 bits = 64 bytes, or 3.2 times as much.If we collect one data row per second during one year, the resulting database sizes will be either 601 or 1,925 megabytes (MiB). If your computer has less than 4 gigabytes of RAM memory, the program might get slow or even crash with the larger database, and in any case you might care about the difference in space on disk.This example also shows that if your dataset is perhaps only a few weeks long with frequency 1 second, or maybe one year with frequency 1 minute, the database file will anyway be so small that there are probably very few reasons to worry about database size.A final related note is that floating-point data to be converted to timestamps in theopentoolflux importcommand should always befloat64. The conversion of floating-point Unix timestamps (in seconds) is designed to preserve 6 decimal places (microseconds), something which requiresfloat64. The__TIMESTAMP__column in the end is always encoded using 64 bits anyway, so there is no space to be saved by parsing the timestamp column as afloat32. Failure to specifyfloat64as data type for the timestamp column raises a helpful error message.Source data file formatHere is an example of what the default source file format looks like:EPOCH_TIME      ALARM_STATUS   solenoid_valves     N2O_dry
1620345675.170  0              5.0000000000E+00    3.3926340875E-01
1620345675.991  0              5.0000000000E+00    3.3928078030E-01
1620345676.605  2              5.0000000000E+00    3.5087647532E-01
1620345677.312  0              6.0000000000E+00    3.3491837412E-01The data files from our Picarro equipment look like this, but with many more columns containing various information. A full example is given here:example/indata.Source data files must be bedelimited text files. The default delimiter, following the data files we get from our Picarro equipment, isone or morewhitespace characters (sep = '\s+'), but other delimiters can be specified using thesepsetting (e.g.,sep = ","for standard csv files).For example, settingsep = ","in the configuration file, an example input data file would beEPOCH_TIME,ALARM_STATUS,solenoid_valves,N2O_dry
1620345675.170,0,5.0000000000E+00,3.3926340875E-01
1620345675.991,0,5.0000000000E+00,3.3928078030E-01
1620345676.605,2,5.0000000000E+00,3.5087647532E-01
1620345677.312,0,6.0000000000E+00,3.3491837412E-01(Technical note: The Picarro source files, following roughly the format shown above, can also be seen asfixed-width files, but since the data fields do not contain whitespace, they can also be parsed as delimited files as described here.)Each source data file must have a one-line header specifying the column names.Notes on timestampsOne of the columns must contain timestamps of the measurements (in the example above, the columEPOCH_TIME). The timestamps can be encoded as:Numeric values, which are interpreted to beUnix timestampsexpressed in seconds. In Picarro data files, theEPOCH_TIMEcolumn is a Unix timestamp in seconds (with three decimals, giving millisecond resolution).String values, which are parsed usingpandas.to_datetime(values, format="ISO8601", utc=True)and then converted to UTC timestamps. This means that timestamp strings can be expressedin UTC using a string such as"2021-12-07T11:00:24.123Z",in any other timezone, e.g.,"2021-12-07 13:00:24.123+0200", orwithout timezone, e.g.,"2021-12-07 13:00:24", which will be interpreted as UTC.When running theopentoolflux importcommand, the timestamp source column is converted to UTC timestamps following the above rules, and renamed to__TIMESTAMP__.Configuration optionsBelow is an example configuration file listing all the configuration options.# The [general] section is optional.[general]# `outdir` is the output directory path, relative to the configuration file.# Optional. Default "opentoolflux".outdir="outdata"# The [import] section is optional. If you already have a Feather database# file, this is not needed.[import]# src lists the path(s) to input data files which should be delimited text# files as explained above. All paths are relative to the configuration file.# When running the `opentoolflux import` command, these paths are searched.src=["path/to/filename.dat",# Path to a specific file"indata/2022/06/01/*.dat",# Unix-style glob patterns using *"indata/2022/**/*.dat",# Also ** globs are supported.]# sep specifies the delimiter used in the data files..sep="\\s+"# One or more whitespace characters (the default).# sep = ","  # Alternative for comma-separated files# sep = "\\t,"  # Alternative for tab-separated files# timestamp_col gives the name of the column containing sample timestamps.# See the README section on timestamps for details on formats.timestamp_col="EPOCH_TIME"# [import.columns] gives the names of the columns to import and their data types.# See the opentoolflux README section on data types for more details.[import.columns]EPOCH_TIME="float64"valve_number="float16"ALARM_STATUS="uint8"CO2="float32"CH4_dry="float32"N2O_dry="float32"# Filters are optional.## A [filters.x] section specifies the filtering to be done on column x.# Filters can specify `allow_only`, `disallow` and `min` and `max`# as shown in examples below. The data matched by a filter is treated# as if it was not present in the database; i.e., data "removed" by# filters do not contribute at all to flux estimates and are not shown# in plots generated.[filters.ALARM_STATUS]disallow=[4]# __TIMESTAMP__ is a special column created in the database to store# the timestamps. This has datetime data type and any values specified in the# filter settings are automatically converted to datetime too.[filters.__TIMESTAMP__]min_value="2022-05-09"max_value="2022-05-10 06:00:00"# Advanced note on data types:# Filtering is done after import, i.e., using the data types specified in# the [import.columns] section above. Values specified in filter settings# are converted to the data types of the  corresponding column; therefore,# in the `valve_number` example below, the `allow_only` values specified# as integers will be converted to `float16`, the specified data type for# that column, before the filter is applied.[filters.valve_number]allow_only=[2,3,6,7][filters.CO2]min=0max=2000# The [measurements] section specifies how to split the database into segments# corresponding to chamber closures.[measurements]# chamber_col is the name of the database column indicating the current chamber.# A new segment starts when the value in this column changes.chamber_col="valve_number"# The time values below (max_gap, min_duration, and max_duration) should be set# to values corresponding to the setup of your experiment.## In our example, the input data have ca 1 Hz sampling frequency, so a gap# of 10 seconds indicates a problem/restart which we treat as a "new" segment;# and the segment length normally should be 20 minutes, so we discard any# segments deviating more than 30 seconds from that.max_gap="00:00:10"# Max 10 seconds between samples.min_duration="00:19:30"# Segment length minimum 19 minutes 30 seconds.max_duration="00:20:30"# Segment length maximum 20 minutes 30 seconds.# The [fluxes] section specifies columns containing concentration data# to be analyzed.[fluxes]gases=["N2O_dry","CH4_dry","CO2"]# the list of gas-concentration columns# Chamber dimensions and sampled gas flow## See notes on units in the README. In summary:# - Concentrations can be in any unit (e.g., ppmv or simple fractions m3/m3).# - V / Q must have unit second. (E.g., if Q is m3/s, then must be in m3.)# - A can have any unit.## The resulting (volumetric) flux estimates will have the units of# [c * V / A] per second, where c is the concentration.# E.g., if concentrations are in ppmv, V in m3 and A in m2, then flux# estimates are in 10^-6 m/s (micrometer per second).## We recommend using straight SI units throughout to minimize confusion.A=0.25# Soil area covered by the chamber, here 0.25 m2V=50e-3# Volume of chamber, here 50 liters in m3Q=4.16e-6# Gas flow drawn from chamber, here 0.25 liters/minute in m3/s# t0_delay is the time delay from chamber closure until the first gas arrives# in the gas analyzer. This delay depends on the tube length which may be# different between chambers.# Optionally, the same t0_delay can be set for all chambers:# t0_delay = "00:06:00"  # 6 minutes# t0_margin specifies an extra duration of data to discard after t0_delay,# to avoid estimation errors if the actual delay is longer which leads to "old"# gas concentrations being erroneously included in the curve fit.# Calculations are made using t0 equal to the closure start plus t0_delay.# See figures generated by `opentoolflux plot flux-fits` for an illustration.t0_margin="00:02:00"# The `t0_delay` can be set to one value, e.g., t0_delay = "00:06:00" as above,# or as a table with different t0_delay for each chamber, as follows.# This is the preferred solution if the delay is substantially different# between chambers.# The left-hand side of these assignments is a string representation# of the chamber value, i.e., in this example the value found in the# `valve_number` column.[fluxes.t0_delay]2="00:04:00"# 4 minutes3="00:06:00"# 6 minutes6="00:05:30"# 5 minutes 30 seconds7="00:06:30"# 6 minutes 30 seconds# The [chamber_labels] section is optional. The labels may correspond to# labels used in the experimental design. If specified, these labels# are used in output file names and printed in figures.[chamber_labels]2="A1"3="A2"6="B1"7="B2"