Package: ukbparse
Noteukbparsehas been superseded byfunpackand will no longer
be developed. Head tohttps://git.fmrib.ox.ac.uk/fsl/ukbparsefor
more information.ukbparseis a Python library for pre-processing of UK BioBank data.ukbparseis developed at the Wellcome Centre for Integrative
Neuroimaging (WIN@FMRIB), University of Oxford.ukbparseis in no way
endorsed, sanctioned, or validated by theUK BioBank.ukbparsecomes bundled with metadata about the variables present in UK
BioBank data sets. This metadata can be obtained from theUK BioBank
online data showcaseInstallationInstallukbparsevia pip:pip install ukbparseOr fromconda-forge:conda install -c conda-forge ukbparseIntroductory notebookTheukbparse_democommand will start a Jupyter Notebook which introduces
the main features provided byukbparse. To run it, you need to install a
few additional dependencies:pip install ukbparse[demo]You can then start the demo by runningukbparse_demo.NoteThe introductory notebook usesbash, so is unlikely to work on
Windows.UsageGeneral usage is as follows:ukbparse [options] output.tsv input1.tsv input2.tsvYou can get information on all of the options by typingukbparse--help.Options can be specified on the command line, and/or stored in a configuration
file. For example, the options in the following command line:ukbparse \
  --overwrite \
  --import_all \
  --log_file log.txt \
  --icd10_map_file icd_codes.tsv \
  --category 10 \
  --category 11 \
  output.tsv input1.tsv input2.tsvCould be stored in a configuration fileconfig.txt:overwrite
import_all
log_file       log.txt
icd10_map_file icd_codes.tsv
category       10
category       11And then executed as follows:ukbparse -cfg config.txt output.tsv input1.tsv input2.tsvCustomisingukbparsecontains a large number of built-in rules which have been
specifically written to pre-process UK BioBank data variables. These rules are
stored in the following files:ukbparse/data/variables_*.tsv: Cleaning rules for individual variablesukbparse/data/datacodings_*.tsv: Cleaning rules for data codingsukbparse/data/types.tsv: Cleaning rules for specific typesukbparse/data/processing.tsv: Processing stepsYou can customise or replace these files as you see fit. You can also pass
your own versions of these files toukbparsevia the--variable_file,--datacoding_file,--type_fileand--processing_filecommand-line
options respectively.ukbparsewill load all variable and datacoding files,
and merge them into a single table which contains the cleaning rules for each
variable.Finally, you can use the--no_builtinsoption to bypass all of the
built-in cleaning and processing rules.OutputThe main output ofukbparseis a plain-text tab-delimited[*]_ file which
contains the input data, after cleaning and processing, potentially with
some columns removed, and new columns added.If you used the--non_numeric_fileoption, the main output file will only
contain the numeric columns; non-numeric columns will be saved to a separate
file.You can use any tool of your choice to load this output file, such as Python,
MATLAB, or Excel. It is also possible to pass the output back intoukbparse.[*]You can change the delimiter via the--tsv_sep/-tsoption.Loading output into MATLABIf you are using MATLAB, you have several options for loading theukbparseoutput. The best option isreadtable, which will load column names, and
will handle both non-numeric data and missing values.  Usereadtablelike
so:data = readtable('out.tsv', 'FileType', 'text');Thereadtablefunction returns atableobject, which stores each column
as a separate vector (or cell-array for non-numeric columns). If you are only
interested in numeric columns, you can retrieve them as an array like this:rawdata =  data(:, vartype('numeric')).Variables;Thereadtablefunction will potentially rename the column names to ensure
that they are are valid MATLAB identifiers. You can retrieve the original
names from thetableobject like so:colnames        = data.Properties.VariableDescriptions;
colnames        = regexp(colnames, '''(.+)''', 'tokens', 'once');
empty           = cellfun(@isempty, colnames);
colnames(empty) = data.Properties.VariableNames(empty);
colnames        = vertcat(colnames{:});If you have used the--description_fileoption, you can load in the
descriptions for each column as follows:descs = readtable('descriptions.tsv', ...
                  'FileType', 'text', ...
                  'Delimiter', '\t',  ...
                  'ReadVariableNames',false);
descs = [descs; {'eid', 'ID'}];
idxs  = cellfun(@(x) find(strcmp(descs.Var1, x)), colnames, ...
                'UniformOutput', false);
idxs  = cell2mat(idxs);
descs = descs.Var2(idxs);TestsTo run the test suite, you need to install some additional dependencies:pip install ukbparse[test]Then you can run the test suite usingpytest:pytestCitingIf you would like to citeukbparse, please refer to itsZenodo page.