Package: ukbcc
ukbccAuthors: Isabell Kiral, Nathalie Willems, Benjamin GoudeyOverviewTool for curation of UK Biobank data to generate cohorts. The tool can filter the main and associated datasets (e.g general practioner clinical data) based on search terms provided by the user. It can be used interactively through a web-based interface, a command line interface, or imported as a module and integrated into a broader pipeline. Additional functionality, such as automatically downloading large data files (bulk data) is also supported.Known IssuesCurrently there are known issues in running very long and complex queries (e.g > 800 datafield:value combinations), resulting in errors from the pandas library. We are working on these issues, and will update subsequent releases with these bug fixes soon.PrerequisitesThe provided tool is developed for Python version 3 and can be imported as a package as described below.Theinteractive modeis developed to be run in the command line and has been developed for and tested on MacOS and Linux systems.InstallationInstalling using pip (or pip3):$ pip3 install ukbccOr clone the repo:$ git clone https://github.com/tool-bin/ukbcc.git
$ python3 setup.py installNB: We strongly recommend using a virtual environment when installing this package and its dependencies. Please see this link for further information:https://docs.python.org/3/tutorial/venv.htmlUsageThere are two ways to use with this module:Running the module from the command line and leveraging theinteractive modefeatures to dynamically generate cohorts.Importing the module into an existing pipeline, and using the functions developed to interact with the UKBB databases.There is more detailed information inour paper.DownloadsIn order to make full use of this module, you will need to download the following files:main_dataset.csv: The main dataset as downloaded from UK Biobank. Please follow UKBB instructions to obtain this file. Utilities to download this file can be found here:http://biobank.ctsu.ox.ac.uk/crystal/download.cgi.Please note that the module assumes that the main dataset is generated as a CSV file!Showcase_Data_Dictionary.csv: This file encodes all the different datafields and their values within the UK Biobank Showcase. The file can be downloaded here:https://raw.githubusercontent.com/tool-bin/ukbcc/master/data_files/showcase.csvShowcase_Codings.csv: This file contains all the coding schemes used in the UK Biobank Showcase. This file can be downloaded here:https://raw.githubusercontent.com/tool-bin/ukbcc/master/data_files/codings.csvreadcodes.csv: A file linking readcodes to descriptions for the GP Clinical data, available from the UKBB data portal. This file can be found in the data_files directory within this repo. A link to download this file is given here:https://raw.githubusercontent.com/tool-bin/ukbcc/master/data_files/readcodes.csvgp_clinical.txt: The full general practioner (GP) clinical data that forms part of the primary care dataset. The full table (gp_clinical) can be downloaded from the UKBB data portal website. Instructions to download this table are provided below.Downloading the GP clinical datasetThe GP clinical dataset can be downloaded directly from the UK Biobank showcase website, through the Data Portal webpage. Instructions for how to download this file are provided below:Log into the UK Biobank showcase website (https://bbams.ndph.ox.ac.uk/ams/resApplications)Navigate to your Project by clicking the "Project" button on the left-hand side of the pageClick on the "Data" tab on the right of the pageClick on the "Go to Showcase" tab - this will take you to the UK Biobank Showcase landing pageClick the "Data Portal" tab and click on the "Connect" button. Note! The user requires access permissions to access the Data Portal webpage - if you do not see this button, you do not have access to this page.Click on the "Table Download" buttonType in "gp_clinical" into the search bar and click the "Fetch Table" buttonClick on the generated link. This will automatically start downloading the gp_clinical table, as tab-separated plain-text file.Provide the path and name of the GP clinical file to the main.py module in order to use this dataset within theinteractivemode of the UKBCC moduleInteractive modeThere are two ways to runinteractive mode:Through a web-based interfaceThrough a command-line interfaceWeb-based interfaceIn order to use the web-based interface, please run the following command from the command line:$ ukbccThe above command will setup the web-based interface and generates a web address where this can be accessed.Follow the instructions on the website to proceed with cohort generation.NB: The web-based interface is built using Plotly Dash, which uses Flask in order to serve the web application. The Flask library uses the default werkzeug development server, which has not been tested for security or performance. Consequently, you will see the following warning when running this command in the command line:"Warning: This is a development server. Do not use app.run_server
in production, use a production WSGI server like gunicorn instead."We recommend using a web-server if you would like to run the UKBCC tool in a production environment. Popular choices include:gunicorn:https://gunicorn.org/uWSIG:https://uwsgi-docs.readthedocs.io/en/latest/Command-line interfaceIn order to use the command-line interface functionality, the module can simple be called from the command line.To start the configuration process, type:$ukbcc_cliYou will be asked to provide certain information. Make sure to provide the full or relative path (and filename if asked).>>Pleasespecifydirectoryforconfigfile[`.`forcurrentdirectory]:
>>Pleasespecifythefullpathandnameofmaindataset:
>>PleasespecifythefullpathandnameoftheGPclinicaldataset-ifyoudonotwanttoquerythisdataset,pleasetype"No">>Pleasespecifythenameofthefiletostorethelistofidsforthecohort:The created config and credentials files can be reused in subsequent runs, using flags:$ukbcc--config./config.confYou will be asked to specify search terms used to generate the cohort (e.gglaucoma,optical cohorence tomography). Provide them as a comma-separated list:>>Pleaseentercomma-separatedsearchterms:glaucoma,opticalcoherencetomographyGo through all fields that may be relevant and decide if a field or condition will be of interest or not. This process can take some time.For all included fields and conditions, you will then be asked to provide logical pointers. Choose if all participants should have a certain condition, none of them, or if every participant should have any of a number of different conditions. Refer to the graphic below for a visual explanation of the logic pointers.Once these selections are made, the module will query the UKBB databases in order to generate a list participants IDs that match the specified criteria. These IDs can then be used in downstream processing and analysis pipelines, for example generating statistics about the cohort (statsmodule), or downloading bulk imaging files for the individuals within the cohort (bulkmodule).There are 3 files that will be created by running the module ininteractive mode:cohort_criteria -- this file contains the dictionary data structure that is created as part of the selection of desired datafield:code combinations (step 2 above)cohort_criteria_updated -- this file contains the update dictionary data structure with the appropriate conditional logic (step 3 above)out_filename -- this file contains the list of participant IDs that match the criteria for the cohort. out_filename is a placeholder for the file name specified during the Configuration process (step 1)The module will write all the relevant files to the specified output directory. As such, the generated dictionary data structure can be updated and reused in other pipeplines (e.g when importing the module within an existing pipeline).Stand-alone modeThe ukbcc module uses dictionaries in order to represent the various datafield:code combinations and conditional logic to be applied in generating a cohort.
This dictionary will be automatically generated through theinteractive mode.
Alternatively, the user can write this dictionary themselves, and use thequerysubmodule to further interact with UKBB databases.
Further information about the expected structure of the dictionary is provided in the docstrings of the functions within this module.
It is recommended the user leverage theinteractive modeif using the ukbcc module for the first time.To learn about how to use modules in this package in your existing pipeline, see example-module notebook in the examples directory in this repo.ContributingAs a collaborator, please make a branch and create a pull request when ready.
To contribute otherwise, please fork directory and create pull requests.
Github issues are also welcome.CitationIf you found this tool useful in your work, please use the following citation:UKBCC: a cohort curation package for UK BiobankIsabell Kiral, Nathalie Willems, Benjamin GoudeybioRxiv 2020.07.12.199810; doi:https://doi.org/10.1101/2020.07.12.199810