Package: ukbiobank-loaders
ukbiobank-loadersThis repository provides an easy way to load UK Biobank data. It is composed of a pre-processing script, which converts the UK Biobank data into parquets that are easier to read,
and a library that provides different methods to access the data.InstallationTo install this package, simply runpipinstallukbiobank-loadersPlease note that python 3.7 or newer is needed.UsageWe will now describe how to use this library. Please note that data can be read from both local directories, and aws s3 directories.Pre-processingThese are the UK Biobank files that are needed in order to run the pre-processing, all saved in the same directory <DATA_FOLDER>:death.txt
death_cause.txt
gp_clinical.txt
gp_scripts.txt
hesin.txt
hesin_diag.txt
hesin_oper.txtAdditionally, also the withdrawn consent file is needed:withdrawn_consent.txtFrom the terminal, runupdate_data.py--raw_dir<DATA_FOLDER>--withdrawn_file<WITHDRAWN_CONSENT_FILE_PATH>--out_dir<OUTPUT_DIR_FOLDER>The processed data will be saved in a folder named<OUTPUT_DIR_FOLDER>/final.We found this process to take about 14 minutes in a pod composed of 4 CPUs and 32GB of RAM. If the process is Killed, it might be
because there is not enough RAM available.Accessing the dataThis is a simple example on how to use the library. Specific documentation about the methods is given below.>>>fromukbb_loaders.loadersimportload
>>>dl=load.DataLoader(data_dir="<OUTPUT_DIR_FOLDER>/final")>>>dl.get_hospital_data("icd10")date_of_visitsourcefeaturevalue
eid681986-04-22icd10N1811681945-05-03icd10N1811681950-04-03icd10N1811681966-08-07icd10N1811671991-03-12icd10N1811..............73NaTicd10N1811481997-06-20icd10N1811481945-03-05icd10N1811481956-02-25icd10N1811481981-04-08icd10N1811Documentation for ukbb_loaders.loadersTable of Contentsukbb_loaders.utilities.utilload_lookupload_mapperukbb_loaders.loaders.loadDataLoader__init__get_hospital_dataget_death_dataget_gp_clinical_dataget_gp_medication_dataukbb_loaders.utilities.utilload_lookupdefload_lookup(lookup_name:str)->pd.DataFrameLoads lookup table.Arguments:lookup_namestr- The name of the lookup table to be loaded.Returns:(pd.DataFrame)- The lookup table of interest.Example:
Load lookup of ICD10 diagnosis codes:load_lookup("ehr_diagnosis_icd10")load_mapperdefload_mapper(mapper_name:str)->pd.DataFrameLoads ontology mapper.Arguments:mapper_namestr- The name of the mapper to be loaded.Returns:(pd.DataFrame)- The mapper of interest.Example:
Load mapping from ICD10 codes to Phecodes:load_mapper("icd10_to_phecodes")ukbb_loaders.loaders.loadLoaders for versioned UKBB data.DataLoader ObjectsclassDataLoader()__init__def__init__(data_dir:str)Class for loading UKBB data.Arguments:data_dirstr- The path to the directory containing the processed data.
Note that on Windows the path must have forward-slashes,
e.g.  "C:/Users/john/Documents/data_dir"get_hospital_datadefget_hospital_data(source:Union[str,List[str]],level=None,patient_list:np.ndarray=None)->pd.DataFrameMethod that fetches hospital data for the UKBB population.Arguments:sourcestr or list- The coding/representation/source we would like to fetch.
It needs to be one or more of:icd10- for fetching all icd10 related diagnoses.icd9- for fetching all icd9 related diagnoses.opcs3- for fetching all opcs4 related operational codes.opcs4- for fetching all opcs4 related operational codes.levellist or string- The level/significance of diagnoses we would like to fetch.
It needs to be one or both of:primary- for fetching only the primary code related to one diagnosis.secondary- for fetching all the secondary (complementary) codes for one
diagnosis.external- For fetching diagnosis codes from external sources.
Defaults to all of them.patient_listnp.ndarray- The patients to fetch characteristics for. If this is empty,
all UKBB patients will be used.Returns:dfpd.DataFrame- A long canonical dataframe with patients as the index and the
following columns:date_of_visit: pandas datetime for each hospital visitfeature: the different codes used (e.g. the different icd10 codes)source: this is relevant to the source the feature is referring to (e.g. icd10)value: the occurrence value for each row combination (initially 1.)get_death_datadefget_death_data(level=None,patient_list:np.ndarray=None)->pd.DataFrameMethod that fetches death information for the UKBB population.Arguments:levellist or string- The level/significance of deaths we would like to fetch.
It needs to be one or both of: primary (main reason of death), secondary. Defaults to both.patient_listnp.ndarray- The patients to fetch characteristics for.
If this is empty, all UKBB patients will be used.Returns:dfpd.DataFrame- A long canonical dataframe with patients as the index and all
recorded death information including death date in the right format.get_gp_clinical_datadefget_gp_clinical_data(source=None,patient_list:np.ndarray=None)Method that fetches GP diagnosis information for the UKBB population.Arguments:sourcestr or list- Whether to load read_2, read_3 or both. Defaults to both.patient_listnp.ndarray- The patients to fetch characteristics for.
If this is empty, all UKBB patients will be used.Returns:dfpd.DataFrame- A long canonical dataframe with patients as the index and all
recorded gp information including date in the right format.get_gp_medication_datadefget_gp_medication_data(patient_list:np.ndarray=None)->pd.DataFrameMethod that fetches GP medication data for the UKBB population.Arguments:patient_listnp.ndarray- The patients to fetch medication data for.
If this is empty, all UKBB patients will be used.Returns:dfpd.DataFrame- A canonical long dataframe with patients as the index and
features as columns.