Package: xena-gdc-etl
Extract, transform and loadGDC dataontoUCSC Xena.Table of ContentsDependenciesInstallationBasic usage with command line toolsAdvanced usage with XenaDataset and its subclassesGDC ETL settingsDocumentationDependenciesSpecific versions mentioned below have been tested. Eariler versions may still work but not guaranteed.Python 2.7, 3.5+This pipeline has been tested with python 2.7, 3.5, 3.6 and 3.7. It may also
work with other python 3 versions since it was originally designed to besingle-source Python 2/3 compatible.Requestsv1.2.3Numpyv1.15.0Pandasv0.23.2Jinja2v2.10.1: used for generating metadata JSON.lxmlv4.2.0: used for parsing TCGA phenotype dataxlrdv1.1.0: used for reading TARGET phenotype dataInstallationFirst clone the repository from GitHub by runninggit clonehttps://github.com/ucscXena/xena-GDC-ETL.git. Now,cdintoxena-GDC-ETLdirectory and install the package using pip:pip install .If you are developing the package, you can usepip’s edit mode for
installation:pip install-e..You can also directly usepipto install the package. To get the latest
code from GitHub master branch, run:pip installgit+https://github.com/ucscXena/xena-GDC-ETL.
To get the latest stable version, run:pip installxena-GDC-ETL.Dependencies can be installed either before or after cloning this repository.
You can install them by runningpip install-rrequirements.txt.In general,gdc.pycontains functionalities related to GDC API, which requires no other modules in this package;xena_dataset.pycontains core functionalities for importing data from GDC to Xena and needs thegdc.pymodule in this package;gdc2xena.pydefinesa command line toolwhich requires bothgdc.pymodule andxena_dataset.pymodule in this package;gdc_check_new.pydefinesa command line toolwhich requiresthegdc.pymodule in this package.Basic usage with command line toolsImport selected project(s) and selected type(s) of data from GDC to Xenaxgeetl[-h][-rROOT][-pPROJECTS[PROJECTS...]|-PNOT_PROJECTS[NOT_PROJECTS...]][-tDATATYPE[DATATYPE...]|-TNOT_DATATYPE[NOT_DATATYPE...]][-DDELETE]This tool will perform a full import of dataset(s) into the root directory (specified by the-roption) with a default directory tree. In general, a full import has 3 steps: downloading raw data, making Xena matrix from raw data and generating matrix associated metadata. Data from each step will be saved to corresponding directories, whose structure is like this:root
└── projects
    ├── "Raw_Data"
    │   └── xena_dtype
    │       ├── data1
    │       ├── data2
    │       ├── ...
    │       └── dataN
    └── "Xena_Matrices"
        ├── projects.xena_dtype(1).tsv
        ├── projects.xena_dtype(1).tsv.json
        ├── projects.xena_dtype(2).tsv
        ├── projects.xena_dtype(2).tsv.json
        ├── ...
        ├── projects.xena_dtype(N).tsv
        └── projects.xena_dtype(N).tsv.jsonA dataset is defined by its project and data type. Projects of interest are provided through-por-Poption, and data types of interest are provided through the-tor-Toption. Multiple inputs separated by whitespace are allowed and will be treated separately with all possible combinations. Valid projects should be valid project_id on GDC. Valid data types includes (without quotation marks): ‘htseq_counts’, ‘htseq_fpkm’, ‘htseq_fpkm-uq’, ‘mirna’, ‘masked_cnv’, ‘muse_snv’, ‘mutect2_snv’, ‘somaticsniper_snv’, ‘varscan2_snv’, ‘phenotype’, and ‘survival’. Upper case options (-Por-T) are mutually exclusive with corresponding lower case options, and they are used to define datasets of interest by excluding selections from either all projects on GDC or all supported data types. For example, the following command line imports 3 types of RNA-seq data for all but FM-AD projects from GDC to /home/user/xena_root:mkdir-p/home/user/xena_rootxgeetl-PFM-AD-thtseq_countshtseq_fpkmhtseq_fpkm-uqNotes:Root directory must be existingPlease check the next section foradvanced usage with XenaDataset and its subclasses, if you want to customize the importing process with selected (rather than all possible) combinations of your input projects and data types or selected (rather than all 3) importing step(s).Generate metadata of a xena matrixxgemetadata--projectTCGA-BRCA--datatypehtseq_counts--matrixpath/to/matrix.tsv--release10This tool generates metadata for a xena matrix. For the shown example, metadata
is generated for the matrixmatrix.tsvfor release10, projectTCGA-BRCAand datatypehtsep_counts. Note that, metadata JSON file is
saved at the same directory as thematrix.tsvfile.Check against a list of updated files for affected dataset(s)xgegdc_check_new[-h]URLThis tool takes in a file (either a URL or a local file readable bypandas.read_csv) of table and read one of its columns named as “New File UUID”. It then checks all file UUIDs in this table on GDC and summarize all their associated project(s), data type(s) and analysis workflow type(s). Such tables are usually provided in GDC’s data release note. With the summarized info, you can design specific imports to just update datasets which are updated on GDC. For example, the following command:xgegdc_check_newhttps://docs.gdc.cancer.gov/Data/Release_Notes/DR9.0_files_swap.txt.gzshould give you:analysis.workflow_typecases.project.project_iddata_typeHTSeq-FPKMTARGET-NBLGeneExpressionQuantificationHTSeq-FPKM-UQTARGET-NBLGeneExpressionQuantificationHTSeq-CountsTARGET-NBLGeneExpressionQuantificationShows the current version of xena_gdc_etlxge--versionCheck equality of two xena matricesxgexena-eqlpath/to/matrix1.tsvpath/to/matrix2.tsvThis tool takes path to two xena matrices and output if they are equal or not.Merge xena matricesxgemerge-xena-fpath/to/matrix1.tsvpath/to/matrix2.tsv-thtseq_counts-opath/to/output-nnew_name.tsv-cTCGA-BRCAThis tool merges xena matrices and outputs the merged matrix. For the given
example the tool will mergematrix1.tsvandmatrix2.tsvmatrices and
store the merged matrix inpath/to/outputdirectory with the namenew_name.tsv. Note that, had the argument-nnot been
specified, the merged matrix would have been saved asTCGA-BRCA.htseq_counts.tsv.Advanced usage with XenaDataset and its subclassesTheXenaDatasetclassThough this is not an abstract class, it is designed as a generalized class representing one Xena dataset and its importing process. For doing an import of GDC data, use itssubclasses, which have preloaded with some default settings, might be simpler.A Xena dataset is defined by its study project (cohort) and the type of data in this dataset. A typical importing process has the following 3 steps:Download raw data from the source.Thedownload_mapproperty defines a dict of raw data to be downloaded, with the key being the URL and the value being the path, including the filename, for saving corresponding downloaded file. Thedownloadmethod will read thedownload_mapand perform the downloading, creating non-existing directories as needed. After downloading all files, a list of paths for downloaded files will be recorded in theraw_data_listproperty. Thedownloadmethod needs only a validdownload_map. It will return the object itself, therefore can be chained withtransform.Transform raw data into valid Xena matrix.One assumption for data transformation is that there might be multiple raw data (in theraw_data_list) supporting the single Xena matrix in a dataset. Therefore, thetransformmethod will first merge data and then process merged matrix as needed. It will open the file one by one accordingly (by extension), and read the file object and transform its data with a function defined byread_raw. The list of transformed single data will be merged and processed by a function defined byraws2matrix, which gives the finalized Xena matrix. Thetransformmethod requires a valid list of raw data, besidesread_rawandraws2matrix. A valid list of raw data can be either explicitly defined byraw_data_listor can be derived fromraw_data_dirwith all files underraw_data_dirbeing treated as raw data. It will return the object itself, therefore can be chained withmetadata.Generate metadata for the new Xena matrix.Metadata for Xena matrix is a JSON file rendered by themetadatamethod withmetadata_vars(dict) through Jinja2 frommetadata_template. This JSON file will be saved under the same directory as the matrix, with a filename being the matrix name plus the ‘.json’ postfix. Themetadatamethod requires an existing file of Xena matrix.root_diris both an optional instantiation arguments and a property. By default, it points to the current working directory. It is worth mentioning that the default directory structure mentioned above is implemented in the class. However, you are free to changed the setting with the following properties:Pass an argument forroot_dirduring instantiation or set theroot_dirproperty explicitly after instantiation.Downloaded raw data will be saved underraw_data_dir.Newly transformed Xena matrix will be saved asmatrixundermatrix_dir. The directory path inmatrixhas the priority overmatrix_dir. By default, Xena matrix will be saved under the “matrix_dir” as “<projects>.<xena_dtype>.tsv”.Metadata will always have the specific pattern of name and be together withmatrix(i.e. no way to change this behavior).Build GDC importing pipelines withGDCOmicset,GDCPhenosetorGDCSurvivalsetclassesGDCOmicset,GDCPhenosetandGDCSurvivalsetare subclasses ofXenaDatasetand are preloaded with settings for importing GDC genomic data, TCGA phenotype data on GDC, TARGET phenotype data on GDC and GDC’s survival data respecitively. These settings can be customized by setting corresponding properties described below. For more details, please check thenext sectionand thedocumentation.The script forgdc2xena.pycommand line is a good example for basic usage of these classes. Similar toXenaDataset, a GDC dataset is defined byprojects, which is one or a list of valid GDC “project_id”. ForGDCOmicset, a dataset should also be defined with one of the supportedxena_dtype(find out with the class methodGDCOmicset.get_supported_dtype()). Thexena_dtypeis critical for aGDCOmicsetobject selecting correct default settings. ForGDCPhenosetandGDCSurvivalset, data type are self-explanatory and cannot be changed. Therefore, you can instantiate these classes like this:fromxena_datasetimportGDCOmicset,GDCPhenoset,GDCSurvivalset,GDCAPIPhenosetgdc_omic_cohort=GDCOmicset('TCGA-BRCA','htsep_counts')# Won't check if the ID is of TCGA program or not.tcga_pheno_cohort=GDCPhenoset('TCGA-BRCA')# Won't check if the ID is of TARGET program or not.target_pheno_cohort=GDCPhenoset('TARGET-NBL')gdc_survival_cohort=GDCSurvivalset('TCGA-BRCA')gdc_api_pheno_cohort=GDCAPIPhenoset('CPTAC-3')With such a dataset object, it is fine to calldownload,transformand/ormetadatamethod(s). These methods will use preloaded settings and save files underroot_diraccordingly. You are free to call/chain some but not all 3 methods; just keep in mind the pre-requisites for each method and set related properties properly. Aside fromdirectory related settingsdescribed above, you can change some default importing settings through the following properties.CustomizeGDCOmicsetAttributesUsageType and Format1Default settingsgdc_filterUsed for deriving defaultdownload_mapas the GDC search filters.dict: the key is 1 GDC available file field and the value is either a string or a list, meaning the value of the file field matches a string or number in (a list)CheckGDC download settingsfor details.gdc_prefixUsed for deriving defaultdownload_mapas the GDC search fields.str: 1 GDC available file field whose value will be the prefix of the filename of corresponding downloaded file.CheckGDC download settingsfor details.download_mapUsed by thedownloadmethod for downloading GDC raw data supporting this dataset.dict: the key is download URL and the value is the desired path for saving the downloaded file.Download URLs are in the pattern of “https://api.gdc.cancer.gov/data/<FILE UUID>”, and paths are in the pattern of “<raw_data_dir>/<value of gdc_prefix>.<GDC file UUID>.<file extension>”.read_rawUsed by thetransformmethod when reading a single GDC raw data.callable: takes only 1 file object as its argument and returns an arbitrary result which will be put in a list and passed on toraws2matrix.CheckGDC genomic transform settingsfor detailsraws2matrixUsed by thetransformmethod and responsible for both merging multiple GDC raw data into one Xena matrix and processing new Xena matrix as needed.callable: takes only 1 list ofread_rawreturns as its argument and returns an object (usually a pandas DataFrame) which has ato_csvmethod for saving as a file.CheckGDC genomic transform settingsfor detailsmetadata_templateUsed by themetadatamethod for rendering metadata by Jinja2.jinja2.environment.Templateorstr: ajinja2.environment.Templateused directly by Jinja2; if it’s a string, it is a path to the template file which will be silently read and converted tojinja2.environment.Template.Resourcesmetadata_varsUsed by themetadatamethod for rendering metadata by Jinja2.dict: used directly by Jinja2 which should match variables inmetadata_template.{
    'project_id': <``projects``>,
    'date': <the time of last modification of ``matrix``>,
    'gdc_release': <``gdc_release``>,
    'xena_cohort': <Xena specific cohort name for TCGA data or GDC project_id for TARGET data, with (for both) "GDC " prefix>
}* The first element of the “url” field in metadata will be “gdc_release” URL, and the second will be specific URL for raw data file if there is only 1 raw data file for this dataset; or it will be just “https://api.gdc.cancer.gov/data/”.gdc_releaseUsed by themetadatamethod for rendering metadata, showing the GDC data release of this dataset.str: an URL pointing to corresponding GDC Data Release Note.Current data release version when thegdc_releaseis being used/called, queried through “https://api.gdc.cancer.gov/status”.1. GDC API Available File Fields:https://docs.gdc.cancer.gov/API/Users_Guide/Appendix_A_Available_Fields/#file-fieldsCustomizeGDCPhenosetfor TCGA projectsTCGA phenotype data for Xena includes both clinical data and biospecimen data, asdetailed below. Downloading and transformation of clinical data and biospecimen data are in fact delegated by two independentGDCOmicsetobject respecitively. Corresponding subdatasets can be accessed throughclin_datasetandbio_datasetattributes and hence can be customized as mentioned above. Because of such complexity of TCGA phenotype data, thedownloadandtransformmethods are coded specifically and overrode corresponding methods of the base class,XenaDataset. Customization for downloading and matrix transformation is very limited and should be done in the following steps:Instantiate aGDCPhenoset;Instantiate and customize one or twoGDCOmicsetobjects for clinical data and/or biospecimen data as needed;Assign customizedGDCOmicsetobjects to corresponding attributes,clin_datasetandbio_dataset;Call desired method(s) (downloadand/ortransform).CustomizedownloadstepThis step can be customized only through customizedclin_datasetandbio_dataset, since the whole downloading process is delegated by these two GDCOmicset objects.CustomizetransformstepThe first part oftransformis delegated bytransformmethods ofclin_datasetandbio_dataset. Therefore, the only way to customized this process is to customizeclin_datasetandbio_dataset. How the two matrices are then merged into one Xena phenotype matrix is hard coded and cannot be customized. It is worth noting that if you want to calltransfrombut skip the downloading step, you will need to defineclin_datasetandbio_datasetbefore callingtransform.CustomizemetadatastepDifferent fromdownloadandtransform, there is no special settings for themetadatamethod ofGDCPhenoset. Therefore, similar to that ofGDCOmicset, this step can be customized throughmetadata_template,metadata_varsandgdc_releaseproperties. And to call just themetadatamethod, an existingmatrixis enough.CustomizeGDCPhenosetforTARGET projectsTARGET phenotype data for Xena contains only the clinical data (no biospecimen data), asdetailed below. The importing process is quite similar to that of aGDCOmicset. You can customizeTARGETPhenosetwithdownload_map,read_raw,raws2matrix,metadata_template,metadata_varsandgdc_releasein the same way as that ofGDCOmicset.CustomizeGDCSurvivalsetGDC data supporting Xena survival matrix does not come any GDC files. It comes from the “analysis/survival” endpoint of GDC API. Therefore, thedownloadandtransformmethods are re-designed, overriding those of the base class,XenaDataset. Aside from redefiningdownloadandtransformmethods, there is no simple way to customizedownloadandtransformsteps. You can still calltransformwithoutdownloadby just defining a valid list of raw data withraw_data_listorraw_data_dir. However, only this first file in the list will be read and used.Different fromdownloadandtransform, there is no special settings for themetadatamethod ofGDCSurvivalset. Therefore, similar to that ofGDCOmicset, this step can be customized throughmetadata_template,metadata_varsandgdc_releaseproperties. To call just themetadatamethod, an existingmatrixis enough.CustomizeGDCAPIPhenosetThe data for this class comes from GDC API only. Therefore, thedownloadandtransformmethods are re-designed, overriding those of the base class,XenaDataset. Aside from redefiningdownloadandtransformmethods,
there is no simple way to customizedownloadandtransformsteps.Different fromdownloadandtransform, there is no special settings
for themetadatamethod ofGDCAPIPhenoset. Therefore, similar to that
ofGDCOmicset, this step can be customized throughmetadata_template,metadata_varsandgdc_releaseproperties. To call just themetadatamethod, an existingmatrixis enough.GDC ETL settingsSettings for downloading/getting raw data (files) from GDCxena_dtypeGDC API endpointGDC data filterFile count/LevelGDC file field for filename prefixdata_typeanalysis.workflow_typehtseq_countsdataGene Expression QuantificationHTSeq - Counts1/Sample vialcases.samples.submitter_idhtseq_fpkmdataGene Expression QuantificationHTSeq - FPKM1/Sample vialcases.samples.submitter_idhtseq_fpkm-uqdataGene Expression QuantificationHTSeq - FPKM-UQ1/Sample vialcases.samples.submitter_idmirnadatamiRNA Expression QuantificationBCGSC miRNA Profiling1/Sample vialcases.samples.submitter_idmirna_isoformdataIsoform Expression QuantificationBCGSC miRNA Profiling1/Sample vialcases.samples.submitter_idcnvdataCopy Number SegmentDNAcopy1/Sample vialcases.samples.submitter_idmasked_cnvdataMasked Copy Number SegmentDNAcopy1/Sample vialcases.samples.submitter_idgisticdataGene Level Copy Number ScoresGISTIC - Copy Number Score1/Projectsubmitter_idstar_countsdataSTARCountsSTAR - Counts1/Sample vialcases.samples.submitter_idmuse_snvdataMasked Somatic MutationMuSE Variant Aggregation and Masking1/Projectsubmitter_idmutect2_snvdataMasked Somatic MutationMuTect2 Variant Aggregation and Masking1/Projectsubmitter_idsomaticsniper_snvdataMasked Somatic MutationSomaticSniper Variant Aggregation and Masking1/Projectsubmitter_idvarscan2_snvdataMasked Somatic MutationVarScan2 Variant Aggregation and Masking1/Projectsubmitter_idclinicaldataClinical SupplementN/A0 or 1/Casecases.submitter_idbiospecimendataBiospecimen SupplementN/A1/Casecases.submitter_idsurvivalanalysis/survivalN/A (filtered by just the “project.project_id”)1 Record/Case (Non-file)N/A (filename will be “<projects>.GDC_survival.tsv”)Settings for transform “Omic” data into Xena matrixxena_dtypeRaw data has header?Select columns (in order)Row indexSkip rows start with?Merge into matrix asProcess matrixhtseq_countsNo1, 2
[Ensembl_ID, Counts]Ensembl_ID_1 new column based on indexAverage if there are multiple data from the same sample vial;log2(counts + 1)htseq_fpkmNo1, 2
[Ensembl_ID, Counts]Ensembl_ID_1 new column based on indexAverage if there are multiple data from the same sample vial;log2(counts + 1)htseq_fpkm-uqNo1, 2
[Ensembl_ID, Counts]Ensembl_ID_1 new column based on indexAverage if there are multiple data from the same sample vial;log2(counts + 1)mirnaYes1, 3
[miRNA_ID, RPM]miRNA_IDN/A1 new column based on indexAverage if there are multiple data from the same sample vial;log2(counts + 1)mirna_isoformYes2, 4
[isoform_coords, RPM]isoform_coordsN/A1 new column based on indexAverage if there are multiple data from the same sample vial;log2(counts + 1)cnvYes2, 3, 4, 6
[Chromosome, Start, End, Segment_Mean]sampleN/ANew rows based on column nameRename columns as:{
    'Chromosome': 'Chrom',
    'Segment_Mean': 'value'
}masked_cnvYes1, 2, 3, 5
[Chromosome, Start, End, Segment_Mean]sampleN/ANew rows based on column nameRename columns as:{
    'Chromosome': 'Chrom',
    'Segment_Mean': 'value'
}gisticYes1
[Ensembl_ID]Ensembl_ID_N/ADrop “Gene ID” and “Cytoband” column;Map “samples.portions.analytes.aliquots.aliquot_id” into “samples.submitter_id” using GDC API and use it as index.muse_snv
mutect2_snv
somaticsniper_snv
varscan2_snvYes13, 37, 5, 6, 7, 40, 42, 52, 1, 11, 16, 111
[Tumor_Seq_Allele2, HGVSp_Short, Chromosome, Start_Position, End_Position, t_depth, t_alt_count, Consequence, Hugo_Symbol, Reference_Allele, Tumor_Sample_Barcode, FILTER]N/A#N/ACalculate variant allele frequency (dna_vaf) by “t_alt_count”/”t_depth”;Delete “t_alt_count” and “t_depth” columns;Trim “Tumor_Sample_Barcode” to sample vial level;Rename columns as:{
    'Hugo_Symbol': 'gene',
    'Chromosome': 'chrom',
    'Start_Position': 'start',
    'End_Position': 'end',
    'Reference_Allele': 'ref',
    'Tumor_Seq_Allele2': 'alt',
    'Tumor_Sample_Barcode': 'sampleid',
    'HGVSp_Short': 'Amino_Acid_Change',
    'Consequence': 'effect',
    'FILTER': 'filter'
}star_countsYes1, 2
[Ensembl_ID, Counts]Ensembl_ID_1 new column based on indexAverage if there are multiple data from the same sample vial;log2(counts + 1)Settings for transform phenotype data into Xena matrixGDC programGDC raw dataRaw data formatSingle data file transformationMerge and matrix processingTCGAClinical Supplement and Biospecimen SupplementBCR XMLFor clincial data, info will be extracted and organized into a per patient based pandas DataFrame. It will have a column named “bcr_patient_barcode” which will be used to join with biospecimen matrix later on.The XML scheme are quite different for different projects. Therefore, to get as much info as possible while still keeping things clear, texts, if any, from all elements that have non-element children are extracted first. After such a “dirty” extraction, two clean ups will be done:For “race” info, it will be converted into a comma separated list of races, in case there are more than one entry in <clin_shared:race_list> in the clinical XML.When there is one or more follow ups, the most recent follow up will be find out. All info in the most recent follow up will be used to replace/add to previously extracted matrix.For biospecimen data, there is one coherent XML scheme for all TCGA projects. There are two parts to be considered for biospecimen data: per sample/sample specific data and patient data (which is common for all samples). Info from both parts will be extracted and finally organized into a per sample based matrix, having a column named “bcr_patient_barcode”, which will be used to join with clinical matrxi later on. In general, info extraction has the following 3 steps:Common patient data will be extracted first, including texts from direct children of <admin:admin> and <bio:patient>. A new field of “primary_diagnosis” will be added by mapping “disease_code” toTCGA study name.Samples from <bio:patient/bio:samples> will be processed and have comman patient data attached one by one. Non-empty texts from direct children of sample will be extracted, i.e. details from nodes like <bio:portions> will be dropped. Samples havingtype code 10are dropped.A column of “bcr_patient_barcode” from <bio:patient/shared:bcr_patient_barcode> will be added to the final biospecimen matrix (same for the whole table).Multiple clinical data are concatenated directly by row with all empty columns removed.Multiple biospecimen data are concatenated directly by row with all empty columns removed.Merged clinical matrix and merged biospecimen matrix are further merged on “bcr_patient_barcode”. For conflict/overlapping columns, non-empty value from the clinical data has the priority.TARGETClinical Supplement onlyXLSXThe excel file is converted to a pandas DataFrame.Multiple DataFrames will be concatenated directly by row, and arriage return and line feed are replaced by a single space.Clinical data is per case(patient) based, while Xena phenotype matrix is per sample based. All related samples for each case/patient will be identified and phenotype data will be mapped to corresponding samples.Settings for transform survival data into Xena matrixGDC survival data is returned as JSON from GDC API. During the downloading process, it can and will be converted directly to pandas DataFrame and saved as tab delimited table. During transformation, columns in “primary” Xena survival matrix can be mapped directly (without further processing/calculation) from the raw table like this:Primary Xena columnGDC source columnOS.timetimeOScensored_PATIENTsubmitter_idGDC survival data is per case(patient) based and so is “primary” Xena survival matrix, while Xena survival matrix is per sample based. All related samples for each case/patient will be identified and survival data will be mapped to corresponding samples.CPTAC-3 CohortCPTAC-3 data consists of RNAseq data (as discussed inGDCOmicset) and
clinical data from the API. The cases and expand for clinical data are
defined in theconstants.pyfile.DocumentationCheck documentation for GDC module and Xena Dataset modulehere.