Package: cdapython
CDA PythonThis library sits on top of the machine generatedCDA Python Clientand offers some syntactic
sugar to make it more pleasant to query the CDA.Read The DocsCDA Read The DocsLaunch in BinderTo try out the example notebook inMyBinder.orgwithout having to install anything, just click on the logo below. This will
launch a Jupyter Notebook instance with our example notebook ready to run.For Testers use this BinderClick on the logo below. This will
launch a Jupyter Notebook instance with our example notebook ready to run.Install the CDA Python library locally:Download and install docker click thislinkor copy urlhttps://www.docker.com/products/docker-desktopto your BrowserOpen Terminal or PowerShell a and navigate to cda-python folder then we will run a docker commanddocker compose up --buildOpen a Browser to this urlhttp://localhost:8888and you are up and running.To change the Portedit the.envfile NOTEBOOK_PORTTo Stop the container from running return to the terminal where the cdapython project is on your keyboard you will clickControl C to stopthe container .To delete the container use this command in the cdapython project directory.docker compose downPip installAlternatively, CDA Python can be installed usingpip. However, this requires python >= 3.6 on your system. To check your version at the command-line runpython -V. To update your version you can download fromhttps://www.python.org/downloads/additional python installation help can be foundhere. Once you have the proper python version, you can run CDA using:pip install git+https://github.com/CancerDataAggregator/cda-python.gitNOTE: We recommend the docker method because pip installation can be a bit more cumbersome, and will not be as closely monitored as the docker installation.BasicsWe will now show you the basic structure ofCDA pythonthrough the use of the most commands:columns(): show all available columns in the table,unique_terms(): for a given column show all unique terms,Q: Executes this query on the public CDA server, andQ.sql: allows you to enter SQL style queries.query: allows you to write long form Q statments with out chaining(Also see exampleIPython notebook)from cdapython import Q, columns, unique_terms


columns() # List column names eg:
# ['days_to_birth',
#  'race',
#  'sex',
#  'ethnicity',
#  'id',
#  'ResearchSubject',
#  'ResearchSubject.Diagnosis',
#  'ResearchSubject.Diagnosis.morphology',
#  'ResearchSubject.Diagnosis.tumor_stage',
#  'ResearchSubject.Diagnosis.tumor_grade',
#  'ResearchSubject.Diagnosis.Treatment',
#  'ResearchSubject.Diagnosis.Treatment.type',
#  'ResearchSubject.Diagnosis.Treatment.outcome',


unique_terms("ResearchSubject.primary_disease_type") # List unique terms for this column eg:
# [None,
#  'Acinar Cell Neoplasms',
#  'Adenomas and Adenocarcinomas',
#  'Adnexal and Skin Appendage Neoplasms',
#  'Basal Cell Neoplasms',
#  'Blood Vessel Tumors',
#  'Breast Invasive Carcinoma',
#  'Chromophobe Renal Cell Carcinoma',
#  'Chronic Myeloproliferative Disorders',
#  'Clear Cell Renal Cell Carcinoma',
#  'Colon Adenocarcinoma',
# ...

q1 = Q('ResearchSubject.primary_disease_type = "Adenomas and Adenocarcinomas"')
r = q1.run()                                 # Executes this query on the public CDA server

# r = q1.run(host="http://localhost:8080")   # Executes on local instance of CDA server
# r = q1.run(limit=2)                        # Limit to two results per page


r.sql   # Return SQL string used to generate the query e.g.
# "SELECT * FROM gdc-bq-sample.cda_mvp.v1, UNNEST(ResearchSubject) AS _ResearchSubject WHERE (_ResearchSubject.primary_disease_type = 'Adenomas and Adenocarcinomas')"

print(r) # Prints some brief information about the result page eg:
#
# Query: SELECT * FROM gdc-bq-sample.cda_mvp.v1, UNNEST(ResearchSubject) AS _ResearchSubject WHERE (_ResearchSubject.# primary_disease_type = 'Adenomas and Adenocarcinomas')
# Offset: 0
# Limit: 2
# Count: 2
# More pages: Yes


r[0] # Returns nth result of this page as a Python dict e.g.
#
# {'days_to_birth': None,
#  'race': None,
#  'sex': None,
#  'ethnicity': None,
#  'id': '4d54f72c-e8ac-44a7-8ab9-9f20001750b3',
#  'ResearchSubject': [{'Diagnosis': [],
#    'Specimen': [],
#    'associated_project': 'CGCI-HTMCP-CC',
#    'id': '4d54f72c-e8ac-44a7-8ab9-9f20001750b3',
#    'primary_disease_type': 'Adenomas and Adenocarcinomas',
#    'identifier': [{'system': 'GDC',
#      'value': '4d54f72c-e8ac-44a7-8ab9-9f20001750b3'}],
#    'primary_disease_site': 'Cervix uteri'}],
#  'Diagnosis': [],
#  'Specimen': [],
#  'associated_project': 'CGCI-HTMCP-CC',
#  'primary_disease_type': 'Adenomas and Adenocarcinomas',
#  'identifier': [{'system': 'GDC',
#    'value': '4d54f72c-e8ac-44a7-8ab9-9f20001750b3'}],
#  'primary_disease_site': 'Cervix uteri'}


r.pretty_print(0) # Prints the nth result nicely
#
# { 'Diagnosis': [],
#   'ResearchSubject': [ { 'Diagnosis': [],
#                          'Specimen': [],
#                          'associated_project': 'CGCI-HTMCP-CC',
#                          'id': '4d54f72c-e8ac-44a7-8ab9-9f20001750b3',
#                          'identifier': [ { 'system': 'GDC',
#                                            'value': '4d54f72c-e8ac-44a7-8ab9-9f20001750b3'}],
#                          'primary_disease_site': 'Cervix uteri',
#                          'primary_disease_type': 'Adenomas and '
#                                                  'Adenocarcinomas'}],
#   'Specimen': [],
#   'associated_project': 'CGCI-HTMCP-CC',
#   'days_to_birth': None,
#   'ethnicity': None,
#   'id': '4d54f72c-e8ac-44a7-8ab9-9f20001750b3',
#   'identifier': [ { 'system': 'GDC',
#                     'value': '4d54f72c-e8ac-44a7-8ab9-9f20001750b3'}],
#   'primary_disease_site': 'Cervix uteri',
#   'primary_disease_type': 'Adenomas and Adenocarcinomas',
#   'race': None,
#   'sex': None}


r2 = r.next_page()  # Fetches the next page of results

print(r2)

# Query: SELECT * FROM gdc-bq-sample.cda_mvp.v1, UNNEST(ResearchSubject) AS _ResearchSubject WHERE (_ResearchSubject.# primary_disease_type = 'Adenomas and Adenocarcinomas')
# Offset: 2
# Limit: 2
# Count: 2
# More pages: Yes

r1 = Q.sql("""
SELECT
*
FROM gdc-bq-sample.cda_mvp.v1, UNNEST(ResearchSubject) AS _ResearchSubject
WHERE (_ResearchSubject.primary_disease_type = 'Adenomas and Adenocarcinomas')
""")

r1.pretty_print(0)
#
#{ 'Diagnosis': [],
#  'ResearchSubject': [ { 'Diagnosis': [],
#                         'Specimen': [],
#                         'associated_project': 'CGCI-HTMCP-CC',
#                         'id': '4d54f72c-e8ac-44a7-8ab9-9f20001750b3',
#                         'identifier': [ { 'system': 'GDC',
#                                           'value': '4d54f72c-e8ac-44a7-8ab9-9f20001750b3'}],
#                         'primary_disease_site': 'Cervix uteri',
#                         'primary_disease_type': 'Adenomas and '
#                                                 'Adenocarcinomas'}],
#  'Specimen': [],
#  'associated_project': 'CGCI-HTMCP-CC',
#  'days_to_birth': None,
#  'ethnicity': None,
#  'id': 'HTMCP-03-06-02177',
#  'id_1': '4d54f72c-e8ac-44a7-8ab9-9f20001750b3',
#  'identifier': [ { 'system': 'GDC',
#                    'value': '4d54f72c-e8ac-44a7-8ab9-9f20001750b3'}],
#  'primary_disease_site': 'Cervix uteri',
#  'primary_disease_type': 'Adenomas and Adenocarcinomas',
#  'race': None,
#  'sex': None}


query('ResearchSubject.identifier.system = "GDC" FROM ResearchSubject.primary_disease_type = "Ovarian Serous Cystadenocarcinoma" AND ResearchSubject.identifier.system = "PDC"')
result = q1.run()Comparison operatorsThe following comparsion operators can be used with theQcommand:operatorDescriptionQ.sql required?=condition equalsno!=condition is not equalno<condition is less thanno>condition is greater thanno<=condition is less than or equal tono>=condition is less than or equal tonolikesimilar to = but always wildcards ('%', '_', etc)yesincompares to a setyesadditionally, more complex SQL can be used with theQ.sqlcommand.A simple querySelect data from TCGA-OV project, with donors over age 50Quick formfrom cdapython import Q

q1 = Q('ResearchSubject.Diagnosis.age_at_diagnosis > 50*365')
q2 = Q('ResearchSubject.associated_project = "TCGA-OV"')

q = q1.AND(q2)
r = q.run()

print(r)

# Query: SELECT * FROM gdc-bq-sample.cda_mvp.v1, UNNEST(ResearchSubject) AS _ResearchSubject, UNNEST(_ResearchSubject.Diagnosis) AS # _Diagnosis WHERE ((_Diagnosis.age_at_diagnosis > 50*365) AND (_ResearchSubject.associated_project = 'TCGA-OV'))
# Offset: 0
# Limit: 1000
# Count: 461
# More pages: No

r.pretty_print(2)
# { 'Diagnosis': [ { 'Treatment': [ { 'outcome': None,
#                                     'type': 'Radiation Therapy, NOS'},
#                                   { 'outcome': None,
#                                     'type': 'Pharmaceutical Therapy, NOS'}],
#                    'age_at_diagnosis': 28779,
#                    'id': 'dc8af98b-03cb-5817-84fa-d86a7f2df8c6',
#                    'morphology': '8441/3',
#                    'primary_diagnosis': 'Serous cystadenocarcinoma, NOS',
#                    'tumor_grade': 'not reported',
#                    'tumor_stage': 'not reported'}],
#   'ResearchSubject': [ { 'Diagnosis': [ { 'Treatment': [ { 'outcome': None,
#                                                            'type': 'Radiation '
#                                                                    'Therapy, '
#                                                                    'NOS'},
#                                                          { 'outcome': None,
#                                                            'type': 'Pharmaceutical '
#                                                                    'Therapy, '
#                                                                    'NOS'}],
#                                           'age_at_diagnosis': 28779,
#                                           'id': 'dc8af98b-03cb-5817-84fa-d86a7f2df8c6',
#                                           'morphology': '8441/3',
#                                           'primary_diagnosis': 'Serous '
#                                                                'cystadenocarcinoma, '
#                                                                'NOS',
#                                           'tumor_grade': 'not reported',
#                                           'tumor_stage': 'not reported'}],
# ...Any given part of a query is expressed as a string of three parts separated by spaces:Q('ResearchSubject.associated_project = "TCGA-OV"')The first part is interpreted as a column name, the second as a comparator and
the third part as a value. If the value is a string, it needs to be put in
quotes.Detailed formFor cases where there may be ambiguity in the quoting, or the right side of the
comparison is another column, the detailed form should be used. Here the three
parts of a query are explicity split apart.from cdapython import Q, Col, Quoted

q1 = Q(Col('ResearchSubject.Diagnosis.age_at_diagnosis'), '>=', 50 * 365)
q2 = Q(Col('ResearchSubject.associated_project'), '=', Quoted('TCGA-OV'))Pointing to a custom CDA instance.run()will execute the query on the public CDA API (https://cda.cda-dev.broadinstitute.org/api/cda/v1/)..run("http://localhost:8080")will execute the query on a CDA server running athttp://localhost:8080.Quick Explanation on UNNEST usage in BigQueryUsing Q in the CDA client will echo the generated SQL statement that may contain multipleUNNESTinclusions
when including a dot(.) structure which may need a quick explanation.
UNNEST is similar to unwind in which embedded data structures must be flattend to appear in a table or Excel file.
Note; The following call using the SQL endpoint is not the preferred method to execute a nested attribute query in BigQuery.
The Q language DSL abstracts the required unnesting that exists in a Record. In BigQuery, structures must be represented in an UNNEST syntax such that:A.B.C.Dmust be unwound toSELECT (_C.D)in the following fashion:SELECT (_C.D)
from TABLE, UNNEST(A) AS _A, UNNEST(_A.B) as _B, UNNEST(_B.C) as _CResearchSubject.Specimen.source_material_typerepresents a complex record that needs to unwound in SQL syntax to be queried on properly when using SQL.SELECT DISTINCT(_Specimen.source_material_type)
FROM gdc-bq-sample.cda_mvp.v3,
UNNEST(ResearchSubject) AS _ResearchSubject,
UNNEST(_ResearchSubject.Specimen) AS _SpecimenDeveloper setupPython 3.7 or higher installedgit clone the repogitclonehttps://github.com/CancerDataAggregator/cda-python.gitopen a terminal or powershell and navigate to cloned directoryexample:cdcda-pythonin the cda-python folder create a virtual environment
usingpython3-mvenvvenvand activate the environment by using
for the mac and push entersource./venv/bin/activateand for windows in powershell and push enter.\venv\Scripts\activatenote to stop using venv typedeactivateProject dependenciesIn your virtual environment install  project requirements
Use pip to install the Python dependencies:pip install -r requirements.txt
pip install -r requirements-dev.txtUsing pip-toolsIn your virtual environment installpipinstallpip-toolswe will use pip-tools to keep track of our dependencies and our requirements.inCommandspip-compilewill build requirements.txt from our setup.py and requirements.in and lockdown the requirements.txtUpdate requirementsjust add dependency to the requirements.in and runpip-compilecdapython local devto install the cdapython Library when you are working on the package locally as a dev
make sure you are in the main cdapython folder from you git clone and in your terminal
navigate to the folder then uselsorDIRto see the files make sure you see the setup.py file.
the setup.py file is used to tell pip
-e editable install so yo can update filespip install -e .Note For runing pytestthere is a global_setting.py here you can change the host serverNoteThis is the spiritual successor of theQuery Translator Prototype