Getting Started with Notebooks in the Precision Health Cloud
By: Christian Di Lorenzo
Tags: Python, Precision Medicine, Matplotlib, Pandas, Exploratory Data Analysis

Starting up notebooks in PHC is easy. Simply select it in the left sidebar, choose a server type, and then start up a Python 3 Jupyter Notebook. For our case, a simple 1 CPU / 4 GB instance will do perfectly fine.

Introduction to the Python SDK

Although the Python SDK can be used anywhere with an API token, we really encourage clinicians and researchers to stay within the HIPAA-protected environment for security and maximum flexibility. Plus, there’s no need to manage authentication credentials in this environment; it’s already there with your specific permissions even across projects and accounts.

To get started, we simply import the library and select our project:

# Update to current SDK version (Recommended not required)

!pip install "phc>=0.18.1" # Import easy modules (underlying modules available in phc.services)

import phc.easy as phc # Set current project (fuzzy search on id, name, or description)

phc.Project.set_current("BRCA")

The Philosophy

For a researcher or clinician, we want the PHC tools to get out of the way as much as possible so that you can focus on your data and your goals. The phc.easy modules facilitate this goal by exposing different entities with very similar interfaces. Likewise, although much of the data is stored in the FHIR format, we are data specialists ourselves and understand the value of tabular data especially for millions of records. The data from most APIs are automatically converted directly to a pandas data frame so that the full power of pandas is unleashed (e.g. filtering, selecting, joining, and transforming).

# Examples of Modules

phc.Observation.get_data_frame(...)

phc.Condition.get_data_frame(...)

phc.Specimen.get_data_frame(...)

Patients

To begin, we can load all of the patient records from the FHIR Search Service. Passing all_results=True will scroll through all of the results available in the project.

patient_df = phc.Patient.get_data_frame(all_results=True)

Columns and counts from the `patient_df` frame

Let’s break down the data here. Tabular data in the form of pandas data frames or typed numpy arrays are vastly easier to work with than normal nested Python dictionaries and arrays. While we store most structured data in the highly-nested JSON structure of FHIR resources, the SDK automatically converts the results into a data frame where each column represents the original location.

For example, the column identifier.type_coding_system__hl7.org/fhir/v2/0203__code traces to the value "ANON" which can be seen in the first raw result below. For FHIR resources, the system or url generally informs the type of value which is why it is included in each column. For nested columns that are not FHIR codes, they can be represented more simply like name_family or address_state .

{'id': '0045349c-69d9-4306-a403-c9c1fa836644',

'name': [{'text': 'A1A0SB LO', 'given': ['A1A0SB'], 'family': 'LO'}],

'extension': [{'url': 'http://hl7.org/fhir/StructureDefinition/us-core-race',

'valueCodeableConcept': {'text': 'race',

'coding': [{'code': '2106-3',

'system': 'http://hl7.org/fhir/v3/Race',

'display': 'white'}]}},

{'url': 'http://hl7.org/fhir/StructureDefinition/us-core-ethnicity',

'valueCodeableConcept': {'text': 'ethnicity',

'coding': [{'code': '2186-5',

'system': 'http://hl7.org/fhir/v3/Ethnicity',

'display': 'not hispanic or latino'}]}}],

'resourceType': 'Patient',

'gender': 'female',

'birthDate': '1937-04-10',

'deceasedBoolean': False,

'identifier': [{'type': {'coding': [{'code': 'ANON',

'system': 'http://hl7.org/fhir/v2/0203'}]},

'value': 'LO-A1-A0SB'}],

'meta': {'tag': [{'system': 'http://lifeomic.com/fhir/dataset',

'code': '34c0fb25-bbde-4eb1-87ed-dd4c7a1ac013'}],

'lastUpdated': '2020-06-18T17:30:56.660Z'},

'address': [{'country': 'united states',

'use': 'home',

'type': 'physical',

'text': 'united states'}]}

TIP: Use the raw flag to see nested structures without the expanded columns: phc.Observation.get_data_frame(raw=True)

Exploring Patients with Visualizations

With all 1,098 patients in a nice pandas data frame, we can easily explore and plot the data in so many rich ways. For this brief demonstration, the demographics will be plotted using matplotlib.

Demographics visualizations for the BRCA-TCGA data set

For gender, race, country, and deceased, the values can be put in a bar chart with the pandas .plot method.

import matplotlib.pyplot as plt # Plot gender counts

patient_df.gender.value_counts().plot.barh(title="Gender")



# Plot race counts

patient_df[COLUMN_RACE].value_counts().plot.barh(title="Race")



# Plot country counts

patient_df.address_country.value_counts().plot.barh(title="Country")



# Plot deceased (by whether a deceased timestamp exists)

(

patient_df["deceasedDateTime.local"].apply(pd.notnull)

.value_counts().plot.barh(title="Deceased")

)

For birth dates, we can produce a simple histogram of the provided years.

# Plot histogram of birth years

(

patient_df["birthDate.local"].dt.year

.dropna().astype(int)

.plot.hist(title="Birth Year")

)

plt.gca().set_xlabel("Year")

plt.gca().set_ylabel("# of Patients")

For visualizing the ages at death, a filtered histogram of years based on the deceasedDateTime will represent a quick snapshot of the distribution.

# Plot age at death

patient_df[

# Filter by those who have passed away

patient_df["deceasedDateTime.local"].notnull()

].pipe(

# Calculate survival in years

lambda df: (

df["deceasedDateTime.local"] - df["birthDate.local"]

).dt.total_seconds() / (60 * 60 * 24 * 365.25)

).plot.hist(title="Age at Death") plt.gca().set_xlabel("Age (years)")

plt.gca().set_ylabel("# of Patients")

Next Steps

In this post, we’ve only scratched the surface of what’s possible. In the BRCA data set, information lives across a host of FHIR resources:

module count

========================== =====

Observation 6601

Specimen 4119

MedicationStatement 2391

DocumentReference 2003

Sequence 1975

Condition 1178

Patient 1098

Encounter 1094

Procedure 1059

MedicationAdministration 53

DiagnosticReport 3

Media 1

The SDK provides many options for querying and extracting Patient records as well as each of these FHIR types and more. All of the code in this post is available in a GitHub repository. Listed below are some helpful links for beginning your journey into PHC.

Links