# Supporting decision making and early prediction of survival for oncology drug development using a pharmacometrics-machine learning based model

Sébastien Benzekry, Mélanie Karlsen, Abdessamad Elkaoutari, René Bruno, Aleš Neubert, François Mercier, Bruno Gomes, Suresh Vatakuti, Peter Curle, Candice Jamois

<figure><img src="image_1.png" /></figure>

## To cite this version:

Sébastien Benzekry, Mélanie Karlsen, Abdessamad Elkaoutari, René Bruno, Aleš Neubert, et al. Supporting decision making and early prediction of survival for oncology drug development using a pharmacometrics-machine learning based model. PAGE - Population Approach Group Europe, Jun 2022, Ljubljana, Slovenia. hal-03921394

HAL Id: hal-03921394

https://inria.hal.science/hal-03921394

Submitted on 3 Jan 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
# Supporting decision making and early prediction of survival for oncology drug development using a pharmacometrics-machine learning based model.

Sébastien Benzekry¹, Mélanie Karlsen¹, Abdessamad El Kaoutari¹, René Bruno², Ales Neubert³, François Mercier³, Martin Stern³, Bruno Gomes³, Suresh Vatakuti³, Peter Curle³ and Candice Jamois³

(1) COMPO Inria – Inserm, Marseille, France; (2) Genentech-Roche, Marseille, France; (3) Roche pRED, Basel, Switzerland

## Thirtieth PAGE meeting, Ljubljana, Slovenia

30 June 2022
# Table of contents

1. Project schematic and objectives

2. Methodology

a. Data

b. Pharmacometrics model
development

c. Machine Learning model
development (Train set)

3. Results

a. Minimal signature model

b. Performance metrics by features
category

c. External model validation (Test set)

4. Summary and Perspectives
# Project Schematic and Objectives

## Prediction of overall survival in NSCLC patients treated with atezolizumab

Roche

### Raw longitudinal data

Tumor Kinetics (TK)

Pharmacodynamic markers (PD)
Albumin, CRP, neutrophils, LDH

<figure><img src="image_2.png" /></figure>

NLME

<figure><img src="image_3.png" /></figure>

### Model-based
individual parameters

TK

PD

<figure><img src="image_4.png" /></figure>

Machine
Learning

### Outputs

Individual prognosis

* C-index

* 12-months survival

### Raw baseline data

Clinical characteristics

Genomic and Transcriptomic data

Feature
selection

### Minimal signature

Clinical
characteristics

Genomic and
Transcriptomic
data

Study level predictions

* Survival curve

* Hazard ratio

NLME: NonLinear Mixed Effect Modeling; CRP: C-reactive protein; LDH: lactate deshydrogenase
<figure><img src="image_5.png" /></figure>

# Data for model development (Train set) and external validation (Test set)

## Four monotherapy studies of atezolizumab in advanced NSCLC

Roche

### Baseline data

* Patients' and disease characteristics

p = 73 parameters

* Transcriptomic and mutational data

p = 58,311 and 395

### Longitudinal data

* Tumor kinetics (TK, SLD)

5,570/3,065 observations

* 4 PD markers (CRP, LDH, Albumin and Neutrophils)

61,296/47,255 observations

<table><thead><tr><th>Study</th><th>Description</th><th>N</th></tr></thead><tbody><tr><td>FIR<br>GO28625</td><td>Phase 2 study for the efficacy and safety of anti-programmed death-ligand 1 (PD-L1) atezolizumab (ATZ) in advanced NSCLC selected by tumor cell (TC) or tumor-infiltrating immune cell (IC) PD-L1 expression</td><td>133</td></tr><tr><td>POPLAR<br>GO28753</td><td>Phase 2 randomised controlled trial (RCT) of ATZ versus docetaxel for patients with previously treated NSCLC (locally advanced or metastatic NSCLC who failed to platinum therapy)</td><td>134</td></tr><tr><td>BIRCH<br>GO28754</td><td>Phase 2 study of ATZ in patients with PD-L1 positive locally advanced or metastatic NSCLC</td><td>595</td></tr><tr><td>Train</td><td></td><td>862</td></tr><tr><td>Test - OAK<br>GO28915</td><td>Phase 3 RCT of ATZ versus docetaxel (DTX) in patients with previously treated NSCLC</td><td>553</td></tr><tr><td>Train + Test</td><td></td><td>1415</td></tr></tbody></table>

Train

Test

NSCLC: Non-Small Cell Lung Cancer; p = number of parameters, N: number of patients treated with atezolizumab (patients from French centers were excluded for legal reasons (N=118); In total, data from 1074 patients from OAK were used as Test set (553 from the ATZ arm, 521 from the DTX arm); PD: Pharmacodynamic; SLD: Sum of the Largest Diameters. CRP: C Reactive Protein; LDH: Lactate Dehydrogenase.

1. Fehrenbacher L et al. Atezolizumab versus docetaxel for patients with previously treated non-small-cell lung cancer (POPLAR): a multicentre, open-label, phase 2 randomised controlled trial. Lancet (2016)

2. Fehrenbacher L et al. Updated Efficacy Analysis Including Secondary Population Results for OAK: A Randomized Phase III Study of Atezolizumab versus Docetaxel in Patients with Previously Treated Advanced Non-Small Cell Lung Cancer. Journal of Thoracic Oncology (2018)

3. Solange Peters et al. Phase II Trial of Atezolizumab As First-Line or Subsequent Therapy for Patients With Programmed Death-Ligand 1-Selected Advanced Non-Small-Cell Lung Cancer (BIRCH). JCO (2017)

4. Spigel D.R et al. FIR: Efficacy, Safety, and Biomarker Analysis of a Phase II Open-Label Study of Atezolizumab in PD-L1-Selected Patients With NSCLC. Journal of Thoracic Oncology (2018)
* Tumor kinetics (TK) : double-exponential model<fnref n="1" />,²

* PD time courses : empirical models³

* ○ constant? linear? hyperbolic? double-exponential?

* Statistical Nonlinear Mixed Effect (NLME) model

* Observation model : constant (TK) or proportional (PD)

$$y_j^i = M(t_j^i; \theta^i) + \epsilon_j^i, \epsilon_j^i \sim N(0, \sigma_j^i)$$

* Inter-individual

$$\ln(\theta^i) = \ln(\theta_{\text{pop}}) + \eta^i, \eta^i \sim N(0, \omega^2)$$

* Population parameters : SAEM algorithm for likelihood maximization

* Individual empirical Bayes estimates (EBEs) from the maximum a posteriori estimator

* Fits performed using the R Monolix2020R1 API

<figure><img src="image_6.png" /></figure>

★ Individual TK and PD model parameters = inputs of the Machine Learning (ML) algorithm

★ Model-derived baseline parameters were excluded (baseline markers already in clinical category)

<footnote n="1">¹Stein et al., 2008 (PSA), ²Claret et al. 2018 (SLD, atezolizumab), ³Gavrilov et al. 2020 (NLR, durvalumab)</footnote>
<figure><img src="image_7.png" /></figure>

# NLME modeling of tumor kinetics and PD markers

## Is there any kinetic pattern in the PD data?

<figure><img src="image_8.png" /></figure>

TK

<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /></figure>

Time (months)

<figure><img src="image_11.png" /></figure>

LDH

<table><thead><tr><th>model</th><th>BICc</th><th>b</th></tr></thead><tbody><tr><td><strong>dexp</strong></td><td>39,886</td><td>0.56</td></tr><tr><td>hyperbolic</td><td>40,915</td><td>0.62</td></tr><tr><td>linear</td><td>42,462</td><td>0.70</td></tr><tr><td>constant</td><td>42,982</td><td>0.74</td></tr></tbody></table>

<figure><img src="image_12.png" /></figure>

Time (months)

<figure><img src="image_13.png" /></figure>

Neutrophils

<table><thead><tr><th>model</th><th>BICc</th><th>b</th></tr></thead><tbody><tr><td><strong>dexp</strong></td><td>102,449</td><td>0.14</td></tr><tr><td>hyperbolic</td><td>102,943</td><td>0.14</td></tr><tr><td>linear</td><td>105,193</td><td>0.17</td></tr><tr><td>constant</td><td>106,249</td><td>0.18</td></tr></tbody></table>

<figure><img src="image_14.png" /></figure>

Time (months)

<figure><img src="image_15.png" /></figure>

CRP

<table><thead><tr><th>model</th><th>BICc</th><th>b</th></tr></thead><tbody><tr><td><strong>dexp</strong></td><td>28,764</td><td>0.21</td></tr><tr><td>hyperbolic</td><td>29,712</td><td>0.22</td></tr><tr><td>linear</td><td>30,020</td><td>0.23</td></tr><tr><td>constant</td><td>31,332</td><td>0.25</td></tr></tbody></table>

<figure><img src="image_16.png" /></figure>

Time (months)

<figure><img src="image_17.png" /></figure>

Albumin

<table><thead><tr><th>model</th><th>BICc</th><th>b</th></tr></thead><tbody><tr><td><strong>hyperbolic</strong></td><td>48,007</td><td>0.056</td></tr><tr><td>dexp</td><td>48,395</td><td>0.058</td></tr><tr><td>linear</td><td>49,436</td><td>0.063</td></tr><tr><td>constant</td><td>49,724</td><td>0.065</td></tr></tbody></table>

<figure><img src="image_18.png" /></figure>

Time (months)

<figure><img src="image_19.png" /></figure>

CR = complete response, PR = partial response, SD = stable disease, PD = progressive disease, SLD = sum of largest diameters, LDH = lactate deshydrogenase, CRP = C-reactive protein
# Methodology - Machine learning model development

* **Preprocess** to handle missing data
  Drop zero-variance or >25% NA columns, dummification, NA imputation, scaling

* **Dimensionality reduction** for FMI and RNAseq data (bootstrap LASSO)

* **Features selection:**

* 5 methods: **LASSO**, random survival forest (RSF) importance, Cox-based and stepwise forward/backward

* 3 strategies: i) all variables, ii) per feature set and iii) pooled selected sets

* **4 survival algorithms tested:**
  Cox, Cox and accelerated failure time with gradient boosting and **random survival forest (RSF)**

* **Evaluation of machine learning models**

* **Model development:** 10-fold cross-validation
    C-index, calibration curves and 12-months survival classification metrics

* **Study-level predictions:** survival curves, **hazard ratios**

**Working principle:** need for a **minimal signature** model with limited number of easily measurable variables
# Reducing number of features to a minimal signature

## Baseline clinical parameters

7 features

<figure><img src="image_20.png" /></figure>

* All features sorted using LASSO

* Incremental models with increasing number of features

* Minimal set of features that reaches the plateau

### Minimal signature baseline characteristics
(p = 11)

CRP

Heart rate

Neutrophils-to-lymphocytes ratio

Neutrophils

Lymphocytes-to-leukocytes ratio

Liver metastases

ECOG (0 vs 1)

PD-L1 (≥ 50% on tumor cells)

Hemoglobin

SLD

Lactate dehydrogenase

7 features

4 features added
because of
established
prognostic/predictive value<fnref n="1" />,<fnref n="2" />,<fnref n="3" />

<footnote n="1">¹Fangfang Wu *et al.* Prognostic value of baseline hemoglobin-to-red blood cell distribution width ratio in small cell lung cancer: A retrospective analysis. Thoracic Cancer. 2020 Apr; 11(4): 888–897</footnote>

<footnote n="2">²Matthen Mathew, Rachael A. Safyan, Catherine A. Shu. PD-L1 as a biomarker in NSCLC: challenges and future directions - Annals of Translational Medicine. 2017 Vol 5, No 18.</footnote>

<footnote n="3">³Bernhard C D. *et al.* Long-term Survival Is Linked to Serum LDH and Partly to Tumour LDH-5 in NSCLC. Anticancer Research April 2010, 30 (4) 1347-1351.</footnote>
<figure><img src="image_21.png" /></figure>

## Prediction metrics by features category

* Each feature set exhibits **differential predictive power**

* RNAseq has low individual predictive power.
  ⇒ discarded

* Model-based **dynamic features** (TK, PD) outperform baseline clinical features, with much less variables

* Model-based PD outperforms TK metrics

<figure><img src="image_22.png" /><figcaption>Cross-validated C-indices by features category</figcaption></figure>

From 10-fold cross-validation on FIR, BIRCH and POPLAR (train data set, N=559)

<figure><img src="image_23.png" /></figure>

A pooled model of 26 features has **very good predictive metrics**

Roche
<figure><img src="image_24.png" /></figure>

# Model-based metrics features are more informative than observed data

## Comparison of predictive performance

* Using SLD, CRP, Albumin, Neutrophils and LDH parameters

* Several cycle landmark times were used (Cycle 3 to Cycle 20)

* ML model learned from the train set truncated at Cycle X

* For all landmarks, the predictive power of the **data-based** versus **model-based** ratio from baseline at Cycle X pre-dose is compared

<figure><img src="image_25.png" /></figure>

Model-based metrics clearly have both **better predictive power** and **narrower uncertainty**

<figure><img src="image_26.png" /></figure>

This illustrates how dynamic modeling allows to **capture the kinetics** and correct for intra-individual stochasticity (noise)

<figure><img src="image_27.png" /></figure>

Model-based ratio from baseline is **more predictive with increasing number of cycles**

### Cross-validation c-indices for **model-** and **observed-** based ratio to baseline

<figure><img src="image_28.png" /></figure>

Note: 10-fold cross-validation on FIR, BIRCH and POPLAR (train data set). Only relative change from baseline used as model metric. Other model-based parameters ignored here for fair comparison
# Methodology - From Train to Test

<figure><img src="image_29.png" /></figure>
<figure><img src="image_30.png" /></figure>

# Performance metrics on test set (OAK)

## Performance metrics for minimal signature model prediction

<figure><img src="image_31.png" /></figure>

C-index = 0.789

AUC-12 months = 0.868

## Prediction of 12-months survival

<figure><img src="image_32.png" /></figure>

Results presented are based on full test set and are prone to immortal time bias

$$ \text{PDL1}^1: \text{AUC} = 0.65 $$

$$ \text{RoPro}^2: \begin{aligned} \text{c-index} &= 0.69 \\ \text{AUC-3 months} &= 0.817 \end{aligned} $$

Train: 559 patients. Test: 396 patients. Performance metrics using full Test set (N= 391 patients). All metrics are computed at 12 months. Positive (1)= death, negative (0)= Alive; NPV: negative predictive value (NPV=TN/ (FN+TN); TN=true negative; FN=false negative; PPV: positive predictive value (PPV=TP/(TP+FP); TP= true positive; FP = False positive. To compute accuracy, censored patients were excluded (i.e., 17/396 patient at 12 months)

<footnote n="1">¹Rizvi, H. et al., J Clin Oncol Molecular Determinants of Response to Anti-Programmed Cell Death (PD)-1 and Anti-Programmed Death-Ligand 1 (PD-L1) Blockade in Patients with Non-Small-Cell Lung Cancer Profiled with Targeted Next-Generation Sequencing. J. Clin. Oncol. 2018, 36, 633–641.</footnote>

<footnote n="2">²Becker, T. et al. An enhanced prognostic score for overall survival of patients with cancer derived from a large real-world cohort. Ann Oncol 31, 1561–1568 (2020).</footnote>
<figure><img src="image_33.png" /></figure>

# The minimal signature model reproduced retrospectively the survival curves of atezolizumab and control arm in the OAK phase III trial

<figure><img src="image_34.png" /></figure>

* The model **predicted well** survival curves of ATZ and control arm (docetaxel) from OAK

* The model was able to **predict ATZ survival benefit** over chemotherapy in OAK (HR < 1 with good match between observed and predicted HR)

<table><thead><tr><td>Observed HR (95%CI)</td><td>D-Light Prediction HR (95%PI)</td></tr></thead><tbody><tr><td>0.765 (0.64 - 0.913)</td><td>0.765 (0.692-0.829)</td></tr></tbody></table>

* The relationship between TK and PD metrics and survival is **not drug specific**

## Observed and predicted survival curves (OAK Phase III)

<figure><img src="image_35.png" /></figure>

Results based on the entire OAK dataset (i.e, 391 patients from atezo arm and 350 patients from docetaxel arm at time=0). 95% prediction intervals from simulating 1000 replicates of model predictions
<figure><img src="image_36.png" /></figure>

# The minimal signature model predicts OS benefit for atezolizumab over docetaxel in OAK 7.5 months after first patient randomized

<figure><img src="image_37.png" /></figure>

5 weeks
(1.25 months)

10 weeks
(2.5 months)

30 weeks
(7.5 months)

90 weeks
(21 months)

100 weeks
(23 months)

<figure><img src="image_38.png" /></figure>

<table><tr><td>Data HR (95% CI)</td><td>NA (NA - NA)</td><td>NA (NA - NA)</td><td>1.04 (0.386 - 2.79)</td><td>0.708 (0.564 - 0.887)</td><td>0.748 (0.606 - 0.922)</td></tr><tr><td>Predicted HR (95% PI)</td><td>0.923 (0.369 - 2.5)</td><td>0.836 (0.547 - 1.4)</td><td>0.802 (0.655 - 0.907)</td><td>0.809 (0.746 - 0.903)</td><td>0.798 (0.715 - 0.89)</td></tr><tr><td>Number of patients (DTX - ATZ)</td><td>11 - 8</td><td>23 - 30</td><td>163 - 183</td><td>352 - 386</td><td>352 - 386</td></tr><tr><td>Median nb of data points (TK/Alb/CRP/LDH/ Neutrophils)</td><td>1/2/2/2/2 - 1/2/1/2/1</td><td>1/3/3/2/2.5 - 1/2/2/2/2</td><td>2/4/4/4/5 - 2/4/4/4/4</td><td>4/8/6/7/8 - 4/10/9/9/10</td><td>4/8/6/7/8 - 4/10/9/9/10</td></tr></table>

Truncation of data is based on the randomization date of the first patient treated in OAK (e.g. at 5 weeks after randomization, 12 patients received atezolizumab (ATZ), 11 docetaxel (DTX); and median TK data points were 2 and 1 for docetaxel and atezo arms respectively)
# Summary and applications for early drug development

Roche

* A **minimal signature** NLME-ML model of **26 features** (clinical, TK and PD model features)

* **Baseline**: CRP, Heart rate, Neutrophils-to-lymphocytes ratio, Neutrophils, Lymphocytes-to-leukocytes ratio, Liver metastases, ECOG (0 vs 1), PD-L1 (≥ 50% on tumor cells), Hemoglobin, SLD, LDH

* **Longitudinal**: tumor kinetics (SLD), albumin, CRP, LDH, neutrophils

* Analysis (preprocess, feature selection, CV, train, predict) fully automated in a **R package** (> 12,000 lines of code)

* Could be **applied to early phase data** to assess the decision to move an asset to a later phase of development

* Potential **extrapolation to other drugs** within the same disease setting;

<figure><img src="image_39.png" /></figure>

**pTS for Combination Therapies**

Faster, more efficient clinical trials
Early data to inform clinical development plan and patient selection

<figure><img src="image_40.png" /></figure>

**Better and Faster Go/no-Go decisions**

Better decision making process by benchmarking efficacy based on early data readouts

<figure><img src="image_41.png" /></figure>

**Patient Matching to Therapy**

**Reduced Cost for Society**

PHC: personalized treatment for every patient
Doing now what patients need next