Application of a risk-management framework for
integration of stromal tumor-infiltrating lymphocytes in
clinical trials

Jan Hudeček, Leonie Voorwerk, Maartje van Seijen, Iris Nederlof, Michiel de Maaker, Jose van den Berg, Koen van de Vijver, Karolina Sikorska, Sylvia Adams, Sandra Demaria, et al.

<figure><img src="image_1.png" /></figure>

To cite this version:

Jan Hudeček, Leonie Voorwerk, Maartje van Seijen, Iris Nederlof, Michiel de Maaker, et al. Application of a risk-management framework for integration of stromal tumor-infiltrating lymphocytes in clinical trials. npj Breast Cancer, 2020, 6 (1), pp.15. 10.1038/s41523-020-0155-1. hal-02882180

HAL Id: hal-02882180

https://hal.science/hal-02882180

Submitted on 7 Jun 2023

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
### REVIEW ARTICLE

OPEN

<figure><img src="image_3.png" /></figure>

Check for updates

# Application of a risk-management framework for integration of stromal tumor-infiltrating lymphocytes in clinical trials

Jan Hudeček<sup>1</sup>, Leonie Voorwerk<sup>2</sup>, Maartje van Seijen<sup>3</sup>, Iris Nederlof<sup>2</sup>, Michiel de Maaker<sup>3</sup>, Jose van den Berg<sup>4</sup>, Koen K. van de Vijver<sup>5</sup>, Karolina Sikorska<sup>6</sup>, Sylvia Adams<sup>7</sup>, Sandra Demaria<sup>8,9</sup>, Giuseppe Viale<sup>10,11</sup>, Torsten O. Nielsen<sup>12</sup>, Sunil S. Badve<sup>13</sup>, Stefan Michiels<sup>14,15</sup>, William Fraser Symmans<sup>16</sup>, Christos Sotiriou<sup>17</sup>, David L. Rimm<sup>18,19</sup>, Stephen M. Hewitt<sup>20</sup>, Carsten Denkert<sup>21,22</sup>, Sibylle Loibl<sup>23</sup>, Sherene Loi<sup>24</sup>, John M. S. Bartlett<sup>25,26,27</sup>, Giancarlo Pruneri<sup>28,29</sup>, Deborah A. Dillon<sup>30</sup>, Maggie C. U. Cheang<sup>31</sup>, Andrew Tutt<sup>32</sup>, Jacqueline A. Hall<sup>33</sup>, Zuzana Kos<sup>34</sup>, Roberto Salgado<sup>24,35</sup>, Marleen Kok<sup>2,36</sup>, Hugo M. Horlings<sup>3,4</sup> and International Immuno-Oncology Biomarker Working Group*

Stromal tumor-infiltrating lymphocytes (sTILs) are a potential predictive biomarker for immunotherapy response in metastatic triple-negative breast cancer (TNBC). To incorporate sTILs into clinical trials and diagnostics, reliable assessment is essential. In this review, we propose a new concept, namely the implementation of a risk-management framework that enables the use of sTILs as a stratification factor in clinical trials. We present the design of a biomarker risk-mitigation workflow that can be applied to any biomarker incorporation in clinical trials. We demonstrate the implementation of this concept using sTILs as an integral biomarker in a single-center phase II immunotherapy trial for metastatic TNBC (TONIC trial, NCT02499367), using this workflow to mitigate risks of suboptimal inclusion of sTILs in this specific trial. In this review, we demonstrate that a web-based scoring platform can mitigate potential risk factors when including sTILs in clinical trials, and we argue that this framework can be applied for any future biomarker-driven clinical trial setting.

npj Breast Cancer (2020)6:15; https://doi.org/10.1038/s41523-020-0155-1

### INTRODUCTION

Clinical trials in cancer research are increasingly incorporating biomarkers, for example, as an inclusion criterion or for stratification of patients to control for confounding factors. Practical challenges, such as interobserver variation in the assessment of biomarkers *during* the execution of the trial, are often overlooked. If not handled appropriately, these challenges can limit the effectiveness and ability to complete the biomarker and drug development process. According to Hall et al.<sup>1</sup>, the risks inherent to biomarker integration can be divided into risks to patients, operational risks, and direct risks to biomarker development. A practical risk-management framework developed by a National Cancer Institute (NCI), National Cancer Research Institute (NCRI), and European Organization for Research and Treatment of Cancer

(EORTC) Working Group<sup>1</sup> was proposed to manage the risks inherent to biomarker integration into clinical trials.

Stromal tumor-infiltrating lymphocytes (sTILs) have been strongly associated with prognosis in early-stage triple-negative breast cancer (TNBC) and HER2-positive breast cancer. In addition, sTILs are predictive for neo-adjuvant chemotherapy response in early breast cancer<sup>2,3</sup>. Furthermore, sTILs correlate with outcome after immune checkpoint blockade in metastatic TNBC<sup>4-6</sup>. The readout of sTILs, however, can be challenging impeding its effective use as a biomarker and its usage in the clinic<sup>7</sup>. The International Immuno-Oncology Biomarker Working Group (hereafter called the TIL Working Group) has provided guidelines for the scoring of sTILs in breast cancer<sup>8</sup>, and the St. Gallen Breast Cancer Conference of 2019 endorsed sTILs being routinely

<footnote n="1"><sup>1</sup>Department of Research IT, The Netherlands Cancer Institute, Amsterdam, The Netherlands. <sup>2</sup>Division of Tumor Biology and Immunology, The Netherlands Cancer Institute, Amsterdam, The Netherlands. <sup>3</sup>Division of Molecular Pathology, The Netherlands Cancer Institute, Amsterdam, The Netherlands. <sup>4</sup>Department of Pathology, The Netherlands Cancer Institute, Amsterdam, The Netherlands. <sup>5</sup>Department of Pathology, Ghent University Hospital, Ghent, Belgium. <sup>6</sup>Department of Biometrics, The Netherlands Cancer Institute, Amsterdam, The Netherlands. <sup>7</sup>Department of Medicine, Perlmutter Cancer Center, New York University School of Medicine, New York, NY, USA. <sup>8</sup>Department of Radiation Oncology, Weill Cornell Medicine, New York, NY, USA. <sup>9</sup>Department of Pathology and Laboratory Medicine, Weill Cornell Medicine, New York, NY, USA. <sup>10</sup>International Breast Cancer Study Group Central Pathology Office, Department of Pathology and Laboratory Medicine, IEO European Institute of Oncology IRCCS, Milan, Italy. <sup>11</sup>University of Milan, Milan, Italy. <sup>12</sup>Department of Pathology and Laboratory Medicine, Genetic Pathology Evaluation Centre, University of British Columbia, Vancouver, BC, Canada. <sup>13</sup>Department of Pathology and Laboratory Medicine, Indiana University Simon Cancer Center, Indianapolis, IN, USA. <sup>14</sup>Service de Biostatistique et d'Epidémiologie, Gustave Roussy, CESP, Université-Paris Sud, Université Paris-Saclay, Villejuif, France. <sup>15</sup>CESP, Fac. de médecine - Univ. Paris-Sud, Fac. de médecine - UVSQ, INSERM, Université Paris-Saclay, Villejuif, France. <sup>16</sup>Department of Pathology, M.D. Anderson Cancer Center, Houston, TX, USA. <sup>17</sup>Breast Cancer Translational Research Laboratory, Institut Jules Bordet, U-CRC, Université Libre de Bruxelles, Brussels, Belgium. <sup>18</sup>Department of Pathology, Yale School of Medicine, New Haven, CT, USA. <sup>19</sup>Department of Medicine, Yale University School of Medicine, New Haven, CT, USA. <sup>20</sup>Laboratory of Pathology, Center for Cancer Research, National Cancer Institute, Bethesda, MD, USA. <sup>21</sup>Institute of Pathology, Charité Universitätsmedizin Berlin, Berlin, Germany. <sup>22</sup>Institute of Pathology, Philipps-University Marburg, Marburg, Germany. <sup>23</sup>German Breast Group, Neu-Isenburg, Germany. <sup>24</sup>Division of Research and Clinical Medicine, Peter MacCallum Cancer Centre, University of Melbourne, Melbourne, VIC, Australia. <sup>25</sup>Ontario Institute for Cancer Research, Toronto, ON, Canada. <sup>26</sup>IGMM, Edinburgh, UK. <sup>27</sup>Edinburgh Cancer Research Centre, Western General Hospital, Edinburgh, UK. <sup>28</sup>Department of Pathology and Laboratory Medicine, IRCCS Fondazion - Instituto Nazionale Tumori, Milan, Italy. <sup>29</sup>School of Medicine, University of Milan, Milan, Italy. <sup>30</sup>Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA. <sup>31</sup>Clinical Trials and Statistics Unit, The Institute of Cancer Research, Surrey, UK. <sup>32</sup>Breast Cancer Now Toby Robins Research Centre, The Institute of Cancer Research, London, UK. <sup>33</sup>Research and Development, Vivactiv Ltd, Chesham, Buckinghamshire, UK. <sup>34</sup>Department of Pathology and Laboratory Medicine, University of Ottawa, Ottawa, ON, Canada. <sup>35</sup>Department of Pathology, GZA-ZNA Ziekenhuizen, Antwerp, Belgium. <sup>36</sup>Department of Medical Oncology, The Netherlands Cancer Institute, Amsterdam, The Netherlands. *A list of authors and their affiliations appears at the end of the paper. †email: roberto@salgado.be</footnote>
characterized in TNBC and reported according to these
guidelines⁸.

## RISKS ASSOCIATED WITH INTEGRATION OF BIOMARKERS IN CLINICAL TRIALS

In contemporary clinical research there is an increasing trend toward the use of biomarker results obtained in daily practice to select patients for inclusion in clinical trials. Although biomarker research is more and more prominent in clinical trials, most biomarkers will not make into the clinic⁹. Therefore, continuous monitoring of the predefined risks and the solutions can improve the quality of the biomarker, which can be applied in a clinical trial setting, as well as in daily practice. The recommendations of the TIL Working Group⁸,¹⁰ for appropriate scoring, and the risk-management framework of the NCI, NCRI, and EORTC Working Groups¹ will help to effectively and efficiently improve the incorporation of biomarkers in clinical trials in first instance.

Several risks are associated with biomarker development and integration of biomarkers in clinical trials. Roughly, risks can be divided into three categories: risks to patient safety, operational risks, and risks to biomarker development. Not all risks are applicable to all clinical trials and upon designing a biomarker-incorporating clinical trial, risks should be defined and mitigation approaches formulated. It is highly recommended that during a clinical trial, risks are not only pre-identified but are also continuously monitored to prevent stagnation in biomarker development¹. For example, incorporating biomarkers in a large multi-center international clinical trial involves different risks than a small single-center trial. In the first case, there might be different legislation regarding data confidentiality, and inter-laboratory variability can be an issue. When incorporating a biomarker as inclusion criterion or stratification factor in clinical trials, rapid turnaround times are needed and the highest level of quality is necessary for correct interpretation of the results. In the next steps of biomarker development, high-quality results are needed to ensure implementation in daily clinical practice.

## USE OF DIGITAL PATHOLOGY IN CLINICAL TRIALS AND DEVELOPMENT OF A NOVEL WEB APPLICATION

In larger trials, usually phase II-III, central pathology review (CPR) plays an important role in the reliable assessment of biomarker scoring. However, logistical issues, such as the sending of tumor blocks or slides, can be time consuming, costly for the pathology laboratory, and error prone with significant consequences for patient inclusion if the wrong material is sent to the central lab. Digital sharing of histology slides and patient data simplifies logistics for CPR¹¹. Besides digital sharing and scoring of slides, digital image analysis and machine learning approaches are emerging in clinical research¹²,¹³. The use of digital pathology or digital evaluation of histology slides most prominently mitigates risks associated with operational processes. It can reduce the number of missing samples, since the sharing of material is simplified; it enables rapid turnaround times; reduces manual errors; and can streamline local versus central assessment of biomarker.

For clinicians and researchers to use digital pathology, applications and websites should be user-friendly and intuitive. As an example, a web-based tool called Slide Score (www.slidescore.com) was developed as a cross-platform web application to facilitate the scoring of whole slide images and tissue microarray (TMA) cores. Application programming interface (API) was implemented that allowed programmatic administration of studies, uploading slides, fetching results, and retrieving pixel data for regions of images. This API enabled automating creation of new studies from internal database system for managing biobanking workflows. Additionally, a plugin was developed for

QuPath¹⁴—open-source image analysis software—which uses this API to run image analysis algorithms on slides stored on the Slide Score platform avoiding the need to download the slides. This web-based platform was used in high-impact projects⁶,¹⁵, for example, for the digital scoring of biomarkers in the first stage of the TONIC trial⁶, and the estimation of the immune infiltrate of tumors of melanoma patients used for single-cell sequencing¹⁵. Furthermore, the web-based platform is currently used for several other types of research, such as interrater variability studies, retrospective TMA, and whole slide scoring and prospective biomarker scoring.

## DESIGN OF A WORKFLOW TO MITIGATE RISKS ASSOCIATED WITH BIOMARKER DEVELOPMENT: AN EXAMPLE

We identified seven distinct risks with the risk-management framework published by Hall et al.¹ as possibly interfering with the quality and integration of prospective sTILs scores in a clinical trial, and designed our workflow accordingly (Table 1). These risks are specific for this trial, but some of them are applicable also to other trials. They span all three categories mentioned above¹ and included (1) poor-quality biopsies, (2) possible loss of data confidentiality, (3) interrater variability, (4) poor sample quality, (5) poor scoring quality, (6) delay in patient registration, and (7) manual errors (Table 1). We then defined solutions to mitigate these risks and integrated these solutions in a workflow that can be applied across clinical trials and across biomarkers (Fig. 1). The workflow can be modified according to local guidelines, research questions, and clinical trial designs. We used the following workflow to obtain timely and reliable sTILs scores (summary in Supplementary Fig. 1).

After obtaining informed consent of a patient, three biopsies of one metastatic lesion (lymph node, skin, liver, or other) were obtained in this trial. Previous research has shown that three 14 G core needle biopsies should be sufficient for accurate breast cancer diagnosis¹⁶. A hematoxylin and eosin (H&E)-stained slide of one biopsy was then evaluated, to ensure that the biopsy contained enough tumor cells (more than 100 cells) for further analysis (risk 1). Next, a high-resolution digital scan was obtained and automatically pseudonymized with study-specific identifiers (risk 2) before uploading to Slide Score. Display of the original labels was masked to ensure confidentiality of all data within Slide Score (Supplementary Fig. 2b). Pathologists and administrators had to login with their username and password to access the slides and were able to add a two-factor authentication application. Four well-trained breast pathologists, based in three different institutes and in two different countries, were notified via email to score each slide using existing sTIL scoring guidelines of the TIL Working Group⁸,¹⁰ to reduce interrater variability (risk 3). sTILs are scored as the percentage of lymphocytes in the total stromal area (in close proximity of the tumor cells). Interrater variability can lead to bias in the results, when assessment of a biomarker is skewed towards either the lower or higher ranges. When there was a disagreement (using a 5% cut-off) a concordance-score was agreed upon (Supplementary Fig. 1). Low-quality, inaccurate collection or processing of samples can result in low sample availability and introduce batch effects or bias in the results (risk 4) and lead to non-consistent scores (risk 5). High quality of samples was ensured by standardization of our workflow in which all steps were performed in the same manner for every biopsy (Supplementary Fig. 1). Oversight of the entire workflow by one person, referred to as the central manager, is essential for timely identification of technical errors. The central manager tracked the timing of the biopsies, notified the pathologists immediately after the scan was uploaded and sent reminders if necessary, kept track of the scores and timing, and noted the score in the patient record for trial office notification. We predefined acceptable timeframes for obtaining the scores of
<table><caption>**Table 1.** Risks with possible high impact identified in a phase II immunotherapy trial⁶ based on the perspectives of Hall et al.¹ with our approach to mitigation of that risk.</caption><thead><tr><th>Type of risk</th><th>Risk</th><th>Description of risk</th><th>Mitigation approach</th></tr></thead><tbody><tr><td>1. Risks to patients</td><td>No stroma or tumor cells in biopsy</td><td>Discomfort and risks associated with a sampling intervention</td><td>Take multiple biopsies from one lesion at the same time (a minimum of three biopsies per lesion⁹) and check amount of tumor cells before analysis of sTILs and inclusion in the trial</td></tr><tr><td>2. Risks to patients</td><td>Loss of data confidentiality</td><td>Patient samples sent to multiple institutions and reviewers</td><td>Pseudonymization should be applied, hide slide labels, implement strict access control, ensure no metadata is linked to a slide</td></tr><tr><td>3. Risks to biomarker development</td><td>Inter-laboratory variability and interobserver variability</td><td>Different methodologies used to score slides, interrater variability</td><td>Use of international guidelines for scoring and training, use consensus score of four expert pathologists from three institutes</td></tr><tr><td>4. Operational risks</td><td>Failure of sample collection, processing, and quality</td><td>Missing or poor-quality samples resulting in poor consensus scoring</td><td>Standardized tissue processing, workflow management</td></tr><tr><td>5. Operational risks</td><td>Inadequate image quality or no ability to access image</td><td>Missing, poor, or inaccurate scoring</td><td>Track the scores of all pathologists and notify when scores are inconsistent</td></tr><tr><td>6. Operational risks/risks to patients</td><td>Long turnaround time</td><td>Delay in patient randomization and treatment</td><td>Timeline tracking incorporated in workflow</td></tr><tr><td>7. Operational risks</td><td>Data management failure</td><td>Errors in collecting manual scores, typos, data conversion issues</td><td>Structured digital scores from pathologist to the analyst</td></tr></tbody></table>

<figure><img src="image_4.png" /><figcaption>**Fig. 1 Organization of a workflow for reliable and timely biomarker scoring in a general single-center or multi-center trial.**</figcaption></figure>

Personnel at individual centers scan the slides after processing by the local pathology department. Digital slides are uploaded to a central web-based repository, such as Slide Score. A study-specific identifier is assigned to each sample. The central manager is notified by the system when new slides are available and requests pathologists to review it. When a consensus score is obtained, the trial office is notified for randomization of the patient.

the reviewers and tracked these during the study progress (risk 6; Supplementary Fig. 1). Pathologists were notified via email the next working day when the slide was not scored yet to minimize the waiting period to start treatment (risk 6). Finally, using Slide Score, we reduced the risks of typos and other manual errors by collecting all slides within one online study group (collection of slides) and a customized scoring form was built to standardize scores and obtain structured data (risk 7).

## IMPLEMENTATION OF WORKFLOW IN THE TONIC TRIAL

The TONIC trial (NCT02499367)⁶ is a phase II, non-comparative randomized multi-cohort single-center trial (full title: Adaptive phase II randomized non-comparative Trial Of Nivolumab after Induction treatment in TNBC patients), designed to assess the efficacy of induction of an anti-cancer immune response by low-dose chemotherapy or irradiation to increase response to anti-PD-1 in patients with metastatic TNBC. In the first part of the trial⁶, patients with metastatic TNBC were randomized to nivolumab (1) without induction or two-week low-dose induction, with (2) irradiation (3 × 8 Gy), (3) cyclophosphamide, (4) cisplatin, or (5) doxorubicin, all followed by nivolumab (anti-PD-1; 3 mg/kg). Based on a Simon's two-stage design¹⁷ and prespecified pick-the-winner criteria, only the doxorubicin cohort was allowed to continue in the second part of the trial⁶. In the second part of the TONIC trial, patients were randomized between anti-PD-1 monotherapy (control group) and two cycles of low-dose doxorubicin (15 mg flat dose, weekly), followed by anti-PD-1 (Supplementary Fig. 2a). Randomization was stratified for sTILs. Stratification is done by dividing patients in two categories, namely sTIL<sub>high</sub> (equal or exceeding 5%) and sTIL<sub>low</sub> (lower than 5%). The cut-off was determined based on data obtained in the first part of the TONIC trial, in which we observed that sTILs were predictive of response to anti-PD-1, both continuous and when a cut-off of 5% was used⁶. These data confirmed the predictive value of sTILs of at least 5% in another trial, which tested the efficacy of anti-PD-1 in patients with metastatic TNBC⁴. The full protocol, including four amendments, and the informed consent form were approved by the medical-ethical committee of The Netherlands Cancer Institute. All patients provided written informed consent before enrollment. The trial was registered on 17 August 2015. The 47 patients of the second part of the trial were randomized between March 2018
and July 2019. Full eligibility criteria and trial procedures have been described previously⁶.

In the second part of the TONIC trial, we could implement our workflow with a focus on accurate and reproducible sTIL scores within a reasonable timeframe after a biopsy was taken (72 h). For all 47 patients included in the trial, reliable sTIL scores were obtained with 45 biopsies scored within the 72-h timeframe (Supplementary Fig. 3). During the course of the study, the server of Slide Score was available 99.9% of the time. Five biopsies had to be re-evaluated due to a discrepancy in the categorical scores, when not all pathologists agreed on the appropriate category of the sTIL score (lower than 5% versus higher or equal to 5%). In three of these cases the score of one pathologist was higher (5 or 10%) than the score of the other two or three pathologists (0–3%). The average sTIL score was obtained and the pathologist causing the disagreement was notified. In the fourth and fifth case, two pathologists scored 5 and 10%, whereas the other pathologists scored 1%. All four pathologists were notified of the disagreement and a consensus score of 5% was obtained. We observed an intraclass correlation coefficient of 0.94 (95% confidence interval (CI): 0.91–0.97) for sTILs as a continuous variable. Interrater agreement for the categorical variable used in the stratification (sTILs <5% or ≥5%) was 0.86 (multirater Fleiss' *k*¹⁸; 95% CI: 0.73–1; Supplementary Fig. 2c). In the anti-PD-1 monotherapy cohort, we observed that 13 out of 23 patients (56.5 %) had sTILs below 5%, as compared to 15 out of 24 patients in the doxorubicin cohort (62.5 %; Fisher's exact test *p* value 0.77). The distribution of the sTIL scores is depicted in Supplementary Fig. 2d. These data indicate effective stratification based on the cut-off of 5%, but a slightly uneven distribution in the higher ranges of sTIL scores (10% or higher) inherent to the use of our cut-off. We observed a median time from biopsy until the scanning of the H&E slide of 30 h (range 24–98 h) and a median time from the biopsy until at least three scores were obtained of 43 h (range 27–106 h). In total, the median time from biopsy until registration in the patient records was 49 h (range 41–106 h; Supplementary Fig. 2e), with 96% of biopsies scored within 72 h. Two biopsies were not scored within the 72-h time limit, due to additional processing of one sample and one delay in registration time due to the absence of the central manager (Supplementary Figs 1 and 3).

## ADVANTAGES AND LIMITATIONS OF A WEB-BASED RISK-MITIGATION WORKFLOW

Our proposed solutions involved standardization of our workflow, obtaining digital images and the use of a web-based tool such as Slide Score for the managing and scoring of digital images. Anticipating the incorporation of digital images in routine diagnostics, our workflow shows that it is feasible for a pathologist to score digital images with high reliability. Moreover, a web-based tool can facilitate the process of coordinated uploading of digital images, pseudonymizing slides, and regulate access to studies and proper data management. Web-based platforms are therefore of high interest in biomarker research and can help with automation that can be transferred to clinical practice in the future.

In this study, we obtained sTIL scores within 72 h after a biopsy was taken, which is a reasonable timeframe for clinicians to start randomization of patients to treatment arms in a clinical trial. We observed an excellent interrater agreement score between our panel of four expert pathologists. In an accompanying paper⁷ we demonstrate using data from three RING studies of the TIL Working Group that the concordance achieved using a risk-management approach as detailed in this study is substantially higher than observed outside this risk-management perspective as observed in the three RING studies and in other published studies¹⁹,²⁰. However, our sample size is small and the four pathologists in the current study were trained and experienced in

the scoring of sTILs in breast cancer. Also, the biopsies used in this study were checked for containing sufficient tumor cells (≥100 cells) before the slide was scored for sTILs, which could have further improved our results. In the future, it is to be expected that computational workflows will further improve the scoring of sTILs¹³. Although we obtained reliable and timely results in 96% of cases, the presence of a central manager is crucial. In one case there was a delay in registration time due to the absence of the central manager. The manual intervention of quality checks, processing of the slides, and data cannot be circumvented in our workflow.

Stratification in this study was performed using sTILs as a binary variable (lower than 5% versus higher or equal to 5%). Consequently, we observed an uneven distribution in continuous sTILs scores between the cohorts (Supplementary Fig. 2d). This was mainly due to more patients with sTILs scores above 10% in the anti-PD-1 monotherapy cohort. Inherent to the use of a binary cut-off for stratification, the median of the continuous measurement might still differ between cohorts. Alternatively, multiple categories for the same variable can be used in stratification. However, this approach generates more strata, with lower number of patients in each stratum, possibly leading to an imbalance in distribution²¹,²². Moreover, at the time of writing of this paper no cut-offs for sTILs are established and/or properly validated for predictive purposes.

During the trial, we continuously monitored whether our strategy was still feasible within the set timeframe by means of regular evaluation by the pathologists and the study coordinators. This led to rapid adjustment of the workflow if needed, ensuring the quality of the sTIL scores. For example, pathologists could easily login remotely and score a digital H&E outside the hospital ensuring that sTILs were still scored within 72 h after biopsy. Ongoing evaluation during the clinical trial is of critical importance for risk mitigation in biomarker research¹.

## FUTURE APPLICATIONS OF THE WORKFLOW

Our strategy can serve as a template for risk management and mitigation of all identified risks in future clinical trials incorporating biomarkers for inclusion, enrichment, or stratification. By no means will risks identified in this study be similar for all clinical trials. Each trial will have its own risks that need to be mitigated, although there will be similarities between the risks across clinical trials. Defining the risks that come with biomarker development will help tested biomarkers eventually make their way to the clinic. However, one may even argue that a similar risk-management strategy can be applied in daily practice. In the BELLINI trial (NCT03815890), two cycles of neo-adjuvant anti-PD-1 are administered in patients with early-stage TNBC or luminal B breast cancer. All patients are required to have at least 5% sTILs in the pretreatment biopsy and patients are thereafter stratified in three sTIL categories. Our workflow will be used to ensure timely and reliable sTIL scores for the right patient selection. By using our workflow, scoring of sTILs is highly standardized, allowing also smaller centers with less extensive experience in sTILs scoring to participate in a clinical trial.

## CONCLUSIONS

In contemporary clinical research there is an increasing trend toward the use of biomarker results obtained in daily practice to select patients for inclusion in clinical trials. Therefore, continuous monitoring of the predefined risks and the solutions can improve the quality of the biomarker, as can be applied in a clinical trial setting, as well as in daily practice. The recommendations of the TIL Working Group⁸,¹⁰ for appropriate scoring, the risk-management framework of the NCI, NCRI, and EORTC Working Groups¹, as well as our proposed strategies to reduce risks will
help to effectively and efficiently improve the incorporation of biomarkers in clinical trials in first instance, herewith illustrated using sTILs as a paradigm of this development.

## DATA AVAILABILITY

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Received: 16 July 2019; Accepted: 18 February 2020;
Published online: 12 May 2020

## REFERENCES

1. Hall, J. A., Salgado, R., Lively, T., Sweep, F. & Schuh, A. A risk-management approach for effective integration of biomarkers in clinical trials: perspectives of an NCI, NCRI, and EORTC working group. *Lancet Oncol.* **15**, e184–193 (2014).

2. Loi, S. et al. Tumor-infiltrating lymphocytes and prognosis: a pooled individual patient analysis of early-stage triple-negative breast cancers. *J. Clin. Oncol.* **37**, 559–569 (2019).

3. Denkert, C. et al. Tumour-infiltrating lymphocytes and prognosis in different subtypes of breast cancer: a pooled analysis of 3771 patients treated with neoadjuvant therapy. *Lancet Oncol.* **19**, 40–50 (2018).

4. Loi, S. et al. LA13—relationship between tumor infiltrating lymphocyte (TIL) levels and response to pembrolizumab (pembro) in metastatic triple-negative breast cancer (mTNBC): results from KEYNOTE-086. *Ann. Oncol.* **28**(Suppl_5), v605–49, https://doi.org/10.1093/annonc/mdx440 (2017).

5. Emens, L. A. et al. Long-term clinical outcomes and biomarker analyses of atezolizumab therapy for patients with metastatic triple-negative breast cancer: a phase 1 study. *JAMA Oncol.* https://doi.org/10.1001/jamaoncol.2018.4224 (2018).

6. Voorwerk, L. et al. Immune induction strategies in metastatic triple-negative breast cancer to enhance the sensitivity to PD-1 blockade: the TONIC trial. *Nat. Med.* https://doi.org/10.1038/s41591-019-0432-4 (2019).

7. Kos, Z. et al. Pitfalls in assessing stromal tumor infiltrating lymphocytes (sTILs) in breast cancer. *npj Breast Cancer*. https://doi.org/10.1038/s41523-020-0156-0 (2020).

8. Salgado, R. et al. The evaluation of tumor-infiltrating lymphocytes (TILs) in breast cancer: recommendations by an International TILs Working Group 2014. *Ann. Oncol.* **26**, 259–271 (2015).

9. Hayes, D. F. et al. Breaking a vicious cycle. *Sci. Transl. Med.* **5**, 196cm196 (2013).

10. Hendry, S. et al. Assessing tumor-infiltrating lymphocytes in solid tumors: a practical review for pathologists and proposal for a standardized method from the International Immunooncology Biomarkers Working Group: Part 1: assessing the host immune response, TILs in invasive breast carcinoma and ductal carcinoma in situ, metastatic tumor deposits and areas for further research. *Adv. Anat. Pathol.* **24**, 235–251 (2017).

11. Mroz, P., Parwani, A. V. & Kulesza, P. Central pathology review for phase III clinical trials: the enabling effect of virtual microscopy. *Arch. Pathol. Lab. Med.* **137**, 492–495 (2013).

12. Pell, R. et al. The use of digital pathology and image analysis in clinical trials. *J. Pathol.* **5**, 81–90 (2019).

13. Amgad, M. et al. Report on computational assessment of tumor infiltrating lymphocytes from the international immuno-oncology biomarker working group. *npj Breast Cancer*. https://doi.org/10.1038/s41523-020-0154-2 (2020).

14. Bankhead, P. et al. QuPath: Open source software for digital pathology image analysis. *Sci. Rep.* **7**, 16878 (2017).

15. Li, H. et al. Dysfunctional CD8 T cells form a proliferative, dynamically regulated compartment within human melanoma. *Cell* **176**, 775–789.e718 (2019).

16. Sauer, G. et al. Ultrasound-guided large-core needle biopsies of breast lesions: analysis of 962 cases to determine the number of samples for reliable tumour classification. *Br. J. Cancer* **92**, 231–235 (2005).

17. Simon, R. Optimal two-stage designs for phase II clinical trials. *Control. Clin. Trials* **10**, 1–10 (1989).

18. Fleiss, J. L. *Statistical Methods for Rates and Proportions* 2nd edn (1981).

19. Tramm, T. et al. Standardized assessment of tumor-infiltrating lymphocytes in breast cancer: an evaluation of inter-observer agreement between pathologists. *Acta Oncol.* **57**, 90–94 (2018).

20. Dieci, V. et al. Association of tumor-infiltrating lymphocytes with distant disease-free survival in the ShortHER randomized adjuvant trial for patients with early HER2+ breast cancer. *Ann. Oncol.* **30**, 418–423 (2019).

21. Therneau, T. M. How many stratification factors are “too many” to use in a randomization plan? *Control. Clin. Trials* **14**, 98–108 (1993).

22. Silcocks, P. How many strata in an RCT? A flexible approach. *Br. J. Cancer* **106**, 1259–1261 (2012).

## ACKNOWLEDGEMENTS

The Department of Pathology of the Netherlands Cancer Institute is thanked for the support of this study and ensuring the rapid turnaround times. The Breast Cancer Research Foundation and Bristol-Myers-Squibb (BMS) are thanked for financial support. We also thank the BMS-International Immuno-Oncology Network (BMS/IL-ON) and the Dutch Cancer Society (NKI2015-7710) for funding the clinical trial costs and this feasibility study (NKI2016-10510). S.L., R.S., and M.K. are supported by a grant from the Breast Cancer Research Foundation (BCRF, NY, US). The following is a list of current members of the International Immuno-Oncology Working Group (TILs Working Group). A member is defined as a person willing to be involved, informed and be part of the activities of the TILs Working Group. The authors alone are responsible for the views expressed in the work of the TIL Working Group and they do not necessarily represent the decisions, policy, or views of their employer.

## AUTHOR CONTRIBUTIONS

J.H. developed Slide Score and wrote the manuscript with L.V., R.S., M.K. and H.M.H. L. V. coordinated study procedures and performed data-analyses. M.v.S., I.N., S.A., S.D., G.V., T.O.N., S.S.B., S.M., W.F.S., C.S., D.L.R., S.H., C.D., S.L., Sh.L., J.M.S.B., G.P., D.A.D., M.C. U.C., A.T., J.A.H. and Z.K. gave critical input. M.d.M. provided logistical support with the sample processing. J.v.d.B., K.K.v.d.V., R.S. and H.M.H. scored the slides. K.S. performed the statistical analysis. M.K. is the principal investigator of the TONIC trial. J.H., L.V., R.S., M.K. and H.M.H. designed this feasibility study. All authors edited and approved the manuscript.

## COMPETING INTERESTS

J.H. is the owner of Slide Score. B.V. L.V., M.v.S., I.N., M.d.M., J.v.d.B., K.K.v.d.V., K.S., S.A., S.D., G.V., S.S.B., S.M., W.F.S., C.S., S.M.H., C.D., S.L., G.P., M.C.U.C., Z.K., and H.M.H. have nothing to disclose. T.O.N. has consulted for Nanostring and received compensation and has intellectual property rights/ownership interests from Bioclassified LLC, not related to the subject material under consideration and received funding support from the Canadian Cancer Society. D.L.R. reports research funding from AstraZeneca, Cepheid, Navigate BioPharma, NextCure, Lilly, and Ultivue; instrument support from Ventana, Akoya/PerkinElmer, and NanoString; advisory board of Amgen, AstraZeneca, Cell Signaling Technology, Cepheid, Daiichi Sankyo, GSK, Konica/Minolta, Merck, NanoString, PerkinElmer, Ventana, and Ultivue; consultancy for Biocept; honorarium and travel support from BMS; royalties from Rarecyte and is a founder and equity holder of PixelGear. Sh.L. receives research funding to her institution from Novartis, Bristol Meyers Squibb, Merck, Roche-Genentech, Puma Biotechnology, Pfizer, and Eli Lilly, acted as consultant (not compensated) to Seattle Genetics, Pfizer, Novartis, BMS, Merck, AstraZeneca, and Roche-Genentech and acted as consultant (paid to her institution) to Aduro Biotech. J.M.S.B. reports research funding from ThermoFisher, Genoptix, Agendia, NanoString Technologies, Stratifier GmbH, and Biotheranostics and advisory roles for Insight Genetics, BioNTech AG, Biotheranostics, Pfizer, RNA Diagnostics, and OncoXchange. J.M.S.B. reports the following patents: Methods and Devices for Predicting Anthracycline Treatment Efficacy, US utility (January 2017; 15/325,472; EPO – 15822898.1; Canada – not yet assigned), Systems, Devices and Methods for Constructing and Using a Biomarker, US utility (January 2017; 15/328,108; EPO – 15824751.0; Canada – not yet assigned), Histone gene module predicts anthracycline benefit (October 2016; PCT/CA2016/000247), 95-Gene Signature of Residual Risk Following Endocrine Treatment (December 2016; PCT/CA2016/000304), Immune Gene Signature Predicts Anthracycline Benefit (December 2016; PCT/CA2016/000305). D.A.D. is on the advisory board and consults for Oncology Analytics Inc., and has consulted for and received travel funds from Novartis for work unrelated to the current manuscript. A.T. reports benefits from ICR's Inventors Scheme associated with patents for one of PARP inhibitors in BRCA1/2-associated cancers. A.T. also reports Honoraria from Pfizer, Vertex, Prime Oncology, and Artios, honoraria and stock in InBioMotion, honoraria and financial support for research from AstraZeneca, Medivation, Myriad Genetics, and Merck Serono. J.A.H. is the director and owner of Vivactiv Ltd. R.S. reports research funding from Roche, Puma, and Merck; advisory board and consultancy for BMS; travel funding from Roche, Merck, and AstraZeneca, outside the scope of this work. M.K. reports funding to the institute from BMS, Roche and an advisory role for BMS, outside the submitted work.
ADDITIONAL INFORMATION

**Supplementary information** is available for this paper at https://doi.org/10.1038/
s41523-020-0155-1.

**Correspondence** and requests for materials should be addressed to R.S.

**Reprints and permission information** is available at http://www.nature.com/
reprints

**Publisher's note** Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

<figure><img src="image_5.png" /></figure>

**Open Access** This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.

© The Author(s) 2020

INTERNATIONAL IMMUNO-ONCOLOGY BIOMARKER WORKING GROUP

Aini Hyytiäinen<fnref n="37" />, Akira I. Hida³⁸, Alastair Thompson³⁹, Alex Lefevre⁴⁰, Alexander J. Lazar⁴¹, Allen Gown⁴², Amy Lo⁴³, Anna Sapino⁴⁴, Anant Madabhushi⁴⁵,⁴⁶, Andre Moreira⁴⁷, Andrea Richardson⁴⁸, Andrea Vingiani⁴⁹, Andrew H. Beck⁵⁰, Andrew M. Bellizzi⁵¹, Angel Guerrero⁵², Anita Grigoriadis⁵³,⁵⁴, Anna Ehinger⁵⁵, Ana Garrido-Castro⁵⁶, Anne Vincent-Salomon⁵⁷, Anne-Vibeke Laenkholm⁵⁸, Ashish Sharma⁵⁹, Ashley Cimino-Mathews⁶⁰, Ashok Srinivasan⁶¹, Balazs Acs⁶², Baljit Singh⁶³, Benjamin Calhoun⁶⁴, Benjamin Haibe-Kans⁶⁵, Benjamin Solomon⁶⁶, Bibhusal Thapa⁶⁷, Brad H. Nelson⁶⁸, Brandon D. Gallas⁶⁹, Carlos Castaneda⁷⁰,⁷¹, Carmen Ballesteros-Merino⁷², Carmen Criscitiello⁷³, Carolien Boeckx⁷⁴, Cecile Colpaert⁷⁵, Cecily Quinn⁷⁶, Chakra S. Chennubhotla⁷⁷, Charles Swanton⁷⁸, Cinzia Solinas⁷⁹, Crispin Hiley⁷⁸, Damien Drubay⁸⁰,⁸¹, Daniel Bethmann⁸², David A. Moore⁸³,⁸⁴, Denis Larsimont⁸⁵, Dhanusha Sabanathan⁸⁶, Dieter Peeters⁸⁷, Dimitrios Zardavas⁸⁸, Doris Höflmayer⁸⁹, Douglas B. Johnson⁹⁰, E. Aubrey Thompson⁹¹, Edi Brogi⁹², Edith Perez⁹³, Ehab A. ElGabry⁹⁴, Elisabeth Specht Stovgaard⁹⁵, Elizabeth F. Blackley⁶⁶, Elvire Roblin⁹⁶,⁹⁷, Emily Reisenbichler¹⁸, Enrique Bellolio⁹⁸,⁹⁹, Eva Balslev⁹⁵, Ewa Chmielik¹⁰⁰, Fabien Gaire¹⁰¹, Fabrice Andre¹⁰², Fang-I Lu¹⁰³, Farid Azmoudeh-Ardalan¹⁰⁴, Federico Rojo¹⁰⁵,¹⁰⁶, Tina Gruosso¹⁰⁷, Francesco Ciompi¹⁰⁸, Franklin Peale¹⁰⁹, Fred R. Hirsch¹¹⁰, Frederick Klauschen²¹, Frédérique Penault-Llorca¹¹¹, Gabriela Acosta Haab¹¹², Gelareh Farshid¹¹³, Gert van den Eynden¹¹⁴, Giuseppe Curigliano¹¹⁵,¹¹⁶, Giuseppe Floris¹¹⁷,¹¹⁸, Glenn Broeckx¹¹⁹, Gonzalez-Ericsson¹²⁰, Harmut Koeppen⁴³, Harry R. Haynes¹²¹, Heather McArthur¹²², Heikki Joensuu¹²³, Helena Olofsson¹²⁴, Huang-Chun Lien¹²⁵, I-Chun Chen¹²⁶,¹²⁷, Ian Cree¹²⁸, Isabel Frahm¹²⁹, Iva Bricc¹³⁰, Jack Chan¹³¹, James Ziai⁴³, Jane Brock¹³², Jelle Wesseling¹³³, Jennifer Giltnane⁴³, Jennifer K. Kerner¹³⁴, Jeppe Thagaard¹³⁵,¹³⁶, Jeremy P. Braybrooke¹³⁷,¹³⁸, Jeroen A. W. M. van der Laak¹⁰⁸, Jerome Lemonnier¹³⁹, Jiping Zha¹⁴⁰, Joana Ribeiro¹⁴¹, Jochen K. Lennerz¹⁴², Jodi M. Carter¹⁴³, Joel Saltz¹⁴⁴, Johan Hartman¹⁴⁵, Johannes Hainfellner¹⁴⁶, John Le Quesne¹⁴⁷, Jonathon W. Juco¹⁴⁸, Jorge Reis-Filho⁹²,¹⁴⁹, Joselyn Sanchez¹⁵⁰, Joseph Sparano¹⁵¹, Joël Cucherousset¹⁵², Juan Carlos Araya⁹⁸, Julien Adam¹⁵³, Justin M. Balko¹⁵⁴, Kai Saeger¹⁵⁵, Kalliopi Siziopikou¹⁵⁶, Karen Willard-Gallo¹⁵⁷, Karsten Weber²³, Katherine L. Pogue-Geile¹⁵⁸, Keith E. Steele¹⁴⁰, Kenneth Emancipator¹⁴⁸, Khalid AbdulJabbar¹⁵⁹, Khalid El Bairi¹⁶⁰, Kim R. M. Blenman¹⁶¹, Kimberly H. Allison¹⁶², Konstanty Korski¹⁶³, Lajos Pusztai¹⁶¹, Laura Comerma¹⁶⁴,¹⁰⁶, Laurence Buisseret¹⁵⁷, Lee A. D. Cooper¹⁶⁵, Leming Shi¹⁶⁶, Loes F. S. Kooreman¹⁶⁷, Luciana Molinero¹⁰⁹, M. Valeria Estrada¹⁶⁸, Magali Lacroix-Triki¹⁶⁹, Maise Al Bakir⁷⁸, Manu M. Sebastian¹⁷⁰, Marc van de Vijver¹⁷¹, Marcelo Luiz Balancin¹⁷²,¹⁷³, Maria Vittoria Dieci¹⁷⁴, Marie-Christine Mathieu¹⁷⁵, Marlon C. Rebelatto¹⁴⁰, Martine Piccart¹⁷⁶, Matthew G. Hanna⁹², Matthew P. Goetz⁹³, Matthias Preusser¹⁴⁶, Mehrnoush Khojasteh¹⁷⁷, Melinda E. Sanders¹⁷⁸, Meredith M. Regan¹⁷⁹,¹⁸⁰, Michael Barnes¹⁸¹, Michael Christie¹⁸², Michael Misialek¹⁸³, Michail Ignatiadis¹⁸⁴, Mieke van Bockstal¹⁸⁵, Miluska Castillo⁷¹, Mohamed Amgad¹⁸⁶, Nadia Harbeck¹⁸⁷, Nadine Tung¹⁸⁸, Nele Laudus¹⁸⁹, Nicolas Sirtaine¹⁹⁰, Nicole Burchardi¹⁹¹, Nils Ternes¹⁴, Nina Radosevic-Robin¹⁹², Oleg Gluz¹⁹³, Oliver Grimm¹⁰¹, Paolo Nuciforo¹⁹⁴, Paul Jank¹⁹⁵, Paula Gonzalez-Ericsson¹⁹⁶, Pawan Kirtani¹⁹⁷, Petar Jelinic¹⁴⁸, Peter H. Watson¹⁹⁸, Peter Savas²⁴,¹⁹⁹, Prudence A. Francis²⁰⁰,²⁰¹, Prudence A. Russell²⁰², Rajendra Singh²⁰³, Rim S. Kim²⁰⁴, Robert H. Pierce²⁰⁵, Robert Hills²⁰⁶, Roberto Leon-Ferre⁹³, Roland de Wind¹⁹⁰, Ruohong Shui²⁰⁷, Sabine De Clercq²⁰⁸, Sam Leung²⁰⁹, Sami Tabbarah²¹⁰, Sandra C. Souza²¹¹, Sandra O'Toole²¹², Sandra Swain²¹³, Sarah Dudgeon²¹⁴, Scooter Willis²¹⁵, Scott Ely²¹⁶, Seong-Rim Kim²¹⁷, Shahinaz Bedri²¹⁸, Sheeba Irshad²¹⁹,²²⁰, Shi-Wei Liu²²¹, Shom Goel²⁰⁰,²⁰¹, Shona Hendry²²², Simonetta Bianchi²²³, Sofia Bragança²²⁴, Soonmyung Paik⁶¹, Stephan Wienert²²⁵, Stephen B. Fox²²², Stephen J. Luen²⁴, Stephen Naber²²⁶, Stuart J. Schnitt³⁰,²²⁷, Luz F. Sua²²⁸, Sunil R. Lakhani²²⁹, Susan Fineberg²³⁰, Teresa Soler²³¹, Thomas Gevaert²³², Timothy D'Alfonso²³³, Tom John²³⁴, Tomohagu Sugie²³⁵, Uday Kurkure²³⁶, Veerle Bossuyt²³⁷, Venkata Manem⁶⁵, Vincente Peg Cámara²³⁸, Weida Tong²³⁹, Weijie Chen⁶⁹, Wentao Yang²⁰⁷, William T. Tran²¹⁰, Yihong Wang²⁴⁰, Yinyin Yuan¹⁵⁹, Yves Allory²⁴¹, Zaheed Husain²⁴² and Zsuzsanna Bago-Horvath²⁴³

<footnote n="37">³⁷Department of Oral and Maxillofacial Diseases, Helsinki, Finland. ³⁸Department of Pathology, Matsuyama Shimin Hospital, Matsuyama, Japan. ³⁹Surgical Oncology, Baylor College of Medicine, Houston, TX, USA. ⁴⁰Roche Diagnostics, Mechelen, Belgium. ⁴¹Departments of Pathology, Genomic Medicine, Dermatology, and Translational Molecular Pathology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA. ⁴²PhenoPath Laboratories, Seattle, WA, USA. ⁴³Research Pathology, Genentech Inc., South San Francisco, CA, USA. ⁴⁴University of Turin/Candiolo Cancer Institute - FPO, IRCCS, Candiolo, Italy. ⁴⁵Case Western Reserve University, Cleveland, OH, USA. ⁴⁶Louis Stokes Cleveland Veterans Health Administration Medical Center, Cleveland, OH, USA. ⁴⁷Pulmonary Pathology, New York University Center for Biospecimen Research and Development, New York University, New York, NY, USA. ⁴⁸Department of Pathology, Johns Hopkins Hospital, Baltimore, MD, USA. ⁴⁹Department of Pathology, Istituto Europeo di Oncologia, University of Milan, Milan, Italy. ⁵⁰PathAI, Inc, Cambridge, MA, USA. ⁵¹Department of Pathology, University of Iowa Hospitals and Clinics, Iowa City, IA, USA. ⁵²Department of Oncology, IVO, Valencia, Spain. ⁵³Cancer Bioinformatics Lab, Cancer Centre at Guy's Hospital, London, UK. ⁵⁴School of Life Sciences and Medicine, King's College London, London, UK. ⁵⁵Lund University, Skane University Hospital, Department of Clinical Sciences Lund, Oncology and Pathology, Lund, Sweden. ⁵⁶Dana Farber Cancer Institute, Boston, MA, USA. ⁵⁷Institut Curie, Paris Sciences Lettres Université, Inserm U934, Department of Pathology, Paris, France. ⁵⁸Department of Surgical Pathology, Zealand University Hospital, Køge, Denmark. ⁵⁹Department of Biomedical Informatics, Emory University, GA, USA. ⁶⁰Departments of Pathology and Oncology, The Johns Hopkins Hospital, Baltimore, MD, USA. ⁶¹National Surgical Adjuvant Breast and Bowel Project Operations Center/NRG Oncology, Pittsburgh, PA, USA. ⁶²Department of Pathology, Karolinska Institutet, Karolinska, Sweden.</footnote>
63Department of Pathology, New York University Langone Medical Centre, New York, NY, USA. 64Department of Pathology and Laboratory Medicine, UNC School of Medicine, Chapel Hill, NC, USA. 65Bioinformatics and Computational Genomics Laboratory, Princess Margaret Cancer Center, Toronto, ON, Canada. 66Department of Medical Oncology, Peter MacCallum Cancer Centre, Melbourne, VIC, Australia. 67Department of Medicine, University of Melbourne, Parkville, VIC, Australia. 68Trev & Joyce Deeley Research Centre, British Columbia Cancer Agency, Victoria, BC, Canada. 69Division of Imaging, Diagnostics, and Software Reliability (DIDSR), Office of Science and Engineering Laboratories (OSEL), Center for Devices and Radiological Health (CDRH), Rockville, MD, USA. 70Department of Research, Instituto Nacional de Enfermedades Neoplásicas, Lima, Peru. 71Department of Research, Instituto Nacional de Enfermedades Neoplásicas, Lima 15038, Peru. 72Providence Cancer Research Center, Portland, Oregon, USA. 73Department of Medical Oncology, Istituto Europeo di Oncologia, Milan, Italy. 74Roche Diagnostics, Mechemel, Belgium. 75Department of Pathology, AZ Turnhout, Turnhout, Belgium. 76Department of Pathology, St Vincent's University Hospital and University College Dublin, Dublin, Ireland. 77Department of Computational and Systems Biology, University of Pittsburgh, Pittsburgh, PA, USA. 78Cancer Research UK Lung Cancer Centre of Excellence, University College London Cancer Institute, University College London, London, UK. 79Azienda AUSL, Regional Hospital of Aosta, Aosta, Italy. 80Université Paris-Sud, Institut National de la Santé et de la Recherche Médicale, Villejuif, France. 81Gustave Roussy, Université Paris-Saclay, Villejuif, France. 82University Hospital Halle (Saale), Institute of Pathology, Halle, (Saale), Germany. 83Department of Pathology, UCL Cancer Institute, UCL, London, UK. 84University College Hospitals NHS Trust, London, UK. 85Department of Pathology, Jules Bordet Institute, Brussels, Belgium. 86Department of Clinical Medicine, Macquarie University, Sydney, Australia. 87HistoGeneX NV, Antwerp, Belgium and AZ Sint-Maarten Hospital, Mechemel, Belgium. 88Oncology Clinical Development, Bristol-Myers Squibb, Princeton, NJ, USA. 89Institut für Pathologie, UK Hamburg, Germany. 90Department of Medicine, Vanderbilt University Medical Centre, Nashville, TN, USA. 91Department of Cancer CV, Jacksonville, FL, USA. 92Department of Pathology, Memorial Sloan Kettering Cancer Center, New York, NY, USA. 93Department of Oncology, Mayo Clinic, Rochester, MN, USA. 94Roche, Tucson, AZ, USA. 95Department of Pathology, Herlev and Gentofte Hospital, Gentofte, Denmark. 96Université Paris-Saclay, Univ. Paris-Sud, Villejuif, France. 97Service de biostatistique et d'épidémiologie, Gustave Roussy, Villejuif, France. 98Department of Pathology, Universidad de La Frontera, Temuco, Chile. 99Departamento de Anatomía Patológica, Universidad de La Frontera, Temuco, Chile. 100Tumor Pathology Department, Maria Sklodowska-Curie Memorial Cancer Center, Gliwice, Poland. 101Pathology and Tissue Analytics, Roche, Neuherberg, Germany. 102Department of Medical Oncology, Gustave Roussy, Villejuif, France. 103Sunnybrook Health Sciences Centre, Toronto, ON, Canada. 104Tehran University of Medical Sciences, Tehran, Iran. 105Pathology Department, Instituto de Investigación Sanitaria Fundación Jiménez Díaz (IIS-FJD), Madrid, Spain. 106GEICAM-Spanish Breast Cancer Research Group, Madrid, Spain. 107Translational Research, Montreal, Canada. 108Computational Pathology Group, Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands. 109Oncology Biomarker Development, Genentech-Roche, Neuherberg, Germany. 110Division of Medical Oncology, Department of Medicine, University of Colorado Anschutz Medical Campus, Aurora, CO, USA. 111Centre de Lutte Contre le cancer - Centre Jean Perrin, Clermont-Ferrand, France. 112Department of Pathology, Hospital de Oncologia Maria Curie, Buenos Aires, Argentina. 113Directorate of Surgical Pathology, SA Pathology, Adelaide, Australia. 114Department of Pathology, GZA-ZNA Ziekenhuizen, Wilrijk, Belgium. 115University of Milano, Instituto Europeo di Oncologia, IRCCS, Milano, Italy. 116Division of Early Drug Development for Innovative Therapy, IEO, European Institute of Oncology IRCCS, Milan, Italy. 117Department of Imaging and Pathology, Laboratory of TranslCal Cell & Tissue Research, Leuven, Belgium. 118KU Leuven-University Hospitals Leuven, Department of Pathology, Leuven, Belgium. 119Department of Pathology, University Hospital of Antwerp, Antwerp, Belgium. 120Paula I, Breast Cancer Research Program, Vanderbilt University Medical Center, Nashville, TN, USA. 121Translational Health Sciences, Department of Cellular Pathology, North Bristol NHS Trust, University of Bristol, Bristol, UK. 122Medical Oncology, Department of Medicine, Cedars-Sinai Medical Center, Los Angeles, CA, USA. 123Helsinki University Central Hospital, Helsinki, Finland. 124Department of Clinical Pathology, Akademiska University Hospital, Uppsala, Sweden. 125Department of Pathology, National Taiwan University Hospital, Taipei, Taiwan. 126Department of Oncology, National Taiwan University Cancer Center, Taipei, Taiwan. 127Graduate Institute of Oncology, College of Medicine, National Taiwan University, Taipei, Taiwan. 128International Agency for Research on Cancer (IARC), World Health Organization, Lyon, France. 129Department of Pathology, Sanatorio Mater Dei, Buenos Aires, Argentina. 130Institute of Pathology, Medical University of Graz, Graz, Austria. 131Department of Oncology, National Cancer Centre Singapore, Singapore, Singapore. 132Department of Pathology, Brigham and Women's Hospital, Boston, MA, USA. 133Department of Pathology, Netherlands Cancer Institute, Amsterdam, The Netherlands. 134PathAI Inc, Cambridge, MA, USA. 135Visiopharm A/S, Hørsholm, Denmark. 136DTU Compute, Department of Applied Mathematics, Technical University of Denmark, Lyngby, Denmark. 137Nuffield Department of Population Health, University of Oxford, Oxford, UK. 138Department of Medical Oncology, University Hospitals Bristol NHS Foundation Trust, Bristol, UK. 139R&D UNICANCER, Paris, France. 140Translational Sciences, MedImmune, Gaithersberg, MD, USA. 141Breast Unit, Champalmaud Clinical Centre, Lisboa, Portugal. 142Department of Pathology, Massachusetts General Hospital, Boston, MA, USA. 143Department of Laboratory Medicine and Pathology, Mayo Clinic, Rochester, MN, USA. 144Biomedical Informatics Department, Stony Brook University, Stony Brook, NY, USA. 145Department of Oncology and Pathology, Karolinska Institutet and University Hospital, Solna, Sweden. 146Department of Medicine, Clinical Division of Oncology, Comprehensive Cancer Centre Vienna, Medical University of Vienna, Vienna, Austria. 147Leicester Cancer Research Centre, University of Leicester, Leicester, and MRC Toxicology Unit, University of Cambridge, Cambridge, UK. 148Merck & Co., Inc, Kenilworth, USA. 149Human Oncology and Pathogenesis Program, Memorial Sloan Kettering Cancer Center, New York, NY, USA. 150Department of Research, Instituto Nacional de Enfermedades Neoplásicas, Lima 15038, Peru. 151Department of Medicine, Department of Obstetrics and Gynecology and Women's Health, Albert Einstein Medical Center, Bronx, USA. 152GHI Le Raincy-Montfermelles, Iles, Ile-de-France, France. 153Department of Pathology, Gustave Roussy, Grand Paris, France. 154Departments of Medicine and Cancer Biology, Vanderbilt University Medical Centre, Nashville, TN, USA. 155VMscope GmbH, Berlin, Germany. 156Department of Pathology, Breast Pathology Section, Northwestern University, Chicago, IL, USA. 157Molecular Immunology Unit, Institut Jules Bordet, Université Libre de Bruxelles, Brussels, Belgium. 158NSABP/NRG Oncology, Pittsburgh, PA, USA. 159Division of Molecular Pathology, Centre for Evolution and Cancer, The Institute of Cancer Research, London, UK. 160Cancer Biomarkers Working Group, Faculty of Medicine and Pharmacy, Université Mohamed Premier, Oujda, Morocco. 161Yale Cancer Center Genetics, Genomics and Epigenetics Program, Yale School of Medicine, New Haven, CT, USA. 162Pathology Department, Stanford University Medical Centre, Stanford, CA, USA. 163Pathology and Tissue Analytics, Roche Innovation Centre Munich, Pernberg, Germany. 164Pathology Department, Hospital del Mar, Parc de Salut Mar, Barcelona, Spain. 165Department of Pathology, Northwestern University Feinberg School of Medicine, Chicago, IL, USA. 166Center for Pharmacogenomics and Fudan-Zhangjiang, Center for Clinical Genomics School of Life Sciences and Shanghai Cancer Center, Fudan University, Fudan, China. 167GROW - School for Oncology and Developmental Biology, Maastricht University Medical Centre and Department of Pathology, Maastricht University Medical Centre, Maastricht, The Netherlands. 168Biorepository and Tissue Technology Shared Resources, University of California San Diego, San Diego, CA, USA. 169Department of Pathology, Gustave Roussy, Villejuif, France. 170Departments of Epigenetics and Molecular Carcinogenesis, The University of Texas MD Anderson Cancer Center, Houston, TX, USA. 171Department of Pathology, Academic Medical Centre, Amsterdam, The Netherlands. 172Department of Pathology, University of São Paulo, São Paulo, Brazil. 173Hospital das Clínicas, São Paulo, Brasil. 174Department of Surgery, Oncology and Gastroenterology, University of Padova, Padua, Italy. 175Department of Medical Biology and Pathology, Gustave Roussy Cancer Campus, Villejuif, France. 176Institut Jules Bordet, Université Libre de Bruxelles, Brussels, Belgium. 177Roche Tissue Diagnostics, Digital Pathology, Santa Clara, CA, USA. 178Department of Pathology, Microbiology and Immunology, Vanderbilt University Medical Centre, Nashville, TN, USA. 179Division of Biostatistics, Dana-Farber Cancer Institute, Boston, MA, USA. 180Harvard Medical School, Boston, MA, USA. 181Roche Diagnostics Information Solutions, Belmont, CA, USA. 182Department of Anatomical Pathology, Royal Melbourne Hospital, Parkville, VIC, Australia. 183Vernon Cancer Center, Newton-Wellesley Hospital, Newton, MA, USA. 184Department of Medical Oncology, Institut Jules Bordet, Université Libre de Bruxelles, Brussels, Belgium. 185Service de pathologique, Cliniques universitaires Saint-Luc, Bruxelles, Belgique. 186Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, GA, USA. 187Breast Center, Dept. OB&GYN and CCC (LMU), University of Munich, Munich, Germany. 188Division of Hematology-Oncology, Beth Israel Deaconess Medical Center, Boston, MA, USA. 189University of Leuven, Leuven, Belgium. 190Department of Pathology, Institut Jules Bordet, Université Libre de Bruxelles, Brussels, Belgium. 191German Breast Group, Neu-Isenburg, Germany. 192Department of Surgical Pathology and Biopathology, Jean Perrin Comprehensive Cancer Centre, Clermont-Ferrand, France. 193Johanniter GmbH - Evangelisches Krankenhaus Bethesda Mönchengladbach, West German Study Group, Mönchengladbach, Germany. 194Molecular Oncology Group, Vall d'Hebron Institute of Oncology, Barcelona, Spain. 195Department of Pathology, University of Marburg, Marburg, Germany. 196Breast Cancer Program, Vanderbilt-Ingraham Cancer Center, Vanderbilt University Medical Center, Nashville, TN, USA. 197Department of Histopathology, Manipal Hospitals Dwarka, New Delhi, India. 198Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, BC, Canada. 199The Sir Peter MacCallum Department of Oncology, University of Melbourne, Melbourne, VIC, Australia. 200Peter MacCallum Cancer Centre, Melbourne, VIC, Australia. 201Sir Peter MacCallum Department of Oncology, University of Melbourne, Melbourne, VIC, Australia. 202Department of Anatomical Pathology, St Vincent's Hospital Melbourne, Fitzroy, VIC, Australia. 203Icahn School of Medicine at Mt. Sinai, New York, NY, USA. 204NRG Oncology/NSABP, Pittsburgh, PA, USA. 205Cancer Immunotherapy Trials Network, Central Laboratory and Program in Immunology, Fred Hutchinson Cancer Research Center, Seattle, WA, USA. 206Clinical Trial Service Unit & Epidemiological Studies Unit, University of Oxford, Oxford, UK. 207Department of Pathology, Fudan University Cancer Center, Shanghai, China. 208Department of Pathology, GZA-ZNA Hospitals, Antwerp, Belgium. 209University of British Columbia, Vancouver, BC, Canada. 210Department of Radiation Oncology, Odette Cancer Centre, Sunnybrook Research Institute, Toronto, ON, Canada. 211Merck Oncology, Kenilworth, NJ, USA. 212The Cancer Research Program, Garvan Institute of Medical Research, Darlinghurst, NSW, Australia. 213Georgetown University Medical Center, Washington, DC, USA. 214FDA/CDRH/OSEL/
Division of Imaging, Diagnostics, and Software Reliability, Silver Spring, MD, USA. ²¹⁵Department of Molecular and Experimental Medicine, Avera Cancer Institute, Sioux Falls, SD, USA. ²¹⁶Translational Medicine, Bristol-Myers Squibb, Princeton, NJ, USA. ²¹⁷National Surgical Adjuvant Breast and Bowel Project Operations Center/NRG Oncology, Pittsburgh, PA, USA. ²¹⁸Anatomic Pathology, Boston, MA, USA. ²¹⁹Guy's Hospital, London, UK. ²²⁰King's College London, London, UK. ²²¹Peking University First Hospital Breast Disease Center, Beijing, China. ²²²Department of Pathology, Peter MacCallum Cancer Centre, Melbourne, VIC, Australia. ²²³Dipartimento di Scienze della Salute (DSS), Firenze, Italy. ²²⁴Department of Oncology, Champalmaud Clinical Centre, Lisbon, Portugal. ²²⁵Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Institute of Pathology, Berlin, Germany. ²²⁶Department of Pathology and Laboratory Medicine, Tufts Medical Center, Boston, MA, USA. ²²⁷Dana-Farber Cancer Institute, Boston, MA, USA. ²²⁸Department of Pathology, Fundación Valle del Lili, Cali, Valle del Cauca, Colombia. ²²⁹The University of Queensland Centre for Clinical Research and Pathology Queensland, Brisbane, QLD, Australia. ²³⁰Department of Pathology, Montefiore Medical Center and the Albert Einstein College of Medicine, Bronx, NY, USA. ²³¹Department of Pathology, University Hospital of Bellvitge, Oncobell, IDIBELL, L'Hospitalet del Llobregat, Barcelona 08908 Catalonia, Spain. ²³²Department of Development and Regeneration, Laboratory of Experimental Urology, KU Leuven, Leuven, Belgium. ²³³Department of Pathology, Memorial Sloan Kettering Cancer Center, New York, NY, USA. ²³⁴Department of Medical Oncology, Austin Health, Heidelberg, VIC, Australia. ²³⁵Department of Surgery, Kansai Medical School, Hirakata, Japan. ²³⁶Roche Tissue Diagnostics, Digital Pathology, Santa Clara, CA, USA. ²³⁷Department of Pathology, Massachusetts General Hospital, Boston, MA, USA. ²³⁸Pathology Department, H.U. Vall d'Hebron, Barcelona, Spain. ²³⁹Division of Bioinformatics and Biostatistics, US Food and Drug Administration, Silver Spring, MD, USA. ²⁴⁰Department of Pathology and Laboratory Medicine, Rhode Island Hospital and Lifespan Medical Center, Providence, RI, USA. ²⁴¹Université Paris-Est, Créteil, France. ²⁴²Praava Health, Dhaka, Bangladesh. ²⁴³Department of Pathology, Medical University of Vienna, Vienna, Austria.