# Radiomics Analysis of 3D Dose Distributions to Predict Toxicity of Radiotherapy for Cervical Cancer

François Lucia, Vincent Bourbonne, Dimitris Visvikis, Omar Miranda, Dorothy M Gujral, Dominique Gouders, Gurvan Dissaux, Olivier Pradier, Florent Tixier, Vincent Jaouen, et al.

<figure><img src="image_1.png" /></figure>

## ► To cite this version:

François Lucia, Vincent Bourbonne, Dimitris Visvikis, Omar Miranda, Dorothy M Gujral, et al. "Radiomics Analysis of 3D Dose Distributions to Predict Toxicity of Radiotherapy for Cervical Cancer." Journal of Personalized Medicine, 2021, 11 (5), pp.398. 10.3390/jpm11050398. [hal-05249379]

HAL Id: hal-05249379

https://hal.univ-brest.fr/hal-05249379v1

Submitted on 11 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Article

# Radiomics Analysis of 3D Dose Distributions to Predict Toxicity of Radiotherapy for Cervical Cancer

François Lucia<sup>1,2,*</sup>, Vincent Bourbonne<sup>1,2 ID</sup>, Dimitris Visvikis<sup>2</sup>, Omar Miranda<sup>1,3</sup>, Dorothy M. Gujral<sup>4,5</sup>, Dominique Gouders<sup>3</sup>, Gurvan Dissaux<sup>1,2</sup>, Olivier Pradier<sup>1,2</sup>, Florent Tixier<sup>2</sup>, Vincent Jaouen<sup>2</sup>, Julien Bert<sup>2,†</sup>, Mathieu Hatt<sup>2,†</sup> and Ulrike Schick<sup>1,2,†</sup>

<figure><img src="image_3.png" /></figure>

check for
updates

**Citation:** Lucia, F.; Bourbonne, V.; Visvikis, D.; Miranda, O.; Gujral, D.M.; Gouders, D.; Dissaux, G.; Pradier, O.; Tixier, F.; Jaouen, V.; et al. Radiomics Analysis of 3D Dose Distributions to Predict Toxicity of Radiotherapy for Cervical Cancer. *J. Pers. Med. 2021*, *11*, 398. https://doi.org/10.3390/jpm11050398

Academic Editor: Gianluca Ingrosso

Received: 26 March 2021

Accepted: 10 May 2021

Published: 11 May 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

<figure><img src="image_4.png" /></figure>

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

**Abstract:** Standard treatment for locally advanced cervical cancer (LACC) is chemoradiotherapy followed by brachytherapy. Despite radiation therapy advances, the toxicity rate remains significant. In this study, we compared the prediction of toxicity events after radiotherapy for locally advanced cervical cancer (LACC), based on either dose-volume histogram (DVH) parameters or the use of a radiomics approach applied to dose maps at the voxel level. Toxicity scores using the Common Terminology Criteria for Adverse Events (CTCAE v4), spatial dose distributions, and usual clinical predictors for the toxicity of 102 patients treated with chemoradiotherapy followed by brachytherapy for LACC were used in this study. In addition to usual DVH parameters, 91 radiomic features were extracted from rectum, bladder and vaginal 3D dose distributions, after discretization into a fixed bin width of 1 Gy. They were evaluated for predictive modelling of rectal, genitourinary (GU) and vaginal toxicities (grade ≥ 2). Logistic Normal Tissue Complication Probability (NTCP) models were derived using clinical parameters only or combinations of clinical, DVH and radiomics. For rectal acute/late toxicities, the area under the curve (AUC) using clinical parameters was 0.53/0.65, which increased to 0.66/0.63, and 0.76/0.87, with the addition of DVH or radiomics parameters, respectively. For GU acute/late toxicities, the AUC increased from 0.55/0.56 (clinical only) to 0.84/0.90 (+DVH) and 0.83/0.96 (clinical + DVH + radiomics). For vaginal acute/late toxicities, the AUC increased from 0.51/0.57 (clinical only) to 0.58/0.72 (+DVH) and 0.82/0.89 (clinical + DVH + radiomics). The predictive performance of NTCP models based on radiomics features was higher than the commonly used clinical and DVH parameters. Dosimetric radiomics analysis is a promising tool for NTCP modelling in radiotherapy.

**Keywords:** radiomics; cervical cancer; toxicity; radiotherapy; NTCP

## 1. Introduction

Cervical cancer is the fourth most frequent cancer in women, with an estimated incidence of 70,000 new cases in 2018 [1]. Standard treatment for locally advanced cervical cancer (LACC) is chemoradiotherapy followed by brachytherapy. Compared with tri-dimensional external beam radiation therapy (EBRT), intensity-modulated RT (IMRT) can deliver an adequate dose to the target structures whilst limiting the dose delivered to pelvic

<footnote n="1">¹ Radiation Oncology Department, University Hospital, CHRU Morvan, 2 avenue Foch, CEDEX, 29609 Brest, France; vincent.bourbonne@chu-brest.fr (V.B.); omar.miranda@chu-brest.fr (O.M.); gurvan.dissaux@chu-brest.fr (G.D.); olivier.pradier@chu-brest.fr (O.P.); ulrike.schick@chu-brest.fr (U.S.)</footnote>

<footnote n="2">² LaTIM, INSERM, UMR 1101, Univ Brest, 29200 Brest, France; dimitris@univ-brest.fr (D.V.); florent.tixier@univ-brest.fr (F.T.); vincent.jaouen@univ-brest.fr (V.J.); julien.bert@univ-brest.fr (J.B.); hatt@univ-brest.fr (M.H.)</footnote>

<footnote n="3">³ Radiation Oncology Department, Cornouaille Hospital, 29000 Quimper, France; dominique.gouders@ch-cornouaille.fr</footnote>

<footnote n="4">⁴ Clinical Oncology Department, Imperial College Healthcare NHS Trust, Charing Cross Hospital, Hammersmith, London W6 8RF, UK; d.gujral@nhs.net</footnote>

<footnote n="5">⁵ Department of Cancer and Surgery, Imperial College London, London SW6 4NE, UK</footnote>

<footnote>* Correspondence: francois.lucia@gmail.com; Tel.: +33-2-98-22-33-98</footnote>

<footnote>† Equally contributed.</footnote>
and abdominal organs-at-risk (OARs), including bowel, rectum and bladder. This results in a lower incidence of early and late gastrointestinal and genitourinary (GU) toxicity [2,3].

Recently, 3D image-guided brachytherapy (IGBT) techniques have been implemented into clinical practice and allow for more favorable dosimetry and a reduced dose to nearby OARs, resulting in lower toxicity and improved local control [4,5]. Despite these advances, the toxicity rate remains significant [6] and these side effects can have an impact on long-term quality of life for these young patients [7,8].

Over the last decade, radiomics has emerged as a translational field of research, aiming at better characterizing medical images' content via high-throughput extraction of engineered features [9–13]. Radiomics analysis has been applied to different imaging modalities, such as magnetic resonance imaging (MRI), positron emission tomography (PET), computerized tomography (CT) or ultrasound [14,15] to improve diagnostic and prognostic multiparametric models [14,16,17]. In radiotherapy (RT) applications specifically, radiomics has mainly been considered within the context of building models for the prediction of response to treatment and disease-free survival [13,18], whereas the prediction of toxicity endpoints has been determined by relying on, for instance, dose-volume histogram (DVH) metrics [19] and treatment-related clinical variables. Until now, the vast majority of studies have considered the application of the radiomics concept to the medical images, not dose distribution maps. Dose maps can also be considered as images with spatial and statistical distributions of dose levels and, therefore, can be analyzed through the radiomics approach with the aim of predicting toxicity endpoints [20].

The construction of models predictive of radiation-induced toxicity is of great interest to improve patients' reported outcome quality of life and to facilitate possible radiotherapy dose escalation. In this study, we compared predictive modelling of toxicity events after RT for LACC, using the DVH parameters and radiomic features from dose maps.

## 2. Materials and Methods

### 2.1. *Patients*

Patients with histologically proven LACC stage IB1-IVA (FIGO 2009 definition) treated with definitive curative chemoradiotherapy (CRT) and subsequent brachytherapy (BT) between August 2010 and December 2016 at two French institutions (Brest and Quimper) were retrospectively included (Table S1, Supplementary Materials). Patients with stage IB1 and IIA1 were included if they had positive lymph nodes. All patients were required to have at least 3 years of follow-up. Exclusion criteria were history of previous chemotherapy or RT and/or metastatic disease.

Collected data included clinical characteristics, presence of positive lymph nodes on PET/CT, tumor size as measured on MRI, body mass index (BMI) before treatment, external beam radiotherapy (EBRT) and brachytherapy dose, presence of symptoms before radiotherapy (baseline), previous abdominal surgery, rectal comorbidities, date and status at last follow-up.

A total of 102 patients were included for retrospective analysis. The cohort was split into a training set (52 patients treated in Brest, 51%) and a testing set (50 patients treated in Quimper, 49%). All patients provided signed permission for the use of their clinical data for scientific purposes and informed consent for the anonymous publication of data. The Institutional Review Board approved this study (29BCR18.0015).

### 2.2. *Treatment*

Consortium guidelines were applied to contour the clinical target volume (CTV), the planning target volume (PTV) and organs at risk [21] (for bladder and rectum, we contoured the full volume). For brachytherapy, the vagina was retrospectively defined according to the method published by Susko et al. [22], as a 3-dimensional (3D) organ using a 0.2 cm fixed brush, expanded to include all vagina identifiable on MRI or CT images. The 0.2 cm brush was chosen because it was the smallest available from the treatment planning system.
All patients received pelvic EBRT or extended field RT to the para-aortic area using high-energy photons (18 MV), at a dose of 45–50.4 Gy with standard fractionation depending on their work-up. In patients with positive pelvic or para-aortic lymph nodes, an image-guided targeted boost was delivered to a dose of 50.4–54 Gy to the involved nodes (7 and 6 patients in training and testing sets, respectively). RT consisted of three-dimensional conformal radiotherapy (3D-CRT) (n = 33 and n = 32 in the training and testing sets, respectively) or IMRT (n = 19 and n = 18 in the training and testing sets, respectively) delivered using a linear accelerator.

Patients received 3–4 fractions of MRI-guided high-dose-rate (HDR) intracavitary brachytherapy every 4 days one week after EBRT completion. The prescribed dose was 7 Gy to the high-risk CTV. None of the patients had interstitial brachytherapy. No patient experienced delays or breaks in RT due to acute toxicity (median RT duration, 49 days; range, 47–53 days). All patients received 4–6 cycles of concomitant chemotherapy with weekly cisplatin (40 mg/m²) or carboplatin (area under the curve (AUC) 2) for those with renal impairment.

## 2.3. Parameters Used for Normal Tissue Complication Probability (NTCP) Modeling

To explore the possible value of dosimetric radiomics in predictive modeling, we focused on the incidence of GU, rectal and vaginal toxicities, using the Common Terminology Criteria for Adverse Events (CTCAE) v4. In the training set, the cumulative rates of acute (within 3 months of completing treatment) and late (later than 3 months after treatment) grade ≥2 GU, rectal and vaginal toxicities were 39%, 14%, 26%, 10%, 24%, and 24%, respectively. In the testing set, they were 42%, 16%, 28%, 14%, 24% and 24%, respectively.

### - Clinical Parameters

Clinical parameters included in the analysis were the following: presence of symptoms before radiotherapy (baseline), age, FIGO stage, TNM staging, previous abdominal surgery, rectal comorbidities.

### - DVH Parameters

Percentage and absolute volumes of rectum, bladder and vaginal receiving x Gy or higher (Vx), with x ranging from 5 to 55 Gy with EBRT and to 140 Gy with cumulate EBRT and BT doses, in 5 Gy steps, were included in the analysis. Dmean, Dmin, Dmax, D1cc, D2cc, D1% and D2% of rectum, bladder and vagina were also recorded for EBRT and BT doses. To account for the biological effects of different fractionation schemes, both the physical doses received with BT and EBRT were converted to EQD2 doses using a linear quadratic model [23] with an α/β ratio of 10 and 3 for dose summation for acute and late effects, respectively [24]. DVH from BT and EBRT were summed according to their relative volume and absolute dose, assuming that hotspots in the BT plan were located at the same location as hotspots in the EBRT plan. We recorded Vx_EBRT and Dx (minimum dose given to the hottest x% volume) EBRT for EBRT doses and Vx_EBRT + BT and Dx_EBRT + BT for cumulative EBRT and BT doses.

### - Radiomic Features

For the purpose of this study, only the dosimetric map of EBRT was used. Doses were discretized according to the fixed bin width (FBW) method with a width of 1 Gy so that a minimum number of 32 bins would be obtained (max dose between 45 Gy and 55 Gy). For subsequent analyses, doses are denoted as “grey levels”.

In total, 91 image biomarker standardization initiative (IBSI)-compliant radiomic features (including first-order intensity metrics and higher order textural features) were extracted in 3D for each rectum, bladder and vagina dose distribution (i.e., volumes of interest corresponding to the manually delineated organs gross tumor volumes (GTV), reported on the corresponding dose maps). More information about feature implementation and benchmarking values can be found in the published literature [11,25,26]. All radiomics features are presented in Table S4 in the Supplementary Materials. Analyses were per-
formed using in-house developed software (MIRAS v1.0, LaTIM, INSERM UMR 1101, Brest, France).

## 2.4. NTCP Modeling

For each toxicity endpoint, all generated NTCP models (including clinical features, DVH and radiomics alone or in association) were created with uni- and multivariate logistic regression. For each toxicity/parameter-set combination, the following workflow was performed to build the final multivariate logistic regression model: First, the features were selected by univariate analysis (*p* < 0.05) without correction for multiple testing in order to have at least a few parameters retained for building the models. The remaining significant features were further filtered based on Spearman's rank test to exclude correlated parameters (>0.80), i.e., containing redundant information. If two or more features were correlated, the feature with the smaller *p*-value from the univariate analyses was retained. Multivariate logistic regression, with backward elimination, was then applied to determine the feature signature. All these steps were carried out using the training set only, for which the performance was evaluated by the AUC of the receiver operating characteristic (ROC) [27,28]. Based on the ROC obtained, the optimal cut-off values were selected in order to optimize accuracy (Youden Index).

The final signatures were evaluated on the testing set reporting balanced accuracy (BAcc) (calculated as the average of sensitivity and specificity) to identify patients with toxicity events. Balanced accuracy was used because of imbalance in the data [29].

All statistical analyses were performed using MedCalc Statistical Software version 15.8 (MedCalc Software bvba, Ostend, Belgium; https://www.medcalc.org; accessed on 10 December 2015). *p*-values below 0.05 were considered significant.

## 3. Results

For all toxicities, models tended to include more radiomic features (average of 1.33 ± 0.52 features) than clinical (average of 0.54 ± 0.29 feature) and DVH variables (average of 1 feature ± 0.55 features) (Table S2, Supplementary Materials).

The final models and the features included are reported in Tables 1–6.

Model development in the training test: as presented in Tables 1–6, the clinical features alone allowed AUCs<sub>clinical</sub> of 0.53, 0.55, 0.51, 0.65, 0.56, and 0.57 for rectal, GU and vaginal acute and late toxicities, respectively. Considering only radiomic dosimetry features, the AUCs were higher for all toxicities compared to clinical parameters (AUC<sub>RA</sub> of 0.72, 0.80, 0.81, 0.86, 0.79, 0.78 for the six endpoints, respectively), and always higher (except for GU late toxicities) than for DVH alone (AUC<sub>DVH</sub> of 0.65, 0.78, 0.58, 0.57, 0.89, 0.72). For all toxicities (except GU acute and late toxicities), the predictive value of the models incorporating clinical and radiomic parameters was higher than that of the corresponding models based on clinical and DVH parameters. The differences in AUC were only statistically significant at the 0.05 level (*p* < 0.04) for acute and late vaginal toxicities.

The trend towards superiority of the models relying on radiomic dosimetry parameters was confirmed by the BAcc of the final models evaluated on the testing cohort, as reported in Table S3, Supplementary Materials. The addition of radiomic dose parameters to clinical + DVH models systematically improved their predictive value (i.e., BAcc<sub>clinical_DVH_RA</sub> > BAcc<sub>clinical_DVH</sub>) except for GU acute toxicities. For late and acute vaginal toxicities, the difference was significant (*p* < 0.04). The best models combining all factors allowed us to reach a BAcc of 78%, 72%, 74%, 64%, 73%, and 78% for rectal, GU and vaginal acute and late toxicities, respectively.
<table><caption>**Table 1.** Results of the multivariate logistic analysis for acute rectal toxicity.</caption><thead><tr><th colspan="6">Rectal Acute</th></tr><tr><th>Features</th><th>P</th><th>OR</th><th>10th-90th Perc. for OR</th><th>AUC</th><th>95% CI</th></tr></thead><tbody><tr><td>Clinical</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• FIGO</td><td>0.66</td><td>1.14</td><td>0.32-5.72</td><td>0.53</td><td>0.39-0.67</td></tr><tr><td>DVH</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• V40_%_EBRT</td><td>0.06</td><td>1.03</td><td>1.00-1.07</td><td>0.65</td><td>0.50-0.78</td></tr><tr><td>RA</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• Inv_diff_norm_cooc</td><td>0.030</td><td>9.61</td><td>3.59-133.99</td><td>0.72</td><td>0.58-0.83</td></tr><tr><td>Clinical + DVH</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• V40_%_EBRT</td><td>0.04</td><td>1.04</td><td>1.00-1.08</td><td>0.66</td><td>0.51-0.79</td></tr><tr><td>Clinical + RA</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• Inv_diff_norm_cooc</td><td>0.02</td><td>13.3</td><td>2.00-89.73</td><td></td><td></td></tr><tr><td>• FIGO</td><td>0.37</td><td>1.88</td><td>0.48-7.43</td><td>0.74</td><td>0.60-0.85</td></tr><tr><td>Clinical + DVH + RA</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• Inv_diff_norm_cooc</td><td>0.04</td><td>63.51</td><td>5.91-104.96</td><td></td><td></td></tr><tr><td>• V45_%_EBRT</td><td>0.30</td><td>1.01</td><td>0.99-1.04</td><td>0.76</td><td>0.61-0.86</td></tr></tbody></table>

OR = odds ratio, AUC = area under the curve, FIGO = International Federation of Gynecology and Obstetrics, Vₓ = percentage of rectum receiving x Gy or higher, DVH = dose-volume histogram, EBRT = external beam radiation therapy, RA = radiomic analysis.

<table><caption>**Table 2.** Results of the multivariate logistic analysis for acute genitourinary toxicity.</caption><thead><tr><th colspan="6">GU Acute</th></tr><tr><th>Features</th><th>P</th><th>OR</th><th>10th-90th Perc. for OR</th><th>AUC</th><th>95% CI</th></tr></thead><tbody><tr><td>Clinical</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• FIGO</td><td>0.55</td><td>1.42</td><td>0.39-6.31</td><td>0.55</td><td>0.40-0.69</td></tr><tr><td>DVH</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• V40_%_EBRT</td><td>0.007</td><td>1.05</td><td>1.01-1.08</td><td></td><td></td></tr><tr><td>• D1 Gy_%_EBRT + BT</td><td>0.46</td><td>1.11</td><td>0.84-1.46</td><td>0.78</td><td>0.64-0.88</td></tr><tr><td>RA</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• Shade</td><td>0.036</td><td>1.00</td><td>1.00-1.01</td><td></td><td></td></tr><tr><td>• ZSNU</td><td>0.29</td><td>0.99</td><td>0.99-1.01</td><td>0.80</td><td>0.66-0.89</td></tr><tr><td>Clinical + DVH</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• FIGO</td><td>0.41</td><td>2.04</td><td>0.38-11.03</td><td></td><td></td></tr><tr><td>• V40_%_EBRT</td><td>0.0041</td><td>1.07</td><td>1.02-1.12</td><td>0.84</td><td>0.71-0.93</td></tr><tr><td>• D1 Gy_%_EBRT + BT</td><td>0.40</td><td>1.15</td><td>0.84-1.58</td><td></td><td></td></tr><tr><td>Clinical + RA</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• ZSNU</td><td>0.13</td><td>0.98</td><td>0.97-1.00</td><td></td><td></td></tr><tr><td>• FIGO</td><td>0.41</td><td>1.92</td><td>0.40-9.26</td><td>0.80</td><td>0.66-0.90</td></tr><tr><td>Clinical + DVH + RA</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• FIGO</td><td>0.42</td><td>1.99</td><td>0.38-10.52</td><td></td><td></td></tr><tr><td>• V40_%_EBRT</td><td>0.27</td><td>1.04</td><td>0.97-1.12</td><td></td><td></td></tr><tr><td>• D1 Gy_%_EBRT + BT</td><td>0.36</td><td>1.16</td><td>0.85-1.59</td><td>0.83</td><td>0.69-0.92</td></tr><tr><td>• ZSNU</td><td>0.46</td><td>0.99</td><td>0.98-1.01</td><td></td><td></td></tr></tbody></table>

OR = odds ratio, AUC = area under the curve, FIGO = International Federation of Gynecology and Obstetrics, Vₓ = percentage of rectum receiving x Gy or higher, Dx minimum dose given to the hottest x% volume, DVH = dose-volume histogram, EBRT = external beam radiation therapy, BT = brachytherapy, RA = radiomic analysis.
<table><caption>**Table 3.** Results of the multivariate logistic analysis for acute vaginal toxicity.</caption><thead><tr><th colspan="6">Vaginal Acute</th></tr><tr><th>Features</th><th>P</th><th>OR</th><th>10th-90th Perc. for OR</th><th>AUC</th><th>95% CI</th></tr></thead><tbody><tr><td>Clinical</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• FIGO</td><td>0.51</td><td>1.35</td><td>0.41-6.14</td><td>0.51</td><td>0.37-0.66</td></tr><tr><td>DVH</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• D1 Gy %_ EBRT + BT</td><td>0.46</td><td>0.80</td><td>1.01-1.08</td><td>0.58</td><td>0.43-0.71</td></tr><tr><td>RA</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• HGLZE</td><td>0.048</td><td>1.00</td><td>1.00-1.01</td><td></td><td></td></tr><tr><td>• LGSRE</td><td>0.057</td><td>1.34</td><td>0.58-1.85</td><td>0.81</td><td>0.68-0.91</td></tr><tr><td>Clinical + DVH</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• D1 Gy %_ EBRT + BT</td><td>0.44</td><td>0.79</td><td>0.44-1.43</td><td>0.58</td><td>0.43-0.71</td></tr><tr><td>Clinical + RA</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• HGLZE</td><td>0.048</td><td>1.00</td><td>1.00-1.01</td><td></td><td></td></tr><tr><td>• LGSRE</td><td>0.053</td><td>1.34</td><td>0.58-1.85</td><td>0.82</td><td>0.69-0.91</td></tr><tr><td>Clinical + DVH + RA</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• HGLZE</td><td>0.048</td><td>1.00</td><td>1.00-1.01</td><td></td><td></td></tr><tr><td>• LGSRE</td><td>0.053</td><td>1.34</td><td>0.58-1.85</td><td>0.82</td><td>0.69-0.91</td></tr></tbody></table>

OR = odds ratio, AUC = area under the curve, FIGO = International Federation of Gynecology and Obstetrics, Vₓ = percentage of rectum receiving x Gy or higher, Dx minimum dose given to the hottest x% volume, DVH = dose-volume histogram, EBRT = external beam radiation therapy, BT = brachytherapy, RA = radiomic analysis.

<table><caption>**Table 4.** Results of the multivariate logistic analysis for late rectal toxicity.</caption><thead><tr><th colspan="6">Rectal Late</th></tr><tr><th>Features</th><th>P</th><th>OR</th><th>10th-90th Perc. for OR</th><th>AUC</th><th>95% CI</th></tr></thead><tbody><tr><td>Clinical</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• FIGO</td><td>0.13</td><td>3.56</td><td>0.69-18.28</td><td>0.65</td><td>0.50-0.78</td></tr><tr><td>DVH</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• D1 Gy %_ EBRT + BT</td><td>0.34</td><td>1.19</td><td>0.84-1.68</td><td>0.57</td><td>0.43-0.71</td></tr><tr><td>RA</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• Coarseness vdif</td><td>0.044</td><td>17.01</td><td>1.69-41.93</td><td></td><td></td></tr><tr><td>• AUC-hist</td><td>0.48</td><td>11.90</td><td>0.013-10.86</td><td>0.86</td><td>0.74-0.94</td></tr><tr><td>Clinical + DVH</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• FIGO</td><td>0.14</td><td>3.41</td><td>0.67-18.21</td><td></td><td></td></tr><tr><td>• D1 Gy %_ EBRT + BT</td><td>0.39</td><td>1.17</td><td>0.82-1.68</td><td>0.63</td><td>0.48-0.76</td></tr><tr><td>Clinical + RA</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• FIGO</td><td>0.069</td><td>5.99</td><td>0.87-41.13</td><td></td><td></td></tr><tr><td>• Coarseness vdif</td><td>0.020</td><td>11.71</td><td>1.54-89.35</td><td>0.87</td><td>0.74-0.95</td></tr><tr><td>Clinical + DVH + RA</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• FIGO</td><td>0.069</td><td>5.99</td><td>0.87-41.13</td><td></td><td></td></tr><tr><td>• Coarseness vdif</td><td>0.020</td><td>11.71</td><td>1.54-89.35</td><td>0.87</td><td>0.74-0.95</td></tr></tbody></table>

OR = odds ratio, AUC = area under the curve, FIGO = International Federation of Gynecology and Obstetrics, Vₓ = percentage of rectum receiving x Gy or higher, Dx minimum dose given to the hottest x% volume, DVH = dose-volume histogram, EBRT = external beam radiation therapy, BT = brachytherapy, RA = radiomic analysis.
<table><caption>**Table 5.** Results of the multivariate logistic analysis for late genitourinary toxicity.</caption><thead><tr><th rowspan="2">Features</th><th colspan="5">GU Late</th></tr><tr><th>P</th><th>OR</th><th>10th-90th Perc. for OR</th><th>AUC</th><th>95% CI</th></tr></thead><tbody><tr><td>Clinical</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• FIGO</td><td>0.54</td><td>1.29</td><td>0.37-4.91</td><td>0.56</td><td>0.42-0.70</td></tr><tr><td>DVH</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• D1 Gy %_EBRT + BT</td><td>0.0046</td><td>1.72</td><td>1.18-2.51</td><td>0.89</td><td>0.77-0.96</td></tr><tr><td>RA</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• Energy_histogram</td><td>0.28</td><td>1.15</td><td>0.36-10.02</td><td></td><td></td></tr><tr><td>• Inv_diff_norm_cooc</td><td>0.13</td><td>5.09</td><td>0.13-19.75</td><td>0.79</td><td>0.66-0.89</td></tr><tr><td>Clinical + DVH</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• FIGO</td><td>0.38</td><td>0.28</td><td>0.017-4.71</td><td></td><td></td></tr><tr><td>• D1 Gy %_EBRT + BT</td><td>0.0050</td><td>1.78</td><td>1.19-2.66</td><td>0.90</td><td>0.78-0.97</td></tr><tr><td>Clinical + RA</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• Energy_histogram</td><td>0.28</td><td>1.15</td><td>0.36-10.02</td><td></td><td></td></tr><tr><td>• Inv_diff_norm_cooc</td><td>0.13</td><td>5.09</td><td>0.13-19.75</td><td>0.79</td><td>0.66-0.89</td></tr><tr><td>Clinical + DVH + RA</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• D1 Gy %_EBRT + BT</td><td>0.029</td><td>2.15</td><td>1.08-4.28</td><td></td><td></td></tr><tr><td>• Energy_histogram</td><td>0.28</td><td>1.14</td><td>0.39-3.76</td><td>0.96</td><td>0.87-1.00</td></tr><tr><td>• Inv_diff_norm_cooc</td><td>0.17</td><td>2.89</td><td>0.88-19.62</td><td></td><td></td></tr></tbody></table>

OR = odds ratio, AUC = area under the curve, FIGO = International Federation of Gynecology and Obstetrics, Vₓ = percentage of rectum receiving x Gy or higher, Dₓ minimum dose given to the hottest x% volume, DVH = dose-volume histogram, EBRT = external beam radiation therapy, BT = brachytherapy, RA = radiomic analysis.

<table><caption>**Table 6.** Results of the multivariate logistic analysis for late vaginal toxicity.</caption><thead><tr><th rowspan="2">Features</th><th colspan="5">Vaginal Late</th></tr><tr><th>P</th><th>OR</th><th>10th-90th Perc. for OR</th><th>AUC</th><th>95% CI</th></tr></thead><tbody><tr><td>Clinical</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• FIGO</td><td>0.38</td><td>1.82</td><td>0.47-6.96</td><td>0.57</td><td>0.42-0.71</td></tr><tr><td>DVH</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• V70 Gy%_EBRT + BT</td><td>0.015</td><td>1.30</td><td>1.05-1.60</td><td>0.72</td><td>0.58-0.84</td></tr><tr><td>RA</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• GLNU area</td><td>0.0097</td><td>0.93</td><td>0.87-0.98</td><td>0.78</td><td>0.64-0.88</td></tr><tr><td>Clinical + DVH</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• V70 Gy%_EBRT + BT</td><td>0.015</td><td>1.30</td><td>1.05-1.60</td><td>0.72</td><td>0.58-0.84</td></tr><tr><td>Clinical + RA</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• FIGO</td><td>0.39</td><td>1.94</td><td>0.42-8.89</td><td></td><td></td></tr><tr><td>• GLNU area</td><td>0.01</td><td>0.92</td><td>0.87-0.98</td><td>0.79</td><td>0.65-0.89</td></tr><tr><td>Clinical + DVH + RA</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>• V70 Gy%_EBRT + BT</td><td>0.013</td><td>1.38</td><td>1.07-1.79</td><td></td><td></td></tr><tr><td>• GLNU area</td><td>0.0092</td><td>0.91</td><td>0.84-0.98</td><td>0.89</td><td>0.77-0.96</td></tr></tbody></table>

OR = odds ratio, AUC = area under the Curve, FIGO = International Federation of Gynecology and Obstetrics, Vₓ = percentage of rectum receiving x Gy or higher, DVH = dose-volume histogram, EBRT = External Beam Radiation Therapy, BT = Brachytherapy, RA = Radiomic Analysis.
## 4. Discussion

To the best of our knowledge, our work is one of the first studies investigating the use of radiomic features of dose distributions to predict toxicity endpoints in LACC. A previous study investigated rectal toxicities but did not investigate GU and vaginal toxicities [30].

Radiomics is the subject of an increasing number of medical studies [1, 10, 12, 19, 29]. In our study, we investigated a recently proposed use of radiomics in radiotherapy [20]. Indeed, for LACC specifically, we compared the predictive performance of NTCP models based on clinical parameters to commonly used DVH parameters of EBRT and brachytherapy, such as Vx and Dmean, and to radiomics features derived from 3D dose distributions, alone or in combination. Adding DVH parameters to clinical parameters systematically improved prediction performance. Furthermore, adding radiomic dose features resulted in incremental improvements for all tests (Tables 1–6).

Our results in terms of DVH parameters are in accordance with previous reports, showing the V70% as predictive for late vaginal toxicity [32], and doses correlated to Dmax (D1cc, D2cc) as predictive for late rectal and GU toxicities [33,34].

DVH parameters commonly used in NTCPs, such as the OAR volume receiving more than a certain dose (V<sub>D</sub>) or Dmean, do not take into account information about the spatial relationship between voxels. Thus, the same V<sub>D</sub> value can correspond to voxels distributed in the OAR or to connected voxels. But the biological impact of the same V<sub>D</sub> may be different. In contrast, radiomic features explicitly take into account spatial relationships in the dose maps and are therefore expected to provide higher differentiating power.

To obtain more reliable and convincing results, we decided to evaluate the value of radiomic dose features in a framework of training and testing in different sets from two centers. Reproducibility is a recognized issue in the field of radiomics, especially with multicenter data [35]. For all data, we used the same dose calculation algorithm, implemented in 3D Slicer [36]. The objective of this study was to evaluate the added value of radiomic dose features in the building of NTCP models. We have detailed our methodological choices to achieve consistency in our approach. However, the robustness of the selected radiomic dose features with respect to variations in the dose calculation algorithm, grid size, and other factors was not evaluated in the present study and should be the subject of future research. Patient-specific variations between the expected dose and the administered dose could have a greater impact than the variations in calculated doses across dose calculation algorithms, the size of the grid applied, etc. To overcome the limitations of NTCP models based on DVH parameters, other dosimetric analyses were also published, such as the pixel-to-pixel comparison of the dose-to-surface map [37].

Two-dimensional dose surface analyses can also be performed with radiomics. Indeed, the frequency histograms of grey levels used in radiomics contain essentially the same information as dose-volume histograms. In this study, dosimetric parameters frequently used in NTCP models, such as Dmean and Vx, were included as DVH parameters, while Variance, Skewness and Kurtosis were considered as radiomic features.

We could have considered parameters such as Dmean and Vx to belong to the set of radiomic features. This approach could improve the predictive performance of NTCP models based on DVH parameters by including radiomics features.

Another method using radiomics to improve NTCP is based on characterizing healthy tissues from CT images through radiomic features which are then correlated with radiation-induced toxicity [38]. The majority of studies published so far focused on head and neck and lung cancers to predict the risk of xerostomia and radiation-induced pneumonitis [39–43]. The best models from these studies were based on the variation in radiomic parameters between pre- and post-treatment CTs. Thus, these models only allow one to predict toxicity at the end of treatment, whereas those derived from dose maps would allow for toxicity prediction at the time of treatment planning, enabling planning adjustments to decrease side effects.

We have followed a previously described methodology [17] consisting of the removal of features based on their correlation, in order to select a subset of radiomic features to be
included in the statistical analyses. Models including more radiomics features could further
improve the predictive performance of the models, which will require the use of more
robust machine-learning techniques to select and combine parameters (e.g., embedded
feature selection with Random Forest or neural networks). This will be the focus of future
studies. Moreover, the relevance of the identified features will also be investigated for other
tumor sites.

Our study has other limitations. First, the size of the two cohorts is small, but is
explained by the low incidence of LACC in our country. The rate of chronic toxicity
may also seem low, but is in accordance with what has been previously reported in the
literature [44,45]. This resulted in limited statistical power that may contribute to explaining
the lack of significance of some results, particularly with regards to chronic rectal toxicity.
We did not apply correction for multiple testing when evaluating features in the training
set, as none would have been significant if we did, and we had to reduce the number of
variables by removing some of them based on their level of Spearman rank correlation.
However, we mitigated the risk of false discovery by evaluating the trained models in
a different, external cohort, which was not used in the training phase. The fact that the
models performed well in the testing set increases the confidence in our findings and
reduces the risk of overfitting when training them. These models, nevertheless, will require
validation in a larger cohort.

Moreover, we only exploited the dosimetry map of external beam radiotherapy. The co-
registration of external beam RT and brachytherapy treatment plans was difficult to achieve
due to the presence of the brachytherapy device on the planning CT undertaken prior to
brachytherapy [46]. In addition, patients treated with external-beam RT in Quimper under-
went their brachytherapy treatment sessions in Brest, precluding the ability to carry out
an external evaluation of the models. A potential solution could be to extract information
from the brachytherapy dose map separately. However, at each brachytherapy session,
a new dosimetry and a new dose map are performed, which poses the same problem of reg-
istration. Thus, although we did not perform radiomic analysis of the brachytherapy dose
maps, the dose delivered by the brachytherapy was still taken into account by including
the DVH parameters associated with the brachytherapy.

Organ motion can also impact the predictive value of radiomic parameters as well as DVH parameters. This is limited by the daily imaging control during treatment, allowing one to check the rectal repletion and the bladder filling.

Finally, it would be interesting to evaluate the combination of radiomic features ex-
tracted from images such as dosimetry CT with those derived from dose maps [38] in order
to achieve the same endpoint prediction (both toxicity and tumor control or recurrence-free
survival). However, this would entail handling an even larger set of features, which would
probably require more complex modeling and larger patient cohorts. This will be the focus
of future studies. Our models might achieve better performance by including the radiomic
parameters extracted from the brachytherapy dose maps. However, they already perform
better than the current models, and safe registration methods need to be validated before
we can consider using the dose maps of the different brachytherapy sessions.

**5. Conclusions**

The 3D dose distributions provided by DVHs provide limited information, as the
information about the spatial relationships between voxel doses is unavailable. The use
of the radiomics approach applied to 3D dose distributions explicitly allows for taking
this into account. This probably explains the higher predictive performance of NTCP
models that include such radiomic dose features instead of common DVH parameters only.
These findings highlight the need to go beyond EBRT planning-based DVH constraints,
and dosimetric radiomics analysis could be a promising tool to personalize radiotherapy
planning. However, the usefulness of the features described here will be investigated in
prospective studies including a larger number of centers.
**Supplementary Materials:** The following are available online at https://www.mdpi.com/article/10.3390/jpm11050398/s1, Table S1: Patients' characteristics, Table S2: Selection rate features, Table S3: Results in testing cohort, Table S4: List of radiomics features. For features detailed definitions and implementation, see Alex Zwanenburg, Martin Vallières, Steffen Löck: Image biomarker standardisation initiative—feature definitions. 2019. https://arxiv.org/abs/1612.07003; accessed on 26 March 2021.

**Author Contributions:** Conceptualization, F.L., V.B. and U.S.; methodology, F.L., M.H. and U.S.; software, F.T., V.J. and J.B.; formal analysis, F.L., V.B. and U.S.; investigation, F.L.; data curation, F.L.; writing—original draft preparation, F.L.; writing—review and editing, F.L., V.B., D.V., O.M., D.M.G., D.G., G.D. and U.S.; supervision, O.P., M.H. and U.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Institutional Review Board of University Hospital of Brest (29BCR18.0015).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

## References

1. Bray, F.; Ferlay, J.; Soerjomataram, I.; Siegel, R.L.; Torre, L.A.; Jemal, A. Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. *CA Cancer J. Clin.* **2018**, *68*, 394–424. [CrossRef] [PubMed]

2. Elson, J.K.; Kachnic, L.A.; Kharofa, J.R. Intensity-modulated radiotherapy improves survival and reduces treatment time in squamous cell carcinoma of the anus: A National Cancer Data Base study. *Cancer* **2018**, *124*, 4383–4392. [CrossRef]

3. Lin, Y.; Chen, K.; Lu, Z.; Zhao, L.; Tao, Y.; Ouyang, Y.; Cao, X. Intensity-modulated radiation therapy for definitive treatment of cervical cancer: A meta-analysis. *Radiat. Oncol.* **2018**, *13*, 177. [CrossRef] [PubMed]

4. Ribeiro, I.; Janssen, H.; De Brabandere, M.; Nulens, A.; De Bal, D.; Vergote, I.; Van Limbergen, E. Long term experience with 3D image guided brachytherapy and clinical outcome in cervical cancer patients. *Radiother. Oncol.* **2016**, *120*, 447–454. [CrossRef] [PubMed]

5. Sun, L.-M.; Huang, E.-Y.; Ko, S.-F.; Wang, C.-J.; Leung, S.W.; Lin, H.; Wu, J.-M.; Wu, I.-H.; Lee, S.P. Computer tomography-assisted three-dimensional technique to assess rectal and bladder wall dose in intracavitary brachytherapy for uterine cervical cancer. *Radiother. Oncol.* **2004**, *71*, 333–337. [CrossRef]

6. Tan, L.-T.; Pötter, R.; Sturdza, A.; Fokdal, L.; Haie-Meder, C.; Schmid, M.; Gregory, D.; Petric, P.; Jürgenliemk-Schulz, I.; Gillham, C.; et al. Change in Patterns of Failure After Image-Guided Brachytherapy for Cervical Cancer: Analysis From the RetroEMBRACE Study. *Int. J. Radiat. Oncol.* **2019**, *104*, 895–902. [CrossRef]

7. Kirchheiner, K.; Pötter, R.; Tanderup, K.; Lindegaard, J.C.; Haie-Meder, C.; Petrič, P.; Mahantshetty, U.; Jürgenliemk-Schulz, I.M.; Rai, B.; Cooper, R.; et al. Health-Related Quality of Life in Locally Advanced Cervical Cancer Patients After Definitive Chemoradiation Therapy Including Image Guided Adaptive Brachytherapy: An Analysis From the EMBRACE Study. *Int. J. Radiat. Oncol.* **2016**, *94*, 1088–1098. [CrossRef]

8. Lind, H.; Waldenström, A.-C.; Dunberger, G.; Al-Abany, M.; Alevronta, E.; Johansson, K.-A.; Olsson, C.; Nyberg, T.; Wilderäng, U.; Steineck, G.; et al. Late symptoms in long-term gynaecological cancer survivors after radiation therapy: A population-based cohort study. *Br. J. Cancer* **2011**, *105*, 737–745. [CrossRef]

9. Yip, S.S.F.; Aerts, H.J.W.L. Applications and limitations of radiomics. *Phys. Med. Biol.* **2016**, *61*, R150–R166. [CrossRef]

10. Hatt, M.; Tixier, F.; Pierce, L.; Kinahan, P.E.; Le Rest, C.C.; Visvikis, D. Characterization of PET/CT images using texture analysis: The past, the present... any future? *Eur. J. Nucl. Med. Mol. Imaging* **2017**, *44*, 151–165. [CrossRef]

11. Vallières, M.; Zwanenburg, A.; Badic, B.; Le Rest, C.C.; Visvikis, D.; Hatt, M. Responsible Radiomics Research for Faster Clinical Translation. *J. Nucl. Med.* **2017**, *59*, 189–193. [CrossRef]

12. LaRue, R.T.H.M.; Defraene, G.; De Ruysscher, D.; Lambin, P.; Van Elmpt, W. Quantitative radiomics studies for tissue characterization: A review of technology and methodological procedures. *Br. J. Radiol.* **2017**, *90*, 20160665. [CrossRef] [PubMed]

13. Aerts, H.J.W.L.; Velazquez, E.R.; Leijenaar, R.T.H.; Parmar, C.; Grossmann, P.; Carvalho, S.; Bussink, J.; Monshouwer, R.; Haibe-Kains, B.; Rietveld, D.; et al. Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. *Nat. Commun.* **2014**, *5*, 4006. [CrossRef] [PubMed]

14. Lucia, F.; Visvikis, D.; Vallières, M.; Desseroit, M.-C.; Miranda, O.; Robin, P.; Bonaffini, P.A.; Alfieri, J.; Masson, I.; Mervoyer, A.; et al. External validation of a combined PET and MRI radiomics model for prediction of recurrence in cervical cancer patients treated with chemoradiotherapy. *Eur. J. Nucl. Med. Mol. Imaging* **2019**, *46*, 864–877. [CrossRef]
15. Reuzé, S.; Orlhac, F.; Chargari, C.; Nioche, C.; Limkin, E.; Riet, F.; Escande, A.; Haie-Meder, C.; Dercle, L.; Gouy, S.; et al. Prediction of cervical cancer recurrence using textural features extracted from 18F-FDG PET images acquired with different scanners. *Oncotarget* **2017**, *8*, 43169–43179. [CrossRef]

16. Kalpathy-Cramer, J.; Mamomov, A.; Zhao, B.; Lu, L.; Cherezov, D.; Napel, S.; Echegaray, S.; Rubin, D.; McNitt-Gray, M.; Lo, P.; et al. Radiomics of Lung Nodules: A Multi-Institutional Study of Robustness and Agreement of Quantitative Imaging Features. *Tomography* **2016**, *2*, 430–437. [CrossRef] [PubMed]

17. Vallieres, M.; Freeman, C.R.; Skamene, S.R.; El Naqa, I. A radiomics model from joint FDG-PET and MRI texture features for the prediction of lung metastases in soft-tissue sarcomas of the extremities. *Phys. Med. Biol.* **2015**, *60*, 5471–5496. [CrossRef]

18. Huynh, E.; Coroller, T.P.; Narayan, V.; Agrawal, V.; Hou, Y.; Romano, J.; Franco, I.; Mak, R.H.; Aerts, H.J. CT-based radiomic analysis of stereotactic body radiation therapy patients with lung cancer. *Radiother. Oncol.* **2016**, *120*, 258–266. [CrossRef] [PubMed]

19. Bentzen, S.M.; Constine, L.S.; Deasy, J.O.; Eisbruch, A.; Jackson, A.; Marks, L.B.; Haken, R.K.T.; Yorke, E.D. Quantitative Analyses of Normal Tissue Effects in the Clinic (QUANTEC): An introduction to the scientific issues. *Int. J. Radiat. Oncol.* **2010**, *76*, S3–S9. [CrossRef] [PubMed]

20. Rossi, L.; Bijman, R.; Schillemans, W.; Aluwini, S.; Cavedon, C.; Witte, M.; Incrocci, L.; Heijmen, B. Texture analysis of 3D dose distributions for predictive modelling of toxicity rates in radiotherapy. *Radiother. Oncol.* **2018**, *129*, 548–553. [CrossRef]

21. Lim, K.; Small, W.; Portelance, L.; Creutzberg, C.; Jürgenliemk-Schulz, I.M.; Mundt, A.; Mell, L.K.; Mayr, N.; Viswanathan, A.; Jhingran, A.; et al. Consensus guidelines for delineation of clinical target volume for intensity-modulated pelvic radiotherapy for the definitive treatment of cervix cancer. *Int. J. Radiat. Oncol.* **2011**, *79*, 348–355. [CrossRef]

22. Susko, M.; Craciunescu, O.; Meltsner, S.; Yang, Y.; Steffey, B.; Cai, J.; Chino, J. Vaginal Dose Is Associated With Toxicity in Image Guided Tandem Ring or Ovoid-Based Brachytherapy. *Int. J. Radiat. Oncol.* **2016**, *94*, 1099–1105. [CrossRef]

23. Bentzen, S.M.; Dörr, W.; Gahbauer, R.; Howell, R.W.; Joiner, M.C.; Jones, B.; Jones, D.T.; van der Kogel, A.J.; Wambersie, A.; Whitmore, G. Bioeffect modeling and equieffective dose concepts in radiation oncology—Terminology, quantities and units. *Radiother. Oncol.* **2012**, *105*, 266–268. [CrossRef]

24. Pötter, R.; Tanderup, K.; Kirisits, C.; De Leeuw, A.; Kirchheiner, K.; Nout, R.; Tan, L.T.; Haie-Meder, C.; Mahantshetty, U.; Segedin, B.; et al. The EMBRACE II study: The outcome and prospect of two decades of evolution within the GEC-ESTRO GYN working group and the EMBRACE studies. *Clin. Transl. Radiat. Oncol.* **2018**, *9*, 48–60. [CrossRef] [PubMed]

25. Zwanenburg, A.; Vallières, M.; Abdalh, M.A.; Aerts, H.J.W.L.; Andrearczyk, V.; Apte, A.; Ashrafinia, S.; Bakas, S.; Beukinga, R.J.; Boellaard, R.; et al. The Image Biomarker Standardization Initiative: Standardized Quantitative Radiomics for High-Throughput Image-based Phenotyping. *Radiology* **2020**, *295*, 328–338. [CrossRef] [PubMed]

26. Zwanenburg, A. Radiomics in nuclear medicine: Robustness, reproducibility, standardization, and how to avoid data analysis traps and replication crisis. *Eur. J. Nucl. Med. Mol. Imaging* **2019**, *46*, 2638–2655. [CrossRef]

27. Steyerberg, E.W.; Harrell, F.E.; Borsboom, G.J.; Eijkemans, M.; Vergouwe, Y.; Habbema, J.F. Internal validation of predictive models: Efficiency of some procedures for logistic regression analysis. *J. Clin. Epidemiol.* **2001**, *54*, 774–781. [CrossRef]

28. Moons, K.G.; Altman, D.G.; Reitsma, J.B.; Ioannidis, J.P.; Macaskill, P.; Steyerberg, E.W.; Vickers, A.J.; Ransohoff, D.F.; Collins, G.S. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): Explanation and elaboration. *Ann. Intern. Med.* **2015**, *162*, W1–W73. [CrossRef] [PubMed]

29. Brodersen, K.H.; Ong, C.S.; Stephan, K.E.; Buhmann, J.M. The balanced accuracy and its posterior distribution. In Proceedings of the 2010 20th International Conference on Pattern Recognition, Istanbul, Turkey, 23–26 August 2010; pp. 3121–3124.

30. Chen, J.; Chen, H.; Zhong, Z.; Wang, Z.; Hrycushko, B.; Zhou, L.; Jiang, S.; Albuquerque, K.; Gu, X.; Zhen, X. Investigating rectal toxicity associated dosimetric features with deformable accumulated rectal surface dose maps for cervical cancer radiotherapy. *Radiat. Oncol.* **2018**, *13*, 125. [CrossRef]

31. Hatt, M.; Tixier, F.; Visvikis, D.; Le Rest, C.C. Radiomics in PET/CT: More Than Meets the Eye? *J. Nucl. Med.* **2016**, *58*, 365–366. [CrossRef]

32. Annede, P.; Mailleux, H.; Sfumato, P.; Ferré, M.; Autret, A.; Cagetti, L.V.; Macagno, A.; Fau, P.; Chargari, C.; Tallet, A.; et al. Multivariate normal tissue complication probability modeling of vaginal late toxicity after brachytherapy for cervical cancer. *Brachytherapy* **2018**, *17*, 922–928. [CrossRef] [PubMed]

33. Kim, Y.; Kim, Y.-J.; Kim, J.-Y.; Lim, Y.K.; Jeong, C.; Jeong, J.; Kim, M.; Lim, M.C.; Seo, S.-S.; Park, S.-Y. Toxicities and dose-volume histogram parameters of MRI-based brachytherapy for cervical cancer. *Brachytherapy* **2017**, *16*, 116–125. [CrossRef]

34. Zhou, Y.; Zhao, L.; Wang, N.; Yong-Chun, Z.; Sun, X.; Zhang, Y.; Li, J.; Li, W.; Liu, J.; Wei, L.; et al. Late rectal toxicity determined by dose-volume parameters in computed tomography-based brachytherapy for locally advanced cervical cancer. *Cancer Med.* **2016**, *5*, 434–441. [CrossRef]

35. Lheureux, S.; Butler, M.O.; Clarke, B.; Cristea, M.C.; Martin, L.P.; Tonkin, K.; Fleming, G.F.; Tinker, A.V.; Hirte, H.W.; Tsores, D.; et al. Association of Ipilimumab With Safety and Antitumor Activity in Women with Metastatic or Recurrent Human Papillomavirus-Related Cervical Carcinoma. *JAMA Oncol.* **2018**, *4*, e173776. [CrossRef] [PubMed]

36. Pinter, C.; Lasso, A.; Wang, A.; Jaffray, D.; Fichtinger, G. SlicerRT: Radiation therapy research toolkit for 3D Slicer. *Med. Phys.* **2012**, *39*, 6332–6338. [CrossRef]
37. Moulton, C.R.; House, M.J.; Lye, V.; Tang, C.I.; Krawiec, M.; Joseph, D.J.; Denham, J.W.; Ebert, M.A. Spatial features of dose-surface maps from deformably-registered plans correlate with late gastrointestinal complications. *Phys. Med. Biol.* **2017**, *62*, 4118–4139.
[CrossRef]

38. El Naqa, I.; Brock, K.; Yu, Y.; Langen, K.; Klein, E.E. On the Fuzziness of Machine Learning, Neural Networks, and Artificial Intelligence in Radiation Oncology. *Int. J. Radiat. Oncol. Biol. Phys.* **2018**, *100*, 1–4. [CrossRef]

39. Gabryś, H.S.; Buettner, F.; Sterzing, F.; Hauswald, H.; Bangert, M. Design and Selection of Machine Learning Methods Using Radiomics and Dosiomics for Normal Tissue Complication Probability Modeling of Xerostomia. *Front. Oncol.* **2018**, *8*, 35.
[CrossRef] [PubMed]

40. Van Dijk, L.V.; Brouwer, C.L.; van der Schaaf, A.; Burgerhof, J.G.M.; Beukinga, R.J.; Langendijk, J.A.; Sijtsema, N.M.; Steen-bakkers, R. CT image biomarkers to improve patient-specific prediction of radiation-induced xerostomia and sticky saliva. *Radiother. Oncol.* **2017**, *122*, 185–191. [CrossRef]

41. Cunliffe, A.; Armato, S.G., 3rd; Castillo, R.; Pham, N.; Guerrero, T.; Al-Hallaq, H.A. Lung texture in serial thoracic computed tomography scans: Correlation of radiomics-based features with radiation therapy dose and radiation pneumonitis development. *Int. J. Radiat. Oncol. Biol. Phys.* **2015**, *91*, 1048–1056. [CrossRef] [PubMed]

42. Moran, A.; Daly, M.E.; Yip, S.S.; Yamamoto, T. Radiomics-based Assessment of Radiation-induced Lung Injury After Stereotactic Body Radiotherapy. *Clin. Lung Cancer* **2017**, *18*, e425–e431. [CrossRef] [PubMed]

43. Krafft, S.P.; Rao, A.; Stingo, F.; Briere, T.M.; Court, L.E.; Liao, Z.; Martel, M.K. The utility of quantitative CT radiomics features for improved prediction of radiation pneumonitis. *Med. Phys.* **2018**, *45*, 5317–5324. [CrossRef] [PubMed]

44. Gandhi, A.K.; Sharma, D.N.; Rath, G.K.; Julka, P.K.; Subramani, V.; Sharma, S.; Manigandan, D.; Laviraj, M.; Kumar, S.; Thulkar, S. Early clinical outcomes and toxicity of intensity modulated versus conventional pelvic radiation therapy for locally advanced cervix carcinoma: A prospective randomized study. *Int. J. Radiat. Oncol.* **2013**, *87*, 542–548. [CrossRef]

45. Naik, A.; Gurjar, O.; Gupta, K.; Singh, K.; Nag, P.; Bhandari, V. Comparison of dosimetric parameters and acute toxicity of intensity-modulated and three-dimensional radiotherapy in patients with cervix carcinoma: A randomized prospective study. *Cancer/Radiothé* **2016**, *20*, 370–376. [CrossRef] [PubMed]

46. Swamidas, J.; Kirisits, C.; De Brabandere, M.; Hellebust, T.P.; Siebert, F.-A.; Tanderup, K. Image registration, contour propagation and dose accumulation of external beam and brachytherapy in gynecological radiotherapy. *Radiother. Oncol.* **2020**, *143*, 1–11.
[CrossRef] [PubMed]