A Multi-Reader Multi-Case Controlled Clinical Trial to Evaluate the Performance Improvement From Computer-aided Tool for the Prognostic Prediction of Colorectal Liver Metastases
Conditions: Colorectal Liver Metastasis (CRLM)
Sponsor: Cancer Institute and Hospital, Chinese Academy of Medical Sciences

 
 
 
 
 
 
     
Official Title of the study：
A Multi-Reader MultiCase Controlled Clinical 
Trial to Evaluate the Performance Improvement 
from Computer -aided Tool for the Prognostic 
Prediction of Colorectal Liver Metastases. 
NCT number:NCT07027605
DATE of the document:2025.06.15
 

A Multi-Reader Multi-Case Controlled Clinical Trial to Evaluate the 
Performance Improvement from Computer-aided Tool for the Prognostic 
Prediction of Colorectal Liver Metastases  
 
1. Synopsis 
1.1. Introduction 
This study aims to evaluate the performance improvement from computer-aided tool for the 
prognostic prediction of colorectal liver metastases (CRLM). 
1.2. Prediction Model Summary 
In 2024, Chen et al. [1] firstly developed clinical prognosis models utilizing the Random Forest 
(RF) approach, along with a web-based tool designed for practical application of these 
predictive models in clinical settings for patients with CRLM undergoing simultaneous 
resection. These models demonstrated excellent performance and can assist in facilitating 
accurate and individualized treatment for CRLM patients. 
1.3. Study Summary 
A multiple reader, multiple case (MRMC) clinical study including a concurrent reading design 
will be conducted to determine the impact of the CRLM tool on physicians’ performance in 
predicting disease progression and death, as characterized by the designated endpoints, when 
the physician’s evaluation is compared with the ground truth. A total of 12 readers will evaluate 
166 patients’ medical records, then made prediction of the risk of post-operative 1, 3, 5-years 
recurrence and death, and provide follow-up timelines and treatment recommendations with or 
without the aid of CRLM prediction tool, by a memory washout period of approximately 5 
weeks. The study will utilize retrospective data that is independent from the training dataset of 
the prediction tool. The terms ‘physicians’ and ‘reader’ are used interchangeably in this protocol. 
The same applies to ‘patients’ and ‘case’. 
1.4. Indication 
Patients with CRLM receiving simultaneous resection. 
1.5. Hypothesis 
This study aims to verify that the CRLM prediction tool improves clinician performance as 
measured by the designated endpoints, when physician’s findings are compared to the ground 
truth. All the objectives and endpoints will be assessed independently. This is a pilot study, the 
hypothesis testing will be conducted only on the primary endpoint. All the other analyses are 
exploratory in nature.  
1.6. Objectives and Endpoints 
 
Primary Objectives 
Primary Endpoints 
To compare aid versus without aid with respect to 
(improved) AUC-ROCs for post-operative prediction 
of post-operative 3-year mortality made by readers. 
AUCs: 
Area 
Under 
the 
Receiver 
Operating 
Characteristic Curve (AUC-ROC)  
Secondary Objective 
Secondary Endpoints 

To evaluate aid versus without aid with respect to 
(improved) AUC-ROCs for post-operative prediction 
of 
post-operative 
1-year 
recurrence, 
3-year 
recurrence, 5-year recurrence, 1-year mortality, and 
5-year mortality made by readers. 
AUCs: 
Area 
Under 
the 
Receiver 
Operating 
Characteristic Curve (AUC-ROC)  
To evaluate aided versus unaided with respect to 
(improved) 
sensitivity 
for 
prediction 
of 
post-operative 1-year, 3-year, and 5-year recurrence 
and mortality made by readers. 
Sensitivity: the ratio of true positives to total (actual) 
positives. 
To evaluate aided versus unaided with respect to 
(improved) 
specificity 
for 
prediction 
of 
post-operative 1-year, 3-year, and 5-year recurrence 
and mortality made by readers. 
Specificity: the ratio of true negatives to total (actual) 
negatives. 
To evaluate aided versus unaided with respect to 
(improved) inter-rater reliability for rating on the 
likelihood of post-operative 1-year, 3-year, and 
5-year recurrence and mortality made by readers. 
Inter-rater reliability: the consistency of ratings 
made by readers on the cases 
Exploratory Objective 
Exploratory Endpoints 
To evaluate aided versus unaided with respect to 
inter-rater reliability for readers’ recommendation on 
initiating surgical treatment; To evaluate aided versus 
unaided with respect to inter-rater reliability for 
readers’ recommendation on the timing of the first 
post-surgery follow-up.  
Inter-rater reliability: the consistency of ratings 
made by readers on the cases 
To evaluate aided versus unaided with respect to 
(improve) decision-making confidence in predicting 
the disease progression and making recommendation 
for a case. The general confidence on prediction 
model in pre and post study will also be evaluated. 
 
Decision-making confidence:  confidence of a 
reader in completing the evaluation for a case 
To evaluate aided versus unaided with respect to 
(reduced) decision-making time in completing a case 
evaluation. 
Decision-making time: duration of a reader in 
completing the evaluation for a case 
 
 
 

 
2. Background and Rationale 
2.1. Clinical & Disease Background 
Colorectal cancer is the third most commonly diagnosed cancer globally and the second leading 
cause of cancer-related deaths[2]. At the time of initial diagnosis, around 20% to 30% of 
colorectal cancer patients present with synchronous liver metastases[3]. This condition often 
results in more aggressive disease progression and poorer survival outcomes for those affected [4]. 
Therefore, improving the prognosis for individuals with synchronous CRLM has become a critical 
challenge and priority in clinical practice. 
2.2. Current Clinical Practice 
The simultaneous resection of colorectal cancer along with liver metastases is the preferred 
strategy for selected patients with synchronous CRLM. This approach offers numerous advantages, 
such as reduced surgical burden, shorter anesthesia time, and improved survival rates compared to 
staged resections[5, 6]. However, it is important to note that not all patients benefit equally from 
this method. Research indicates that approximately 62.8% of CRLM patients experience 
postoperative recurrence, with relapse rates exceeding 50% within the first two years. Furthermore, 
the median progression-free survival is only 11.7 months, which correlates strongly with worse 
overall survival rates [5, 6]. 
Consequently, early identification of CRLM patients undergoing simultaneous resection who 
are at high risk for postoperative recurrence and mortality is critical for clinicians. This enables the 
development of tailored follow-up and treatment strategies. There is a pressing need for effective 
outcome prediction tools for CRLM patients undergoing simultaneous resection in specific 
clinical contexts, as this could significantly enhance clinical decision-making processes. Accurate 
prognostic assessments are essential for devising appropriate treatment and follow-up plans. 
 
2.3. Prediction Model Background 
High-performing predictive models are vital tools that can aid clinicians in developing 
treatment strategies and creating effective follow-up plans to improve the prognosis of patients 
with CRLM. Although several institutions have created prognostic nomograms to predict 
outcomes for CRLM patients[7-10], there is a lack of studies specifically addressing those 
undergoing simultaneous resection. In previous research, Chen et al. [1]firstly establish RF models 
that incorporate a wide range of variables, including demographic, clinical, laboratory, and genetic 
data, and to develop a web-based prediction tool designed to forecast long-term outcomes for 
CRLM patients undergoing simultaneous resection. 
The RF models showed significant enhancements in predictive performance during both 
internal and external validation compared to traditional CRS scores. The AUC values for the 
survival model at 1 year, 3 years, and 5 years outperformed those of other models, such as CERR 
score, GAME score, and modified clinical score, reported in previous studies [11-13]. 
Furthermore, Chen et al. developed a user-friendly web-based prediction tool based on patients' 
baseline characteristics for clinical application. The practical implications of this prediction tool 
are considerable, as it provides clinicians with accurate risk assessments tailored to a patient’s 
tumor characteristics, ultimately facilitating personalized treatment interventions and follow-up 
plans. 
 

 
 
 
 
 
 
2.4. Similar Studies  
No similar study to evaluate the clinical benefit of a prognostic prediction model in clinical 
practice. 
 
3. Study Design 
 
3.1. Data Collection Procedure 
This is a retrospective MRMC study which employs a fully crossed design in which all readers 
review medical records from all cases in two visits separated by a memory washout period of 
approximately 1 month. Readers are divided into two groups (A and B) roughly reflecting similar 
reader qualifications and experience. Training and instructions will be provided to each reader so 
that readers work from a set of given rules including a consistent process for recording their 
evaluations 
3.2. Rationale for Endpoints  
Primary endpoints: 
The Area Under the Receiver Operating Characteristic Curve (AUC-ROC) for predicting 
post-operative 3-year mortality, comparing clinician performance with and without the aid of the 
prediction model according to risk scores: (low risk)＜10%, 2 (Moderate Risk)10%-30%, 3 (High 
Risk)30-70% and 4 (Critical Risk)＞70%.  
Secondary endpoints:  
The AUC-ROC for predicting post-operative 1-year recurrence, 3-year recurrence, 5-year 
recurrence, 1-year mortality, and 5-year mortality, comparing clinician performance with and 
without the aid of the prediction model according to risk scores: (low risk)＜10%, 2 (Moderate 
Risk)10%-30%, 3 (High Risk)30-70% and 4 (Critical Risk)＞70%.  
After surgery, all patients were routinely monitored for recurrence through thoracic, 
abdominal, and pelvic computed tomography, magnetic resonance imaging of the liver, and serum 
carcinoembryonic antigen (CEA) measurements. The first postoperative assessment took place 
one month post-surgery, and subsequent evaluations were conducted in the outpatient clinic every 
three months. The rates of recurrence or mortality at one year, three years, and five years 
post-operation were determined based on these earlier follow-up reviews. 
 
4. Sample Population 
4.1. Reader Characteristics 
A reader study with 12 participating physicians (4 Junior Physician, 4 mid-level Physician and 4 
Senior Physician) from the Department of Surgical Oncology of the Digestive Tract will be 
conducted. The standardized training will be provided to all readers how to use medical record 
system to get patients data, and how to complete the questionnaire.  
4.2. Case Population 

The study will target 166 CRLM patients receiving simultaneous resection. 
Inclusion Criteria: 
- ≥ 18 years old 
- confirmation of histologically diagnosed liver metastases of colorectal adenocarcinoma  
- receiving colorectal resection with simultaneous liver resection. 
 
Exclusion Criteria: 
-presence of other malignancies 
-absence of follow-up data 
-patients who were followed up postoperatively for less than 5 years and had no occurrences of 
death. 
 
5. Study Procedure 
5.1. Ground Truth 
The recurrent and modality status, and the date of experiencing the events of each patient have 
been documented. The readers will be blinded to the ground truth. 
5.2. Reading Procedure 
The standardized training will be provided to all readers how to use medical record system to get 
patients data, and how to complete the questionnaire. Reading will be conducted in a blinded 
fashion in the sense that the readers will not have access to the identification information and 
post-operative recurrence and modality status during either the aided or unaided portion of the 
reading. During the reading process, whether aided or unaided, the readers will only have access 
to anonymous medical information relevant to surgery, as well as information pertaining to the 
period prior to the surgical procedure. This includes medical records, demographics, imaging 
examinations, preoperative biochemical markers, surgery-related variables, tumor pathological 
characteristics, genetic markers, and treatment information from the medical records system. In 
the aided session, the individualized predicated KM curves and landmark survival probabilities 
based on prediction models for the case will be presented in addition to the information provided 
in unaided session.  
Each reader will be asked to complete a questionnaire about the risk score of post-operative 
1-year, 3-year, and 5-year recurrence or mortality, their confidence of the making the decision and 
the recommended timing for the first follow-up examination after surgery. Time from starting to 
read the medical history to completing the questionnaire will be recorded. 
 Patient cases will be equally and randomly split between Dataset A and Dataset B. The readers 
are also equally and randomly split between Group A and Group B. During each session each 
reader reads all the cases assigned to the group they belong to in that session. There will be at least 
a one-month washout period between sessions.   
 
 
 
 
 
 
 

 
 
 
 
 
 
 
Study Schema like below 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
5.3. Study discontinuities 
Several extenuating circumstances exist where readers may not be able to read all cases expected 
of them. Table below summarize several such scenarios and responses. 
Study Discontinuity Scenarios and responses  
Scenario 
Protocol 
A reader is not able to complete all the cases 
assigned to his group in session 1, but remains 
committed to continue reading in session 2.  
Additional cases will be assigned to either existing readers or new 
readers to supplement the cases that reader could not complete. 
A reader withdraws after session 1 and is not 
able to participate in session 2. 
A new reader will be identified to replace this reader. This new 
reader will complete both session 1 and 2.  
A reader completes session 1 but can only 
partially complete session 2.  
Depending on the number of unread cases in session 2, 
investigator will opt to either (1) forego the unmatched cases 
Reader Group A: 
Interprets Dataset A in 
unaided 
scenario 
and 
Dataset 
B 
in 
aided 
scenario 
Reader Group B: 
Interprets Dataset A in 
aided 
scenario 
and 
Dataset B in unaided 
scenario 
1st Study Session 
Reader Group A: 
Interprets Dataset A in 
aided 
scenario 
and 
Dataset B in unaided 
scenario 
2nd Study Session 
Minimum 5 weeks separation between sessions to allow for memory wash-out 
Reader Group B: 
Interprets Dataset A in 
unaided 
scenario 
and 
Dataset 
B 
in 
aided 
scenario 

which will reduce sample size and statistical power, or (2) assign 
unmatched cases to new or existing readers. In the latter scenario, 
the replacement reader will need to complete both session 1 and 2 
with the assigned cases, including the washout period.  
 
 
5.4  Safety Assessment  
There are no safety concerns of physicians on using the prediction tools and making evaluation for 
patients. As cases are retrospectively evaluated, patients would not experience safety concern 
through incorrect evaluation or action of a reader in this study. 
 
 
5.5. Data Security 
All case data used in this study is completely anonymized. This includes any identifying 
information (e.g. patient name) displayed to readers. All data collected in this reader study will be 
securely maintained by Cancer Hospital, Chinese Academy of Medical Sciences. 
 
6. Statistical Plan 
 
Study Design Overview 
A retrospective, MRMC study, employs a fully crossed design in which 
all readers (physicians) review medical records from all cases (patients) 
in two visits separated by a memory washout period of approximately 5 
weeks. 
Analysis Population 
The intention-to-treat (ITT) population will serve as the population for 
primary efficacy analysis. All randomized participants (readers and 
cases) will be included in this population. 
Primary Endpoints 
AUC-ROC 
Secondary Endpoints 
Sensitivity 
Specificity 
Interrater reliability 
Statistical Methods 
The primary endpoints, between reading conditions, will be compared 
using the standard MRMC analysis of variance (ANOVA) method of 
Obuchowski and Rockette (OR)[14], where both reader and case are 
random effects. 
Interim Analyses 
No interim analysis of study endpoints is planned. 
Multiplicity 
Other analyses will be exploratory in nature. No multiplicity adjustment 
will be implemented. Nominal p-value will be reported. 
Sample Size and Power 
A sample size of 166 cases (90 experienced OS events within 3 years), 

and 12 readers was selected for this study. The AUC for predicting 
3-year mortality is designed for two-sided α = 0.05 and power of at 
least80% to detect a 0.05 absolute advantage in AUC for reading with 
an assistant of the CRLM prediction tool.  
 
 
 
6.1. Response for Analyses/Blinding 
The readers will be blinded to the ground truth of the cases’ recurrence or mortality status they 
will be asked to review. Readers will also not have access to results from other readers or 
information that could influence the diagnosis decision. The statistical analysis of the data 
obtained from this study will be the responsibility of the Sponsor and conducted by an unblinded 
statistician. 
 
6.2. Analysis Endpoints/Objectives 
The following endpoints are used to compare the reader’s performance for assessing patients’ 
disease progression and death under two reading scenarios (i.e. with the aid of the CRLM 
prediction tool vs. without aid). 
 
6.2.1 Primary Endpoint 
AUCs: Area Under the Receiver Operating Characteristic Curve (AUC-ROC)  
Objective: To compare aid versus without aid with respect to (improved) AUC-ROCs for 
post-operative prediction of post-operative 3-year mortality made by readers. 
 
6.2.2 Secondary Endpoints 
 
AUCs: Area Under the Receiver Operating Characteristic Curve (AUC-ROC)  
Objective: To evaluate aid versus without aid with respect to (improved) AUC-ROCs for 
post-operative prediction of post-operative 1-year recurrence, 3-year recurrence, 5-year recurrence, 
1-year mortality, and 5-year mortality made by readers. 
 
Sensitivity: the ratio of true positives to total (actual) positives. 
Objective: To evaluate aided versus unaided with respect to (improved) sensitivity for prediction 
of post-operative 1-year, 3-year, and 5-year recurrence and mortality made by readers. 
 
Specificity: the ratio of true negatives to total (actual) negatives. 
Objective: To evaluate aided versus unaided with respect to (improved) specificity for prediction 
of post-operative 1-year, 3-year, and 5-year recurrence and mortality made by readers. 

 
Inter-rater reliability: the consistency of ratings made by readers on the cases 
Objective: To evaluate aided versus unaided with respect to (improved) inter-rater reliability for 
rating on the likelihood of post-operative 1-year, 3-year, and 5-year recurrence and mortality made 
by readers. 
 
6.2.3 Exploratory Endpoints 
 
Inter-rater reliability: the consistency of ratings made by readers on the cases 
Objective: To evaluate aided versus unaided with respect to inter-rater reliability for readers’ 
recommendation on initiating surgical treatment; To evaluate aided versus unaided with respect to 
inter-rater reliability for readers’ recommendation on the timing of the first post-surgery 
follow-up.  
 
Decision-making confidence:  confidence of a reader in completing the evaluation for a case 
Objective: To evaluate aided versus unaided with respect to (improve) decision-making 
confidence in predicting the disease progression and making recommendation for a case. The 
general confidence on prediction model in pre and post study will also be evaluated. 
 
Decision-making time: duration of a reader in completing the evaluation for a case 
Objective: To evaluate aided versus unaided with respect to (reduced) decision-making time in 
completing a case evaluation. 
 
 
6.3. Randomization/Treatment Assignment 
This is a retrospective MRMC study, which employs a fully crossed design in which all cases will 
be evaluated the same way by all study readers, such that there are no treatment assignments or 
treatment groups. 
The 12 readers will be randomly allocated into two groups, reader group A and B. The 166 
cases will be randomly allocated into two subsets of 83 cases each, case subsets A and B. The 
cases within a case subset will be assigned to each reader in a randomized case order. Each reader 
will read one case subset with prediction tool aided and another case subset as unaided during 
Session 1, and the complementary unaided and aided during Session 2. 
Study case identification numbers and study reader numbers will be assigned to all cases and 
readers and used to protect their identities in statistical analysis and in reporting of results. 
 
6.4. Statistical Methods 
 
6.4.1 Statistical Methods for Efficacy Analyses 
 

6.4.1.1 Primary Endpoints 
AUC-ROCs for 1-year, 3-year, 5-year of recurrence and mortality will be estimated for each 
reader in each review condition (aided or unaided, and the difference between them) based on the 
reader's interpretations of all cases. The account for right-censor data, the inverse probability of 
censoring weighting estimate was used for the cumulative/dynamic time-dependent ROC curve 
approach by Blanche et al. [15].  
AUC-ROCs between reading conditions will be compared using the standard MRMC analysis of 
variance (ANOVA) method of Obuchowski and Rockette (OR), where both reader and case are 
random effects, to ensure generalization of the study results to both the population of readers and 
the population of cases. 
The average AUCs within each reading condition, as well as its standard error and two-sided 95% 
confidence intervals (CIs) will be reported. The average difference in AUCs, its standard error, 
corresponding two-sided 95% CIs, and statistical inference will be computed, referencing to the 
approach in Obuchowski and Rockette. No adjustments for covariates are planned. 
Sensitivity analysis may be conducted using OR approach with only reader (sensitivity analysis 1) 
and only case (sensitivity analysis 2) as random effects. 
 
6.4.1.2 Secondary Endpoints 
Analyses of sensitivity and specificity will also use standard MRMC ANOVA methods to compare 
reader performance under different reading condition and use two-sided 95% CIs to quantify 
uncertainty. Sensitivity and specificity each provide additional information needed to understand 
the expected impact of using the prediction tool on clinical practice. Their analysis and reporting, 
including confidence intervals, is thus required to provide a complete description of reader 
performance. 
Analysis of Inter-rater reliability or other ordinal outcomes collected under different reading 
condition will use generalized linear mixed model (for appropriate link function) with reading 
condition as a covariate and both reader and case are random effects. 
 
6.4.1.3 Missing Data 
Readers are required to complete evaluating all cases in the two sessions, and missing reader data 
will be prevented. If any, missing reader data will be reviewed and reported (considered as 
protocol deviations). No imputation is planned for any study data. The analysis methods would 
accommodate missing reader data under missing at random assumption. 
 
6.4.2 Statistical Methods for Safety Analyses  
No adverse events are anticipated in this study using retrospective cases. Readers are not expected 
to report any adverse events.  
 
6.4.3 Summary of Baseline Characteristics  
The demographic and baseline characteristics of the cases will be assessed by the use of tables 

and/or graphs. The baseline characteristic of readers (e.g. confidence on prediction aided tool) will 
also be assessed in the same way. No statistical tests will be performed on these characteristics. 
 
6.5. Interim Analysis 
No interim analysis of study endpoints are planned. The final analysis will be triggered when all 
the readers complete and record their findings for all the assigned cases at Session 2. 
 
 
6.6 Multiplicity 
The study allocated two-sided α = 0.05 to test AUC for 3-year mortality between reading 
conditions. All the other analyses are exploratory in nature. No multiplicity adjustment will be 
implemented. The nominal p-value will be reported. 
 
6.7. Sample Size and Power Calculation 
This study is well powered for the primary endpoint AUC of 3-year mortality prediction. A sample 
size of 166 cases and 12 readers was selected for this study. The method of Obuchowski[14] is 
used to determine the number of readers required in a fully crossed design to provide at least 80% 
power to detect a 0.05 absolute advantage in AUC for reading, at statistical significance level α = 
0.05 (two-sided). The disease cases to non-disease cases (for 3-year mortality) will be randomly 
selected with an approximately 1:1 ratio from the patient dataset pool, which is independent from 
the dataset used to training the prediction tool. A balanced disease vs non-disease design may not 
reflect the actual disease prevalence in the real-world setting but enhance statistical efficiency in 
assessing the accuracy of diagnostic tests. The sample size calculations conservatively assume that 
both readers and cases are random effects.  
Due to this being a pilot study, there were no estimate of variance components. Variance 
component estimates (via Obuchowski and Rockette approach using jackknife method) from 
previous MRMC studies that use a similar study design as the proposed (regardless of the original 
endpoint types) were leveraged in the power and sample size calculation of this study. All power 
estimates are dependent upon the appropriateness of the assumed variance components. 
 
6.8. Subgroup Analysis 
 
To determine whether the advantage of a prediction tool in assisting physicians is consistent across 
various subgroups, the between reading conditions differences in AUCs (with a nominal 95% CI) 
will be estimated within each category of the following classification variables: 
● 
Reader Subgroups 
○ 
Reader expertise (junior physician, mid-level physician, senior physician) 
● 
Case Subgroups 
○ 
Age at baseline (<60, >=60) 
● 
Primary Tumor site Subgroups 

○ 
 Primary tumor site (right half-colon, sigmoid colon, and rectum) 
 
Appendix: Patient Evaluation Questionnaire 
 
 
 
 
 
 
Reference 
[1] Q. Chen, J. Chen, Y. Deng, X. Bi, J. Zhao, J. Zhou, Z. Huang, J. Cai, B. Xing, Y. Li, K. Li, H. Zhao, 
Personalized prediction of postoperative complication and survival among Colorectal Liver Metastases 
Patients Receiving Simultaneous Resection using machine learning approaches: A multi-center study, 
Cancer letters, 593 (2024) 216967. 
[2] F. Bray, M. Laversanne, H. Sung, J. Ferlay, R.L. Siegel, I. Soerjomataram, A. Jemal, Global cancer 
statistics 2022: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 
countries, CA: a cancer journal for clinicians, 74 (2024) 229-263. 
[3] N. Machairas, M. Di Martino, F. Primavesi, P. Underwood, M. de Santibanes, I. 
Ntanasis-Stathopoulos, I. Urban, D.I. Tsilimigras, A.K. Siriwardena, A.E. Frampton, T.M. Pawlik, 
Simultaneous resection for colorectal cancer with synchronous liver metastases: current 
state-of-the-art, Journal of gastrointestinal surgery : official journal of the Society for Surgery of the 
Alimentary Tract, 28 (2024) 577-586. 
[4] K. Kataoka, K. Takahashi, J. Takeuchi, K. Ito, N. Beppu, W. Ceelen, Y. Kanemitsu, Y. Ajioka, I. Endo, K. 
Hasegawa, K. Takahashi, M. Ikeda, Correlation between recurrence-free survival and overall survival 
after upfront surgery for resected colorectal liver metastases, The British journal of surgery, 110 (2023) 
864-869. 
[5] Y. Wu, A. Mao, H. Wang, G. Fang, J. Zhou, X. He, S. Cai, L. Wang, Association of Simultaneous vs 
Delayed Resection of Liver Metastasis With Complications and Survival Among Adults With Colorectal 
Cancer, JAMA network open, 5 (2022) e2231956. 
[6] Q. Chen, Y. Deng, Y. Li, J. Chen, R. Zhang, L. Yang, R. Guo, B. Xing, P. Ding, J. Cai, H. Zhao, Association 
of preoperative aspartate aminotransferase to platelet ratio index with outcomes and tumour 
microenvironment among colorectal cancer with liver metastases, Cancer letters, 588 (2024) 216778. 
[7] K. Imai, M.A. Allard, C. Castro Benitez, E. Vibert, A. Sa Cunha, D. Cherqui, D. Castaing, H. Bismuth, H. 
Baba, R. Adam, Nomogram for prediction of prognosis in patients with initially unresectable colorectal 
liver metastases, The British journal of surgery, 103 (2016) 590-599. 
[8] W. Liu, K. Wang, Y. Han, J.Y. Liang, Y.H. Li, B.C. Xing, Nomogram predicted disease free survival for 
colorectal liver metastasis patients with preoperative chemotherapy followed by hepatic resection, 
European journal of surgical oncology : the journal of the European Society of Surgical Oncology and 
the British Association of Surgical Oncology, 45 (2019) 2070-2077. 
[9] Q. Chen, R. Mao, J. Zhao, X. Bi, Z. Li, Z. Huang, Y. Zhang, J. Zhou, H. Zhao, J. Cai, Nomograms 
incorporating preoperative RDW level for the prediction of postoperative complications and survival in 
colorectal liver metastases after resection, Annals of palliative medicine, 10 (2021) 4143-4158. 
[10] W. Liu, W. Zhang, Y. Xu, Y.H. Li, B.C. Xing, A Prognostic Scoring System to Predict Survival Outcome 
of Resectable Colorectal Liver Metastases in this Modern Era, Annals of surgical oncology, 28 (2021) 

7709-7718. 
[11] G.A. Margonis, K. Sasaki, S. Gholami, Y. Kim, N. Andreatos, N. Rezaee, A. Deshwar, S. Buettner, P.J. 
Allen, T.P. Kingham, T.M. Pawlik, J. He, J.L. Cameron, W.R. Jarnagin, C.L. Wolfgang, M.I. D'Angelica, M.J. 
Weiss, Genetic And Morphological Evaluation (GAME) score for patients with colorectal liver 
metastases, The British journal of surgery, 105 (2018) 1210-1220. 
[12] K.W. Brudvik, R.P. Jones, F. Giuliante, J. Shindoh, G. Passot, M.H. Chung, J. Song, L. Li, V.J. 
Dagenborg, A. Fretland Å, B. Røsok, A.M. De Rose, F. Ardito, B. Edwin, E. Panettieri, L.M. Larocca, S. 
Yamashita, C. Conrad, T.A. Aloia, G.J. Poston, B.A. Bjørnbeth, J.N. Vauthey, RAS Mutation Clinical Risk 
Score to Predict Survival After Resection of Colorectal Liver Metastases, Annals of surgery, 269 (2019) 
120-126. 
[13] Y. Chen, W. Chang, L. Ren, J. Chen, W. Tang, T. Liu, M. Jian, Y. Liu, Y. Wei, J. Xu, Comprehensive 
Evaluation of Relapse Risk (CERR) Score for Colorectal Liver Metastases: Development and Validation, 
The oncologist, 25 (2020) e1031-e1041. 
[14] N.A.O. Jr, H.E.R.J.C.i.S.-S. Jr, Computation, Hypothesis testing of diagnostic accuracy for multiple 
readers and multiple tests an anova approach with dependent observations, 24 (1995) 285-308. 
[15] Blanche, Paul, Aurélien Latouche, and Vivian Viallon. "Time-dependent AUC with 
right-censored data: a survey." Risk assessment and evaluation of predictions (2013): 239-251.