R Data Packages of Multi-Reader Multi-Case 

Studies and Simulation Tools to Support the 

Development of Reader Performance 

Evaluation Methods

Si Wen, Qi Gong, Brandon Gallas

FDA/CDRH/OSEL Division of Imaging, Diagnostics, and 

Software Reliability
2

Outline

• MRMC studies and analysis

– Multi-reader, Multi-case

• ROC: Receiver Operating Characteristic curves

• R Data packages

– What and how

• Show examples of MRMC data analysis

• Why

– Disseminate our research data, methods, and tools

– Enables other investigators to develop methods and tools
3

Background

• Medical device evaluation

– Assessing new device performance

V.S.

V.S.

Comparing two modalities 

in a medical task – Assessing computer aided diagnosis devices

AI detected potential lesion 

Image source: (1) https://www.independentimaging.com/digital-x-rays-vs-traditional-x-rays/ (2) https://www.featurepics.com/online/Doctor-Holding-Ray-Picture298823.aspx

(3)(4) Morra, Lia, et al. "Breast cancer: computer-aided detection with digital breast tomosynthesis." Radiology 277.1 (2015): 56-63. https://www.ncbi.nlm.nih.gov/pubmed/25961633

(1) (2) 

(3) (4) 
4

Motivation

• Design a study to compare two 

modalities for a medical task

– Ask clinicians to review cases under 

the two different modalities

• Random factors in the study:

– Cases: randomly select samples 

from intended case population

– Readers: randomly select samples 

from intended reader population

• Multi-Reader Multi-Case 

(MRMC) Study

– Each case is reviewed by multiple 

clinicians/readers

– Each reader review multiple cases

• MRMC analysis

– Account for reader and case 

variability

– Generalize results to the population 

of readers and cases
5

Multi-Reader Multi-Case Studies

• Application

– Comparing manual scores from clinicians by using different medical device

– Comparing machine learning algorithm result with manual scores

– Comparing the results from different machine learning algorithms

Medical Imaging and other technologies
6

• Data Structure

Multi-Reader Multi-Case Studies

Truth 1 1 1 1 1 0 0 0 0 0

90 70 100 40 50 10 20 80 60 30

0 50 100

High confidence 

of negative case

low confidence High confidence 

of positive case

70 80 100 90 50 20 60 40 30 10

Modality A

80 50 100 70 20 30 10 60 40 90

30 80 100 90 40 60 70 20 50 10

Modality B

readerID caseID modalityID confidenceScore

reader1 case1 A 90

reader1 case2 A 70

reader1 case3 A 100

⋮ ⋮ ⋮ ⋮

reader2 case1 A 70

reader2 case2 A 80

reader2 case3 A 100

⋮ ⋮ ⋮ ⋮

reader1 case1 B 80

reader1 case2 B 50

reader1 case3 B 100

⋮ ⋮ ⋮ ⋮

reader2 case7 B 70

reader2 case8 B 20

reader2 case9 B 50

reader2 case10 B 10

Reader study result

caseID Truth

case1 1

case2 1

case3 1

case4 1

case5 1

case6 0

case7 0

case8 0

case9 0

case10 0

Ground truth
7

• Receiver Operating Characteristic (ROC) analysis

Predicted as positive

Multi-Reader Multi-Case Studies

Truth 1 1 1 1 1 0 0 0 0 0

90 70 100 40 50 10 20 80 60 30

1-Specificity

Sensitivity

0 0.2 0.4 0.6 0.8 1

0.2

0.4

0.6

0.8

1

ROC curve

Area Under 

this ROC curve 

(AUC) 

10 20 30 40 50 60 70 80 90 100

Predicted as negative

t=100 Truth=1 Truth=0

Score≥100 1 0

Score<100 4 5

Sensitivity = 1/5 = 0.2 Specificity = 5/5=1

• AUC represents the 

probability that a 

random positive case 

is positioned to the 

right(>) of a random 

negative case

t=90 Truth=1 Truth=0

Score≥90 2 0

Score<90 3 5

Sensitivity = 2/5 = 0.4 Specificity = 5/5=1
8

• MRMC ROC analysis for two modalities

Truth 1 1 1 1 1 0 0 0 0 0

90 70 100 40 50 10 20 80 60 30

1-Specificity

Sensitivity

0 0.2 0.4 0.6 0.8 1

0.2

0.4

0.6

0.8

1

70 80 100 90 50 20 60 40 30 10

80 50 100 70 20 30 10 60 40 90

30 80 100 90 40 60 70 20 50 10

reader1 in Modality A: AUC = 0.80

reader2 in Modality A: AUC = 0.96

reader1 in Modality B: AUC = 0.68

reader2 in Modality B: AUC = 0.76

Reader-averaged AUC in modality A 

= 

層

匝

岫0.Β0 + 0.Γヶ岻=0.88

Modality A

Modality B

Reader-averaged AUC in modality B

= 

層

匝

岫0.ヶΒ + 0.Αヶ岻 =0.72

• Difference of reader-averaged AUC = 宋. 掻掻 − 宋. 挿匝 = 宋. 層掃

• Variance of difference of reader-averaged AUC - hard

Multi-Reader Multi-Case Studies
9

Multi-Reader Multi-Case Studies

Time consuming and labor intensive 

not much data is publicly available

Difficulties

Complicated correlation structure 

when computing variance of 

reader-averaged AUC

MRMC data storage and 

distribution 

R data packages

MRMC performance analysis 

iMRMC R package

Solutions
10

R data package for MRMC study

• To store and distribute data along with its documentation

– Well-organized documentation: description, format, source, etc.

– Public available: CRAN(The Comprehensive R Archive Network https://cran.rproject.org/) or GitHub (https://github.com/) 

• To provide use case for existing and future functions/methods

– Pre-processed reproducible data that can be easily accessed in R

• To help establishing future studies

– Acting as a baseline for similar study

– Providing information for sizing future study in this area
11

Building up an R data package

• Tool: devtools (https://www.rstudio.com/products/rpackages/devtools/)

• Steps: 

– devtools::create() create the package skeleton

– devtools::use_data_raw() create data-raw/

– Put data preprocessing script into .R file, which reads in raw data and at 

calls usethis::use_data() to save .rda formatted data files in data/
12

MRMC R data packages

• Structure

– colorScaleStudyData (https://github.com/DIDSR/colorScaleStudyData)

• Description (required as a R package) contains the package name, title, and description

• Readm.md (for GitHub) contains the instruction information of the package
13

MRMC R data packages

• Structure

– colorScaleStudyData

(https://github.com/DIDSR/colorScaleStudyData)

raw data

Read in raw data 

and output .rda

file in ../data

• Data-raw contains the raw dataset and functions to output the R format data
14

MRMC R data packages

• Structure

– colorScaleStudyData (https://github.com/DIDSR/colorScaleStudyData)

raw data

Read in raw data 

and output .rda

file in ../data

• Data contains the R format dataset can be read in R easily 
15

• The documentation is edited in the data.R file in R

MRMC R data packages

• Documentation

– colorScaleStudyData (https://github.com/DIDSR/colorScaleStudyData)

• devtools::document() generate .Rd manuals in man
16

MRMC R data packages

• Build the package

– devtools:: build() build source 

package and generate .tar.gz file

• No ./data-raw and .Rproj in the zip file

– In Rstudio, you can also build your 

package by click Build & Reload
17

MRMC R data packages - Summary

• To store and distribute data along with its documentation

– Structure

– Documentation

– How to create

• To provide use case for functions in existing and future packages

– How to use : apply doIMRMC in iMRMC package to the data
18

MRMC Data Analysis

• iMRMC package (https://github.com/DIDSR/iMRMC)

– A stand-alone, precompiled Java application developed by Dr. Brandon 

Gallas’ lab

– To analyze and size MRMC reader studies that compare the difference 

AUCs from two modalities, including the U-statistics estimation of 

variance of the difference of reader-averaged AUC

– iMRMC R package: an R package that utilizes the core Java application 

and output the R list object that contains all the analysis result.

(https://cran.r-project.org/web/packages/iMRMC/index.html)
19

MRMC Data Analysis

• iMRMC R package core functions:

– doIMRMC:

• Takes ROC data as a data frame and runs a MRMC analysis based on U-statistics

– sim.gRoeMetz:

• Simulate an MRMC data set of an ROC study comparing two modalities

• Allow testing other new methods
20

MRMC Data Analysis

• Apply doIMRMC in iMRMC package to color scale study data

https://github.com/DIDSR/colorScaleStudyData/blob/master/inst/extra/applyIMRMC.pdf
21

MRMC Data Analysis

• Apply doIMRMC in iMRMC package to color scale study data

– AUC and variance of AUC for each modality by U-statistics method

https://github.com/DIDSR/colorScaleStudyData/blob/master/inst/extra/applyIMRMC.pdf
22

MRMC Data Analysis

• Apply doIMRMC in iMRMC package to color scale study data

– Difference of AUC and variance of difference of AUC for pairwise 

comparison of different modalities by U-statistics method

https://github.com/DIDSR/colorScaleStudyData/blob/master/inst/extra/applyIMRMC.pdf
23

MRMC Data Analysis

• R-markdown files 

– Demonstrate the use of functions and data

– colorScaleStudyData

(https://github.com/DIDSR/colorScaleStudyData/blob/master/inst/extra

/applyIMRMC.Rmd)

• More package application

– viperData R data package (https://didsr.github.io/viperData/ )
24

MRMC Data Analysis

• iMRMC R package core functions:

– doIMRMC:

• Takes ROC data as a data frame and runs a MRMC analysis based on U-statistics

– sim.gRoeMetz:

• Simulate an MRMC data set of an ROC study comparing two modalities

• Allow testing other new methods

26

Simulation Tool for MRMC Data

• Apply sim.gRoeMetz to simulate MRMC data

⋮
27

Summary

• Multi-Reader Multi-Case (MRMC) study

– Data storage and distribution: R data Package

– MRMC ROC analysis: doIMRMC in iMRMC package

– MRMC simulation tool: sim.gRoeMetz in iMRMC package
28

Reference Links

• colorScaleStudyData R data package https://github.com/DIDSR/colorScaleStudyData

• viperData R data package https://didsr.github.io/viperData/

• iMRMC package https://github.com/DIDSR/iMRMC

• iMRMC R package https://cran.r-project.org/web/packages/iMRMC/index.html

• References:

– Gallas, Brandon D., et al. "Impact of prevalence and case distribution in lab-based diagnostic imaging studies." Journal of 

Medical Imaging 6.1 (2019): 015501. https://www.spiedigitallibrary.org/journals/Journal-of-Medical-Imaging/volume6/issue-1/015501/Impact-of-prevalence-and-case-distribution-in-lab-based-diagnostic/10.1117/1.JMI.6.1.015501.full

– Gallas, Brandon D., and Stephen L. Hillis. "Generalized Roe and Metz receiver operating characteristic model: analytic link 

between simulated decision scores and empirical AUC variances and covariances." Journal of Medical Imaging 1.3 (2014): 

031006. https://www.spiedigitallibrary.org/journals/journal-of-medical-imaging/volume-1/issue-3/031006/GeneralizedRoe-and-Metz-receiver-operating-characteristic-model--analytic/10.1117/1.JMI.1.3.031006.full

– Gallas, Brandon D., et al. "A framework for random-effects ROC analysis: biases with the bootstrap and other variance 

estimators." Communications in Statistics—Theory and Methods 38.15 (2009): 2586-2603. 

https://www.tandfonline.com/doi/abs/10.1080/03610920802610084

– Gallas, Brandon D., et al. "Evaluating imaging and computer-aided detection and diagnosis devices at the FDA." Academic 

radiology 19.4 (2012): 463-477. https://www.sciencedirect.com/science/article/pii/S1076633211006246
29

Thank you!

Q&A