Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: The authors introduce RadGraph which associates full-text radiology reports with clinical entities that makes use of a novel information extraction scheme to structure radiology reports. They introduce and employ the schema and use a deep learning model, RadGraph Benchmark, with MIMIC-CXR and CheXpert datasets. This contribution is an important one in the field of natural language processing and computer vision with potential applications in a wide range of other fields in medicine.
strengths: - multiple datasets used
- association between full text radiology reports and imaging 
- board-certified radiologist annotations used
weaknesses: - no validation in other countries; MIMIC and CheXpert are both USA
correctness: The claims seem correct; I would include some word on sample size limitations in the limitations section; this goes in line with the fact that datasets from the same country were used, likely from pretty similar populations given BIDMC and Stanford are in Boston and Palo Alto, respectively
clarity: Written very well
relation to prior work: Builds upon prior work as detailed in the paper; the authors seem judicious about their claims but nonetheless push forward the technology in the field
additional feedback: Great work!
review_title: Review of "RadGraph: Extracting Clinical Entities and Relations from Radiology Reports"

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: Jain et al present a novel information extraction schema for free text radiology reports and apply this to existing datasets (MIMIC-CXR and CheXpert) to generate a new, curated dataset of structured clinical annotations. Next, they train a deep learning benchmark, RadGraph, to automatically extract clinical entities and relationships. They thoroughly document dataset curation and benchmark their deep learning model to show excellent performance at identification of clinical entities and relationships. Their densely annotated, expert curated development and test datasets are released along with an inference dataset with automatically generated annotations, which are all freely available on Physio.net. Overall, the work is very strong and a significant contribution to natural language processing for radiology report annotation. In addition, the datasets have potential to stimulate multiple lines of work in the NLP, computer vision, and multi-modal learning community.

Updated review:
The revised manuscript is strong and addresses my previous concerns.
strengths: Their novel information extraction schema is clinically relevant, logical, and addresses limitations in previous schemas. Their deep learning model is rigorously bench-marked with excellent performance. The datasets are collected with high-fidelity, well-documented, and freely available. The work is broadly relevant to the medical NLP and computer vision communities with the potential stimulate multiple studies and lines of research. 
weaknesses: Not a weakness, per se, but inclusion of de-identified demographic metadata such as age or binned age (45-50), gender/sex, or ethnicity in future versions/releases would allow studies to examine whether RadGraph or other algorithms show similar performance across different demographic groups or whether hidden disparities exist.


Updated review:
The revised manuscript addresses this issue with the addition of Table 2.
correctness: The dataset is rigorously constructed by expert radiologists after three labeling/training pilots. The submission includes a Datasheet, ML Reproducibility Checklist, supplement, and is hosted on a public data repository.
clarity: Overall, the manuscript was well-written with a detailed description of the dataset and methods as well as compelling results. Manuscript formatting adheres to the guidelines, and no spelling or grammar errors were identified.
relation to prior work: This is a significant contribution that has the potential to assist the development of large scale, structured annotation of free-text radiology reports. Relevance to previous work is discussed.
additional feedback: Very nice work, it was a pleasure to review! A couple of general comments for future releases of the dataset, if applicable. It could be useful if the metadata included some de-identified demographic information such as age or binned age (45-50), gender/sex, or ethnicity. The reason is that future studies may wish to test whether algorithms show similar performance across different demographic groups or whether there are hidden disparities. However, the reviewer understands that this information either may not be available since the work is partially derived from existing datasets or that the information may be available in the original datasets and that future studies can map to these metadata if needed. Also, the size and utility of the dataset could be enhanced in the future if there was the ability to crowd-source radiologist verification of the annotations in the inference dataset, which could then be incorporated in new dataset releases. The reviewer acknowledges this is not the intent of the work, and it would require significant effort, but it is worth considering for future releases.
review_title: Excellent contribution and dataset for automated annotation of radiology free text reports

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: 1. The paper presents a new dataset RadGraph that consists is a dataset of entities and relation in full-text chest X-ray
    * train set consists of radiologist annotations for 500 radiology reports
    * test set consists two independent sets of radiologist annotations for 100 independent reports
2. The authors present a baseline deep learning model that achieves 0.82 and 0.73 F1 scores
3. Inference dataset generated by RadGraph benchmark model is released freely
strengths: * This paper presents a new dataset of entities and relations in full-chest X-ray radiology reports based on a new information extraction schema designed to structure radiology reports. The released datasets are annotated by board-certified radiologists. The authors also present baseline models trained on these datasets. These dataset could be important  to further research in information extraction from text for both clinicians and researchers.
* The paper also describes and provides baseline models PURE and DYGIE++ approaches with BERT. 
* The dataset and models are externally released
weaknesses: The paper is well written and checks all major boxes. below are some suggestions:
* Recently deep learning model in medical computing have shown to provide disparate performance across patient demographics. It would be great if the dataset is balanced across population groups and if not this is documented  and added as  a limitation.
* While the paper does a good job of explaining the technical aspects of the problem and the solution I think it could expand on the description of clinical significance.
* A broader impact section should be added

correctness: * The development and test dataset is prepared by collecting annotations from two radiologist ensuring the correctness of the dataset
* The schema is also revised based on radiologist feedback. It is designed to contain high coverage of clinically relevant information and to simplify the annotation task
clarity: The paper is well written and easy to follow.
relation to prior work: Related work section provides adequate information on existing words. Existing labels do not capture fine-grained information such as entities and relations. Also, some works are limited in report coverage and generalizability. The dataset claims to improve upon these works which are later demonstrated via results.
additional feedback: I recommend adding a broader impact section detailing any limitations or ethical concerns when using this dataset.
review_title: This paper presents a new dataset RadGraph that consists of entities and relations extracted from medial test. Benchmark models are training on this dataset an released for external use. These dataset and models will be very useful to the community.

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: The paper contributes an entity-relation graph of radiology reports from chest XRays. The annotations on a subset of this dataset were made by certified radiologists. This subset is used to train a neural model and annotate the rest of the dataset. Accuracy of the neural model is analyzed and is found to have an F1 in the range [0.73,0.82].
strengths: 1. The paper is very well-written and a joy to read.
2. The paper enriches existing datasets of through annotations that are trained on a smaller radiologist annotated data.
3. I like the fact that the authors clearly list out the limitations of their data annotation procedure in Section 4.5
4. Section 6.2 is insightful. It clearly explains disagreements even among radiologists and intricacies of Mimic-CXR vs. CheXpert on annotation difficulty.
weaknesses: 1. A discussion and empirical demonstration of how this annotated dataset can used for downstream healthcare tasks would have been nice. While the current case-studies are based on machine learning, it is not clear how doctors would use the dataset.

2. The authors use three relations of Suggestive Of, Located At, and Modify. Why did the authors limit themselves to only these three? Are these three exhaustive enough to cover all possible relationships in the dataset? Were these relationships decided by radiologists? 


correctness: I do not have any concerns with the claims made in the paper.
clarity: The paper is well written
relation to prior work: The paper clearly contextualizes the work and contributions made over existing work/datasets.
additional feedback: All my feedback is outlined in the strengths, weaknesses, and Documentation fields.
review_title: Review of the paper titled "RadGraph: Extracting Clinical Entities and Relations from Radiology Reports The "

Meta-review: Dear authors,

thank you very much for your submission and fruitful discussion with the reviewers.

The final score are 9/9/9/9 (with confidences 3/4/4/3) which is obviously a clear accept. 

All reviewers agreed in the importance of the data, the work and scientific rigor that went into creating and annotating the data.

The only real limitation was the limitation of US data, which cannot be changed easily. 



Decision: Accept