Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 10: Top 5% of accepted papers, seminal paper
review_content: summary and contributions: The dataset presents significant contributions and progress compared to the previous ones. It also has great potential for many applications in both healthcare insights and benchmarking of ML models. The manuscript also presents detailed steps of how the annotation was performed and its reliability.
strengths: **1. This dataset would greatly improve model explainability:** Being able to associate location and disease annotation would greatly help explain the prediction. The dataset itself would also be useful in many other downstream applications and benchmarking. \
**2. Knowledge graph:** The data structured as a scene graph makes understanding it much easier. \
**3. Creation of the gold standard dataset:** The author made an effort to create manual checking using the gold standard dataset to further validate most parts of the silver dataset. This allows users to know which bounding boxes/relationships are more reliable than others. \
**4: Demonstrated immediate applications:** The author demonstrated two useful applications of the dataset.
weaknesses: Many of the concerns have been addressed from the response to the first round review. In addition to those, a figure (main or supplementary is fine) to visually summarize some aspects and distribution of the annotations in this dataset would help readers understand the dataset quicker, especially if there is an imbalance or rare events to be aware of. In relation to the gold-standard dataset, a similar figure would help to assure that the distributions in the gold standard dataset, which is used to validate the accuracy of the silver annotation, is mostly representative of the silver one.

Two minor things that I might have missed is that it is not clear how the severity in the text was taken into account, if at all, in the annotations. The other is why is the model not able to distinguish laterality and what does it mean that the left-right information cross sentence boundaries.
correctness: The author has conducted automatic annotation of the data in a sound way also with gold-standard annotation for comparison. The limitation of the silver dataset, especially those due to NLP, was duly noted. Those that have lower accuracy are also mentioned.
clarity: The paper is well written and easy to follow. 
relation to prior work: The manuscript laid out very clearly in Table 1 the distinction of this dataset compared to past contributions.
additional feedback: All suggestions (if needed) were written right after the comments.
review_title: This would be a valuable dataset for the community and for benchmarking

final_score: rating: 10: Top 5% of accepted papers, seminal paper
review_content: summary and contributions: The authors describe a comprehensive scene graph dataset for Chest X-Rays (CXR), which address limitations of prior CXR datasets by incorporating semantic information for each exam (i.e. localized bounding boxes for relevant anatomy, sequential information from prior CXR for each patient, etc). The Chest ImaGenome Dataset is constructed by taking an existing CXR dataset (MIMIC-CXR) and using an automatic text-analysis pipeline to create a scene graph for 242,272 CXR images. The labels extracted by the automatic pipeline are validated against a “gold standard” dataset, which is manually labeled by clinicians. The dataset is released on the PhysioNet website with in-depth information about the scene graph structure stored in JSON format and iPython notebooks with example scripts.
strengths: This dataset is the first of its kind in the sense that it contains semantic diagnostic information for each CXR exam. Most other CXR databases only include an image with a label denoting the pathology in the image. With the Chest ImaGenome dataset, a machine learning researcher will be able to develop models that fuse together more complete information, much like a radiologist would when reading a case. The example use cases provided in the text are compelling, such as using the ImaGenome dataset to construct a new task of determining whether the pathology has worsened or improved since the last exam. I am excited to see what other use cases the ML community will develop with this dataset.
weaknesses: There is some information missing about how the Gold Standard dataset was collected and annotated. For example, did only one radiologist read each image? Since some of the categories contain information that is subjective to each clinician, how were differences in the reads resolved? Or did the clinicians come to a consensus? Please clarify these questions in the text.
correctness: The authors correctly state in the title, and throughout the text, that they have released a dataset (not a benchmark). The claims about the dataset are sound.
clarity: Yes, the paper is well written and presents information in an organized fashion.
relation to prior work: It is clearly discussed in the "Related Works" section how this work relates to prior publicly avaliable CXR databases and the advantages that CXR ImaGenome has over those databases.
additional feedback: -	On page 3, Methods section, first paragraph, last sentence: there is a typo. “HIPPA” should be “HIPAA”
-	On page 5, Silver Dataset section, the authors state: “we manually validated or corrected the bounding boxes for 1,071 CXR images…” Who is we? Clinicians? What level of experience? Please clarify this in the text.
review_title: The Chest ImaGenome dataset is an important step forward for developing diagnostic machine learning models that fuse together important contextual information, much like a clinician would

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The author constructed the Chest ImaGenome dataset, a large multi-model (text and image) that consists of 242,072 chest X-ray (CXR) images, with a scene graph data structure to describe the images. Local annotations are obtained using a joint rule-based natural language processing (NLP) and atlas-based bounding box detection pipeline. 
strengths: -	The dataset provides information about the change between sequential CXR exams. This information would help to develop algorithms that consider the progress of the patient.
-	Compared with other larger scale CXR dataset, this data set have the advantage of providing local annotation which is the atlas-based bounding box. This information help us to localize the CXR attributes.

weaknesses: As described in the manuscript, the dataset has some weaknesses:
-	The source is only U.S. hospital. Thus, the dataset lack of generalization.
-	The NLP and bounding box extraction have the limitation of accuracy.
-	The information of change between sequential CXR images cam is missed from the report.
The accuracy of object-object comparisons is relatively low (Table 4). I wonder if it can be used.
The atlas-based bounding box is not spatial precise enough to be used for the training detection model.

correctness: The proposed dataset is constructed in a sound way.
clarity: The paper is well-written. The dataset and experiments were clearly described.
relation to prior work: The paper compares with other datasets such as MIMIC and CheXpert. Furthermore, the author compared detailed the differences between Chest ImaGenome and Visual Genome datasets in Table 2.
additional feedback: Evaluating anatomy object detection for CXR images: the annotator would localize the bounding box based on the medical report's global “weak” label. If the annotations don’t agree with the report, how could we treat these cases?
review_title: Chest ImaGenome Dataset for Clinical Reasoning

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper contributes a Chest ImaGenome Dataset with a scene graph data structure to describe 242,072 images. Two datasets would be released as a part of this paper: Gold dataset + Silver dataset. The gold dataset is a manually constructed dataset for 500 patients. And the silver dataset has been constructed automatically by algorithms and tested for correctness on the gold dataset.
strengths: The authors do a great job of identifying the problem and addressing it through the dataset. Special shoutout to Table-1 for putting all the related work in context visually, and highlighting why the work stands out.

The dataset size is significantly large and hence would be extremely helpful to the community. Good results on the gold dataset highlight the strength of the algorithm/pipeline making the silver dataset trustworthy.

Releasing both the gold + silver datasets makes the work even stronger.

Stating the assumptions used to make the dataset is a very good practice!


weaknesses: Overall, the quality of the research is excellent, the importance of work is well understood, but the writing needs to be improved.

The paper exceeds the prescribed page limit (10 pages instead of 9). This should be taken care, and corrected!

It would be good to represent the workflow followed to prepare the data diagrammatically, especially as it involves multiple steps and gold/silver datasets.

The conclusion section is missing in the paper
correctness: The work seems to be sound and correct
clarity: Addressed in the above sections
relation to prior work: Addressed in the above sections
additional feedback: I think it would be nice to include a summary of the results of Table 4 (from PhysioNet) into the main paper. As a reader it's very important for me to know how good was the bounding box algorithm. 

UPDATE: Great job incorporating the feedback! Looking forward to the camera-ready version
review_title: Review of "Chest ImaGenome Dataset for Clinical Reasoning"

Meta-review: This paper presents a dataset of chest X-Ray images with VisualGenome-style scene graph annotations. All reviewers acknowledge the value and significance of the dataset and gave positive review scores, two of them are 10. Although the dataset mainly consists of silver (automatic) annotation, it also incorporates a portion of gold annotation to discuss the quality of the former, which is highly appreciated. Overall, this is a very good paper that will significantly contribute to promote researches toward high-quality medical image diagnosis in an explainable manner.
Decision: Accept