Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper presents a dataset of quantitative MRI with knee scans, images, and dense labels to create a localized tissue-wise qMRI biomarker maps benchmarking of the MRI reconstruction. The paper benchmarks MRI reconstruction and segmentation models using both traditional and new deep learning image and label metrics. 
strengths: * The paper is well-motivated with clinical contributions. The collection of large-scale labelled knee segmentation and reconstruction data is difficult and as a result benchmarking would benefit corresponding clinical applications.

* The authors divide their evaluation methods into Raw Data Benchmarking Track and the DICOM Benchmarking track.

* The data collection and pre-processing is explained. 

* The authors report accuracies over different knee segments which is helpful in potentially understanding of potential diseases.
weaknesses: The model is used is U-Net which has proven to show great results for segmentations and reconstruction. Currently there are better models nn-Unet and residual based U-Net and analyses on such models is missing. While the dataset produced is large, its functionality remains undetermined if results are not compared with those of other knee datasets.
correctness: Yes, the claims made in the submission are correct. The evaluation methods and experiment design are appropriate.
clarity: The paper is well written.
relation to prior work: Prior works have been well discussed. 
additional feedback: It would be better to provide comparison with existing benchmarks and newer models.
review_title: Review

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: The authors propose a new dataset for quantitative MRI scans, the corresponding scanner-generated DICOM images, segmentations of tissues, and bounding box annotations for sixteen clinically relevant pathologies. They also provide a benchmark evaluation method for an accurate and consistent metric to measure tissue health. finally they extensively test the dataset for various tasks.

strengths: 1. The dataset provides a large subset of patients that can be very useful in clinical research and is useful for various clinical research areas (segmentation, reconstruction, classification, detection). 
2. An extensive evaluation of the dataset was carried out by the authors and the results are clearly analyzed.

weaknesses: 1. The data is missing information on the annotators:
    a) How the annotators were selected 
    b) The ground truth criteria the annotators used 
    c) How disagreements in annotations were resolved. 
2. The authors also failed to provide Information on how the dataset will be accessed in future. 

correctness: Although the paper was very well written and the motivation clearly described in the paper, The reviewer failed to see a detailed description on how the dataset was constructed. Therefore it is difficult to determine the overall accuracy of the dataset.
clarity: The paper was very well written, clearly organized and easy to follow
relation to prior work: The authors clearly described previous works and how their work differs from them
additional feedback: Overall this is a very well written paper. the motivation for the paper is clear and the reviewer believes it is a significant contribution, however, the reviewer has a few concerns on the annotators of the dataset (i.e. how the annotators were selected, The ground truth criteria the annotators used, and how disagreements in annotations were resolved). Furthermore, more information is needed on how the dataset will be accessed in the future, and the plans for maintaining the dataset.

EDIT. Thank you for the response, the review has been updated
review_title: Review of SKM-TEA

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper provides a multi-task knee MRI dataset of 155 patients. This includes the 'full resolution' k-space data, the scanner and SENSE reconstructions, the quantitative (T2) maps, as well as dense segmentations of some anatomies as well as bounding box annotations of some anomalies. Therefore, the paper boasts of a wide gamut of data types from different parts of the MR reconstruction and analysis pipeline. Additionally, thanks to the availability of the parameter maps, this work proposes an evaluation protocol closer to clinical usage, instead of the traditional reconstruction and analysis measures.        
strengths: 1. The multi-task nature of the dataset. As highlighted in the paper, presence of k-space data along with clinical labels has the potential to open up numerous interesting research questions.

2. The care given to differentiating scanner-based reconstructions (using proprietary software) and the SENSE reconstructions, and cautioning the reader on their corresponding usage is admirable.

3. The paper is very well written and easy to follow.  
weaknesses: 1. The size of the dataset (155 patients), with scans from a single centre and manufacturer still makes trustworthy clinical translation difficult. To be fair, the authors acknowledge this in the conclusions. However, a more solid claim is needed to ensure that the dataset will actually evolve over time, given its potential.  

2. The benchmarking over the subset of tasks seems rushed. Why were only a subset of tasks benchmarked? Benchmarking classification/anomaly-detection would have been a more clinical relevant step. Also, there exist cutting edge techniques for k-space to image reconstruction. An explanation is warranted for why only U-Net and unrolled networks were used. Also, no intuitions or learnings are provided based on the observed performances.

3. Information on annotation procedure and inter-rater variability is missing. This could answer the question: How reliable are the segmentation masks or the bounding boxes or even the diagnoses? 
correctness: Yes, the dataset is constructed properly. Why was this split chosen? Was it random? Based on the provided table in the supplement and the well-written text, it is clear to deduce the experimental design. However, an improved version of Table 6 (supplementary) mentioning the actual 'Tasks' for which the dataset can be used (incl. but not exclusively, of course) would be very helpful.
clarity: Yes, the paper is very well-written.
relation to prior work: Table 1 and Section 3 do the job. Why not have actual dataset sizes in Table 1? Is it hard to determine for the other datasets? 
additional feedback: In my opinion, the work has immense potential. In its current form, even though the dataset is complete, the benchmarking seems incomplete. The latter not being the primary focus of the dataset track is why I'm recommending an 'accept'. However, a complete benchmark plus the dataset would make this work very strong.
review_title: Good dataset covering data from k-space to quantitative biomarkers. Has potential.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors open source a new quantitative MRI dataset along with baseline benchmarks and relevant code. They provided 150 patients with k-space raw data; the scanner reconstructed DICOMs, manually annotated segmentation, and the corresponding bounding boxes. The clear differentiation of this dataset is that it provides for a single patient all the different steps of the MRI imaging modality from acquisition to pathology analysis. 

The paper is well written with a clear motivation for the dataset creation, highlighting the different use cases and reporting on a few example baseline models for benchmarking the use cases. As such, I don't have any major concerns about the paper. It reads well. I only list a few minor typos and clarification comments. 
strengths: - Comprehensive and meaningful dataset, which helps facilitate the development of AI-based MR reconstruction and quantification of pathologies. 
- Well written paper with clear documentation of the dataset description, acquisition protocols, use cases, and example results. 
- Bias towards developing quantitative T2 relaxation-based quantification, a clinically relevant biomarker for detecting an early onset of OA.
- Code to replicate the baseline results. 

weaknesses: The authors themselves list the weakness of the paper and do a good job of mentioning how in future work will address each of the weaknesses. 
- The limited size of the cohort and its scope limited to just a particular region, hospital, and scanner type. 




correctness: Yes 
clarity: The paper is very well written clearly highlighting all aspects of the data collection usages and its limitations. Further, the suggested use cases and baseline benchmarks and code provide interested users a good starting point to innovate. 
relation to prior work: The authors clearly compare with the existing datasets for knee MRI and highlight what their differentiation is. 
additional feedback: A few minor typos and clarifications: 
- Lines 49-50  exists no large-scale public datasets that include both the raw MRI data **along and** multiple clinically- relevant observations for each image.
- Lines 113-114 Classification/Detection: The MRNet and the Osteoarthritis Initiative datasets provide labels for classification of knee MRI abnormalities - Table 1 shows classification crossed out. Please correct the discrepancy. 
- Line 205 reconstruction tasks in clinically pertinent regions. When using these labels in the context of **the this**
- Clarification - For the reconstruction track to validate the T2 relaxation, I assume the sub-regions quantification would be based on the ground truth. 
- Section 6.4 Raw Data Track – Reconstruction - It's interesting that some of the experiments (Unrolled (E1E2)) had higher accuracy and lower variance in the 8x  than the 6x. It is surprising that a more aggressive sampling by a factor of 2x produces a higher accuracy. Do the authors have a hypothesis as to why that is happening? It would be a useful addition to the paper.
review_title: Knee MRI dataset for end-to-end clinically meaningful AI development 

Meta-review: The reviewers all liked the paper. The authors' response clarified some important points. In view of that, the authors are strongly invited to take the feedback on board for the final version.
Decision: Accept