Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: The authors present a large, diverse dataset of abdominal CT and MRI scans with pixel-level annotations of 15 organs. The paper comprehensively explains the data collection procedures and the dataset's limitations, present their use cases, and provides baseline experiments for the community to build upon. 

Overall this is an excellent resource for further developing CAD for multi-organ abdominal segmentation. 
strengths: - A large, diverse dataset for a crucially important CAD problem in medical imaging. 
- Clear experiments and comparisons explain AMOS's benefits over the existing open source abdominal datasets. 
- they provide clear use cases for their dataset and how they envision the community can use it. 
-they provide strong baselines and plan to release the code making this dataset easier to use. They have posed this dataset as a MICCAI challenge, which will also help spread the use in the community. 
weaknesses: The authors have captured the restrictions of this work pretty well.  
- As mentioned in the paper, it is a subset of the Asian population with data from two hospital systems with similar protocols for multiple scanners. 

correctness: Yes, the claims in the submission are correct and sound with reasonable baselines and benchmarks. 
clarity: Yes, the paper is well written. 
relation to prior work: Yes, the paper has good evaluations on comparison with other datasets and motivation for creating this dataset. 
additional feedback: Minor comments: 
- the authors decided not to include normals and focus only on diseased cases. Having normals would also have been beneficial, but that is in no way discounting the value of this dataset. 
- the minor typos in the paper, which a good revision can quickly fix. 
- On lines 188-189: The authors mention they pre-trained the models between AMOS and BTCV and tested them on the others. Did they further fine-tune the algorithm using the training set of the other datasets or use the pre-trained models as is? Can you clarify? 
- On lines 322-323: Can the authors clarify what they mean by solving privacy and social effects by obtaining the research ethics committee approval? It is unclear what social and privacy issues the committee helped solve.
review_title: AMOS dataset for abdominal segmentation from CT and MRI scans. 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents AMOS a large scale abdominal multi-organ dataset from diverse clinical scenarios to serve as a benchmark to study robust segmentation algorithms. The dataset is comprehensively annotated with semi-automatic annotation workflow along with clinical experts to reduce the bias. Authors establish the diversity and scale of the dataset by comparing AMOS with existing abdominal datasets along the different parameters: organs, modality, scans, slices and annotations per scan. Further, few experiments were setup to benchmark several state-of-the-art medical segmentation models and evaluate them on existing abdominal datasets. The paper also focuses on out of distribution model generalization, transfer learning and cross modality learning, further substantiate it with experiment results. The authors also elucidate the data curation process and address its limitations.
strengths: -The dataset presented by the authors contains the maximum annotations per scan as stated in *Table 1* , which is a significantly larger and at the same diverse as well (with reference to *Figure 4*) than the previous datasets, thus can serve a good benchmark in future

-The authors have meticulously conducted experiments across different state-of-the-art models

-The dataset presented by the authors shows potential in transfer learning and cross-modality learning a fair setup



weaknesses: * With reference to page number 6, Table 3, the authors left some important multi-organ dataset namely *AbdomentCT-1K* dataset for benchmarking, which leaves the comparison somewhat incomplete, does that mean a pre-trained model on *AbdomentCT-1K* 
outperforms over pre-trained *AMOS* or the performance gain is marginal compared ?


* As stated on line number *257*:\
         *The results demonstrate that there is indeed a data distribution shift caused by collecting MRI samples from different scanners that 
         hinders the model from performing consistently on two sub-sets*\
         The authors could have conducted a few experiments specifically comparing AMOS-MRI with other MRI datasets rather than stating 
         Out of Distribution (OOD) as stated in Table 6. Further to tackle this ood, do authors  intend to add more data in future or have any plan to mitigate the OOD?


 
correctness: Please correct the Appendix B.5 to Appendix B.4 throughout the paper 

**Misc:**
\
W.r.t Appendix,On line number 573:
*normalize truncated voxels values by subtracting 50 and dividing by 141*

Any specific reason why did you chose these values for normalization ?
clarity: The paper clearly explains the data acquisition and annotation procedure. It also highlights the vital dataset statistics and experiment setup to benchmark other SOTA models.

Overall well articulated.


relation to prior work: 
The authors provide adequate context along with vital statistics among abdominal datasets to substantiate the difference across various parameters. Also, the authors have highlighted all the distinguishing features of this work in relation to prior art. 


additional feedback: With reference to line number 115-117

*"To meet this purpose, we cautiously select data, generated
116 by two institutes from 2018 to 2021, based on the following criteria: 1) Patients should be diagnosed
117 with abdominal tumors/abnormalities, while the ones with normal abdomen will be excluded"*

It would be great if you can include the data with normal abdomen to provide even more diversity and generalization for transfer learning tasks
review_title: Abdominal Multi-Organ Dataset for Medical Image Segmentation

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: The paper contributes a publicly available dataset of CT and MRI Image and segmentation pairs with 15 different organs segmented for all of the images. There are 100 MRI and 500 CT scans in total. This contributes to fulfil an important gap where various datasets of single organs are available individually but multiple organ segmentation are not available in the amount contributed by this paper. This helps in training data hungry deep learning algorithms for segmenting organ boundaries in abdomen region which can be useful for reducing manual segmentation effort. 
Benchmarking of few representative baselines on segmentation tasks, transfer learning, cross-modality learning etc. are also done.  
strengths: As stated earlier, the size of the dataset and the number of organs segmented opens avenue for designing and validating methods for multiple organ segmentation taking their relative position and shape/size into account. Multivendor scans with varied diseases conditions adds to the challenge of learning and helps in benchmarking robust algorithms. 

Consultation with multiple senior radiologists when finalizing the segmentation boundary is a big plus. Many of the datasets that involve only a single domain expert, for example some mentioned in this paper, may inherit the bias of the expert, which may then be learnt/amplified by the learning algorithms. 

Publicly available leader board as hosted in grand-challenge.org is useful in tracking the state-of-the-art. 
weaknesses: Not necessarily a weakness, but the use of default parameters for training many of the baselines (no hyperparameter tuning), might hinder making a rigorous statement about the experiment results. 
correctness: Yes, the dataset is a very useful contribution in terms of number of organs segmented and the number of samples on which the full spectrum of organs is segmented.  Required caution has been made in making the dataset bias free and preserving anonimity.
clarity: Yes, the major points come through clearly. There are few typos, redundancies, word choice and omissions. 
relation to prior work: Yes, relationship with various other public multi-organ, multi-modal and uni-modal datasets are discussed. Network architectures in the literature related to multi-organ(and partial organ) segmentation are also discussed.
additional feedback: Line 184 mentions Fig 4. But fig 4 is in supplementary text. Perhaps, it is better to indicate that the figure is in the relevant appendix section (not in the main text).  

Section 3.3 Evaluation Metrics. Please mention at what tolerance level was the NSD calculated?  

Line 248 What do you mean “we have also observed a significant gap between mDice and mNSD as previously mentioned by [15]?” Perhaps, concrete examples of the difference (for which organs, these metrics were significantly different?)  and interpretation of what that means. 

 Line 226-227 “All models are optimized by the empirical risk minimization (ERM), which trains the model to minimize the average training loss.” This sentence may be redundant. It is a given. 

### Typo and Grammar: 

Line 25 Nerveless -> Nevertheless  
Table 7 performacne -> performance 

Line 144 Training data can **differs** the testing data -> training data can **differ from** the testing data 

Line 157 Perhaps, Data Format is a better paragraph title here rather than Data Distribution. 

Line 188 and its **closed datasets,** BTCV -> and its **closest dataset,** BTCN  

Line 268 Since AMOS **owns** more modalities -> Since AMOS **contains** more modalities 

Please check for any additional typos in the draft.
review_title: Very useful contribution to Medical Imaging Community 

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper proposed a large-scale abdominal organ segmentation dataset with transparent data collection, annotation, split, and distribution details. Furthermore, the data in AMOS are collected from multi-center, multi-vendor, multi-modality, multi-phase, multi-disease patients, each with voxel-level annotations of 15 abdominal organs. The dataset construction is transparent and reasonable enough. Further, the authors utilized different types of baseline methods for dataset evaluation. 
strengths: Compared with other existing abdomen organ datasets, AMOS contains a larger number of scans. Further, AMOS provides abundant annotations of multi-modality, multi-center, multi-scanner, multi-phase, and multi-disease patients, covering 15 organs. Further, the dataset is comprehensively evaluated by different types of baseline methods, including the CNN-based methods, Transformer-based methods, and hybrid ones. This comprehensive experiment setting strongly proved the generalization capability of the dataset.
weaknesses: The quality of the dataset and paper content is excellent. More detailed information could be clarified in the paper.  For example, during the annotation review by humans, due to the decision making being subjective to each radiologist, is there any way to deal with the differences between different radiologists? Could the paper release the standards for decision making between different radiologists.

correctness: The dataset is constructed in a sound and correct way.

clarity: Yes, the paper is well written and presents content in an organized way.

relation to prior work: It is clearly discussed in the ‘Related Works’ section how this work relates to prior publicly available datasets and the previous methods for abdominal single- and multi-organ segmentation.
additional feedback: The paper is very well written. Here is a minor remarks about typo:

In 3.1 Dataset Contribution → Data Overview 

“Following the standard clinical acquisition protocols, … at SLonggang District 109 People’s Hospital and Longgang District People’s Hospital,”

It seems “S” incorrectly occurs here? Should it be “Longgang District” rather than “SLonggang District”?
review_title: A valuable and well-constructed dataset for the community and for benchmarking

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors introduce a dataset for the segmentation of multiple organs in the abdomen, from real-world clinical 3D CT and MRI scans. The authors name their dataset AMOS, and it is meant to address two limitations in previous benchmark datasets: small-scale and lack of diversity. AMOS contains 500 CT images and 100 MRIs, each containing segmentations for 15 organs. The data come from two centers and eight scanners and include several different diseases related to abdominal organs. Importantly, healthy abdominal scans are not included in this dataset. The authors also conduct several experiments that evaluate the dataset in relation to previous datasets and benchmark several segmentation models on the AMOS dataset. This is impressive work that must have required close collaboration between computer scientists and clinicians -- kudos on the great interdisciplinary work!
strengths: The AMOS dataset has many strengths. Most of all, the dataset is diverse. It includes 5 CT scanners, 3 MRI scanners, and 2 institutions. This is tremendous strength of AMOS, because it can help expose overfitting or lack of generalization in models (the authors emphasize this in the section "Data Splits"). In addition, this is a dataset from Chinese hospitals (that are presumably serving mostly Chinese patients), so this will add much-needed diversity to the field.
weaknesses: This work overall has few weaknesses in my opinion, but there is one very important weakness: dataset availability and accessibility. The dataset is currently available via the Grand Challenge website, after also submitting a Google form and being accepted as a user. But are there any expectations or guarantees for this the AMOS22 challenge site to remain available to the public? The authors should consider a different way of sharing the data -- one that does not require an account on the Grand Challenge website. In addition, potential users of the data need to submit a form to access the data. Can the authors clarify what would disqualify someone from gaining access to the data? If at all possible, I would highly encourage the authors to consider making their data available on a platform like Zenodo.

Another weakness is that the appendix seems incomplete. Only sections A.1 and A.2 are included (and A.1 is empty). The table of contents of the appendix suggests that it includes important details about the dataset, including about acquisition and annotation.

Last, no code repository is included with this submission. The code to conduct the benchmarking experiments would foster reproducibility. _Without this code, the manuscript is not reproducible._
correctness: The dataset appears to be constructed in a sound way, and the benchmark experiments seem appropriate.
clarity: Yes, overall the paper is very well written. I have a few specific comments about clarity:
1. I do not understand the phrase on lines 67-68: "with data volumes ranging from tens to hundreds." Please clarify this in the text.
2. The authors frequently reference the "mDice" metric. How does this differ from the Dice coefficient? Please clarify in the text.
relation to prior work: Yes, the authors nicely summarize previous work and describe the novelty of their work. However, the paragraph titled "Methods for abdominal multi-organ segmentation" should almost certainly include a reference to the U-Net. In addition, the authors write (lines 98-100) that "However, we empirically found that developing and validating methods with limited data still hinders the potential power of modern deep models, leading to unfair/inaccurate comparisons and estimates of different methods", but it is unclear whether they are referencing this work or a previous work of theirs. Can the authors please clarify?
additional feedback: There should be a citation for lines 31-33: "The tremendous cost also restricts previous works to only acquiring data samples from a single-center, phase, scanner, and disease. Models trained with homogeneous and inadequate datasets tend to perform catastrophically when the test data distribution shifts under different clinical scenarios."

In lines 75-77, the authors write "Unlike the above works, our AMOS contains 600 CT/MRI scans with 15 types of organ annotation, making it the most comprehensive and large-scale benchmark of its kind to date." In the previous sentence, however, the authors mention the AbdomenCT-1K dataset, which includes 1,112 scans. I am under the impression that with the phrase "most comprehensive and large-scale benchmark of its kind to date", the authors are implying that their dataset is the largest of its kind to date. Can the authors please rephrase this to make it clear that it is not the largest --- but it does seem to be the most comprehensive in terms of number of organs annotated. And it is indeed a good sized dataset, especially considering the size of related datasets.

The authors reference the U-Net and nnUNet in "Methods for single abdominal organ segmentation." It is true that the U-Net can be used for single-organ segmentation, but it can also be used for multi-organ segmentation. The authors should clarify this in this section.

In the paragraph on "benchmark results", the authors mention that segmentations of small organs are still relatively poor, and the authors highlight a need to improve this. The authors might consider referencing methods that are designed to address segmenting small objects, like the Tversky loss or Focal loss of FocalTversky loss.

Below, I included a few minor comments.

Spelling / word choice comments:
- line 20: "computer-aided diagnosis applications" could probably be "computer-aided clinical applications"
- line 25: "Nerveless" should probably be "Nevertheless" (?)
- lines 93-94: "a bunch of works" seems too casual for a scientific manuscript.
- line 188: should "closed" be "closest"?
- line 205: "better score" should probably be "higher score"
review_title: A multi-center, multi-scanner dataset of 600 CT and MRI scans, with labels for 15 organs

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This work presents AMOS, a large-scale dataset that includes segmentation masks for 15 different organs for 500 CT scans and 100 MRIs. In addition to the large scale, AMOS is also relatively diverse, including medical images generated from 8 different scanners from 2 different hospitals in China. Such a heterogeneous dataset can promote algorithmic robustness and out-of-distribution study, which are important topics for medical images. 
strengths: - Scale: the dataset is currently the largest abdominal segmentation dataset available. Additionally, AMOS provides segmentation masks for the most number of organs. 
- Diversity: Most abdominal organ segmentation datasets available are curated from hospitals in the US or Europe. AMOS improves the diversity of current datasets by providing images from the Asian population. 
- The dataset also includes scanner information (form 8 scanners) and provides an out-of-distribution test set, which allows robustness and out-of-distribution studies. 
- The authors provide a suite of strong baselines to benchmark their dataset, including convolution and transformer-based models.
- The Transfer learning experiment (Table 3) future demonstrates the value of AMOS for medical image segmentation.  
- All figures and tables are well made. The animated figure (Figure 1) is a nice touch. 

weaknesses: - The manuscript has a truncated Appendix, which left out a lot of crucial implementation and dataset details. 
- To better test for robustness and fairness, the dataset should include more metadata such as patient demographics. The accompanying reports can also be really useful for the research community. 
- Unclear why only diseased patients are included, especially since no disease label or diseased level segmentations are provided. These disease level labels are tremendously helpful for the medical imaging research community. 
- More implementation and data processing details are needed. For instance, CT scans are known to be sensitive to the viewing window level and window width. Such information should be included to ensure reproducibility. 
- No Github link is provided. Thus reproducibility is further limited.
- The authors use “default training and testing configurations” for the baseline method. This isn't the optimal approach since neural networks are known to be sensitive to hyperparameters. Thus more extensive hyperparameter search is necessary. 
- Table 6 compares models trained from scratch with models pretrained with AMOS. A more comprehensive comparison should include models that are imagenet pretrained. 
- The authors write in section 4.3 that “the CT samples collected by different scanners in our work are almost indistinguishable”, and proved their point using results from table 5. However, this contradicts their claim of having a diverse dataset that can be tested or OOD by providing scanner information. 

correctness: The data collection is constructed nicely, however, the authors should specify why only diseased patients are included. The evaluation methods are comprehensive but the authors should perform more hyperparameter tuning. 
clarity: Writing 
- The paper is organized well. However, typographical and grammatical errors should be corrected.
- Certain wording and sentences are hard to understand without further clarification. For example, L182 - what are slices spacing and size range? Table 4 caption - what is the [1x128x128x128] feature map?


Tables/Figures
- Please bold best performances in all tables
- Table 7 comes before table 6, the labels should be flipped

Dataset checklist: 
- Where exactly is the potential negative societal impact provided?
- Paper did not provide codebase 

Appendix
- Missing text for A.1 
- Missing everything after A.2 

relation to prior work: The paper provides a comprehensive review of prior work (summarized in Table 1), and showed improvement in scale and diversity. The authors also included strong baselines from multiple representative segmentation models. 
additional feedback: N/A
review_title: This work presents a large-scale multi-label abdominal segmentation dataset. While the dataset is useful, some crucial information is missing, which hinders reproducibility and better understanding of this dataset. 

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: The authors constructed an abdominal database for multi-organ segmentation with 660 CT/MRI exams and 74.000 slides from 8 distinct CT and MRI scanners in two Hong Kong centers. 
strengths: To date, this is the largest such database that is being shared with the research community. The project provides a complete annotation workflow for the labeling process. A number of baseline models were developed to evaluate the dataset.

weaknesses: Labels provided are sparse. There are no sociodemographic features that will enable interrogation for algorithm bias of downstream models. The absence of information on comorbidities is also a limitation of the dataset. 
correctness: It is misleading to characterize the database as "diverse" given that sociodemographic labels and comorbidities are missing. But the dataset was meticulously constructed.
clarity: Submission is well-written. However, there are a few questions that need further elaboration:
●  	How is the performance in the first stage of annotation using a segmentation model trained with a small number of images?
●  	Was de-identification performed on the images?
relation to prior work: Comparison to previous related work on medical image segmentation is comprehensive.
additional feedback: Appendix B.5 is missing.
review_title: AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper provides a comprehensive dataset for abdominal multi-organ image segmentation. The dataset is supported with statistical analysis and experimentation to evaluate its suitability over existing datasets. The paper is well written and provides updated relevant literature. The authors have given necessary details about dataset construction, data evaluation and splitting. This dataset will help research in the field of abdominal multi-organ segmentation. It adds a variety of images to the existing data. Many state-of-the-art methods are used to benchmark the dataset.
strengths: Wide variety in data collection. Labels for a large number of organs. Large size of the dataset. Benchmarking with state-of-the-art methods. Ablative study and dataset exploration.
weaknesses: More experiments are required. Dataset marginally stands out in comparison to existing datasets with the present set of experiments.
correctness: Although many experiments have been conducted on this dataset, however, it is hard to conclude the superiority/hardness of this dataset over other existing ones. In this regard, authors may have shown more results involving other methods than UNet and conducted the same experiments (Table 3, 5, 6, 7) over other different datasets or under the reversed setting.

I believe that this paper will be a valuable contribution to the field, and I recommend its acceptance with some clarifications. Below I have a few questions and comments for the authors.

- How is the second challenge about lack-of-diversity addressed? And is there any effect of these diverse features (of the dataset) on the performance of models? It should be quantified through experiments.
- Table 1 may also have a column indicating the diversity available in the datasets. Here, authors should also mention the class imbalance in the dataset.
- Table 3 presents an interesting transfer learning analysis. Please tell why other methods ([6],[16]) were not used than UNet.
- In Table 1, why are MSD datasets written separately per organ? Isn't it one dataset and the largest dataset?
- In Table 3, why are results on MSD dataset better than AMOS? Why is the performance of models pretrained on MSD dataset not shown for AMOS and BTCV?
- In Table 7, I suspect that what authors call OOD is just the test split and not truly the OOD data since the test and train distributions are similar. This may be the reason behind better performance on OOD data than ID for CT images in Table 7. If that is the case, authors should conduct pure OOD experiments. A more stringent setting could be testing on AbdomentCT-1K, Chaos, Word, or DenseVNet.
- From Table 8, can we conclude that we don't need a very high number of CT images to train the network? As the performance of (10CT/30MRI) is at par with (200CT/40MRI). What do these results indicate about the ground-truth data deficiency problem? Based on results from Table 8, I think it is not very conclusive to say how large of a dataset we need to overcome ground-truth deficiency problem?
- Authors may write a small para about: How feasible is the extension of this dataset? What points need to be considered while extending it?
- Further experiments should be conducted to show transferability results with methods [6] and [16], in Table 6.
- Please check the order of tables. Table 6 should not appear after table 7.
clarity: Yes
relation to prior work: Yes, authors have covered sufficient depth and width of recent literature.
additional feedback: None
review_title: This paper proposes a large-scale abdominal multi-organ dataset under versatile scanning conditions for medical image segmentation. This dataset may overcome the ongoing ground-truth data deficiency problem for training large-scale models on diverse clinical scenarios. Each instance in the dataset is labeled with voxel-level annotations of 15 abdominal organs. Experiments were conducted for cross-modality and transferability of the dataset.

Meta-review: metareview: The authors have done a good job in addressing the review comments and all the 8 reviewers are positively in support of accepting this work. I would like to reiterate to authors to please check if their data collection follows standard ethical guidelines of medical data collection and release One example of guideline can be - EU guideline - https://ec.europa.eu/info/funding-tenders/opportunities/docs/2021-2027/horizon/guidance/ethics-and-data-protection_he_en.pdf
confidence: 5: The area chair is absolutely certain
Decision: Accept