Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: In this paper, the authors collect and present a new ordinal classification dataset for benchmarking mammographic masking of cancer. The dataset (named CSAW-M) is collected from 10,020 patients from Sweden and is the current largest public mammographic dataset with potential masking annotation. The authors also trained baseline deep learning models for evaluation and showed that the estimated masking is much more productive of women diagnosed with interval and large invasive cancers, than its breast density counterparts. The main contribution of this paper is the proposed dataset. Considering its large scale and the novelty in the annotation and its clinical value, it would benefit the research in the community.

I have read the rebuttal from the authors and for more details please see my response below.
strengths: + The scale of the proposed dataset is very large, compared to prior datasets, which could be beneficial for the research community.

+ The annotation (8 levels of masking potential) of the proposed dataset is novel and has real clinical value. The ordinal labels could also be used beyond the studied mammographic problem, e.g. benchmarking other ordinal classification models.

+ The data collection part of the dataset is well presented with sufficient details.

+ In addition to the proposed dataset, the authors also perform evaluations with baseline models on this dataset. The evaluation validates the strength of the proposed dataset with its novel annotation and also provides baseline models for following-up research.

+ The dataset is publically available for non-commercial use.

+ The proposed dataset has potential positive ethical and social implications, e.g. help with the automatic analysis of mammograms and research in breast cancer.
weaknesses: - It is a bit unclear how the dataset is hosted at the SciLifeLab. Besides, the mentioned "controlled, less biased setting" (L58) lacks details. It would also be better to provide more details of the "planned service" (L59).

- Although the collected dataset is quite large, it was collected only from one region (three centers of the Stockholm region). This could possibly introduce bias and there should have been a discussion on this.

- The collected data is from the age range of 40-74 (L90), but it is unclear how does this compare to existing datasets. It would be better to summarize and compare this as well in Table 1.

- Although more details are provided in the Appendix, the main paper lacks sufficient details (e.g. the preprocessing part) of how to construct the dataset.

- The images in the training set are only one expert annotation per image and the distribution of the expert experience for the whole training set is not provided. I know this is a large-scale dataset and it is costing to have multiple experts annotate each image, but only one expert annotation per image would inevitably introduce bias and affected by the experience of the expert. It would be much better if 1-2 more experts could be included for the annotation of each image. Or otherwise, an annotation distribution (analysis) in terms of the expert experience should be provided.

- In the test set, the "ground truth" is computed as the "median of the annotation". This is a bit inappropriate, especially considering the experience of the experts ranges from 2 years to 30 years.

- It is unclear why the masking level is defined to be eight and why has to be such an even number. The current description (L142-145) is a bit unconvincing.

- It is not mentioned how the proposed dataset will be used ethically and responsibly.
correctness: The proposed dataset is generally constructed in a sound way. My only concern is the number of experts used to annotate each image and the definition of the masking level.
clarity: The paper is well written and easy to follow.
relation to prior work: The authors presented a discussion on how this work differs from previous contributions, though more details could be provided as mentioned above, e.g. the comparison on the age range and other metrics.
additional feedback: * It would be better to move some content regarding the data processing and annotation analysis from the appendix to the main paper if possible, or clearly mention in the main paper what information is provided in the appendix.
review_title: Comments from reviewer (updated 19 Jul.)

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: Here, the authors provide a new dataset of mammographic images, which focuses on masking potential as annotated by specialists. Their dataset provides several contributions over previous public mammography datasets: in addition to containing 1-2 orders of magnitude more images, they also focus on providing gold-standard masking annotations over pixel-level cancer ROIs.
strengths: I find the data to be well-organized with easily understandable annotations and an abundance of metadata. Overall, this dataset is a valuable resource that could potentially establish new lines of research, both in studying masking as a risk factor for mammography-specific applications, and in developing ordinal methods in machine learning. The manuscript is well-written, and explains their labeling procedure in detail. 
weaknesses: One weakness is the limitations that the authors had with labeling the images: the training dataset images are each only annotated by one expert, as opposed to the test datasets that are a median of five experts. However, this is an understandable limitation given the nature of this dataset: labels from medical experts are just more expensive and difficult to procure than tasks that can be conducted by laypeople. I appreciate the authors outlining their logic in how they distributed their resources in light of these limitations, and it is evident that extensive thought has gone into making the best of a limited situation - I doubt many other medical image datasets of similar scale would be more thoroughly labeled than this one. 

The main problem that I noticed with this dataset comes from browsing the data directly: 

1. I noticed that a substantial portion of the images contain text at the top corner, presumably from the imaging instrument. I'm concerned that this text could be a confounder especially for deep learning models that may learn to memorize this text as a shortcut over actually examining the imaged breast. Could the authors provide a better explanation of what this text is, and if they expect any correlation with the text and masking or cancer? Does the text provide any potential of leakage of information or shortcuts in predicting the test datasets? Are there any best practices for preprocessing out this text, if necessary? 

I think this is an issue that could potentially reduce the utility of this dataset for machine learning purposes, so I have given the paper a weak accept for now, but I am happy to improve my score if the authors can satisfactorily address this point. 

I also have some additional thoughts on how the characterization of the dataset could be improved (but are non-essential), especially as it could better describe the exact nature of the ordinal problem here, that I will detail in the "Feedback to Authors" section. 
correctness: Yes, the dataset is constructed in a sound way. As I discussed in the weaknesses, there are resource limitations in the collection of labels like these; the authors have proposed, in my opinion, sound principles in soliciting standardized labels from medical experts in light of these limitations. 
clarity: Yes - I appreciate the authors' clear writing. 
relation to prior work: Yes - similar mammography datasets are cited, and the authors make a clear distinction between their work and these datasets. 
additional feedback: 1. It looks like about half of the experts attempted to bin the all images evenly, while others have a more normal distribution to their labeling. Since there is still a heavy tail in the distribution of images even when undersampling their data, I expect the way this would interact with the classes is that for the experts that binned images evenly, you'd see a higher range of variability in the images of label 8, than in the images of label 3, for example. I'm less clear about what the intra-class distribution of variability would look like with the median labels: could the authors directly test this (e.g. one way I can think of is by looking at how percent identity distributes across each of the 8 ordinal labels, understanding this is an imperfect proxy)? I feel like this would be important to characterizing the exact nature of the ordinal challenge here: my feeling is that with the even binning strategy, there may be more overlap or less of a gap between most images of label 3 versus label 4, where there'd be a bigger difference between 7 and 8. (If the authors want to go the bonus mile, I would like to see how this interacts with the metrics for agreement between reviewers in Table 3 - are there certain pairs of adjacent classes where the disagreement is concentrated, or is it evenly distributed? And do the models mirror this confusion? The F1-scores provide some insight into this, but not directly.)

2. Could the authors clarify if identifying as a woman was an inclusion criterion for their images or for screening at the hospital site? If not, I would suggest that the authors use more accurate and inclusive nomenclature than "women" in their manuscript (e.g. "participants", "patients", or "people at risk for breast cancer"), given that other people including trans men and nonbinary people also receive mammograms.
review_title: Review of "CSAW-M: An Ordinal Classification Dataset for Benchmarking Mammographic Masking of Cancer"

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper presents a new mammographic dataset called CSAW-M which happens to be a subset of CSAW, a large mammogram dataset.   CSAW-M contains data from a whopping 10,000+ individuals aged between 40 and 74 together with masks manually outlined  by 5 specialists.  As mentioned early in the introduction, masking is a phenomenon in which a tumor is hidden by surrounding [dense] tissues and thus more likely to be missed.  Images were also graded according to 8 level of masking.   The dataset is divided in 3 subsets, namely a public train, a public test and a private test dataset.  The data is stored on an online repo called "SciLifeLab".

The authors also trained 2 generic ResNet models and report results within the inter-expert variability (especially for the ordinal classification task).


strengths: This proposition has many strengths:

* First and foremost, it is VERY large.  With 10,000+ patients, this dataset is the largest of its kind and thus has the potential of having a bold impact on the community.
* Annotations from multiple experts give a lot of credibility to the groundtruth.  It also provides inter-expert variability intervals.
* The dataset is well balanced and the images were selected via a sound evaluation protocol summarized by a flowchart provided in the appendix.
* source code for the annotation tool and baseline models have been rendered public
*  Considering that breast cancer is one of the most prominent cancer but whose survival rate is high when diagnosed early, it is easy to understand that better masking prediction will lead to more accurate early diagnostics and thus better survival rates.
*  The manual annotation procedure is thorough and explained in great details.






weaknesses: In my view, this work has two main limitations:

* that dataset is not really opened as it can only be used by non-profit organizations.  This drastically reduces its future impact on *real* clinical applications.  As a promoter of open data my self, it is of critical importance that source code (like Linux, python, tensorflow, etc) as well as datasets (like ADNI, PPMI, etc) be open to the entire community and, in particular, to those that bring solutions to the clinical world (namely private companies).

* I haven't been able to appreciate the online system the authors claim to have.  The dataset is hosted on a Swedish hosting system (SciLifeLab) that a priori only Swedish University members  can access.  I sent an email for a login but haven't got any feedbacks...  Their login access policy seams to be yet another effort to draw a line between those who can access the data and those who cannot.  I wish a different system like Kaggle or grand-challenge.org had been used.
correctness: As mentioned before, the dataset is well balanced and the images were selected in a systematic way (details in the appendix).  Same with the manual annotation procedure which the authors have explained in great details.  Also, the fact that the images are in png format (instead of DICOM for example) makes it very easy to process.

clarity: Yes, very well written.  Good job!
relation to prior work: In my opinion, Table 1 says it all:  the proposed dataset is between 1 and 2 orders of magnitude larger than what has been publicly released before.   CSAW-M is thus an important step forward.
additional feedback: It is overall a good work that deserves to be published.
review_title: New mammographic dataset

Meta-review: The authors introduce a dataset mammographic dataset for benchmarking models in the domain of breast cancer. The reviewers noted the extensive scope of the dataset as well as its significance and there is a consensus on acceptance.
Decision: Accept