Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 3: Clear rejection
review_content: summary and contributions: This paper provides a data set of SRH images from a number of types of brain tumors as well as control tissue. It also provides several benchmarks for accuracy of classification of these images at the individual patch, slide, and patient level. 
strengths: * The community needs more medical imaging data sets, and this is one covers a large number of tumor types and patients. 
* The benchmarking seems of very high quality and is very thoroughly described. 
weaknesses: * The "ground truth" dataset itself was categorized based on a neural network. This is not a problem in general, but the description of exactly how this was done is by far the weakest part of the paper. Who were they manually checked by? How much of the data was checked at each stage? It is hard to evaluate any of the results if the ground truth is not certain, and in a paper proposing a data set and that it should be used as a benchmark it is critical to understand this.
* The data comes from a single institution and a single commercial instrument. It is not clear to this reviewer whether one or many companies produce such instruments, but if this data only helps purchasers of a single (or a few) commercial instruments, the breadth of its utility seems limited to me. 
* There is not a current finalized data use agreement. Exactly how "open" OpenSRH will be cannot be determined until this is true. 
* The data is currently structured as either a single extremely large tarball (>300GB) or per-patient. The latter is appreciated, but without information about which patient(s) have which tumor types, it isn't clear how would would know which patients to sub-sample. The metadata.json files are not available as far as I can tell except in the large tarball.
correctness: The benchmark evaluation and experimental design appear correct to me. Per the "Weaknesses" section, the dataset does not contain enough information to assess this. 
clarity: Yes.
relation to prior work: Yes.
additional feedback: None
review_title: NeurIPS 2022 Track Datasets and Benchmarks Paper264 Reviewer YwAc

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The authors introduce a public dataset of stimulated Raman histology images of brain tumors (OpenSRH). The authors claim that this is the first dataset of its kind (I cannot confirm or deny this, because I am not familiar with previous literature in Raman histology datasets). This dataset includes over 1,300 whole slide images from 307 patients. The authors also establish classification and representation learning benchmarks on this dataset. This is a rich dataset, which includes raw acquired data, patch-wise tumor classifications for the whole slide images, and processed patches ready for ML pipelines. Overall, this is an excellent and exciting contribution to the field.
strengths: The authors provide strong motivation for their work -- specifically that the pathologic diagnosis of brain tumors is usually unknown until surgery, whereas it is known prior to surgery for other cancers. The authors also successfully motivate the use of stimulated Raman histology over hematoxylin and eosin-stained histology (such as frozen sections that might be used for diagnosis during surgery). Another strength is in the description of stimulated Raman histology and SRH imaging. Moreover, the data and code are all available (to reviewers at the moment, and to the public in the near future), and the appendix provides meticulous documentation of the data and training.
weaknesses: There are very few weaknesses I can identify in this work. The authors have done an excellent job. One weakness I would like to point out is in the evaluation metrics. The authors use accuracy (and variations) and mean average precision. In addition to these, I highly suggest including a metric that highlights false negatives. False negatives would be unacceptable in a diagnostic task, so the authors should provide benchmark evaluation of this and highlight the amount of false negatives. 

In addition, I would caution against using accuracy as an evaluation metric in this case, because the labels are unbalanced (based on the number of slides in Figure 3). The authors should report metrics that properly handle unbalanced data, like AUROC and AUPR (or related metrics).
correctness: Yes, the claims made in the submission are correct to me.
clarity: Yes, the paper is very well written. There is one section that I did not understand fully (lines 247-249): "Interestingly, the patient-level metrics for pretrained ViT-S were prominently high, while the patch-level metrics were comparatively worse. We believe these results may be due to poor aggregation of patch-level predictions." If there is poor aggregation of patch-level predictions, wouldn't one expect poor patient-level predictions as well? Perhaps I am misunderstanding here. Can the authors please clarify?
relation to prior work: The authors provide a great summary of previous work. Indeed, much of the related prior work is in hematoxylin and eosin-stained histology images. One large claim that the authors make is that OpenSRH is the first public SRH dataset for brain tumors. I cannot verify this because I have insufficient knowledge of existing SRH datasets.
additional feedback: The authors indicate that data will be available via Google Drive and Amazon AWS S3. In my experience, downloading data from Google Drive has been inconvenient, because it does not allow for programmatic downloads (e.g., with wget or curl on a remote machine). I would encourage the authors to consider a different hosting mechanism. Perhaps DropBox, Zenodo, or an AWS S3 bucket that does not require an account.

Can the authors indicate the range of patient ages in the dataset? Some brain tumors are much more common in children, while others (like metastases) would be more likely in adults. It is possible that the tissue or tumor microenvironment of a younger patient could be different than that of an older patient, and this is why I am asking about the range of ages.

In lines 125-126, the authors write that "SRH image acquisition is invariant to patient demographic features, clinical workforce, and geographic location." Can the authors please explain this claim? How are they certain that SRH image acquisition is invariant to all of these factors?

Regarding partitioning data into train/val, are the data split by patient or by slide? The data should be split by patient, so that no patient is present in more than one data split (otherwise there could be leakage between splits). --- After reading Figure 9 in the Appendix, I see that the data is split by patient. But I will let the authors confirm that this is indeed the case.

On line 145, the authors indicate that the two channels of an SRH image are co-registered. How robust is this co-registration? Does it every fail?

On lines 200-201, the authors write that ViT requires larger data than ImageNet-1k --- I agree with this. In the future, the authors can consider evaluating transfer learning from a vision transformer pretrained on ImageNet-22K (they are publicly available).
review_title: First public dataset of stimulated Raman histology images of brain tumors

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The medical problem being tackled in this paper seems exciting and novel (given the reviewer's knowledge about medicine which is not much). The SRH dataset is apparently unique and would allow real-time brain tumour segmentation and classification with significant consequences for patients.
This being said, the reviewer is not convinced that this paper suits the NeurIPS conference's or the NeurIPS Datasets & Benchmarks Track's objectives.
While the medical added value of the paper could be interesting (from what the reviewer can only guess), the interest for the machine learning community seems pretty limited: the dataset is a medical imaging repository and the techniques being benchmarked are very classical computer vision models and contrastive learning methods. It is not clear to the reviewer that this new dataset will foster particularly innovative discoveries on the machine learning front.

strengths: The medical problem is well stated and seems important.
The github repository is well conceived and user-friendly while the dataset is easy to access and well constructed.
The approach of the tumour segmentation and classification tasks is rigorous.

weaknesses: As mentioned above, the reviewer does question the added value of the dataset to ML's innovation.

correctness: The dataset seems constructed is a very sound as well as innovative way.
Benchmarks for the studied segmentation and classification tasks and contrastive learning techniques are classical and seem pretty natural.

clarity: The paper is very clear and well written. The medical objective is well explained for non-specialists in medical imaging.
The supplementary material is well conceived and very useful.
Access to the dataset and the github repo are user-friendly.

relation to prior work: The differentiation of this dataset vs. other histopathology repositories is pretty appealing and well explained.
additional feedback: NA
review_title: Very interesting on the medical front. Not sure about the machine learning side.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper introduces OpenSRH,  a public dataset including 1300+ clinical SRH images from 300+ brain tumor patients, and a benchmark for two computer vision tasks: multiclass histologic brain tumor classification and patch-based contrastive representation learning. 
The proposed workflow demonstrated in the benchmarking, and the dataset aims to accelerate the development of fast, reliable, and accessible intraoperative diagnosis practices for brain tumor surgeries.
strengths: - The authors well documented the need for this dataset through their prior work and sound arguments regarding the specifications of an ideal system for brain tumor surgeries. 

- The paper included two well-documented and explained benchmarks that exemplify their proposed workflow and the significance of the dataset. Additionally, enough details are provided for reproducibility.
weaknesses: - The paper recommends a workflow for classifying brain tumors using Machine Learning instead of existing resource-intensive workflows. However, the paper doesn't touch on the accuracy of their automated process compared to the existing anywhere in the paper. I think a brief mention of this is needed as a current limitation if there is a dip in accuracy or a strong point if the accuracy is improved.

- The paper didn't highlight or explicitly elaborate on their results in both tables (1 - 2), although I think the results are rich enough for a rich discussion and comparison of the results.
correctness: The claims made in the paper seem to be correct.
clarity: - Overall, the reading flow of this paper was easy. Highlighted bullet points with meaningful descriptions of what to expect in the following paragraphs were beneficial. An excellent example of this is section (4) data description.

In section 5.3, the authors mentioned, "We did observe improved performance when using ImageNet pretraining." However, based on table 1, the patient-level accuracy does not follow this observation. A more detailed analysis is needed. 

In Tables 1 and 2, the authors may make the best result in each column bold. 
relation to prior work: The paper provides detailed background knowledge about SRH and its applications in brain tumor surgeries, allowing non-experts to understand this paper's significance quickly. 
Previous related works are clearly explained and smoothly embedded into this paper. The authors discussed the current difficulties in ML for SRH.
additional feedback: N/A
review_title: A 300+ patients brain tumor dataset with benchmark

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: Overall, I have enjoyed this paper very much, and believe that it provides an excellent contribution to the literature/NeurIPS.

Major points:
- rapid intra-operative histopathological assessment of tumor samples seems incredibly desirable/valuable
- two aspects are highlighted: increased speed (savings of approx. 30 minutes/75% of time after a sample is taken and transferred to the slide) and through digitization and AI/computer vision techniques a reduced dependence on human neuropathology expert opinion; it would be important to test this with a pre-clinical prospective study (what is the influence of AI generated labels/diagnoses on surgical decision making; what are benefits/harms compared to current standard of care; when and how can over-reliance on AI lead to detrimental outcomes, etc.)
- the dataset, together with the workflow/code from the authors will allow rapid development of these protocols/studies/trials to demonstrate the clinical value of SRH as an integral part of tumor surgery, and given the open access (upon request, as it seems) will allow other groups to increase the speed of adoption of this technology

strengths: - manuscript very clearly lays out the advance made by switching from frozen sections to SRH imaging
- methods are well described and results well visualized; the main findings can be readily extracted: SRH reduces the time from sample extraction to usable information by about 30 minutes (75%); and using contrastive learning on a (frozen) computer vision model (head) significantly improves the classification accuracy of patch-level inference to (almost) patient-level accuracy (of the simple across-patch aggregated data); this second finding (patch-level inference matches simple-aggregate patient-level inference) might be highlighted in the abstract and discussion to make it stand out even more clearly
- data was collected for specific research/technology development interest, making it a very strong contribution to the field
- data and code is (or will be) shared for further collaboration and development

weaknesses: The only remaining question I had while reading occurred in lines 86-87 -- could the AI approach also be combined with light-microscopic (traditional) histopathology? it would seem important to separate the two contributions to the field (SRH as a speed up, and AI as a reduction in pathology expert reliance) for a more fine-grained evaluation of the claims; what would it take to train a similar AI on (high-resolution digitized) traditional H&E slides? And given how much a SRH scanner costs (and the lack of expert pathologist availability), is that the greater (or lesser) bottleneck in current practice?
correctness: From what I could tell, the claims seem correct, and I have no concerns at present; however, given that the data cannot currently be requested (the Google form states this will be at a later date), I was not able to inspect the dataset itself.
clarity: Only some **minor** points noted here:

- line 139, the unit (3x3 mm3) suggests either a planar or cubic specimen, please clarify
- lines 147 and 148: into how many such 300x300 patches would a typical slide be "cut" during preprocessing (to give readers a sense of the density of the underlying data, as well as over how much the aggregation function later needs to work)? have the authors tried different patch sizes (if so, what were the findings; if not, maybe it's worth considering, given that information as to tissue "class membership" may reside at different spatial frequencies/scales)? incidentally, what is the mm- (or micrometer) size of a 300x300 patch?
- plus, out of curiosity: have the authors tried (partially) overlapping windows? if not, why were non-overlapping windows chosen?
- lines 148-151: if the third channel is fully explained by channels 1 and 2, how does it add to the network performance? also, assuming that channels 1 and 2 are bound between 0 and some arbitrary upper value/unit (for full luminescence of the sensor, say, 1); the third channel would have double the width (between -1 and 1); how is that coded in the input of the data?
- line 153: who performed the manual labeling? (multiple experts? if so, what was their concordance? this might help further convince readers of the quality of the dataset)
- line 182: what function was used for the image resizing (linear/cubic interpolation, sinc/spline, etc.)?
- line 224: why does the number in dimensions for the hypersphere differ this much across the two model types (ResNet vs. ViT-S)?

relation to prior work: Since the authors already cite Dong Li's work (ref #3), maybe they would also like to include https://www.sciencedirect.com/science/article/pii/S1878875021003156 as a paper that already demonstrates the value of SRH specifically?
additional feedback: - this is a wonderful contribution to the literature, demonstrating state-of-the-art technology brought to the bedside; I congratulate the authors, and hope that more rapid and reliable intra-operative tumor diagnostic can soon become available for brain tumor patients everywhere
- figure 4: while maybe not entirely easy to achieve, it would be wonderful to see a confusion matrix on the patch level (specifically, how many patches of slides with non-normal tissue are mis-labeled), to get a sense of how necessary the aggregation into slide/patient-level classification is
- lines 248-249: would it be instructive (for future development) to show some of the slides (with classified as well as manually labeled patches from the cross-validation part of the training)? This might give readers some idea as to how a potentially improved aggregation function might look like
- line 258: for readers interested in applying this technique in their own setting, it might be beneficial to add (somewhere in the manuscript) the typical cost for obtaining the necessary equipment (SRH slide scanner, computer hardware necessary for inference, etc.), as well providing a clear path for collaboration (if already planned, maybe add a URL to the part of the opensrh.mlins.org site here?)
- figure 5: while definitely some extra work, it would be a great visualization to pick out some of the patches (across the different diagnoses among the validation images) and show where they "land" within the tSNE to help readers appreciate how the difference in "patterns" present in a patch might translate into the clustering location in the 2D projection
review_title: Reducing surgery wait-time via SRH and providing expert-level tumor classification important improvement

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: The paper introduces a new resource for brain optical imaging, namely OpenSRH, which is valuable for optimizing brain tumor surgery. The paper clearly explain the motivation of OpenSRH, which is easy enough to follow even without any prior knowledge in brain surgery or optical imaging. The paper provides sufficient details regarding the data collection methods and the dataset statistics. In addition, the paper develop two benchmarks from the dataset, i.e., histologic brain tumor classification and contrastive representation learning benchmarks. The experiment protocol for both benchmark is well-descibed and sufficient analysis is presented from the results.
strengths: - The published dataset opens up an interesting opportunity for optimizing brain tumor surgery through SRH imaging
- The established benchmark motivates future work to address the domain gap on the existing pretraining data and aggregation methods to better model slide-level and patient-level prediction
- The task described in the paper is well-motivated and well-written
weaknesses: **Patient-level prediction vs patient-level prediction** Despite the poor aggregation method is already briefly discussed on 6.2, I would expect there would be simple experiment to validate this phenomenon further by, for example, trying different aggregation methods during inference.
correctness: The dataset and the benchmark are well-designed and clearly explained
clarity: The paper is well-written, motivating, and easy to follow
relation to prior work: The paper clearly described prior works from other digital and computational pathology research and other SRH imaging studies. 
The paper further explain what is required for future research and how the paper accommodates this requirement. 
additional feedback: - Appendix E: "except for the unfrozen model weights in finetuning experiments.", does the fine-tuning experiment refers to the section 5? I think this needs to be revised to make it clearer
- The domain gap cap also be described by explaining the gap from the linear evaluation and the full fine-tuning of the ImageNet models.
review_title: Worthwhile resource with detailed description and thorough experiments

Meta-review: metareview: There are contrasting review rating ranging from 9 to 3. The reasonable concerns about this paper not been well validated for ML community. The proposed datasets has been benchmarked with some classical and some relevant DL methods. I have taken into account the expertise of reviewers and their concerns carefully. I do agree with some reviewers opinion about this dataset will be importance to medical CV community. I do agree with authors response "The open questions for the ML community that OpenSRH may foster innovative discoveries include the following: 1) domain adaptation between SRH images and other histology images such as H&E images in the large scale TCGA project; 2) using multiple instance learning (MIL) to avoid expensive dense patch annotations; 3) different aggregation methods for patch-based training, including clustering, attention, or MIL; 4) self supervised learning and comparing different augmentation strategies for SRH images; and 5) data efficient training of ViT architectures using SRH data. " Even though the authors have not verified these innovative applications on the openSRH dataset. I will go with acceptance opinion of few reviewers for this paper.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept