Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: The authors combined existing ophthalmology datasets, and introduced additional biomarkers / labels. The authors identified ML tasks that are relevant to patient care, and benchmarked classification performance on image data, biomarkers, and multimodal inputs.
strengths: - The authors identified good ML tasks relevant to Ophthalmology patient care, and how they can be incorporated into patient care
- The authors showed good summaries of the labels / biomarkers created for the data.
weaknesses: - It would be nice if the authors can elaborate more on the clinical significance of the biomarkers, why were these markers chosen, and what are the value ranges of the biomarkers and their implications.
- It is unclear about the grader’s qualification, and how the biomarkers are acquired (is there only one grader for each scan or are there multiple graders for each scan)
- There are standard deviations for the balanced accuracy. Why aren’t they included for other metrics?
- It is unclear if there are any domain shift between the data collected from two different studies. They study different conditions, which are also the labels of the classification task. Any domain shift between the datasets would compromise the classification result. It is also unclear why these two datasets are selected, and what are the clinical impact of such a combined dataset.
- Table 1 is confusing, it is unclear if there is any relationships between the merged rows in image modalities and label modalities.
- Time series data in the dataset correspond to visits, which have an average of 16 data points, and may have different frequencies and intervals. It us unclear how these data contribute to the classification of patients and there are no benchmarks included in the paper based on the time series data.
correctness: The dataset seems to be properly constructed and the experimentation seems to be reasonable
clarity: The paper seems to be well written, but table 1 is confusing. 
relation to prior work: The authors described how the work relates to prior work and compared it to existing datasets
additional feedback: N/A
review_title: New biomarkers and classification benchmarks for ophthalmology data

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: Authors presented a longitudinal multi-modal dataset comprising of 2D fundus image, 3D OCT scans, clinical labels and biomarkers collected from patients undergoing treatment for Diabetic Retinopathy or Diabetic Macular Edema. Authors have also presented baseline results for tasks such as DR/DME detection, disease classification, biomarker detection and clinical outcome prediction.

strengths: - Longitudinal data covering multiple modalities
- Facilities research in several different directions: disease detection and classification, treatment progression as well as understanding the relationship between multiple modalities. 
weaknesses: - Authors explicitly state this, data is collected from one geographical location and could be biased.
- Sample size in terms of the number of patients. 

correctness: Yes
clarity: The paper is well written. The machine learning tasks mentioned in the Section 4 could elaborate on the inference task.
relation to prior work: Yes
additional feedback: This sort of longitudinal datasets with multimodal data are very important and hard to acquire. The concerns I have with this dataset are the limitations in terms of the number of patients and the lack of geographic diversity.
review_title: OLIVES Dataset: Ophthalmic Labels for Investigating Visual Eye Semantics 

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper presents OLIVES, an OCT and near-IR fundus dataset that includes clinical labels, biomarker labels, disease labels, and time-series patient treatment information from associated clinical trials. The dataset contains the information of 96 eyes averaged over a period of at least two years, with each eye treated for an average of 66 weeks and 7 injections. Benchmark experiments, benchmark models, and baseline results are presented.
strengths: This dataset contains 96 eyes and an average of 16 visits per patient, and 1268 fundas eye images. Figure 1 clearly illustrates the clinical practice described in section 1.

This dataset is collected over a long period of time. The long spanning of time series data allows future researchers to perform experiments on predictive models. 

Good level of details on data collection and hyperparameters used. 

The authors have discussed related work in different aspects. All mentioned research was properly referenced.

Compared with existing datasets, OLIVES contains a comprehensive set of modalities and is large enough in volume to be leveraged by ML algorithms. According to the paper, this is currently the largest and most diverse dataset of its kind.

weaknesses: The entire paper is hard to follow for reviewers who are not experts in biology because of the extensive use of abbreviations of biological terminologies. I understand that this paper is targeted toward an audience in the domain of biology/medicine. Still, to facilitate interdisciplinary research, it would be great if the authors could include in their appendix the corresponding full names of the abbreviations used in the paper.

I would suggest the authors reorganize section 4.1. Table 2 presents experiments with increasing balanced accuracy, but section 4.1 presents different tasks in different orders, which makes readers hard to follow. Would be great if the authors could indicate which ML model is used for which task in the tables.

Table 3 is unclear at first glance. It would be clearer if the authors could discuss the first three models in detail in the corresponding section. Also, it would be better to mention that table six is in the appendix. At first glance, I thought the authors forgot to present table six in the paper. Also, in the last section of section 4, there is no “figure c.3” in appendix “c.3”. The overall paper needs more careful review. 

In the discussion section, the authors should consider elaborating more upon the ethical implications of this study.

correctness: Yes.
clarity: Yes.
relation to prior work: Yes.
additional feedback:  N/A.
review_title: OLIVES Dataset: Ophthalmic Labels for Investigating Visual Eye Semantics

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper provides medical data from different modalities with potentially positive impact on medical research and treatments.

The authors train different ML models to analyze the ability of the presented data to detect the relevant diseases ( DR/DME ), as well as  predicting the effects of the successive treatment and the final occular state.

They explain the technical details of their experiments and their outcomes.

strengths: The paper provides medical data from different modalities with potentially positive impact on medical research and treatments.

The authors train different ML models to analyze the ability of the presented data to detect the relevant diseases ( DR/DME ), as well as  predicting the effects of the successive treatment and the final occular state.

They explain the technical details of their experiments and their outcomes.

The paper & the pesented dataset seem good, grounded and valuable to me, but it is hard for me as someone without any medical background to evaluate the medical analysis and justifications made in this paper.
weaknesses: It is unclear to me, why they trained the vision models used to test the abilities of the dataset with a ResNet 18 backbone, which is pretty small and old compared to 2022 ML standards.


correctness: In general, it is hard for me as someone without any medical background to evaluate the medical analysis and justifications made in this paper.
clarity: yes, the writing style seem good and clear to me
relation to prior work: I cannot judge this, cause this medical field is new to me.
additional feedback: It would be interesting to evaluate the dataset using newer, potentially better models e.g. by finetuning the Image encoder backbone of a CLIP model.
review_title: Well-written paper with potentially positive impact on medical research and treatments

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The authors provided an Ophthalmic dataset with OCT and near-IR fundus images, including clinical labels, biomarker labels, disease labels, and time-series patient treatment information from associated clinical trials. The authors introduced the OLIVES dataset to bridge the gap between existing ophthalmic datasets and the clinical diagnosis and treatment process. 


strengths: The Paper is well written and correctly addresses the problem statement.
The Paper has introduced a dataset with three modalities and shown its scope in the field of ML.

weaknesses: 1. The size of the dataset introduced is small.
2. The data is collected from two trials, PRIME and TREX. The authors have not mentioned the differences, which may/may not affect the model evaluation with collected samples.
3. The significance of the clinical features such as BCVA etc., should have been better explained to draw the comparison across the modalities.
4. Clinical labels and biomarkers are associated with each eye. How is the relation across the two modalities developed for the datasets? Per my understanding, there should be a 'Patient id' with 'left' and 'right' eyes and corresponding clinical labels and biomarkers associated with each sample (eye).
5. Are the mentioned three modalities correspond to the same patient? This means there are three samples across three modalities for each patient
6. Results in terms of sensitivity and specificity are missing, which are important for evaluating the ML model for disease diagnosis.
7. In Table 3, many inputs used to train the model have shown a random accuracy for binary classification. This proves the insignificance of these features and contradicts the author's claims. Similar results are found in table 7 in the supplementary.

8. Data collected from a single centre might encourage data bias.

correctness: yes
clarity: yes
relation to prior work: yes
additional feedback:
review_title: provided Ophthalmic dataset with OCT and near-IR fundus images that includes clinical labels, biomarker labels, disease labels, and time-series patient treatment information from associated clinical trials

Meta-review: metareview: The reviewers struggled to find a consensus for this paper. Concerns about applicability of the dataset due to domain shift, issues with the data collection, size of the dataset, and clarity of the paper were raised. At the same time, I believe that despite its size, the value of longitudinal data for diagnostics is extremely valuable to the community. And the authors have made efforts to improve the readability, therefore I recommend accept.
confidence: 3: The area chair is somewhat confident
Decision: Accept