Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper pursues the research question of how to evaluate representation learning for echocardiographic data (ultrasound images of the heart). This is tough because available open-access datasets, while valuable, each cover only a subset of the supervised tasks of interest and may not offer too many labeled examples for their chosen tasks.

The paper shows how to build representation learning benchmarks for many tasks of interest by remixing 3 open-access datasets: EchoNet from Stanford in California; CAMUS from University Hospital of St Etienne in France; and TMED from Tufts Medical Center in Massachusetts.

The first contribution is a suite of 25 specific "visual task adaption" benchmarks, listed in Table 2. Each involves a source and a target choice of dataset-view-label. 

The second contribution is an evaluation protocol that helps grade overall representations, via an average across tasks and task-categories (Eq 1). The idea is to be able to assess the "overall usefulness" of a representation across tasks, via a single metric.

Experiments show how their contributions enable insights about whether pretraining helps (Fig 3), which deep architecture might be best (Tab 3), and whether pretraining on medical images is better than generic images (Fig 4).
strengths: Strengths are:

* Tackles an important problem for an exciting application area (echocardiography); could spur significant ML methods development that is *useful* for improving the efficiency and quality of patient care

* Broad coverage of tasks of interest within echocardiography, ranging from segmentation for structure, function estimation, view recognition, and diagnosis prediction

* Specific experiments that show benefits of pretraining on medical tasks (3DSeg8) vs generic images (ImageNet) are valuable

* Experiments that assess performance versus target set data size (as in Fig 3a) are quite valuable, esp. for thinking about combining public data with in-house private labeled sets that are small

weaknesses: **Update after author response on 2022-09-02**:

I have raised my score to an "accept", because

* W0, W2, and W3 below have been completely addressed. 
* W1 has been mostly addressed (see my detailed comments in response to the authors for a remaining minor issue related to how CAMUS test data is used)

**Original review submitted in July 2022**

The key issues I see with the present paper are:

* W0: Confusing task definitions for a few tasks in Table 2
* W1: Missing target-only baseline in Fig 4 and Tab 4; makes hard to assess quality
* W2: Reproducibility is poor: key experimental design details / hyperparameters not available
* W3: Documentation not ready; seems to have many completely blank sections

See detailed subsections labeled "W__" below for elaboration on each of these weaknesses.

I think these are very addressable in the response period, and I look forward to engaging with the authors via discussion.

This work has lots of promise; I just want to emphasize that at present, the reproducibility and documentation really need to improve for this to be accepted.
correctness: I will raise some concerns about experimental quality here

### W1: Missing target-only baseline in Fig 4 and Tab 4; makes hard to assess quality

When evaluating whether a source task helps, it is important to consider the baseline of just "Training from scratch". While done in Fig 3, this is missing in Fig 4 and Tab 4. Makes it very difficult to know if any of this pretraining is worthwhile.

I'm particularly wondering about the view classification results in Fig 4, which appear to score in the 0.5-0.75 range for AUROC for the CXX4 target task (distinguish between A2C and A4C views). Previous publications indicate that view classification can be done much better: a 2018 study [*] reports 97% accuracy in a 15-way view classification task using 267 studies and a much simpler model (VGG16, not ResNet50). Perhaps there's an easy explanation (how many studies in target are available for fine-tuning?), but I think more explanation is needed here.


[*] Madani et al '18. Fast and accurate view classification of echocardiograms using deep learning.


clarity: 
### W0: Confusing task definitions for a few tasks in Table 2

Part 1: How can you fix a view type when classifying views?

In Table 2, look at tasks with prefix CA44____. Decoding this means that we should use CAMUS data images from view "A4C" to classify views.... but shouldn't we not specify the type of view in advance if we want to classify it? Is this just a typo, and it should be CXX4____ instead?

Part 2: Need to clarify if any task is multi-class (> 2 categories)

For example, in TMED data (see Tab 1 of Huang et al), the AS task has 3 labels (no AS, mild-to-moderate AS, and severe AS). Is your AS classification task using all 3 labels, or a binary task? If using all 3, how do you compute an AUROC score... what do you average?


### Minor issues 

Please address in revision, but no need to discuss in response period

- Should clarify that the EchoNet dataset used is from Stanford (current main paper doesn't specify the institution), and should clarify that it is "EchoNet Dynamic". This distinguishses from the recently released EchoNet LVH dataset: https://echonet.github.io/lvh/
- Under "Evaluation metrics" (line 187), mention that for all metrics, higher is better?
- Bolding policy in Tab 3 needs clarity.... why bold .893 but not .892 in first line?

relation to prior work: While there have been several general-purpose vision benchmarks that asssess representations across tasks (see lit review in App B), this is the first one I'm aware of specific to echocardiography.

Some prior work has already shown that open-access echo datasets are inter-operable (e.g. train on Stanford, test on Unity). But the broad scope of this benchmark showing how many inter-operable tasks are possible is new to me.

additional feedback: ### 1) Any specific reason not to also include the Unity dataset of echos from the UK? 

https://data.unityimaging.net/

Looks inter-operable with the datasets chosen: has PLAX views, measures EF, etc.

Not being aware of this resource until now is a fine answer, just seems natural way to extend the current work.

### 2) Can you suggest specific weights to assign to each task when doing the ETAB expectation in Eq 1?

I worry that specifying the ETAB score so generically has problems. 
I'd rather see specific weights recommended for each task, or just define it in terms of uniform weights.
Most users of this benchmark will have far less domain knowledge about echocardiography than the authors, so their ability to pick sensible weights is questionable.
review_title: Very promising work <strike>if poor reproducibility and lack of documentation can be addressed</strike>

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: The paper proposes a benchmark approach for representation learning of visual data in cardiac ultrasound. It provides 25 adaptation benchmarks in which a source task is performed on a source dataset, and then learned representation is employed for a target task on a target dataset. ETAB involves three datasets: EchoNet, CAMUS and TMED. The proposed benchmark suite captures a degree of adaptation on multiple datasets; nevertheless, the impact of this suite remains questionable. 
strengths: - The paper involves three well-known public datasets: EchoNet, CAMUS and TMED.
- Adequate classification of adaptation tasks, including clinical prediction, view recognition, cardiac function estimation and cardiac structure identification.
- Comprehensive experimental setup with the use of popular methods such as ResNet50, U-Net etc
- Provided some basic insights on transfer and representation learning in Echocardiography.
weaknesses: - The paper presents a benchmarking suite to investigate the task adaptation of representation learning methods which is simplistic with applications in a very limited context.
- The adaptation benchmarks were proposed based on limited views/knowledge of the existing datasets, which might not be exhaustive and "unified" for tasks in Echocardiography.
- Whereas enabling adaptation of pre-trained models are good to have, how does ETAB impact the existing/new workflows in developing representation learning methods?
- Limited discussion on theoretical contributions and practical implications of the paper in real-world scenarios.

correctness: Yes
clarity: Yes
relation to prior work: The paper did not discuss previous contributions.
additional feedback: NIL
review_title: Pre-train possibilities in Echocardiography

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper provides an echocardiographic task adaptation benchmark for the various clinically-relevant task using publicly available datasets. Their benchmark evaluates some model architectures, pretraining techniques to standardize the evaluation method for clinical usage.
strengths: This paper shows powerful usage in specific clinical tasks like Cardiac struct. identification, estimation, and clinical predictions. Translating the clinical evaluation protocols into ML-friendly tasks will bridge the gap in deep learning applications in the medical domain.

weaknesses: - **Novelty**: as addressed in the paper, "the domain-specific analogue of the general benchmark for vision tasks developed in [19,20]". Though the ETAB definition involving the formula is slightly different, it does not increase superiority in methodology. 

- **Methods are not comprehensive**: benchmarking a dataset should consider SOTA methodology in model architecture and pretraining. Though the paper mentioned ViT and Swin-ViT in line 173, it's not included in the main context and supplementary. In the baseline for the cardiac function estimation task, it only mentioned LSTM to handle the video frames without justification. The other methods for video data like ViViT should at least be mentioned. Other methods like semi-supervised learning and contrastive learning, which achieved a good performance recently, should also be considered as a good pretraining strategy in addition to transfer learning. 

- **Pipeline not user-friendly**: Github repo does not provide a user-friendly interface to reproduce the result. All the links at the bottem are invalid.
correctness: The claims appear to be correct, and the experiments are conducted in a sound way. The experimental design and evaluation are fine.
clarity: Including formula (1) does more harm than good. Evaluation metrics in Sec 4.1 is more straightforward than that formula.

Figure 3 should indicate what's meaning of x axis.


relation to prior work: As addressed in the related work, it's a domain-specific analogue to [19, 20]. 
additional feedback: Can you try to put the related work in the main context?
review_title: Review for ETAB

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This is a very well written paper and I believe that this benchmarking setup can later be used for other medical specialties. The use of AI in medicine requires a higher threshold of security and testing than other use cases because the difference between right and wrong output can mean life or death. The authors of this paper are suggesting a novel standardized approach to solving this problem.
strengths: Novelty. Well written. Strong main experiment and sub-experiment.
weaknesses: The GitHub website is not as complete as I would like it to be. However, I can understand that it is constantly being updated.
correctness: I am not a computer scientist. From a clinician-scientist point of view, I believe that the method and design are very plausible.
clarity: It is well written for clinicians and computer scientists.
relation to prior work: Yes it does.
additional feedback: Thank you for your amazing work.
review_title: Well written paper.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The work presents a benchmark suite of tasks for evaluating the performance of learned visual representations for use in echocardiography. This benchmark, the echocardiographic task adaptation benchmark (ETAB), provides a meta-dataset constructed from 3 existing echocardiogam research datasets. ETAB formalizes several categories of source and target task categories, and defines an aggregate ETAB score for estimating performance across a these benchmark categories. 

This benchmark enables evaluating a number of backbones and other representation learning approaches across a heterogenous mix of echocardiographic tasks and datasets, which is critical for assessing robustness and other properties of representation learning methods. 
strengths: - The benchmark is very promising. Moving beyond traditional benchmark datasets, which are often narrowly constrained to iid assumptions by task and dataset, is critical for practical development of applied medical ML. 
- Incorporating small training sizes (16+ training examples) is a nice focus area, as few-shot medical benchmarks is a critical research area. 
- The downstream tasks cover a nice range of realistic use cases (segmentation, cardiac function estimation, view classification, clinical attributes). 
- A metadata with a formal evaluation protocol across datasets would be a nice contribution
weaknesses: The most significant weaknesses of this work are
- A lack of clarity outlining the structure of the overall experimental pipeline in terms of measurable sub-components. 
- Reported results are terse or in some cases missing (i.e. all non-ResNet50 performance numbers). Currently the manuscript lacks a full  breakdown of results even for the ResNet50 backbone use case that forms most of the paper's results. This incomplete reporting makes it difficult to follow the overall merits of the benchmark. 

Clarity of results could be substantially improved. The paper proposes an evaluation strategy that covers a large number of model configurations, spanning architectures, datasets, pretraining strategies, and downstream task performance inclusive of some adaptation/transfer approaches. This is a very large cross product. While the gestalt ETAB score provides some useful assessment of representation learning quality (to the points raised in lines 146-147 around how individual scores assess specific transfer learning scenarios) a complete reporting of the individual source/target scores would make this paper much easier to follow. 

Figures 3 and 4 include subsets of tasks. Why not report performance for all task categories?  In general, I feel the paper is missing reporting for many intermediate results (e.g., performance metrics for the source models) as well as not providing a consistent report for all downstream tasks. 
 
The paper only includes results for ResNet50 architectures. Lines 173-176 states that other backbone architectures were evaluated, including vision transformers (ViT-L/16, Swin-ViT) and multi-layer perceptrons (MLP-mixer), however these performed worse than the ResNet50. While these results are stated as included in the supplementary material (line 176) they are actually not included. The supplement (which is quite short) includes pointers to the ETAB GitHub, but many links are broken and no results or leaderboard are available. I feel this is a substantial limitation and the paper would be substantially improved if we could observe ETAB scores across more than 1 family of backbone architectures.  I also find that not including the performance for the transformer and MLP architectures due to inferior performance to the ResNet50 is counter to the utility of a leaderboard benchmark, where these backbones are very reasonable baselines and increasing in popularity.  

Some of the datasets may have availability issues. While EchoNet has a complete DUA process and provides archival information the other 2 datasets seem problematic. The "Tufts Medical Echocardiogram Dataset" is currently a dead link. CAMUS is from a challenge dataset and it's unclear from the website if the data is available to researchers who are not participating in the challenge (which ends in 31 Dec 2024). I was unable to fine an explicit discussion of licensing terms for CAMUS. Sometimes medical challenge datasets are only available for a given challenge. What guarantees do you have that these datasets will be available to researchers moving forward? 
correctness: - Item 3(c) in the checklist asks for error bars with respect to running with multiple seeds, but none of the reported experiment results report any measure of standard error or standard deviation. 

- There do not appear to be any obvious errors in work.
clarity: While compact for bucketing results or various model configurations for reporting results, I found the benchmark task coding (e.g., EA40 CA22) very frustrating to parse while reading. This seems more useful as syntax for submitting to a leaderboard vs. something that enables clearer discussion for highlighting results in a manuscript. 

I think the benchwork would benefit considerably from anchoring on a single attribute across source and target (same/different task, same/different pretraining dataset) and discussing the performance differences.  

Given the mix and heterogeneity of the datasets, Table 1 would benefit from more summary information (or at least a more complete table in the supplement). Currently it's very hard to get a gloss of these 3 datasets in their "off-the-shelf" state. For example, what are the resolution of the images? What is a normalized unit of samples, such as number of total frames, that can be used to compare their relative scales? What are the counts of unlabeled vs. labeled across task categories (segmentation, view classification, severity ratings). Some of these details are found in Table 2, but key properties, like TMED not containing any cardiac function or structure tasks, containing only still images, etc. would be much clearer if Table 1 had clearer attribute columns.   

Figure 3 was confusing. Subfigure A contains results for 3 benchmark segmentation models/task . Why these three and not all models? Why are the 3 benchmarks (EA40CA40, CA40CA20, CA40CA42) for A different than (CA40CA20, EA40CA20, EA40CA22) used for B? Subfigure B contains gold standard (top) and U-Net predictions (bottom), but what does gold standard mean for the echocaediogram image view? If the point is to show the deltas between gold and predicted segmentation masks, there are clearer plots you could use here. 
relation to prior work: This work builds on 3 existing echocardiographic datasets by proposing the ETAB benchmarking suite. 

A related work section is provided in the supplemental materials that provides a high-level overview of the state of deep learning for echocardiography. 

additional feedback: - There is no author statement bearing all responsibility in case of violation of rights, etc., and confirmation of the data license.

- Given the experiments run, the manuscript should include an answer to section 3(d) in the checklist on the compute required to run experiments. Feels like the answer should be [Yes] (or at least [No] + justification) instead of [N/A]

- Given the scale of n in Figure 3a x-axis (n=16 to 256) it would be clearer to just use the original integer vs. the log_10 transform and label the axis something more intuitive like "n training examples".
review_title: A promising medical imaging benchmark, but needs additional work  

Meta-review: metareview: The consensus among the reviewers was that this work covers an important topic and a broad number of tasks. There were concerns about the documentation, reproducibility, and accessibility of the dataset. But the authors have done a good job in addressing most of these concerns with documentation, an API, and example notebooks.
confidence: 3: The area chair is somewhat confident
Decision: Accept