Venue: MIDL 2022 Conference

Reviews:

final_score: rating: 2: Weak reject
review_content: summary: This work presents a form of meta-study exploring four classes of existing deep learning paradigms based
on CNNs and the representations learned by these methods under different data augmentation strategies. Th
e authors study the difference in learned representations of weakly-supervised, unsupervised-, regularize
d and self-supervised learning settings. Experiments are demonstrated on 64x64 crops of grey-scale images of drug treated cancer cells. The importance of the learned representations are evaluated based on downstream classification and clustering performances.
strengths: * The overall objective of studying representation learning for image analysis under the various learning
 paradigms is interesting.
* The dataset and the downstream tasks chosen for this task are relevant and could be insightful.
* The source-code for all the models are open-source.

weaknesses: * **Comparing learned representations**: While the objective of this work is quite interesting, the experimental set-up might not be appropriate to compare the representations across different learning paradigms. By shifting the focus on comparing the representations between supervised, unsupervised and self-supervised methods with varying complexities, and in some cases differing size of latent representations, the message is somehow muddled. This work could have simply focused on studying the representations learned by each of the methods, and contrasting their peculiarities, if at all. Also, the assumption that somehow the intermediate representation in a weakly supervised method is the same as that of an unsupervised/self-supervised method is ambiguous. For instance, if the downstream task is to perform classification the learned representations at some intermediate layer is not most interesting as the final decisions could well be performed in the last layers.               

* **Assumption of a Euclidean space**: The notion that the learned, high dimensional semantic space is flat (Euclidean) and distances in this space could be comparable across methods is a bit problematic [1].

* **Discussion of results**: This meta-study provides scope for discussing very interesting properties of the different methods, which in my opinion are not fully explored. By only focusing on the downstream task performance the authors overlook the impact of different learning strategies. 

* Table 2 is too dense and not sufficiently well explained. Authors say that higher-is-better but in places, lower values are in bold (Ex: first row, last column is 0.20 compared to 0.17 in first column). Why is that? There are several more instances like this. 
* For model b in Fig. 2, how is BCE used as reconstruction loss? The intensity values are in grey-scale, I think.

[1] Arvanitidis, Georgios, Lars Kai Hansen, and Søren Hauberg. "Latent Space Oddity: on the Curvature of Deep Generative Models." International Conference on Learning Representations. 2018.


detailed_comments: See points above.
recommendation: Poster
review_title: null

final_score: rating: 4: Weak accept
review_content: summary: This paper analyzed the effect of representation learning by controlling learning paradigms, data augmentations, and cropping strategies. The learned representation was validated in three ways: (i) class similarity, (ii) linear probe, and (iii) clustering. The downstream task was a binary classification between drug and control performed on a private dataset. The results suggested that regularized learning required more computational power but achieved the most informative representation; self-supervised learning showed comparable representation while being faster to train. Data augmentation and multiple cropping facilitated representation learning.
strengths: + The experimental design is comprehensive and compact.

+ Many technical details can be found in the manuscript and appendix.

+ The conclusions are supported by both quantitative and qualitative results.
weaknesses: - The results were only validated on only one (private) dataset. The generalizability of the conclusions to other tasks, datasets, learning paradigms, augmentation strategies remains unclear.

- The terminology used in the manuscript is confusing and inaccurate.
detailed_comments: * In Figure 7, what do the numbers in the legend stand for? Why aren’t they ordered?

* The consistent terminology is very important for such an analysis paper so that the readers can easily capture the objectives and settings of the study. This terminology is better to follow the convention in the existing literature. A new word is introduced only when it is necessary (there is absolutely no other word used by peers that can describe this concept). Please improve the word choice in the paper as suggested in the answers below.
recommendation: Poster
review_title: null

Meta-review: metareview: All reviewers have brought up concerns about this study; however, after rebuttal, I think the authors have clarified a few concepts, e.g., reconstruction loss, by adding new references as well. After rebuttal, I think at least two reviewers are accepting the work and I recommend a weak acceptance to this work.
confidence: 3: The area chair is somewhat confident
Decision: Accept (Poster)