Contrastive learning of cell state dynamics in response to perturbations

Authors: Soorya Pradeep, Alishba Imran, Ziwen Liu, Eduardo Hirata-Miyasaki, Taylla Milena Theodoro, Ivan E. Ivanov, Madhura Bhave, Sudip Khadka, Hunter Woosley, Carolina Arias, Shalin B. Mehta
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: contrastive learning, dynamics, cell biology

Abstract:
We introduce dynaCLR, a self-supervised framework for modeling cell and organelle dynamics via contrastive learning of representations of time-lapse datasets. Live cell imaging of cells and organelles is widely used to analyze cellular responses to perturbations. Supervised modeling of dynamic cell states encoded in 3D time-lapse data is laborious and prone to bias. dy- naCLR leverages single-cell tracking and time-aware contrastive sampling to map images of cells at neighboring time points to neighboring embed- dings. We illustrate the features and applications of dynaCLR with the following experiments: analyzing the kinetics of viral infection in human cells, detecting transient changes in cell morphology due to cell division, and mapping the dynamics of organelles due to viral infection. Temporally regularized embeddings computed with dynaCLR models enable efficient and quantitative annotation, classification, clustering, or interpretation of the cell states. The models reliably embed, i.e., generalize to, data from un- seen experiments with different microscopes and imaging contrasts. Models trained with dynaCLR consistently achieve > 95% accuracy in mitosis and infection state classification, enable the detection of transient cell states and reliably embed unseen experiments. dynaCLR provides a flexible framework for comparative analysis of cell state dynamics due to perturbations, such as infection, gene knockouts, and drugs. We provide PyTorch-based implementations of the model training and inference pipeline and a napari plugin user interface for the visualization and annotation of trajectories of cells in the real space and the embedding space.

Decision: reject
Reviewer scores: 5, 5, 3

Reviews and discussion:
<thread>
creation date:
1730574374415

thread_type:
official_review

<thread_object>
creator: reviewer_ua5v

<content>
summary:
The paper describes an application of contrastive learning on a microscopy dataset to model single cell dynamics. The method introduces time-aware sampling to the traditional contrastive learning methodology by making use of cell-tracking. Three downstream applications of the learned embeddings are shown which demonstrate how the method can be used to analyze cell-state dynamics in response to perturbations. In addition, experiments showing generalization and interpretability of embeddings are performed.

strengths:
1.	The paper addresses an interesting topic where self-supervised learning has the potential to be extremely beneficial. Getting annotations for large amounts of microscopy data on a single-cell level is difficult and there is a broad push in the field towards the kind of self-supervised approach described here, so this work is well placed.
2.	The utilization of cell tracking to improve self-supervised learning is an interesting idea; cell-identity detection is potentially a useful proxy task that could aid in learning good representations.
3.	The biological setup and experiments are well thought out and the dataset would be beneficial to researchers as a benchmark, given simple readouts like infection rate.
4.	The addition of generalization experiments (though only on one additional set of data) provides some grounding to the results – without this section I don’t think the paper has much weight. I would encourage the authors to extend this to more datasets if possible.

weaknesses:
1.	In general, this is written like a paper in a scientific journal rather than a machine learning conference. While it is important to provide the necessary biological context, the paper lacks the mathematical discourse required to be suitable in a machine-learning setting. For example, could you add a theoretical or mathematical description of ideas such as ‘smoothness’ or ‘richness’ of the latent space that you have mentioned?  
2.	I’m not quite sure that the quantitative results match up with the claims made. For example, performance scores in Table 1 are better for the ‘classical’ contrastive learning approach as opposed to the ‘cell-aware’ and ‘time-aware’ approaches. The arguments made based on the Euclidean distance plots in Figure 2 are strenuous, and I don’t agree with the qualitative conclusions made about time-regularized contrastive learning being better than the classical approach from these results (I will add more on this in the next points). Overall, I don’t think the contributions here are significant enough and the arguments don’t hold enough weight for me to accept this paper. 
3.	The whole ‘temporal continuity’ argument seems a little counterintuitive to me. The ‘time-aware’ loss function is designed to encourage the model to ignore differences at the time-scale of the time hyperparameter Tau, but you expect the model to maintain temporal differences at time-scales t > Tau. I think Tau needs to be much smaller than the time-scale of your expected changes for this to work. If you expect changes in cells to happen over a few hours, the hyperparameter Tau being 30 minutes doesn’t make sense – your model is just going to smooth out all temporal information. I know you are limited by the fact that you can’t take images, say, every 5 minutes, but the current setup doesn’t make sense to me.
4.	Just to add to the previous argument – in your current setup, it matters how long you train your model for, since overtraining could completely smooth out all temporal information. I don’t even know how you would design experiments to determine what a good number of epochs to train is without having an idea of how much temporal smoothness is ‘correct’. For example, in the limit of an infinite number of epochs under this loss function, all temporal information would be removed from your model. I just don’t see how you could both encourage temporal smoothness while maintaining a temporally faithful embedding when your data has a temporal resolution of 30 minutes over 24 hours.
5.	In the case of ‘time and cell-aware sampling’, how do you ensure that the model is robust to imaging factors like brightness or sensor noise? The point of such augmentations in the classical contrastive learning case is to teach the model to ignore spurious confounders which may actually show up in your real data – but the ‘time and cell-aware’ model never learns to do this. Is this model actually good at generalizing? What happens of your imaging conditions change a little? Would this model still work? I think many questions need to be answered here.
6.	I think the novelty proposed here lacks conceptual correctness in my view. With traditional ways of using temporal information in self-supervised learning, like predicting the temporal order of randomly flipped images, it makes sense how these would lead to a temporally unbiased and meaningful embedding space. My arguments in the points above highlight my concerns on why the proposed method may not be doing the same.
7.	Without actual ground truth annotations of fine-grained temporality, the only way to assess the quality of the embeddings is through the results on downstream temporal tasks. However, the classical contrastive sampling variant seems to perform better on the downstream infection classification task, which tells me that the classical method is leading to better quality embeddings than the proposed method.

questions:
1.	I’m a little bit confused about the quantitative results. Specifically, you write “Therefore, we evaluate the accuracy of visual representation learned by our method using a biologically relevant benchmark: accuracy of the classification of the cell states with 3 hours of expert annotations. We compare our method with two baseline methods: fully supervised time-agnostic semantic segmentation of the infection state and time-agnostic contrastive learning. Compared to the ≈ 80% accuracy achieved by the supervised model and 60 − 65% accuracy achieved by the time-agnostic contrastive learning, DynaCLR models consistently achieve ≈ 95% accuracy.” I only see results in Table 1 that show classical contrastive sampling having an accuracy of 98.8%. Can you clarify where the results of the supervised model are and why there is a mismatch between the claimed values of 60-65% and the values shown in the table?
2.	When you say, “For the experiments in this paper, we set τ = 30 min, as we found this captures significant cell changes while maintaining temporal continuity without over-sampling.” Does this mean you tested with different τ values? If you did any experiments on this it would be good to include those, and would contribute to my concerns in the weaknesses section on temporal continuity and temporal faithfulness.
3.	In Section 3.3 you claim, “We compute (2) and (3) to evaluate the temporal evolution of the embeddings as a gradual, steady increase signifies strong temporal smoothness. In both figure 2 a) and 2 b), cell and time-aware sampling shows a smooth rising curve with minimal fluctuations. The classical contrastive method exhibits a rapid initial increase followed by plateaus, while the cell-aware approach shows intermittent fluctuations.” I’m not sure I see in Figure 2 what is described here. It seems like the curves for Euclidean distance are pretty similar between the different strategies. In fact, in the UMAP version it seems like the cell and time-aware version is most fluctuating (whether UMAP is an appropriate space in which to perform this is another question, since it is known to not maintain global structure). Can you please elaborate on this? I’m also unsure what you mean by “richness” of representation in Figure 2C and how you determined this.
4.	Can you elaborate the results of the integrated gradients based interpretability in Figure A1? What exactly is the model focusing on in the two channels? Can you provide evidence that this is relevant to the output task?
5.	You say “We observe a clear transition of cell states from interphase to mitosis as we follow the cells in UMAP space, particularly in models trained solely with the phase channel and incorporating temporal regularization” I kind of see what you’re trying to say but it’s still very unclear. Can you provide a mathematical comparison of the two models and show that one model is ‘smoother’ than the other over all cell division events?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730631457196

thread_type:
official_review

<thread_object>
creator: reviewer_awz5

<content>
summary:
In this paper, the authors propose a framework for modeling cell dynamics in response to perturbations. They suggested several downstream tasks for the learned representation, such as the analysis of viral infection kinetics in human cells, detecting transient changes in cell morphology, and mapping organelle dynamics due to viral infection. Furthermore, they reported that the proposed framework achieves an accuracy of over 95% for infection state classification, outperforming the supervised setting.

strengths:
- The paper is well-written, following a clear structure that enhances readability and comprehension.
- The learned representation was used in several dowstream tasks.

weaknesses:
- The technical novelty of the paper is limited, as DynaCLR may be considered a straightforward application of the triplet loss with varied sampling strategies.
-  Some figures (Fig. 2, Fig. 3, and Fig. 6) are not thoroughly explained; a more detailed description would enhance clarity.

questions:
- The choice of the triplet loss is not justified. Why is this loss more suitable for this task than alternatives like NT-Xent or InfoNCE?
- In Section 4.1.1, could you clarify ithe origin of the independent test data?
- In Section 4.2, the smooth transitions and tight clustering of division events are not immediately evident in Figure 4; additional support for these claims would be beneficial.
- In Section 4.3, it remains unclear how the referenced figures demonstrate that "the encoder learns meaningful features that describe cell dynamics."
- Figure 6 would benefit from a more detailed explanation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704650211

thread_type:
official_review

<thread_object>
creator: reviewer_ojpv

<content>
summary:
The authors present a self-supervised framework for leveraging contrastive learning to model cell state dynamics from time-lapse imaging. 
The model allows temporally adjacent states to be mapped closely together helping achieve accurate, efficient, and label-free analysis of dynamic cell states under perturbations like viral infection. The paper presents a unique application of contrastive learning in cellular imaging and stands out for its temporal coherence and robustness in infection state classification.

strengths:
- The use of temporally-aware contrastive learning enables efficient modeling of time-dependent cellular changes, showing improved performance over traditional contrastive methods, especially under perturbative conditions.
- The framework facilitates rapid annotation of cell states, potentially decreasing the reliance on human-intensive and subjective labeling, a significant advancement over prior approaches.
- By using a cell-aware approach, the framework attempts to address the intrinsic heterogeneity across cell populations, an advantage over traditional time-agnostic contrastive approaches.

weaknesses:
- The choice of a 30-minute interval as the temporal offset might not generalize across other biological systems with different dynamics, limiting the model adaptability.
- The reliance on phase and fluorescence imaging could constrain its utility where alternate modalities are necessary.
- Since cell-aware and time-aware sampling use specific tracked cells, the embeddings may risk overfitting to individual cell trajectories instead of generalized dynamics.
- Although contrastive learning was chosen, the paper lacks in-depth comparisons with generative methods that authors summarize in the related work.
- By setting a fixed temporal offset, the model may miss capturing events that unfold asynchronously or at variable rates in different cells.
- Models relying on phase channels for cell division detection may struggle with subtler morphological changes that require fluorescence markers.

questions:
- How does the model perform for other tasks beyond infection classification? Like, for example, tracking mitotic spindle dynamics during first cell division in embryonic development? 

- How does the model handle potential noise or artifacts in the time-lapse imaging data? How are hyperparameters tuned and how sensitive is the model to the choice of these hyperparameters?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732596043586

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of the revision

comment:
We want to thank all the reviewers for their detailed and constructive feedback and look forward to an active discussion over the next few days.

The reviewers recognized the timeliness of the problem we are addressing and the rigor of our solution. 

___“ temporally-aware contrastive learning enables efficient modeling of time-dependent cellular changes”, “framework facilitates rapid annotation of cell states, potentially decreasing the reliance on human-intensive and subjective labeling, a significant advancement over prior approaches”___  (reviewer oJPV);

___“The learned representation was used in several dowstream tasks.”  (reviewer: aWZ5)___

___“The addition of generalization experiments (though only on one additional set of data) provides some grounding to the results – without this section I don’t think the paper has much weight. I would encourage the authors to extend this to more datasets if possible.”___ (reviewer: UA5v).

The key constructive feedback from reviewers concerns the method's novelty and generalizability. We have added new results aligned with the paper's original scope on the generalized embedding models trained with public and in-house datasets. We respond to the reviewers' feedback individually. Below is a summary of the main changes to the structure and content of the paper.

* The main figures and appendix figures report the following data :
   * Fig 1: Overview of dynaCLR framework.
      - Appendix Fig 1: napari plugin for interactive annotation of cell states in embedding space.
   * Fig 2: Evaluation of time-aware contrastive sampling using public dataset (ALFI)
      - Appendix Fig 2: Displacement, dynamic range, and smoothness of cell tracks in the embedding as a function of the contrastive sampling strategy.
      - Appendix Fig 3: PHATE visualizations of embeddings with contrastive sampling strategies.
      - Appendix Fig 4: Evaluation of contrastive loss using in-house (cellular response to infection) data.
      - Appendix Fig 5: Number of significant principal components of embeddings as a function of contrastive sampling strategy.
   * Fig. 3: Evaluation of generalization of the learned embeddings across cell types and microscopes using public (ALFI) and in-house (cell and organelle response to infection) data.
      - Appendix Fig 6: robustness of dynaCLR embeddings to tracking errors in the data.
      - Appdneix Fig 7: detecting cell division events in infected cells.
      - Appendix Fig 8: visual inspection of remodeling of endoplasmic reticulum due to infection.
   * Fig 4: Explanation of learned embeddings with engineered features and occlusion-based class attribution.


* We report computational experiments to show that the time-interval hyperparameter can be changed to tune embeddings' dynamic range and smoothness (Fig. 2 and Table 1).

* In an exciting development, we find that the dynaCLR embedding model generalizes across unseen microscopes and contrast methods (Fig. 3). We think that the self-supervised learning of temporally smooth representations of 4D (3D, multi-channel) biological datasets is a valuable novel contribution of this work.

* NT-Xent and triplet loss lead to embeddings with similar structures  (Appendix Fig 4), and time-aware contrastive sampling improves the dynamic range and smoothness of embeddings independent of the loss function.

* We improve the writing throughout to highlight novel methodological aspects of the work, namely, 
  * The first use of time-aware contrastive sampling to learn embeddings from 4D tensors.
  * Application to diverse biological datasets in which cell states are labeled by a human or via an experimental fluorescent marker.

* We now use precise mathematical notation to clarify the contrastive sampling strategy, the metric of dynamic range in embedding space, and the metric of smoothness in embedding space.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper describes a self-supervised framework for modeling cell and organelle dynamics in time-lapse microscopy datasets called dynaCLR. The reviewers unanimously recommend rejection, citing a lack of significant methodological novelty and a lack of sufficient rigor in presentation from a machine learning paper.