ToRL: Topology-preserving Representation Learning Of Object Deformations From Images

Authors: Tonmoy Hossain, Miaomiao Zhang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Representation Learning, Deformations, Topology

Abstract:
Representation learning of object deformations from images has been a long-standing challenge in various image or video analysis tasks. Existing deep neural networks typically focus on visual features (e.g., intensity and texture), but they often fail to capture the underlying geometric and topological structures of objects. This limitation becomes especially critical in areas, such as medical imaging and 3D modeling, where maintaining the structural integrity of objects is essential for accuracy and generalization across diverse datasets. In this paper, we introduce ToRL, a novel *Topology-preserving Representation Learning* model that, for the first time, offers an explicit mechanism for modeling intricate object topology in the latent feature space. We develop a comprehensive learning framework that captures object deformations via learned transformation groups in the latent space. Each layer of our network's decoder is carefully designed with an integrated smooth composition module, ensuring that topological properties are preserved throughout the learning process. Moreover, in contrast to a few related works that rely on a reference image to predict object deformations during inference, our approach eliminates this impractical requirement. To validate ToRL's effectiveness, we conduct extensive multi-class classification experiments across a wide range of datasets, including synthetic 2D images, real 3D brain magnetic resonance imaging (MRI) scans, real 3D adrenal computed tomography (CT) shapes, and \textcolor{blue}{real 2D facial expression images}. Experimental results demonstrate that ToRL outperforms state-of-the-art methods, setting a new way to enforce topological consistency in representation learning. Our code is available at - https://anonymous.4open.science/r/ToRL-44BF/

Decision: reject
Reviewer scores: 5, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730075191445

thread_type:
official_review

<thread_object>
creator: reviewer_tdx8

<content>
summary:
Existing deep learning methods focus on visual features (like intensity or texture) but fail to preserve geometric and topological structures of objects, which is crucial for applications in medical imaging and 3D modeling. The paper introduces ToRL, a new topology-preserving representation learning framework that models complex object deformations directly from images without relying on a reference image during inference.

contributions:

1. The proposed method models intricate geometric deformations in the latent space using learned transformation groups.
This latent space representation preserves structural and topological integrity during training and decoding.

2. The decoder architecture integrates skip connections with a novel smooth group composition module. This ensures that deformations remain smooth and topologically consistent across network layers.

3. Unlike other models that require a reference image for deformation prediction, ToRL removes this dependency, improving practical usability.

strengths:
1. ToRL explicitly preserves geometric and topological consistency throughout the representation learning process, which is essential for high-stakes applications such as medical imaging and 3D modeling.

2. Unlike many previous methods, ToRL eliminates the need for reference images during inference, making it more practical and efficient for real-world applications.

3. The decoder’s design with smooth transformations ensures structural continuity and smoothness, addressing the challenges of traditional feature fusion methods like simple concatenation or addition.

4. The results look good to me as it improves the classification performance and the ablation study is comprehensive.

weaknesses:
1. I am concerned with the complexity. The incorporation of group transformations in the latent space and smooth group composition can increase computational overhead, making the method slower for large-scale datasets or real-time applications.

2. The custom decoder design and latent transformation modules may make it more difficult to integrate with standard architectures (like ResNet or UNet) without extensive modification.

3. Although ToRL is validated on several datasets, it might benefit from additional tests on larger or more diverse real-world datasets to further demonstrate its robustness and generalizability.

4. The strong focus on topology preservation could lead to overfitting in scenarios where topological consistency is not essential, potentially reducing the model’s flexibility.

questions:
It would be great if the authors could explain more on the computational complexity of the proposed method, and evaluate on more diverse real-world datasets.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730397301853

thread_type:
official_review

<thread_object>
creator: reviewer_wezs

<content>
summary:
The paper presents a novel model and method for image representation that aims to preserve topological information. The authors highlight two main contributions: (i) developing an explicit mechanism for maintaining topological integrity within the latent space, and (ii) integrating a smooth group composition module into the skip connections of a U-shaped network to ensure topology preservation. The performance of ToRL is compared with other methods on downstream classification tasks. However, it should be noted that only classification tasks were evaluated. The main issue with this paper is that everything is artificial: the data, the tasks, and the method are all designed based on these artificial settings.

------

### Update given the author responses:  
Thank you for the responses! While I still believe that the issues I highlighted — including those related to experiments, writing, and methodology — persist, I am willing to acknowledge that the quality of this work has improved. I have therefore raised my score (from 3 to 5) to reflect my perspective and leave the final decision to the AC.

strengths:
1. The overall approach is somewhat inspiring and can essentially be regarded as an SVF  in the feature space.
2. The experiments demonstrate the effectiveness of the proposed representation learning method by showing improvements in downstream classification tasks

weaknesses:
1. Lack of real-world applications: The data, tasks, and methods used are all based on artificial settings, similar to early spatial transformer models. Although diffeomorphism is introduced to address classification task with spatial distortion, there is no extensive exploration of SO(3) transformations or other variations, despite substantial literature in this area. There are no experiments involving real applications related to diffeomorphisms, such as registration. The classification experiments are overly simplistic, and there is no post-hoc analysis to determine if the model truly preserves topology. In fact, due to the discrete errors in SVF, it is not theoretically a true diffeomorphism, yet no analysis is provided. As a result, this paper is neither theoretical nor practical.

2. Poor writing structure and unclear contributions: The paper spends considerable space discussing diffeomorphisms, which are not directly related to its contributions, without clearly highlighting what has been contributed. The design of TGM, which could be a key part of the method, is barely mentioned. In contrast, irrelevant network architecture details with many similarities to existing models (e.g., UNet, UNet++) are overly emphasized.

3. Factual errors or lack of evidence: There is a significant factual error regarding AdrenalMNIST3D, which is a binary shape dataset, but the paper incorrectly refers to it as "3D Abdominal MRIs." If the author have visualized the data, no one will make mistakes on distinguish binary shape and MRI. In fact, even for the source, the shape is from CT scans, not MRI. This raises concerns about other potential factual errors that I may not find. Furthermore, the claim that "our approach can be easily applied to other parameterizations of diffeomorphisms" is unsupported. TGM is not adequately described, making it difficult for readers to understand its details, and it is questionable whether other NeuralODE-based methods can be seamlessly integrated into this approach.

4. Computational cost not reported or compared: The paper does not compare the parameter count and computational load against state-of-the-art methods, which may result in unfair comparisons. Additionally, the experiments do not provide comparisons of the model's parameter count and computational load (e.g., FLOPS) before and after incorporating the smooth group composition module. Therefore, it is unclear whether the observed performance improvements are due to the proposed module or simply an increase in parameters and computational cost.

questions:
See weakness. Please clarify if I've misunderstood.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730592798108

thread_type:
official_review

<thread_object>
creator: reviewer_w891

<content>
summary:
This paper introduces ToRL, a novel topology-preserving representation learning model for capturing object deformations from images. Unlike existing deep neural networks that primarily focus on visual features, ToRL explicitly models object topology in the latent feature space through learned transformation groups. The model incorporates a novel decoder architecture with a smooth composition module to preserve topological properties during the learning process. Notably, ToRL eliminates the need for reference images during inference, which is a limitation in existing approaches. The authors validate their method through extensive experiments on multiple datasets, including 2D shapes and 3D medical images, demonstrating superior performance in classification tasks and better preservation of topological properties compared to state-of-the-art methods.

strengths:
1.	The technical contribution is novel and well-formulated, with a clear mathematical foundation for the transformation groups and their properties in the latent space.
2.	The elimination of the reference image requirement during the inference addresses a practical limitation in existing methods, making the approach more applicable to real-world scenarios.
3.	The experimental validation is comprehensive.

weaknesses:
1.	The mathematical formulation, while thorough, could benefit from more intuitive explanations or visualizations to help readers better understand the paper.
2.	The individual contributions of different components within the ToRL architecture are not analyzed. In particular, how different choices of transformation groups might affect the model's performance is not investigated.
3.	Although the paper claims superiority in preserving topological properties, additional metrics for assessing topological preservation are needed.
4.	The paper lacks a detailed analysis of the model's robustness to different types of input perturbations or variations in image quality, which is particularly relevant for medical imaging applications where image acquisition conditions can vary significantly.
5.	Beyond classification, are there any downstream tasks that could better showcase the capabilities to preserve the topology?

questions:
See the weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716214127

thread_type:
official_review

<thread_object>
creator: reviewer_2hvs

<content>
summary:
This work presents a representation learning for image deformations by learning transformation groups in the latent space, and innovates the preservation of topology of object in the decoder by incorporating a smooth group composition module. Experiments on various databases along with the comparison with SOTA models are given.

strengths:
The work proses a novel representation learning to preserve object topology in deformation. The whole paper is well-written.

weaknesses:
1.	The descriptions and proofs of the given formula (2), (5), (6) are insufficient. Readers cannot fully understand the reason of the design.
2.	The qualitative experiments results are incomplete. No figures show the results on 3D Brains.

questions:
1.	Please explain and verify the key formula proposed by this work. And explain the originality of the formula. For example, in formula (2), what is the second part? What is the relation to LDDMM?
2.	Please give more qualitative results, especially for 3D Brains database.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732304080874

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author Rebuttal: General Response

comment:
We sincerely thank all reviewers' valuable comments, insightful suggestions, and valuable feedback. We are encouraged that the reviewers recognize the

- **model/technical contribution novel and well-formulated** *(2hVs, W891, tDX8)*
- **importance of eliminating the reference image requirement** *(W891, tDX8)*
- **experimental validation and ablation study comprehensive** *(W891, Wezs, tDX8)*. 

Besides, reviewers **appreciate our novel decoder design** for ensuring structural continuity *(tDX8)* and **mathematical foundation clear** *(W891)*, and **overall approach inspiring** *(Wezs)*. This helped improve our submission and strengthen our claims.

We have carefully addressed all the reviewers' concerns and hope the improvements and clarifications will be considered. Their feedback has helped improve our manuscript, and the changes are summarized below. 

- Added qualitative visualizations on brain dataset **[2hVs]** *(Fig. 6)* 

- Performed experiments on diverse real-world 2D facial expression datasets having diverse image qualities. **[Wezs, W891, tDX8]** *(Tab. 1/2/3/4; Fig. 5)* 

- Reported computation costs of all models, including parameter counts, computational load (FLOPS), training/testing times, and their trade-offs with performance accuracy. **[Wezs, tDX8]** *(Tab. 5)*

- Validated individual components of ToRL. **[W891]** *(Appendix Fig. 8/9)* 

- Evaluated additional metrics to assess the necessity of topology preservation. **[W891]** *(Tab. 4)* 

- Performed experiments on the model’s robustness to input perturbations. **[W891]**  *(Fig. 8)* 


For clarity, we highlight the revised part of the manuscript in blue color.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This submission introduces a topology-preserving representation learning framework that captures object deformations from images by explicitly modeling transformation groups in the latent feature space. The method incorporates a smooth group composition module in the decoder to preserve topological properties and thus removes dependency on reference images during inference. Experimental work investigates 2D shape and 3D medical imaging datasets, demonstrating improved performance in classification tasks and topology preservation in comparison with existing approaches. 

This paper was discussed at length with the SAC. After reviewing the paper, rebuttal and resulting discussions the consensus was that although the review process has helped to strengthen the work, open reviewer concerns relating to experiments, writing, methodology can likely be better addressed with deeper revisions that will benefit from an additional round of review. AC therefore recommends rejection on this occasion however authors are encouraged to take onboard feedback for re-submission to an alternative future venue.