Title: Homeomorphism Alignment in Two Spaces for Unsupervised Domain Adaptation
Authors: Lihua Zhou
Mao Ye
Siying Xiao
Xu-Qian Fan
Xiatian Zhu
Ferrante Neri
Venue: ICLR 2023
Keywords: Homeomorphism Alignment
Unsupervised Domain Adaptation
Self-supervised Learning

Abstract: The existing unsupervised domain adaptation methods always align the features from the source and target domains explicitly or implicitly in a common space, i.e., the domain invariant space. Explicit distribution matching always ignores the discriminability of the learned features, while implicit distribution matching such as self-supervised learning suffers from the pseudo-label noises. It is difficult to find a common space which maintains discriminative structure of the source and target domain data when aligning the data distributions. We propose a novel approach dubbed as HomeomorphisM Alignment (HMA) so that the source and target features can be aligned in two different spaces. Specifically, an invertible neural network based homeomorphism is constructed. Distribution matching method is used as a sewing up tool for connecting homeomorphism mapping between the source and target feature spaces. Theoretically, we show this mapping can preserve data topological structure, i.e., the samples in the same cluster are still in the same projected cluster. Based on this property, we adapt the model by the cross entropy of transformed and original source features and prediction consistency between target features and transformed target features. Extensive experiments demonstrate that our method can achieve the state-of-the-art results.

TL;DR: A new appraoch uses Homeomorphism property to do Unsupervised Domain Adaptation.

Review discussion:
Strengths:
- The idea of learning a homeomorphism between the source and target feature space is new.
- The method can be used with different distribution alignment strategies.
- The method improves the distribution alignment strategy it is applied to.
Weaknesses:
1) Related work:
1.1) The discussion of the related work is incomplete and somewhat misleading, particularly in the introduction. The authors state that " the methods based on distribution alignment only consider the transferability and ignore the discriminability of the feature". This is not quite true. As a matter of fact, the goal of methods such as CDAN and CAN, or even CaCo, all cited in this paper, is to account for the category labels during the alignement.
1.2) Several methods that also account for discriminability during alignment are missing.
For example, Saito et al., MCD, CVPR 2018; Lee et al., SWD, CVPR 2019; Lu et al., STAR, CVPR 2020.
1.3) There exist a growing literature on the use of optimal transport for domain adaptation, starting with Courty et al., 2015 (Optimal Transport for Domain Adaptation).
For example, Damoradan et al., DeepJDOT, ECCV 2016, also aim to take the categories into account in the alignment process. More generally, I wonder if there is a deeper connection between optimal transport and the idea of homeomorphisms used here.
2) Methodology:
2.1) The motivation behind the second term in the loss of Eq. 4 (Loss_S) is not entirely clear to me. The authors showed that the topology of the feature space is maintained by the homeomorphism.
Therefore, why is it necessary to also encourage a correct classification after mapping to the target space? Based on theory, if the classification is correct in the source space, it should also be correct in the target space.
2.2) For similar reasons, the necessity of the loss in Eq. 5 is not clear to me.
Furthermore, the choice of the L_2 norm seems a bit arbitrary to enforce consistency between classifier predictions. Why not use the cross-entropy?
3) Experiments:
3.1) While the results show an improvement over the distribution alignment strategies the method is applied to, the improvement over CAN is small in two out of three cases. This tends to show a diminishing return: As the distribution alignment method becomes increasingly effective, the impact of the proposed method decreases.
3.2) Although the authors perform several ablation studies, I suggest them to validate the loss terms they used, as discussed in comments 2.1 and 2.2 above.
- The paper is mostly clear, although English could be improved.
- The description of the INN in Section 4.1, together with Fig. 2 could be clarified (the link between the figure and the text is not obvious).
- The motivation behind the loss terms in Eqs. 4 and 5 should be improved.
- The method appears to be novel, although I would appreciate a discussion of its relation to optimal transport, if any.
- The method seems reproducible, with some effort for the INN and training parts.
NO.
Q1. The discussion of the related work is incomplete and somewhat misleading, particularly in the introduction. The authors state that " the methods based on distribution alignment only consider the transferability and ignore the discriminability of the feature". This is not quite true. As a matter of fact, the goal of methods such as CDAN and CAN, or even CaCo, all cited in this paper, is to account for the category labels during the alignement.
A1: Thanks for this suggestion and we agree it is not sufficiently accurate. These mentioned methods are all based on conditional distribution alignment which considers the category information in marginal distribution alignment for discrimination.
What we intended to express is that, by performing feature alignment either via marginal distribution alignment or conditional distribution alignment, existing methods tend to align the features of different distributions in one space;
As a result, the original feature structure will be inevitably distorted, potentially destroying useful features. Some similar hypothesis can be found in BSP(ICML,2019), DWL (CVPR2021) and SRDC (CVPR2020). We have made a more clear presentation in the new version.
Q2. Several methods that also account for discriminability during alignment are missing.
For example, Saito et al., MCD, CVPR 2018; Lee et al., SWD, CVPR 2019; Lu et al., STAR, CVPR 2020.
A2: Thanks. We have added these methods in Section A.1.
Q3. There exist a growing literature on the use of optimal transport for domain adaptation. More generally, I wonder if there is a deeper connection between optimal transport and the idea of homeomorphisms used here.
A3: Thanks for this comment. We have added these works and discussion in Section A.1.
Q4. The motivation behind the second term in the loss of Eq. 4 (Loss_S) is not entirely clear to me. The authors showed that the topology of the feature space is maintained by the homeomorphism.
Therefore, why is it necessary to also encourage a correct classification after mapping to the target space? Based on theory, if the classification is correct in the source space, it should also be correct in the target space.
A4: Thanks. We will further clarify.
Specifically, the topology of feature space is maintained by the homeomorphism, in practice obtained by training an INN with neither feature extractor $F$ nor classifier $C$ updated in this process.
Therefore, the domain shift between source and target domains is still preserved. Based on the homeomorphism, the structure information can be transformed across domains.
Specifically, $f^{s2t}$ has the same label information with  $f^s$, which can be used to train the feature extractor $F$ and classifier $C$.
At the same time, $f^{s2t}$ has the same distribution as $f^t$, so classifying  $f^{s2t}$ is essentially classifying the target domain. This is how we realize the domain adaptation. It is also clarified in the revised version.
Q5. For similar reasons, the necessity of the loss in Eq. 5 is not clear to me.
Furthermore, the choice of the L_2 norm seems a bit arbitrary to enforce consistency between classifier predictions. Why not use the cross-entropy?
A5: The reason for Eq. 5 is that due to no labels in the target domain, using unsupervised learning loss is a natural strategy for domain adaptation. As the homeomorphism means  that their structure is preserved, using consistency constraint is just straight.
L2 norm is common in domain adaptation e.g., those based on the mean teacher. As suggested, we have additionally provided the results using cross-entropy at Appendix.
Q6.
While the results show an improvement over the distribution alignment strategies the method is applied to, the improvement over CAN is small in two out of three cases. This tends to show a diminishing return: As the distribution alignment method becomes increasingly effective, the impact of the proposed method decreases.
A6: Given less room for improvement left, in general boosting would become more challenging for all the methods.
In other words, when the performance starts to saturate, further gain becomes harder.
Further, the gain of our method is still good (1.4% over CAN on office-home).
Q7.
Although the authors perform several ablation studies, I suggest them to validate the loss terms they used, as discussed in comments 2.1 and 2.2 above.
A7: Thanks, we have now added the suggested studies using cross-entropy for Eq (5) at Appendix.
Q8. the paper is mostly clear, although English could be improved.
A8: Thanks for all the suggestions. We will further improve the presentation.
Q9. the description of the INN in Section 4.1, together with Fig. 2 could be clarified (the link between the figure and the text is not obvious).
A9: Great suggestion. We have improved text-figure link.
Q10. The motivation behind the loss terms in Eqs. 4 and 5 should be improved.
A10: Thanks. We have revised this motivation part.
Strengths: Theoretical justification for the topology-preserving mapping.
Integration of current ideas such as invertible neural networks as a vehicle for homeomorphic mapping.
Weaknesses: Insufficient experimental results. The Office31 dataset is twelve years old (Saenko, et al., introduced this dataset in ECCV 2020!). The performance of the proposed approach on this dataset is mixed. The performance on VISTA is better, but not by much for many categories.
The paper is written well.
Novelty is low as the proposed approach builds on invertible neural networks.
The underlying theory is of ICLR quality.
NO.
Q1. I will encourage the authors to do more experiments on challenging datasets like DomainNet.
A1: Thanks. As you suggested, we have supplemented our experiments on DomainNet, which is shown in Table 7 of Appendix in the revised version. Compared with the state-of-the-arts: MCD (Saito et al., 2018), BNM (Cui et al., 2020), SWD (Lee et al., 2019) and CGDM (Du et al., 2021), our method achieves the SOTA results.
Q2. The performance of the proposed approach on this dataset (office-31) is mixed.
A2: This is due to the diversity of transfer tasks, with less consistency across all the methods including ours.
However, our HMA(CAN) can still excel on 5 out of 6 tasks on this dataset.
Q3. The performance on VISTA is better, but not by much for many categories.
A3. We consider the gain by our method is good across the categories with an average of 0.9%. Please also note the gain of 7-8% over both latest CVPR 2022 methods CaCo and SUDA.
Overall, we set new state of the art on this challenging benchmark.
Q4. Novelty is low as the proposed approach builds on invertible neural networks.
A4. We note that the key contribution of this work is introduction of a homeomorphic alignment-based strategy, a totally different, simple yet superior framework for unsupervised domain adaptation. Whilst INN itself is not new, it is just an implementation of our framework, with the use of INN being also novel in UDA to our best knowledge.
Strength:
1. It is a novel angle in leveraging homeomorphism mapping to consider the cross-domain distribution alignment problem. And the authors considered INN to simulate the HMA and provide a practical training framework to the UDA.
2. The story is written well with several exploration questions at the beginning and guide the readers to gradually focus on the points the authors propose.
3. The coverage of the UDA is adequate and the authors conducted sufficient analysis regarding the way of alignment, i.e., marginal distribution alignment, conditional distribution alignment, bijection mapping alignment, etc.
4. There are extensive empirical study across 3 widely applied domain adaptation datasets to demonstrate the effectiveness of the proposed method.
Weakness:
1.
though INN the authors mentioned is straight-forward to verify theoretically it is a homeomorphism, it would be good from some empirical evidence to show the homeomorphism property.
2. the experiment dataset is encouraged to be conducted on DomainNet as well, where the number of domains are more and the number of classes are 365, which is an order higher than Office-31, Office-home and Visda-17.
3. the improvement from HMA+CAN to CAN is marginal, across all the three datasets evaluation.
The paper provides a novel view of using homeomorphism and leveraged INN to achieve the homeomorphism and demonstrated the success application in unsupervised domain adaptation. The illustration is clear regarding the technical pipelines with illustrative figures and concise equations.
NO.
N/A
Q1:Though INN the authors mentioned is straight-forward to verify theoretically it is a homeomorphism, it would be good from some empirical evidence to show the homeomorphism property.
A1: Thanks for this good suggestion.
However, as the learning of our model takes place in the latent high-dimensional feature space of complex image data, it is not straightforward to visualize. For conceptual illustration, we have now provided a visualization on a set of 2-dimensional structured toy data (please see A.7 in Appendix of the revised manuscript). Please let us know any suggestion on this and we are more than happy to attempt.
Q2: The experiment dataset is encouraged to be conducted on DomainNet as well, where the number of domains are more and the number of classes are 365, which is an order higher than Office-31, Office-home and Visda-17.
A2: Thanks.  As you suggested, we have supplemented our experiments on DomainNet, which is shown in Table 7 of Appendix in the revised version. Compared with the state-of-the-arts: MCD (Saito et al., 2018), BNM (Cui et al., 2020), SWD (Lee et al., 2019) and CGDM (Du et al., 2021), our method achieves the SOTA results.
Q3: The improvement from HMA+CAN to CAN is marginal, across all the three datasets evaluation.
A3: We consider that the gain by our method is already good across different benchmarks.
For example, the gain of 0.9% by ours on Visda-17, vs 0.7% by BCDM (Li et al., 2021) over its best competitor STAR (Lu et al., 2019).
Also note that, our HMA is a model generic framework, whilst most of existing methods (e.g. BCDM) are not.
Thanks for the authors response. I basically do not have further comment on this submission and remain my current rating.
We thank all the reviewers for insightful comments on our work. We appreciate the following positive feedback:
(1) It is a novel angle and the idea of learning a homeomorphism is new.  (reviewer 1hQA, UL2Y).
(2)The story is written well. (reviewer 1hQA, XjCb)
(3)The coverage of the UDA is adequate and the authors conducted sufficient analysis regarding the way of alignment. There are extensive empirical studies. (reviewer 1hQA)
(4)Theoretical justification for the topology-preserving mapping.  Integration of current ideas such as invertible neural networks as a vehicle for homeomorphic mapping. (reviewer XjCb)
(5)The method can be used with different distribution alignment strategies and improves the distribution alignment strategy it is applied to. (reviewer UL2Y)
We address all the comments of each reviewer in detail below. We thank all reviewers for the first round of comments, please feel free to let us know if further clarifications and experiments are needed. We would really appreciate it if the reviewers could consider raising the scores accordingly in reflection of our responses and updates.
Dear Reviewers,
First of all, we appreciate all the reviewing time and efforts spent on our work. All the reviews are helpful for improving our paper.
With our best appreciation, we have now made detailed response to all the thoughtful comments/suggestions.
While we consider that our current response could have addressed all the concerns raised hopefully, it is most critical that the reviewers can kindly read our response and tell us how the issues have been addressed and left concerns if any.
Any thoughtful and constructive comments/suggestions are welcome. And we will address them by our best efforts.
Best regards,
Authors
Dear Reviewers,
Please make sure that your reviews acknowledge authors’ responses and reflect your current evaluation of the paper. This is particularly important if you didn’t directly engage with the authors during the discussion phase (so the authors don’t know if their response changed your evaluation) or if you expressed an intention to update your rating but did not do so.
Cheers,
AC
Strength:
1. Introducing homeomorphism mapping for domain adaptation to prevent the deterioration of data topological structure is novel and reasonable.
2. The paper is written and structured well.
3. The numeric experimental results convincingly justify its advantages over existing methods.
Weakness:
1. No direct evidence to support that homeomorphism mapping preserves the data topological structure.
1.1. No clear definition of the claimed topological structure that is to be preserved. If the cluster/group structure denotes the topological structure, as claimed by the authors, then clustering-based or locality-preserving methods should all be able alleviate this issue. What’s the essential advantage of the HMA?
1.2. The only toy example in Figure 3 is too simple to convincingly justify the preserving of the topological structure while figure6-8 also fail to validate it.
2. The method is not theoretically supported.
3. As pointed out by other reviewers, this is a connection between the proposed homeomorphisms and optimal transport, but it is not mentioned.
4. The motivations need to be further clarified to distinguish the preservation of the topological structure of both domains and the discriminative structure of the representation.
• The paper is mostly clear and well-structured.
• The overall idea of homeomorphism mapping to preserve the structure is novel.
• Seems reproducible, with some effort for the INN.
NO.
N/A
After discussion with other reviewers, I am impressed that the experimental results still do not convincingly support the claims and justify the advantages over existing ones. The connection to Bi-classifier adversarial training and optimal transport needs further clarification.
In summary, considering that a major revision is still needed, I would like to vote for a weak rejection.
Dear reviewer X6x3:
First of all, we appreciate your reviewing time and efforts spent on our work. All the reviews are helpful for improving our paper. We appreciate the following positive feedback: (1) Introducing homeomorphism mapping for domain adaptation to prevent the deterioration of data topological structure is novel and reasonable. (2) The paper is written and structured well. (3) The numeric experimental results convincingly justify its advantages over existing methods.
For your concerns, we explain one by one as follows:
Q1: No direct evidence to support that homeomorphism mapping preserves the data topological structure.
A1: Thanks. We mathematically proved that homeomorphism mapping preserves the data topological structure, which can be found in the Proof of Theorm 1 in Section A.2.
At the same time, we also took a toy example to show this property in 2-D dimension.
Q1.1: No clear definition of the claimed topological structure that is to be preserved. If the cluster/group structure denotes the topological structure, as claimed by the authors, then clustering-based or locality-preserving methods should all be able alleviate this issue. What’s the essential advantage of the HMA?
A1.1: Thanks. 1) Topological structure is a mathematical concept, please see the details in Topology [Munkres, 2000]. Typically, a feature extractor (e.g., a typical neural network) trained in the source domain with label supervision can guarantee that the features of the same category are in one cluster. In this condition, in this work we aim to maintain this cluster structure when the source features are transformed into the target feature space. We will supplement this exact definition in the next revision.
2) Existing clustering-based or locality-preserving methods usually pose constraints on the data, but not on the projection.
As a result, they do not guarantee the topological structure preservation after projection. We will further clarify this subtle but significant differences in the revision.
Q1.2: The only toy example in Figure 3 is too simple to convincingly justify the preserving of the topological structure while figure6-8 also fail to validate it.
A1.2: 1) For conceptual illustration, we give a toy example. In subsequent versions, we will modify this experiment by introducing more data and more clusters. Please note, all the experiments on real-world data we have given are implicit validation of our method, although not visually (please suggest the ways for validation if any and we really appreciate).
2) Figure 5-8 examines the discrimination of the extracted features, and does not involve topology structure preservation. Figure 5-6 mainly further analyzes the ablation study and Figure 7-8 shows that compared with previous methods, our method can better pay attention to the category information of the image.
Q2:  The method is not theoretically supported.
A2: Our method is divided into three steps, the first step is to construct homeomorphism, the second step is to train homeomorphism, and the third step is to use the properties of homeomorphism to train feature extractors and classifiers.
For the first step, our theory 1 proves that homeomorphism can preserve the topological structure, that is, the data in one cluster is still in one cluster after transformation, and will not be redistributed in multiple clusters.
For the second step, we use existing DA methods to align the transformed source distribution with the target distribution, and the transformed target distribution with the source distribution, such that homeomorphisms can connect the same categories of data from both domains. This part can be theoretically guaranteed by previous DA methods.
For the third step, we exploit the properties of theory 1 to feature extractors and classifiers.
We therefore consider our method to be theoretically supported.
Q3: As pointed out by other reviewers, this is a connection between the proposed homeomorphisms and optimal transport, but it is not mentioned.
A3: In our current release, we added a comparison with optimal transport in A.1, such as the these methods align the distribution in a common space while our method aligns the distribution in two different spaces; these methods assume rigidly one-to-one alignment across domains while our method considers a class-wise alignment; these methods learn a transformation by solving a linear programming problem while our method constructs a homeomorphism mapping by invertible neural network.
Q4. The motivations need to be further clarified to distinguish the preservation of the topological structure of both domains and the discriminative structure of the representation.
A4: This problem was also mentioned by the reviewer UL2Y before. In order not to confuse readers, we have modified the introduction. And we will further modify relevant parts to improve the readability.
The innovation of our method is that the use of homeomorphism mapping can preserve topological structure for UDA tasks. We have proved this from both theory and experiments, and also achieved sota results on public datasets.
And recent most of the existing research focus on statistic moment matching and adversarial learning, which could serve as baselines in our framework. We also reviewed many recent plugin-like methods, such as  CaCo (Huang et al., CVPR2022) and SUDA (Zhang et al., CVPR2022), and none of which were tested on Bi-classifier adversarial training and optimal transport. This does not deny the value of evaluating these methods, and absolutely good to have. What we intent to express that this could be considered as some additional evaluation beyond a standard method-focused article, as this work is not an experimental survey paper.
However, we appreciate for this comment whilst also please allow us to comment with our respect and politeness.

Meta-review:
role: PC
data: metareview:_summary,_strengths_and_weaknesses: This paper introduces the idea of homeomorphism alignment into UDA where one find a common space which maintains discriminative structure of the source and target domain data when aligning the data distributions. 

Strength: 
Homeomorphism alignment is novel. Meanwhile, there are certain theoretical analyses to connect the Invertible Neural Network (INN) with this homeomorphism alignment, which is a good application of INN. I think this could potentially benefit the research community for more useful ideas.

Weakness:
There is indeed some experiment missing, such as CAN in Table 7. The bi-classifier-based methods are discussed but not really put in perspective of the proposed method. The authors did this, to some degree, in their response above, mentioning the distortion of the feature space. However, it would be more convincing to evidence this distortion, for example with t-SNE plots (the ones in the current version of the appendix do not show bi-classifier-based methods). The discussion of the optimal transport-based methods is clearer, but there is no clear empirical evidence of the benefits of the proposed method over the optimal transport-based ones. No direct evidence to support that homeomorphism mapping preserves the data topological structure. Section 3 on  ANALYSIS OF DISTRIBUTION ALIGNMENT based on the results on the Office-31 only. It may draw a biased conclusion if we only evaluate on one dataset.
justification_for_why_not_higher_score: Missing some important experimental results and support the claims. Fail to discuss its connection to previous works, especially the optimal transport and discussion of bi-classifier adversarial learning and optimal transport.
justification_for_why_not_lower_score: The proposed method is novel. No one explore before. There are certain theoretical analyses to connect the Invertible Neural Network (INN) with this homeomorphism alignment, which is a good application of INN.
summary_of_AC-reviewer_meeting: In the meeting, all the reviewers acknowledge its novelty in terms of methodology. The authors have addressed the concerns of more experimental results on the DomainNet. However, the CAN results are missing which weakens the results. Another concern that is still not addressed is the discussion of its connection to previous works, especially the optimal transport and discussion of bi-classifier adversarial learning and optimal transport. This weakness is pointed out by XjCb, UL2Y, and X6x3. However the reviewer 1hQA still insists that the experiment is, empirically, this method shows the top performance but not necessarily compelling better results. To this end, the method might need some further consideration.  There are certain theoretical analyses to connect the Invertible Neural Network (INN) with this homeomorphism alignment, which is a good application of INN. I think this could potentially benefit the research community for more useful ideas.

After several rounds of discussions, we all converge on rejection. 

Decision: Reject