Title: IDEA: An Invariant Perspective for Efficient Domain Adaptive Image Retrieval
Venue: NeurIPS 2023
Accepted: yes
Keywords: domain adaption

binary descriptor

causal inference
Abstract: In this paper, we investigate the problem of unsupervised domain adaptive hashing, which leverage knowledge from a label-rich source domain to expedite learning to hash on a label-scarce target domain. Although numerous existing approaches attempt to incorporate transfer learning techniques into deep hashing frameworks, they often neglect the essential invariance for adequate alignment between these two domains. Worse yet, these methods fail to distinguish between causal and non-causal effects embedded in images, rendering cross-domain retrieval ineffective. To address these challenges, we propose an Invariance-acquired Domain AdaptivE HAshing (IDEA) model. Our IDEA first decomposes each image into a causal feature representing label information, and a non-causal feature indicating domain information. Subsequently, we generate discriminative hash codes using causal features with consistency learning on both source and target domains. More importantly, we employ a generative model for synthetic samples to simulate the intervention of various non-causal effects, ultimately minimizing their impact on hash codes for domain invariance. Comprehensive experiments conducted on benchmark datasets validate the superior performance of our IDEA compared to a variety of competitive baselines.

IDEA: An Invariant Perspective for Efficient Domain Adaptive Image Retrieval

Abstract

In this paper, we study the problem of unsupervised domain adaptive retrieval, which transfers retrieval models from a label-rich source domain to a label-scarce target domain. Although there exist numerous approaches that incorporate transfer learning techniques into deep hashing frameworks, they often overlook the crucial invariance needed for adequate alignment between these two domains. Even worse, these methods fail to distinguish between causal and non-causal effects embedded in images, making cross-domain retrieval ineffective. To address these challenges, we propose an Invariance-acquired Domain Adaptive Hashing (IDEA) model. Our IDEA first decomposes each image into a causal feature representing label information and a non-causal feature indicating domain information. We then generate discriminative hash codes using causal features with consistency learning on both source and target domains. More importantly, we employ a generative model for synthetic samples to simulate the intervention of various non-causal effects, thereby minimizing their impact on hash codes for domain invariance. Comprehensive experiments conducted on benchmark datasets confirm the superior performance of our proposed IDEA compared to a variety of competitive baselines.

1 Introduction

Approximate nearest neighbor (ANN) search [73, 77, 49, 29, 40] has been a pivotal research problem within the field of deep learning for decades. Recently, learning to hash [41, 80, 68, 20] has come to the fore in ANN search engines due to its high computational and memory efficiency. The main objective of learning to hash is to convert high-dimensional data points into compact low-dimensional binary codes while maintaining semantic similarity. Consequently, this approach mitigates the requirement for computationally expensive pairwise distance computations, replacing them with economical Hamming distance calculations using bit-wise XOR and bit-count operations [60].

Recent hashing techniques can primarily be classified into two types, namely supervised approaches [72, 44, 67, 48, 75, 8, 31, 52] and unsupervised approaches [32, 76, 46, 69, 33, 61, 10]. Supervised hashing approaches incorporate annotation data into the similarity learning algorithms, generally leading to better performance than unsupervised ones. These supervised methods primarily focus on minimizing a spectrum of similarity-preserving losses and quantization losses [11, 78] to generate highly discriminative hash codes. Furthermore, several innovative proxy-based techniques have been introduced where a proxy is constructed in the Hamming space for each class, and hash

codes are enforced to approximate their respective proxies. These methods have demonstrated encouraging results in efficient image retrieval.

However, these supervised approaches rely on an assumption that the distributions between the training and query data are the same, which is often violated in real-world scenarios. For instance, when an ANN search engine trained on benchmark images, is employed to search query images from mobile phones, retrieval performance could potentially diminish substantially [45, 22, 64]. In light of this, this paper concentrates on the problem of domain adaptive retrieval, which aims to enhance the performance of retrieval systems by leveraging both labeled source instances and unlabeled target samples. In literature, a variety of domain adaptive hashing methods have been proposed [71, 66, 64, 47, 74, 21, 58], which typically incorporate domain adaptation strategies into deep hashing frameworks. They adopt the memory bank [64] and adversarial learning [58, 36] to achieve alignment across source and target domains. Moreover, pseudo-labeling [19, 58] is implemented to extract knowledge from unlabeled target samples, thus addressing the challenge of label scarcity in the target domain.

Unfortunately, existing approaches have two intrinsic limitations that hinder their effectiveness as follows: (1) Entanglement of causal and non-causal effects. Current methods tend to recklessly focus on correlations between image data and semantic labels. This could result in an entanglement of causal and non-causal factors. Consequently, the hash codes derived from these deep features may lack interpretability and be suboptimal for subsequent retrieval tasks. (2) Neglect of invariant patterns . While adversarial learning is commonly used to implicitly align hash codes between source and target domains, these methods often fail to explicitly capture the invariant patterns present in vision representations. This could lead to ineffective domain alignment of hash codes due to the influence of underlying variant factors. As a result, an effective hashing framework under the principle of invariant learning [2, 5] is highly desired.

In this paper, we explore domain adaptive hashing through a causal analysis lens, proposing a model named Invariance-Acquired Domain Adaptive Hashing (IDEA) for efficient image retrieval under domain shift. The core of domain adaptive hashing lies in discerning causal and non-causal effects within images. To generate high-quality and interpretable binary codes, we disentangle causal and non-causal features within each image, guided by the principles of the information bottleneck. On one hand, we maximize the mutual information between causal features and label information. On the other hand, we retain the most information in hidden features for non-causal features while minimizing the information in labels. These causal features are used to generate discriminative hash codes through consistency learning across both source and target domains. To further mitigate the non-causal effects, we incorporate a generative model that simulates the intervention of various non-causal effects, thereby encouraging hash codes to remain sufficiently invariant to different non-causal components. Leveraging invariant learning on causal effects, our IDEA is capable of generating domain-invariant hash codes that facilitate efficient cross-domain retrieval. Empirical experiments conducted on a variety of benchmark datasets validate the superior performance of our IDEA compared to competitive baselines. The contribution of this paper is summarized as follows:

Problem Connection. We pioneer a novel perspective that connects invariant learning with domain adaptive hashing for efficient image retrieval.

Novel Methodology. Our IDEA not only disentangles causal and non-causal features in each image following the principle of the information bottleneck, but also ensures hash codes are sufficiently invariant to the intervention of non-causal features.

High Performance. Comprehensive experiments across numerous datasets demonstrate that our IDEA outperforms a range of competitive baselines in different settings.

2 Related Works

Learning to Hash. The task of efficient image retrieval has recently witnessed significant interest [49, 29, 40, 37], leading to the development of deep hashing methods [41, 80, 68, 20], including both supervised [72, 44, 67, 48, 75, 8, 31, 52, 54] and unsupervised approaches [32, 76, 46, 69, 33, 61, 56, 10, 55]. Deep unsupervised hashing methods typically reconstruct semantic structures using similarity metrics, subsequently generating similarity-preserving binary codes. Self-supervised methods have also been developed to enhance the performance. The incorporation of label information has notably improved the performance of deep supervised hashing methods. Early attempts at optimizing the

hashing network often involved pairwise and triplet losses based on similarity structures [44, 67, 30, 4]. Another line of study has utilized point-wise optimization, establishing proxies for each category within the Hamming space and compelling hash codes to approximate these proxies [72, 13]. However, these methods typically overlook potential distribution shift in practical scenarios, which can significantly degrade retrieval performance. This limitation has prompted research into domain adaptive hashing.

Unsupervised Domain Adaptation. Unsupervised Domain Adaptation (UDA) has long been a formidable challenge in machine learning and computer vision [38, 51, 14]. Early methods often explicitly reduce distribution discrepancies for domain alignment [43, 27]. An alternative approach utilizes adversarial learning for implicit domain alignment, incorporating a gradient reversal layer and a domain discriminator to engage in a minimax game [15, 36]. Recently, researchers have extended this topic towards efficient domain adaptive retrieval [71, 66, 64, 47, 74, 21, 58, 59], proposing a number of transferable hashing methods to address this problem. These methods typically integrate domain adaptation models into deep hashing frameworks and generate similarity structures for learning semantic knowledge on the target domain. Despite these advancements, these methods often fall short due to the entanglement of causal and non-causal effects and the neglect of invariant patterns. In response to these challenges, we propose an effective approach IDEA.

Invariant Learning. Invariant learning [2, 5, 9] aims to identify invariant correlations between inputs and targets under domain shift, while concurrently eliminating spurious and variant relationships. This concept has been explored in the context of out-of-distribution generalization. Under specific assumptions, invariant learning has demonstrated significant potential for model generalization following causal theory. Invariant risk minimization [2] (IRM) has been proposed to regularize neural networks to remain stable under the environmental variance, showing superior performance when compared to empirical risk minimization [12] (ERM). Moreover, MVDC [65] learns from frequency spectrum to generate causal features for domain generalization for cross-domain object detection. Invariant learning has also been extended to address challenges in graph domains [28, 70, 6, 53, 63]. For instance, CIGA [6] leverages distinct subgraphs for graph contrastive learning, thereby achieving superior out-of-distribution graph classification performance. Our study establishes a connection between invariant learning and domain adaptive hashing, generating domain-invariant and discriminative hash codes for cross-domain image retrieval. To the best of our knowledge, this is the first work to apply invariant learning for transferable retrieval tasks, and is proven effective for potential application.

3 Problem Definition

Given a source domain D s = { ( x s i , y s i ) } N s i =1 with N s fully-labeled images and a target domain D t = { ( x t j ) } N t j =1 with unlabeled N t images, both domain share a common label space Y = { 1 , 2 , · · · , C } despite potential distribution shifts. The objective is to develop a hashing-based retrieval model that projects an input image x onto a compact binary code b ∈ {-1 , 1 } L , where L represents the code length. This model should ensure that similar images are mapped to comparable binary codes within the Hamming space, and its performance should be evaluated in both single-domain and cross-domain retrieval systems. In a single-domain retrieval system, both query and database images originate from the target domain, whereas, in a cross-domain retrieval system, the query images are drawn from the target domain D t and the database images are sourced from the source domain D s .

4 Method

4.1 Framework Overview

We address the problem of unsupervised domain adaptive retrieval. While several domain adaptive hashing algorithms have been proposed, they often neglect invariant learning during the hash code generation process. This oversight often leads to significant domain discrepancy in the resulting hash codes. More-

Y

C

D

H

X

NC

Y

:

Semantic labels

NC

:

Non-causal part

X

: Image

D

:

Relationships between domains

H

:

Hash code

C

:

Causal part

over, these methods indiscriminately focus on correlations between image data and semantic labels,

Source

domain

Non-Causal

feature

Target

domain

.

.

.

Generative

model

1

-1

-1

-1

1

-1

-1

1

-1

1

1

-1

-1

-1

1

-1

-1

1

-1

1

Hash Codes

Encoder

Causal

feature

Disentanglement

Loss

Non-Causal

feature

Intervene

Reconstruct

failing to disentangle the causal and non-causal effects encoded in images. This results in learnt hash codes that are not only suboptimal for downstream retrieval tasks, but are also difficult to interpret.

In this study, we present a novel approach named IDEA, to tackle these issues. The fundamental idea is to incorporate invariant learning into the optimization of the hashing network. Specifically, we first introduce a structural causal model, as illustrated in Figure 1, and analyze the correlation within this problem. We then separate each image into causal and non-causal features using an information bottleneck. The causal features are subsequently used for hash code production. Lastly, we introduce interventions on non-causal effects and limit their influence on invariant hash codes. More details can be found in Figure 2.

4.2 Structure Causal Model

To start, we apply a causal perspective to domain adaptive hashing and construct a structural causal model [39] (SCM) to depict the image generation process under domain shifts. This model is used to illustrate the relationships between domains D , semantic labels Y , causal parts C , non-causal parts NC , images X , and hash codes H . The specific relationships are elaborated as follows:

Y → C . Semantic labels produce the causal part which should be invariant to domain distributions.

D → NC . Domain information provides the non-causal part which is changeable across domains.

C → X ← NC . Each image is generated by combining both causal part and non-causal part.

NC C → H . To obtain domain-invariant hash codes, we should generate hash codes using causal features. However, there could be additional relationships between C and NC . For example, C and NC can be simultaneously influenced by a cause. Therefore, C acts as a confounder between NC and H , i.e., resulting in falsely related variables.

The essence of domain adaptive hashing lies in learning domain-invariant and discriminative binary codes. From the structural causal model, we need to construct mappings ϕ ( · ) and ψ ( · ) which fulfill the following conditions:

where ⊥ symbolizes the independence under the given condition. Inferred from Eqn. 1, we are required to address the following challenges: 1) The first term necessitates us to disentangle the causal and non-causal components of each image; 2) The second term requires us to learn discriminative hash codes from causal components; 3) The final term advocates for the generation of hash codes that are independent of the influence of non-causal components. With these goals in mind, we propose a novel deep hashing method IDEA for domain adaptive retrieval.

4.3 Causality-acquired Disentanglement

In this part, we apply the information bottleneck principle [1] to disentangle each image into causal and non-causal features. This approach facilitates the extraction of multifaceted latent factors embedded within images, enabling the generation of high-quality and interpretable binary codes. As suggested by the structural causal model, causal features should encapsulate label information, while non-causal features should reflect domain discrepancy.

In detail, we first introduce a feature extractor F ( · ) , which removes the last classification layer in a popular neural network backbone (e.g., VGG-F [50]) to generate hidden features, i.e., z = F ( x ) . Then, two different MLPs g c ( · ) and g n ( · ) are utilized to generate two features, respectively, i.e., f c = g c ( z ) and f n = g n ( z ) . Let F c and Y denote the random variables of causal features and semantic labels, and we maximize the mutual information between F c and Y . To achieve this, we turn to InfoNCE [18, 3, 7], which construct positive pairs and negative pairs from the joint distribution p ( F c , Y ) ∈ D s and the product of marginal distributions p ( F c ) p ( Y ) ∈ D s , respectively. Since label information is not available in D t , we merely utilize source data in this module. In particular, an estimator T c is introduced to estimate the lower bound of mutual information, the target objective is written as:

where T c ( · , · ) comes from a bi-linear function with a weight matrix W c to calculate the probability of being a positive pair, i.e., T c ( f c , y ) = f c W c y . The estimator and the causal head g c are optimized jointly to obtain effective causal features with high correlation to label information.

Moreover, to generate non-causal features, we minimize the mutual information between F n and Y with the maximum mutual information with hidden features Z following the principle of information bottleneck. In formulation, we need to minimize:

where β is a parameter to balance the loss. Similarly, it is infeasible to minimize Eqn. 3 directly. Therefore, we calculate the upper bound and lower bound of I ( F n , Y ) and I ( F n , Z ) , respectively. Here, we first define:

which can be shown as an upper bound of I ( F n , Y ) . More detail is shown in Appendix. Moreover, we introduce a variational function q ( y | f n ) to approximate p ( y | f n ) . To measure the lower bound of I ( F n , Z ) , we introduce a different estimator T n ( f n , z ) = f n W n z and the upper bound of I ( F n , Z ) is written as follows:

By combining Eqn. 16 and Eqn. 5, we rewrite the objective in Eqn. 3 as:

From Eqn. 6 , we utilize adversarial learning to optimize non-causal features, which generates reserve gradients for estimator T n when minimizing L n MI . The overall loss for disentanglement is summarized as:

By minimizing L D , we can successfully disentangle causal and non-causal elements embedded in images, which can guide us to generate high-quality and interpretable hash codes.

4.4 Consistency Learning for Discriminative Hash Codes

To learn discriminative hash codes, we leverage causal features given their strong correlation with label information for hash code generation. Here, consistency learning is introduced, based on the principle that similar images should be mapped to similar hash codes to enable effective image retrieval [4, 25]. This strategy ensures that our model can capture and preserve the inherent semantic relationships in the data, thus facilitating more accurate and effective retrieval.

In particular, we utilize an MLP ψ ( · ) to map causal features to hash codes and enforce similar images to have similar binary codes. Here, we resort to consistency learning for optimization, which constructs positives via (1) two images sharing the same label in the source domain; (2) two augmented views from the same image in the target domain. In practice, we generate two augmented views for each image in a mini-batch B = B s ∪ B t , and define the positive set for source sample as:

where ˆ x s i denotes the augmented source sample. Then, we provide a hashing consistency learning objective for source domain:

where ˆ b s i = ψ ( g c ( F (ˆ x s i ))) and τ is a temperature parameter. Moreover, the hashing consistency learning objective in the target domain is:

Finally, the consistency learning objective is derived by combining both source and target domains:

The introduction of consistency learning offers two primary benefits for hash code generation: (1) By maximizing the consistency of hash codes between similar images, our IDEA ensures the generation of similarity-preserving hash codes, which is essential for effective search engines. (2) By minimizing the similarity of hash codes derived from dissimilar images, our IDEA encourages hash codes to be uniformly distributed in the Hamming space, which enhances the capacity of each hash bit.

4.5 Invariance under Intervention

However, in practice, causal parts and non-causal-part could be still closely related. For example, waterbirds are typically paired with water backgrounds in the training set. To further reduce the potential hidden effects of non-causal parts, we introduce invariant learning [28, 70, 6], which first introduce a generative model to do interventions and then encourages hash codes to be sufficiently invariant to different non-causal parts.

In detail, a generative model G ( · , · ) is first learnt to reconstruct original images using both causal and non-causal features. To optimize the generative model, the reconstruction loss in a mini-batch is written in the formulation of:

With the well-trained model, we can simulate the intervention by utilizing different non-causal features. Here, we first generate synthetic images using f c i and f n k :

where do ( f n m ) denotes to impose non-causal features from a different sample x n m in B . Then we feed the synthetic sample into the hashing network to generate hash codes, b + m i = ψ ( g c ( F ( x + m i ))) and then minimize the variance under different intervention. The objective can be written as:

where ¯ b i = E x m ∈B [ b + m i ] is the mean of synthetic hash codes. With effective invariant learning, we can generate domain-invariant binary codes for effective cross-domain retrieval.

Algorithm 1 Training Algorithm of IDEA

Require:

Source data D s , target data D s , code length L ;

Ensure: c

The hashing network ψ ( g ( F ( · ))) ;

1: Warm up our backbone by minimizing L D and L RE ;

2: repeat

3: Sample a mini-batch B s and B t from D s and D t , respectively;

4: Generate two augmented views for each sample;

5: Calculate positive set by Eqn. 8;

6: Generate synthetic samples under intervention by Eqn. 13;

7: Calculate the final loss objective in Eqn. 15;

8: Reverse the gradients T n ;

9: Update parameters by gradient descent;

10: until convergence

4.6 Summary

Finally, we summarize our framework with the following overall training objective as:

Moreover, the derivation of sign ( · ) is zero for any non-zero value, and therefore it is difficult to compatible with the gradient propagation. To tackle this challenges, we adopt tanh ( · ) to replace sign ( · ) during optimization, which produces approximate hash codes v = tanh ( ψ ( g c ( F ( x )))) for training. Our model is first warmed up by the disentanglement module optimized by the reconstruction loss. Then we optimize the whole network with mini-batch stochastic gradient descent and the detailed progress of our IDEA can be found in Algorithm 1.

5 Experiment

5.1 Experimental Settings

Baselines. We adopt a variety of state-of-the-art approaches for performance comparison, including six unsupervised hashing approaches (i.e., SH [62], ITQ [17], DSH [34], LSH [16], SGH [24], OCH [35]) and seven transfer hashing approaches (i.e., ITQ + [79], LapITQ + [79], GTH-g [74], DAPH [21], PWCF [22], DHLing [64], and PEACE [58]). Some results of ITQ + and LapITQ + are omitted because prior information is not accessible for sample pair construction.

Datasets. Experiments are conducted on different benchmark datasets: (1) Office-Home dataset [57]: This dataset contains examples from four domains (Ar, Cl, Pr, Re), each with 65 object categories. Two domains are selected as the source and target, resulting in six transferable image retrieval tasks

P2R

C2R

R2A

R2P

R2C

D2W

IDEA

IDEA

IDEA

IDEA

IDEA

Precision

0.6

PEACE

DHLing

PWCF

Precision

0.6

PEACE

DHLing

PWCF

Precision

0.6

PEACE

DHLing

PWCF

Precision

0.6

PEACE

DHLing

PWCF

Precision

0.6

PEACE

DHLing

PWCF

Precision

0.6

0.4

0.4

0.4

0.4

0.4

0.4

IDEA

0.2

0.2

0.2

0.2

0.2

0.2

DHLing

PEACE

PWCF

0.0

0.0

0.2

0.4

Recall

0.6

0.8

1.0

0.0

0'0

0.2

0.4

Recall

0.6

0.8

1.0

0.0

0.0

0.2

0.4

Recall

0.6

0.81.0

0.0

0.0

0.2

0.4

Recall

0.6

0.8

1.0

0.0

0.0

0.2

0.4

Recall

0.6

0.8

1.0

0.0

0.0

0.2

0.4

Recall

0.6

0.8

1.0

(P2R, C2R, R2A, R2P, R2C, A2R). (2) Office-31 dataset [42]: This widely used benchmark dataset contains over 4000 examples classified into 31 classes, which are from three domains (Am, We, Ds). We randomly select two domains as the source and target, resulting in six transferable image retrieval tasks (A2D, A2W, W2D, D2A, W2A, D2W). (3) Digits dataset : We focus on MNIST [26] and USPS [23], each containing ten handwritten digits. Each sample is re-scaled to 16 × 16. We select one dataset as the source and the other as the target, resulting in two transferable image retrieval tasks (MNIST2USPS and USPS2MNIST).

Implementation Details & Evaluation metrics. We perform experiments on an A100-40GB GPU. 10% of the target examples are used as test queries and the remaining target and source samples are viewed as the training set. As for the database, source domain data is adopted in cross-domain retrieval while target domain data is adopted in single-domain retrieval. The hashing network is optimized using mini-batch SGD with momentum. The batch size is set to 36 and the learning rate is fixed as 0.001. We evaluate the retrieval performance using four common metrics: mean average precision (MAP), precision-recall curve, top-N accuracy curve, and top-N recall curve. MAP is the primary metric for evaluating retrieval accuracy, while the precision-recall curve provides an overall performance indicator at different recall levels. The top-N accuracy curve shows the accuracy for different numbers of retrieval instances. As the key metrics, MAP and precision-recall curve can reflect the overall performance of our model in retrieval tasks. We also analyze the top-N accuracy and recall curves to evaluate the performance at different number of retrieved results. These comprehensive evaluations allow us to have an in-depth understanding of the model's retrieval ability.

5.2 Empirical Results

Performance Comparison. We present the results of our experiments on several benchmark datasets in Tables 1 and Table 2, where we report the mean average precision (MAP) scores achieved by our IDEA and other state-of-the-art methods. Our analysis of the results leads to the following conclusions. On the Office-Home and Office31 datasets, our IDEA achieves an outstanding average MAP score of 57.03, which significantly outperforms the second-best method, PEACE, with an average MAP score of 51.22. Furthermore, IDEA achieves the highest MAP scores across all query types, including P2R, C2R, R2A, R2P, R2C, A2R, A2D, A2W, W2D, D2A, W2A, and D2W. On the

60

DHLing

60

PEACE

60

IDEA

Mean AveragePrecision

Sensitivity Analysis

Mean Average Precision

55

Sensitivity Analysis

40

40

40

49

A2W

20

20

20

48

C2R

D2A

W2A

0

0

0

A2D

5

-20

20

20

47

R2A

-40

-40

-40

46

60

40-30-20-10

0

10

20

30

40

-60

40-30-20-10

0

10

2030

40

-60

40-30-20-10

0

10

20

30

40

0.01 0.05

0.10

0.20

0.30

530.10

0.30

0.50

1

0.70

06°0

β

Digits dataset, IDEA consistently achieves the highest MAP scores for all hash code lengths, ranging from 16 to 128 bits. This demonstrates the consistency of our method across different code lengths. For instance, on the MNIST2USPS task, IDEA achieves an average MAP score of 74.34 for 128-bit codes, which is significantly higher than the second-best method, PEACE, with an average MAP score of 68.16. Overall, our experimental results demonstrate that the proposed IDEA is effective and outperforms state-of-the-art methods on various benchmark datasets. These findings highlight the potential of our method for practical applications in cross-domain image retrieval tasks. We present the recall-precision curves of four compared approches (PWCF, DHLing, PEACE, and IDEA) on two cross-domain retrieval tasks in Figure 3. Furthermore, in most cases, the curve of IDEA is higher than that of the other three baselines, indicating that the binary codes generated by IDEA works better for the hash table lookup strategy.

Qualitative Results. Here we provide a qualitative analysis of the model. Specifically, we use t-SNE visualization to display the hash codes learned by IDEA, PEACE, and DHLing (see the three subfigures on the left side of Figure 4). When compared to the PEACE and DHLing baseline methods, our IDEA-generated hash codes show more discriminative structures, with binary codes of different categories being better separated. These results indicate that the proposed method can produce hash codes with better discriminative power, which facilitates more effective image retrieval.

Parameter Sensitivity. In this part, we conduct sensitivity analysis for two important hyperparameters, which is shown in the right subplot of the Figure 4. We can observe that the MAP values for all three cross-domain retrieval tasks slightly decrease as the β parameter increases, indicating a negative impact on model performance. However, the decrease is not significant, allowing for an increase in β to balance loss terms without causing a significant reduction in model performance. Similarly, the MAP values increase and then decrease as τ increases, demonstrating the potential for improved model performance. However, exceeding an optimal τ range may have a negative impact on retrieval performance. Thus, determining the optimal τ value through experimentation is necessary for achieving optimal retrieval performance in practical applications. In our experiments, we set β and τ to 0 . 1 and 0 . 5 , respectively.

Ablation Study. We have introduced several variants of our IDEA to investigate the impact of each components: (1) IDEA w/o L RE removes the original image reconstruction loss from the generated model, (2) IDEA w/o L CL removes the consistency learning loss, (3) IDEA w/o L V removes the hash code variance under intervention, and (4) IDEA w/o L MI removes the mutual information loss. By summarizing the

results of these model variants in Table 3, We conclude, the ablation experiments show that removing any of the four components (i.e., L RE , L CL , L V , L MI ) leads to a drop in performance on all datasets and tasks. Notably, L RE and L CL have the most significant impact. Our IDEA achieves the best or comparable performance on all datasets and tasks, demonstrating its effectiveness in learning disentangled representations for domain generalization. Specifically, it achieves the highest MAP on Office-Home and Digits datasets and the highest accuracy on the A2W task of the Office31 dataset, outperforming the ablated variants.

6 Conclusion

This work studies the problem of domain adaptive retrieval and proposes a novel method named IDEA to solve the problem. Our IDEA first disentangle causal and non-causal features within each image, guided by the principles of the information bottleneck. Moreover, these causal features are used to generate discriminative binary codes through consistency learning across both source and target domains. To generate domain-invariant binary codes, we simulates the intervention of various non-causal effects and encourage the invariance of hash codes. Extensive experiments on benchmark datasets validate the superiority of IDEA.

Broader Impacts and Limitations. This work improves the performance cross-domain retrieval, which can benefit real-world search engineers. Moreover, our work provide a new direction of incorporating invariant learning into cross-domain retrieval problems. One limitation of our work is that IDEA cannot tackle the open-set scenarios where target samples may come from unseen classes and we would extend our IDEA to more generalization scenarios in our future works. In particular, more advanced techniques such as modes in AICG and multimodal large-scale learning can be incorporated into our IDEA to improve the generalizability in a wider range of scenarios. In addition, we would explore more detailed visualization and interpretability analysis of invariant learning in the scope of cross-modal retrieval.

References

Rana Ali Amjad and Bernhard C Geiger. Learning representations for neural network-based classification using the information bottleneck principle. IEEE Transactions on Pattern Analysis and Machine Intelligence , 42(9):2225-2239, 2019.

Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893 , 2019.

Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In ICML , pages 531-540, 2018.

Zhangjie Cao, Ziping Sun, Mingsheng Long, Jianmin Wang, and Philip S Yu. Deep priority hashing. In ACMMM , 2018.

Shiyu Chang, Yang Zhang, Mo Yu, and Tommi Jaakkola. Invariant rationalization. In ICML , pages 1448-1458, 2020.

Yongqiang Chen, Yonggang Zhang, Yatao Bian, Han Yang, Kaili Ma, Binghui Xie, Tongliang Liu, Bo Han, and James Cheng. Learning causally invariant representations for out-ofdistribution generalization on graphs. In NeurIPS , pages 22131-22148, 2022.

Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. Club: A contrastive log-ratio upper bound of mutual information. In ICML , pages 1779-1788, 2020.

Lianhua Chi, Bin Li, Xingquan Zhu, Shirui Pan, and Ling Chen. Hashing for adaptive real-time graph stream classification with concept drifts. IEEE Transactions on Cybernetics , 48(5):15911604, 2017.

Elliot Creager, Jörn-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. In ICML , pages 2189-2200, 2021.

Cheng Deng, Erkun Yang, Tongliang Liu, Jie Li, Wei Liu, and Dacheng Tao. Unsupervised semantic-preserving adversarial hashing for image search. IEEE Transactions on Image Processing , 28(8):4032-4044, 2019.

Khoa D Doan, Peng Yang, and Ping Li. One loss for quantization: Deep hashing with discrete wasserstein distributional matching. In CVPR , pages 9447-9457, 2022.

Michele Donini, Luca Oneto, Shai Ben-David, John S Shawe-Taylor, and Massimiliano Pontil. Empirical risk minimization under fairness constraints. In NeurIPS , 2018.

Lixin Fan, KamWoh Ng, Ce Ju, Tianyu Zhang, and Chee Seng Chan. Deep polarized network for supervised learning of accurate binary hashing codes. In IJCAI , 2020.

Kaituo Feng, Changsheng Li, Xiaolu Zhang, and Jun Zhou. Towards open temporal graph neural networks. arXiv preprint arXiv:2303.15015 , 2023.

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. 17(1):2096-2030, 2016.

A. Gionis, Piotr Indyk, and Rajeev Motwani. Similarity search in high dimensions via hashing. In VLDB , 1999.

Yunchao Gong, Svetlana Lazebnik, Albert Gordo, and Florent Perronnin. Iterative quantization: A procrustean approach to learning binary codes for large-scale image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence , 35:2916-2929, 2013.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR , 2020.

Tao He, Yuan-Fang Li, Lianli Gao, Dongxiang Zhang, and Jingkuan Song. One network for multi-domains: domain adaptive hashing with intersectant generative adversarial networks. In IJCAI , 2019.

Jiun Tian Hoe, Kam Woh Ng, Tianyu Zhang, Chee Seng Chan, Yi-Zhe Song, and Tao Xiang. One loss for all: Deep hashing with a single cosine similarity based learning objective. In NeurIPS , 2021.

Fuxiang Huang, Lei Zhang, and Xinbo Gao. Domain adaptation preconceived hashing for unconstrained visual retrieval. IEEE Transactions on Neural Networks and Learning Systems , 2021.

Fuxiang Huang, Lei Zhang, Yang Yang, and Xichuan Zhou. Probability weighted compact feature for domain adaptive retrieval. In CVPR , 2020.

Jonathan J. Hull. A database for handwritten text recognition research. IEEE Transactions on pattern analysis and machine intelligence , 16(5):550-554, 1994.

Qing-Yuan Jiang and Wu-Jun Li. Scalable graph hashing with feature transformation. In IJCAI , 2015.

Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NeurIPS , pages 18661-18673, 2020.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Chen-Yu Lee, Tanmay Batra, Mohammad Haris Baig, and Daniel Ulbricht. Sliced wasserstein discrepancy for unsupervised domain adaptation. In CVPR , 2019.

Haoyang Li, Ziwei Zhang, Xin Wang, and Wenwu Zhu. Learning invariant graph representations for out-of-distribution generalization. In NeurIPS , 2022.

Wen Li, Ying Zhang, Yifang Sun, Wei Wang, Mingjie Li, Wenjie Zhang, and Xuemin Lin. Approximate nearest neighbor search on high dimensional data-experiments, analyses, and improvement. IEEE Transactions on Knowledge and Data Engineering , 32(8):1475-1488, 2019.

Wu-Jun Li, Sheng Wang, and Wang-Cheng Kang. Feature learning based deep supervised hashing with pairwise labels. In AAAI , 2016.

Xuan Li, Wei Wu, Yun-Hao Yuan, Shirui Pan, and Xiaobo Shen. Online unsupervised cross-view discrete hashing for large-scale retrieval. Applied Intelligence , pages 1-13, 2022.

Yunqiang Li and Jan van Gemert. Deep unsupervised image hashing by maximizing bit entropy. In AAAI , 2021.

Qinghong Lin, Xiaojun Chen, Qin Zhang, Shaotian Cai, Wenzhe Zhao, and Hongfa Wang. Deep unsupervised hashing with latent semantic components. In AAAI , 2022.

Yuetan Lin, Deng Cai, and Cheng Li. Density sensitive hashing. IEEE Trans. Cybern. , 44:13621371, 2014.

Hong Liu, R. Ji, Jingdong Wang, and Chunhua Shen. Ordinal constraint binary coding for approximate nearest neighbor search. IEEE Transactions on Pattern Analysis and Machine Intelligence , 41:941-955, 2019.

Fuchen Long, Ting Yao, Qi Dai, Xinmei Tian, Jiebo Luo, and Tao Mei. Deep domain adaptation hashing with adversarial learning. In SIGIR , 2018.

Xiao Luo, Haixin Wang, Daqing Wu, Chong Chen, Minghua Deng, Jianqiang Huang, and Xian-Sheng Hua. A survey on deep hashing methods. ACM Transactions on Knowledge Discovery from Data , 17(1):1-50, 2023.

Xinhong Ma, Junyu Gao, and Changsheng Xu. Active universal domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8968-8977, 2021.

Judea Pearl et al. Models, reasoning and inference. Cambridge, UK: CambridgeUniversityPress , 19(2), 2000.

Liudmila Prokhorenkova and Aleksandr Shekhovtsov. Graph-based nearest neighbor search: From practice to theory. In ICML , pages 7803-7813, 2020.

Chaitanya Ryali, John Hopfield, Leopold Grinberg, and Dmitry Krotov. Bio-inspired hashing for unsupervised similarity search. In ICML , pages 8295-8306, 2020.

Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In Computer Vision-ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part IV 11 , pages 213-226. Springer, 2010.

Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In CVPR , 2018.

Fumin Shen, Xin Gao, Li Liu, Yang Yang, and Heng Tao Shen. Deep asymmetric pairwise hashing. In ACMMM , 2017.

Xiaohui Shen, Zhe Lin, Jonathan Brandt, and Ying Wu. Mobile product image search by automatic query object extraction. In ECCV , 2012.

Yuming Shen, Jie Qin, Jiaxin Chen, Mengyang Yu, Li Liu, Fan Zhu, Fumin Shen, and Ling Shao. Auto-encoding twin-bottleneck hashing. In CVPR , 2020.

Weiwei Shi, Yihong Gong, Badong Chen, and Xinhong Hei. Transductive semisupervised deep hashing. IEEE Transactions on Neural Networks and Learning Systems , 2021.

Yang Shi, Xiushan Nie, Xingbo Liu, Li Zou, and Yilong Yin. Supervised adaptive similarity matrix hashing. IEEE Transactions on Image Processing , 31:2755-2766, 2022.

Harsha Vardhan Simhadri, George Williams, Martin Aumüller, Matthijs Douze, Artem Babenko, Dmitry Baranchuk, Qi Chen, Lucas Hosseini, Ravishankar Krishnaswamny, Gopal Srinivasa, et al. Results of the neurips'21 challenge on billion-scale approximate nearest neighbor search. In NeurIPS Competitions and Demonstrations Track , pages 177-189, 2022.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Ankit Singh. Clda: Contrastive learning for semi-supervised domain adaptation. In NeurIPS , pages 5089-5101, 2021.

Shupeng Su, Chao Zhang, Kai Han, and Yonghong Tian. Greedy hash: Towards fast optimization for accurate hash coding in cnn. In NeurIPS , 2018.

Yongduo Sui, Xiang Wang, Jiancan Wu, Min Lin, Xiangnan He, and Tat-Seng Chua. Causal attention for interpretable and generalizable graph classification. In KDD , pages 1696-1705, 2022.

Rong-Cheng Tu, Xian-Ling Mao, Jia-Nan Guo, Wei Wei, and Heyan Huang. Partial-softmax loss based deep hashing. In Proc. of WWW , pages 2869-2878, 2021.

Rong-Cheng Tu, Xian-Ling Mao, Kevin Qinghong Lin, Chengfei Cai, Weize Qin, Wei Wei, Hongfa Wang, and Heyan Huang. Unsupervised hashing with semantic concept mining. SIGMOD , 1(1):1-19, 2023.

Rong-Cheng Tu, Xianling Mao, and Wei Wei. Mls3rduh: Deep unsupervised hashing via manifold based local semantic similarity structure reconstructing. In IJCAI , 2020.

Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR , 2017.

Haixin Wang, Jinan Sun, Xiao Luo, Wei Xiang, Shikun Zhang, Chong Chen, and Xian-Sheng Hua. Towards effective domain adaptive retrieval. IEEE Transactions on Image Processing , 2023.

Haixin Wang, Jinan Sun, Xiang Wei, Shikun Zhang, Chong Chen, Xian-Sheng Hua, and Xiao Luo. Dance: Learning a domain adaptive framework for deep hashing. In TheWebConf , pages 3319-3330, 2023.

Meng Wang, Tongtong Wu, and Guilin Qi. A hash learning framework for search-oriented knowledge graph embedding. In ECAI , pages 921-928. 2020.

Ziwei Wang, Han Xiao, Yueqi Duan, Jie Zhou, and Jiwen Lu. Learning deep binary descriptors via bitwise interaction mining. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Yair Weiss, Antonio Torralba, and Rob Fergus. Spectral hashing. In NeurIPS , 2008.

Ying-Xin Wu, Xiang Wang, An Zhang, Xiangnan He, and Tat-Seng Chua. Discovering invariant rationales for graph neural networks. In ICLR , 2022.

Haifeng Xia, Taotao Jing, Chen Chen, and Zhengming Ding. Semi-supervised domain adaptive retrieval via discriminative hashing learning. In ACMMM , 2021.

Mingjun Xu, Lingyun Qin, Weijie Chen, Shiliang Pu, and Lei Zhang. Multi-view adversarial discriminator: Mine the non-causal factors for object detection in unseen domains. In CVPR , pages 8103-8112, 2023.

Renjun Xu, Pelen Liu, Yin Zhang, Fang Cai, Jindong Wang, Shuoying Liang, Heting Ying, and Jianwei Yin. Joint partial optimal transport for open set domain adaptation. In IJCAI , 2021.

Cheng Yan, Guansong Pang, Xiao Bai, Chunhua Shen, Jun Zhou, and Edwin Hancock. Deep hashing by discriminating hard examples. In ACMMM , 2019.

Chenggang Yan, Biao Gong, Yuxuan Wei, and Yue Gao. Deep multi-view enhancement hashing for image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43(4):1445-1451, 2020.

Erkun Yang, Tongliang Liu, Cheng Deng, Wei Liu, and Dacheng Tao. Distillhash: Unsupervised deep hashing by distilling data pairs. In CVPR , 2019.

Nianzu Yang, Kaipeng Zeng, Qitian Wu, Xiaosong Jia, and Junchi Yan. Learning substructure invariance for out-of-distribution molecular representations. In NeurIPS , 2022.

Chaohui Yu, Jindong Wang, Yiqiang Chen, and Zijing Wu. Accelerating deep unsupervised domain adaptation with transfer channel pruning. In 2019 International Joint Conference on Neural Networks (IJCNN) , pages 1-8. IEEE, 2019.

Li Yuan, Tao Wang, Xiaopeng Zhang, Francis EH Tay, Zequn Jie, Wei Liu, and Jiashi Feng. Central similarity quantization for efficient image and video retrieval. In CVPR , 2020.

Jing Zhang, Xiaokang Zhang, Jifan Yu, Jian Tang, Jie Tang, Cuiping Li, and Hong Chen. Subgraph retrieval enhanced model for multi-hop knowledge base question answering. In ACL , 2022.

Lei Zhang, Ji Liu, Yang Yang, Fuxiang Huang, Feiping Nie, and David Dian Zhang. Optimal projection guided transfer hashing for image retrieval. IEEE Transactions on Circuits and Systems for Video Technology , 30:3788-3802, 2020.

Qi Zhang, Liang Hu, Longbing Cao, Chongyang Shi, Shoujin Wang, and Dora D Liu. A probabilistic code balance constraint with compactness and informativeness enhancement for deep supervised hashing. 2022.

Wanqian Zhang, Dayan Wu, Yu Zhou, Bo Li, Weiping Wang, and Dan Meng. Deep unsupervised hybrid-similarity hadamard hashing. In ACMMM , 2020.

Liangli Zhen, Peng Hu, Xu Wang, and Dezhong Peng. Deep supervised cross-modal retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019.

Feng Zheng, Cheng Deng, and Heng Huang. Binarized neural networks for resource-efficient hashing with minimizing quantization loss. In IJCAI , pages 1032-1040, 2019.

Joey Tianyi Zhou, Heng Zhao, Xi Peng, Meng Fang, Zheng Qin, and R. Goh. Transfer hashing: From shallow to deep. IEEE Transactions on Neural Networks and Learning Systems , 29:6191-6201, 2018.

Xiaosu Zhu, Jingkuan Song, Yu Lei, Lianli Gao, and Hengtao Shen. A lower bound of hash codes' performance. In NeurIPS , pages 29166-29178, 2022.

A The Upper Bound of Mutual Information

We define:

Then, we show that ˆ I ( F n , Y ) is an upper bound of I ( F n , Y ) . In formulation, we calculate their difference as follows:

where the last inequality holds due to Jensen's Inequality with a convex function log( · ) .

B The Details of Baselines

These baselines for model comparison are introduced as below:

SH [62] is based on the theory of graph Laplacian. The binary code of a new data point is efficiently computed by the spectral analysis, enabling the out-of-distribution extension for unseen samples.

ITQ [17] is an uncomplicated and effective alternative minimization algorithm. It can be incorporated into both supervised and unsupervised learning processes.

LSH [16] is an important method in compressing high-dimensional data. It aims to map similar data points into the same bucket.

DSH [34] is a model variant of LSH, which generates multiple view of the data point for metric learning using random projections.

SGH [24] aims to compress high-dimensional data in a bit-wise manner, which is efficient for large-scale semantic similarity learning.

OCH [35] leverage the tensor product to generate an ordinal graph, which guides the learning to generate more related hash codes.

ITQ + [79] is the early effort to implement transfer learning for hashing study, which learns a slack mapping from the source domain and achieves domain alignment through the quantization loss.

LapITQ + [79] is an extension of ITQ + [79] by transferring the geometric relationship in the source domain to generate high-quality hash codes.

GTH-g [74] selects the best hash mapping for target data using source data following the principle of maximum likelihood estimation.

DAPH [21] utilizes a domain-invariant feature projection to reduce the domain discrepancy.

PWCF [22] leverages a Bayesian model to learn discriminative hash codes and then infer the similarity structure using histogram features.

DHLing [64] utilizes learning to cluster to optimize hash code within a single domain and then reduces the domain shift using the memory bank.

PEACE [58] learns the semantics on target data using pseudo-labeling techniques and then minimizes the domain shift using both implicit and explicit manners.

C The Details of Datasets

Three popular benchmarks, i.e., Office-Home dataset, Office-31 and Digits, are adopted for performance comparison. Their details are elaborated as follows:

Office-Home dataset [57] consists of image samples from four different domains, i.e., Art (A), Clipart (C), Product (P) and Real-World (R). There are 65 categories in the dataset. We select two domains as the source and target, respectively, which can produce six transferable image retrieval tasks (P2R, C2R, R2A, R2P, R2C, A2R).

Office-31 dataset [42] is comprised of 4110 image samples from three different domains, i.e., Amazon (A), Webcam (W) and Dslr (D). Each domain corresponds to a platform where the images are collected. Similarly, we randomly select two domains as the source and target, which can result in six transferable image retrieval tasks (A2D, A2W, W2D, D2A, W2A, D2W).

Digits consists of two datasets, MNIST [26] and USPS [23] and each contains handwritten digits from ten classes. The digits in both datasets are captured under different conditions. Here we consider two transferable image retrieval tasks (MNIST2USPS and USPS2MNIST).

D Additional Experimental Results

D.1 Performance of Single-domain Retrieval

We first explore the results on single-domain retrieval, Table 4 compares the average mean average precision (MAP) of different hashing methods under different bit lengths for the single-domain retrieval task, i.e., Amazon → Dslr. The table is divided into two parts: the first part shows the performance of unsupervised hashing methods, including SH, ITQ, DSH, LSH, SGH, and OCH, while the second part shows the performance of transfer learning hashing methods, including ITQ+, LapITQ+, GTH-g, DAPH, PWCF, DHLing, and PEACE. Additionally, the results of IDEA method are highlighted with bold font and pink background, indicating that it achieves the highest MAP values across all hash code lengths. The table shows that IDEA performs better than other hashing methods for simplifying image descriptions and improving image retrieval efficiency. As the hash

A2R

A2D

A2W

IDEA

IDEA

IDEA

PEACE

PEACE

PEACE

DHLing

DHLing

DHLing

Precision

0.6

PWCF

Precision

0.6

PWCF

Precision

0.6

PWCF

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0.0

0.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Recall

Recall

Recall

W2D

D2A

W2A

IDEA

IDEA

IDEA

PEACE

PEACE

PEACE

DHLing

BuTHI

uTH

Precision

0.6

PWCF

Precision

0.6

PWCF

Precision

0.6

PWCF

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0.0

0.0

0.0

0.2

0.4(

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Recall

Recall

Recall

code length increases, the MAP value of IDEA also increases, ultimately surpassing the performance of all other hashing methods at various hash code lengths. This indicates that IDEA has stronger feature extraction and hashing encoding capabilities and has significant practical value for image retrieval.

Table 5 compares the average MAP of different hashing methods under different bit lengths for the single-domain retrieval task, i.e., Product → Real. We can see that IDEA performs better than other hashing methods at all hash code lengths, especially at higher bit lengths such as 96, 128. Furthermore, the table highlights the results of IDEA in bold font and pink background, indicating that it achieves the highest MAP score at all bit lengths, namely 34.88, 44.83, 48.38, 49.91, 53.55, and 54.40. This further highlights the strong performances and advantages of IDEA, especially in learning compact hash codes and improving retrieval efficiency for single-domain retrieval tasks.

Table 6 compares the average MAP of different hashing methods under different bit lengths for the single-domain retrieval task MNIST → USPS. We can see that IDEA achieves the highest MAP values compared to other methods at all bit lengths, especially at higher bit lengths such as 96 and 128 bits, with MAP values of 74.47 and 76.73 respectively, far exceeding the MAP values of other methods. Additionally, the table highlights the results of IDEA in bold font and pink background, indicating that it achieves the highest MAP values at all bit lengths. Therefore, combining the information from the table, we can conclude that IDEA demonstrates outstanding performance in the

P2R

C2R

R2A

40

20

0.60

0.6

IDEA

0.6

IDEA

0.55

IDEA

Precision

0.5

PEACE

PEACE

0.50

PEACE

0.4

DHLing

0.4

DHLing

DHLing

PWCF

0.3

PWCF

PWCF

0.3

0.2

0.30

0.2

0.1

0.25

0.1

0.0

0.20

0

50 100 150 200 250 300 350 400

0

50100150 200 250300350400

0

50 100 150 200 250300350 400

NumberofTopRanked Samples

Number ofTopRankedSamples

NumberofTopRanked Samples

R2P

R2C

A2R

20

0.50

0.6

IDEA

0.45

IDEA

0.5

IDEA

PEACE

Precision

0.40

PEACE

PEACE

DHLing

0.35

DHLing

DHLing

0.3

0.4

PWCF

0.25

0.30

PWCF

0.3

PWCF

0.20

0.2

0.2

0.15

0.1

0.10

0.1

050 100 150 200 250 300 350 400

050 100 150 200 250 300 350 400

050 100 150 200 250 300 350 400

NumberofTopRankedSamples

NumberofTopRankedSamples

NumberofTopRankedSamples

A2D

A2W

W2D

0.6

IDEA

0.5

IDEA

0.8

IDEA

20

PEACE

PEACE

PEACE

0.4

DHLing

DHLing

0.5

DHLing

0.3

PWCF

0.3

PWCF

0.4

PWCF

0.3

0.2

0.2

0.2

0.1

0.1

0.1

50 100 150 200 250 300 350 400

050 100 150 200 250 300 350 400

050 100 150 200 250 300 350 400

NumberofTopRankedSamples

Number of Top Ranked Samples

NumberofTopRankedSamples

D2A

W2A

D2W

0.6

0.6

0.9

Precision

0.5

PEACE

IDEA

Precision

0.5

PEACE

IDEA

L0

0.8

PEACE

IDEA

0.4

DHLing

0.4

DHLing

0.6

0.5

DHLing

0.3

PWCF

0.3

PWCF

0.41

PWCF

0.2

0.2

0.3

0.2

0.1

0.1

0.1

0

50100150200250300350400

050 100 150 200 250 300 350 400

0

50100 150200250300350400

NumberofTopRankedSamples

NumberofTopRankedSamples

NumberofTopRankedSamples

single-domain retrieval task from MNIST → USPS, significantly enhancing retrieval efficiency and accuracy. This result suggests that IDEA can be a promising approach for improving image retrieval in similar domains.

D.2 Performance of Precision-Recall Curves

Additional Precision-Recall curves for six image retrieval tasks, namely A2R, A2D, A2W, W2D, D2A, and W2A, can be found in this section. These tasks involve retrieving images from one domain and matching them to images from another domain. The Precision-Recall curves offer a visual representation of the balance between precision and recall for each task. As illustrated in Figure 5, these curves give insight into the effectiveness of our approach in dealing with different cross-domain retrieval scenarios. In particular, the curves highlight the strengths of our method in achieving high precision and recall values for each task. The supplementary material provides a detailed analysis of the experimental results, which further demonstrates the robustness and generalized ability of our approach. Overall, these additional results reinforce the effectiveness of our method in improving cross-domain image retrieval performance.

D.3 Performance of Top-N Precision Curve and Top-N Recall Curve

The Top-N precision curve and Top-N recall curve depicted in Figure 6 and Figure 7, respectively, provide valuable insights into the performance of our IDEA algorithm compared to other methods in

P2R

C2R

R2A

0.9

0.9

0.8

0.8

IDEA

IDEA

0.7

0.61

PEACE

Precision

0.8

PEACE

PEACE

IDEA

0.5

0.41

DHLing

0.6

DHLing

DHLing

PWCF

0.4

PWCF

0.2

PWCF

0.2

0.2

0.1

0.1

050 100 150 200 250 300 350 400

050 100 150 200 250 300 350 400

050 100 150 200 250300350 400

Number ofTop Ranked Samples

Number ofTop Ranked Samples

R2P

R2C

A2R

1.0

IDEA

0.8

0.8

0.8

Precision

Precision

PEACE

Precision

0.6

IDEA

0.6

DHLing

0.6

IDEA

.0.41

PEACE

DHLing

0.4

PWCF

0.4

PEACE

DHLing

0.2

PWCF

0.2

0.2

PWCF

0

50100 150200250300350400

0

50 100 150 200 250 300 350 400

0

50100 150200250300350400

NumberofTopRankedSamples

NumberofTop RankedSamples

Number ofTopRanked Samples

A2D

A2W

W2D

1.0

IDEA

0.9

IDEA

IDEA

Precision

0.8

PEACE

0.8

PEACE

Precision

0.8

PEACE

0.6

DHLing

0.6

DHLing

0.6

DHLing

0.5

0.41

PWCF

PWCF

PWCF

0.4

0.3

0.2

0.2

0.2

0.1

050100 150 200 250300350 400

0

50 100 150 200 250 300350 400

0

50 100 150 200 250 300 350 400

NumberofTopRankedSamples

NumberofTopRanked Samples

NumberofTopRankedSamples

D2A

W2A

D2W

0.9

1.0

Precision

0.7

0.8

PEACE

IDEA

0.8

0.9

PEACE

IDEA

Precision

0.8

PEACE

IDEA

0.5

0.6

DHLing

0.6

0.51

DHLing

0.6

DHLing

0.41

PWCF

0.41

PWCF

10.4

PWCF

0.3

0.3

0.2

0.2

0.2

050 100 150 200 250300 350 400

0

50100 150200250300350400

0

50100 150200250300350400

NumberofTopRankedSamples

NumberofTopRankedSamples

NumberofTopRankedSamples

image retrieval tasks. Based on the analysis of these curves, we can conclude that our IDEA algorithm outperforms other comparison methods. This is attributed to the fact that the calculation of the Top-N precision and Top-N recall curves is based on the Hamming distance ranking, and the IDEA algorithm achieves better performance under this metric. Therefore, from the perspective of evaluation metrics, the IDEA algorithm is superior to other comparison methods, as it offers superior precision and recall values for various cross-domain retrieval tasks. Our extensive experimental results further validate the effectiveness of our approach in improving the performance of cross-domain image retrieval.

D.4 Ablation Study

We conduct more ablation studies on more datasets and the results are shown in Table 7. The results can validate our superiority again.

Phone

Bike

Desktop

computer

Ours

P@10

100%

PEACE

P@10

90%

PEACE

P@10

80%

PEACE

P@10

60%

Ours

P@10

90%

Ours

P@10

100%

Query

Top 10 Retrieved Images

六场

DELL

D.5 Performance of Different Architectures

We compare the performance of both PEACE and IDEA with different network backbones, i.e., VGGF and ViT. The compared results are shown in Table 8. From the results, we can have the following observations. Firstly, our IDEA can still achieve better performance under different architectures. Secondly, with strong networks, the performance of our IDEA can improve a lot.

D.6 Case Study

Figure 8 demonstrates the top 10 retrieval results of the proposed IDEA and the best baseline method PEACE for three query samples on the Office-31 dataset. The blue box indicates the query, the green box indicates the correctly retrieved result, and the red box indicates the incorrectly retrieved result. The results demonstrate that IDEA performs better than PEACE in providing more relevant and user-desired examples, with the latter typically used for cross-domain retrieval. For instance, compared to PEACE, our IDEA returns more examples of the query "Phone," further validating its success in image retrieval tasks. Overall, these findings emphasize the effectiveness of our IDEA method in improving cross-domain image retrieval performance.

Peer reviews:
comments: SUMMARY:
This paper proposes to use invariant learning based on causality inference for domain adaptive retrieval. In the proposed IDEA model, a feature disentanglement module is deployed for obtaining causal and non-causal features. A generative model is designed with non-causal features intervention for reconstructing images. Experiments show the good effectiveness.

STRENGTHS:
+Deploying casuality perspective into domain adaptive hashing retrieval is a new point.
+The proposed model is simple in methodology with encoder and decoder under intervention.
+Experiments and comparisons to related methods show the superiority of the proposed model.

WEAKNESSES:
-How about the experimental setting? Is the setting the same as some related models, such as PWCF [22]? because the performance is much better than previous ones. It can be discussed.
-Causality perspective is a common sense in domain generalization and out-of-distribution, although it is under-studied in domain adaptive hashing retrieval.
-Please discuss the following work, because this work also considers to remove non-causal features. The idea is a little similar. Additionally, the authors claim that the non-causal features are domain information, but it may not be correct, because non-casual feature may still be domain invariant information. Causal feature should be contained in domain invariant features.
"Multi-view Adversarial Discriminator: Mine the Non-causal Factors for Object Detection in Unseen Domains, CVPR 2023."

QUESTIONS:
I would lean to accept this paper, after addressing the above point in weakness, because I think this paper has some novelty.

LIMITATIONS:
I have not found obvious issue.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on an unsupervised domain adaptation method for deep hashing. This paper proposes to disentangle each image into causal and non-causal features, where casual features represent label information and non-causal features represent domain information. The causal features are used to compute hash codes, while both causal and non-causal features are used to reconstruct the image. The disentanglement loss is to minimize the mutual information between non-causal features and label and maximize the mutual information between non-causal features and the hidden features. As a result, the proposed method performs better than prior methods in multiple benchmarks.\n\nSTRENGTHS:\n- the proposed method outperforms state-of-the-art methods on cross-domain retrieval\n- the structural causal model on learning hash codes and the disentanglement loss on learning hash codes are interesting. This way is intuitively able to disentangle causal and non-causal features from an image. I believe this can be one of the regularization term for future retrieval/hashing methods.\n\nWEAKNESSES:\n- while using non-causal features from other samples for the reconstruction of hash codes seems intuitive, no experiments or analysis on the proposed intervention scheme\n- it is unknown whether the extracted features for hash codes are indeed the causal part. \n- no interpretability analysis. While the author claims that their method can generate high-quality interpretable hash codes, the only evaluation is on performance analysis, with no other quantitative and qualitative analysis to claim that the hash codes are interpretable.\n- no visualization of the causal and non-causal features, not interpretable on how those features correlate to which part of the image\n- although testing with VGG features for unsupervised hashing is a standard since this method involves disentangling features, the author should test with more different kinds of architectures such as ResNet or Vision Transformer. It is because VGG features are globally pooled features, thus already not directly interpretable.\n- no synthetic samples have been shown. As the author uses an intervention scheme on generating images and then learns to extract hash codes that have low variance, what will the samples look like?\n\nQUESTIONS:\n- how effective is this proposed method on vision transformer-based method where attention mechanism is involved? will causal features only attend to certain parts? will it overlap with non-causal features?\n- can synthetic images show that different non-causal features can generate different images from different domains? \n- why do you need a warm-up for learning Eq. 15? What causes the instability at the beginning of the training?\n\nLIMITATIONS:\n- the author mentioned that their method cannot tackle the open-set scenarios, the proposed solution is for seen classes scenario only", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper proposes a novel method called Invariance-acquired Domain Adaptive Hashing for generating high-quality and interpretable hash codes. The approach incorporates the concepts of causal and non-causal features, leveraging the information bottleneck principle and consistency learning to optimize the hash network. The paper presents the training algorithm and experimental settings for IDEA and provides qualitative and quantitative analysis of the experimental results. Overall, this paper proposes a novel approach to address certain challenges in hash code generation and achieves promising performance on multiple benchmark datasets.

STRENGTHS:
1. A structure causal model is adopted to explain the problem of unsupervised domain adaptation image retrieval.
2. This paper employs mutual information as a metric to achieve feature disentanglement, which provides a solid theoretical foundation for the paper.
3. The experimental results demonstrate that the proposed method outperforms the compared methods with a large margin.

WEAKNESSES:
1. The contribution of the structure causal model is not clear. In fact, there is no usage of casual inference for image retrieval.
2. The paper does not provide the convergence analysis of mutual information. Mutual information is sensitive to noise and outliers in the data. Thus, it is necessary to verify the convergence of the proposed method.
3. In the sensitivity analysis, the paper does not provide the recommend range of \beta and \tao.

QUESTIONS:
1. More clear contribution.
2. Convergence analysis.

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- In this research, the authors investigate the problem of unsupervised domain adaptation for hashing, which aims to expedite learning on a target domain with limited label information by leveraging knowledge from a source domain with abundant labels. The IDEA model begins by decomposing each image into a causal feature that captures label information and a non-causal feature that represents domain information. The authors then utilize consistency learning on both source and target domains to generate discriminative hash codes based on the causal features. Additionally, the authors employ a generative model for synthetic samples to simulate various non-causal effects, ultimately minimizing their impact on the domain invariant hash codes.\n- The authors conduct extensive experiments on benchmark datasets to evaluate the performance of their IDEA model against a variety of competitive baselines. The results demonstrate that the IDEA approach outperforms the others, showcasing its superiority in handling unsupervised domain adaptation for hashing.\n\nSTRENGTHS:\n- Clarity: The authors have clearly and concisely explained their methods, results, and conclusions in a way that is easy for other researchers to understand and replicate.\n- Quality of analysis: The authors have conducted extensive experiments and analyses, and the experimental data demonstrate the effectiveness and persuasiveness of the IDEA method.\n- Theoretical analysis: The authors have provided a thorough theoretical analysis of the research problem, exploring new insights and frameworks that contribute to the advancement of knowledge in the field.\n\nWEAKNESSES:\n- from the Ablation Study(line 310) on six benchmark cross-domain retrieval tasks, it can be observed that the effectiveness of these four Loss functions cannot be well demonstrated. It is suggested to consider incorporating more permutations and combinations in order to verify the validity of each component.\n\nQUESTIONS:\n1. The IDEA pipeline has numerous modules and loss functions, which may make it challenging to transfer and fine-tune for other tasks or datasets. Could the authors provide some explanations and clarifications on this matter?\n\nLIMITATIONS:\n- One potential drawback of this approach is that it may not be effective in open-set scenarios, where target samples could potentially come from unseen classes.\n- To address this limitation, more advanced techniques and models (such as those in AICG and multimodal large-scale models) can be incorporated into the research work to improve the generalizability of the method in a wider range of scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

comments: SUMMARY:
This work studies apply hashing for domain adaptive image retrieval. Specifically, the authors propose Invariance-acquired Domain Adaptive Hashing (IDEA) to consider alignment invariance, and causal effects. IDEA decomposes each image into causal and non-causal features, and introduce invariant learning to minimize the variance under different intervention. The experiments validate the effectiveness of the proposed method.

STRENGTHS:
- The problem is interesting.
- The techniques seem correct.

WEAKNESSES:
- The motivation of this work seems to be not stated clearly. The author claim invariance needed, however it lacks explanation in whether and why it is important in hashing. It also applies for the use of causal feature.

- This work is a simple combination of several techniques, e.g., contrastive learning, causal learning, invariance learning. It does not provide new insights to me.

- I have some concerns on the empirical results. In ablation study, the most important modules claimed in this work, i.e., L_V, L_MI have little improvement on the performance, and thus the importance of the two modules is questionable.

QUESTIONS:
Plz see weakness

LIMITATIONS:
The work discusses the limitations of this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-12