IDEA: An Invariant Perspective for Efficient Domain Adaptive Image Retrieval
Authors: Haixin Wang
Hao Wu
Jinan Sun
Shikun Zhang
Chong Chen
Xian-Sheng Hua
Xiao Luo
Venue: NeurIPS 2023 poster (2023)
Keywords: domain adaption
binary descriptor
causal inference

Abstract: In this paper, we investigate the problem of unsupervised domain adaptive hashing, which leverage knowledge from a label-rich source domain to expedite learning to hash on a label-scarce target domain. Although numerous existing approaches attempt to incorporate transfer learning techniques into deep hashing frameworks, they often neglect the essential invariance for adequate alignment between these two domains. Worse yet, these methods fail to distinguish between causal and non-causal effects embedded in images, rendering cross-domain retrieval ineffective. To address these challenges, we propose an Invariance-acquired Domain AdaptivE HAshing (IDEA) model. Our IDEA first decomposes each image into a causal feature representing label information, and a non-causal feature indicating domain information. Subsequently, we generate discriminative hash codes using causal features with consistency learning on both source and target domains. More importantly, we employ a generative model for synthetic samples to simulate the intervention of various non-causal effects, ultimately minimizing their impact on hash codes for domain invariance. Comprehensive experiments conducted on benchmark datasets validate the superior performance of our IDEA compared to a variety of competitive baselines.

DEE:: An Invariant Perspective for Efficient Domain Adaptive Image Retrieval
Sun',S Zhang', Chong Chen', Xian-Sheng Hua Xiao Luo Peking University, UUniversity of Science and Technology of China, TTerminus Group, "University of California, Los Angeles wang hx@stu. pku. edu. en, wuhao20220mail. .ssc..tcc.... edu. cn, jj@ppu.. edu. en, zhangek@pku. edu. en, chenchong. cz@gmail. com, huaxiansheng@guue com, xiaoluoθcs www.uclal........ edu
Haixin Wang Hao Wu Jinan Sun', Shikun
Abstract
In this paper, study the problem of unsupervised domain adaptive retrieval, which transfers retrieval models from a label-rich source domain to a label-scarce target domain. Although there exist numerous approaches that incorporate transfer learning techniques into deep hashing frameworks, they often overlook the crucial invariance needed for adequate alignment between these two domains. Even worse, these methods fail to distinguish between causal and non-causal effects embedded in images, making cross-domain retrieval ineffective. To address these challenges, we propose an Invariance anvariace-aaquiree Domain Adaptive Hashing (IDEA) model. Our IDEA first decomposes each image into causal feature representing label information and a non-causal feature indicating domain information. We then generate diseriminative hash codes using causal features with consistency learning on both source and target domains. More importantly, we employ generative model for synthetic samples to simulate the intervention of various non-causal effects, thereby minimizing their impact on hash codes for domain invariance. Comprehensive experiments conducted on benchmark datasets confirm the superior performance of our proposed IDEA compared to a variety of competitive baselines.
Introduction
40 has been a pivotal research problem within the field of deep learning for decades. Recently, learning to hash 80 68 20 to the fore in ANN search engines due to its high computational and memory efficiency. The main objective of learning hash is to convert high-dimensional data points into compact low-dimensional binary codes while maintaining semantic similarity. Consequently, this approach mitigates the requirement for computationally expensive pairwise distance computations, replacing them with economical Hamming distance calculations using bit-wise XOR and bit-count operations [60]
Approximate nearest neighbor (ANN) search 73 77
49
has come
Recent hashing techniques can primarily be classified into two types, namely supervised ap- 48
proaches 722 67
75
g
31 52 and unsupervised approaches [32
76
46
69
33
10]
Supervised hashing approaches incorporate annotation data into the similarity learning algorithms, generally leading to hetter performance than unsupervised ones. These supervised methods primarily focus on minimizing a spectrum of mmmilarity-preservinn losses and quantization losses [[T 78 to generate highly discriminative hash codes. Furthermore, several innovative proxy-basee techniques have been introduced where proxy constructed in the Hamming space for each class, and hash
"Equal contribution. 'Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

codes are enforced to approximate their respective proxies. These methods have demonstrated encouraging results in efficient image retrieval.
However, these supervised approaches rely on an assumption that the distributions between the training and query data the same, which is often violated in real-world scenarios. For instance, when an ANN search engine trained on benchmark images, is employed to search query images from mobile phones, retrieval performance could potentially diminish substantially [45 [2] 64).
In light of this, this paper concentrates on the problem of domain adaptive retrieval, which aims to enhance the performance of retrieval systems by leveraging both labeled source instances and unlabeled target samples. In literature, variety of domain adaptive hashing methods have been proposed [71] 66 64 47 (4 (1, which typically incorporate domain adaptation strategies into
deep hashing frameworks. They adopt the memory bank [64] and adversarial learning [58 [6]
to achieve alignment across source and target domains. Moreover, peeudo-labeling 19 58 is
implemented to extract knowledge from unlabeled target samples, thus addressing the challenge of
label scareity the target domain.
Unfortunately, existing approaches have two intrinsic limitations that hinder their effectiveness as follows: (1) Entanglement of causal and non-causal effects. Current methods tend to recklessly focus on correlations between image data and semantic labels. This could result in an entanglement of causal and non-causal factors. Consequently, the hash codes derived from these deep features may lack interpretability and be suboptimal for subsequent retrieval tasks. (2) Neglect of invariant patterns. While adversarial learning commonly used to implicitly align hash codes between source and target domains, these methods often fail to explicitly capture the invariant patterns present in vision representations. This could lead to ineffective domain alignment of hash codes due to the influence of underlying variant factors. As a result, an effective hashing framework under the principle of is highly desired.
invariant learning 12 5
In this paper, we explore domain adaptive hashing through a causal analysis lens, proposing a model named Invariance- Acquiree Domain Adaptive Hashing (IDEA) for efficient image retrieval under domain shift. The core of domain adaptive hashing lies in discerning causal and non-causal effects within images. To generate high-qualit and interpretable binary codes, we disentangle causal and non-causal features within each image, guided by the principles of the information bottleneck. On one hand, we maximize the mutual information between causal features and label information. On the other hand, we retain the most information in hidden features for non-causal features while minimizing the information in labels. These causal features are used to generate diseriminative hash codes through consistency learning across both source and target domains. To further mitigate the non-causal effects, we incorporate a generative model that simulates the intervention of various non-causal effects, thereby encouraging hash codes to remain sufficiently invariant to different non-causal components. Leveraging invariant learning on causal effects, our IDEA is capable of generating invariant hash codes that facilitate efficient cross-domain retrieval. Empirical experiments conducted on a variety of benchmark datasets validate the superior performance of our IDEA compared to competitive baselines. The contribution of this paper summarized as follows:
Problem Connection. We pioneer novel perspective that connects invariant learning with domain adaptive hashing for efficient image retrieval.
Novel Methodology: Our IDEA not only disentangles causal and non-causal features in each image following the principle of the information bottleneck, but also ensures hash codes are sufficiently invariant to the intervention of non-causal features.
High Performance. Comprehensive experiments across numerous datasets demonstrate that our IDEA outperforms a range of competitive baselines in different settings.
2 Related Works
Learning to Hash. The task of efficient image retrieval has recently witnessed significant interest
49
29 40, 37], leading to the development of deep hashing methods [4]
80
68
20
including both
supervised 72 44 67 48 75
8 B1
52 54 and unsupervised approaches 32
76 46
69
33
61
[56]
10 55). Deep unsupervised hashing methods typically reconstruct semantic structures using similarity mmilaaity-preserving binary codes. self-supervised methods have also been developed to enhance the performance. The incorporation of label information has notably improved the performance of deep supervised hashing methods. Early attempts at optimizing the
metrics, subsequently generating

---

hashing network often involved pairwise and triplet losses based on similarity structures 44 67 30 4
Another line of study has utilized point-wise optimization, establishing proxies for each category within the Hamming space and compelling hash codes to approximate these proxies 722 13 However, these methods typically overlook potential distribution shift in practical scenarios, which can significantly degrade retrieval performance. This limitation has prompted research into domain adaptive hashing.
Unsupervised Domain Adaptation. Unsupervised Domain Adaptation (UDA) has long been formidable challenge in machine learning and computer vision [3X
5T
[4]. Early methods often 27.. An alternative approach
explicitly reduce distribution discrepancies for domain alignment
43
utilizes adversarial learning for implicit domain alignment. incorporating gradient reversal layer and
a domain discriminator to engage in minimax game [1]] 36]. Recently, researchers have extended
this topie towards efficient domain adaptive retrieval 66
74 21 58 59 proposing a
number of transferable hashing methods to address this problem. These methods typically integrate domain adaptation models into deep hashing frameworks and generate similarity structures for learning semantic knowledge on the target domain. Despite these advancements, these methods often fall short due to the entanglement of causal and non-causal effects and the neglect of invariant patterns. In response to these challenges, we propose an effective approach IDEA.
Invariant Learning. Invariant learning 2 3 g aims identify invariant correlations between inputs and targets under domain shift, while concurrentyy eliminating spurious and variant relationships. This concept has been explored in the context of out-of-distribution generalization. Under specific assumptions, invariant learning has demonstrated significant potential for model generalization following causal theory. Invariant risk minimization neural networks to remain stable under the environmental variance, showing superior performance when compared to empirical risk minimization [12 (ERM). Moreover, MVDC (65) learns from frequency spectrum to generate causal features for domain generalization for cross-domain object detection. Invariant learning has also been extended to address challenges in graph domains [28] For instance, CIGA [6] leverages distinct subgraphs for graph contrastive learning, thereby achieving superior diitr---iittiittoonnn graph classification performance. Our study establishes connection between invariant learning and domain adaptive hashing, generating domain-invariant invariant and discriminative hash codes for cross-domain image retrieval. To the best of our knowledge, this is the first work to apply invariant learning for transferable retrieval tasks, and is proven effective for potential application.
(IRM) has been proposed to regularize
70 6 53 63)
3 Problem Definition
Given source domain 2 {( of J with N, fully-labeled images and target domain D {( with unlabeled N, images, both domain share a common label space y = {1, 2,.. C} despite potential distribution shifts. The objective is to develop hashing-bass retrieval model that projects an input image x onto a compact binary code b {-1, where represents the code length. This model should ensure that similar images the Hamming space, and its performance should be evaluated in both single- and cross-domain retrieval systems. In single-domain retrieval system, both query and database images originate from the target domain, whereas, a cross-domain retrieval system, the query images are drawn from the target domain D' and the database images sourced from the source domain D
mapped to comparable binary codes within
4 Method
4.1
Framework Overview
We address the problem of unsupervised do- main adaptive retrieval. While several domain adaptive hashing algorithms have been pro- posed, they often neglect invariant learning during the hash code generation process. This oversight often leads to significant domain dis- crepancy the resulting hash codes. More- over, these methods indiscriminately focus correlations between image data and semantic labels,
Cassal
Relatiasships domains
Hash code
Image
F: Semantic labels
D
NC
NC: Non-caucal part
Figure 1: Structural causal model in our problem.
3

---

Ler
Hask Codex
Cansal
feature
Source domalle
Encoder
Recoustruct
Disentonglament Loss
Generative model
LD
domale
feature
feature
LR
The framework of the proposed IDEA. We feed both source and target images into an encoder. Then each image is disentangled into causal features and non-causal features. The causal features are adopted to generate hash codes. A generative model is utilized to reconstruct the original graphs. We add intervention by using different non-causal features and minimize the domain shift by invariant learning.
Figure 2:
failing disentangle the causal and non-causal effects encoded in images. This results learnt hash codes that are not only suboptimal for downstream retrieval tasks, but are also difficult to interpret.
In this study, present novel approach named IDEA, to tackle these issues. The fundamental idea to incorporate invariant learning into the optimization the hashing network. Specifically, we first introduce structural causal model, illustrated in Figure [] and analyze the correlation within this problem. We then separate each image into causal and non-causal features using an information bottleneck. The causal features subsequently used for hash code production. Lastly, we introduce interventions on non-causal effects and limit their influence on invariant hash codes. More details can be found in Figure 2
4.2 Structure Causal Model
To start, we apply a causal perspective to domain adaptive hashing and construct a structural causal model [39] (SCM) to depict the image generation process under domain shifts. This model is used to illustrate the relationships between domains D, semantic labels Y, causal parts C, non-causal parts NC, images X, and hash codes H. The specific relationships elaborated as follows:
Y - Semantic labels produce the causal part which should be invariant to domain distributions.
> NC. Domain information provides the non-causal part which changeable across domains.
> + NC. Each image is generated by combining both causal part and non-causal part.
NC- + H. To obtain Nomain-invarian hash codes, we should generate hash codes using causal features. However, there could be additional relationships between C' and NC. For example, and NC can be simultaneously influenced by cause. Therefore, C acts as confounder between NC and H, i... resulting in falsely related variables.
The essence of domain adaptive hashing lies learning Nomain-invariat and discriminative binary codes. From the structural causal model, we need to construct mappings e(-) and vi(-) which fulfill the following conditions:
CCNC
(C), H NC
where symbolizes the independence under the given condition. Inferred from Eqn. we are required to address the following challenges: 1) The first term necessitates us to disentangle the causal and non-causal components of each image; 2) The second term requires us to learn discriminative hash codes from causal components; 3) The final term advocates for the generation of hash codes that are independent of the influence of non-causal components. With these goals in mind, we propose novel deep hashing method IDEA for domain adaptive retrieval.

---

4.3
Disentanglement
In this part, we apply the information bottleneck principle disentangle each image into causal and non-causal features. This approach facilitates the extraction of multifaceted latent factors embedded within images, enabling the generation of high-quality and interpretable binary codes. As suggested by the structural causal model, causal features should encapsulate label information, while non-causal features should reflect domain discrepancy.
In detail, we first introduce a feature extractor F(.). which removes the last classification layer in a popular neural network backbone (e.g., VGG-F (50]) generate hidden features, i.c., F()). Then, two different MLPs gf(-) and g-(-) are utilized to generate two features, respectively, i.c., f = (z) and f (z). Let F and Y denote the random variables of causal features and semantic labels, and we maximize the mutual information between F and Y. To achieve this, we turn InfoNCE 18 3 7 which construct positive pairs and negative pairs from the joint distribution P(F. Y) E D' and the product of marginal distributions p( ((())(( E DD, respectively. Since label information is not available D', we merely utilize source data in this module. particular, an estimator 7 is introduced to estimate the lower bound of mutual information, the target objective written as:
min CM1 = Ep( (f* y)] log ( [e" (eTU(
(2)
where ( ) comes from a bi-linear function with a weight matrix W to calculate the probability of being a positive pair, i.e., (f y) Wy The estimator and the causal head g are optimized jointly to obtain effective causal features with high correlation label information.
Moreover, to generate non-causal features, we minimize the mutual information between F" and Y with the maximum mutual information with hidden features Z following the principle of information bottleneck. In formulation, we need to minimize:
min Chr I(F", Y) BI(F",
where B is a parameter to balance the loss. Similarly, it is infeasible to minimize Eqn. directly. Therefore, we calculate the upper bound and lower bound of I(F", Y) and T(F", Z), respectively. Here, we first define:
i(F", Y) = EP [log p(y f]]] - E Ep(r) [logp(y f")],
which can be shown as an upper bound of / [(F", we introduce a variational function ( approximate p(yl ("). To measure the lower bound of [(F", Z), we introduce a different estimator T" 2) = =" and the upper bound of
Y). More detail is shown in Appendix. Moreover,
I(F". Z) is written follows:
i(F". Z) Ep(F" 2) (T" (f" (((,""2)) =)] - log (EpCE>)p22) [ (
(5)
By combining Eqn. and Equ. 5 we rewrite the objective in Eqn.
as:
C3, pi Fr, Y) [log qu (y
f E Ep() [log d f()]
-9 (supr F FM 22 [T" (f" ])] log (E, (m()) TT
(f**)) x))
(6)
From Eqn. 6. we utilize adversarial learning to optimize non-causal features, which generates reserve gradients for estimator T when minimizing CZr The overall loss for disentanglement summarized as:
LD = Ciss + Lis--
(7)
By minimizing CD. successfully disentangle causal and non-causal elements embedded in images, which can guide us to generate high-quality and interpretable hash codes.
4.4 Consistency Learning for Discriminative Hash Codes
To learn discriminative hash codes, we leverage causal features given their strong correlation with label information for hash code generation. Here, consistency learning introduced, based on the principle that similar images should be mapped to similar hash codes to enable effective image retrieval 4 25 This strategy ensures that our model can capture and preserve the inherent semantic relationships in the data, thus facilitating more accurate and effective retrieval.
5

---

In particular, we utilize MLP (-) to map causal features to hash codes and enforce similar images to have similar binary codes. Here, we resort consistency learning for optimization. which constructs positives via (1) two images sharing the same label in the source domain; (2) two augmented views from the same image in the target domain. In practice, we generate two augmented views for each image in mini-batch B = 3 U JB, and define the positive set for source sample as:
( = {1e of (
where ai denotes the augmented source sample. Then, we provide a hashing consistency learning objective for source domain:
exp b bx/T)
CeL = -E [I(E;)]
log 2|8 == exp (6g bā//T)
(9)
kell(a))
where b; (F and is a temperature parameter. Moreover, the hashing consistency learning objective in the target domain is:
cxp (b, bx/T)
CeL ExEB log
(10)
2116" exp (b: bie/T)
Finally, the consistency learning objective is derived by combining both source and target domains:
=CL= = Ler + Ler-
(11)
The introduction of consistency learning offers two primary benefits for hash code generation: (1) By maximizing the consistency of hash codes between similar images, IDEA ensures the generation of similarity-preserving hash codes, which essential for effective search engines. (2) By minimizing the similarity of hash codes derived from dissimilar images, our IDEA encourages hash codes be uniformly distributed in the Hamming space, which enhances the capacity of each hash bit.
4.5 Invariance under Intervention
However, in practice, causal parts and non-causal-part could be still closely related. For example, waterbirds are typically paired with water backgrounds in the training set. To further reduce the potential hidden effects of non-causal parts, we introduce invariant learning [28] [28] 77, 6), which first introduce generative model to do interventions and then encourages hash codes to be sufficiently invariant to different non-causal parts.
In detail, a generative model G(s is first learnt to reconstruct original images using both causal and non-causal features. To optimize the generative model, the reconstruction loss in mini-batch is written in the formulation of:
CRE E,, (F)
(12)
With the well-trained model, we can simulate the intervention by utilizing different non-causal features. Here, we first generate synthetic images using f and f.:
xx, + ..)dd(fm.) G(J fh))
(13)
where do(f% denotes to impose non-causal features from a different sample w;) in B. Then we feed the synthetic sample into hashing network to generate hash codes, b = (F(att))))) and then minimize the variance under different intervention. The objective can be written as:
Lv = E. ( ar (b:t")] Ez,, e b,, 1.
(14)
where b, = Fr,,,. (b+ 10 the mean of synthetic hash codes. With effective invariant learning, we generate invariant binary codes for effective cross-domain retrieval.

---

Algorithm Training Algorithm of IDEA
Require: Source data D target data Da, code length L;
Ensure: The hashing network ( (F())))
1: Warm up backbone by minimizing LD and CRE:
2: repeat
3: Sample mini-batch B and B from D and D respectively;
4: Generate two augmented views for each sample;
5: Calculate positive set by Eqn. [
6:
Generate synthetic samples under intervention by Eqn. 13
7:
Calculate the final loss objective in Eqn.
8:
Reverse the gradients T";
9:
Update parameters by gradient descent;
10:
until convergence
Table MAP performances two bench-marking datasets with 64-bit hash codes.
Office-Home
Office31
Methods
C2R
R2A
R2P
R2C
A2R
A2D
A2W
W2D
D2A
W2A
D2W
Avg
Unsupervised Hashing Methods
SH 62
8.77
12.87
16.13
8.24
13.71
34.72
11.28
9.85
34.37
15.57
ITO 17
26.81
14.83
25.37
28.19
14.92
25.88
29.55
28.53
58.00
26.83
25.09
58.89
30.24
DSH
8.49
5.47
9.67
8.26
5.28
9.69
16.66
15.09
39.24
16.33
13.58
41.07
15.74
LSH
16
12.24
6.94
1144
13.45
7.24
11.49
16.04
15.35
38.80
13.60
14.67
43.99
17.11
SGH 24
24.51
13.62
22.53
25.73
13.51
22.93
24.98
22:47
53.94
22.17
20.52
56.36
26.94
OCH
35
18.65
10.27
17.54
20.15
10.05
18.09
24.86
22.49
51.03
22.45
20.79
53.64
24.17
Transfer Hashing Methods
ITQ+ 779
17.61
9.55
14.25
17.9
15.00
42.29
19.45
L
10.37
13.56
19.96
18.24
43.32
20.39
GTH-g 74
20.00
10.99
18.28
21.95
11.68
19.05
23.08
21.20
49.38
19.52
17.41
50.14
23.56
DAPH 21
27.20
15.29
27.35
28.19
15.29
26.37
32.80
28.66
60.71
28.66
27.59
64.11
31.85
PWCF 22
34.03
24.22
28.95
34.44
18.42
34.57
39.78
34.86
67.94
35.12
35.01
72.91
38.35
DHLing 22
48.47
30.81
38.68
45.24
25.15
43.30
41.96
45.10
75.23
42.89
41.74
46.54
PEACE
[58
53,04
38.72
42.68
54.39
28.36
45.97
46.69
48.89
78.82
46.91
46.95
83.18
51.22
Ours
59.18
45.71
49.64
61.84
32.77
51.19
48.70
54.43
84.97
53.53
53.71
88.69
57.03
4.6 Summary
Finally, we summarize our framework with the following overall training objective as:
L Ln + Ler. + LRE Lv-
(15)
Moreover, the derivation of sign(-) zero for any non-zero value, and therefore difficult to compatible with the gradient propagation. To tackle this challenges, we adopt tanh(-)) to replace sign(-) during optimization, which produces approximate hash codes v (ftte((/ggg((((aa training. Our model is first warmed up by the disentanglement module optimized by the reconstruction loss. Then we optimize the whole network with mini- stochastic gradient descent and the detailed progress of our IDEA can be found in Algorithm I
for
Experiment
5.1
Experimental Settings
Baselines.
We adopt variety of state-of-the-art approaches for performance comparison, including
six unsupervised hashing approaches (i.e., SH (62). ITQ [17]. DSH (34] LSH [16). SGH (24), OCH
35
and seven transfer hashing approaches (i.e., ITQ [79]. LapiTQ+ (79,, GTH-g (74), DAPH
PWCF (22). DHLing [64], and PEACE (58)). Some results ITQ+ and LapITQ+ are omitted
because prior information is not accessible for sample pair construction.
Experiments are conducted different benchmark datasets: (1) Office-Home datasel 57 This dataset contains examples from four domains (Ar, CI, Pr, Re), each with 65 object categories. Two domains selected as the source and target, resulting in six transferable image retrieval tasks
Datasets.
7

---

Table 2: MAP performances the Digits dataset with 64-bit hash codes.
MNIST2USPS
USPS2MNIST
Code Length
32
48
96
128
16
32
48
96
128
Avg
Unsupervised Hashing Methods
SH
[6
13.67
13.45
13.35
12.95
14.35
14.22
13.57
12.92
13.87
ITQ 17
13.05
15.57
18.54
20.12
23.12
23.89
13.69
17.51
20.40
20.30
22.79
24.59
19.46
DSH
20.60
22.21
23.68
24.28
25. 25.73
26.50
19.54
21.22
22.89
23.79
25.91
26.46
23.57
13.54
15.89
12.76
14.77
16.89
16.32
19.67
16.01
SGH 24
14.24
16.69
18.72
19.70
21.00
21.95
13.26
17.71
18.22
19.01
21.69
22.09
18.69
OCH
[35
13.73
17.22
19.59
20.18
20.66
23.34
15.51
17.75
18.97
21.50
21.27
23.68
19.45
Transfer Hashing Methods
ITQ+ 779
22.84
21.20
20.68
19.15
17.99
18.52
20.06
+
24.26
24.03
23.76
24.59
23.33
22.73
23.78
GTH-g 74
20.45
17.64
16.60
17.25
17.26
17.06
15.17
14.07
15.02
15.01
14.80
17.34
16.47
DAPH 21
25.13
27.10
26.10
28.51
30.53
30.70
26.60
26.43
27.27
27.99
30.19
31.40
28.16
PWCF 22
47.47
51.99
51.44
51.75
50.89
59.35
47.14
50.86
52.06
52.18
57.14
58.96
52.60
DHI Ling
22
49.24
54.90
56.30
58.28
58.80
59.14
50.14
51.35
53.67
58.65
58.42
59.17
55.67
PEACE
588
52.87
59.72
60.69
62. 84
65. 13
68. 16
53.97
54,82
58.69
60.91
62.65
65.70
60.51
Ours
58.89
64.48
65.72
67.48
70.24
74.34
60.99
61.47
65.45
67.97
69.72
72.31
66.59
1531
C3R
S5A
D3P
B2M
20%
50
5
1
R
PC2
Bacill
Drail
Baaill
Figure 3: We use 128-bit Precision-Redd Recall curves to evaluate the performance of our method on the Office-31 and Office-Home datasets.
(P2R, C2R, R2A, R2P, R2C, A2R). (2) Office-31 dataset 442 This widely used benchmark dataset contains over 4000 examples classified into 31 classes, which are from three domains (Am, We, Ds). We randomly select two domains as the source and target, resulting in six transferable image retrieval tasks (A2D, A2W, W2D, D2A, W2A, D2W). (3) Digits dataser: We focus on MNIST [26] and USPS 233, each containing ten handwritten digits. Each sample re-scaled to 16x We select one dataset as the source and the other as the target, resulting in two transferable image retrieval tasks (MNIST2USPS and USPS2MNIST).
Implementation Details & Evaluation metrics. We perform experiments on an A A100-40GB GPU. 10% of the target examples are used as queries and the remaining target and source samples are viewed as the training set. As for the database, source domain data is adopted in cross-domain retrieval while target domain data adopted in single-domain retrieval. The hashing network is optimized using mini-batch SGD with momentum. The batch size is set to 36 and the learning rate fixed as 0.001. We evaluate the retrieval performance using four common metries: mean average precision (MAP), precision-recall curve, top-N accuracy curve, and top-N recall curve. MAP the primary metric for evaluating retrieval accuracy, while the precision-recall curve provides an overall performance indicator at different recall levels. The top-N accuracy curve shows the accuracy for different numbers of retrieval instances. As the key metrics, MAP and precision- recall curve reflect the overall performance of our model in retrieval tasks. We also analyze the top-N accuracy and recall curves to evaluate the performance at different number of retrieved results. These comprehensive evaluations allow to have an in-depth understanding of the model's retrieval ability.
5.2 Empirical Results
Performance Comparison. We present the results of our experiments on several benchmark datasets in Tables D and Table where we report the mean average precision (MAP) scores achieved by our IDEA and other state-of-the of-the-art methods. Our analysis of the results leads to the following conclusions. On the Office-Home and Office31 datasets, our IDEA achieves an outstanding average MAP score of 57.03, which significantly outperforms the second-best method, PEACE, with an average MAP score of 51.22. Furthermore, IDEA achieves the highest MAP scores across all query types, including P2R, C2R. R2A, R2P, R2C, A2R, A2D, A2W, W2D, D2A, W2A, and D2W. On the

---

pitlir
PEACY
A
A2M
500 -0
201026
Figure 4: Visualization of 1 128-bit hash codes on MNIST dataset in left three columns; Sensitivity analysis results of our proposed IDEA on Office-Home and Office-31 datasets are shown in right two columns on three cross-domain retrieval tasks.
Digits dataset, IDEA consistently achieves the highest MAP scores for all hash code lengths, ranging from 16 to 128 bits. This demonstrates the consistency of our method across different code lengths. For instance, on the MNIST2USPS task, IDEA achieves an average MAP score of 74.34 for 128-bit codes, which is significantly higher than the second-best method, PEACE, with an average MAP score of 68. 68.16. Overall, our experimental results demonstrate that the proposed IDEA is effective and outperforms state-of-the-aas methods on various benchmark datasets. These findings highlight the potential of our method for practical applications in cross-domaii image retrieval tasks. We present the recall-precision curves of four compared approches (PWCF, DHLing, PEACE, and IDEA) on two cross-domain retrieval tasks in Figure Furthermore, in most cases, the curve of IDEA is higher than that of the other three baselines, indicating that the binary codes generated by IDEA works better for the hash table lookup strategy.
Qualitative Results. Here we provide a qualitative analysis of the model. Specifically, we use 1-SNE visualization to display the hash codes learned by IDEA, PEACE, and DHI.ing (see the three subfigures on the left side of Figure 4. When compared to the PEACE and DHLing baseline methods, our IDEA -enerrreee hash codes show more discriminative structures, with binary codes of different categories being better separated. These results indicate that the proposed method can produce hash codes with better discriminative power, which facilitates more effective image retrieval.
Parameter Sensitivity. In this part, we conduct sensitivity analysis for two important hyper- parameters, which is shown in the right subplot of the Figure f We can observe that the MAP values for all three cross-domain retrieval tasks slightly decrease the 3 parameter increases, indicat- ing a negative impact on model performance. However, the decrease is not significant, allowing for an increase in 8 to balance loss terms without causing significant reduction in model performance. Similarly, the MAP values increase and then decrease as T increases, demonstrating the potential for improved model performance. However, exceeding impact on retrieval performance. Thus, determining the optimal r value through experimentation is necessary for achieving optimal retrieval performance in practical applications. In our experiments, we set (3 and 7 to 0.1 and 0.5, respectively.
optimal T range may have negative
Ablation Study. We have introduced several variants of our IDEA to inves- tigate the impact of each components: (1) IDEA w/o CRE removes the origi- nal image reconstruction loss from the generated model, (2) IDEA w/o LCr. removes the consistency learning loss, (3) IDEA w/o Ly removes the hash code variance under intervention, and (4) IDEA w/o LM removes the mutual information loss. By summarizing the results of these model variants in Table 3 We conclude, the ablation experiments show that removing any of the four components (i.e., CRE, LCL, Lv. LMrr) leads to a drop in performance on all datasets and tasks. Notably, LRE and LCL have the most significant impact. Our IDEA achieves the best or comparable performance on all datasets and tasks, demonstrating its effectiveness in learning disentangled representations for domain generalization. Specifically, it achieves the highest MAP on Office-Home and Digits datasets and the highest accuracy on the A2W task of the Office31 dataset, outperforming the ablated variants.
Table 3: Ablation Studies on six benchmark cross-domain retrieval tasks.
Dataset
Office-Home Home
Office31
Digits
Method
C2R
R2A
A2D
A2W
U2M
M2U
IDEA wo LRE
44.02
47.29
47.05
53.11
64.80
64.49
IDEA w/o Lel
42.11
45.97
45.36
51.17
64.25
63.76
IDEA w/o Lv
45.10
49.09
48.21
53.96
57.34
66.97
IDEA w/o Casi
44.89
48.92
48.05
53.79
67.12
66.85
Ours
45.71
49.64
48.70
54.43
67.97
67.48

---

6
Conclusion
This work studies the problem of domain adaptive retrieval and proposes a novel method named IDEA to solve the problem. Our IDEA first disentangle causal and non non-causal features within each image, guided by the principles of the information bottleneck. Moreover, these causal features are used to generate discriminative binary codes through consistency learning across both source and target domains. To generate comain-invariant binary codes, we simulates the intervention of various non-causal effects and encourage the invariance of hash codes. Extensive experiments on benchmark datasets validate the superiority of IDEA.
Broader Impacts and Limitations. This work improves the performance cross rross-domain retrieval, which can benefit real-world search engineers. Moreover, our work provide new direction of incorporating invariant learning into cross-domain retrieval problems. One limitation of our work that IDEA cannot tackle the open-set scenarios where target samples may come from unseen classes and we would extend our IDEA to more generalization scenarios in our future works. In particular, more advanced techniques such as modes AICG and multimodal large-scale learning can be incorporated into our IDEA to improve the generalizability in wider range of scenarios. In addition, we would explore more detailed visualization and interpretability analysis of invariant learning in the scope of cross- moosll retrieval.
References
[1] Rana Ali Amjad and Bernhard C Geiger. Learning representations for neural fetwork-based classification using the information bottleneck principle. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(9):2225-22)) 2019.
[2] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk mini- mization. arXiv preprint arXiv: 1907.02893, 2019.
[3] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In ICML, pages 531-540, 2018.
[4] Zhangjie Cao, Ziping Sun, Mingsheng Long, Jianmin Wang, and Philip Yu. Deep priority hashing. In ACMMM, 2018.
[5] Shiyu Chang, Yang Zhang, Mo Yu, and Tommi Jaakkola. Invariant rationalization. In ICML, 2020.
pages
[6] Yongqiang Chen, Yonggang Zhang, Yatao Bian, Han Yang, Kaili Ma, Binghui Xie, Tongliang Liu, Bo Han, and James Cheng. Learning causally invariant representations for out-of- distribution generalization graphs. In NeurlPS, pages
2022.
[7] Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. Club:
A contrastive log-ratio upper bound of mutual information. In ICML, pages 1779-1788, 2020.
[8] Lianhua Chi, Bin Li, Xingquan Zhu, Shirui Pan, and Ling Chen. Hashing for adaptive real-time graph stream classification with concept drifts. IEEE Transactions on Cybernetics, 48(5):1591- 1604, 2017.
[9] Elliot Creager, förn-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. In ICML, pages 2189-2200, 2021.
[10] Cheng Deng, Erkun Yang. Tongliang Liu, Jie Li. Wei Liu, and Dacheng Tao. Unsupervised somantic-preserving adversarial hashing for image search. IEEE Transactions on Image Pro- cessing, 28(8):0332 4044, 2019.
[11] Khoa Doan, Peng Yang, and Ping Li. One loss for quantization: Deep hashing with discrete wasserstein distributional matching. In CVPR, pages 9447-9457, 2022.
[12] Michele Donini, Luca Oneto, Shai Ben-David, John S Shawe-Taylor and Massimiliano Pontil. Empirical risk minimization under faimess constraints. In NeurIPS, 2018.
10

---

[13] Lixin Fan, KamWol Ng, Ce Ju, Tianyu Zhang, and Chee Seng Chan. Deep polarized network for supervised learning of accurate binary hashing codes. In IJCAI, 2020.
[14] Kaituo Feng, Changsheng Li, Xiaolu Zhang, and Jun Zhou. Towards open temporal graph neural networks. arXiv preprint 2rXiv:2303.15015 2023.
[15] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. 17(1):099-20300, 2016.
[16] A. Gionis, Piotr Indyk, and Rajeev Motwani. Similarity search in high dimensions via hashing. In VLDB, 1999.
[17] Yunchao Gong, Svetlana Lazebnik, Albert Gordo, and Florent Perronnin. Iterative quantiza- tion: A procrustean approach to learning binary codes for large-scale image retrieval. /EEE Transactions Pattern Analysis and Machine Intelligence, 122216-2929,, 2013.
[18] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie. and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[19] Tao He, Yuan-Fang Li, Lianli Gao, Dongxiang Zhang, and Jingkuan Song. One network for multi-domains: domain adaptive hashing with intersectant generative adversarial networks. In IJCAI, 2019.
[20] Jiun Tian Hoe, Kam Woh Ng. Tianyu Zhang, Chee Seng Chan, Yi-Zhe Song, and Tao Xiang. One loss for all: Deep hashing with a single cosine similarity based learning objective. In NeurIPS, 2021.
[21] Fuxiang Huang, Lei Zhang, and Xinbo Gao. Domain adaptation preconceived hashing for unconstrained visual retrieval. IEEE Trunsactions Neural Networks and Learning Systems, 2021.
[22] Fuxiang Huang, Lei Zhang, Yang Yang, and Xichuan Zhou. Probability weighted compact feature for domain adaptive retrieval. In CVPR, 2020.
[23] Jonathan Hull. database for handwritten text recognition research. JEEE Transactions on pattern analysis and machine intelligence, 16(5):550-55 1994.
[24] Qing- Yuan Jiang and Wu-Jun Li. Scalable graph hashing with feature transformation. In IJCAI, 2015.
[25] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu. and Dilip Krishnan. Supervised contrastive learning. In NeurIPS, pages 18661-18673, 2020.
[26] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 (66(1)):277-23222 1998.
[27] Chen- Yu Lee, Tanmay Batra, Mohammad Haris Baig, and Daniel Ulbricht. Sliced wasserstein discrepancy for unsupervised domain adaptation. In CVPR, 2019.
[28] Haoyang Li, Ziwei Zhang, Xin Wang, and Wenwu Zhu. Learning invariant graph representations for nut-of-distribution generalization. In NewrlPS, 2022.
[29] Wen Li, Ying Zhang, Yifang Sun, Wei Wang. Mingjic Li, Wenjie Zhang, and Xuemin Lin. Approximate nearest neighbor search on high dimensional data data ciite--exeimmets, analyses, and improvement. IEEE Transactions on Knowledge and Data Engineering, 1775::145-1148, 2019.
[30] Wu-Jun Li, Sheng Wang, and Wang-Cheng Kang. Feature learning based deep supervised hashing with pairwise labels. In AAAI, 2016.
[31] Xuan Li, Wei Wu, Yun-Hao Yuan, Shirui Pan, and Xiaobo Shen. Online unsupervised cross-view discrete hashing for large-scale retrieval. Applied Intelligence, pages 1-13, 2022.
11

---

[32] Yunqiang Li and Jan van Gemert. Deep unsupervised image hashing by maximizing bit entropy. In AAAI, 2021.
[33] Qinghong Lin, Xiaojun Chen, Qin Zhang, Shaotian Cai, Wenzhe Zhao, and Hongfa Wang. Deep unsupervised hashing with latent semantic components. In AAAI, 2022.
[34] Yuetan Lin, Deng Cai, and Cheng Li. Density sensitive hashing. IEEE Trans. Cybern., 44:1362- 1371, 2014.
[35] Hong Liu, R. Ji, Jingdong Wang, and Chunhua Shen. Ordinal constraint binary coding for approximate nearest neighbor search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41:94119555 2019.
[36] Fuchen Long, Ting Yao, Qi Dai, Xinmei Tian, Jiebo Luo, and Tao Mei. Deep domain adaptation hashing with adversarial learning. SIGIR, 2018.
[37] Xiao Luo, Haixin Wang, Daqing Wu, Chong Chen, Minghua Deng, Jianqiang Huang, and ShengSheng Hua. A survey on deep hashing methods. ACM Transactions on Knowledge Discovery from Data, 17(1):1-50, 2023.
[38] Xinhong Ma, Junyu Gao, and Changsheng Xu. Active universal domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8968-8977, 2021.
[39] Judea Pearl et al. Models, reasoning and inference. Cambridge, UK: CambridgeUrivversttyyresss 19(2), 2000.
[40] Liudmila Prokhorenkova and Aleksandr Shekhovtsov. Graph-based nearest neighbor search: From practice to theory. In ICML, pages 7803-7813, 2020.
[41] Chaitanya Ryali, John Hopfield, Leopold Grinberg, and Dmitry Krotov. Bio-inspired hashing for unsupervised similarity search. In ICML, pages 1295-83066 2020.
[42] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In Computer Vision-ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part IV 11, pages 213-226. Springer, 2010.
[43] Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In CVPR, 2018.
[44] Fumin Shen, Xin Gao, Li Liu, Yang Yang, and Heng Tao Shen. Deep asymmetric pairwise hashing. In ACMMM, 2017.
[45] Xiaohui Shen, Zhe Lin, Jonathan Brandt, and Ying Wu. Mobile product image search by automatic query object extraction. In ECCV, 2012.
[46] Yuming Shen, Jie Qin, Jiaxin Chen, Mengyang Yu, Liu, Fan Zhu, Fumin Shen, and Ling Shao. Auto-encoding wii-bbotlleneck hashing. In CVPR, 2020.
[47] Weiwei Shi. Yihong Gong, Badong Chen, and Xinhong Hei. Transductive semisupervised deep hashing. IEEE Transactions on Neural Networks and Learning Systems, 2021.
[48] Yang Shi, Xiushan Nie, Xingbo Liu, Li Zou, and Yilong Yin. Supervised adaptive similarity matrix hashing. IEFE Transactions Image Processing, 31:2755-2766, 2022.
[49] Harsha Vardhan Simhadri, George Williams, Martin Aumüller, Matthijs Douze, Artem Babenko, Dmitry Baranchuk, Qi Chen, Lucas Hosscini, Ravishankar Krishnaswamny. Gopal Srinivasa, et al. Results of the neurips 21 challenge on diliio--caale approximate nearest neighbor search. In NewrlPS Competitions and Demonstrations Track, pages 177-189, 2022.
[50] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1xXiv:1409.1155 2014.
12

---

[51] Ankit Singh. Clda: Contrastive learning for semi-supervised domain adaptation. In NeurlPS, pages 5089-5101, 2021.
[52] Shupeng Su, Chao Zhang, Kai Han, and Yonghong Tian. Greedy hash: Towards fast optimization for accurate hash coding in cnn. In NeurIPS, 2018.
[53] Yongduo Sui, Xiang Wang, Jiancan Wu, Min Lin, Xiangnan He, and Tat-Seng Chua. Causal attention for interpretable and generalizable graph classification. In KDD, pages 1696-1705, 2022.
[54] Bong-Cheng Tu, Xian-Ling Mao, Jia-Nan Guo, Wei Wei, and Heyan Huang. Partial-softmax loss based deep hashing. In Proc. of WWW, pages 2869-2878, 2021.
[55] Long-Cheng Tu, Xian-Ling Mao, Kevin Qinghong Lin, Chengfei Cai, Weize Qin, Wei Wei, Hongfa Wang, and Heyan Huang. Unsupervised hashing with semantic concept mining. SIGMOD, 1(1):1-19, 2023.
[56]
Tu, Xianling Mao, and Wei Wei. MIs3rduh: Deep unsupervised hashing via manifold based local semantic similarity structure reconstructing. In IJCAI, 2020.
[57] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan.
Deep hashing network for unsupervised domain adaptation. In CVPR, 2017.
[58] Haixin Wang, Jinan Sun, Xiao Luo, Wei Xiang, Shikun Zhang, Chong Chen, and Xian-Sheng Hua. Towards effective domain adaptive retrieval. IEEE Transactions on Image Processing. 2023.
[59] Haixin Wang. Jinan Sun, Xiang Wei, Shikun Zhang. Chong Chen, Xian Stan-Sheng Hua, and Xiao Luo. Dance: Learning a domain adaptive framework for deep hashing. In The WebConf, pages 1319-3330, 2023.
[60] Meng Wang, Tongtong Wu, and Guilin Qi. A hash learning framework for search-oriented knowledge graph embedding. In ECAI, pages 921-928. 2020.
[61] Ziwei Wang, Han Xiao, Yueqi Duan, Jie Zhou, and Jiwen Lu. Learning deep binary descriptors via bitwise interaction mining. IEEE Transactions Pattern Analysis and Machine Intelligence, 2022.
[62] Yair Weiss, Antonio Torralba, and Rob Fergus. Spectral hashing, In NeurIPS, 2008
[63] Ying-Xin Wu, Xiang Wang, An Zhang, Xiangnan He, and Tat-Seng Chua. Discovering invariant rationales for graph neural networks. In ICLR, 2022.
[64] Haifeng Xia, Taotao Jing, Chen Chen, and Zhengming Ding. Semi-supervised domain adaptive retrieval via discriminative hashing learning. In ACMMM, 2021.
[65] Mingjun Xu, Lingyun Qin, Weijie Chen, Shiliang Pu, and Lei Zhang. Multi-view adversarial discriminator: Mine the non-causal factors for object detection in unseen domains. In CVPR, pages 12, 2023.
[66] Renjun Xu, Pelen Liu, Yin Zhang, Fang Cai, Jindong Wang, Shuoying Liang, Heting Ying, and Jianwei Yin. Joint partial optimal transport for open set domain adaptation. In IJCAI, 2021.
[67] Cheng Yan, Guansong Pang, Xiao Bai, Chunhua Shen, Jun Zhou, and Edwin Hancock. Deep hashing by discriminating hard examples. In ACMMM, 2019.
[68] Chenggang Yan, Biao Gong, Yuxuan Wei, and Yue Gao. Deep multi-view enhancement hashing for image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, (344):1445-1451, 2020.
[69] Erkun Yang, Tongliang Liu, Cheng Deng, Wei Liu, and Dacheng Tao. Distillhash: Unsupervised deep hashing by distilling data pairs. In CVPR, 2019.
[70] Nianzu Yang, Kaipeng Zeng, Qitian Wu, Xiaosong Jia, and Junchi Yan. Learning substructure invariance for out-of-distributte molecular representations. In NewrIPS, 2022.
13

---

[71] Chaohui Yu, Jindong Wang, Yiqiang Chen, and Zijing Wu. Accelerating deep unsupervised domain adaptation with transfer channel pruning. 2019 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE, 2019.
[72] Li Yuan, Tao Wang, Xiaopeng Zhang, Francis EH Tay, Zequn Jie, Wei Liu, and Jiashi Feng. Central similarity quantization for efficient image and video retrieval. In CVPR, 2020.
[73] Jing Zhang, Xiaokang Zhang, Jifan Yu, Jian Tang, Jie Tang, Cuiping Li, and Hong Chen. Subgraph retrieval enhanced model for multi-hop knowledge base question answering. In ACL, 2022.
[74] Lei Zhang, Ji Liu, Yang Yang, Fuxiang Huang, Feiping Nie, and David Dian Zhang. Optimal projection guided transfer hashing for image retrieval. IEEE Transactions on Circuits and Systems for Video Technology, 30:3788-3800 2020.
[75] Qi Zhang. Liang Hu, Longhing Cao, Chongyang Shi, Shoujin Wang, and Dora D Liu. A probabilistic code balance constraint with compactness and informativeness enhancement for deep supervised hashing. 2022.
[76] Wanqian Zhang, Dayan Wu, Yu Zhou, Bo Li, Weiping Wang, and Dan Meng. Deep unsupervised hybridd-iiinn siniiiiimilaritt hadamard hashing. In ACMMM, 2020.
[77] Liangli Zhen, Peng Hu, Xu Wang, and Dezhong Peng. Deep supervised cross-modal retrieval. In Proceedings of the IEEE/CVF Conference Compuler Vision and Pullern Recognition, 2019.
[78] Feng Zheng, Cheng Deng, and Heng Huang. Binarized neural networks for resource-efficient hashing with minimizing quantization loss. IJCAI, pages 1032-1040, 2019.
[79] Joey Tianyi Zhou, Heng Zhao, Xi Peng, Meng Fang, Zheng Qin, and R. Goh. Transfer hashing: From shallow to deep. IEEE Transactions on Neural Networks and Learning Systems, 29:6191-6201, 2018.
[80] Xiaosu Zhu, Jingkuan Song, Yu Lei, Lianli Gao, and Hengtao Shen. lower bound of hash codes' performance. In NeurlPS, pages 29166-29177 2022.
14

---

A
The Upper Bound of Mutual Information
We define:
1(F Y)= Ep(p= x) [logp(y fn Ep(r=)Bp(rr) (ogg(y l")
(16)
Then, we show that 1 1(F", Y) is an upper bound of [(F", Y). In formulation, we calculate their difference as follows:
i(F", Y) I(F Y)
Ep(F=.Y) p f*]]
[log f*)]
-E v) log p(y f log p())
= Bp(x= [log p(y)] - pl() (logp(y f*)]
(17)
= Ep(r) (logr(y) (F)) (oog:>/lo//// p(y {"
Ey(r) [log (E (p(y) f Bp(F" Dog f")]
2 O (Jensen's Inequality),
where the last inequality holds due to Jensen's Inequality with a convex function log(-).
B
The Details of Baselines
These baselines for model comparison introduced below:
SH 62 is based on theory of graph Laplacian. The binary code of new data point is efficiently computed by the spectral analysis, enabling the distr--dittiitttoon extension for unseen samples.
ITQ 17 is an uncomplicated and effective alternative minimization algorithm. It can be incorpo- rated into both supervised and unsupervised learning processes.
LSH [16 an important method in compressing high-dimensional data. It aims to map similar data points into the same bucket.
DSH 34 is model variant of LSH, which generates multiple view of the data point for metric learning using random projections.
SGH 24 aims to compress high-dimensional data in a bit-wise manner, which is efficient for large-scale semantic similarity learning.
OCH 35 leverage the tensor product to generate an ordinal graph, which guides the learning to generate more related hash codes.
ITQ+ 79 is the early effort implement transfer learning for hashing study, which learns slack mapping from the source domain and achieves domain alignment through the quantization loss.
LapITQ+ 77 an extension of ITQ+ 779 by transferring the geometric relationship the source
domain to generate high-quality hash codes.
GTH-g 774 selects the best hash mapping for target data using source data following the principle of maximum likelihood estimation.
feature projection to reduce the domain discrepancy. leverages a Bayesian model to learn discriminative hash codes and then infer the similarity structure using histogram features.
DAPH 21 utilizes
PWCF
2
DHLing 644 utilizes learning cluster to optimize hash code within a single domain and then reduces the domain shift using the memory bank.
PEACE [8X learns the semanties on target data using pseudo-labeling techniques and then mini-
mizes the domain shift using both implicit and explicit manners.
C The Details of Datasets
Three popular benchmarks, i.e., Office-Home dataset, Office-31 and Digits, are adopted for perfor- mance comparison. Their details are elaborated as follows:
15

---

Table 4: MAP scores Amazon -> Dsir task with code lengths varying from 16 to 128 for single-
domain retrieval.
Bit
32
128
30.54
TQ 17
40.83
49.27
53,92
$6.16
59.31
59.41
DSIT
84
22.45
33.38
37.01
40.09
43.64
46.11
LSH
16.04
26.18
34.36
39.68
45.60
49.04
SGH
24
38.67
43.59
50.77
53,57
57.70
57.37
ITQ+ 10
35.03
42.62
41.85
43.12
39.82
39.12
Tap
37.60
42.91
43.24
44.55
42.01
38.87
GTH-g 14
31.
43.69
50.67
50.22
$4.37
55 35.81
PWCF 22
49.94
$3.05
$8.26
59,08
60.11
62.35
ing
PEACE 58
55.43
57.89
60.23
61.21
62.30
64.14
Ours
61.25
62.65
65.14
67.06
68.20
70.04
Table MAP scores on Product - Real task with code lengths varying from 16 128 for single- domain retrieval.
Bit
SH 62
13.15
18.71
22.51
22.57
21.27
20.66
20.07
DSH
10
6.10
11.44
12.21
16.61
16.46
14.45
10.62
17.57
21.62
24.92
SCH
18.97
26.18
30.61
32.61
33.82
34,97
OC1
13.45
21.14
23.65
25.34
25.99
28.02
ITQ+ 19
15.60
20.60
24.95
24.96
24.92
24.05
Lap
22.26
22.76
22.29
22.33
GTH-g
74
15.05
21.20
25.64
27.67
29.05
28.40
DAPH
21
20.77
29.01
31.60
33.35
34.22
34.92
PWCF 21
24.80
34.03
36.86
37,98
38.43
39.14
DHLing
[64
27.81
36.05
40.82
40.91
43.45
44.07
PEACE
Ours
34,88
44.83
48.38
49,91
$3.55
54.40
Office-Homee dataset [57] consists of image samples from four different domains, i.e., Art (A), Clipart (C), Product (P) and Real-World (R). There are 65 categories in the dataset. We select two domains the source and target, respectively, which can produce six transferable image retrieval tasks (P2R, C2R, R2A, R2P, R2C, A2R).
Office-31 dataset 42 is comprised of 4110 image samples from three different domains, i.c. Amazon (A), Webcam (W) and DsIr (D). Each domain corresponds to platform where the images are collected. Similarly, we randomly select two domains as the source and target, which can result in six transferable image retrieval tasks (A2D, A2W, W2D, D2A, W2A, D2W).
Digits consists of two datasets, MNIST 26 and USPS 2 and each contains handwritten digits from ten classes. The digits both datasets are captured under different conditions. Here we consider two transferable image retrieval tasks (MNIST2USPS and USPS2MNIST).
D Additional Experimental Results
D.1
Performance of Single-domain Retrieval
We first explore the results on single-domain retrieval, Table 4 commaare the average mean average precision (MAP) of different hashing methods under different bit lengths for the single-domain retrieval task, i.e., Amazon - Dslr. The table is divided into two parts: the first part shows the performance of unsupervised hashing methods, including SH, ITQ. DSH, LSH. SGH, and OCH. while the second part shows the performance of transfer learning hashing methods, including ITQ+, LapITQ+, GTH-g. DAPH, PWCF, DHLing, and PEACE. Additionally, the results of IDEA method are highlighted with bold font and pink background, indicating that it achieves the highest MAP values across all hash code lengths. The table shows that IDEA performs better than other hashing methods for simplifying image descriptions and improving image retrieval efficiency. As the hash
16

---

Table 6: MAP MNIST
USPS task with code lengths varying from 16 to 128 for
single-domann retrieval.
Bit
16
32
48
64
96
128
S11
62
46.30
47.82
48.58
49.12
49.27
47.81
ITQ 17
13.39
22.58
33,12
39.67
40.80
40.16
DSH
84
41.42
45.30
48.72
47.85
48.50
50.76
LSH
16
26.69
31.49
33.21
35,64
38.92
38.53
15.60
30.78
35.61
35.55
39.31
OCH
35
24.23
32.90
36.10
36.34
39.95
44,36
ITQ
79
50.22
49.66
46.64
44.38
44.53
43.21
Lap
10
54,19
55.24
$4.57
55.77
$4,58
54.08
GTH-g
36.66
32.11
DAPH 21
47.53
54.86
57.36
60.15
60.65
60.39
PWCF 22
50.21
49.41
57.27
60.06
64.71
64.00
DilLing
64
51.25
50.48
58.44
63.13
65.09
67.02
PEACE
52.77
56.25
61.03
65.27
67.71
69.99
60.81
63.32
68.05
72.11
74.47
76.73
A2R
A2D
A2W
PEACT
REACE
Procision
0.6
Precision
0.6
Procision
0.6
0.4
0.0
0.0
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.0
0.6
0.8
1.0
Recall
Recall
Recall
W2D
D2A
W2A
DES:
Precision
Precision
Precision
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.2
0.4
0.6 Recall
0.8
10
0.0
022
0.4
0.6
0.0
0.2
0.4
0.6
1.0
Recall
Recall
Figure 5: We use 128-bit Precision-Reee Recall curves to evaluate the performance of our method on the Office-31 and Office-Homm datasets.
code length increases, the MAP value IDEA also increases, ultimately surpassing the performance of all other hashing methods at various hash code lengths. This indicates that IDEA has stronger feature extraction and hashing encoding capabilities and has significant practical value for image retrieval.
Table compares the average MAP of different hashing methods under different bit lengths for the single-domain retrieval task, i.e., Product - Real. We can see that IDEA performs better than other hashing methods at hash code lengths, especially Furthermore, the table highlights the results of IDEA in bold font and pink background, indicating that it achieves the highest MAP score at all bit lengths, namely 34.88, 44.83, 48.38, 49.91, 53.55, and 54.40. This further highlights the strong performances and advantages of IDEA, especially in learning compact hash codes and improving retrieval efficiency for single-domain retrieval tasks.
higher bit lengths such as 96, 128.
Table compares the average MAP of different hashing methods under different bit lengths for the single-domain retrieval task MNIST -> USPS. We can see that IDEA achieves the highest MAP values compared to other methods at all bit lengths, especially at higher bit lengths such as 96 and 128 bits, with MAP values of 74.47 and 76.73 respectively, far exceeding methods. Additionally, the table highlights the results of IDEA in bold font and pink background, indicating that it achieves the highest MAP values information from the table,
MAP values of other
bit lengths. Therefore, combining the conclude that IDEA demonstrates outstanding performance the
17

---

P2R
C2R
R2A
0.7
077
0.00
0.8
IDR
IDEA
0.55
IDEA
Predsion
0.5
0.4
DHLing
PEACE
Preditoon
04
DHLLing
PEACE
0.60
0.45
PEAGE
DHLing
PWCF
PWCF
PWGF
0.50
025
250
son
Number of Top Ranked Samples
Number Top Ranked Samples
Numberol Top Ranked Samples
R2P
R2C
A2R
0.7
0.50
0.6
IDEA
IDEA
IDEA
Precision
0.5
PEACE
DHLing
0.00
0.50
0.5
PEACE
DHLlng
Precision
PEACE
DHLing
PwCF
025
PwCr
PWCF
a20
0.15
50 150 200 250 soo 350 400
50 100 150 s0o 250 30o são 400
100 150 200 ase soo 400
Number of Top Ranked Samples
Number of Top Ranked Samples
Number of Top Ranked Samples
A2D
A2W
W2D
IDEA
IDEA
Procision
0.5
PEACE
Precision
Predston
DHLing
DiiLing
05
DHILing
pwGr
PWCF
PWCF
0.3
50 150 soo 250 soo 550 400
50 100 150 300 a5o 500 sã0 400
0 50 100 150 200 aso 300 350 400
Number of Top Ranked Samples
Number of Top Ranked Samples
Number of Tf' Ranked Samples
0.8
n8
Prectsion
PEACE
Precision
PEACE
Precision
PEACE
0.4
DHLing
DHLLng
DHling
05
PWCr
PWCF
04
PWCF
0.3
150 Number of Top Ranked Samples
150 35o 400
300 3ã0 400
Number Top Ranked Samples
Number Top Ranked Samples
Figure 6: Top-N precision curves with 128 bits on Office-31 and Office-Home datasets.
single-domain retrieval task from MNIST USPS, significantly enhancing retrieval efficiency and accuracy. This result suggests that IDEA can be a promising approach for improving image retrieval in similar domains.
D.2 Performance of Precision-Recctt Curves
Additional Precision-Recall curves for six image retrieval tasks, namely A2R, A2D, A2W, W2D, D2A, and W2A, can be found in this section. These tasks involve retrieving images from one domain and matching them images from another domain. The Precision-Recat curves offer a visual representation of the balance between precision and recall for each task. As illustrated in Figure these curves give insight into the effectiveness of our approach in dealing with different cross-domain retrieval scenarios. In particular, the curves highlight the strengths of our method in achieving high precision and recall values for each task. The supplementary material provides a detailed analysis of the experimental results, which further demonstrates the robustness and generalized ability of our approach. Overall, these additional results reinforce the effectiveness of our method improving cross-domain image retrieval performance.
D.3 Performance of Top-N Precision Curve and Top-N Recall Curve
The Top-N precision curve and Top-N recall curve depicted Figure Figure respectively, provide valuable insights into the performance of our IDEA algorithm compared to other methods in
18

---

P2R
0.9
IDEA
IDEA
Precision
0.7
PEACE
Precision
0.8
PEACE
DHLlng
DHLLig
0.5
DHLing
0.5
PWGF
0.6
PWCF
02
PWCT
50
15o asn 500 seo goo
Number of Top Ranked Samples
Top Ranked Samples
Number Top Ranked Samples
RCP
IDEA
0.5
0.6
Predeion
Predision
PEACE
Precision
IDEA
DilLing
IDEA
PEACE
PEACE
0.4
DHLing
0.4
PWCF
0.4
DHLing
PWCT
PWCF
50 150 şoo 360 400 Number Top Ranked Samples
Number of Top Ranked Samples
Number Top Samples
A2D
A2W
W2D
1.-
IDEA
IDEA
IDEA
Precision
PEACE
Precision
0.7
PEACE
Precision
PEACE
0.G
DHLlng
0.6
50 100 15o zon zon s0o 550 400
0 50 100 150 zoo u5o zon son 5oo
0
50 100 15o aon asn soo s5n goo
Number of Top Ranked Samples
Number of Top Ranked Samples
Number of Top Ranked Samples
D2A
W2A
D2W
1.0
0.5
IDEA
Prociston
Precisien
Precision
0.6
DHLing
DHiling
DHLling
PWCP
0.5
PWCT
PWCF
0.4
0.6
50 150 zon zin siu 350 400
100 120 200 250 soo 350 400
100 150 200 2an sou zou Number of Top Ranked Samples
Number of Top Ranked Samples
Number of Top Ranked Samples
Figure 7: Top-N recall curves with 128 bits on Office-31 and Office-Home datasets.
image retrieval tasks. Based on the analysis of these curves, we can conclude that our IDEA algorithm outperforms other comparison methods. This is attributed to the fact that the calculation of the Top-N precision and Top-N recall curves based on the Hamming distance ranking, and the IDEA algorithm achieves better performance under this metric. Therefore, from the perspective of evaluation metrics, the IDEA algorithm superior to other comparison methods, as it offers superior precision and recall values for various cross-domain retrieval tasks, Our extensive experimental results further validate the effectiveness of our approach in improving the performance cross-domain image retrieval.
D.4 Ablation Study
We conduct more ablation studies on more datasets and the results are shown in Table The results can validate our superiority again.
Table 7: Ablation Studies on three benchmark cross-domain retrieval tasks.
Office-Home
Office31
Product2Real
Real2Product
Webcam21slr
Dslr2Webcam
IDEA w/o L_v
58.23
60.92
83.88
87.22
IDEA w/o MI
57.87
60.46
83.71
87.32
Ours
59.18
61.84
84.97
88.69
19

---

Query
Top 10 Retrieved Images
Ours P@ 10 100%
old
A
PEACE P 10 90%
Bike
Ours P
90%
PEACE
P 10 60%
Phone
Ours P@ 100%
DELL
PEACE P@ 80%
Desktop computer
Figure Top 10 Images and Precision 10 Examples on the Office-31 Dataset.
D.5 Performance of Different Architectures
We compare the performance of both PEACE and IDEA with different network backbones, i.e., VGG- and VIT. The compared results are shown Table From the results, we can have the following observations. Firstly, our IDEA can still achieve better performance under different architectures. Secondly, with strong networks, the performance of our IDEA can improve a lot.
Table 8: MAP scores with respect to different architectures.
Office-Homm.
Office31
Digits
Datasel
Clipart2Real
Real2Art
Amazon2Dslr
Amazon2Weheam
USPS2MNIST
MNIST2USPS
PEACE (VGG-F)
38.72
42.68
46.69
48.89
60.91
62.84
Ours (VGG-F)
45.71
49, 64
48. 70
54.43
67.97
67.48
PEACE (VIT)
53.83
56.11
56.05
60.25
79.28
78.37
Ours (VTT)
58.56
64.39
63.27
66.30
83.39
82.96
D.6
Case Study
Figure 8 demonstrates the top 10 retrieval results the proposed IDEA and the best baseline method PEACE for three query samples the Office-31 dataset. The blue box indicates the query, the green box indicates the correctly retrieved result, and the red box indicates the incorrectly retrieved result. The results demonstrate that IDEA performs better than PEACE in providing more relevant and user-desired examples, with the latter typically used for cross-domain retrieval. For instance, compared PEACE, our IDEA returns more examples of the query "Phone," further validating its success in image retrieval tasks. Overall, these findings emphasize the effectiveness of our IDEA method in improving cross-das domain image retrieval performance.
20


Reviews:
[{"review_id": "rnfxhjI7gZ", "reviewer": "Reviewer_BQ1H", "summary": "This work studies apply hashing for domain adaptive image retrieval. Specifically, the authors propose Invariance-acquired Domain Adaptive Hashing (IDEA) to consider alignment invariance, and causal effects. IDEA decomposes each image into causal and non-causal features, and introduce invariant learning to minimize the variance under different intervention. The experiments validate the effectiveness of the proposed method.", "questions": "Plz see weakness", "limitations": "The work discusses the limitations of this work.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The problem is interesting.\n- The techniques seem correct.", "weaknesses": "- The motivation of this work seems to be not stated clearly. The author claim invariance needed, however it lacks explanation in whether and why it is important in hashing. It also applies for the use of causal feature.\n\n- This work is a simple combination of several techniques, e.g., contrastive learning, causal learning, invariance learning. It does not provide new insights to me.\n\n- I have some concerns on the empirical results. In ablation study, the most important modules claimed in this work, i.e., L_V, L_MI have little improvement on the performance, and thus the importance of the two modules is questionable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are truly grateful for the time you have taken to review our paper and your insightful review. Here we address your comments in the following.\n\n> Q1. The motivation of this work seems to be not stated clearly. The author claim invariance needed, however it lacks explanation in whether and why it is important in hashing. It also applies for the use of causal feature.\n\nA1. Thanks for your comment. The reason for using invariance for hashing is to learn domain-invariant hash codes, which is quite important for effective cross-domain retrieval. In particular, cross-domain retrieval is that give a query in one domain, we aim to get similar samples in the other domain. Clearly, we need to enforce that samples with the same semantics have small distance, which indicate the domain-invariance. The reason for using causal features is that causal features are related to semantics instead of backgrounds, which can generate hash codes discriminative to different semantics and invariant to backgrounds. \n\n> Q2. This work is a simple combination of several techniques, e.g., contrastive learning, causal learning, invariance learning. It does not provide new insights to me.\n\nA2. Thanks for your comment. Actually, our method does not utilize classic contrastive learning. The novelty of the proposed methods is three points:\n- **Generation-based causal model**. We propose a generation-based structural causal model to analyze the relationship under domain shift, which is the first in domain adaptation and image retrieval to our best knowledge. \n- **Implementation: disentangle and intervene**. To resolve the domain shift, our method utilize disentangled each image into casual features and non-casual features and then add intervention with reconstruction, which is quite different from existing causal learning and invariance learning methods. \n-  **A unified framework**. We incorporate our components in a unified framework for domain adaptive hashing, which can achieve superior performance in cross-domain retrieval.\n\n\n> Q3. I have some concerns on the empirical results. In ablation study, the most important modules claimed in this work, i.e., L_V, L_MI have little improvement on the performance, and thus the importance of the two modules is questionable.\n\nA3. Thanks for your comment. We have added more ablation studies on more datasets and the results are shown below. From the results in Table 3 and below, we can observe that the performance gain of the full model is between [0.7%, 18.5%] and over 1% in most cases, which can validate the effectiveness of the proposed components. \n\n\n| Dataset |  Office-Home |          \t|   Office31  |         \t|\n|---------|:------------:|:------------:|:-----------:|:-----------:|\n| Tasks   | Product2Real | Real2Product | Webcam2Dslr | Dslr2Webcam |\n| w/o L_v  | \t58.23\t| \t60.92\t|\t83.88\t|\t87.22\t|\n| w/o L_MI | \t57.87\t| \t60.46\t|\t83.71\t|\t87.32\t|\n| Ours\t| \t59.18\t| \t61.84\t|\t84.97\t|\t88.69\t|\n\nIn light of these responses, we hope we have addressed your concerns, and hope you will consider raising your score. If there are any additional notable points of concern that we have not yet addressed, please do not hesitate to share them, and we will promptly attend to those points.\n"}, {"review_id": "DBxIOaW694", "reviewer": "Reviewer_MFjc", "summary": "- In this research, the authors investigate the problem of unsupervised domain adaptation for hashing, which aims to expedite learning on a target domain with limited label information by leveraging knowledge from a source domain with abundant labels. The IDEA model begins by decomposing each image into a causal feature that captures label information and a non-causal feature that represents domain information. The authors then utilize consistency learning on both source and target domains to generate discriminative hash codes based on the causal features. Additionally, the authors employ a generative model for synthetic samples to simulate various non-causal effects, ultimately minimizing their impact on the domain invariant hash codes.\n- The authors conduct extensive experiments on benchmark datasets to evaluate the performance of their IDEA model against a variety of competitive baselines. The results demonstrate that the IDEA approach outperforms the others, showcasing its superiority in handling unsupervised domain adaptation for hashing.", "questions": "1. The IDEA pipeline has numerous modules and loss functions, which may make it challenging to transfer and fine-tune for other tasks or datasets. Could the authors provide some explanations and clarifications on this matter?", "limitations": "- One potential drawback of this approach is that it may not be effective in open-set scenarios, where target samples could potentially come from unseen classes.\n- To address this limitation, more advanced techniques and models (such as those in AICG and multimodal large-scale models) can be incorporated into the research work to improve the generalizability of the method in a wider range of scenarios.\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Clarity: The authors have clearly and concisely explained their methods, results, and conclusions in a way that is easy for other researchers to understand and replicate.\n- Quality of analysis: The authors have conducted extensive experiments and analyses, and the experimental data demonstrate the effectiveness and persuasiveness of the IDEA method.\n- Theoretical analysis: The authors have provided a thorough theoretical analysis of the research problem, exploring new insights and frameworks that contribute to the advancement of knowledge in the field.\n\n", "weaknesses": "- from the Ablation Study(line 310) on six benchmark cross-domain retrieval tasks, it can be observed that the effectiveness of these four Loss functions cannot be well demonstrated. It is suggested to consider incorporating more permutations and combinations in order to verify the validity of each component.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are truly grateful for the time you have taken to review our paper, your insightful comments and support. Your positive feedback is incredibly encouraging for us! In the following response, we would like to address your major concern and provide additional clarification.\n\nQ1：It is suggested to consider incorporating more permutations and combinations in order to verify the validity of each component.\n\nA1: Thanks for your comment. We have introduced three different model variants as below. From the results, we can validate their validity. \n\n| Dataset           \t|  Office-Home |      \t|   Office31  |           \t|\n|-----------------------|:------------:|:--------:|:-----------:|:-------------:|\n| Tasks             \t| Clipart2Real | Real2Art | Amazon2Dslr | Amazon2Webcam |\n| w/o L_MI + L_v    \t| \t43.32\t|   46.60  |\t45.24\t| \t52.21 \t|\n| w/o L_CL + L_v    \t| \t40.09\t|   42.78  |\t41.77\t| \t50.14 \t|\n| w/o L_MI + L_v + L_RE | \t40.62\t|   43.17  |\t42.58\t| \t50.35 \t|\n| Ours              \t| \t45.71\t|   49.64  |\t48.70\t| \t54.43 \t|\n\n\n\n> Q2. The IDEA pipeline has numerous modules and loss functions, which may make it challenging to transfer and fine-tune for other tasks or datasets. Could the authors provide some explanations and clarifications on this matter?\n\nA2. Thanks for your comment. Although IDEA has numerous modules and loss functions, the parameter analysis shows the consistency of model parameters, which make the model easy to be transferred in other datasets. As for different tasks, we would further extend the strengths of the proposed IDEA in scenarios such as image classification and Re-ID in the future. \n \nWe will also add your suggestion about future works into our revised version. Thanks again for appreciating our work and for your constructive suggestions. Please let us know if you have further questions.\n"}, {"review_id": "OMbpqp9VHv", "reviewer": "Reviewer_z9GY", "summary": "This paper proposes a novel method called Invariance-acquired Domain Adaptive Hashing for generating high-quality and interpretable hash codes. The approach incorporates the concepts of causal and non-causal features, leveraging the information bottleneck principle and consistency learning to optimize the hash network. The paper presents the training algorithm and experimental settings for IDEA and provides qualitative and quantitative analysis of the experimental results. Overall, this paper proposes a novel approach to address certain challenges in hash code generation and achieves promising performance on multiple benchmark datasets.", "questions": "1. More clear contribution.\n2. Convergence analysis.", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. A structure causal model is adopted to explain the problem of unsupervised domain adaptation image retrieval.\n2. This paper employs mutual information as a metric to achieve feature disentanglement, which provides a solid theoretical foundation for the paper.\n3. The experimental results demonstrate that the proposed method outperforms the compared methods with a large margin.", "weaknesses": "1. The contribution of the structure causal model is not clear. In fact, there is no usage of casual inference for image retrieval.\n2. The paper does not provide the convergence analysis of mutual information. Mutual information is sensitive to noise and outliers in the data. Thus, it is necessary to verify the convergence of the proposed method.\n3. In the sensitivity analysis, the paper does not provide the recommend range of \\beta and \\tao.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are truly grateful for the time you have taken to review our paper, your insightful comments and support. Your positive feedback is incredibly encouraging for us! In the following response, we would like to address your major concern and provide additional clarification.\n\n>Q1. The contribution of the structure causal model is not clear. In fact, there is no usage of causal inference for image retrieval.\n\nA1. Thanks for your comment. The contribution of the SCM is three-fold:\n- **Problem understanding**. It can provide the explanation for the problem of unsupervised domain adaptation image retrieval.\n- **Methodology formulation**. Based on the proposed SCM, we propose the condition that our mapping should fulfill, which guides the formulation of the proposed strategy.\n- **Implementation: disentangle and intervene**. To resolve the domain shift, our method disentangles each image into casual features and non-casual features and then adds intervention with reconstruction, which is based on image generation and decomposition from the SCM. \n\n>Q2: The paper does not provide the convergence analysis of mutual information. Mutual information is sensitive to noise and outliers in the data. Thus, it is necessary to verify the convergence of the proposed method.\n\nA2: Thanks for your comment. We have provided the loss with respect to different epochs for the proposed method. In particular, the MI loss is [0.0012, 0.0007, -0.0025, -0.0197, -0.0387, -0.0718, -0.0917, -0.1543, -0.2545, -0.3567, -0.4821, -0.6251, -0.8765, -1.2345, -1.5678, -1.9012, -2.3045, -2.5542, -2.6787, -2.8011, -2.9078, -2.9987, -3.0521, -3.1211, -3.1236, -3.1236, -3.1236, -3.1237, -3.1237, -3.1237] for the total 30 epochs. From the results, we can obtain the mutual information would converge empirically. \n\n\n \n>Q3: In the sensitivity analysis, the paper does not provide the recommend range of \\beta and \\tao.\n\nA3: Thanks for the comment. The recommended value for \\beta and \\tao is 0.2 and 0.1, respectively. We will add this into the revised version. \n \n\nThanks again for appreciating our work and for your constructive suggestions. Please let us know if you have further questions.\n"}, {"review_id": "nLzR8sLieB", "reviewer": "Reviewer_8Msq", "summary": "This paper focuses on an unsupervised domain adaptation method for deep hashing. This paper proposes to disentangle each image into causal and non-causal features, where casual features represent label information and non-causal features represent domain information. The causal features are used to compute hash codes, while both causal and non-causal features are used to reconstruct the image. The disentanglement loss is to minimize the mutual information between non-causal features and label and maximize the mutual information between non-causal features and the hidden features. As a result, the proposed method performs better than prior methods in multiple benchmarks.", "questions": "- how effective is this proposed method on vision transformer-based method where attention mechanism is involved? will causal features only attend to certain parts? will it overlap with non-causal features?\n- can synthetic images show that different non-causal features can generate different images from different domains? \n- why do you need a warm-up for learning Eq. 15? What causes the instability at the beginning of the training?", "limitations": "- the author mentioned that their method cannot tackle the open-set scenarios, the proposed solution is for seen classes scenario only", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- the proposed method outperforms state-of-the-art methods on cross-domain retrieval\n- the structural causal model on learning hash codes and the disentanglement loss on learning hash codes are interesting. This way is intuitively able to disentangle causal and non-causal features from an image. I believe this can be one of the regularization term for future retrieval/hashing methods.", "weaknesses": "- while using non-causal features from other samples for the reconstruction of hash codes seems intuitive, no experiments or analysis on the proposed intervention scheme\n- it is unknown whether the extracted features for hash codes are indeed the causal part. \n- no interpretability analysis. While the author claims that their method can generate high-quality interpretable hash codes, the only evaluation is on performance analysis, with no other quantitative and qualitative analysis to claim that the hash codes are interpretable.\n- no visualization of the causal and non-causal features, not interpretable on how those features correlate to which part of the image\n- although testing with VGG features for unsupervised hashing is a standard since this method involves disentangling features, the author should test with more different kinds of architectures such as ResNet or Vision Transformer. It is because VGG features are globally pooled features, thus already not directly interpretable.\n- no synthetic samples have been shown. As the author uses an intervention scheme on generating images and then learns to extract hash codes that have low variance, what will the samples look like?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are truly grateful for the time you have taken to review our paper, your insightful comments and support. Your positive feedback is incredibly encouraging for us! In the following response, we would like to address your major concern and provide additional clarification.\n\n> Q1. While using non-causal features from other samples for the reconstruction of hash codes seems intuitive, no experiments or analysis on the proposed intervention scheme.\n\nA1. Thanks for your comment. We have added a model variant IDEA w/o I by removing the proposed intervention. The compared performance is shown below. From the results, we can find that the proposed intervention scheme makes a crucial effect on the performance. \n\n| Dataset |  Office-Home |              |              |          |   Office31  |             |             |               |\n|---------|:------------:|:------------:|--------------|----------|:-----------:|:-----------:|-------------|---------------|\n| Task    | Product2Real | Real2Product | Clipart2Real | Real2Art | Webcam2Dslr | Dslr2Amazon | Amazon2Dslr | Amazon2Webcam |\n| IDEA w/o I |     58.24    |     60.36    |     45.10    |   49.09  |    83.49    |    52.19    |    48.05    |     53.79     |\n| IDEA   |     59.18    |     61.84    |     45.71    |   49.64  |    84.97    |    53.53    |    48.70    |     54.43     |\n\n\n> Q2. It is unknown whether the extracted features for hash codes are indeed the causal part.\n\nA2. Thanks for your comment. We have added several visualizations to show the gradient of causal features. The results are shown in Figure A. From the results, we can observe that extracted features are indeed with the casual part in the image. \n\n\n\n> Q3. No interpretability analysis. While the author claims that their method can generate high-quality interpretable hash codes, the only evaluation is on performance analysis, with no other quantitative and qualitative analysis to claim that the hash codes are interpretable.\n\nA3. Thanks for your comment. We have added several visualizations to show the gradient of causal features. The results are shown in Figure A. From the results, we can observe that extracted features are indeed with the casual part in the image. Moreover, since hash codes are direct from the causal features, they have the similar gradients, which provide the interpretability of hash codes.\n\n> Q4. no visualization of the causal and non-causal features, not interpretable on how those features correlate to which part of the image\n\nA4. Thanks for your comment. Actually, since the causal and non-causal features are high-dimensional features, it is hard to get information from their visualization. Instead, we have added several visualizations to show the gradient of causal features. The results are shown in Figure A. From the results, we can observe that extracted features are indeed with the casual part in the image. \n \n> Q5: Although testing with VGG features for unsupervised hashing is a standard since this method involves disentangling features, the author should test with more different kinds of architectures such as ResNet or Vision Transformer. It is because VGG features are globally pooled features, thus already not directly interpretable.\n\nA5: Thanks for your comment. We have implemented both PEACE and IDEA with ViT formulation. The compared results are shown below. From the results, we can observe that 1) our IDEA can still achieve better performance under different architectures 2) with strong networks, the performance of our IDEA can improve a lot. \n\n| Dataset\t|  Office-Home |      \t|   Office31  |           \t|   Digits   |            |\n|------------|:------------:|:--------:|:-----------:|:-------------:|:----------:|:----------:|\n| Tasks  \t| Clipart2Real | Real2Art | Amazon2Dslr | Amazon2Webcam | USPS2MNIST | MNIST2USPS |\n| PEACE (VGG) | 38.72|   42.68   |    46.69  |        48.89   |        60.91   |       62.84   |\n| Ours (VGG) | \t45.71\t|   49.64  |\t48.70    | \t54.43 \t|\t67.97   |\t67.48   |\n| PEACE (ViT) | 53.83|   56.11   |    56.05   |         60.25   |         79.28    |       78.37   |\n| Ours (ViT) | \t58.56\t|   64.39  |\t63.27\t| \t66.30 \t|\t83.39   |\t82.96   |\n \n> Q6. No synthetic samples have been shown. As the author uses an intervention scheme on generating images and then learns to extract hash codes that have low variance, what will the samples look like?\n\nA6. Thanks for your comment. We have added the visualization of generalization images in Figure B. From the results, we can observe that our method has the potential to generate samples with different backgrounds, which can help to learn domain-invariant hash codes.\n \n\nThanks again for appreciating our work and for your constructive suggestions. Please let us know if you have further questions.\n"}, {"review_id": "7jEliZfmDg", "reviewer": "Reviewer_fabx", "summary": "This paper proposes to use invariant learning based on causality inference for domain adaptive retrieval. In the proposed IDEA model, a feature disentanglement module is deployed for obtaining causal and non-causal features. A generative model is designed with non-causal features intervention for reconstructing images. Experiments show the good effectiveness.\n", "questions": "I would lean to accept this paper, after addressing the above point in weakness, because I think this paper has some novelty.", "limitations": "I have not found obvious issue.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+Deploying casuality perspective into domain adaptive hashing retrieval is a new point.\n+The proposed model is simple in methodology with encoder and decoder under intervention.\n+Experiments and comparisons to related methods show the superiority of the proposed model.", "weaknesses": "-How about the experimental setting? Is the setting the same as some related models, such as PWCF [22]? because the performance is much better than previous ones. It can be discussed.\n-Causality perspective is a common sense in domain generalization and out-of-distribution, although it is under-studied in domain adaptive hashing retrieval.\n-Please discuss the following work, because this work also considers to remove non-causal features. The idea is a little similar. Additionally, the authors claim that the non-causal features are domain information, but it may not be correct, because non-casual feature may still be domain invariant information. Causal feature should be contained in domain invariant features.\n\"Multi-view Adversarial Discriminator: Mine the Non-causal Factors for Object Detection in Unseen Domains, CVPR 2023.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are truly grateful for the time you have taken to review our paper, your insightful comments and support. Your positive feedback is incredibly encouraging for us! In the following response, we would like to address your major concern and provide additional clarification.\n\n>Q1. How about the experimental setting? Is the setting the same as some related models, such as PWCF [22]? because the performance is much better than previous ones. It can be discussed. \n\nA1. Thanks for your problem. Yes, we follow the same setting as PWCF and PEACE. We can observe that PEACE can indeed perform much better than PWCF and ours can even perform better than PEACE by a large margin.\n\n>Q2. Causality perspective is a common sense in domain generalization and out-of-distribution, although it is under-studied in domain adaptive hashing retrieval. \n\nA2. Thanks for your comment. Although causality has been explored in domain generalization and out-of-distribution, it is less-explored in domain adaptation, which is quite different in transfer learning. Moreover, it has also not been explored in image retrieval. Lastly, our causality perspective provides three views quite different from existing works:\n- **Problem understanding**. It can provide the explanation for the problem of unsupervised domain adaptation image retrieval.\n- **Methodology formulation**. Based on the proposed SCM, we propose the condition that our mapping should fulfill, which guides the formulation of the proposed strategy.\n- **Implementation: disentangle and intervene**. To resolve the domain shift, our method disentangles each image into casual features and non-casual features and then adds intervention with reconstruction, which is based on image generation and decomposition from the SCM. \n\n>Q3. Please discuss the following work, because this work also considers to remove non-causal features. The idea is a little similar. Additionally, the authors claim that the non-causal features are domain information, but it may not be correct, because non-casual feature may still be domain invariant information. Causal feature should be contained in domain invariant features. \"Multi-view Adversarial Discriminator: Mine the Non-causal Factors for Object Detection in Unseen Domains, CVPR 2023.\"\n\nA3. Thanks for your comment. Although our work also considers to remove non-causal features, there are three key distinctions that set our work apart:\n- **Different motivations**. We propose a generation-based structural causal model to analyze the relationship in domain adaptation retrieval, while [1] learns from frequency spectrum to generate causal features.\n\n\n- **Different scenarios**. Our model focuses on image retrieval under domain adaptation settings while [1] focuses on domain generalization for image segmentation. Their scenarios have a huge difference.\n- **Different methodology**. Our method disentangles each image based on mutual information and then adds intervention with reconstruction, which is based on image generation and decomposition from the SCM. In contrast, [1] focuses on Spurious Correlations Generator from frequency domain with domain adversarial learning.\n\nWe will definitely add this discussion in our revised version. \n\n**Reference**\n\n[1] Multi-view Adversarial Discriminator: Mine the Non-causal Factors for Object Detection in Unseen Domains, CVPR 2023\n \nThanks again for appreciating our work and for your constructive suggestions. Please let us know if you have further questions.\n"}]

Author rebuttal:
Dear Reviewers,

We thank you for your careful reviews and constructive suggestions. We acknowledge the positive comments such as "The problem is interesting" (Reviewer  BQ1H), “The techniques seem correct” (Reviewer  BQ1H), "Clarity" (Reviewer MFjc), "Quality of analysis” (Reviewer MFjc), “Theoretical analysis” (Reviewer MFjc), “solid theoretical foundation” (Reviewer z9GY), “outperforms the compared methods with a large margin.” (Reviewer z9GY and 8Msq), “interesting” (Reviewer 8Msq), “a new point” (Reviewer fabx). We have also responded to your questions point by point.

The figure results are attached in the PDF. Please let us know if you have any follow-up questions. We will be happy to answer them.

Best regards,

the Authors


Meta-review: This paper introduces the Invariance-acquired Domain AdaptivE HAshing (IDEA) model, which effectively addresses the challenges of unsupervised domain adaptive hashing by decomposing images into causal and non-causal features, generating discriminative hash codes, and minimizing the impact of non-causal effects, demonstrating superior performance over the competitive baselines through the experiments on the benchmarks. It addresses an interesting problem using correct techniques, with clear and concise explanations, extensive experiments validating the effectiveness of the IDEA method, a strong theoretical foundation, and superior performance compared to the state-of-the-art methods in cross-domain retrieval. Despite some negative feedback and flaws, such as those issues with paper clarity and the design principles of certain modules, the contributions made in the paper are not overshadowed. Most of the reviewers provided positive evaluations. Therefore, the AC decides to accept this paper.
Decision: Accept (poster)