SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haixin Wang $^{1,*}$ , Hao Wu $^{2,*}$ , Jinan Sun $^{1}$ , Shikun Zhang $^{1}$ , Chong Chen $^{1}$ , Xian-Sheng Hua $^{3}$ , Xiao Luo $^{4,\dagger}$
<sup>1</sup>Peking University, <sup>2</sup>University of Science and Technology of China, <sup>3</sup>Terminus Group, <sup>4</sup>University of California, Los Angeles wang.hx@stu.pku.edu.cn, wuhao2022@mail.ustc.edu.cn, sjn@pku.edu.cn, zhangsk@pku.edu.cn, chenchong.cz@gmail.com, huaxiansheng@gmail.com, xiaoluo@cs.ucla.edu
In this paper, we study the problem of unsupervised domain adaptive retrieval, which transfers retrieval models from a label-rich source domain to a label-scarce target domain. Although there exist numerous approaches that incorporate transfer learning techniques into deep hashing frameworks, they often overlook the crucial invariance needed for adequate alignment between these two domains. Even worse, these methods fail to distinguish between causal and non-causal effects embedded in images, making cross-domain retrieval ineffective. To address these challenges, we propose an Invariance-acquired Domain Adaptive Hashing (IDEA) model. Our IDEA first decomposes each image into a causal feature representing label information and a non-causal feature indicating domain information. We then generate discriminative hash codes using causal features with consistency learning on both source and target domains. More importantly, we employ a generative model for synthetic samples to simulate the intervention of various non-causal effects, thereby minimizing their impact on hash codes for domain invariance. Comprehensive experiments conducted on benchmark datasets confirm the superior performance of our proposed IDEA compared to a variety of competitive baselines.
Approximate nearest neighbor (ANN) search [73, 77, 49, 29, 40] has been a pivotal research problem within the field of deep learning for decades. Recently, learning to hash [41, 80, 68, 20] has come to the fore in ANN search engines due to its high computational and memory efficiency. The main objective of learning to hash is to convert high-dimensional data points into compact low-dimensional binary codes while maintaining semantic similarity. Consequently, this approach mitigates the requirement for computationally expensive pairwise distance computations, replacing them with economical Hamming distance calculations using bit-wise XOR and bit-count operations [60].
Recent hashing techniques can primarily be classified into two types, namely supervised approaches [72, 44, 67, 48, 75, 8, 31, 52] and unsupervised approaches [32, 76, 46, 69, 33, 61, 10]. Supervised hashing approaches incorporate annotation data into the similarity learning algorithms, generally leading to better performance than unsupervised ones. These supervised methods primarily focus on minimizing a spectrum of similarity-preserving losses and quantization losses [11, 78] to generate highly discriminative hash codes. Furthermore, several innovative proxy-based techniques have been introduced where a proxy is constructed in the Hamming space for each class, and hash
codes are enforced to approximate their respective proxies. These methods have demonstrated encouraging results in efficient image retrieval.
However, these supervised approaches rely on an assumption that the distributions between the training and query data are the same, which is often violated in real-world scenarios. For instance, when an ANN search engine trained on benchmark images, is employed to search query images from mobile phones, retrieval performance could potentially diminish substantially [45, 22, 64]. In light of this, this paper concentrates on the problem of domain adaptive retrieval, which aims to enhance the performance of retrieval systems by leveraging both labeled source instances and unlabeled target samples. In literature, a variety of domain adaptive hashing methods have been proposed [71, 66, 64, 47, 74, 21, 58], which typically incorporate domain adaptation strategies into deep hashing frameworks. They adopt the memory bank [64] and adversarial learning [58, 36] to achieve alignment across source and target domains. Moreover, pseudo-labeling [19, 58] is implemented to extract knowledge from unlabeled target samples, thus addressing the challenge of label scarcity in the target domain.
Unfortunately, existing approaches have two intrinsic limitations that hinder their effectiveness as follows: (1) Entanglement of causal and non-causal effects. Current methods tend to recklessly focus on correlations between image data and semantic labels. This could result in an entanglement of causal and non-causal factors. Consequently, the hash codes derived from these deep features may lack interpretability and be suboptimal for subsequent retrieval tasks. (2) Neglect of invariant patterns. While adversarial learning is commonly used to implicitly align hash codes between source and target domains, these methods often fail to explicitly capture the invariant patterns present in vision representations. This could lead to ineffective domain alignment of hash codes due to the influence of underlying variant factors. As a result, an effective hashing framework under the principle of invariant learning [2, 5] is highly desired.
In this paper, we explore domain adaptive hashing through a causal analysis lens, proposing a model named Invariance-Acquired Domain Adaptive Hashing (IDEA) for efficient image retrieval under domain shift. The core of domain adaptive hashing lies in discerning causal and non-causal effects within images. To generate high-quality and interpretable binary codes, we disentangle causal and non-causal features within each image, guided by the principles of the information bottleneck. On one hand, we maximize the mutual information between causal features and label information. On the other hand, we retain the most information in hidden features for non-causal features while minimizing the information in labels. These causal features are used to generate discriminative hash codes through consistency learning across both source and target domains. To further mitigate the non-causal effects, we incorporate a generative model that simulates the intervention of various non-causal effects, thereby encouraging hash codes to remain sufficiently invariant to different non-causal components. Leveraging invariant learning on causal effects, our IDEA is capable of generating domain-invariant hash codes that facilitate efficient cross-domain retrieval. Empirical experiments conducted on a variety of benchmark datasets validate the superior performance of our IDEA compared to competitive baselines. The contribution of this paper is summarized as follows:
- Problem Connection. We pioneer a novel perspective that connects invariant learning with domain adaptive hashing for efficient image retrieval.
- Novel Methodology. Our IDEA not only disentangles causal and non-causal features in each image following the principle of the information bottleneck, but also ensures hash codes are sufficiently invariant to the intervention of non-causal features.
- High Performance. Comprehensive experiments across numerous datasets demonstrate that our IDEA outperforms a range of competitive baselines in different settings.
Learning to Hash. The task of efficient image retrieval has recently witnessed significant interest [49, 29, 40, 37], leading to the development of deep hashing methods [41, 80, 68, 20], including both supervised [72, 44, 67, 48, 75, 8, 31, 52, 54] and unsupervised approaches [32, 76, 46, 69, 33, 61, 56, 10, 55]. Deep unsupervised hashing methods typically reconstruct semantic structures using similarity metrics, subsequently generating similarity-preserving binary codes. Self-supervised methods have also been developed to enhance the performance. The incorporation of label information has notably improved the performance of deep supervised hashing methods. Early attempts at optimizing the
hashing network often involved pairwise and triplet losses based on similarity structures [44, 67, 30, 4]. Another line of study has utilized point-wise optimization, establishing proxies for each category within the Hamming space and compelling hash codes to approximate these proxies [72, 13]. However, these methods typically overlook potential distribution shift in practical scenarios, which can significantly degrade retrieval performance. This limitation has prompted research into domain adaptive hashing.
Unsupervised Domain Adaptation. Unsupervised Domain Adaptation (UDA) has long been a formidable challenge in machine learning and computer vision [38, 51, 14]. Early methods often explicitly reduce distribution discrepancies for domain alignment [43, 27]. An alternative approach utilizes adversarial learning for implicit domain alignment, incorporating a gradient reversal layer and a domain discriminator to engage in a minimax game [15, 36]. Recently, researchers have extended this topic towards efficient domain adaptive retrieval [71, 66, 64, 47, 74, 21, 58, 59], proposing a number of transferable hashing methods to address this problem. These methods typically integrate domain adaptation models into deep hashing frameworks and generate similarity structures for learning semantic knowledge on the target domain. Despite these advancements, these methods often fall short due to the entanglement of causal and non-causal effects and the neglect of invariant patterns. In response to these challenges, we propose an effective approach IDEA.
Invariant Learning. Invariant learning [2, 5, 9] aims to identify invariant correlations between inputs and targets under domain shift, while concurrently eliminating spurious and variant relationships. This concept has been explored in the context of out-of-distribution generalization. Under specific assumptions, invariant learning has demonstrated significant potential for model generalization following causal theory. Invariant risk minimization [2] (IRM) has been proposed to regularize neural networks to remain stable under the environmental variance, showing superior performance when compared to empirical risk minimization [12] (ERM). Moreover, MVDC [65] learns from frequency spectrum to generate causal features for domain generalization for cross-domain object detection. Invariant learning has also been extended to address challenges in graph domains [28, 70, 6, 53, 63]. For instance, CIGA [6] leverages distinct subgraphs for graph contrastive learning, thereby achieving superior out-of-distribution graph classification performance. Our study establishes a connection between invariant learning and domain adaptive hashing, generating domain-invariant and discriminative hash codes for cross-domain image retrieval. To the best of our knowledge, this is the first work to apply invariant learning for transferable retrieval tasks, and is proven effective for potential application.
Given a source domain  $\mathcal{D}^s = \{(\pmb{x}_i^s, y_i^s)\}_{i=1}^{N_s}$  with  $N_s$  fully-labeled images and a target domain  $\mathcal{D}^t = \{(\pmb{x}_j^t)\}_{j=1}^{N_t}$  with unlabeled  $N_t$  images, both domain share a common label space  $\mathcal{Y} = \{1, 2, \dots, C\}$  despite potential distribution shifts. The objective is to develop a hashing-based retrieval model that projects an input image  $\pmb{x}$  onto a compact binary code  $\pmb{b} \in \{-1, 1\}^L$ , where  $L$  represents the code length. This model should ensure that similar images are mapped to comparable binary codes within the Hamming space, and its performance should be evaluated in both single-domain and cross-domain retrieval systems. In a single-domain retrieval system, both query and database images originate from the target domain, whereas, in a cross-domain retrieval system, the query images are drawn from the target domain  $\mathcal{D}^t$  and the database images are sourced from the source domain  $\mathcal{D}^s$ .
We address the problem of unsupervised domain adaptive retrieval. While several domain adaptive hashing algorithms have been proposed, they often neglect invariant learning during the hash code generation process. This oversight often leads to significant domain dis
crepancy in the resulting hash codes. Moreover, these methods indiscriminately focus on
Figure 1: Structural causal model in our problem. Correlations between image data and semantic labels,
Figure 2: The framework of the proposed IDEA. We feed both source and target images into an encoder. Then each image is disentangled into causal features and a non-causal features. The causal features are adopted to generate hash codes. A generative model is utilized to reconstruct the original graphs. We add intervention by using a different non-causal features and minimize the domain shift by invariant learning.
failing to disentangle the causal and non-causal effects encoded in images. This results in learnt hash codes that are not only suboptimal for downstream retrieval tasks, but are also difficult to interpret.
In this study, we present a novel approach named IDEA, to tackle these issues. The fundamental idea is to incorporate invariant learning into the optimization of the hashing network. Specifically, we first introduce a structural causal model, as illustrated in Figure 1, and analyze the correlation within this problem. We then separate each image into causal and non-causal features using an information bottleneck. The causal features are subsequently used for hash code production. Lastly, we introduce interventions on non-causal effects and limit their influence on invariant hash codes. More details can be found in Figure 2.
4.2 Structure Causal Model
To start, we apply a causal perspective to domain adaptive hashing and construct a structural causal model [39] (SCM) to depict the image generation process under domain shifts. This model is used to illustrate the relationships between domains  $D$ , semantic labels  $Y$ , causal parts  $C$ , non-causal parts  $NC$ , images  $X$ , and hash codes  $H$ . The specific relationships are elaborated as follows:
-  $Y \rightarrow C$ . Semantic labels produce the causal part which should be invariant to domain distributions.
-  $D \to NC$ . Domain information provides the non-causal part which is changeable across domains.
-  $C \to X \gets NC$ . Each image is generated by combining both causal part and non-causal part.
-  $NC \to C \to H$ . To obtain domain-invariant hash codes, we should generate hash codes using causal features. However, there could be additional relationships between  $C$  and  $NC$ . For example,  $C$  and  $NC$  can be simultaneously influenced by a cause. Therefore,  $C$  acts as a confounder between  $NC$  and  $H$ , i.e., resulting in falsely related variables.
The essence of domain adaptive hashing lies in learning domain-invariant and discriminative binary codes. From the structural causal model, we need to construct mappings  $\phi(\cdot)$  and  $\psi(\cdot)$  which fulfill the following conditions:
$$
[ C, N C ] = \phi (X), H = \psi (C), H \perp N C \mid C, \tag {1}
$$
where  $\perp$  symbolizes the independence under the given condition. Inferred from Eqn. 1, we are required to address the following challenges: 1) The first term necessitates us to disentangle the causal and non-causal components of each image; 2) The second term requires us to learn discriminative hash codes from causal components; 3) The final term advocates for the generation of hash codes that are independent of the influence of non-causal components. With these goals in mind, we propose a novel deep hashing method IDEA for domain adaptive retrieval.
4.3 Causality-acquired Disentanglement
In this part, we apply the information bottleneck principle [1] to disentangle each image into causal and non-causal features. This approach facilitates the extraction of multifaceted latent factors embedded within images, enabling the generation of high-quality and interpretable binary codes. As suggested by the structural causal model, causal features should encapsulate label information, while non-causal features should reflect domain discrepancy.
In detail, we first introduce a feature extractor  $F(\cdot)$ , which removes the last classification layer in a popular neural network backbone (e.g., VGG-F [50]) to generate hidden features, i.e.,  $z = F(x)$ . Then, two different MLPs  $g^{c}(\cdot)$  and  $g^{n}(\cdot)$  are utilized to generate two features, respectively, i.e.,  $\pmb{f}^{c} = g^{c}(\pmb{z})$  and  $\pmb{f}^{n} = g^{n}(\pmb{z})$ . Let  $\pmb{F}^{c}$  and  $\pmb{Y}$  denote the random variables of causal features and semantic labels, and we maximize the mutual information between  $\pmb{F}^{c}$  and  $\pmb{Y}$ . To achieve this, we turn to InfoNCE [18, 3, 7], which construct positive pairs and negative pairs from the joint distribution  $p(\pmb{F}^{c},\pmb{Y})\in \mathcal{D}^{s}$  and the product of marginal distributions  $p(\pmb{F}^{c})p(\pmb{Y})\in \mathcal{D}^{s}$ , respectively. Since label information is not available in  $\mathcal{D}^t$ , we merely utilize source data in this module. In particular, an estimator  $T^c$  is introduced to estimate the lower bound of mutual information, the target objective is written as:
$$
\min  \mathcal {L} _ {M I} ^ {c} = \mathbb {E} _ {p (\boldsymbol {F} ^ {c}, \boldsymbol {Y})} \left[ T ^ {c} \left(\boldsymbol {f} ^ {c}, \boldsymbol {y}\right) \right] - \log \left(\mathbb {E} _ {p \left(\boldsymbol {F} ^ {c}\right) p (\boldsymbol {Y})} \left[ e ^ {T ^ {c} \left(\boldsymbol {f} ^ {c}, \boldsymbol {y}\right)} \right]\right), \tag {2}
$$
where  $T^{c}(\cdot, \cdot)$  comes from a bi-linear function with a weight matrix  $W^{c}$  to calculate the probability of being a positive pair, i.e.,  $T^{c}(\pmb{f}^{c}, \pmb{y}) = \pmb{f}^{c}\pmb{W}^{c}\pmb{y}$ . The estimator and the causal head  $g^{c}$  are optimized jointly to obtain effective causal features with high correlation to label information.
Moreover, to generate non-causal features, we minimize the mutual information between  $\pmb{F}^n$  and  $\pmb{Y}$  with the maximum mutual information with hidden features  $Z$  following the principle of information bottleneck. In formulation, we need to minimize:
$$
\min  \mathcal {L} _ {M I} ^ {n} = I (\boldsymbol {F} ^ {n}, \boldsymbol {Y}) - \beta I (\boldsymbol {F} ^ {n}, \boldsymbol {Z}), \tag {3}
$$
where  $\beta$  is a parameter to balance the loss. Similarly, it is infeasible to minimize Eqn. 3 directly. Therefore, we calculate the upper bound and lower bound of  $I(\pmb{F}^n, \pmb{Y})$  and  $I(\pmb{F}^n, \pmb{Z})$ , respectively. Here, we first define:
$$
\hat {I} \left(\boldsymbol {F} ^ {n}, \boldsymbol {Y}\right) = \mathbb {E} _ {p \left(\boldsymbol {F} ^ {n}, \boldsymbol {Y}\right)} \left[ \log p (\boldsymbol {y} \mid \boldsymbol {f} ^ {n}) \right] - \mathbb {E} _ {p \left(\boldsymbol {F} ^ {n}\right)} \mathbb {E} _ {p (\boldsymbol {Y})} \left[ \log p (\boldsymbol {y} \mid \boldsymbol {f} ^ {n}) \right], \tag {4}
$$
which can be shown as an upper bound of  $I(\pmb{F}^n, \pmb{Y})$ . More detail is shown in Appendix. Moreover, we introduce a variational function  $q(y|\pmb{f}^n)$  to approximate  $p(y|\pmb{f}^n)$ . To measure the lower bound of  $I(\pmb{F}^n, \pmb{Z})$ , we introduce a different estimator  $T^n(\pmb{f}^n, \pmb{z}) = \pmb{f}^n\pmb{W}^n\pmb{z}$  and the upper bound of  $I(\pmb{F}^n, \pmb{Z})$  is written as follows:
$$
\hat {I} \left(\boldsymbol {F} ^ {n}, \boldsymbol {Z}\right) = \mathbb {E} _ {p \left(\boldsymbol {F} ^ {n}, Z\right)} \left[ T ^ {n} \left(\boldsymbol {f} ^ {n}, \boldsymbol {z}\right) \right] - \log \left(\mathbb {E} _ {p \left(\boldsymbol {F} ^ {n}\right) p (\boldsymbol {Z})} \left[ e ^ {T ^ {n} \left(\boldsymbol {f} ^ {c}, \boldsymbol {z}\right)} \right]\right), \tag {5}
$$
By combining Eqn. 16 and Eqn. 5, we rewrite the objective in Eqn. 3 as:
$$
\begin{array}{l} \mathcal {L} _ {M I} ^ {n} = \mathbb {E} _ {p (\boldsymbol {F} ^ {n}, \boldsymbol {Y})} [ \log q _ {\mu} (\boldsymbol {y} \mid \boldsymbol {f} ^ {n}) ] - \mathbb {E} _ {p (\boldsymbol {F} ^ {n})} \mathbb {E} _ {p (\boldsymbol {Y})} [ \log q (\boldsymbol {y} \mid \boldsymbol {f} ^ {n}) ] \\ - \beta \left(\sup  _ {T ^ {n}} \mathbb {E} _ {p (\boldsymbol {F} ^ {n}, \boldsymbol {Z})} [ T ^ {n} (\boldsymbol {f} ^ {n}, \boldsymbol {z}) ] - \log \left(\mathbb {E} _ {p (\boldsymbol {F} ^ {n}) p (\boldsymbol {Z})} [ e ^ {T ^ {n} (\boldsymbol {f} ^ {n}, \boldsymbol {z})) ]\right)\right), \end{array} \tag {6}
$$
From Eqn. 6, we utilize adversarial learning to optimize non-causal features, which generates reserve gradients for estimator  $T^n$  when minimizing  $\mathcal{L}_{MI}^n$ . The overall loss for disentanglement is summarized as:
$$
\mathcal {L} _ {D} = \mathcal {L} _ {M I} ^ {c} + \mathcal {L} _ {M I} ^ {n}. \tag {7}
$$
By minimizing  $\mathcal{L}_D$ , we can successfully disentangle causal and non-causal elements embedded in images, which can guide us to generate high-quality and interpretable hash codes.
4.4 Consistency Learning for Discriminative Hash Codes
To learn discriminative hash codes, we leverage causal features given their strong correlation with label information for hash code generation. Here, consistency learning is introduced, based on the principle that similar images should be mapped to similar hash codes to enable effective image retrieval [4, 25]. This strategy ensures that our model can capture and preserve the inherent semantic relationships in the data, thus facilitating more accurate and effective retrieval.
In particular, we utilize an MLP  $\psi(\cdot)$  to map causal features to hash codes and enforce similar images to have similar binary codes. Here, we resort to consistency learning for optimization, which constructs positives via (1) two images sharing the same label in the source domain; (2) two augmented views from the same image in the target domain. In practice, we generate two augmented views for each image in a mini-batch  $\mathcal{B} = \mathcal{B}^s \cup \mathcal{B}^t$ , and define the positive set for source sample as:
$$
\Pi \left(\hat {\boldsymbol {x}} _ {i} ^ {s}\right) = \left\{k \mid \hat {y} _ {i} ^ {s} = \hat {y} _ {j} ^ {s} \right\}, \tag {8}
$$
where  $\hat{\pmb{x}}_i^s$  denotes the augmented source sample. Then, we provide a hashing consistency learning objective for source domain:
$$
\mathcal {L} _ {C L} ^ {s} = - \mathbb {E} _ {\boldsymbol {x} \in \mathcal {B} ^ {s}} \frac {1}{| \Pi \left(\hat {\boldsymbol {x}} _ {i} ^ {s}\right) |} \sum_ {k \in \Pi \left(\hat {\boldsymbol {x}} _ {i} ^ {s}\right)} \log \frac {\exp \left(\hat {\boldsymbol {b}} _ {i} \cdot \hat {\boldsymbol {b}} _ {k} / \tau\right)}{\sum_ {i = 1} ^ {2 | \mathcal {B} ^ {s} |} \exp \left(\hat {\boldsymbol {b}} _ {i} ^ {s} \cdot \hat {\boldsymbol {b}} _ {k ^ {\prime}} ^ {s} / \tau\right)}, \tag {9}
$$
where  $\hat{\pmb{b}}_i^s = \psi (g^c (F(\hat{\pmb{x}}_i^s)))$  and  $\tau$  is a temperature parameter. Moreover, the hashing consistency learning objective in the target domain is:
$$
\mathcal {L} _ {C L} ^ {t} = - \mathbb {E} _ {\boldsymbol {x} \in \mathcal {B} ^ {t}} \log \frac {\exp (\hat {\boldsymbol {b}} _ {i} \cdot \hat {\boldsymbol {b}} _ {k} / \tau)}{\sum_ {i = 1} ^ {2 | \mathcal {B} ^ {s} |} \exp (\hat {\boldsymbol {b}} _ {i} ^ {s} \cdot \hat {\boldsymbol {b}} _ {k ^ {\prime}} ^ {s} / \tau)}. \tag {10}
$$
Finally, the consistency learning objective is derived by combining both source and target domains:
$$
\mathcal {L} _ {C L} = \mathcal {L} _ {C L} ^ {s} + \mathcal {L} _ {C L} ^ {t}. \tag {11}
$$
The introduction of consistency learning offers two primary benefits for hash code generation: (1) By maximizing the consistency of hash codes between similar images, our IDEA ensures the generation of similarity-preserving hash codes, which is essential for effective search engines. (2) By minimizing the similarity of hash codes derived from dissimilar images, our IDEA encourages hash codes to be uniformly distributed in the Hamming space, which enhances the capacity of each hash bit.
4.5 Invariance under Intervention