SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhiqing Xiao $^{13}$ , Haobo Wang $^{23}$ , Ying Jin $^{4}$ , Lei Feng $^{5}$ , Gang Chen $^{13}$ , Fei Huang $^{6}$ , Junbo Zhao $^{13*}$
<sup>1</sup> College of Computer Science and Technology, Zhejiang University
$^{2}$  School of Software Technology, Zhejiang University
<sup>3</sup> Key Lab of Intelligent Computing based Big Data of Zhejiang Province, Zhejiang University
<sup>4</sup> CUHK-SenseTime Joint Lab, The Chinese University of Hong Kong
<sup>5</sup> School of Computer Science and Engineering, Nanyang Technological University
6 Alibaba Group
{zhiqing.xiao, wanghaobo, cg, j.zhao}@zju.edu.cn,
{sherryying003,lfengqaq,feirhuang}@gmail.com
Unsupervised domain adaptation (UDA) is a pivotal form in machine learning to extend the in-domain model to the distinctive target domains where the data distributions differ. Most prior works focus on capturing the inter-domain transferability but largely overlook rich intra-domain structures, which empirically results in even worse discriminability. In this work, we introduce a novel graph Spectral Alignment (SPA) framework to tackle the tradeoff. The core of our method is briefly condensed as follows: (i)-by casting the DA problem to graph primitives, SPA composes a coarse graph alignment mechanism with a novel spectral regularizer towards aligning the domain graphs in eigenspaces; (ii)-we further develop a fine-grained message propagation module — upon a novel neighbor-aware self-training mechanism — in order for enhanced discriminability in the target domain. On standardized benchmarks, the extensive experiments of SPA demonstrate that its performance has surpassed the existing cutting-edge DA methods. Coupled with dense model analysis, we conclude that our approach indeed possesses superior efficacy, robustness, discriminability, and transferability. Code and data are available at: https://github.com/CrownX/SPA.
Domain adaptation (DA) problem is a widely studied area in computer vision field [20, 19, 73, 65, 40], which aims to transfer knowledge from label-rich source domains to label-scare target domains where dataset shift [32] or domain shift [72] exists. The existing unsupervised domain adaptation (UDA) methods usually explore the idea of learning domain-invariant feature representations based on the theoretical analysis [3]. These methods can be generally categorized several types, i.e., moment matching methods [44, 51, 70], and adversarial learning methods [18, 66, 19, 52].
The most essential challenge of domain adaptation is how to find a suitable utilization of intra-domain information and inter-domain information to properly align target samples. More specifically, it is a trade-off between discriminating data samples of different categories within a domain to the greatest extent possible, and learning transferable features across domains with the existence of domain shift. To achieve this, adversarial learning methods implicitly mitigate the domain shift by driving the feature extractor to extract indistinguishable features and fool the domain classifier.
The adversarial DA methods have been increasingly developed and followed by a series of works [14, 45, 8]. However, it is very unexpected that the remarkable transferability of DANN [18] is enhanced at the expense of worse discriminability [10, 38].
To mitigate this problem, there is a promising line of studies that explore graph-based UDA algorithms [9, 87, 54]. The core idea is to establish correlation graphs within domains and leverage the rich topological information to connect inter-domain samples and decrease their distances. In this way, self-correlation graphs with intra-domain information are constructed, which exhibits the homophily property whereby nearby samples tend to receive similar predictions [55]. There comes the problem that is how to transfer the inter-domain information with these rich topological information. Graph matching is a direct solution to inter-domain alignment problems. Explicit graph matching methods are usually designed to find sample-to-sample mapping relations, requiring multiple matching stages for nodes and edges respectively [9], or complicated structure mapping with an attention matrix [54]. Despite the promise, we find that such a point-wise matching strategy can be restrictive and inflexible. In effect, UDA does not require an exact mapping plan from one node to another one in different domains. Its expectation is to align the entire feature space such that label information of source domains can be transferred and utilized in target domains.
In this work, we introduce a novel graph spectral alignment perspective for UDA, hierarchically solving the aforementioned problem. In a nutshell, our method encapsulates a coarse graph alignment module and a fine-grained message propagation module, jointly balancing inter-domain transferability and intra-domain discriminability. Core to our method, we propose a novel spectral regularizer that projects domain graphs into eigenspaces and aligns them based on their eigenvalues. This gives rise to coarse-grained topological structures transfer across domains but in a more intrinsic way than restrictive point-wise matching. Thereafter, we perform the fine-grained message passing in the target domain via a neighbor-aware self-training mechanism. By then, our algorithm is able to refine the transferred topological structure to produce a discriminative domain classifier.
We conduct extensive evaluations on several benchmark datasets including DomainNet, OfficeHome, Office31, and VisDA2017. The experimental results show that our method consistently outperforms existing state-of-the-art domain adaptation methods, improving the accuracy on  $8.6\%$  on original DomainNet dataset and about  $2.6\%$  on OfficeHome dataset. Furthermore, the comprehensive model analysis demonstrates the superiority of our method in efficacy, robustness, discriminability and transferability.
Problem Description. Given source domain data  $\mathcal{D}_s = \{(x_i^s,y_i^s)\}_{i = 1}^{N_s}$  of  $N_{s}$  labeled samples associated with  $C_s$  categories from  $\mathcal{X}_s\times \mathcal{Y}_s$  and target domain data  $\mathcal{D}_t = \{x_i^t\}_{i = 1}^{N_t}$  of  $N_{t}$  unlabeled samples associated with  $C_t$  categories from  $\mathcal{X}_t$ . We assume that the domains share the same feature and label space but follow different marginal data distributions, following the Covariate Shift [67]; that is,  $P(\mathcal{X}_s)\neq P(\mathcal{X}_t)$  but  $P(\mathcal{Y}_s\mid \mathcal{X}_s) = P(\mathcal{Y}_t\mid \mathcal{X}_t)$ . Domain adaptation just occurs when the underlying distributions corresponding to the source and target domains in the shared label space are different but similar enough to make sense the transfer [15]. The goal of unsupervised domain adaptation is to predict the label  $\{y_i^t\}_{i = 1}^{N_t}$  in the target domain, where  $y_{i}^{t}\in \mathcal{V}_{t}$ , and the source task is  $\mathcal{X}_s\to \mathcal{Y}_s$  assumed to be the same with the target task  $\mathcal{X}_t\rightarrow \mathcal{Y}_t$ .
Adversarial Domain Adaptation. The family of adversarial domain adaptation methods, e.g., Domain Adversarial Neural Network (DANN) [18] have become significantly influential in domain adaptation. The fundamental idea behind is to learn transferable features that explicitly reduce the domain shift. Similar to standard supervised classification methods, these approaches include a feature extractor  $F(\cdot)$  and a category classifier  $C(\cdot)$ . Additionally, a domain classifier  $D(\cdot)$  is trained to distinguish the source domain from the target domain and meanwhile the feature extractor  $F(\cdot)$  is trained to confuse the domain classifier and learn domain-invariant features. The supervised classification loss  $\mathcal{L}_{cls}$  and domain adversarial loss  $\mathcal{L}_{adv}$  are presented as bellow:
$$
\begin{array}{l} \mathcal {L} _ {c l s} = \mathbb {E} _ {\left(x _ {i} ^ {s}, y _ {i} ^ {s}\right) \sim \mathcal {D} _ {\mathcal {S}}} \mathcal {L} _ {c e} \left(C \left(F \left(x _ {i} ^ {s}\right)\right), y _ {i} ^ {s}\right) \\ \mathcal {L} _ {a d v} = \mathbb {E} _ {F \left(x _ {i} ^ {s}\right) \sim \tilde {\mathcal {D}} _ {s}} \log \left[ D \left(F \left(x _ {i} ^ {s}\right)\right) \right] \tag {1} \\ + \mathbb {E} _ {F \left(x _ {i} ^ {t}\right) \sim \tilde {\mathcal {D}} _ {t}} \log \left[ 1 - D \left(F \left(x _ {i} ^ {t}\right)\right) \right] \\ \end{array}
$$
where  $\tilde{\mathcal{D}}_s$  and  $\tilde{\mathcal{D}}_t$  denote the induced feature distributions of  $\mathcal{D}_s$  and  $\mathcal{D}_t$  respectively, and  $\mathcal{L}_{ce}(\cdot ,\cdot)$  is the cross-entropy loss function.
In this section, we will give a specific introduction to our approach. The overall pipeline is shown in Figure 1. Our method is able to effectively utilize both intra-domain and inter-domain relations simultaneously for domain adaptation tasks. Specifically, based on our constructed dynamic graphs in Section 3.1, we propose a novel framework that utilizes graph spectra to align inter-domain relations in Section 3.2, and leverages intra-domain relations via neighbor-aware propagation mechanism in Section 3.3. Our approach enables us to effectively capture the underlying domain distributions while ensuring that the learned features are transferable across domains.
Figure 1: The overall architecture. The final objective integrates supervised classification loss  $\mathcal{L}_{cls}$ , domain adversarial loss  $\mathcal{L}_{adv}$ , neighbor-aware propagation loss  $\mathcal{L}_{nap}$ , and graph spectral alignment loss  $\mathcal{L}_{spa}$ .
3.1 Dynamic Graph Construction
Images and text data inherently contain rich sequential or spatial structures that can be effectively represented using graphs. By constructing graphs based on the data, both intra-domain relations and inter-domain relations can be exploited. For instance, semantic and spatial relations among detected objects in an image [43], cross-modal relations among images and sentences [11], cross-domain relations among images and texts [9, 27] have been successfully modeled using graph-based representations. In this paper, leveraging self-correlation graphs enables us to model the relations between different samples within domain and capture the underlying data distributions.
Our self-correlation graphs are constructed on source features and target features respectively. A feature extractor  $F(\cdot)$  can be designed to learn source features  $f_{s}$  and target features  $f_{t}$  from source domain samples  $\mathbf{x}_s$  and target domain samples  $\mathbf{x}_t$  respectively, i.e.,  $f_{s} = F(x_{s})$  and  $f_{t} = F(x_{t})$ . Given the extracted features  $f_{s}$ , we aim to construct a undirected and weighted graph  $\mathcal{G}_s = (\mathcal{V}_s,\mathcal{E}_s)$ . Each vertex  $v_{i}\in \mathcal{V}_{s}$  is represented by a feature vector  $f_{i}^{s}$ . Each weighted edge  $e_{i,j}\in \mathcal{E}_s$  can be formulated as a relation between a pair of entities  $\delta (f_i^s,f_j^s)$ , where  $\delta (\cdot)$  denotes a metric function. With the extracted features  $f_{t}$ , another graph  $\mathcal{G}_s = (\mathcal{V}_t,\mathcal{E}_t)$  can be constructed in the same way. Note that both  $f_{s}$  and  $f_{t}$  keep evolving along with the update of parameter  $\theta$  during the training process. Their adjacency matrices are denoted as  $\mathbf{A}_s$  and  $\mathbf{A}_t$  respectively. As is well-known, the adjacency matrix of a graph contains all of its topological information. By representing these graphs in domain adaptation tasks, we can directly obtain the intra-domain relations and regard inter-domain alignment as a graph matching problem [9].
3.2 Graph Spectral Alignment
In this section, we will introduce how to align inter-domain relations for source domain graphs and target domain graphs. In domain adaptation scenarios where domain shift exits, if we directly construct a graph with source and target domain features together, we can only obtain a graph beyond the homophily assumption. In this way, a necessary domain alignment comes out.
As stated in Section 3.1, based on our correlation graphs, the inter-domain alignment can be regard as a graph matching problem. Explicit graph matching methods aim to find a one-to-one correspondence between nodes or edges of two graphs and typically involve solving a combinatorial optimization problem for node matching and edge matching respectively [9]. Nevertheless, our goal is often to align the distributions of the source and target domains and learn domain-invariant features, instead of requiring such an complicated matching approach. Therefore, we prefer a implicit graph alignment method, avoiding multiple stages of explicit graph matching.
The distance between the spectra of graphs is able to measure how far apart the spectrum of a graph with  $n$  vertices can be from the spectrum of any other graph with  $n$  vertices [1, 69, 22], leading to a simplification of measuring the discrepancy of graphs. Inspired by this, we give the definitions of graph laplacians and spectral distances:
Definition 1. (GRAPH LAPLACIANS [77]). Let  $\mathcal{G} = (\mathcal{V},\mathcal{E})$  be a finite graph with vertices  $\mathcal{V}$  and weighted edges  $\mathcal{E}$ . Let  $\phi :\mathcal{V}\to \mathcal{R}$  be a function of the vertices taking values in a ring and  $\gamma :\mathcal{E}\rightarrow \mathcal{R}$  be a weighting function of weighed edges. Then, the graph Laplacian  $\Delta$  acting on  $\phi$  and  $\gamma$  is defined by
$$
(\Delta_ {\gamma} \phi) (v) = \sum_ {w: d (w, v) = 1} \gamma_ {w v} [ \phi (v) - \phi (w) ]
$$
where  $d(w, v)$  is the graph distance between vertices  $w$  and  $v$ , and  $\gamma_{wv}$  is the weight value on the edge  $wv \in \mathcal{E}$ .
Definition 2. (SPECTRAL DISTANCES). Given two simple and nonisomorphic graphs  $\mathcal{G}_s$  and  $\mathcal{G}_t$  on  $n$  vertices with the spectra of Laplacians  $\Lambda_s = \{\lambda_i^s\}_{i=1}^n$  with  $\lambda_1^s \geq \lambda_2^s \geq \dots \geq \lambda_n^s$  and  $\Lambda_t = \{\lambda_i^t\}_{i=1}^n$  with  $\lambda_1^t \geq \lambda_2^t \geq \dots \geq \lambda_n^t$  respectively. Define the spectral distance between  $\mathcal{G}_s$  and  $\mathcal{G}_t$  as
$$
\sigma \left(\mathcal {G} _ {s}, \mathcal {G} _ {t}\right) = \left\| \Lambda_ {s} - \Lambda_ {t} \right\| _ {p}, \quad p \geq 1
$$
For a simple undirected graph with a finite number of vertices and edges, the definition 1 is just identical to the Laplacian matrix. With the adjacency matrix  $\mathbf{A}_s$  of source domain graph  $\mathcal{G}_s$ , we can obtain its Laplacian matrix  $\mathbf{L}_s$  and its Laplacian eigenvalues  $\Lambda_s$ . Similarly, we yield Laplacian matrix  $\mathbf{L}_t$  and Laplacian eigenvalues  $\Lambda_t$ . Following the definition Def.2, we can calculate the spectral distances between source domain graph  $\mathcal{G}_s$  and target domain  $\mathcal{G}_t$ , and thus the graph spectral penalty is defined as below:
$$
\mathcal {L} _ {g s a} = \sigma \left(\mathcal {G} _ {s}, \mathcal {G} _ {t}\right) \tag {2}
$$
This spectral penalty measures the discrepancy of two graphs on spectrum space. Minimizing this penalty decreases the distance of source domain graphs and target domain graphs. It can also be regarded as a regularizer and easy to combine with existing domain adaptation methods.
More Insights. Graph Laplacian filters are a type of signal processing filter to apply a smoothing operation to the signal on a graph by taking advantage of the local neighborhood structure of the graph represented by the Laplacian matrix [6]. A graph filter can be denoted as  $(f*g)_{\mathcal{G}} = Ug_{\Lambda}U^{T}f$  where  $f$  is a signal on graph  $\mathcal{G}$ , and  $\Lambda$  and  $U$  is the eigenvalues and eigenvectors of Laplacian matrix  $L = U\Lambda U^T$ . This filter is also the foundation of classic graph neural networks [7, 16, 36] and usually infers the labels of unlabeled nodes with the information from the labeled ones in a graph. Similar to the hypothesis in [91], source features and target features will be aligned into the same eigenspace along with the learning process and finally only differs slightly in the eigenvalues  $g_{\Lambda}$ .
3.3 Neighbor-aware Propagation Mechanism
In this section, we will introduce how to exploit intra-domain relations within target domain graphs. The well-trained source domain naturally forms tight clusters in the latent space. After aligning via the aforementioned graph spectral penalty, the rich topological information is coarsely transferred to the target domain. To perform the fine-grained intra-domain alignment, we take a further step by encouraging message propagation within the target domain graph.
Intuitively, we adopt the weighted  $k$ -Nearest-Neighbor (KNN) classification algorithm [83] to generate pseudo-label for target domain graph. We focus on unlabelled target samples  $\{x_i^t\}_{i=1}^{N_t}$  and thus omit the domain subscript  $t$  for clarity. Let  $p_i = C(F(x_i))$  denotes the  $C_t$ -dimensional prediction of unlabelled target domain sample  $x_i$  and  $p_i^m$  denotes its mapped probability stored in the memory
bank. A vote is obtained from the top  $k$  nearest labelled samples for each unlabelled sample  $x_{i} \in \mathbf{x}$ , which is denoted as  $\mathcal{N}_i$ . The vote of each neighbor  $j \in \mathcal{N}_i$  is weighted by the corresponding predicted probabilities  $p_{j,c}$  that data sample  $x_{j}$  will be identified as category  $c$ . Therefore, the voted probability  $p_{i,c}$  of data sample  $x_{i}$  corresponding to class  $c$  can be defined as  $q_{i,c} = \sum_{j \neq i, j \in \mathcal{N}_i} p_{j,c}^m$  and we yield the normalized probability  $\hat{q}_{i,c} = q_{i,c} / \sum_{m=1}^{C_t} q_{i,m}$  and the pseudo-label  $\hat{y}$  for data sample  $x_{i}$  can be calculated as  $\hat{y}_i = \arg \max_c \hat{q}_{i,c}$ . Considering different neighborhoods  $\mathcal{N}_i$  lie in different local density, we should expect a larger weight for the target data in higher local density [48]. Obviously, a larger  $\hat{q}_{i,c}$  means the data sample  $x_{i}$  lies in a neighborhood of higher density. In this way, we directly utilize the category-normalized probability  $\hat{q}_{i,c}$  as the confidence value for each pseudo-label and thus the weighted cross-entropy loss based on the pseudo-labels is formulated as:
$$
\mathcal {L} _ {n a p} = - \alpha \cdot \frac {1}{N _ {t}} \sum_ {i = 1} ^ {N _ {t}} \hat {q} _ {i, \hat {y} _ {i}} \log p _ {i, \hat {y} _ {i}} \tag {3}
$$
where  $\alpha$  is the coefficient term properly designed to grow along with iterations to mitigate the noises in the pseudo-labels at early iterations and avoid the error accumulation [39]. Note that our neighbor-aware propagation loss only depends on unlabeled target data samples, just following the classic self-training method [31].
Memory Bank. We design the memory bank to store prediction probabilities and their associated feature vectors mapped by their target data indices. Before storing them, we firstly apply the sharpening technique to fix the ambiguity in the predictions of target domain data [5, 24]:
$$
\tilde {p} _ {j, c} = p _ {j, c} ^ {- \tau} / \sum_ {x = 1} ^ {C _ {t}} p _ {j, x} ^ {- \tau} \tag {4}
$$
where  $\tau$  is the temperature to scale the prediction probabilities. As  $\tau \rightarrow 0$ , the probability will collapse to a point mass [39]. Then, we utilize L2-norm to normalize feature vectors  $f_{i}$ . Finally, we store the sharpened prediction  $\tilde{p}_i$  and its associated normalized feature  $\| f_i\|$  to the memory bank via  $\beta$ -exponential moving averaging (EMA) strategy, updating them for each iteration. The predictions stored in the memory bank  $p_i^m$  are utilized to approximate real-time probability for pseudo-label generation, and the feature vectors  $f_i^m$  stored in the memory bank are employed to construct graphs before neighbor-aware propagation.
The Final Objective. At the end of our approach, let us integrate all of these losses together, i.e., supervised classification loss  $\mathcal{L}_{cls}$  and domain adversarial loss  $\mathcal{L}_{adv}$  described in Eq.1, the loss of neighbor-aware propagation mechanism  $\mathcal{L}_{nap}$  in Eq.3, and the loss of graph spectral alignment  $\mathcal{L}_{gsa}$  in Eq.2. Finally, we can obtain the final objective as follows:
$$
\mathcal {L} _ {\text {t o t a l}} = \mathcal {L} _ {\text {c l s}} + \mathcal {L} _ {\text {a d v}} + \mathcal {L} _ {\text {g s a}} + \mathcal {L} _ {\text {n a p}} \tag {5}
$$
For the sake of simplicity, we leave out the loss scale for  $\mathcal{L}_{adv}$  and  $\mathcal{L}_{gsa}$  here. Just as the coefficient term  $\alpha$  in  $\mathcal{L}_{nap}$ , these two losses are also adjusted with specific scales in the implementation. For more detailed implementation specifics, please refer to our source code. Note that we only describe the problem of unsupervised domain adaptation here our approach can easily to extend to semi-supervised domain adaptation scenario. Concerning the labeled data, we employ the standard cross-entropy loss with label-smoothing regularization [71] in our implementation. We also give different trials of similarity metrics and graph laplacians. More concrete details are in the following section.
In this section, we present our main empirical results to show the effectiveness of our method. To evaluate the effectiveness of our architecture, we conduct comprehensive experiments under unsupervised domain adaptation, semi-supervised domain adaptation settings. The results of other compared methods are directly reported from the original papers. More experiments can be found in the Appendix.
Datasets. We conduct experiments on 4 benchmark datasets: 1) Office31 [63] is a widely-used benchmark for visual DA. It contains 4,652 images of 31 office environment categories from three
domains: Amazon (A), DSLR (D), and Webcam (W), which correspond to online website, digital SLR camera and web camera images respectively. 2) OfficeHome [76] is a challenging dataset that consists of images of everyday objects from four different domains: Artistic (A), Clipart (C), Product (P), and Real-World (R). Each domain contains 65 object categories in office and home environments, amounting to 15,500 images around. Following the typical settings [10], we evaluate methods on one-source to one-target domain adaptation scenario, resulting in 12 adaptation cases in total. 3) VisDA2017 [58] is a large-scale benchmark that attempts to bridge the significant synthetic-to-real domain gap with over 280,000 images across 12 categories. The source domain has 152,397 synthetic images generated by rendering from 3D models. The target domain has 55,388 real object images collected from Microsoft COCO [49]. Following the typical settings [56], we evaluate methods on synthetic-to-real task and present test accuracy for each category. 4) DomainNet [57] is a large-scale dataset containing about 600,000 images across 345 categories, which span 6 domains with large domain gap: Clipart (C), Infograph (I), Painting (P), Quickdraw (Q), Real (R), and Sketch (S). Following the settings in [28], we compare various methods for 12 tasks among C, P, R, S domains on the original DomainNet dataset.
Implementation details. We use PyTorch and tlib toolbox [28] to implement our method and fine-tune ResNet pre-trained on ImageNet [25, 26]. Following the standard protocols for unsupervised domain adaptation in previous methods [48, 56], we use the same backbone networks for fair comparisons. For Office31 and OfficeHome dataset, we use ResNet-50 as the backbone network. For VisDA2017 and DomainNet dataset, we use ResNet-101 as the backbone network. We adopt mini-batch stochastic gradient descent (SGD) with a momentum of 0.9, a weight decay of 0.005, and an initial learning rate of 0.01, following the same learning rate schedule in [48].
4.2 Result Comparisons