SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mixue Xie<sup>1</sup> Shuang Li<sup>1,*</sup> Longhui Yuan<sup>1</sup> Chi Harold Liu<sup>1</sup> Zehui Dai<sup>2</sup>  
<sup>1</sup>Beijing Institute of Technology, China <sup>2</sup>Lazada Search & Monetisation Tech, China {mxxie, shuangli, longhuiyuan}@bit.edu.cn, liuchi02@gmail.com zehui.dzh@alibaba-inc.com
The capability of generalizing to out-of-distribution data is crucial for the deployment of machine learning models in the real world. Existing domain generalization (DG) mainly embarks on offline and discrete scenarios, where multiple source domains are simultaneously accessible and the distribution shift among domains is abrupt and violent. Nevertheless, such setting may not be universally applicable to all real-world applications, as there are cases where the data distribution gradually changes over time due to various factors, e.g., the process of aging. Additionally, as the domain constantly evolves, new domains will continually emerge. Re-training and updating models with both new and previous domains using existing DG methods can be resource-intensive and inefficient. Therefore, in this paper, we present a problem formulation for Continual Domain Generalization over Temporal Drift (CDGTD). CDGTD addresses the challenge of gradually shifting data distributions over time, where domains arrive sequentially and models can only access the data of the current domain. The goal is to generalize to unseen domains that are not too far into the future. To this end, we propose an Evolving Standardization (EvoS) method, which characterizes the evolving pattern of feature distribution and mitigates the distribution shift by standardizing features with generated statistics of corresponding domain. Specifically, inspired by the powerful ability of transformers to model sequence relations, we design a multi-scale attention module (MSAM) to learn the evolving pattern under sliding time windows of different lengths. MSAM can generate statistics of current domain based on the statistics of previous domains and the learned evolving pattern. Experiments on multiple real-world datasets including images and texts validate the efficacy of our EvoS.
In real-world applications, the assumption that training and testing data conform to the same distribution, a prerequisite for the success of contemporary deep learning methods, is seldom valid. A prime example of this can be found in autonomous driving, where a vehicle may traverse environments that switch from daylight to nightfall or from urban to rural. As environmental conditions change, the issue of distribution shift [5, 4, 51, 62] arises. Moreover, directly utilizing a model trained on in-distribution (ID) data in an out-of-distribution (OOD) context often results in catastrophic performance deterioration. Consequently, ensuring that models perform well on OOD data has emerged as a crucial challenge for the widespread deployment of machine learning models in the real world.
To cope with the distribution shift, two dominant paradigms have been systematically explored depending on the availability of target (test) domain. One is domain adaptation (DA), which aims to assist the model learning on an unlabeled or label-scarce target domain by transferring the knowledge
from a related and label-rich source domain [15, 34, 29, 35]. Yet, target data are not always known or available in advance. The other is domain generalization (DG), the goal of which is to learn a model capable of generalizing to any unseen domain by using date from multiple related but distinct source domains [62, 36, 27, 63, 26]. Though the second paradigm DG has attained some encouraging outcomes, its configuration is limited to offline and discrete scenarios where multiple source domains can be accessed simultaneously, and the distribution shift among domains is sudden and severe. For instance, the prevalent benchmark PACS [25] in current DG methods comprises four distinct domain styles - "Art", "Cartoon", "Photo" and "Sketch". Nevertheless, this type of configuration may not be suitable for all real-world applications. There are also some cases that the data distribution gradually evolves over time, the distribution shift arising from which is referred to as temporal drift [56].
On one hand, the real-world scenarios often exhibit underlying evolutionary patterns [59, 44]. For example, annual or monthly weather data can be utilized for weather forecasting [12]. The evolutionary patterns can be exploited to enhance generalization capabilities towards future unseen domains that are not too distant. However, current DG methods often fail to consider these patterns, leading to suboptimal performance. On the other hand, since data distribution is constantly evolving, new domains will continue to emerge. Consequently, it is imperative to efficiently utilize these new domains to further enhance the model's performance for practical applications. For instance, in the context of advertisement recommendation, user browsing data for various products continually surfaces. How can we leverage these newly collected data to enable more accurate advertisement recommendations tailored to each user's preferences in the days to come? One simple approach is to store data from previous domains and retrain the model using both the new and previous domains with the existing DG techniques. However, such way may consume huge training resources due to the accumulation of data and is of low efficiency, especially for scenarios with rapid data accumulation.
In this paper, we formulate the aforementioned problems as Continual Domain Generalization over Temporal Drifts (CDGTD), where the data distribution gradually drifts with the passage of time and the domain arrives sequentially. And the goal of CDGTD is to generalize well on future unseen domains that are not too far under the circumstance that only current domain is available at current times point while data from previous domains are inaccessible. Although some temporal / evolving DG methods [44, 59, 3, 41] have been proposed to handle the temporal drift, most of them [59, 44, 41] works in non-incremental setting, i.e., multiple domains can be accessed simultaneously. Instead, we design an Evolving Standardization (EvoS) approach specialized for CDGTD.
For CDGTD, there are two main challenges: how to characterize the evolving pattern in the incremental setting and how to achieve generalization using the learned evolving pattern. For the former, we draw inspiration from the sequence modeling capability of transformers [31] and design a multi-scale attention module (MSAM) to learn the evolving pattern underlying the feature distribution of domains. Specifically, we store the statistics (i.e., mean and variance of features) of previous domains and use sliding time windows of different lengths over the statistic tokens to obtain multi-scale information. Then multi-scale statistic tokens are fed into MSAM to generate statistics of current domain. MSAM is trained over the whole sequence of domains to learn the evolving pattern. Here, we integrate multiscale information, with the consideration that some evolving patterns may be better characterized at different time intervals, e.g., seasonal climate and daily weather. For the latter challenge, in order to mitigate the temporal drift, each domain is transformed into a common normal distribution by conducting feature standardization with the generated statistics of corresponding domain. Besides, considering that the feature encoder may suffer from catastrophic forgetting and overfitting to current domain, we constrain it to learn a shared feature space among domains via the adversarial learning.
Contributions: 1) We formulate a promising but challenging problem of continual domain generalization over temporal drift (CDGTD), which has seldom been explored, compared with traditional DG. 2) An evolving standardization (EvoS) approach is specially proposed for CDGTD, which can characterize the evolving pattern and further achieve generalization by conducting the feature standardization. 3) Experiments on multiple real-world datasets with different models verify the effectiveness of EvoS and the flexibility to be applied on different models.
Domain Generalization (DG) aims to learn a model that can generalize well to any unseen target domains by leveraging data from multiple source domains. In recent years, a wide range of DG
methods have been proposed [6, 39, 27, 26, 63, 19, 37]. According to the strategies of improving generalization, existing DG methods roughly fall into the three categories. Representation learning based methods [7, 39, 50, 27, 27, 2, 17, 42] aim to learn domain-invariant or domain-shared representations to enable models to generalize to unseen target domains. Data augmentation/generation based methods [53, 61, 63, 57, 32] focus on manipulating inputs to facilitate learning general representations. Differently, other DG methods instead employ general learning strategies like meta-learning [26] and ensemble learning [21] to improve generalization ability. With full access to all source domains, these DG methods have achieved promising performance on unseen domains with abrupt and violent distribution shifts. Unfortunately, without the consideration of the evolutionary pattern of domains over time, existing DG methods are usually less efficient under the scenario of temporal drift.
Continual Learning (CL) is a machine learning paradigm, where models learn continuously from a stream of tasks over time and meanwhile try to retain performances on all seen tasks [22, 1, 24, 9, 46, 47]. Existing CL methods can be roughly categorized into replay-based [46, 10, 52], regularization-based [22, 60, 30] and structure-based [45, 14] methods. In addition, the combination of CL and DA has attracted lots of attention [23, 17, 33, 43, 55]. Yet, combining CL and DG remains underdeveloped. In this work, we propose CDGTD which continually trains the model on sequential domains, but the goal is to generalize well on novel domains in the near future. This distinct objective from CL yields different challenges: the modeling of underlying evolutionary patterns of temporal domains and how to utilize these patterns to mitigate the distribution shifts in forthcoming domains.
Test Time Adaptation (TTA) focuses on adapting the source-pretrained model during testing phase with test data [40, 49, 20, 54, 58]. For example, [49] corrects the activation statistics of batch normalization using test data. T3A [20] uses a back-propagation-free manner to adjust only the weights of the linear classifier during test time. By contrast, CDGTD optimizes the model during training phase with sequential domains and emphasizes on generalization without using test data.
**Evolving / Temporal Domain Generalization** has emerged in recent years, aiming to tackle the problem of generalizing on temporally drifted domains, where the environment evolves dynamically over time [41, 3, 44, 59]. GI [41] proposes a time-sensitive model architecture to capture the time-varying data distribution and the model is supervised with the first-order Taylor expansion of the learned function to advance the generalization in the near feature. DRAIN [3] launches a Bayesian framework to model the concept drift and utilizes a recurrent neural network to dynamically generate network parameters to adapt the evolving pattern of domains. LSSAE [44] incorporates variational inference to explore the evolving patterns of covariate shift and label shift in the latent space. The goal of learning evolutionary patterns from source domains and generalizing to domains in the near future is similar to our EvoS. The main difference is that the model in our EvoS is incrementally trained on sequentially arriving domains, considering the low efficiency of "offline" training with accumulated domains. By contrast, the aforementioned methods [41, 44] require multiple source domains to be simultaneously available. Besides, the Taylor expansion in [41], variational inference in [44] and network parameters generation in [3] make them hard to expand on large models.
3 Evolving Standardization for Continual Domain Generalization over Temporal Drift
3.1 Problem Formulation of CDGTD
Here, we take the  $C$ -class classification problem as an example, where  $X$  and  $Y$  denote the data and label space, respectively. Suppose that  $T$  source (training) domains  $\{\mathcal{D}^1,\mathcal{D}^2,\dots ,\mathcal{D}^T\}$  sequentially arrive, which are sampled from distributions at  $T$  different times points  $\mathfrak{t}_1 < \mathfrak{t}_2 < \dots < \mathfrak{t}_T$ . At time point  $\mathfrak{t}_t$ ,  $t\in \{1,2,\dots ,T\}$ , only the domain  $\mathcal{D}^t = \{\pmb {x}_i^t,y_i^t\}_{i = 1}^{N^t}$  is accessible, where  $\pmb{x}_i^t\in X$ ,  $y_{i}^{t}\in Y$  and  $N^t$  is the number of training samples in  $\mathcal{D}^t$ . Previous and future domains are unavailable. In addition, as previous temporal/evolving DG methods [44, 41, 3], we further assume that the data distribution  $P(X,Y)$  of domains evolves temporally, i.e., the distribution of domains changes along the time following certain patterns. The goal of CDGTD is to enable the model, composed of a feature encoder  $\mathcal{E}:x\to f\in \mathbb{R}^{d_f}$  ( $d_{f}$  is the dimension of features) and a classifier  $\mathcal{C}:f\rightarrow y\in \{0,1,\dots ,C - 1\}$ , to generalize well on  $K$  unseen target (test) domains in the near future:  $\{\mathcal{D}^{T + k}\}_{k = 1}^{K}$ . To this end, two main challenges need to be addressed. One is to characterize the evolving pattern of domains, which we address by designing a multi-scale attention module (MSAM)
Figure 1: Overview of EvoS. Memory pool  $\mathcal{M}$  stores previously generated domain statistics (i.e.,  $\pmb{\mu}^i$  (mean) and  $\pmb{a}^i$  (logarithm of standard deviation) of features,  $1\leq i\leq t - 1$ ). With the  $t$ -th domain  $\mathcal{D}^t$  available, loss  $\mathcal{L}_{stan}^t$  is minimized to train the multi-scale attention module (MSAM) to generate statistics  $\hat{\pmb{\mu}}^t$  and  $\hat{\pmb{a}}^t$  that approach the real ones via leveraging historical statistics in  $\mathcal{M}$ . Besides, we sample features from historical feature distributions as the proxy of previous domains to conduct adversarial training with current domain, encouraging to learn a shared feature space.
inspired from the sequence modeling ability of transformers. The other is the generalization, which is realized by transforming domains into a common normal distribution via the feature standardization.
3.2 EvoS: Evolving Standardization
Single-scale Attention. Inspired by the capability of the transformer to model the relationships among sequences [31], we introduce its attention mechanism to model the evolving pattern among sequential domains. Before presenting our multi-scale attention module (MSAM), we first introduce how single-scale attention works. Without loss of much generality, we assume that the feature distribution of domain  $\mathcal{D}^t$  follows the normal distribution, which is characterized by the mean vector  $\pmb{\mu}^t\in \mathbb{R}^{d_f}$  and the standard deviation vector  $\sigma^t\in \mathbb{R}^{d_f}$ . Now, our goal turns out to be learning the evolving pattern of the feature statistics (i.e.,  $\pmb{\mu}^t$  and  $\pmb{\sigma}^t$ ). In practice, to ensure that  $\pmb{\sigma}^t$  is non-negative, we choose to learn  $\pmb{a}^t$  (the logarithm of  $\pmb{\sigma}^t$ ). In this way,  $\pmb{\sigma}^t = \exp (\pmb{a}^t)$  is always non-negative. Besides, considering the unavailability of previous domains, we store the learned statistics at time point  $t_t$  into a memory pool  $\mathcal{M}$  for future uses once the training procedure finishes on domain  $\mathcal{D}^t$ . Having the feature statistics of previous  $t - 1$  domains ready, we then utilize the multi-head self-attention module  $\mathcal{A}$  to generate the statistics of domain  $\mathcal{D}^t$  at time point  $t_t$ .
Concretely, given statistic tokens  $\{s^i\}_{i = 1}^{t - 1}$  of previous  $t - 1$  domains, where  $\pmb {s}^i = [\pmb {\mu}^i,\pmb {a}^i ]\in \mathbb{R}^{2d_f}$  is the token of concatenated statistics from domain  $\mathcal{D}^i$ , the output of  $\mathcal{A}$  is expressed as
$$
\begin{array}{l} \hat {\boldsymbol {S}} ^ {t} = \mathcal {A} (\boldsymbol {S} ^ {t - 1}) = [ \mathrm {S A} _ {1} (\boldsymbol {S} ^ {t - 1}), \mathrm {S A} _ {2} (\boldsymbol {S} ^ {t - 1}), \dots , \mathrm {S A} _ {n _ {h}} (\boldsymbol {S} ^ {t - 1}) ] \mathbf {W} _ {f c}, \quad \mathbf {W} _ {f c} \in \mathbb {R} ^ {(n _ {h} \cdot d _ {h}) \times 2 d _ {f}}, \\ = [ \hat {\boldsymbol {s}} ^ {1}; \hat {\boldsymbol {s}} ^ {2}; \dots ; \hat {\boldsymbol {s}} ^ {t - 1} ], \\ \end{array}
$$
$$
\boldsymbol {S} ^ {t - 1} = \left[ \boldsymbol {s} ^ {1}; \boldsymbol {s} ^ {2}; \dots ; \boldsymbol {s} ^ {t - 1} \right], \quad \boldsymbol {S} ^ {t - 1} \in \mathbb {R} ^ {(t - 1) \times 2 d _ {f}} \tag {1}
$$
where  $\hat{s}^i$  is the  $i$ -th output statistic token of  $\mathcal{A}$ ,  $n_h$  and  $d_h$  are the number and feature dimension of heads in the multi-head self-attention module and  $\mathbf{W}_{fc}$  is the learnable parameters of  $\mathcal{A}$  to convert feature dimensions.  $\mathrm{SA}_i(\cdot)$  is the self-attention of the  $i$ -th head, which operates as follows:
$$
\begin{array}{l} [ \boldsymbol {S} _ {i, q} ^ {t - 1}, \boldsymbol {S} _ {i, k} ^ {t - 1}, \boldsymbol {S} _ {i, v} ^ {t - 1} ] = \boldsymbol {S} ^ {t - 1} \mathbf {W} _ {q k v} ^ {i}, \quad \mathbf {W} _ {q k v} ^ {i} \in \mathbb {R} ^ {2 d _ {f} \times 3 d _ {h}} \\ \operatorname {S A} _ {i} \left(\boldsymbol {S} ^ {t - 1}\right) = \operatorname {s o f t m a x} \left(\boldsymbol {S} _ {i, q} ^ {t - 1} \boldsymbol {S} _ {i, k} ^ {t - 1} ^ {\top} / \sqrt {d} _ {h}\right) \boldsymbol {S} _ {i, v} ^ {t - 1}, \tag {2} \\ \end{array}
$$
where  $\mathbf{W}_{qkv}^{i}$  is the learnable parameters of the  $i$ -th head and  $\boldsymbol{S}_{i,q}^{t-1}, \boldsymbol{S}_{i,k}^{t-1}, \boldsymbol{S}_{i,v}^{t-1}$  are the query, key and value embeddings of  $\boldsymbol{S}^{t-1}$ . Finally, we use the average of all output statistics tokens of the attention module  $\mathcal{A}$  as the generated statistic token  $\hat{s}^t$  for time points  $\mathfrak{t}_t$ :
$$
[ \hat {\boldsymbol {\mu}} ^ {t}, \hat {\boldsymbol {a}} ^ {t} ] = \hat {\boldsymbol {s}} ^ {t} = a v g (\hat {\boldsymbol {S}} ^ {t}) = \frac {1}{t - 1} \sum_ {i = 1} ^ {t - 1} \hat {\boldsymbol {s}} ^ {i}. \tag {3}
$$
By continuously generating new statistics from historical ones throughout the whole sequence of domains, the attention module  $\mathcal{A}$  is expected to learn the evolving pattern of feature distributions. Note that we directly introduce learnable statistic vectors  $\hat{\pmb{\mu}}^1,\hat{\pmb{a}}^1$  and  $\hat{\pmb{\mu}}^2,\hat{\pmb{a}}^2$  for time points  $\mathfrak{t}_1$  and  $\mathfrak{t}_2$ , respectively, since the tokens of historical statistics are not enough for the attention module  $\mathcal{A}$  to work. That is, the attention module  $\mathcal{A}$  is only used at time points  $\mathfrak{t}_t,t\geq 3$ .
Multi-scale Attention Module (MSAM). The above details how to model the evolving pattern using single-scale attention. However, it is limited to learn the pattern using a sliding time window of length 1 over the statistic tokens. Sometimes, the evolutionary pattern may be better captured by considering longer time intervals. For instance, the data of a season in each time window would be more suitable for capturing the evolving pattern of seasonal climate, rather than data of a single day in each time window. Considering this, we introduce multi-scale attention in Fig. 1, which leverages information from observation windows of different lengths to better model the evolving pattern.
Specifically, for the multi-head self-attention module  $\mathcal{A}_w$  responsible for the time window of length  $w$ , we slide the time window with stride 1 over historical  $t - 1$  statistics tokens and concatenate the statistic tokens in each window to obtain the input  $\ddot{\pmb{S}}_w^{t - 1}$  for the attention module  $\mathcal{A}_w$  at time point  $t_t$ :
$$
\ddot {\boldsymbol {s}} _ {w} ^ {i} = \left[ \boldsymbol {s} ^ {i}, \boldsymbol {s} ^ {i + 1} \dots , \boldsymbol {s} ^ {i + w - 1} \right], \quad \ddot {\boldsymbol {s}} _ {w} ^ {i} \in \mathbb {R} ^ {w \cdot 2 d _ {f}}, \quad (1 \leq i \leq t - w) \wedge (t \geq w + 2)
$$
$$
\ddot {\boldsymbol {S}} _ {w} ^ {t - 1} = \left[ \ddot {\boldsymbol {s}} _ {w} ^ {1}; \ddot {\boldsymbol {s}} _ {w} ^ {2}; \dots ; \ddot {\boldsymbol {s}} _ {w} ^ {t - w} \right], \quad \ddot {\boldsymbol {S}} _ {w} ^ {t - 1} \in \mathbb {R} ^ {(t - w) \times (w \cdot 2 d _ {f})}. \tag {4}
$$
And similarly, we use the average output  $\check{s}_w^{t - w + 1}$  of the attention module  $\mathcal{A}_w$  as the generated statistics for the sliding time window at the next time point, the formulation of which is expressed as
$$
\breve {\boldsymbol {S}} _ {w} ^ {t} = \mathcal {A} _ {w} (\ddot {\boldsymbol {S}} _ {w} ^ {t - 1}) = [ \breve {\boldsymbol {s}} _ {w} ^ {1}; \breve {\boldsymbol {s}} _ {w} ^ {2}; \dots ; \breve {\boldsymbol {s}} _ {w} ^ {t - w} ], \quad \breve {\boldsymbol {S}} _ {w} ^ {t} \in \mathbb {R} ^ {(t - w) \times (w \cdot 2 d _ {f})}
$$
$$
\check {\boldsymbol {s}} _ {w} ^ {t - w + 1} = a v g \left(\check {\boldsymbol {S}} _ {w} ^ {t}\right) = \frac {1}{t - w} \sum_ {i = 1} ^ {t - w} \check {\boldsymbol {s}} _ {w} ^ {i}, \quad \check {\boldsymbol {s}} _ {w} ^ {t - w + 1} \in \mathbb {R} ^ {w \cdot 2 d _ {f}}. \tag {5}
$$
Note that in MSAM, the attention module  $\mathcal{A}_w$  is to predict the statistics in the next sliding time window of length  $w$ , instead of one statistic token as in the single-scale attention. This may encourage the attention module to also capture the domain relationships within the window.
Then, we split  $\check{s}_w^{t - w + 1}$  into  $w$  parts:  $[\hat{s}_w^{t - w + 1},\dots ,\hat{s}_w^{t - 1},\hat{s}_w^t ] = \check{s}_w^{t - w + 1}$ , where  $\hat{s}_w^j\in \mathbb{R}^{2d_f}$  can be regarded as the predicted statistic token for time point  $\mathfrak{t}_j$  using the attention module  $\mathcal{A}_w,j = t - w + 1,\dots ,t - 1,t$ . Finally, the generated statistic token  $\hat{s}^t$  for time point  $\mathfrak{t}_t$  using MSAM is denoted as the average of predicted statistic tokens for time point  $\mathfrak{t}_t$  at different time window lengths:
$$
[ \hat {\boldsymbol {\mu}} ^ {t}, \hat {\boldsymbol {a}} ^ {t} ] = \hat {\boldsymbol {s}} ^ {t} = \frac {1}{W} \sum_ {w = 1} ^ {W} \hat {\boldsymbol {s}} _ {w} ^ {t}, \tag {6}
$$
where  $W$  is the maximum length of the sliding time window. Generally, MSAM integrates evolving patterns learned at different scales, contributing better estimation of future feature distributions.
Feature Standardization. For CDGTD, the second main challenge is generalization. To this end, we leverage the generated feature statistics by MSAM to transform the distribution of corresponding domain into a standard normal distribution, by which the temporal drift is mitigated. Concretely, for feature  $\pmb{f}_i^t = \mathcal{E}(\pmb{x}_i^t)$ , its standardized feature  $\pmb{z}_i^t$  via the feature standardization is formulated as
$$
\boldsymbol {z} _ {i} ^ {t} = \frac {\boldsymbol {f} _ {i} ^ {t} - \hat {\boldsymbol {\mu}} ^ {t}}{\hat {\boldsymbol {\sigma}} ^ {t}} = \frac {\boldsymbol {f} _ {i} ^ {t} - \hat {\boldsymbol {\mu}} ^ {t}}{\exp (\hat {\boldsymbol {a}} ^ {t})}. \tag {7}
$$
Another benefit of feature standardization is that it allows classifier to be trained on a common feature distribution, thus avoiding the problem of overfitting to current domain and catastrophic forgetting.
3.3 Model Training
This section outlines the four losses  $\mathcal{L}_{ce}^{t}$ ,  $\mathcal{L}_{stan}^{t}$ ,  $\mathcal{L}_{con}^{t}$  and  $\mathcal{L}_{adv}^{t}$  involved in the training stage at time point  $t_{t}$ . The first one is the essential loss to supervise the model learning for specific tasks, e.g., the following cross-entropy loss  $\mathcal{L}_{ce}^{t}$  for classification tasks in this paper.