AdaRNN: Adaptive Learning and Forecasting for Time Series∗
Yuntao Du1 , Jindong Wang2 , Wenjie Feng3 , Sinno Pan4 , Tao Qin2 , Renjun Xu5 , Chongjun Wang1
1 Nanjing University, Nanjing, China

2 Microsoft Research Asia, Beijing, China

3 Institute of Data Science, National University of Singapore 4 Nanyang Technological University 5 Zhejiang University

dz1833005@smail.nju.edu.cn,jindong.wang@microsoft.com
A

arXiv:2108.04443v2 [cs.LG] 11 Aug 2021

ABSTRACT
Though time series forecasting has a wide range of real-world applications, it is a very challenging task. This is because the statisticalproperties of a time series can vary with time, causing the distribution to change temporally, which is known as the distribution
shift problem in the machine learning community. By far, it still
remains unexplored to model time series from the distribution-shift
perspective. In this paper, we formulate the Temporal Covariate
Shift (TCS) problem for the time series forecasting. We propose
Adaptive RNNs (AdaRNN) to tackle the TCS problem. AdaRNN issequentially composed of two modules. The first module is referred
to as Temporal Distribution Characterization, which aims to bettercharacterize the distribution information in a time series. The second module is termed as Temporal Distribution Matching, whichaims to reduce the distribution mismatch in the time series to learn
an RNN-based adaptive time series prediction model. AdaRNN is a
general framework with flexible distribution distances integrated.
Experiments on human activity recognition, air quality prediction,
household power consumption, and financial analysis show that
AdaRNN outperforms some state-of-the-art methods by 2.6% interms of accuracy on classification tasks and by 9.0% in terms ofthe mean squared error on regression tasks. We also show thatthe temporal distribution matching module can be extended to theTransformer architecture to further boost its performance.

CCS CONCEPTS• Computing methodologies → Transfer learning; Artificial
intelligence.

KEYWORDS
Time series, multi-task learning, transfer learning
ACM Reference Format:
Yuntao Du1 , Jindong Wang2 , Wenjie Feng3 , Sinno Pan4 , Tao Qin2 , Renjun
Xu5 , Chongjun Wang1 . 2021. AdaRNN: Adaptive Learning and Forecastingfor Time Series. In Proceedings of the 30th ACM Int’l Conf. on Information
and Knowledge Management (CIKM ’21), November 1–5, 2021, Virtual Event,Australia. ACM, New York, NY, USA, 11 pages. https://doi.org/10.1145/
XXXXXX.XXXXXX∗ Corresponding author: J. Wang and W. Feng. This work is done when Y. Du was anintern at MSRA.

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.
CIKM ’21, November 1–5, 2021, Virtual Event, Australia.
© 2021 Association for Computing Machinery.
ACM ISBN 978-1-4503-8446-9/21/11. . . $15.00
https://doi.org/10.1145/XXXXXX.XXXXXX

C

B

Unseen test𝑡

Raw data

𝑷𝑨

𝑷𝑩

?
𝑷𝑪

Probability
distribution

𝑷𝑻𝒆𝒔𝒕

Temporal Covariate Shift: 𝑷𝑨 ≠ 𝑷𝑩 ≠ 𝑷𝑪 ≠ 𝑷𝑻𝒆𝒔𝒕
Figure 1: The temporal covariate shift problem in nonstationary time series. The raw time series data (𝑥) is multivariate in reality. At time intervals 𝐴, 𝐵, 𝐶 and the unseen
test data, the distributions are different: 𝑃𝐴 (𝑥) ≠ 𝑃𝐵 (𝑥) ≠𝑃𝐶 (𝑥) ≠ 𝑃𝑇 𝑒𝑠𝑡 (𝑥). With the distribution changing over time,
how to build an accurate and adaptive model?

1

INTRODUCTION

Time series (TS) data occur naturally in countless domains including
financial analysis [55], medical analysis [29], weather condition
prediction [46], and renewable energy production [5]. Forecastingis one of the most sought-after tasks on analyzing time series data(arguably the most difficult one as well) due to its importance inindustrial, society and scientific applications. For instance, given
the historical air quality data of a city for the last five days, how to
predict the air quality in the future more accurately?
In real applications, it is natural that the statistical properties
of TS are changing over time, i.e., the non-stationary TS. Overthe years, various research efforts have been made for building
reliable and accurate models for the non-stationary TS. Traditional approaches such as hidden Markov models (HMMs) [43],dynamic Bayesian networks [35], Kalman filters [9], and otherstatistical models (e.g. ARIMA [19]), have made great progress. Recently, better performance is achieved by the recurrent neural networks (RNNs) [37, 46]. RNNs make no assumptions on the temporal
structure and can find highly non-linear and complex dependence
relation in TS.
The non-stationary property of time series implies the data distribution changes over time. Given the example in Figure 1, data
distributions 𝑃 (𝑥) vary for different intervals 𝐴, 𝐵, and 𝐶 where 𝑥, 𝑦
are samples and predictions respectively; Especially for the test data
which is unseen during training, its distribution is also different
from the training data and makes the prediction more exacerbated.
The conditional distribution 𝑃 (𝑦|𝑥), however, is usually considered
to be unchanged for this scenario, which is reasonable in many
real applications. For instance, in stock prediction, it is naturalthat the market is fluctuating which changes the financial factors

CIKM ’21, November 1–5, 2021, Virtual Event, Australia.

(𝑃 (𝑥)), while the economic laws remain unchanged (𝑃 (𝑦|𝑥)). It is
an inevitable result that the afore-mentioned methods have inferior
performance and poor generalization [22] since the distribution
shift issue violates their basic I.I.D assumption.
Unfortunately, it remains unexplored to model the time series
from the distribution perspective. The main challenges of the problem lies in two aspects. First, how to characterize the distribution
in the data to maximally harness the common knowledge in thesevaried distributions? Second, how to invent an RNN-based distribution matching algorithm to maximally reduce their distribution
divergence while capturing the temporal dependency?
In this paper, we formally define the Temporal Covariate Shift(TCS) in Figure 1, which is a more practical and challenging setting to model time series data. Based on our analysis on TCS, we
propose AdaRNN, a novel framework to learn an accurate andadaptive prediction model. AdaRNN is composed of two modules.
Firstly, to better characterize the distribution information in TS,
we propose a temporal distribution characterization (TDC) algorithm to split the training data into 𝐾 most diverse periods that arewith large distribution gap inspired by the principle of maximum
entropy. After that, we propose a temporal distribution matching(TDM) algorithm to dynamically reduce distribution divergence
using a RNN-based model. Experiments on activity recognition, air
quality prediction, household power consumption and stock priceprediction show that our AdaRNN outperforms the state-of-the-art
baselines by 2.6% in terms of an accuracy on classification tasks
and by 9.0% in terms of RMSE on regression tasks. AdaRNN isagnostic to both RNN structures (i.e., RNNs, LSTMs, and GRUs) anddistribution matching distances (e.g., cosine distance, MMD [3], or
adversarial discrepancy [12]). AdaRNN can also be extended to theTransformer architecture to further boost its performance.
To sum up, our main contributions are as follows,• Novel problem: For the first time, we propose to model
the time series from the distribution perspective, then we
postulate and formulate the Temporal Covariate Shift (TCS)
problem in non-stationary time series, which is more realistic
and challenging.
• General framework: To solve TCS, we propose a generalframework AdaRNN that learns an accurate and adaptivemodel by proposing the temporal distribution characterization and Temporal Distribution matching algorithms.
• Effectiveness: We conduct extensive experiments on human activity recognition, air quality prediction, household
power consumption, and stock price prediction datasets. Results show that AdaRNN outperforms the state-of-the-art
baselines on both classification and regression tasks.

2 RELATED WORK
2.1 Time series analysis
Traditional methods for time series classification or regression
include distance-based [13, 32], feature-based [38], and ensemble
methods [27]. The distance-based methods measure the distanceand similarity of (segments of) raw series data with some metric,
like the Euclidean distance or Dynamic time wrapping (DTW) [20].
The feature-based methods capture the global/local patterns in thetime series rely on a set of manually extracted or learned features.

Du, et al.

The ensemble methods combine multiple weak classifiers to boost
the performance of the models.
However, involving heavy crafting on data pre-processing andlabor-intensive feature engineering makes those methods struggling to cope with large-scale data and the performance is also
limited for more complex patterns. Recurrent Neural Networks(RNNs), such as Gated Recurrent Unit (GRU) and Long Short Term
Memory (LSTM) have been popular due to their ability to extract
high quality features automatically and handle the long term dependencies. These deep methods tackle TS classification or forecastingby leveraging attention mechanisms [23, 33] or tensor factorization [40] for capturing shared information among series. Another
trend is to model uncertainty by combining deep learning andState space models [37]. Furthermore, some methods [24, 37, 46]
adopt seq2seq models for multi-step predictions. Unlike those methods based on statistical views, our AdaRNN models TS from theperspective of the distributions. Recently, the Transformers structure [45] is proposed for sequence learning with the self-attention
mechanism. The vanilla Transformer can be easily modified fortime series prediction, while it still fails for distribution matching.
Time series segmentation and clustering are two similar topics
to our temporal distribution characterization. Time series segmentation [6, 28] aim to separate time series into several pieces that canbe used to discover the underlying patterns; it mainly uses changepoint detection algorithms, include sliding windows, bottom-up,
and top-down methods. The methods in segmentation almost don’t
utilize the distribution matching scheme and can’t adapt to our
problem. Time-series clustering [16, 34], on the other hand, aims
at finding different groups consisting of similar (segments of) timeseries, which is clearly different from our situation.

2.2

Distribution matching

When the training and test data are coming from different distributions, it is common practice to adapt some domain adaptation(DA) algorithms to bridge their distribution gap, such that domaininvariant representations can be learned. DA often performs instance re-weighting or feature transfer to reduce the distribution
divergence in training and test data [12, 44, 47, 48, 56]. Similar to
DA in general methodology, domain generalization (DG) [49] also
learns a domain-invariant model on multiple source domains in thehope that it can generalize well to the target domain [2, 26, 30]. The
only difference between DA and DG is that DA assumes the test datais accessible while DG does not. It is clear that our problem settings
are different from DA and DG. Firstly, DA and DG does not built for
temporal distribution characterization since the domains are givena prior in their problems. Secondly, most DA and DG methods arefor classification tasks using CNNs than RNNs. Thus, they might
fail to capture the long-range temporal dependency in time series.
Several methods applied pre-training to RNNs [4, 8, 11, 15, 51]
which is similar to CNN-based pre-training.

3

PROBLEM FORMULATION

Problem 1 (𝑟 -step ahead prediction). Given a time series of𝑛 segments D = {x𝑖 , y𝑖 }𝑛𝑖=1 , where x𝑖 = {𝑥𝑖1, · · · , 𝑥𝑖𝑚𝑖 } ∈ R𝑝×𝑚𝑖 is

AdaRNN: Adaptive Learning and Forecasting for Time Series

1σ 𝐷(𝒟σ𝑖,𝑗 𝑖𝐷(𝒟maxmax, 𝒟𝑗 )𝑖 , 𝒟𝑗 )
𝑖,𝑗𝐾
max
𝐾 σ
𝑖,𝑗 𝑑(𝒟𝑖 , 𝒟𝑗 )𝑡𝑡 𝑡 𝑡 𝑡𝑡𝑉𝑉
σ𝑉𝑖,𝑗𝛼𝛴𝑖,𝑗
𝑡𝐷(𝒉
ℒ=𝑡𝑑𝑚
𝛼𝑖,𝑗
σ𝑖,𝑗=𝛴𝑡=1ℒℒ
𝒉, 𝑗𝒉)𝑡 )𝑖 , 𝒉𝑗 )𝑡=1
𝑡𝑑𝑚
𝑖 ,𝑡𝑖𝐷(𝒉𝑡𝑑𝑚 = σ𝑖,𝑗 𝛴𝑡=1 𝛼𝑖,𝑗 𝐷(𝒉𝑗

1𝑛 1𝑛1𝑛 …
𝑛1 𝑛…
𝐾−1
𝐾 𝑛𝐾
𝑛𝐾𝑛𝐾−1
𝑛𝑛𝐾+1
1
2 …

𝑇𝑇
𝑇
11
2
12 2
𝛼𝛼
𝛼𝑖,𝑗
𝛼𝛼
𝛼𝛼𝑖,𝑗
𝑖,𝑗𝑖,𝑗𝑖,𝑗𝑖,𝑗 𝛼𝑖,𝑗𝑖,𝑗𝑖,𝑗𝛼

Shareddecoder
decoder
SharedShared
decoder

Boosting-based
Boosting-based
Boosting-based
Importance
Evaluation
Importance
Evaluation
Importance
Evaluation𝒊 , 𝒊𝒚𝒊 )𝒊 𝒊 𝒊
σσ
ෝ, 𝒚
ℒℒ
=𝑝𝑟𝑒𝑑
ෝ)
ℒ
= σ𝑖 𝑚𝑠𝑒(𝒚𝑝𝑟𝑒𝑑𝑖 𝑚𝑠𝑒(𝒚ෝ ) ,𝒚𝑝𝑟𝑒𝑑 =
𝑖 𝑚𝑠𝑒(𝒚𝑖 𝑖𝑖𝑖𝑖 𝑖 𝑦
𝑖
𝑖
𝑖
𝑦ො𝑡−1
ො
𝑦
ො
𝑦ො𝑡𝑡+1 𝑦ො𝑡+1
𝑡𝑦
𝑦ො𝑡−1 𝑦ො𝑡−1
ො𝑡 𝑡+11

1

Adaptive
RNN
model
Adaptive
RNN
model
Adaptive
RNN
model
TemporalTemporalTemporalDistribution
Distribution
Distribution
Matching(TDM)
Matching(TDM)
Matching(TDM)
Pre-training
Pre-training
Pre-training

𝐾

…… …

encoder
Sharedencoder
SharedShared
encoder𝒟𝒟
1 1 𝒟1 …
… …𝒟𝒟𝐾 𝐾 𝒟𝐾
11
2𝑉𝑉
12 2
𝑇
𝒉𝒉
𝑖 𝑖 𝒉𝒉𝑖 𝑖 𝒉𝒉𝑖 𝑖 𝒉𝑖𝑖𝒉
𝑖𝒉

TemporalTemporalTemporalDistribution
Distribution
Distribution
Characterization(TDC)
Characterization(TDC)
Characterization(TDC)
Raw
trainingdata
Raw
training
datadata
Raw
training

CIKM ’21, November 1–5, 2021, Virtual Event, Australia.

…… …
𝒙𝑛 +1 ⋯ 𝒙𝑛
Raw
trainingdata
Raw
training
datadata
Raw
training

11𝒉𝑗𝒉
𝒉12 2 𝒉2𝑉𝒋 𝑉 𝒉𝑇𝒋𝒋𝑗 𝒉𝑗𝑗𝒉𝑗 𝒉𝑗𝒉⋯⋯𝒙𝑛𝒙𝑗 ⋯ 𝒙
+1
𝒙𝑖−1
𝒙+1
𝒙𝑛𝑖 𝒙𝑛𝒙𝑗−1
𝒙+1
𝑛𝑖−1
𝑛𝑗−1
𝑛𝑖−1⋯
+1𝒙𝑖𝑛⋯
𝑛𝑗−1
+1𝑛
𝑛
𝑖
…
𝒟𝒟
𝒟
𝑖 𝑖 𝒟
… … 𝒟𝑗 𝑗 𝒟𝑗𝑗 𝑗
𝑖

…… …

(c)
Distribution
Matching(TDM)(a)
ofofAdaRNN
TemporalDistribution(a) Overview
of(b)(b)
Temporal
distribution
characteriTemporal
distribution
matching(TDM)(c)TemporalTemporalDistribution
Matching(TDM)(a)Overview
Overview
AdaRNN
TemporalDistribution(c)(c)Temporal
Distribution
Matching(TDM)(a)
Overview
ofAdaRNN
AdaRNN(b)(b)
TemporalDistribution
zation(TDC)
Characterization(TDC)
Characterization(TDC)
Characterization (TDC)
Figure 2: The framework of AdaRNN.
a 𝑝-variate segment of length 𝑚𝑖 1 , and y𝑖 = (𝑦𝑖1, . . . , 𝑦𝑖𝑐 ) ∈ R𝑐 is thecorresponding label.
Learn a precise prediction model M : x𝑖 → y𝑖 on the future𝑟 ∈ N+ steps 2 for segments {x 𝑗 }𝑛+𝑟𝑗=𝑛+1 in the same time series.
In most existing prediction approaches, all time series segments,{x𝑖 }𝑛+𝑟𝑖=1 , are assumed to follow the same data distribution; a prediction model learned from all labeled training segments are supposedto perform well on the unlabeled testing segments. However, thisassumption is too strong to hold in many real-world applications.
In this work, we consider a more practical problem setting: thedistributions of the training segments and the test segments can bedifferent, and the distributions of the training segments can also
be changed over time as shown in Figure 1. In the following, weformally define our problem.

vary with the change of market (i.e., different distributions in different periods of time), some common knowledge, e.g., the economic
laws and patterns (𝑃 (𝑦|𝑥)) in different periods, can still be utilized
to make precise predictions over time.
However, the number of periods 𝐾 and the boundaries of each period under TCS are usually unknown in practice. Therefore, before
designing a model to capture commonality among different periods
in training, we need to first discover the periods by comparing
their underlying data distributions such that segments in the same
period follow the same data distributions. The formal definition of
the problem studied in this work is described as follows.

Note that the definition of covariate shift is for non-time series
data. We now extend its definition for time series data as follows,which is referred to as temporal covariate shift (TCS) in this work.

Problem 2 (𝑟 -step TS prediction under TCS). Given a timeseries of 𝑛 segments with corresponding labels, D = {x𝑖 , y𝑖 }𝑛𝑖=1 , fortraining. Suppose there exist an unknown number of 𝐾 underlying
periods in the time series, so that segments in the same period 𝑖 follow
the same data distribution 𝑃 D𝑖 (x, 𝑦), while for different time periods
1 ≤ 𝑖 ≠ 𝑗 ≤ 𝐾, 𝑃 D𝑖 (x) ≠ 𝑃 D 𝑗 (x) and 𝑃 D𝑖 (𝑦|x) = 𝑃 D 𝑗 (𝑦|x).
Our goal is to automatically discover the 𝐾 periods in the training
time series data, and learn a prediction model M : x𝑖 → y𝑖 by
exploiting the commonality among different time periods, such that it
makes precise predictions on the future 𝑟 segments, D𝑡𝑠𝑡 = {x 𝑗 }𝑛+𝑟𝑗=𝑛+1 .
Here we assume that the test segments are in the same time period,
𝑃 D𝑡𝑠𝑡 (x) ≠ 𝑃 D𝑖 (x) and 𝑃 D𝑡𝑠𝑡 (𝑦|x) = 𝑃 D𝑖 (𝑦|x) for any 1 ≤ 𝑖 ≤ 𝐾.

Definition 2 (Temporal Covariate Shift). Given a time series
data D with 𝑛 labeled segments. Suppose it can be split into 𝐾 periods
𝑘+1
or intervals, i.e., D = {D1, · · · , D𝐾 }, where D𝑘 = {x𝑖 , y𝑖 }𝑛𝑖=𝑛,𝑘 +1
𝑛 1 = 0 and 𝑛𝑘+1 = 𝑛. Temporal Covariate Shift (TCS) is referred to thecase that all the segments in the same period 𝑖 follow the same data
distribution 𝑃 D𝑖 (x, 𝑦), while for different time periods 1 ≤ 𝑖 ≠ 𝑗 ≤ 𝐾,
𝑃 D𝑖 (x) ≠ 𝑃 D 𝑗 (x) and 𝑃 D𝑖 (𝑦|x) = 𝑃 D 𝑗 (𝑦|x).

Note that learning the periods underlying the training time series
data is challenging as both the number of periods and the corresponding boundaries are unknown, and the search space is huge.
Afterwards, with the identified periods, the next challenge is how
to learn a good generalizable prediction model M to be used in the
future periods. If the periods are not discovered accurately, it may
affect the generalization performance of the final prediction model.

Definition 1 (Covariate Shift [41]). Given the training andthe test data from two distributions 𝑃𝑡𝑟𝑎𝑖𝑛 (x, 𝑦), 𝑃𝑡𝑒𝑠𝑡 (x, 𝑦), covariate
shift is referred to the case that the marginal probability distributions are different, and the conditional distributions are the same, i.e.,
𝑃𝑡𝑟𝑎𝑖𝑛 (x) ≠ 𝑃𝑡𝑒𝑠𝑡 (x), and 𝑃𝑡𝑟𝑎𝑖𝑛 (𝑦|x) = 𝑃𝑡𝑒𝑠𝑡 (𝑦|x).

To learn a prediction model with good generalization performance under TCS, a crucial research issue is to capture the common
knowledge shared among different periods of D [22]. Take the stock
price prediction as an example. Though the financial factors (𝑃 (𝑥))1 In general, 𝑚𝑖 may be different for different time segments.
2 When 𝑟 = 1, it becomes the one-step prediction problem.

4

OUR PROPOSED MODEL: ADARNNIn this section, we present our proposed framework, termed Adaptive RNNs (AdaRNN), to solve the temporal covariate shift problem.
Figure 2(a) depicts the overview of AdaRNN. As we can see, itmainly consists of two novel algorithms:

CIKM ’21, November 1–5, 2021, Virtual Event, Australia.

Du, et al.

• Temporal Distribution Characterization (TDC): to characterize
the distribution information in TS;• Temporal Distribution Matching (TDM): to match the distributions of the discovered periods to build a time series
prediction model M.
Given the training data, AdaRNN first utilizes TDC to split it into
the periods which fully characterize its distribution information.
It then applies the TDM module to perform distribution matchingamong the periods to build a generalized prediction model M. The
learned M is finally used to make 𝑟 -step predictions on new data.
The rationale behind AdaRNN is as follows. In TDC, the model
M is expected to work under the worst distribution scenario where
the distribution gaps among the different periods are large, thus
the optimal split of periods can be determined by maximizing their
dissimilarity. In TDM, M utilizes the common knowledge of thelearned time periods by matching their distributions via RNNs witha regularization term to make precise future predictions.

4.1

Temporal distribution characterization

Based on the principle of maximum entropy [17], maximizing theutilization of shared knowledge underlying a times series under
temporal covariate shift can be done by finding periods which aremost dissimilar to each other, which is also considered as the worst
case of temporal covariate shift since the cross-period distributions
are the most diverse. Therefore, as Figure 2(b) shows, TDC achieves
this goal for splitting the TS by solving an optimization problem
whose objective can be formulated as:
max

max

1

0<𝐾 ≤𝐾0 𝑛 1 ,··· ,𝑛𝐾 𝐾∑︁𝑑 (D𝑖 , D 𝑗 )1≤𝑖≠𝑗 ≤𝐾

s.t. ∀ 𝑖, Δ1 < |D𝑖 | < Δ2 ;∑︁

(1)
|D𝑖 | = 𝑛case, it would have better generalization ability on unseen test data.
This assumption has also been validated in theoretical analysis oftime series models [22, 31] that diversity matters in TS modeling.
In general, the time series splitting optimization problem in (1)is computationally intractable and may not have a closed-form
solution. By selecting a proper distance metric, however, the optimization problem (1) can be solved with dynamic programming(DP) [36]. In this work, by considering the scalability and the efficiency issues on large-scale data, we resort to a greedy algorithm
to solve (1). Specifically, for efficient computation and avoiding
trivial solutions, we evenly split the time series into 𝑁 = 10 parts,
where each part is the minimal-unit period that cannot be split
anymore. We then randomly search the value of 𝐾 in {2, 3, 5, 7, 10}.
Given 𝐾, we choose each period of length 𝑛 𝑗 based on a greedy
strategy. Denote the start and the end points of the time series
by 𝐴 and 𝐵, respectively. We first consider 𝐾 = 2 by choosing 1
splitting point (denote it as 𝐶) from the 9 candidate splitting points
via maximizing the distribution distance 𝑑 (𝑆𝐴𝐶 , 𝑆𝐶𝐵 ). After 𝐶 isdetermined, we consider 𝐾 = 3 and use the same strategy to selectanother point 𝐷. A similar strategy is applied to different values of𝐾. Our experiments show that this algorithm can select the more
appropriate periods than random splits. Also, it shows that when 𝐾
is very large or very small, the final performance of the prediction
model becomes poor (Section 5.6.1).

4.2

Temporal distribution matching

Given the learned time periods, the TDM module is designed tolearn the common knowledge shared by different periods via matching their distributions. Thus, the learned model M is expected togeneralize well on unseen test data compared with the methods
which only rely on local or statistical information.
The loss of TDM for prediction, L𝑝𝑟𝑒𝑑 , can be formulated as:𝑖where 𝑑 is a distance metric, Δ1 and Δ2 are predefined parameters to
avoid trivial solutions (e.g., very small values or very large values
may fail to capture the distribution information), and 𝐾0 is thehyperparameter to avoid over-splitting. The metric 𝑑 (·, ·) in Eq. (1)can be any distance function, e.g., Euclidean or Editing distance, orsome distribution-based distance / divergence, like MMD [14] andKL-divergence. We will introduce how to select 𝑑 (·, ·) in Section 4.3.
The learning goal of the optimization problem (1) is to maximize
the averaged period-wise distribution distances by searching 𝐾 and
the corresponding periods so that the distributions of each period
are as diverse as possible and the learned prediction model has
better a more generalization ability.
We explain in detail for the splitting scheme in (1) with the principle of maximum entropy (ME). Why we need the most dissimilar
periods instead of the most similar ones? According to ME, whenno prior assumptions are made on the splitting of the time series
data, it is reasonable that the distributions of each period become
as diverse as possible to maximize the entropy of the total distributions. This allows more general and flexible modeling of the future
data. Therefore, since we have no prior information on the test
data which is unseen during training, it is more reasonable to traina model at the worst case, which can be simulated using diverse
period distributions. If the model is able to learn from the worst

L𝑝𝑟𝑒𝑑 (𝜃 ) =| D𝑗 |
𝐾
1 ∑︁ 1 ∑︁𝑗𝑗ℓ (y𝑖 , M (x𝑖 ; 𝜃 )),𝐾 𝑗=1 |D 𝑗 | 𝑖=1(2)

where (x𝑖 , y𝑖 ) denotes the 𝑖-th labeled segment from period D 𝑗 ,ℓ (·, ·) is a loss function, e.g., the MSE loss, and 𝜃 denotes the learnable model parameters.
However, minimizing (2) can only learn the predictive knowledge
from each period, which cannot reduce the distribution diversities
among different periods to harness the common knowledge. A naïve
solution is to match the distributions in each period-pair D𝑖 and D 𝑗
by adopting some popular distribution matching distance (𝑑 (·, ·)) as
the regularizer. Following existing work on domain adaptation [12,42] that distribution matching is usually performed on high-level
representations, we apply the distribution matching regularization
term on the final outputs of the cell of RNN models. Formally, we use
H = {h𝑡 }𝑉𝑡=1 ∈ R𝑉 ×𝑞 to denote the 𝑉 hidden states of an RNN withfeature dimension 𝑞. Then, the period-wise distribution matchingon the final hidden states for a pair (D𝑖 , D 𝑗 ) can be represented as:𝑗𝑗

L𝑑𝑚 (D𝑖 , D 𝑗 ; 𝜃 ) = 𝑑 (h𝑉𝑖 , h𝑉𝑗 ; 𝜃 ).
Unfortunately, the above regularization term fails to capture thetemporal dependency of each hidden state in the RNN. As eachhidden state only contains partial distribution information of an
input sequence, each hidden state of the RNN should be consideredAdaRNN: Adaptive Learning and Forecasting for Time Series

CIKM ’21, November 1–5, 2021, Virtual Event, Australia.

while constructing a distribution matching regularizer as describedin the following section.
4.2.1 Temporal distribution matching. As shown in Figure 2(c), we
propose the Temporal Distribution Matching (TDM) module to adaptively match the distributions between the RNN cells of two periods
while capturing the temporal dependencies. TDM introduces theimportance vector 𝜶 ∈ R𝑉 to learn the relative importance of 𝑉 hidden states inside the RNN, where all the hidden states are weighted
with a normalized 𝜶 . Note that for each pair of periods, there isan 𝜶 , and we omit the subscript if there is no confusion. In thisway, we can dynamically reduce the distribution divergence ofcross-periods.
Given a period-pair (D𝑖 , D 𝑗 ), the loss of temporal distribution
matching is formulated as:
L𝑡𝑑𝑚 (D𝑖 , D 𝑗 ; 𝜃 ) =

𝑉
∑︁

𝑡
𝛼𝑖,𝑗
𝑑 (h𝑖𝑡 , h𝑡𝑗 ; 𝜃 ),(3)

h𝑖𝑡 = 𝛿 (x𝑖𝑡 , h𝑖𝑡 −1 ).

(4)

By integrating (2) and (3), the final objective of temporal distribution matching (one RNN layer) is:
𝑖≠𝑗
∑︁
2
L
(D𝑖 , D 𝑗 ; 𝜃, 𝜶 ), (5)𝐾 (𝐾 − 1) 𝑖,𝑗 𝑡𝑑𝑚

where 𝜆 is a trade-off hyper-parameter. Note that in the second
term, we compute the average of the distribution distances of all
pairwise periods. For computation, we take a mini-batch of D𝑖 andD 𝑗 to perform forward operation in RNN layers and concatenate
all hidden features. Then, we can perform TDM using (5).
4.2.2 Boosting-based importance evaluation. We propose a Boostingbased importance evaluation algorithm to learn 𝜶 . Before that, wefirst perform pre-training on the network parameter 𝜃 using thefully-labeled data across all periods, i.e., use (2). This is to learn
better hidden state representations to facilitate the learning of 𝜶 .
We denote the pre-trained parameter as 𝜃 0 .
With 𝜃 0 , TDM uses a boosting-based [39] procedure to learn
the hidden state importance. Initially, for each RNN layer, all theweights are initialized as the same value in the same layer, i.e.,
𝜶𝑖,𝑗 = {1/𝑉 }𝑉 . We choose the cross-domain distribution distanceas the indicator of boosting. If the distribution distance of epoch𝑡,(𝑛+1)𝑛 + 1 is larger than that of epoch 𝑛, we increase the value of 𝛼𝑖,𝑗to enlarge its effect for reducing the distribution divergence. Otherwise, we keep it unchanged. The boosting is represented as:𝑡,(𝑛)𝑡,(𝑛) 𝑡,(𝑛−1)𝑡,(𝑛)𝑡,(𝑛−1)) 𝑑𝑖,𝑗 ≥ 𝑑𝑖,𝑗𝛼𝑖,𝑗 × 𝐺 (𝑑𝑖,𝑗 , 𝑑𝑖,𝑗𝑡,(𝑛+1)
𝛼𝑖,𝑗
=
(6)
𝑡,(𝑛)

𝛼𝑖,𝑗
otherwise

where
𝑡,(𝑛)

𝑡,(𝑛−1), 𝑑𝑖,𝑗is the updating function computed by distribution matching loss
𝑡,(𝑛)
𝑡,(𝑛)
at different learning stages. And 𝑑𝑖,𝑗 = 𝐷 (h𝑖𝑡 , h𝑡𝑗 ; 𝛼𝑖,𝑗 ) is the
distribution distance at time step 𝑡 in epoch 𝑛. 𝜎 (·) is the sigmoid
function. It is easy to see that 𝐺 (·, ·) > 1, ensuring the incrementof the importance. Similar to naive method, we normalize it as𝑡,(𝑛+1)𝑡 denotes the distribution importance between the periods
where 𝛼𝑖,𝑗D𝑖 and D 𝑗 at state 𝑡.
All the hidden states of the RNN can be easily computed byfollowing the standard RNN computation. Denote by 𝛿 (·) the computation of a next hidden state based on a previous state. The statecomputation can be formulated as

𝐺 (𝑑𝑖,𝑗Require: A time series dataset D, 𝜆, max iteration number 𝐼𝑡𝑒𝑟
1: Perform temporal distribution characterization by solving (1)to get 𝐾 periods {D𝑖 }’s.
2: End-to-end pre-training by minimizing (2) with 𝜆 = 0
3: For iteration 𝑖 = 1 to 𝐼𝑡𝑒𝑟 :4:
Learn the value of 𝜶 and 𝜃 by solving (5) and using (6)5: End For
6: return Optimal parameters 𝜃 ∗ , 𝜶 ∗ .

𝑡,(𝑛+1)𝑡 =1

L (𝜃, 𝜶 ) = L𝑝𝑟𝑒𝑑 (𝜃 ) + 𝜆Algorithm 1 AdaRNN𝑡,(𝑛)) = (1 + 𝜎 (𝑑𝑖,𝑗𝑡,(𝑛−1)− 𝑑𝑖,𝑗

))

(7)

𝛼𝑖,𝑗

𝛼

= Í𝑉 𝑖,𝑗 𝑡,(𝑛+1) . By (3) and (6), we can learn the value of 𝜶 .
𝑗 =1 𝛼𝑖,𝑗Remark 1. The objective (3) is defined on one RNN layer, it can beeasily implemented for multiple layers to achieve multi-layer distribution matching. As for inference, we simply use the optimal network
parameter 𝜃 ∗ . The computation of domain-wise distribution distances
can also be made efficient following existing work [26].
Remark 2. In fact, there is also a naïve approach to acquire 𝜶 byusing another network with weight W𝑖,𝑗 ∈ R2𝑞×𝑉 that takes (H𝑖 , H 𝑗 )as inputs and output 𝜶 . For instance, 𝜶 can be learned as 𝜶𝑖,𝑗 =𝑔(W𝑖,𝑗 ⊙ [H𝑖 , H 𝑗 ] ⊤ ) where 𝑔 is an activation and normalization
function and ⊙ is element-wise production. After computing 𝛼𝑖,𝑗 ,we normalize it by a softmax function. Unfortunately, this naïve
approach does not work in reality for two reasons. First, since 𝜶𝑖,𝑗 and𝜃 are highly correlated and at early training stage, the hidden state
representations learned by 𝜃 are less meaningful, which will result ininsufficient learning of 𝜶𝑖,𝑗 . Second, it is expensive to learn such W𝑖,𝑗for each domain pair in RNN models.

4.3

Computation of the distribution distance

AdaRNN is agnostic to RNN structures (vanilla RNNs, LSTMs, andGRUs) and the distribution distances, i.e., the function 𝑑 (·, ·). Technically, 𝑑 (·, ·) in TDC and TDM can be the same choice. We adopt
several popular distances: cosine distance, MMD [14] and adversarial distance [12]. Also note that the TDC and TDM are currently
two separate stages for controlling computation complexity. It shallbe a future work to end-to-end optimize these two stages. The complete process of AdaRNN is presented in Algorithm 1. The detailed
implementations of these distances are presented in Appendix.

5 EXPERIMENTS
5.1 Setup
We mainly use GRUs (Gated Recurrent Units) [7] as the base RNN
cells for its high performance in real applications3 . We test AdaRNN
in four real-world datasets as summarized in Table 1.
We compare AdaRNN with four categories of methods:
3 It is natural to use simple RNNs or LSTMs.

CIKM ’21, November 1–5, 2021, Virtual Event, Australia.

Du, et al.

Table 1: Information of the datasets used in this paper.
DatasetUCI activity
Air quality
Electric power
Stock price#Train
6,000
29,232
1,235,964
778,200#Valid
1,352
2,904
413,280
271,690

#Test
2,947
2,832
413,202504,870

dim. (𝑝)
128
6
7
360

Human activity recognition

The UCI smartphone activity [1] is a time-series dataset for classification task. It is collected by thirty volunteers wearing a smartphone to record the accelerometer, gyroscope, and magnetometersignals while performing six different activities (walking, upstairs,
downstairs, sitting, standing, and lying). We use 7, 352 instances fortraining and 2, 947 instances as the test following [1]. It is clear thattraining and testing sets have different distributions and TCS exists
in the training set due to different activities and users. We firstly
clip the data using sliding window with the size of 128. As there are
9 channels for each instance, we regard it a sequence with lengthof 128 and feature of 9 dimension. For this task, we use accuracy
4 STRIPE is an improved and stronger version of DILATE [46].

MethodLightGBMGRU
MMD-RNN
DANN-RNN
AdaRNN

Task
Classification
Regression
Regression
Regression• Traditional TS modeling methods, including ARIMA [53],FBProphet [10], LightGBM, and GRU [7];• Latest TS models, including LSTNet [23] and STRIPE [24]4 ;• Transformer [45] by removing its mask operation;• Variants of popular domain adaptation / generalization methods and we denote them as “MMD-RNN” and “DANN-RNN”.
Note that the comparison is mainly for the final prediction performance using temporal distribution matching after obtaining thetime periods using TDC, which the evaluation of TDC is in Section 5.6.1. Transformer [45] was originally proposed for machinetranslation. Therefore, we alter it by removing the masks for TS databy following implementation 5 . We use 4 encoder layers of Transformer with 8 heads for self-attention tuned by cross-validation toensure that it achieves the best performance on the datasets. Existing DA / DG methods cannot be directly applied to our problem,since most of them are for CNN-based classification tasks. Therefore, we use variants of them. Specifically, we use “MMD-RNN” todenote popular MMD-based methods such as MEDA [47, 48] andMMD-AAE [26]. We use “DANN-RNN” to denote popular domainadversarial training methods such as RevGrad [12] and DAAN [52].
These two distances are widely adopted in recent DA and DG literature. We change their backbone network to be RNNs which are
the same as other comparison methods and then add the MMD oradversarial distances to the final output of the RNN layers.
All methods use the same train-valid-test split and periods after TDC and the final results are averaged on five random runs.
Optimal hyperparameters are tuned on the validation set. Notethat to balance the efficiency and performance in AdaRNN, welimit the range of 𝐾 in {2, 3, 5, 7, 10} to acquire the best splitting of
periods. We set the pre-training epochs to be 10 in TDM. We use
different distance function 𝑑 and the main results are based on the
MMD [14]. Other distances are discussed later. Code of AdaRNN
is at https://github.com/jindongwang/transferlearning/tree/master/
code/deep/adarnn.

5.2

Table 2: Results on UCI time series classification dataset.

5 https://timeseriestransformer.readthedocs.io/en/latest/README.html

ACC
84.11
85.68
86.39
85.88
88.44

P
83.73
85.62
86.80
85.59
88.71

R
83.63
85.51
86.26
85.62
88.59

F1
84.91
85.46
86.38
85.56
88.63

AUC
90.23
91.33
91.77
91.41
93.19

(ACC), precision (P), recall (R), F1, and area under curve (AUC) as
the evaluation metrics.
The network is composed of two-layer GRUs with the hidden
state dimension 32 and a FC layer with output to be 𝑟 = 6. We
optimize by Adam optimizer with the learning rate of 0.005 with abatch size 64. We set 𝜆 = 0.1. On this dataset, we did not run thelatest methods such as LSTNet and STRIPE as well as FBProphet
and ARIMA since they are built for regression tasks. Transformer
is not compared since we did not get good performance.
Table 2 shows the results on this classification task. It is obviousthat AdaRNN achieves the best performance on all evaluation
metrics that outperforms the adversarial-based DANN method by
2.56% in accuracy and 3.07% in F1 score. This means that AdaRNN
is effective in non-stationary time series classification tasks.

5.3

Air quality prediction

The air-quality dataset [54] contains hourly air quality informationcollected from 12 stations in Beijing from 03/2013 to 02/2017. We
randomly chose four stations (Dongsi, Tiantan, Nongzhanguan, andDingling) and select six features (PM2.5, PM10, S02, NO2, CO, andO3). Since there are some missing data, we simply fill the empty
slots using averaged values. Then, the dataset is normalized before
feeding into the network to scale all features into the same range.
This process is accomplished by max-min normalization and ranges
data between 0 and 1.
The network is composed of two-layer GRUs with the dimension of the hidden state to be 64 and two bottleneck layers whichcontains two FC layers, and an FC layer with output to be 𝑟 = 1.
We set 𝜆 = 0.5. We optimize all the models by Adam optimizer withthe learning rate of 0.005, we set the batch size to be 36.
Table 3 presents the results where the last column “Δ(%)” denotes
the average increment over the baseline GRU. It can be seen that the
proposed method achieves the best results in all four stations. Our
AdaRNN significantly outperforms the strong baseline STRIPE [24]by 8.97% in RMSE reduction.

5.4

Household electric power consumption

This dataset [18] contains 2,075,260 measurements gathered between 16/12/2006 and 26/11/2010 with a one-minute samplingrate. It contains several attributes (global active power, global reactive power, voltage, global intensity, and three substrings). We get
2,062,346 measurements for the whole dataset after removing all
the null data. We construct the train/valid/test sets with a ratio of
6 : 2 : 2, i.e., train on data from 16/12/2006 to 24/04/2009 and test
on data from 25/02/2010 to 26/11/2010 and the rest is valid set. We
use the data of 5 consecutive days to predict the value of next day.
The network is composed of two-layer GRUs with the dimension
set to be 64 and two bottleneck layers which contains two FC

AdaRNN: Adaptive Learning and Forecasting for Time Series

CIKM ’21, November 1–5, 2021, Virtual Event, Australia.

Table 3: Results on air quality prediction (left) and household electric power consumption prediction (right).
Tiantan
RMSE
MAE

Nongzhanguan
RMSE
MAE

Dingling
RMSE
MAE

Δ(%)

Electric Power0.18660.1811
0.05100.03600.0356
0.05870.05440.03390.03650.02950.14340.14140.04750.01830.02140.04120.05190.02330.02040.01640.15510.15570.04590.02670.02740.0436
0.05480.02260.02480.0196

0.0932
0.09220.03470.02880.02910.03220.05990.02630.03040.02330.00-61.31-59.97-11.08-61.20-64.60-73.57

0.080
0.093
0.082
0.080
0.080
0.080
0.079
0.086
0.077

0.14030.1356
0.03800.02670.02550.03900.0651
0.02200.02160.01850.11190.10820.03480.01330.01570.02890.06510.01640.01480.0112

3

0.08

0.03
Distance

Dingling
Dongsi0.12210.1156
0.03300.01970.02030.03190.06960.01810.01540.0122

0.03

0.04

RMSE

Naive
Boosting
Pre-train + naive
Pre-train + boost

0.02

RMSE

Best results

2.5

RMSE

Distance

RMSE

0.06
0.04

0.07360.07090.02440.01680.0211
0.02100.07050.01630.01390.0150

0.03

Naive

RMSE

FBProphet [10]
ARIMA
GRU
MMD-RNN
DANN-RNN
LightGBM
LSTNet [23]Transformer [45]
STRIPE [24]AdaRNN

Dongsi
RMSE
MAE

Ours

0.025

0.02

0.02
0

2

3

5

7

10

#Domain(a) #Domain in TDC

2Split 1

Split 2

Ours

0.01

Split

(b) Different domain split

0.01

0.02
DongsiCosineTiantanAdv

MMD CORAL

Distance

Station(c) Temporal distribution matching(d) Effectiveness of 𝛼

Figure 3: Ablation study. (a) Analysis of 𝐾 for temporal distribution characterization. (b) Different domain split strategies. (c)Comparison of w. or w./o. adaptive weights in distribution matching. (d) Effectiveness of temporal distribution matching.
layers, and a FC layer with output to be 𝑟 = 1. We set 𝜆 = 0.001. We
optimize all the models by Adam optimizer with the learning rateof 0.0005 and batch size of 36.
The right part of Table 3 shows the RMSE of predicting thehousehold power consumption using different methods. Again, our
AdaRNN achieves the best performance.

5.5

Stock price prediction

We further evaluate the performance of AdaRNN on a large private
finance dataset where the task is to predict the stock price given
the historical data from 01/2007 to 12/2019. This dataset contains
over 2.8M samples with 360 financial factors as the features. We use
the data from 01/2017 to 12/2019 as the test set, and the data from
01/2007 to 12/2014 as the training set with the rest as the validation
set. We use information coefficient (IC) and information ratio (IR)as the evaluation metrics following existing work [21]. The detailed
description and computation of these metrics are in appendix.
The network is composed of two-layer GRUs with the dimensionof hidden state to be 64 and two bottleneck layers which contains
one FC layers, and a FC layer with output to be 𝑟 = 1. We optimize
all the models by Adam optimizer with the learning rate of 0.0002,
we set the batch size to be 800 and 𝜆 = 0.5.
Table 4 shows that our proposed AdaRNN outperforms othercomparison methods in all metrics. While achieving the best IC
values, its RankIC can still be higher than other methods. Thismeans that AdaRNN is relatively steady in real applications.

5.6

Ablation study

5.6.1 Temporal distribution characterization. To evaluate TDC, we
design two experiments. Firstly, we compare the results of 𝐾 =2, 3, 5, 7, 10 to analyze the importance of 𝐾 in Figure 3(a). We see

Table 4: Results on stock price prediction.
MethodLightGBMGRU
MMD-RNN
DANN-RNN
Transformer [45]
STRIPE [24]AdaRNN

IC
0.063
0.106
0.107
0.106
0.104
0.108
0.115

ICIR
0.551
0.965
0.962
0.964
0.935
0.987
1.071

RankIC
0.056
0.098
0.101
0.103
0.099
0.101
0.110

RankICIR
0.502
0.925
0.926
0.924
0.920
0.946
1.035

RawIC
0.064
0.109
0.108
0.107
0.101
0.109
0.116

RawICIR
0.568
0.986
0.967
0.966
0.946
0.993
1.077

that the performance is changing with the increment of 𝐾 and thebest result is achieved when 𝐾 = 3, 5 for two stations on air quality
prediction, respectively. Then, when 𝐾 continues to increase, theperformance drops. It means that 𝐾 is important in characterizing
the distribution information in non-stationary TS.
Secondly, Figure 3(b) shows the comparison of domain split
algorithms using the same distribution matching method when𝐾 = 2. Here, “Split 1” denotes a random split and “Split 2” denotes
the split where all periods are with similar distributions (i.e., just the
opposite of TDC). The “distance” denotes the distribution distance.
The results indicate that our TDC achieves the best performance in
RMSE with the largest distribution distance. The results of othervalues of 𝐾 are also the same. Therefore, it is important that we
split the periods to the worst case when the distributions are themost diverse. Both experiments show that TDC could effectively
characterize the distribution information in time series. Evaluationof TDC using other distances are in appendix.
5.6.2 Temporal distribution matching. We analyze the TDM module by evaluating the performance of pre-training and the boostingbased method. We compare four variants of TDM: (1) naïve approach (i.e., learning 𝜶 by a feed-forward layer as discussed inremark 2) w./o pre-training, (2) boosting w./o pre-training (3) pretraining + naïve, (4) pre-training + boosting (Ours).

CIKM ’21, November 1–5, 2021, Virtual Event, Australia.

0.04

8
CosineAdv

MMD

Du, et al.

10-3

0.13
MMD

CORAL

Adv.

0.05

0.01
STRIPE

AdaRNN

Ours

MSE
TDM

0.008

500
RMSE
Time (s)

0.04

400

S2

S3

0.004
0.002

0.07

0
S1

0.006

0.09
2

0.01

RMSE

4

Loss

RMSE

Distance

RMSE
0.02

0.11

S4

Station(a) Different distribution distance

20

40

60

80

100

0
6

12

24

20

40

Hour

Epoch

(b) Change of distance

18

60

80

0.03

300

0.02

200

0.01

100

0

100

STRIPEGRU

Epoch(c) Multi-step prediction

Time (s)

6

0.03

0

AdaRNN

Method(d) Convergence(e) Training time

Figure 4: (a) Multi-step prediction of air quality. (b) TDM is agnostic to distribution distances in different stations. (c) The
change of distribution distance. (d) Convergence analysis. (e) Training time.
1

1
Prediction

0.8

0.6

0.6

0.4
0.2

Prediction

0.4
0.2

0

0
20

40

60

80

100

20

40

60

80

Point

Point

(a) MMD-RNN(b) DANN-RNN

1

100

0.2
Groundtruth

Prediction

DANN

0.8

MMD

Ours

60

80

0.15

Value

0.6
0.4

0.1
0.05

0.2
0

0
20

40

60

80

Point

(c) AdaRNN

5.7

Groundtruth

0.8

Value

Value

Groundtruth

Value

Figure 3(c) shows the performance on two stations. We observethat the pre-training technique helps both naïve and boostingbased methods to get better performance. More importantly, ourboosting-based method can achieve the best performance with pretraining. Compared with the STRIPE method from Table 3, we seethat almost all four variants of TDM can achieve better performance
than STRIPE. We also observe that by manually setting the same
importance to each hidden state, the results are not comparable.
To summarize, it is necessary to consider the importance of eachhidden state when trying to build an adaptive RNN model.
To evaluate the importance vector 𝜶 , we compare the resultsof with or without importance vector in Figure 3(d). The results
indicate that by taking the same distribution distance, our AdaRNN
achieve the best results. This demonstrates that the different distributions of each hidden state is important to model in RNN-based
time series modeling, to which our AdaRNN is capable of dynamically evaluating their importance to achieve the best performance.

100

20

40

100

Point

(d) Differences

Further analysis

5.7.1 Distribution distance. AdaRNN is agnostic to the distribution
distance. Here we evaluate this property by running TDM with four
different kinds of distribution distances: MMD [3], CORAL [42],Cosine, and domain-adversarial distance [12]. Figure 4(a) presents
the results on three stations for air quality prediction. Combining
the results from Table 3, we observe that AdaRNN is robust to
the choice of distribution distance while achieving the best performance. Moreover, Figure 4(b) shows that by adding the importanceweights 𝜶 , our AdaRNN achieves lowest distribution distances
compared to MMD and adversarial training.
5.7.2 Multi-step prediction. We analyze the performance of AdaRNN
for multi-step (𝑟 > 1) air quality prediction and compare withSTRIPE [24] by predicting air quality index in next 6, 12, 18, 24 hours(the output sequence length is 6, 12, 18, 24 accordingly). Figure 4(c)shows the results on one station. We see that AdaRNN achieves
the best performance for multi-step prediction. In addition, as theprediction steps increase (6 → 12 → 18 → 24), the performancesof all methods are becoming worse since it is more difficult to model
the longer sequences and also affected by the cumulative errors.
5.7.3 Convergence and time complexity analysis. We analyze theconvergence of AdaRNN by recording its training loss for prediction and distribution matching in Figure 4(d). We see that AdaRNN
can be easily trained with quick convergence. On the other hand,
AdaRNN is based on RNNs which are autoregressive models. So
it is reasonable that AdaRNN takes more time for training thanFigure 5: Prediction vs. ground truth.
traditional RNNs. Figure 4(e) shows the training time comparison
between AdaRNN, GRU, and STRIPE [24]. We see that other thanthe best performance, the training time of AdaRNN is still comparable to existing methods. The inference time of AdaRNN is thesame as existing RNN-based models since we only need to compute
Eq. (2).

5.8

Case study: prediction vs. ground truth

Taking the station Dongsi as an example, we show the prediction vs.
ground truth values of different methods in Figure 5, where the redcircles denote the bad cases. We see that DANN and MMD achieve
worse results than our AdaRNN since they have relatively worse
predictions in many points. Especially in Figure 5(d), we show theabsolute difference between the prediction of each method and theground-truth. We can clearly see that our AdaRNN outperforms
all methods with much less mistakes.

6

EXTENSIONS

Apart from RNN-based models, in this section, we add the adaptive
importance weights to each hidden representations in the Transformer [45] model, which is claimed more powerful than RNNbased models in the latest studies. We add the importance weights
𝜶 to the representations of each self-attention block and term thisextension as AdaTransformer. As shown in Table 5, this pilot study

AdaRNN: Adaptive Learning and Forecasting for Time Series

CIKM ’21, November 1–5, 2021, Virtual Event, Australia.

shows the power of the AdaRNN in Transformers. We believe more
improvements can be made in the future.
Table 5: Results of vanilla and adaptive transformer
MethodVanilla Transformer
AdaTransformer

Station 1
0.028
0.025

Station 2
0.035
0.030

Table 6: Results of different distances in TDC
Split
Split1
Split2
Ours|D1 |/|D2 |0.75/0.25
0.25/0.75
0.66/0.33𝑑(MMD)
0.898
0.910
0.932

𝑑(DTW)
2.54
2.60
2.79

𝑑(CORAL)
9.9999884E-1
9.9999902E-1
9.9999919E-1

𝑑(Cosine)
0.97480.9966
0.9861RMSE
0.02330.02390.0164domain discriminator denoted by 𝐷 is introduced. The domain
adversarial objective is defined as:
𝑙𝑎𝑑𝑣 (h𝑠 , h𝑡 ) = E[log[𝐷 (h𝑠 )]] + E[log[1 − 𝐷 (h𝑡 )]],

7

CONCLUSIONS AND FUTURE WORKThis paper proposed to study the more realistic and challenging temporal covariate shift (TCS) problem for non-stationary time series.
We proposed the AdaRNN framework to learn an adaptive RNN
model for better generalization. AdaRNN consists of two novel
algorithms: temporal distribution characterization to characterize
the distribution information in TS and temporal distribution matching to build the generalized RNN model by distribution matching.
Extensive experiments show its effectiveness in three datasets.
In the future, we plan to extend this work from both algorithm
and application level. First, explore deeper extension of AdaRNN
to Transformer for better performance. Second, integrate the twostages of AdaRNN in an end-to-end network for easier training.

A

DISTRIBUTION DISTANCE

We adopt four widely used distribution distance functions following
existing work on DA/DG, i.e., cosine, MMD [3], CORAL [42], anddomain adversarial discrepancy [12].
Cosine. Cosine distance is widely used in time series data [25,50] to measure the similarity between two distributions. Given thedata h𝑠 and h𝑡 , cosine distance is defined as:⟨h𝑠 , h𝑡 ⟩𝑑𝑐𝑜𝑠𝑖𝑛𝑒 (h𝑠 , h𝑡 ) = 1 −.
(8)||h𝑠 || · ||h𝑡 ||Maximum Mean Discrepancy (MMD). The MMD [14] is a
non-parametric distance between two probability distributions.
MMD measures the distance of the mean embeddings of the samples from two distributions in a Reproducing Kernel Hilbert Space(RKHS). MMD can be empirically estimated by𝑑𝑚𝑚𝑑 (h𝑠 , h𝑡 ) =−𝑛𝑠 +𝑛𝑡
𝑛𝑠
1 ∑︁
1 ∑︁
,
ℎ
)
+
𝑘 (ℎ𝑠𝑖 , ℎ𝑠 𝑗 )
𝑘
(ℎ
𝑠
𝑠
𝑗
𝑖
𝑛𝑠2 𝑖,𝑗 =1
𝑛𝑡2 𝑖,𝑗 =𝑛𝑠 +1
𝑛∑︁
𝑠 +𝑛𝑡
2
𝑘 (ℎ𝑠𝑖 , ℎ𝑠 𝑗 ),𝑛𝑠 𝑛𝑡 𝑗 =𝑛 +1

MAE
0.01790.01970.0112(9)(11)

where E denotes expectation. Hence, the adversarial discrepancy is
𝑑𝑎𝑑𝑣 (h𝑠 , h𝑡 ) = −𝑙𝑎𝑑𝑣 (h𝑠 , h𝑡 ).

B

(12)

DIFFERENT DISTANCES IN TDC

We analyze temporal distribution characterization (TDC) by using
different distance functions. We set 𝐾 = 2 for TDC and compare
its performance with one random domain split and the opposite
domain split (i.e., with the smaller distribution distance).
The results in Table 6 indicate that our TDC algorithm achieves
the best performance in RMSE and MAE with the largest distribution distance under most distance functions. When using cosine
distance, our split achieves the second largest distribution distancebut it gets the best performance in RMSE and MAE. The results
show that TDC could effectively characterize the distribution information and is agnostic to distance functions.

C

STOCK PRICE PREDICTION: METRICS

The evaluation metrics for stock price prediction have been widelyadopted in previous work [21]. The information coefficient (IC)describes the correlation between predicted and actual stock returns,
sometimes used to measure the contribution of a financial analyst.
The IC can range from 1.0 to −1.0. An IC of +1.0 indicates a perfectlinear relationship between predicted and actual returns, while an
IC of 0.0 indicates no linear relationship. An IC of −1.0 indicatesthat the analyst always fails at making a correct prediction. IC is
defined as:
𝐼𝐶 = 𝑐𝑜𝑟𝑟 (𝑓𝑡 −1, 𝑟𝑡 ),(13)
where 𝑓𝑡 −1 is the factor value of the stock in period 𝑡 −1 and 𝑟𝑡 is thestock return rate of period 𝑡. 𝑐𝑜𝑟𝑟 (·, ·) is the correlation functions.
Besides, there are also some variants of IC, such as rankIC, which is𝑓𝑟𝑎𝑛𝑘𝐼𝐶 = 𝑐𝑜𝑟𝑟 (𝑜𝑟𝑑𝑒𝑟 𝑡 −1, 𝑜𝑟𝑑𝑒𝑟 𝑟𝑡 ),

(14)

𝑠

where 𝑘 (·, ·) is the kernel function such as RBF kernel and linear
kernel, and 𝑛𝑠 = |ℎ𝑠 |, 𝑛𝑡 = |ℎ𝑡 | are the number of the data from two
distributions, respectively.
Deep Correlation Alignment (CORAL). The CORAL distance[42] is defined as the distance of the second-order statistic (covariance) of the samples from two distributions:1𝑑𝑐𝑜𝑟𝑎𝑙 (h𝑠 , h𝑡 ) = 2 ||C𝑠 − C𝑡 || 2𝐹 ,(10)
4𝑞
where 𝑞 is the dimension of the features, and C𝑠 and C𝑡 are thecovariance matrices of the two distributions, respectively.
Domain Adversarial Discrepancy. The domain discrepancy
can be parameterized as a neural network and it is referred to as thedomain adversarial discrepancy [12]. An additional network named

where 𝑜𝑟𝑑𝑒𝑟 𝑡 −1 is the factor rank of each stock in period 𝑡 − 1 and𝑜𝑟𝑑𝑒𝑟 𝑟𝑡 is the ranking of each stock’s return rate in period 𝑡.
The information ratio (IR) of an actively managed portfolio isdefined as the portfolio’s mean return over a benchmark portfolio
return divided by the tracking error (standard deviation of theexcess return), which is defined as√
𝐼𝑅 = 𝐼𝐶 × 𝐵𝑅,(15)𝑓where BR is breadth or number of independent information sources.
We also call IR the ICIR. Based on IC and ICIR, we further use
RankIC and RankICIR to evaluate the robustness of models against
permuations, and use RawIC and RawICIR to denote the performance of unnormalized data. To sum up, the higher they are, thebetter the model is.

CIKM ’21, November 1–5, 2021, Virtual Event, Australia.

REFERENCES[1] Bandar Almaslukh, Jalal AlMuhtadi, and Abdelmonim Artoli. 2017. An effective deep autoencoder approach for online smartphone-based human activity
recognition. Int. J. Comput. Sci. Netw. Secur 17, 4 (2017), 160–165.
[2] Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. 2018. Metareg:
Towards domain generalization using meta-regularization. In NeurIPS. 998–1008.
[3] Karsten M Borgwardt, Arthur Gretton, Malte J Rasch, Hans-Peter Kriegel, Bernhard Schölkopf, and Alex J Smola. 2006. Integrating structured biological data bykernel maximum mean discrepancy. Bioinformatics 22, 14 (2006), e49–e57.
[4] Lingzhen Chen and Alessandro Moschitti. 2019. Transfer learning for sequencelabeling using source model and target data. In AAAI, Vol. 33. 6260–6267.
[5] Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy
Schuetz, and Walter Stewart. 2016. Retain: An interpretable predictive model for
healthcare using reverse time attention mechanism. In NeurIPS. 3504–3512.
[6] Fu-Lai Chung, Tak-Chung Fu, Vincent Ng, and Robert WP Luk. 2004. An evolutionary approach to pattern-based time series segmentation. IEEE transactions
on evolutionary computation 8, 5 (2004), 471–489.
[7] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014.
Empirical evaluation of gated recurrent neural networks on sequence modeling.
arXiv preprint arXiv:1412.3555 (2014).
[8] Wanyun Cui, Guangyu Zheng, Zhiqiang Shen, S. Jiang, and Wei Wang. 2019.
Transfer Learning for Sequences via Learning to Collocate. In ICLR.
[9] Emmanuel de Bézenac, Syama Sundar Rangapuram, Konstantinos Benidis,
Michael Bohlke-Schneider, Richard Kurle, Lorenzo Stella, Hilaf Hasson, Patrick
Gallinari, and Tim Januschowski. 2020. Normalizing Kalman Filters for Multivariate Time Series Analysis. NeurIPS 33 (2020).
[10] Facebook. Accessed in Jan 2021. FBProphet. https://facebook.github.io/prophet/.
[11] Hassan Ismail Fawaz, G. Forestier, Jonathan Weber, L. Idoumghar, and PierreAlain Muller. 2018. Transfer learning for time series classification. Big Data
(2018), 1367–1376.
[12] Yaroslav Ganin, E. Ustinova, Hana Ajakan, P. Germain, H. Larochelle, F. Laviolette,
M. Marchand, and V. Lempitsky. 2016. Domain-Adversarial Training of Neural
Networks. JMLR 17 (2016), 59:1–59:35.
[13] Tomasz Górecki and Maciej Luczak. 2015. Multivariate time series classification
with parametric derivative dynamic time warping. Expert Syst. Appl. 42 (2015),
2305–2312.
[14] A. Gretton, Bharath K. Sriperumbudur, D. Sejdinovic, Heiko Strathmann, S. Balakrishnan, M. Pontil, and K. Fukumizu. 2012. Optimal kernel choice for large-scale
two-sample tests. In NeurIPS.
[15] Priyanka Gupta, P. Malhotra, L. Vig, and G. Shroff. 2018. Transfer Learning
for Clinical Time Series Analysis using Recurrent Neural Networks. In ML for
Medicine and Healthcare Workshop at KDD.
[16] David Hallac, Sagar Vare, Stephen Boyd, and Jure Leskovec. 2017. Toeplitz inverse
covariance-based clustering of multivariate time series data. In SIGKDD. 215–223.
[17] Edwin T Jaynes. 1982. On the rationale of maximum-entropy methods. Proc.
IEEE 70, 9 (1982), 939–952.
[18] Kaggle. Accessed in Jan 2021. Household electric power consumption dataset.
https://www.kaggle.com/uciml/electric-power-consumption-data-set.
[19] Konstantinos Kalpakis, Dhiral Gada, and Vasundhara Puttagunta. 2001. Distancemeasures for effective clustering of ARIMA time-series. In ICDM. IEEE, 273–280.
[20] Eamonn Keogh and Chotirat Ann Ratanamahatana. 2005. Exact indexing ofdynamic time warping. Knowl. Inf. Syst. 7, 3 (2005), 358–386.
[21] Kazuhiro Kohara, Tsutomu Ishikawa, Yoshimi Fukuhara, and Yukihiro Nakamura.
1997. Stock price prediction using prior knowledge and neural networks. Intell.
Syst. Account. Finance Manag. 6, 1 (1997), 11–22.
[22] Vitaly Kuznetsov and Mehryar Mohri. 2014. Generalization bounds for timeseries prediction with non-stationary processes. In ALT. Springer, 260–274.
[23] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. 2018. Modelinglong-and short-term temporal patterns with deep neural networks. In SIGIR.
95–104.
[24] Vincent Le Guen and Nicolas Thome. 2020. Probabilistic Time Series Forecastingwith Structured Shape and Temporal Diversity. In Advances in Neural Information
Processing Systems.
[25] Qi Lei, Jinfeng Yi, Roman Vaculín, Lingfei Wu, and Inderjit S. Dhillon. 2019.
Similarity Preserving Representation Learning for Time Series Clustering. In
IJCAI.
[26] Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. 2018. Domain generalization with adversarial feature learning. In CVPR. 5400–5409.
[27] J. Lines and Anthony J. Bagnall. 2014. Time series classification with ensemblesof elastic distance measures. Data Min. Knowl. Discov. 29 (2014), 565–592.
[28] Xiaoyan Liu, Zhenjiang Lin, and Huaiqing Wang. 2008. Novel online methods
for time series segmentation. IEEE TKDE 20, 12 (2008), 1616–1626.
[29] Yasuko Matsubara, Yasushi Sakurai, Willem G Van Panhuis, and Christos Faloutsos. 2014. FUNNEL: automatic mining of spatially coevolving epidemics. In KDD.
105–114.
[30] Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. 2013. Domain
generalization via invariant feature representation. In ICML. 10–18.

Du, et al.

[31] Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. 2021.
Meta-learning framework with applications to zero-shot time-series forecasting.
In AAAI.
[32] C. Orsenigo and C. Vercellis. 2010. Combining discrete SVM and fixed cardinality
warping distances for multivariate time series classification. Pattern Recognit. 43(2010), 3787–3794.
[33] Yao Qin, Dongjin Song, Haifeng Chen, Wei Cheng, Guofei Jiang, and Garrison
Cottrell. 2017. A dual-stage attention-based recurrent neural network for timeseries prediction. In AAAI.
[34] Goce Ristanoski, Wei Liu, and James Bailey. 2013. Time series forecasting using
distribution enhanced linear regression. In PAKDD. 484–495.
[35] Joshua W Robinson, Alexander J Hartemink, and Zoubin Ghahramani. 2010.
Learning Non-Stationary Dynamic Bayesian Networks. Journal of Machine
Learning Research 11, 12 (2010).
[36] Sheldon M Ross. 2014. Introduction to stochastic dynamic programming. Academicpress.
[37] David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. 2020.
DeepAR: Probabilistic forecasting with autoregressive recurrent networks. Int. J.
Forecast 36, 3 (2020), 1181–1191.
[38] P. Schäfer. 2015. Scalable time series classification. Data Min. Knowl. Discov. 30(2015), 1273–1298.
[39] Robert E Schapire. 2003. The boosting approach to machine learning: An overview.
Nonlinear estimation and classification (2003), 149–171.
[40] Rajat Sen, Hsiang-Fu Yu, and Inderjit S Dhillon. 2019. Think globally, act locally:
A deep neural network approach to high-dimensional time series forecasting. In
NeurIPS. 4837–4846.
[41] Hidetoshi Shimodaira. 2000. Improving predictive inference under covariate
shift by weighting the log-likelihood function. Journal of statistical planning andinference 90, 2 (2000), 227–244.
[42] Baochen Sun and Kate Saenko. 2016. Deep CORAL: Correlation Alignment for
Deep Domain Adaptation. In ECCV.
[43] Kerem Sinan Tuncel and Mustafa Gokce Baydogan. 2018. Autoregressive forests
for multivariate time series modeling. Pattern recognition 73 (2018), 202–215.
[44] E. Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. 2017. Adversarial
Discriminative Domain Adaptation. In CVPR. 2962–2971.
[45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all
you need. In Advances in neural information processing systems. 5998–6008.
[46] LE Vincent and Nicolas Thome. 2019. Shape and Time Distortion Loss for Training
Deep Time Series Forecasting Models. In NeurIPS. 4189–4201.
[47] Jindong Wang, Yiqiang Chen, Wenjie Feng, Han Yu, Meiyu Huang, and Qiang
Yang. 2020. Transfer learning with dynamic distribution adaptation. ACM
Transactions on Intelligent Systems and Technology (TIST) 11, 1 (2020), 1–25.
[48] Jindong Wang, Wenjie Feng, Yiqiang Chen, Han Yu, Meiyu Huang, and Philip S Yu.
2018. Visual domain adaptation with manifold embedded distribution alignment.
In Proceedings of the 26th ACM international conference on Multimedia. 402–410.
[49] Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Wenjun Zeng, and Tao
Qin. 2021. Generalizing to Unseen Domains: A Survey on Domain Generalization.
In International Joint Conference on Artificial Intelligence (IJCAI).
[50] Zhiguang Wang and Tim Oates. 2015. Imaging Time-Series to Improve Classification and Imputation. In IJCAI.
[51] Z. Yang, R. Salakhutdinov, and William W. Cohen. 2017. Transfer Learning for
Sequence Tagging with Hierarchical Recurrent Networks. In ICLR.
[52] Chaohui Yu, Jindong Wang, Yiqiang Chen, and Meiyu Huang. 2019. Transfer
learning with dynamic adversarial adaptation network. In 2019 IEEE International
Conference on Data Mining (ICDM). IEEE, 778–786.
[53] G Peter Zhang. 2003. Time series forecasting using a hybrid ARIMA and neuralnetwork model. Neurocomputing 50 (2003), 159–175.
[54] Shuyi Zhang, Bin Guo, Anlan Dong, Jing He, Ziping Xu, and Song Xi Chen. 2017.
Cautionary tales on air-quality improvement in Beijing. Proc. Math. Phys. Eng.
Sci. 473, 2205 (2017), 20170457.
[55] Yunyue Zhu and Dennis Shasha. 2002. Statstream: Statistical monitoring of
thousands of data streams in real time. In VLDB. Elsevier, 358–369.
[56] Yongchun Zhu, Fuzhen Zhuang, Jindong Wang, Guolin Ke, Jingwu Chen, Jiang
Bian, Hui Xiong, and Qing He. 2020. Deep subdomain adaptation network forimage classification. IEEE transactions on neural networks and learning systems
32, 4 (2020), 1713–1722.