Feature-aligned N-BEATS with Sinkhorn divergence
Authors: Joonhun Lee
Myeongho Jeon
Myungjoo Kang
Kyunghyun Park
Venue: ICLR 2024 spotlight (2024)
Keywords: Time series forecasting
Deep learning
Domain generalization
Representation learning
Sinkhorn divergence
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose Feature-aligned N-BEATS with Sinkhorn divergence, a domain generalization model for time series forecasting within representation learning.

Abstract: We propose Feature-aligned N-BEATS as a domain-generalized time series forecasting model. It is a nontrivial extension of N-BEATS with doubly residual stacking principle (Oreshkin et al. [45]) into a representation learning framework. In particular, it revolves around marginal feature probability measures induced by the intricate composition of residual and feature extracting operators of N-BEATS in each stack and aligns them stack-wise via an approximate of an optimal transport distance referred to as the Sinkhorn divergence. The training loss consists of an empirical risk minimization from multiple source domains, i.e., forecasting loss, and an alignment loss calculated with the Sinkhorn divergence, which allows the model to learn invariant features stack-wise across multiple source data sequences while retaining N-BEATS’s interpretable design and forecasting power. Comprehensive experimental evaluations with ablation studies are provided and the corresponding results demonstrate the proposed model’s forecasting and generalization capabilities.

Published as a conference paper at ICLR 2024
FEATURE-ALIGNED N-BEATS WITH SINKHORN DI- VERGENCE
Joonhun Lee", Myyongho Jeon" & Myungjoo Kangl Seoul National University
Kyunghyun Park
Nanyang Technological University kyunghyun- pa par.parkkkkeee rk@nku.eeee edu..eduu.......
i ceguy71 18, andyjeon, mamo:/jsoommmaaeaannn..
ABSTRACT
We propose Feature-aligned N-BEATS as a domain-generalizee time series fore- casting model. It is nontrivial extension of N-BEATS with doubly residual stack- ing principle (Oreshkin al. [45]) into representation learning framework. particular, it revolves around marginal feature probability measures induced by the intricate composition of residual and feature extracting operators N-BEATS in each stack and aligns them stack-wise via an approximate of optimal trans- port distance referred to as the Sinkhorn divergence. The training loss consists of an empirical risk minimization from multiple source domains, i.c., forecasting loss, and an alignment loss calculated with the Sinkhorn divergence, which allows the model learn invariant features stack-wise across multiple source data se- quences while retaining N-BEATS's interpretable design and forecasting power. Comprehensive experimental evaluations with ablation studies are provided and the corresponding results demonstrate the proposed model's 's forecasting and gen- eralization capabilities.
INTRODUCTION
Machine learning models typically presume that the loss minimization from training data results in reasonable performance on target environment, i.c., empirical risk minimization [56]. However, when using such models in the real world, the target environment is likely to deviate from the training data, which poses a significant challenge for well-adaptive model to the target environment. This related to the concept of domain shift [49].
A substantial body of research has been dedicated to developing frameworks that can accommodate the domain shift issue [6. 7, 20]. In particular, classification tasks have been the predominant focus [30, 32, 58, 59, 66]. As an integral way for modeling sequential data in broad domains such as finance, operation research, climate modeling, and biostatistics, time series forecasting has been a big part of machine learning fields. Nevertheless, the potential domain shift issue for common forecasting tasks has not been considered intensively compared to classification tasks, but only a few articles addressing this can be named [25, 26).
The goal of this article is to propose a resolution for the domain shift issue within time series fore- casting tasks, namely a lomain-generalized time series forecasting model. In particular, the pro- posed model is built upon deep learning model which is N-BEATS [45, 46], and representation learning toolkit which is the feature alignment. N-BEATS revolves around doubly residual stack- ing principle and enhances the forecasting capabilities of multilayer perceptron (MLP) architectures without resorting to any traditional machine learning methods. On the other hand, it is well-known that aligning marginal feature measures enables machine learning models to capture invariant fea- tures across distinctive domains [8]. Indeed, in the context of classification tasks, many references [30, 38, 42, 65] demonstrated that the feature alignment mitigates the domain shift issue.
It is important to highlight that the model is not straightforward combination of the established components but a nontrivial extension that poses several challenges. First, N-BEATS does not allow the feature alignment in 'one-shot" unlike the aforementioned references. This is because it is hierarchical multi stackisgtacing architecture in which each stack consists of several blocks and con- nected to each other by residual operations and feature extractions. In response to this, we devise the
"quual contribution
1 co-corresponding authors

---

Published as a conference paper at ICLR 2024
stack-wise alignment that is minimization of divergences between marginal feature measures a stack-wise basis. The stack-wise aligmment enables the model to learn feature invariance with an ideal frequency of propagation. Indeed, instead of aligning every block for each stack, single align- ment for each stack mitigates gradient vanishing/exploding issue [47] via sparsely propagating loss while preserving the interpretabillty of N-BEATS and ample semantic coverage [45, Section 3.3].
Second, the stack-wise alignment demands an efficient and accurate method for measuring diver- gence between measures. Indeed, the alignment is inspired by the error analysis of general domain generalization models given in [1, Theorem 1]. in which empirical risk minimization loss and pair- wise H-divergence loss between marginal feature measures are the trainable components among the total error terms without any target domain information. In particular, since the feature align- ment requires the calculation of pairwise divergences for multiple stacks (due to the doubly residual stacking principle), the computational load steeply increases as either the number of source domains or that of stacks increases. On the other hand, from the perspective of accuracy and efficiency, the H-divergence is notoriously challenging to be used in practice [6, 28, 31, 54].
For suitable toolkit, we adopt the Sinkhorn divergence which is efficient approximation for the classic optimal transport distances [17, 21, 50J. This choice is motivated by the substantial theoreti- cal evidences of optimal transport distances. Indeed, in the adversarial framework, optimal transport distances have been essential for theoretical evidences and calculation of divergences between push- forward measures induced by a generator and a target measure [21, 53, 62, 66]. In particular, the computational efficiency of the Sinkhorn divergence and fluent theoretical results by [13, 14, 17, 21] are crucial for our choice among other optimal transport distances. Thereby, the training objective is to minimize the empirical risk and the stack-wise Sinkhorn divergences (Section 3.3).
Contributions. To provide an informative procedure of stack-wise feature alignment, we introduce a concrete mathematical formulation of N-BEATS (Section 2), which enables to define the pushfor- ward feature measures induced by the intricate residual operations and the feature extractions for each stack (Section 3.1). From this, we make use of theoretical properties of optimal transport prob- lems to show a representation learning bound for the stack-wise feature alignment with the Sinkhorn divergence (Theorem 3.6). which justifies the feasibility of Feature-aligned N-BEATS. To embrace comprehensive domain generalization scenarios, we use real-world data and evaluate the proposed method under three distinct protocols based the domain shift degree. We show that the model consistently outperforms other forecasting models. In particular, our method exhibits outstanding generalization capabilities under severe domain shift cases (Table 1). We further conduct ablation studies to support the choice of the Sinkhorn divergence in our model (Table 2).
Related literature. For time series forecasting, deep learning architectures including recurrent neu- ral networks [4, 9, 24, 51, 52] and convolutional neural networks [11, 34] have achieved significant progress. Recently, a prominent shift has been observed towards transformer architectures leverag- ing self-attention mechanisms [33, 35, 60, 61, 67, 68]. Despite their innovations, concerns have been raised regarding the inherent permutation invariance in self-attention, which potentially leads to the loss of temporal information [63]. On the other hand, [10, 45] empirically show that MLP-based architectures would mitigate such disadvantage and even surpass the transformer-based models.
Regarding the domain shifts for time series modeling, [25] proposed a technique that selects samples from source domains resembling the target domain, and employs regularization to encourage learn- ing domain invariance. [26] designed a shared attention module paired with a domain discriminator to capture domain invariance. [46] explored domain generalization from a meta-learning perspec- tive without the information on the target domain. Nonetheless, an explicit toolkit and concrete formulation for domain generalization are not considered therein.
The remainder of the article is organized as follows. In Section 2, we set the domain generalization problem in the context of time series forecasting, review the doubly residual stacking architecture of N-BEATS, and introduce the error analysis for domain generalization models. Section 3 is de- voted defining the marginal feature measures inspiring the stack-wise alignment, introducing the Sinkhorn divergence together with the corresponding representation learning bound, and presenting the training objective with the corresponding algorithm. In particular, Figure 1 therein illustrates the overall architecture of Feature-aligned N-BEATS. In Section 4, comprehensive experimental evalu- ations are provided. Section 5 concludes the paper. Other technical descriptions, visualized results, and ablation studies are given in Appendix.
2

---

Published as a conference paper at ICLR 2024
2 BACKGROUND
Notations. Let X == R and y := IR be the input and output spaces, respectively, where a and denote the lookback and forecast horizons, respectively. Let 2 = IR" be the latent space with y representing the feature dimension. We further denote by Σ Z a subspace of Z. All the aforementioned spaces are equipped with the Euclidean norm ||-Il- Define by P := P(X x V) the set of all Borel joint probability measures on X' x y. For any TP E P, denotes by Px and Py corresponding marginal probability measures on X' and Y respectively. We further define by P(X)
and P(Z) () the of all Borel probability measures X' and 2, respectively.
Domain generalization in time series forecasting, There are multiple source domains {D*} with 2 2 and target (unseen) domain D Assume that each D is equipped with go E P and the same holds for D" with pp" E P and that sequential data for each domain are sampled from corresponding joint distribution. Let I y x y + R be a loss function. Then, the objective is to derive prediction model 3 X ) such that (($)-a+1,,"" ..) "141," for ($t-a+1,'.. .B) ($1+1,-... S(+0) ~ p",, by leveraging [PO] Jk=1, i.c.
K
inf ((8), with C(3) = 8
Σ
E (ax))~ [(3()).))]
(2.1)
K
Doubly residual stacking architecture. The main architecture of N-BEATS equipped with the doubly residual stacking principle from [10, 45] is summarized as follows: for M, L E N, the model comprises M stacks, with each stack consisting of blocks. The blocks share the same model weight within each respective stack and recurrently operated based on the double residual stacking principle. More precisely, an m-th stack derives the principle in a way that for Im,11 E X,
Um E(E 1hth (zm.))) Im,l Fmnd-1 (5T") a) (Zm,11)), 1= ...... L, (2.2)
where vith : X + Z extracts features (zm.)) E Z from the inputs Tm,l E X for each layer I, and (ET", ktm) Z + y x X generates both forecasts (4T" o (zm,) E y and backcasts (5T" (Im..) E X branches. Note that im E Y' represents the m-th forecast obtained through the hierarchical aggregation of each block's 's forecast, and that the last backeast Lm.L E X, derived by residual sequence from blocks, serves as an input for the next stack, except for the case m M.
Once the hierarchical aggregation of all stacks and the residual operations are completed, the model for the doubly residual stacking architecture is given as follows: for (x., v) PTT and 21,1 = x,
M
y R B(:; V E, F = Σ Dms
Lm,1 == Im-1, m m=a 2,..., ...,,,
(2.3)
m=11
subject to Im and mm given in (2.2), where
V 7 m=
E = (ETH) M Imml: 5y== (5Th y
Im=ls
(2.4)
implemented by fully connected layers. For further details, refer to Appendix A.
Domain-invariant feature representation. After the investigation on the error analysis for domain adaptation models by [64], an extended version for domain generalization models is provided by [1]. This provides us an insight for developing a domain generalization toolkit within the context of doubly residual stacking models.
In the following, we restate Theorem in [1]. To that end, we introduce some notations. Let H be the set of hypothesis functions h X - [0 and let H == {sgn( [(-) - n²(-)) - f): h, h e H. t E [o, 1]}. The H-divergence is defined by (F (P,) == 2 suphER Exer, [(((a)=1)]] Ex-ry [1 (x(())=))] for any Px E P(X). The H-divergence dx(:; is defined analogously, with H. replaced by H. Furthermore, denote by (+) H. + R and RT((), H + R the expected risk under the source measures pk l 1, I,..., K, and the target measure IPT respectively.
3

---

Published as a conference paper at ICLR 2024
Proposition 2.1. Let AK be a (K-I)-dimensoonal simplex such that each component π repre- sents convex weight. Set A == {E n;P Ak} and let T arg miggiiee EN dr (P X Px). Then, the following holds: for any H.
7i Ex= Ex=1
R² (1) E = R (h) dx (P2,, Px) +
dz x Px) + ((pT.Pz):
max
i.jE (l....).. inj
(B Ex ( (x)),, Benzy E Tt. - ft (x)])} ...,, , K, denotes a true fubelingfuncttin under PP, i.e, y - (x) for (a, y) ~ PPt and similarly 5 denotes true labeling function under pl,
with A(rT == min BT where k = -=,,....
While the upper bound of (-) consists of four terms, only the first and third terms (representing the source risks {?){_ and the pairwise divergences (dz((xx Px) across all marginal
feature measures, respectively) are learnable without the target domain information.
METHOD
3.1
MARGINAL FEATURE MEASURES
Aligning marginal feature measures is a predominant approach in domain-invariant representation defined as push- forward measures induced by given feature map g X Z from { 1-x i.e, 9# P& (E) x gg (FE) for any Borel set E in Z.
learning [20, 55]. In particular, the marginal feature measures {9 TPk 1K
However, defining such measures for doubly residual architectures poses some challenges. Indeed, IM as discussed in Section 2, N-BEATS includes multiple feature extractors V/ = {viT Jm== as defined in (2.2), where each extractor t takes a sampled input passing through multiple residual operations of previous stacks and the input is recurrently processed within each stack by the residual operations B, and Ey. The scaling factor represents Nomain-speciffe characteristics that exhibit noticeable vari- ations. This can lead to an excessive focus scale adjustments in the aligning process, potentially neglecting crucial features, such as seasonality or trend.
To resolve these difficulties, propose stack-wise alignment of feature measures on subspace 2 C 2. This involves defining measures for each stack through the compositions of feature extrac- tions in V = {15Tm) M backcasting operators in E ( j given in (2.2), and a normaliza- tion function.
Definition 3.1. Let a: Z -
z be a normalizing function satisfying C, -ipLishciizz continuity, i.c., [z)()) - a22))l < Co - z'|l for z, E Z. Given vhth : + Z defined in (2.2), the operators pm X' and gdh X > Z are defined as:
m (x) = ( v/
(3.1)
g Th (x) = (e o (1 o (- (1 o ..... (r))(1))))))
(3.2)
where (z) denotes L-times composition of rm with (rmT)( = x for L - 1 = 0 and g"h = (1iTh a (rm)((-11) for m = 1. Then the set of marginal feature measures in the m-th stack, m 1, M, is defined by
{(o g0 + P
where each (a g is pushforward of P E (K induced by o gr :X' x
Remark 3.2. The normalization function o helps the model to learn invariant features by mitigating the influence of the scale information of each domain. Furthermore, the Lipschitz condition on prevents gradient explosion during model updates. There are two representatives for a: (i) softmax:
Z - = (0, 1)² where softmax(2), (2); = = / et = 1.... y; (ii) tanh: 2 - 2 = 1-1,
where tanh(2), (z), 2e - () + 1), j = 7. Both are 1-Lipschitz continuous, i.e., C. In Appendix G (see Table 9), we provide the ablation study under these functions, in addition to the case without the normalization.
Remark 3.3. Embedding feature alignment 'block-wise' for every stack results in recurrent op- erations within each stack and redundant gradient flows. This redundancy cause exploding or
4

---

Published as a conference paper at ICLR 2024
vanishing gradients for long-term forecasting [47]. Our stack-wiss feature alignment addresses these problems by sparsely propagating the loss. It also maintains ample alignment coverage related to se- mantics since the stack serves as a semantic extraction unit in [45]. Further heuristic demonstration is provided in Appendix G.I.
The operator gd in (3.2) accumulates features up to the m-th stack accounting for the previous m m-1 residual operations. Despite the complex composition of and E1, the fully connected layers in them exhibit L. Lipschitz continuity [57, Section 6], which ensures the Lipschitz continuity of g" From this observation and Remark 3.2, we state the lemma below, with its proof in Appendix B:
Lemma 3.4. Let Co > 0 be given in Definition 3.1. Denote for m = 1,.... M by Cm > 0 and Cm.t > O the Lipschitz constants of ttth and 5f: respectively. Then (or o gmm) is Coogm Lipschitz continuous with
Congmn Cm + 1 (1 C, for m m= 22 M,
and Coogm = Cm + Cm, for ===1.
By the doubly residual principle, {g LM are inseparable for V and Ey. However, the stack- wise alignment via regularizing {g" Jm=1 potentially deteriorates the backcasting power of E₁ which could lead to performance degradation of the model. Instead, regularizing exclusively on feature extractors V. More precisely, this alignment of marginal feature measures from Definition 3.1 is defined follows: given E (5T JM
conduct the alignment by
1m-1
M
inf
max m=1 JeLL, K], iff ((00g"))PP) (o
Σ
(3.3)
where d(::)) P(2) P( - R4 a divergence or distance between given measures. The illustration of the stack-wise alignment is provided in Figure 3 (in Appendix A).
Note that the third term in Proposition 2.1, i.e., ma..,jE(1......... (I)...), iØ dæ (Pi Px)), and the stack-wise alignment in (3.3) are perfectly matched once -( is specified as the H-divergence. However, the empirical estimation for the H-divergence is notoriously difficult [6, 32, 54). These concerns become even more pronounced in the proposed method due to the stack-wise alignment necessitat- ing M (K- 1)/2-times calculation of pairwise divergence, implying heavy computational load. Meanwhile, substantial body of literature regarding the domain invariant feature learning adopts other alternatives for the H-divergence, and among them [28, 29, 53, 66], optimal transport distances have been dominant due to their in-depth theoretical ground. In line with this, in the following sec- tion, we introduce an optimal transport distance as a relevant choice for d(.:.).
3.2 SINKHORN DIVERGENCE ON MEASURES
In the adversarial framework [21. 53, 62, 66]. optimal transport distances have been adopted for training generators to make corresponding pushforward measures close to given target measure. In particular, the Sinkhorn divergence, an approximate of an entropic regularized optimal transport distance, is shown to be an efficient method address intensive calculations of divergence between empirical measures [13, 17, 21J. As the stack-wise alignment given in (3.3) leverages on a number of calculations of divergences and hence requires an efficient and accurate toolkit for feasible training, we adopt the Sinkhorn divergence as the relevant one for d(.,-).
To define the Sinkhorn divergence, let us introduce the regularized quadratic Wasserstein-2 distance. To that end, let be the entropic regularization degree and IH,, vi 2) the space of all couplings, ie., transportation plans, the marginals of which are respectively μ, E P(ž). Then the regularized quadratic Wasserstein-2 distance defined on 2 is defined as follows: for 2 O.
W (H,) == inf U (Ix- y/l² + e elo x;)
(
dr(x,y)
)) di (1,8)
(3.4)
(du(z)dv(z)
By replacing Ž with X, one can define by We,a the corresponding regularized distance on X.
The entropic term attached with in (3.4) is known to improve computational stability of the Wasserstein-2 distance, whereas it causes a bias on corresponding estimator. To alleviate this, ac- cording to [12], we adopt the following debiased version of the regularized distance:
5

---

Published as a conference paper at ICLR 2024
Definition 3.5. For é 2 0, the Sinkhorn divergence is
We,e(14 v) = W. N,, 2 (W.
+W,,)) 14, v E P(ž).
(3.5)
Using the duality the regularized optimal transport distance from [48, Remark 18 in Section 4.4] and the Lipschitz continuity of {ar " from Lemma 3.4, we present the following theorem, substantiating the well-definedness and feasibility of our stack-wise alignment W VV,)) The
Jm=1
proof is provided in Appendix B.
Theorem 3.6. Let Crogm > O be as in Lemma 3.4 and define C == Em max ) Then the following holds: for 0,
M
Σ
W ( o 9Th (o))))))+/ Px SC
We, (Pxx, Pz)
max (JE(1, m=11
max A.EEL, .Y, it)
K it)
In [44, Lemma 3 & Proposition 6], representation learning bounds under the maximum mean dis- crepancy and the regularized distance in (3.4) are investigated for single-layered fully connected network. With similar motivation, Theorem 3.6 represents a learning bound for the stack-wise align- ment loss under the Sinkhorn divergence as the entropic regularized distance between source do- mains' measures. While the Lipschitz continuity {ar o g" ["== allows nice bound, there exists room for having tighter bound by deriving the smallest Lipschitz constant [57] and applying the spectral normalization [40], which will be left for the future extension. Further discussions on the choice of the Sinkhorn divergence and Theorem 3.6 are provided in Appendix C.
3.3 TRAINING OBJECTIVE AND ALGORITHM
From Sections 3.1 and 3.2, we define the training objective and corresponding algorithm. To that end, denote by Φ {dm 1m O := mm, JMM and O {6.m. 1m the parameters sets of the fully connected neural networks in the residual operators in V E.. and E, given (2.4). Then corresponding parameterized forms of the operators are given by
M
(D) fv)T (-; (m)) M (9,) {5[" (:: (m. }
ET(O)) = {ef" (:: (m,,t))) M
Then denote by g == g" (:3 (dn H (0n, J 1). m m= I,..., M, the parameterized version of 9thh given in (3.2). Let L( :))) be the parameterized form of the forecasting loss given in (2.1) and (-) be that of the aligmment loss given in (3.3) under the Sinkhorn divergence, i.c..
M
Latign 00, = Σ
max (J K).
W. Wez((o o ( (a o De Px).
(3.6)
ma
We then provide the following training objective La(Ø, 04, O+) L(3( O2, 0+)) A Oy). (3.7)
Stack St.
To update (Φ. 0₁ θ₁) according to (3.7), we calculate m-th stack divergence W z ((oog gh , p x (oog ) Px)) as its empirical (f( m.(i),
Stukk
Um
counterpart W (ug 1 e M,,,,) where the corresponding empir- (l) (K) KK
Sackk
ical measures (1 e, ===1 are given as follow: for l = 1, K,
P4.00
(.)k)
Σ
6 6,0
with (k) = o
on (K)
diveggeee
B
Bo with stack Gerces
1=1
()))) are sampled from Dk and B and 8g de- note a mini-batch size and the Dirac measure centered on z E ž, respectively.
where ((1)0)
(k)
Figure Illustration of
Feature-aligned N-BEATS.

---

Published as a conference paper at ICLR 2024
As mentioned in Section 3.1, the alignment loss Lalign (Ø, ①+) is minimized by updating + for given O, while {9H,e, I==1 are inseparable for & and O, At the same time, the forecasting loss ( 04, +4)) is minimized by updating (Φ, 1 θ To bring them together, we adopt the following alternatively updating optimization inspired from [19, Section 3.1]:
i t = arg min ((3(2,, .4, O+)), 4- := arg min LA (1, 01, 07). ,,
(3.8)
The training procedure on (3.8) is summarized in Algorithm 1 and the overall model architecture is illustrated Figure where we highlight the stack-wist alignment process (with 'red' color) not appearing in the original N-BEATS (see Figure 1 in [45]).
Algorithm 1: Training Feature-aligned N-BEATS.
Requires: yy (learning rate), B (mini-batch size); Initialize D, O₁ O,; while not converged do
(k) (K) ) from D & Initialize ] c- k= L. for 1 to M do
Sample {((),
for 1 to K do
Compute e (3% (z) ) Update 0% -/ + (9 (1 (o( fm..)) b 1.... end end
(k) Compute (Ma,9, Jm=1, h = I...., K: Update & such that for I,.... M.
@m On + em
M (>
max
W
We (Pote Ho, (
n=1 (JE(L.... -),, 18s
10
Update (0, Us, H4) such that m 1,...
(om, o,, 0m..t) &- (0m, 1 0m,,)
K Rm
8
V Om (1bh)) )
12 end
4 EXPERIMENTS
Evaluation details. Our evaluation protocol lies on two principles: (i) real-world scenarios and (ii) examination of various domain shift environments between the source and target domains. For (i), we use financial data from the Federal Reserve Economic Data (FRED)* and weather data from the National Centers for Environmental Information (NCEI)? For (ii), let define set of semantically similar domains as superdomain denoted by A,, c.g., FRED, NCEI. We then categorize the domain shift scenarios into out-domain generalization (ODG), drosi-domain generalization (CDG), and in-domain generalization (IDG) such that
ODG: {p A.
Shin (KZ1), DTd Aş;
CDG: (D* A.. (D*J S A (it5) D. E S. (D*) = =0:
IDG: (D*) A:
(1=1), DJ E S. n =
DD==
The domain shift from source target becomes increasingly pronounced in the sequence of IDG, CDG, and ODG, making it even more challenging to generalize. For detailed data configuration and domain specifications, refer to Appendix D.
Benchmarks. We compare our proposed approach with deep learning-bast baased models, including transformer (e.g.. Informer [67], Autoformer [61]), MLP-based models (e.g., LTSF-Linear models [63] with NLinear and Dlinear) and N-BEATS based models (e.g., N-BEATS [45] and N-HiTS [10])- Note that since the aforementioned time series models addressing domain shift [25, 26] still requires target domain data (due to their domain-adapted" framework), we do not consider their models into our cammain-gennralizeed protocol.
Experimental details. We adopt the symmetric mean absolute percentage error (SMAPE) for C(3(s-))) given in (3.7) and use the sofumax function for a given in Definition 3.1. The Sinkhorn
https: fred. sulliiseeddor
https: /tt.//ncii.///////////// noaa. nova.gov

---

Published as a conference paper at ICLR 2024
Domain generalization performance. The performance across all combinations of each ODG, CDG, and IDG scenario is provided (as the average of scenarios for each FRED and NCEI). The detailed description for N-BEATS-G and N-BEATS- is provided in Appendix A. The notation + FA' stands for feature alignment. Each evaluation is conducted three times, with different random seeds. Values over 10,000 are labeled as 'NA'. Runtime is measured for a single iteration.
Table 1:
Mothood
N-HITS
FA (Ours)
N-BEATS-I
+ FA (Oun)
N-BEATS-G + FA (Ours]
NLincar
DLincar
Autofommer
Informer
ODG
FRED
SMAPE
MASH
0.00
0.148
IJ157
0.134
0.899
0.232
0.05
0.214
0,061
0.172
0.1.0
0.059
48.150
0.176
2,214.48
0.307
0.570
NA
1.214
NA
0.723
0.713
0,814
0.724
0.722
0.718
1,112
1.302
1.293
1.630
MASE
0.561
0,512
0.754
2.737
33311
5.784
CDG
FRED
SMAPE
0.124
0.058
0.057
0.123
0.064
0,181
0.179
0,062
0.059
0.139
0.133
0.058
60,929
0.176
2.554.27
0.536
0.893
NA
1.143
NA
NCHI
SMAPE
MASE
0.581
0.718
0.482
0.731
0.822
0.755
0.608
0.763
0.582
2.734
2.787
3.233
4.147
IDG
FRED
KMAPE
0.119
0.059
0.115
0.062
0.137
0.064
0.136
0,0883
0.143
0.119
509.71
0.197
1,217.50
0.843
1.001
0.813
NA
NCEI
SMAPE
MASE
0.593
0.718
0,591
0.715
0.713
1.011
0.715
1.039
0.726
0.712
0.714
0,591
0.997
3.722
0.772
3.634
3.573
1.268
2,979
1.505
Runtine (soc)
0.26
0.80
0.32
0.97
0.16
0.68
0.04
0.05
0.58
0.50
divergence implemented by GeomLoss from [16] is utilized, and is set to be 0.0025. The Adam optimizer [27] is employed for implementing the optimization given in (3.8). The lookback horizon, forecast horizon, and the number source domains are set to be a = 50, B 10, and K 3, respectively (noting that it depends on the characteristics of source domains' datasets). Further- more, the number of stacks and blocks, and the dimension of feature space are set to be M = 3, I= 4, and y 512, respectively (noting that it is consistent with N-BEATS [45]). Others are deter- mined through grid search, and the SMAPE and MASE are adopted as evaluation metrics. Additional implementation details and definitions provided in Appendix D.
Domain generalization performance. As shown Table 1, the proposed stack-wise feature align- ment significantly improves the domain shift issue within the deep residual stacking architectures with outstanding performance compared to other benchmarks. In particular, we highlight that the improvement more significant in ODG where the domain shift from source to target is severely pronounced. That being said, the proposed comain-ggnneallin model can perform and adapt well in very severe situation without any information on the target environment. Other detailed analysis on the results are discussed in Appendix E.
Ablation study divergences. Table 2 provides the ablation results on the choice of divergence (or distances) for the proposed stack-wise feature alignment, in which the benchmarks consist of the classic (not regular- ized) Wasserstein-2 distance (WD), the maxi- mum mean discrepancy (MMD), and the Kull- back-Leibler divergence (KL) and further sen- sitivity analysis on the Sinkhorn divergence (SD) with respect to > O is also provided. Due to the heavy running cost for implement- ing WD cases (see Runtime with 314.30 in Table 2) and the training instability associated with KL cases (see Table 6), we consider the target domain case for 'exchange rate' (within FRED) and the several source domain scenarios for ODG, CDG and IDG (see Appendix D for the details on the source domains' combinations). For the same reasons, the baseline model is fixed to N-BEATS-G. The entire results are provided in Tables 6 and 7.
Table 2: Ablation study on divergences.
Divergences
WD
(D 0)
MMD
KL.
1c-5
2.5c-3
1c-1
0.031
0.040
0.032
0.033
0.035
0.045
MASR
0.022
0.059
0.022
0.022
0.055
0.057
CDG
SMAPE
0.028
0.058
0.026
0.040
0,028
0.027
0.029
0.039
0.030
IDG
SMAPE
0.024
0.024
0.024
0.025
0.025
0.026
0.049
0.050
0.050
0.050
0.051
0,049
Runtimo (sec)
314.30
0.68
0.81
0.53
As the Sinkhorn divergence is an accurate approximate of the Wasserstein-2 distance (see Definition 3.5), the similar results for the two cases in Table 2 seem to be reasonable. On the other hand, their computational costs are incomparable. That being said, the Sinkhorn divergence is the computation- ally feasible and accurate toolkit for the proposed stack-wise alignment with optimal transport based divergence, while some instability issue (see [5, 21I) would come out for extremely small > 0 (i.c., =le-5 in Table 2). In comparison with the MMD and KI. cases (see Tables 6 and well for the
8

---

Published as a conference paper at ICLR 2024
Block
Block
Block
Block 4
Stack
Stack
Stack 3
0 (w/o)
=
A-
Figure 2: Visualization invariant feature learning. In the aligned scenario (w), the interconnection between green and red instances, particularly at X = 3, becomes visible. Contrastingly, in the non- aligned scenario (w/o), we observe a pronounced dispersion, especially of the blue instances within the initial two stacks at À = 3, resulting in heightened inter-domain entropy.
entire results), the Sinkhorn divergence case and consistent results in overall domain shift scenarios. From these empirical evidences, we hence conclude that the choice of the Sinkhorn divergence allows the model bring both the abundant theoretical advantages of optimal transport problems and the practical feasibility.
be marginally better but shows stable
Visualization representation learning. To visualize representations, i.e., samples of marginal feature measures observed from N-BEATS-G with and without aligmment, we use the uniform mani- fold approximation and projection (UMAP) introduced by [39]. To minimize the effect of unaligned scale information, the softmax function is employed to remove the scale information and instead emphasize the semantic relationship across domains. As illustrated proximity between instances and the substantial upsurge in the entropy of domains. For other cases on N-BEATS-I and N-HiTS, please refer to Figure 8 in Appendix F.
Figure 2, we observe the
Other results. On top of the aforementioned results, further experiments are provided in Appendix G, which supports our choices and assumptions on the proposed model. The followings summarize the corresponding results: Comparison of stack-wise and block-wise feature alignment (Appendix G.1); Comparison of several normalization functions (Appendix G.2); Evaluation of the model under marginal (or the absence of) domain shift (Appendix G.3); Evaluation on Tourism [3], M3 [36], and
M4 [37] datasets (Appendix G.4). On top of that, we report the train and validation losses in Figure
5 supporting the stable optimization procedure. Furthermore, we provide the visual samples of forecasting results in Figure and make use of the interpretability of the feature-aligned N-BEATS to present Figure 7 (see Appendix F).
5 DISCUSSION AND EXTENSIONS
There are some unresolved theoretical parts in the current article such as a convergence analysis for the training loss (given in (3.8)) with the empirical risk minimization and the stack-wise feature alignment, filling the gap between the Sinkhorn divergence and the H-divergence adopted in the emror analysis of domain generalization models (given in Proposition 2.1), and the instability issue coming from the small entropic parameter c > 0 in the Sinkhorn divergence (see Table 2).
On the other hand, there many rooms for an extension of the proposed domain-generalizer time series forecasting model such as the 'conditional' feature measure alignment in [65] and 'adversarial representation learning framework' in [30]. Moreover, considering the utilization of 'moments' as distribution measurements in [22] and mitigating distribution mismatches through the 'contrastive loss' in [41] would represent meaningful avenues for future research.

---

Published as a conference paper at ICLR 2024
Acknowledgement. K. Park gratefully acknowledges support of the Presidential Postdoctoral Fellowship of Nanyang Technological University. M. Kang was supported by the NRF grant (022RRIA230108871 and the MSIT/IITP [No. 1711117093; 2021-0-00077; 2021-0-01343, Ar- tificial Intelligence Graduate School Program of SNUJ.
REFERENCES
[I] I. Albuquerque, J. Monteiro, M. Darvishi, H. Falk, and I. Mitliagkas, Generalizing to unseen domains via distribution matching. arXiv preprint arXiv:1911.00804, 2019.
[2] L. Ambrosio, N. Gigli, and G. Savaré. Gradient flows: in metric spaces and in the space of probability measures. Springer Science & Business Media, 2005.
[3] G. Athanasopoulos, R. J. Hyndman, H. Song, and D. C. Wu. The tourism forecasting compe- tition. International Journal of Forecasting, 27(3):822-844 2011.
[4] K. Bandara, C. Bergmeir, and S. Smyl. Forecasting across time series databases using recurrent neural networks on groups of similar series: A clustering approach. Expert Systems with Applications, 140:1 12896, 2020.
[5] H. Bao and S. Sakaue. Sparse regularized optimal transport with deformed q-entropy. Entropy, 24(11):1634 2022.
[6] S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan. A theory of learning from different domains. Machine Learning, 79:151-175, 2010.
[7] Ben-David, Blitzer, K. Crammer, and Percira. Analysis representations for domain adaptation. Advances in Neural Information Processing Systems, 19, 2006.
[8] Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspec- tives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):198-11828 2013.
[9] W. Cao, Wang. J. Li, H. Zhou, L. Li, and Y. Li. Brits: Bidirectional recurrent imputation for time series. Advances in Neural Information Processing Systems, 31, 2018.
[10] C. Challu, K. G. Olivares, B. N. Oreshkin, F. G. Ramirez, M. M. Canseco, and A. Dubrawski. Nhits: Neural hierarchical interpolation for time series forecasting. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 6989-6999 6697, 2023.
[11] Y. Chen, Y. Kang, Y. Chen, and Z. Wang. Probabilistic forecasting with temporal convolutional neural network. Neurvcomputing. 399:491-501, 2020.
[12] L. Chizat, P. Roussillon, F. Léger, F-X. Vialard, and G. Peyré. Faster wasserstein distance estimation with the sinkhorn divergence. Advances in Neural Information Processing Systems, 33:2257-2269, 2020.
[13] S. Di Marino and A. Gerolin. Optimal transport losses and sinkhorn algorithm with general convex regularization. arXiv preprint 0rXiv:2007.00976, 2020.
[14] R. M. Dudley. The speed of mean glivenko-cantelli convergence. The Annals of Mathematical Statistics, 40(1):40-50, 1969.
[15] H. Federer. Geometric measure theory. Classics in Mathematics. Springer, 2014.
[16] J. Feydy. Geometric data analysis, beyond convolutions. Applied Mathematics, 2020.
[17] J. Feydy, T. Séjoumé, F.-X. Vialard, S.-i. Amari, A. Trouvé, and G. Peyré. Interpolating between optimal transport and mmd using sinkhorn divergences. The 22nd International Conference Artificial Intelligence and Statistics, pages 2681-2690. PMLR, 2019.
[18] R. Flamary, N. Courty, A. Gramfort, M. Z. Alaya, A. Boisbunon, S. Chambon, L. Chapel, A. Corenflos, K. Fatras, N. Fournier, et al. Pot: Python optimal transport. Journal of Machine Learning Research, 22(1):3571-3578 2021.
10

---

Published as a conference paper at ICLR 2024
[19] Y. Ganin and V. Lempitsky. Unsupervised domain adaptation by backpropagation. In Interna- tional Conference on Machine Learning, pages 1 1180-1 1189. PMLR, 2015.
[20] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. March, and
V. Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17(59): 1-35, 2016.
[21] A. Genevay, G. Peyré, and M. Cuturi. Learning generative models with sinkhorn divergences. In International Conference on Artificial Intelligence and Statistics, pages 1608-1617. PMLR, 2018.
[22] M. Ghifary, D. Balduzzi, W. B. Kleijn, and M. Zhang. Scatter component analysis: unified framework for domain adaptation and domain generalization. IEEE Transactions Pattern Analysis and Machine Intelligence. 39(7):1414-1493) 2016.
[23] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(1):723-775) 2012.
2012.
[24] H. Hewamalage, C. Bergmeir, and K. Bandara. Recurrent neural networks for time series fore- casting: Current status and future directions. International Journal of Forecasting, 37(1))388- 427, 2021.
[25] H. Hu, M. Tang, and C. Bai. Datsing: Data augmented time series forecasting with adversarial domain adaptation. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 2061-2064, 2020.
[26] X. Jin, Y. Park, D. Maddix, H. Wang, and Y. Wang. Domain adaptation for time series fore- casting via attention sharing. In International Conference on Machine Learning, pages 10280- 10297. PMLR, 2022.
[27] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations, 2015.
[28] T.-N. Le, A. Habrard, and M. Sebban. Deep multi-wasserstein unsupervised domain adapta- tion. Pattern Recognition Letters, 125:249-255 225,, 2019.
[29] C.-Y. Lee, T. Batra, M. H. Baig, and D. Ulbricht. Sliced wasserstein discrepancy for unsuper- vised domain adaptation. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10285-10295, 2019.
[30] H. Li, S. Pan, S. Wang, and A. C. Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5400-5409, 2018.
[31] Y. Li. D. E. Carlson, et al. Extracting relationships by multi-domain matching. Advances in Neural Information Processing Systems, 31, 2018.
[32] Y. Li, X. Tian, M. Gong, Y. Liu, T. Liu, K. Zhang, and D. Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference Computer Vision, pages 624-639, 2018.
[33] B. Lim, S. Ö. Ank, N. Loeff, and T. Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 1784-177481 2021.
[34] M. Liu, A. Zeng. M. Chen, Z. Xu, Q. Lai, L. Ma, and Q Xu. Scinet: Time series modeling and forecasting with sample convolution and interaction. Advances in Neural Information Processing Systems, 55:5816-5828, 2022.
[35] K. Madhusudhanan, J. Burchert, N. Moong-Trung S. Born, and L. Schmidt- Thiemme. U-net in- spired transformer architecture for far horizon time series forecasting. In Joint European Con- ference on Machine Learning and Knowledge Discovery in Databases, pages 36-52. Springer,
2022.
11

---

Published as a conference paper at ICLR 2024
[36] S. Makridakis and M. Hibon. The c3-competition: results, conclusions and implications. International Journal of Forecasting, 16(4):451-47) 2000.
[37] S. Makridakis, E. Spiliotis, and V. Assimakopoulos. The m4 competition: Results, findings, conclusion and way forward. International Journal of Forecasting, 34(4):802-808, 2018.
[38] Matsuura and T. Harada. Domain generalization using a mixture of multiple latent domains. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 1 11749- 11756, 2020.
[39] L. McInnes, J. Healy, N. Saul, and L.. Großberger. Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29):861, 2018.
2018.
[40] T. Miyato, Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
[41] S. Motiian, M. Piccirilli, D. A. Adjeroh, and G. Doretto. Unified deep supervised domain adap- tation and generalization. In Proceedings of the IEEE International Conference on Computer Vision, pages 5715-5725, 2017.
[42] K. Muandet, D. Balduzzi, and B. Schölkopf. Domain generalization via invariant feature rep- resentation. In International Conference on Machine Learning, pages 10-18. PMLR, 2013.
[43] V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In International Conference on Machine Learning, pages 807-814, 2010.
[44] L. Oneto, M. Donini, G. Luise, C. Ciliberto, A. Maurer, and M. Pontil. Exploiting mmd and sinkhorn divergences for fair and transferable representation learning. Advances in Neural Information Processing Systems, 33:15360- -5370, 2020.
[45] B. N. Oreshkin, D. Carpov, N. Chapados, and Y. Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. International Conference on Learning Representations, 2019.
[46] B. N. Oreshkin, D. Carpov, N. Chapados, and Y. Bengio. Meta-learning framework with applications to zero-shot time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages
2021.
[47] R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training recurrent neural networks.
In International Conference on Machine Learning, pages 1310-1318. Pmlr, 2013.
[48] G. Peyré, M. Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends@ in Machine Learning, 11(5-6):355-60 5075 2019.
[49] J. Hhinonero-Candelle M. Sugiyama, A. Schwaighofer, and N. D. Lawrence. Dataset shift in machine learning. Mit Press, 2008.
[50] A. Ramdas, N. García Trillos, and M. Cuturi. On wasserstein two-sample testing and related families of nonparametric tests. Entropy, 19(2):47,2015
[51] S. S. Rangapuram, M. W. Seeger, J. Gasthaus, L. Stella, Y. Wang, and T. Januschowski. Deep state space models for time series forecasting. Advances in Neural Information Processing Systems, 31, 2018.
[52] D. Salinas, V. Flunkert, J. Gasthaus, and T. Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3):1181- 1191, 2020.
[53] Shen, Qu, W. Zhang, and Y. Yu. Wasserstein distance guided representation learning for domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 32, 2018
[54] C. Shui, Q. Chen, J. Wen, F. Zhou, C. Gagné, and B. Wang. A novel domain adaptation theory with shannnnn divergence. Knowledge-Based Systems, 257: 109808, 2022.
12

---

Published as a conference paper at ICLR 2024
[55] C. Shui, B. Wang, and C. Gagné. On the benefits of representation regularization in invariance based domain generalization. Machine Learning. 111(3):895 (115),895-91,,0 2022.
[56] V. Vapnik. Principles of risk minimization for learning theory. Advances in Neural Information Processing Systems, 4, 1991.
[57] A. Virmaux and K. Scaman. Lipschitzreguulariy regularity of deep neural networks: analysis and efficient estimation. Advances in Neural Information Processing Systems, 31, 2018.
[58] J. Wang, C. Lan, C. Liu, Y. Ouyang, T. Qin, W. Lu, Y. Chen, W. Zeng, and P. Yu. Generalizing to unseen domains: A survey on domain generalization. IEEE Transactions on Knowledge and Data Engineering, 2022.
[59] M. Wang and W. Deng. Deep visual domain adaptation: A survey. Neurocompatinng 312:135- 153, 2018.
[60] G. Woo, C. Liu, D. Sahoo, A. Kumar, and S. Hoi. Etsformer: Exponential smoothing trans- formers for time-series forecasting. arXiv preprint arXiv:2202.01381, 2022.
[61] H. Wu, J. Xu, J. Wang, and M. Long. Autoformer: Decomposition transformers with auto- correlation for long- long-term series forecasting. Advances in Neural Information Processing Sys- tems, 0222419-22400 2021.
[62] T. Xu, L. K. Wenliang, M. Munn, and B. Acciaio. Cot-gan: Generating sequential data via causal optimal transport. Advances Neural Information Processing Systems, 33:8798-8809, 2020.
[63] A. Zeng, M. Chen, L. Zhang, and Q. Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 11121-
11 11828 2023.
[64] H. Zhao, R. T. Des Combes, K. Zhang, and G. Gordon. On learning invariant representations for domain adaptation. International Conference on Machine Learning, pages 7523-7532. PMLR, 2019.
[65] Zhao, M. Gong, T. Liu, H. Fu, and D. Tao. Domain generalization via entropy regularization. Advances in Neural Information Processing Systems, 33:16096-16107, 2020.
[66] F. Zhou, Z. Jiang, C. Shui, B. Wang, and B. Chaib-draa. Domain generalization via optimal transport with metric similarity learning. Neurocomputing, 456:469-480, 2021.
[67] H. Zhou, S. Zhang, Peng. S. Zhang, J. Li, H. Xiong, and W. Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11 11106-11115, 2021.
[68] T. Zhou, Z. Ma, Q. Wen, X. Wang. L. Sun, and R. Jin. Fedformer: Frequency enhanced decom- posed transformer for long- series forecasting. In International Conference on Machine Learning, pages 27268 27286. PMLR, 2022.
13

---

Published as a conference paper at ICLR 2024
APPENDIX
A FURTHER DETAILS ON FEATURE-ALIGNED N-BEATS
The m-th residual operators (10 "5" (5 x m I..., M, are given by
6 (x) == (FC oFC oF. FC (x),
I E X = R
(T) (2) = V m, W m,
2 E 2 RR,
(A.1)
(f) (z) == V m.t W m,t²
2 E E IR?.
These operators correspond to the m-th stack and involve fully connected layers denoted by FC with RELU activation function [43]. Specifically, (x)) is defined as RELu m + Dm,n), where W m.n and bm, are trainable weight and bias parameters, respectively. The matrix W m. RRXXX7 (resp. W m.t E RTT*7) represents trainable linear projection layer for forecasting (resp. backcasting) operations. For the parameters B3 a, and y denoting to the forecast horizon, lookback horizon, and latent space dimension, respectively, V "m.J E 8x74 (resp. V mm, E Raxm) represents a forecast basis (resp. backcast basis) matrix, given by
Vm, == (v (Vm.to-... ) E Rx3 with v1 Vidd:... V m.
R.
(A.2)
(resp. Vm,t== == (v (m,t m E Raxm with (v M mm, E
v (resp. v 1) is forecast basis (resp. backeast basis) vector. and each v
Note that V mm. and Vm,t are set to be non-trainable parameter sets that embrace vital information for time series forecasting purposes, including trends and seasonality. These parameter sets are based on [45]. The basis expansion representations in (A.1) with flexible adjustments in (A.2) allow the model to capture the relevant patterns in the sequential data.
N-BEATS-G & N-BEATS-L The main difference between N-BEATS-G G and N-BEATS-I I lies on the utilization of Vm.it and Vn,t. More precisely, N-BEATS-G does not incorporate any spe- cialized time series-specific knowledge but employs the identity matrices for V m. and V mm,t In contrast, N-BEATS-I captures trend and seasonality information, which derives the interpretabil- ity. Specifically, Vm., is given by mm (1,t, t*)), where t }0, 1, . 2,
T)T. This choice is motivated by the characteristic of trends, which are typically represented by monotonic or slowly varying functions. For the seasonality, Vm.. is defined using a peri- odic function, (specifically the Fourier series), so that Vm,it (1, cos(t. c0s [8/2 1 1Jt)), sin(tt),, ... sin(2m] |//22 1|t))) T. The dimension of V mm is determined by adjusting 1Jt),
the interval between c2st and cos c2m [3/2 - as well as sin(2 and sin [ l[t). This formulation incorporates the notion of sinusoidal waveforms to capture the periodic nature of seasonality. The formulation of V m. is identical to that of V 1m with the only difference being the replacement of with 3 and 74 with 7t-
Lipschitz continuity of residual operators. Since each 1tth defined in (A.1) is an N N-layered fully connected network with the 1-Lipschitz continuous activation, i.c., RELU, we can apply [57, Section 6] have explicit representation for the (Rademacher) Lipschitz constant C.
of 4.th [15, Theorem 1.6. Furthermore, the forecasting and backcasting operators, E and em are matrix operators, and we can calculate their Lipschitz constants Cm.1 and Cm, by us-
ing the matrix norm induced by the Fuclidean norm I| i.e., Cm. = ||Vm,, Wm, > 0 and Gm, [m,, W.,,WW..tt Wm 0.
Detailed illustration of stack-wise feature alignment. In addition the above-presented N- BEATS, we incorporate the concept of learning invariance for domain generalization, which is re-
ferred as Feature-aligned N-BEATS. We provide the illustration of Feature-aligned N-BEATS in Figure 3 which is detailed version of Figure I.
B PROOFS OF LEMMA 3.4 AND THEOREM 3.6
Proof of Lemma 3.4. From the definition of o g in Definition 3.1 and the Lipschitz continuity of and ttth with corresponding constants C, > and C. 0, it follows for every 1,..., M,
14

---

Published as a conference paper at ICLR 2024
Serpting
sf (177 10% 10%--1271 spft
Stack
S Ir optineee (wthh (3.79)
Stack
(9
(4) aiggedd feedure roeaauree, Le., Jor eo in Ie M. (eogh1ah
D"
Stack M
Figure 3: Illustration Feature-aligned N-BEATS (noting that it is detailed version of Figure 1).
that for any 2, E.VE,
110 o gx - a o CCll"()))) 9
S cCoCm (( (r (m--11((1) .... (12))(1) (x)
(B.1)
-(r o (r (m-)))(1) ... ( (1))
Based on the residual operation in (3.1), i.e, (=)) x (5th (x)) and considering the Lipschitz continuity of a and r with respective constants C. > 0 and Cm > 0. we can establish the following inequality for every m 1,.... .,, and any x; y E X,
(n""())) - r""((") (y)ll & || - yll + [|(T o (x) - (E7 atth (v)ll $ (1 + Cm))a - yll-
Applying this to 1-times composition of rm, i.e., (m-1), we have that for any a. X,
|| ( (T- (z)))())))))))(())))))))) (z) - (r (
(T- 13 (v)l & (1+ Cm,1 Cm) - all.
Using the same arguments for the remaining compositions ( 1)()))( (B.1), deduce that for any x, y X,
(r1)(() in
|a (x) oog" (v)l SC Cm Cm
II (1 Cr Cn, vl1-
Proof of Theorem 3.6. We first note that from the nonnegativity of the entropy term in the regular- ized Wasserstein distance W 2. i.e., for every x E II( vz 2).
f JaxZ elog dı((z.) (dp(t)dv()))
dJ (z, v) 2 0,
it is clear that W ( v) 2 0 for every 11 P(ž). Moreover, from the definition of W
ie., W. (H, = W W.WZ(H, (M, JW (v, (M, H)), for u, P(ž), it follows that for every l,..., M and any i.j [I...., , K} such that i j.j.
W Px (a o gzm) xxt S W. W. g (")
g px (a o g" g(r) 1m
(B.2)
Let C(ž) be the set of all real-valued continuous functions on 2 and C(X: o nr) be defined by
( (1 X - ( s.t. fooog
15

---

Published as a conference paper at ICLR 2024
Then, from the dual representation in [48, Remark 4.18 in Section 4.4] based on the Lagrangian method and the integral property of pushforward measures in [2, Section 5.2], it follows for every m =0==..... ,.....,a .M that given Px P E (P with i * j.
W. W.z(lo o 1")) Px (Tog))
[ + ( [
sup T)EC(()
-c -e/ _xz
-ll (((o o )²1)(( (+
(B.3)
{ f(0)dPxx + h(y)dPx(()
sup (.
-(( (2)
d @ (z,8))
- Xxx
Consider the following regularized optimal transport problem:
inf xEI(FI, ,x)) {T -o0
(I, (")") =
(r( v) )dT x ) dr (T(2)1))
+clog
dP' (x))
Then, from the dual representation, as in (B.3), it follows that
( f())dF((())) + f M(y)dPx(()
(P' (P,,P") sup (.EE((N))
-e {f xxxx
(B.4)
where C(X) denotes the set of all continuous real-valued functions X. From the dual representa- tions in (B.3) and (B.4) and the relation that C(X:: a gm S C(x)) it follows that
W Wezz(lo e g"h """"
S We,x (P'x (P²,P";
(B.5)
On the other hand, from the first order optimality condition and the continuity of a o g presented in Lemma 3.4, it follows that the optimal potentials ,", hh* E C(X)) which realize the supremum in (B.4), are given by, respectively,
(x: = - log (f (100WW-llloo""""==100000"1"111
a (v)) xEX, X
== Ko di (x)) VEX,
which can be represented by / = j a o and h² = h5 o mm respectively, where J TV E C(²) are given by, respectively,
| e
7*(x)= = = log
x 2
io = -clog
This ensures that f" C(22 " C Ō(2). Hence we establish that (B.5) holds equality.
16

---

Published as a conference paper at ICLR 2024
From this and the Lipschitz continuity of g" with the constant Coogm > in Lemma 3.4, it follows that
We ((a o H ( " = We,x (P', PP; gm
( xxxx
(oogm -v12 elog
dr (x y)
)) dr dz(z, )
< inf PH:x)
dll (x))
I Imax|C 1) P.).
Therefore, we have shown that
Σ
max
W
S C
max (..,( iz
(e (P)))
(J it
m=1
with EM
max(Coegm 13. Combining this with (B.2) concludes the proof.
C SOME REMARKS ON SECTION 3.2
Remark C.I. Theorem 3.6 supports that the Sinkhorn-based alignment involving intricate dou- bly residual stacking architecture feasible, far the pair-wise divergence of source domains' measures can be 'a priori' estimated under some suitable divergence (i.e., the entropic regularized Wasserstein distance in the right-hand side of the inequality therein). Indeed, the PoT library in- troduced by [18] can be used for calculation of the entropic regularized Wasserstein distances. On the other hand, the proposed Sinkhorn- -baed alignment loss implemented by GeomLoss of [16] that is known to be significantly efficient and accurate approximate algorithm and will be the main calculation toolkit in the model.
Remark C.2. For sequential data generation, [62] introduced a causality constraint within optimal transport distances and used the Sinkhorn divergence as approximate for the causality constrained optimal transport distances. However, we do not consider the constraint but adopt the Sinkhorn divergence for approximate of the classic optimal transport distance in (3.4). This is because unlike the reference, there no inference for the causality between pushforward feature measures from the source domains.
D DETAILED EXPERIMENTAL INFORMATION OF SECTION 4
Experimental environment. We conduct all experiments using the specifications below:
CPU: Intel(R) Xeon(R) Platinum 8163 CPU 2.50GHz.
GPU: NVIDIA TITAN RTX.
All analyses in Section E utilize the same environment. The comprehensive software setup can be found on GitHub3.
Dataset configuration. The training data generation follows the steps detailed below:
1. Retrieve financial data {commodity, income, interest rate, exchange rate} from the Federal Reserve Economic Data (FRED), and weather data fpressure, rain, cemperatures wind) from the National Centers for Environmental Information (NCEI). Sub- sequently, we designate finance and weather as the superdomain A, with the subordinate data categorized individual domains.
2. Process each data point into sliding window of defined dimensions [o, 31, e.g., [50, 10), with the sliding stride of 1. Each segment is treated as an individual instance.
3. To alleviate any potential concerns arising from data imbalance, we establish a predetermined quantity of 75,000 instances for each domain through random sampling, thereby guaranteeing independence from such considerations.
https://gittubb.com/
fan-beats
17

---

Published as a conference paper at ICLR 2024
1.0%
Quarlerly
5,0%
monthly 10 10.8%
weeky 10. 10.2%%
dely 65,8
88.8%
(a)
(b)
Figure 4: Visualization of frequency distribution. (a) FRED, and (b) NCEL.
We randomly split each domain into three sets: 70% for training, 10% for validation, and 20% for testing.
worth noting that our dataset consists entirely of real-world data and covers a wide range of frequencies, including daily, weekly, monthly, quarterly, and yearly observations. The graphical representation of the frequency distribution for instances is depicted in Figure 4.
ODG scenarios involve no overlap between the source and target domains with respect to the super- domain, c.g., {D) (pressure, rain, temperature aure), and D commodity, For fair comparisons, the number of source domains is standardized to three across CDG and IDG scenarios. In CDG, we select one domain from the target domain's superdomain (p 2) and two domains from the other superdomain as sources, e.g., {D) Jk=1 {commodi ty, iittt..iincee income, pressure J. and D = ra To evaluate IDG, we designate one target domain and consider the remaining domains within the same superdomain source domains, e.g.. {D) (pressure, rain, temperature), and D w wd.
Note that each selected combination of source domains for Table 2 is (rain, temperature, wind) for ODG, coomodity. temperature, wind) for CDG, and (commodi tt, income, interest rate] for IDG.
Evaluation metrics. For our experiments, employ two evaluation metrics. Given H = N NXX with where N represents the number of instances, the metrics are defined as:
V1
1/H x > and MASE 1/(H-1)
SMAPE
X
H
16+1
Hyperparameters. Considering the scope of our experimental configuration that bring total of 184 experimental cases for each model, we adopt a suitable range of hyperparameters, detailed in Table 3, to achieve the performance results presented in Table 1.
Training and validation loss plots. Feature-aligned N-BEATS is complicated architecture based on the doubly residual stacking principle with an additional feature-aligning procedure. To investi- gate the stability of training, we analyze the training and validation loss plots. Figure 5 indicates that the gradients are well-behaved during training. This stable optimization is regarded to the Lemma 3.4 presented in Section 3.1.
18

---

Published as a conference paper at ICLR 2024
Table 3: Hyperparameters.
Hyperparameter
Considered Values
Lookhack horizon.
a 50
Forecast horizon.
B 10
Number of stacks.
M
Number of blocks in each stack.
L RELU
Activation function.
Feature dimension.
7 512
Loss function.
sMAPE
Regularizing temperature.
(0.1, 1,3)
Leurning rate scheduling.
CycliciR (base 1r-2e-7, max 1r-2e-5,
mode="triangular2"
scep.size.up=10/
Batch size.
B - 2
Number of iterations.
1,000
Type of stacks used for BEATS-I.
[Trnd, Seasonality, Seasonality]
Numher polynomials and harmonics used N-BEATS-L.
2
Pooling method used N-HiTS.
MaxPcol1d
Interpolation method used for N-HITS.
interpolate (mode "linear")
Kemel size used for N-HITS.
2
5
124
Al,
AsA
1.10
100
Figure 5: Training and validation loss plots. (a) Total loss, (b) forecasting loss, and (c) alignment loss. From top to bottom, each row illustrates the losses of N-BEATS-G G, N-BEATS-I, and N-HiTS, respectively. Losses are reported every 10 iterations.
19

---

Published as a conference paper at ICLR 2024
E DETAILED EXPERIMENTAL RESULTS OF SECTION 4
Tables 4 and 5 contain the extended experimental results summarized in Table The suitability of various measures of dispersion within our proposed framework explored in Table 2 is presented in Tables 6 and with more details. Specifically, we consider the commonly used metries for the representation learning framework: for μ.v P(2).
Kullback-Leibler divergence (KL):
(L(u||D)= log [l(d)) v(dz) u(dz),
Maximum mean discrepancy (MMD):
sup fEF ((E)))(()))) - F(o))ids)).
MMDF
where JF represents class of functions f - Notably, F can be delineated the unit ball in reproducing kernel Hilbert space. For detailed description and insights into other possible function classes, refer [23, Sections 2.2, 7.1, and 7.2].
Table Domain generalization performance of Feature-aligned N-BEATS (noting that it represents entire stats corresponding to the averaged ones given in Table 1). The first two columns denote the target domain for each experiment. This is followed in subsequent tables as well.
Methods
N-HITS
FA (Ours)
N-BEATS-1
FA (Ours)
N-BEATS-G
FA (Ours)
Metries
SMAPE
MASE
SMAPE
MASE
SMAPE
MASE
SMAPE
MASE
SMAPE
MASE
SMAPE
MASE
ODG
Commodity
0.136
0.049
0.103
0.046
0.279
0.072
0.258
0.069
0.195
0.052
0.136
0.049
FRED
Income
0.299
0.057
0.075
0.305
Interest rate
0.120
0.074
0.100)
0.070
0.200
0.144
0.189
0.046
0,148
0.073
0.120
0.073
Exchange rate
0.035
0.059
0.034
0.058
0.078
0.078
0.075
0.069
0,040
0.062
0.039
Average
0.148
0.060
0.134
0.057
0.232
0.069
0.214
0.065
0.172
0.061
0.150
0.059
Pressure
0.368
0.255
0,350
0.250
0.759
0.396
0.409
0.305
0,368
0.255
0.367
0.255
NCH
Rain
1.821
1.099
1.804
0.910
1.798
1.600
1.793
1.430
1.818
1.100
1.806
0,918
Temperature
0.247
0.245
0.245
0.243
0.247
0.353
0.246
0.255
0.247
0.245
0.246
0.244
Wind
0.455
0.645
0.454
0.644
0.451
0.665
0.449
0.662
0.455
0.645
0.454
0.645
Average
0.723
0.561
0.713
0.512
0,814
0.754
0.724
0.663
0.722
0.561
0.718
0.516
Commodity
0.082
0.047
0.081
0.144
0.195
0.060
0.197
0.059
0.119
0.046
0.107
0.046
FRED
Income
0.296
0.054
0.295
0.053
0.327
0.072
0.323
0.070
0.301
0.055
0.295
0.053
Interest rate
0.086
0.074
0.085
0.074
0.051
0.146
0.054
0.077
0.077
Exchange rale
0.032
0.056
0.029
0.055
0.054
0.074
0.055
0.063
0.032
0.056
0.031
0.055
Average
0.124
0.058
0.123
0.057
0.062
0.059
Pressure
0.375
0.264
0.372
0.260
0.408
0.307
0.405
0.299
0,540
0.374
0.372
0.263
NCEI
1.169
2.091
1.807
1.169
Temperature
0.334
0.243
0,245
0.242
0.275
0.245
0,240
0.244
0.253
0.245
0.245
0.243
0.453
0.643
0.452
0.643
0.441
0.646
0.439
0.646
0.453
0.643
0.452
0.643
Average
0.742
0.581
0.718
0.482
0.731
0.822
0.718
0.755
0.763
0.608
0.718
0.582
IDG
Commodity
0.083
0.045
0.068
0.043
0.125
0.053
0.126
0.053
0.165
0.047
0.080
0.044
FRED
Income
0.299
0.058
0.297
0.054
0.305
0.055
0.302
0.055
0.301
0.056
0.298
0.055
Interest rate
0.072
0.081
0.071
0.081
0.088
0.084
0.086
0.091
0.080
0.083
0.074
0.081
Exchaane
0.024
0.051
0.024
0,050
0.028
0.056
0.029
0.056
0.025
0.051
0.024
0,050
Average
0.119
0.059
0.115
0.057
0.137
0.062
0.136
0.064
0. 0.343
0.081
0.119
0.058
Pressure
0.394
0.276
0.384
0.272
0.392
0.266
0.389
0.263
0.384
0.276
0.384
0.276
Rain
1.776
1,211
1.776
1.208
1.792
2.922
1,805
3.046
1.818
1.690
1.776
1.208
Temperature
0.247
0.242
0.227
0.242
Wind
0.455
0.641
0.452
0.640
0.433
0.622
0.434
0.623
0.454
0.641
0.452
0.640
Average
0.718
0.593
0.715
0.715
1.039
0.726
0.712
0.714
20

---

Published as a conference paper at ICLR 2024
Table 5: Domain generalization performance of competing models (noting that it represents entire corresponding the averaged ones given in Table 1). Note that 'NA' indicates an anomalous emor exceeding 10,000, not divergence of training, as in Table 1.
Methods
NLincar
DLincar
Autoformer
Informer
Metrics
SMAPE
MASE
SMAPE
MASE
SMAPE
MASE
SMAPE
MASE
ODG
Commodity
0.666
17.941
0.657
18.084
0.830
22.325
1.248
40.580
FRED
Income
0.003
162.32
0.185
8,752.66
0.762
NA
1.275
NA
Interest rate
0.022
9.138
0.190
83.204
0.363
144.97
1.209
2, 2,184.78
Exchange
0.013
3.189
0.196
3.979
0.326
4.531
1.124
27.990
Average
0.176
48.147
0.307
2,214.48
0.570
NA
1.214
NA
Pressure
0,954
3.837
1,300
4.237
1.168
4.749
1.414
8,284
NCEI
Rain
1.038
1.089
1.231
1.169
1.175
0.897
1.783
1.162
Temperature
1.136
4.399
1.340
4.572
1.352
5,761
1.616
10.885
Wind
1.320
1.623
1.337
1.497
1.476
1.836
1.706
2.803
Average
1.112
2.737
1.302
2.869
1.293
3.311
1.630
5,784
CDG
Commodity
0.664
18.166
0.855
21.661
0.829
21.360
1,353
40,584
FRED
Income
0.004
212.60
0.203
9,979.14 14
0.995
NA
1.204
NA
Interest rate
0.023
9.364
0.587
210.91
0.957
1,695.19
1.040
2,058.07
Exchange rate
0.014
3,586
0.499
5.352
0.792
14.055
0.974
28.157
Average
0.176
60.929
0.536
2,554.27
0.893
NA
1.143
NA
Pressure
0.923
3,810
1.055
4.100
1.127
4,769
1.222
5,851
NCEI
Rain
1.037
1.137
1.033
1.125
1.201
0.899
1.523
1.149
Temperature
1.114
4.340
1.106
4.431
1.304
5.609
1.439
7.358
Wind
1.310
1.649
1.151
1.491
1.458
1.655
1.565
2.228
Average
1.096
2.734
1.086
2.787
1.273
3.233
1.437
4.147
IDG
Commodity
0.671
30.210
1.139
38, 28.929
0.835
21.462
1.652
34.784
FRED
Income
0.026
1,951.16
0.043
4,232.97
1,500
NA
0.704
NA
Interest rate
0.079
52.007
1.111
586.24
0.943
475.12
0.593
333.18
Exchange rate
0.013
5,443
1.080
11.873
0.726
5.815
0.424
5.255
Average
0.197
509.71
0.843
1,217.50
1,001
NA
0.843
NA
Pressure
0.868
5.271
0.698
5.318
1.280
6.992
1.906
4.614
NCEI
Rain
0,900
1,368
0.709
1.301
1.009
0.918
1.463
1.081
Temperature
1.014
6.055
0.768
5.753
1.310
4.788
1.080
4.235
Wind
1.206
2.195
0.913
2.084
1.472
1.592
1.570
1.986
Average
0.997
3.722
0.772
3.614
1.268
3.573
1.505
2.979
21

---

Published as a conference paper at ICLR 2024
Table 6: Ablation study on other divergences (noting that represents entire stats containing the ones given in Table 2)
Models
N-H175
S-BEATS-
B-BATS-
Divergences
WD
MMD
KL
WD
MMD
KL
WD
MMD
KL
Meriic
AMAR
MASE
WMAPS
MAAR
AMARS
MASE
MASE
MASK
<WARE
MASE
KMARE
+MARS
MARE
KMARP
MA46
DIG
Commediy
0.003
0/46
00.10
0..00
0.447
0.257
0.009
0.246
0.006
0.222
0.156
0.009
0.357
0.333
0.050
Inam::
0.297
0.055
0.332
0.055
0.293
0.050
0.334
0,375
0.318
0.022
0.330
0887
0.305
0.055
0.305
0.655
0.300
0.082
bainco mas
0.000
0.070
0.107
0.883
0.100
0.70
0.19
0.666
0.885
0.821
0.19
0.46
0.119
0.073
0.123
0.077
0.120
0,073
Exchange oee
0J134
0105
0.134
01158
0.840
03551
0.75
0.699
0.718
0.075
0.701
0979
0.09
0.800
0.61
0.09
0.03
0.087
Average
0.250
0411
0.306
N.8
0.255
0.367
0.2g
Rais
1,907
0.910
1833
0.919
Suy
N6N
1.589
1450
1,300
1.728
N18
108
133g
0.918
1518
1.006
1S51
0.951
Tempperit
0.345
0.345
0.345
0.344
0.46
0.948
0.345
0.54
0.56
0.364
0358
03385
0.58
0,37
0.444
0.255
0.64
S3N
0,400
0.622
6,450
0.63
800
150
0444
0.66
8459
0641
0.1%
0.566
Aeergge
0.714
0.513
0.715
N6N
0.738
0.738
N1N
0.766
0.515
0.577
CDG
Comsuodiy
0
01344
0.185
0.445
N68
9197
0,59
0.187
0057
NaN
N4h
0.00
0.96
0,19
0.0%
N4N
NNN
Eachurgga mase
Prossure
Average
Incoue
Rais
0.123
0.372
0,829
1.296
0.055
0.056
0.783
0.368
0115
0.829
0.371
0.25
1.815
0.86
0055
0.361
580
0180
2805
000
0314
0289
0055
0061
0070
6.054
0.176
0.406
03333
1.791
0.086
0.316
0.61
0.71
1.934
0644
0315
No..
0000
0133
0081
0004
0885
0.53
0200
033
0281
6300
0060
0000
1068
100
0000
0,00
NN
NN
0367
0,61
NNN
NX
NIN
Wend
0.683
0.43
0.451
0.45
558
188
1357
0466
0.400
0.66
806
100
0.85
0451
8241
NUN
Avirage
00716
0.483
07273
0.433
0538
0.355
0.719
0.755
N0N
N000
0.318
031
036
NIN
126
Commodiy
0,068
0.943
0J868
0,94
SoN
N68
13156
0.55
0.333
0/500
N68
N45
0.000
01964
0060
0009
NNN
NNN
Eachuings mak
Snteresi uane
Inoone
0,073
0.297
0.054
0.030
01081
0.29
0.025
0.072
6355
0/080
0.011
36h
368
888
0352
9086
2024
0391
0.555
0956
0035
0.386
0/06
0.25
0.69
0.55
N.5
N.N
N.5
198
N05
0.04
0.397
0.555
0.98
0.599
0054
0.300
8263
0000
0.80
0.56
NIX.
NuN
NGN
NIN
Pressare
Raie
1.782
1.306
0.272
0.403
1.849
0.266
1.431
0.55
0.50
1.800
1.645
0.364
1.790
1.811
6,259
03900
1367
1,308
0.376
0378
1517
118
655
NIN
Venspendaaee
0.245
0.342
0.247
0.341
N6N
8230
6.227
0224
0226
NN
150
0265
0.241
0365
036
NN
NNN
0.640
0.254
N6N
6,33
6.13
0.64
N.X
198
0461
0.50
0453
8641
MN
NGN
Average
Table 7: Ablation study on the Sinkhorn divergence with several values on (noting that it repre- sents entire stats containing the ones given in Table 2).
Models
N-HITS
N-BEATS-1
N-BEATS-G
Values
le-1
le.5
le-1
Metries
SMAPE
MASE
SMAPE
MASE
SMAPE
MASE
SMAPE
MASE
SMAPE
MASE
SMAPE
MASE
ODG
Commodity
0.103
0.046
0.1099
0.047
0.257
0.069
0.121
0.097
0.110
0.047
0.112
0.074
FRED
Income
0.297
0.055
0.297
0.129
0.334
0.075
0.329
0.267
0,302
0.057
0.304
0.204
Interest rate
0.100
0.070
0.105
0.946
0.046
0.116
0.094
0.106
0.074
0.107
0.072
Exchange rale
0.034
0.058
0,036
0.015
0.075
0.069
0.040
0.031
0.036
0.060
0.037
0.024
Average
0.057
0.137
0.152
0.139
0.060
Pressure
0.348
0,250
0,356
0.156
0.408
0.305
0.393
0.322
0,364
0.254
0.364
0.246
NCBI
1.795
0.910
1.790
1,980
1.603
0.944
1.832
Temperature
0.244
0.243
0,246
0.106
0.245
0.255
0,272
0.219
0.247
0.244
0.252
0,167
Wind
0.452
0.644
0.450
0.195
0.448
0.662
0.498
0.404
0.457
0.645
0.461
0.308
Average
1.072
0.580
1,073
0.466
1.099
0.868
1,187
0.963
1.086
0.599
1.098
0.735
CDG
Commodity
0.081
0.044
0.087
0.057
0.197
0.059
0.093
0.085
0.087
0.045
0.088
0.066
FRED
Income
0.295
0.053
0.298
0.193
0.323
0.070
0.318
0.290
0.298
0.056
0.302
0.225
Interest rate
0.085
0.074
0.090
0.058
0.146
0.054
0.096
0.086
0.090
0.078
0.091
0.067
Exchange
0.029
0.055
0.030
0.020
0.055
0.063
0.032
0.030
0.030
0.057
0.030
0.023
Average
0.123
0.057
0.126
0.082
0.180
0.062
0.135
0.123
0.126
0.059
0.128
0.095
Pressure
0.372
0.260
0.369
0.240
0.393
0.361
0.367
0.261
0.374
NCHI
Rain
1,803
0.783
1.790
1.163
1.787
1.831
1.910
1.747
1,801
0.965
1,816
1,355
Temperature
0.245
0.242
0.245
0.159
0,240
0.244
0.261
0.239
0.246
0.244
0.248
Wind
0.452
0.643
0.451
0.293
0.439
0.646
0.481
0.440
0.454
0.643
0.457
0.341
Average
1.088
0.522
1,080
0.702
1.096
1.065
1.152
1.054
0.613
1.095
0.818
IDG
Commodity
0.068
0.043
0.070
0.056
0.126
0.053
0.071
0.092
0.070
0.044
0.070
0.055
FRED
Income
0.297
0.054
0.304
0.240
0.302
0.055
0.308
0.397
0.299
0.058
0.303
0.238
Interest rate
0.071
0.081
0.072
0.058
0.086
0.091
0.073
0.095
0.073
0.085
0.072
0,057
Exchange rate
0.024
0.050
0.025
0.020
0.029
0.056
0.025
0.033
0.025
0.050
0.025
0.020
Average
0.115
0.057
0.118
0.094
0.136
0.064
0.119
0.154
0.117
0.059
0.118
0.093
Pressure
0.384
0.273
0.389
0.310
0.389
0.263
0.395
0.512
0.386
0.277
0.388
0.307
NCEI
Rain
1.776
1.212
1.785
1.422
1.805
3.046
1.811
2.348
1.781
1.326
1.781
1.409
Temperature
0.247
0.243
0.247
0.196
0.231
0.227
0.250
0.323
0.246
0.241
0.246
0.194
Wind
0.452
0.642
0.452
0.360
0.434
0.623
0.459
0.595
0.452
0.640
0.451
0.387
Average
1.080
0.743
1.087
0.866
1.097
1.655
1.103
1.430
1.084
0.802
1.085
0.858
22

---

Published as a conference paper at ICLR 2024
F VISUALIZATION ON FORECASTING, INTERPRETABILITY AND REPRESENTATION
Visual comparison of forecasts. We visually compare our models to the N-BEATS- models, i.e., N-BEATS-G, N-BEATS-I, and N-HiTS. As illustrated in Figure 6, incorporating feature align- ment remarkably enhances generalizability, allowing the models to produce finer forecast details. Notably, while baseline models suffer significant performance degradation in the ODG and CDG scenarios, aelgeee-aligned N-BEATS evidences the benefits the feature alignment.
In-domain Generalization
Cross-domain Generalization
Out-domain Generalization
GT N-BEATS-G
lookbeakk Toraraa foranad
GT - FA
mmm
lookback forecast
(a)
Why
Wan mm m
(b)
(c)
Figure 6: Visual comparison forecasts. (a) N-BEATS-G, (b) N-BEATS-I, and (c) N-HITS. Results are averaged across source domain combinations, with standard deviations.
23

---

Published as a conference paper at ICLR 2024
Visual analysis of interprefability. Figure 7 exhibits the interpretability of the proposed method by presenting the final output of the model and intermediate stack forecasts. N-BEATS-I and N-HiTS presented in Appendix A have interpretabilityy. More specifically. N-BEATS-I explicitly captures trend and seasonality information using polynomial and harmonic basis functions, respectively. N- HiTS employs Fourier decomposition and utilizes its stacks for hierarchical forecasting based frequencies. Preserving these core architectures during the alignment procedure, Feature-aligned N-BEATS still retains interpretability.
M
W
M
W
W
M
(8)
(b)
Figure 7: Visual analysis of interpretability. (a) Model forecasts, (b) stack forecasts of N-BEATS- I. and (c) N-HiTS. Note that N-BEATS-I utilizes single trend stack and two seasonality stacks, sequentially.
24

---

Published as a conference paper at ICLR 2024
Visualization of representation. We further investigate the representational landscape, analyze the samples of pushforward measure from N-BEATS-I and N-HiTS. Adopting visualization tech- niques for both aligned and non-aligned instances as depicted in Figure 2, we configure UMAP with 5 neighbors, a minimum distance of 0.1, and employ the Fuclidean metric. Similar to N-BEATS- G, discern observations in Figure 8 pertaining N-BEATS-I and N-HiTS: (1) instances coalesce, residing closer to one another, and (2) an evident surge in domain entropy, from both N-BEATS-I and N-HiTS.
A - (w/o)
A 0.5 (w) (a)
A- (w)
(wlo)
1
A-3 (w)
Figure 8: Visualization of extracted features. (a) N-BEATS-L, and (b) N-HiTS. For both (a) and (b), former plots illustrate increased inter-instance proximity, while subsequent ones depict inflated entropy.
25

---

Published as a conference paper at ICLR 2024
G ABLATION STUDIES
G.1 1 STACK-WISE VS BLOCK-WISE ALIGNMENTS
As mentioned in Remark 3.3, redundant gradient flows from recurrent architecture potentially causes gradient explosion or vanishing. To empirically validate this insight applied to our approach, we contrast stack-wise and block-wise feature alignments, as shown in Table 8. Notably, although stack-wise alignment generally outperform its counterpart, we do not observe the aforementioned problems, which could be identified by divergence training. N-BEATS-I with block-wise align- ment even demonstrates superior performance. Two plausible explanations are: (1) the limited number of stacks, and (2) the operational differences between the trend and seasonality modules in N-BEATS-I, which might help alleviating redundancy issue. Nonetheless, primary objective of generalizing the recurrent model across various domains appears achievable through ttack-wise alignment.
Table 8: Ablation study alignment frequency (i.e., stack wise block alignments)
Models
N-HITS
N-REATS-I
N-BEATS-G
Alignments
Block-wise
Stack-wise (Ours)
Block-wise
Stack-wise (Ours)
Block-wise
Stack-wise (Ours)
Metrics
SMAPE
MASE
SMAPE
MASE
SMAPE
MASE
SMAPE
MASE
sMAPE
MASE
SMAPE
MASE
ODG
Commodity
0.137
0.103
0.133
0.049
0.258
0.069
0.049
0.049
FRED
Income
0.306
0.056
0.298
0.055
0.305
0.055
0.335
0.075
0.306
0.056
0.304
0.055
Exchange rate
0.040
0.060
0.034
0.058
0.040
0.060
0.075
0.069
0.040
0.060
0.039
0.060
Average
0.151
0,060
0.134
0.057
0.059
0.214
0.065
0.060
0.059
Pressure
0.368
0.255
0.350
0.250
0.367
0.254
0.409
0.305
0.368
0.255
0.367
0.255
NCEI
Rain
1,806
1.094
1,804
0.910
1,806
1.094
1,793
1.430
1.807
1.091
1,806
0.918
Temperature
0.247
0.245
0.245
0.243
0.247
0.245
0.246
0.255
0.247
0.245
0.246
0,244
Wind
0.456
0.645
0.454
0.644
0.456
0.645
0.449
0.662
0.457
0.645
0.454
0.645
Average
0.719
0.560
0.713
0.719
0.560
0.724
0.663
0.559
0.516
CDG
Commodity
0.102
0.046
0.081
0.044
0.105
0.046
0.197
0.059
0.108
0.047
0.107
0,046
FRED
Income
0.301
0.054
0.295
0.053
0.301
0.055
0.323
0.070
0.301
0.054
0.295
0.053
0.077
Excchange rale
0.031
0.056
0.029
0.055
0.031
0.098
0.055
0.063
0.032
0.055
0.031
0.055
Average
0.123
Pressure
0.371
0,263
0,372
0.260
0.370
0.262
0.405
0.299
0.373
0.264
0,372
0,263
NCEI
Rain
1.809
1.144
1,803
0.783
1.808
1.148
1.787
1.831
1.804
1.181
1.804
1.178
Temperature
0.246
0.244
0.245
0.242
0.246
0.244
0.240
0.244
0.246
0.244
0.245
0.243
Wind
0.453
0.644
0.452
0.643
0.454
0.644
0.439
0.646
0.453
0.643
0.452
0.643
Average
0.720
0.574
0.718
0,482
0.720
0.575
0.718
0.755
0.719
0.583
0.718
0.582
IDG
Commodity
0.081
0.045
0.068
0.043
0.075
0.044
0.126
0.053
0.074
0.044
0.080
0.044
FRED
Income
0.302
0.056
0.297
0.054
0.301
0.056
0.302
0.055
0.302
0.056
0.298
0.055
Interest rate
0.079
0.083
0.071
0.081
0.079
0.082
0.086
0.091
0.079
0.083
0.081
Exchange rate
0.025
0.051
0.024
0.050
0.025
0.051
0.029
0.056
0.025
0.051
0.024
0,050
Average
Pressure
0.384
0.276
0.384
0.272
0.383
0.277
0.389
0.263
0.384
0.276
0.384
0.276
NCB
Rain
1.818
1.776
Temperature
0.247
0.243
0.247
0.242
0.245
0.242
0.231
0.227
0.246
0.243
0.245
0.241
Wind
0.453
0.641
0.452
0.640
0.453
0.641
0.434
0.623
0.453
0.641
0.452
0.640
Average
0.726
0.710
0.715
0.591
0.720
0.674
0.715
1.039
0.725
0.709
0.714
0.591
26

---

Published as a conference paper at ICLR 2024
G.2 NORMALIZATION FUNCTIONS
According to the Table 9, Feature-aligned N-BEATS generally achieves superior performance when utilizing softmax function. However, there are instances where tanh function or even the absence of normalization yields better results compared the softmax. This suggests that while scale is predominant instance-wise attribute, it may exhibit domain-dependent characteristics under certain conditions. Aligning this scale is therefore necessary. This entails that the softmax, tanh, and to not normalize offer different levels of flexibility in modulating or completely disregarding the scale information, implying spectrum of capacities in aligning domain-spectie -speciic attributes.
Table 9: Ablation study on normalization functions.
Niemalooon
Nomz
tunb
sofermee (Omo)
Nenz
tauh
softetee (Curs)
None
tach
weftnnet (Qam)
Metrics
SMATS
MASE
SMAPS
MASE
SHATS
MASC
AMAAE
MASI
AMATE
MASE
AMATE
MASI
3MARE
MASE
SMAFE
MASE
SHATE
MASE
DDC
Commodity
0.056
0.239
0.055
0.319
0.063
0.355
0.306
0.056
0.305
0.304
0,055
Insrest uno
0.301
0.711
0.101
0.771
0.190
0.770
9.191
0/675
0.895
0.348
0.189
0/846
0.120
0.072
0.123
16074
0.120
0.073
Exchurnga mate
0J334
0.86
0.834
0
0J834
0358
0.72
0651
0,777
0.074
0.055
0859
0.041
0.061
0.63
0.009
0.099
0.0M)
Avarage
0156
0:135
0357
0212
0:218
0064
0.314
6:129
0
0.130
0.989
0.15
0.91
0.10
1.800
1.541
1.795
1.075
1554
1.306
0.913
Tenspenutaee
0.247
0.244
0.245
0.344
0.345
0.243
0348
0.253
0240
0.256
0.246
0.255
0.247
0.345
0.268
0.345
0.245
0.244
Wand
0.455
0.645
0.455
0.644
0.444
0.644
0452
0.600
CA51
0.651
8.49
0.652
0.456
0.545
0457
0645
0.444
0.645
Avrragg
0.514
0.711
0:514
0.783
0.512
0.737
0.710
0.165
0.755
0.85
0.717
0.983
0.730
0.718
0.566
Commodity
01833
01945
8J881
BJ344
0J811
03444
0.19
0,099
0.00
0.611
0.197
0.099
0.108
0.067
0.00
0.107
fnoom:
0.055
0.245
605
0383
0.115
6.119
0.664
0.115
0.700
6-100
0.054
0.60
0.295
R.83
Innevest
0.056
0/129
0005
0055
0,55
0.05%
0005
0005
0.301
0.078
0382
1000
0.100
0.07
Avverage
0.123
0.038
0,123
0.12
0.65
0,556
6.182
0083
6139
0252
00366
0.356
8000
0131
0L0SS
0.373
0.260
0.314
0.260
0.322
0:580
0.16
C410
03115
0.65
0299
0.255
0.32
0.55
0.356
0372
0.20
Rale
1,998
01311
1.955
0151
1.93
0781
1400
2193
1,800
21144
1287
1251
1885
1.15
1305
1.066
1884
1.174
Teraperuture
0.453
0.42
8.439
0.64
0.55
0,64
Averages
0.720
0.519
0.739
0.519
0.768
0.483
0335
0341
6:715
0.359
6.188
0.55
0.337
0.52
0738
0.360
0.718
0.582
IDG
Commoday
0.044
0,065
0.443
0.124
0,055
0.136
0.045
0.83
0.000
0,044
Inom:
0.200
0.97
01:30
0,05
0.297
0354
0310
0,59
6.36
69%
630
0055
0.300
0396
030
8003
0.796
0.951
Iaviod may
0,073
01081
0.073
Ium
(J71
0081
0087
0887
0.00
0009
0086
0091
0.000
0381
8080
0060
0.04
0.08
Exchaage uate
0.025
0101
0.025
0,59
0.24
0050
0035
0.55
0/40
0055
0029
0956
9.055
0.002
0.06
0.51
0.24
0.800
Average
0.155
0.41
0062
0136
0.13
0.009
0123
0,58
Main
1.77%
12211
1.7%
1.285
1.7%
1.28
1.65
1312
1315
1346
1355
1.266
1818
1571
1815
1.66
1.766
1.06
Tertperurure
0.247
0.342
0.31
0332
0.47
0.322
0.36
0335
0335
6331
8331
6.377
0.36
0.41
0366
634
0335
0.31
Ward
0.454
0641
0.254
0.40
0.43
0640
9441
0431
0/411
0632
0.444
6.23
0.553
0.62
8453
0640
0.00
0.000
0.718
0.991
0.715
0.598
0.911
0.33
6035
1.44
0315
1839
0.72
0.707
6721
0716
0714
0.591
G.3 SUBTLE DOMAIN SHIFT
Although the domain generalization commonly focuses on the domain shift problems, models may not perform as expected when the domain shift between source and target data is minimal. In some cases where the data from both domains align closely, fitting source domain without invariant feature learning even can be beneficial. To examine this concern, we extend our analysis to the generalizability of Feature-aligned N-BEATS under such conditions. Table 10 demonstrates, while our model remains competitive, there is performance degradation observed in certain instances.
Tabbe 10: Evaluation under subtle domain shift. "F' and 'N' represent the FRED and NCEI datasets, respectively. The number of domains associated with each dataset is denoted accordingly, e.g., 'F3' represents three source domains from FRED. We conduct experiments by considering all possible
combinations for each case.
Methods
N-HITS
+ EA (Ours)
N-BEATS-1
FAA (Ours)
N-BEATS-G
+ FA (Ouns)
Metries
SMAPE
MASE
SMAPE
MASE
SMAPE
MASE
SMAPE
MASE
SMAPE
MASE
SMAPE
MASE
F3
0.023
2.055
0.023
2.055
0.025
2.028
0.027
2.061
0.024
2.064
0.024
2.066
F2N1
0.236
0.244
0.236
0.240
0.210
0.221
0.209
0.220
0.236
0.241
0.236
0.244
FIN2
0.235
0.240
0.235
0.240
0.209
0.220
0.209
0.219
0.235
0.240
0.234
0.241
N3
0.243
0.243
0.243
0.243
0.220
0.224
0.221
0.225
0.242
0.243
0.241
0.242
Average
0.184
0.695
0.184
0.694
0.166
0.673
0.166
0.680
0.184
0.697
0.184
0.698
27

---

Published as a conference paper at ICLR 2024
G.4 TOURISM, M3 AND M4 DATASETS
We extend experimental scope include three additional datasets: Tourism [3], M3 [36], and M4 [37]. Models are trained on two datasets and tested on the remaining dataset, enabling us to evaluate both ODG (M3, M4 + Tourism) and CDG (M3, Tourism M4 and M4, Tourism M3) scenarios. Our proposed methods consistently outperform N-BEATS models, demonstrating their generalization ability.
Table 11:
Domain generalization performance on Tourism, M3 and M4 datasets. The first column indicates the target domain.
Methods
N-HiTS
FA (Ours)
N-BEATS-I
FA (Ours)
N-BEATS-C
+ (Ours)
Metries
SMAPE
MASE
SMAPE
MASE
SMAPE
MASE
SMAPE
MASE
SMAPE
MASE
SMAPE
MASE
ODG
Tourism
0.437
0.122
0.427
0.117
0.382
0.104
0.372
0.098
0,440
0.125
0.427
0.121
CDG
M3
0.357
0.296
0.356
0.286
0.294
0.355
0.284
0.343
0.364
0.296
0.352
0.285
M4
0.097
0.015
0.091
0.009
0.152
0.093
0.148
0.086
0.091
0.014
0.084
0.009
28


Reviews:
[{"review_id": "kyrws9kSIV", "reviewer": "Reviewer_F4wY", "summary": "* The paper proposes a new time series model \"Feature-aligned N-BEATS\", which is an extension of existing N-BEATS model targeting domain-generalized time series and learning invariant features.\n* The novelty of this model is the introduction of alignment across different stacks using Sinkhorn divergence. The training loss includes both empirical risk minimization from multiple domains and an alignment loss calculated with Sinkhorn divergence to promote invariance.\n* The paper also includes experiments and ablation studies, showing the model's effective forecasting and generalization capabilities​.", "questions": "Please see above section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is well-written and easy to follow.\n* The proposed idea is simple but yet effective given the experimental results.\n* The distance measure choice of Sinkhorn divergence seems promising. It shows non-trivial improvements in the ablation study and also takes much less time compared with standard Wasserstein distance.", "weaknesses": "I checked the proposed architecture and the experimental results. They look convincing.\n\nMy concern is that the idea is not that \"new\". Imposing penalties on divergence is commonly used to promote model invariances. The novelty might be the application and the extension on N-BEATS model at the stacking level. I am not sure whether this contribution is enough. Another concern is that the analysis (theorem 3.6) seems to be direct extensions of the literature [41] which may diminish the contribution as well.\n\nIn general, I still believe it's a good paper and the proposed method seems working and elegant. However, my review might be superficial given I am not familiar with this domain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cEBvHkOXhl", "reviewer": "Reviewer_adrw", "summary": "The paper proposes the advancement of N-BEATs, a pure deep learning based method for time series forecasting. The novelty of the approach is a combination of stack-wise feature alignment (FA) and N-BEATs. The training objective is to minimize the empirical risk and the stack-wise Sinkhorn divergences (Section 3.3) as an approximation of optimal transport measures for feature alignment.\n\nWell written introduction points out that it is not a straightforward combination of the established components but a nontrivial extension of NBEATs that poses several challenges. The paper lays down background on NBEATs and FA in rather technical manner. Never the less it stresses the main challenges nicely and the way presented approach mitigates them, including proofs of the main theoretical steps, Lemma 3.4, Theorem 3.6 (Sinkhorn div.), eventually resulting in the Algorithm 1. The experimental section test the model on two real world datasets (FRED, NCEI) against three domain shift scenarios:  out-domain generalization (ODG), cross-domain generalization (CDG), and in-domain generalization (IDG) with on-par or superior (confidence intervals are not present) results.", "questions": "1. Authors proposed stack-wise feature alignment using Sinkhorn divergence (original feature alignment uses KL divergence or Jensen-Shannon divergence). Can authors elaborate on the difference?\n\n2. Would naive feature alignment applied on N-BEATS work? Why not? That is, why should reader consider using proposed method?\n\n3. Paper already suggests using approximation to H-divergence which is computational expensive. So even more importantly, what are computation expenses related to proposed stack-wise approach compared to N-BEATS?\n\n4. Results are only showing visually rather marginal gains (see Weaknesses) against the compared methods with differences on 2nd or even third decimal places (I know, it's percentage mean error, but still ...). Could authors elaborate more on the positive side of the results and put into context increased computational costs? In general I would suggest to discuss limitations and practical usefulness of the method.\n\nSome further suggestions: \n- Consider bringing, possibly simplified, model architecture (Fig 2 from Appendix) to the main body instead of a rather cumbersome technical description Section 2 \n- Focus more on novelty of the proposed approach in Sections 1-3.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ A novel practical method with available code supported by theoretical results\n+ Timely taps into lively research field and improves on previous existing results on pure deep learning based time series forecasting interesting to ML community", "weaknesses": "- Overall presentation: The main contributions are rather buried in the technical exposition of sections 1-3. The main idea (adding stack-wise FA through minimising Sinkhorn div. added to the N-BEATS loss) could be simply laid out and perhaps visualised while some of the technical details may be put into Appendices. I would consider starting from algorithm and Section 3.3. and than relate it to principles in a backward manner as opposed to a current forward exposition that simply takes too long and has several leaps of faith anyways (see below). Authors are encouraged to lighten the exposition up in the main paper body and increase the rigour int he Appendix. More over, if paper claims “extraordinary results” bringing some results (e.g.,  Fig.4 in Appendix or simplified Fig.1) forward into the Introduction could increase motivation of the readers.\n\n- Due to multitude of approximation and heuristics used, i.e., Sinkhorn instead of H-divergence, Definition 3.5, Remark 3.8,  “... Instead, we conduct the alignment by regularizing exclusively on feature extractors $\\Psi$ ...” the work effectively strongly relies on experimental results. Experimental results seems only marginally better compared to selected alternatives (Tables 1 and 2), however. These should be extended to increase the validity of the results.\n \n- I believe important but unclear computational cost increase compared to alternatives (cost increase is only assumed by me but expected since method adds “FA” and more on top)\n\n- One of the major features that N-BEATS algorithm provides is an Interpretability. The paper avoids any analysis of interpretability after adding FA", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bSmbv1IHMk", "reviewer": "Reviewer_g8xd", "summary": "The paper introduces a new algorithm for performing time series forecasting under domain shift. It does so by combining the N-BEATS deep learning model for time series forecasting with a feature alignment regularization that is intended to select for robust models across a variety of domains. While both approaches had been previously introduced and popularized in separate bodies of literature, the combination of the two approaches is nontrivial due to the challenges of implementing feature alignment on a stack-like architecture.\n\nConcretely, the authors parameterize the N-BEATS architecture with a collection of feature mappings $\\phi^m$, forecast generators $\\xi_{\\downarrow}$, and backcast generators $\\xi_{\\uparrow}$. Each stack produces a prediction of the forecast $\\hat{y}$ by repeatedly computing residual features and mapping those to forecasts and backcasts, whose residuals are used to generate further features. The predictions of multiple stacks are combined together. The paper introduces a feature alignment regularization term, which is small when the  features extracted by each stack are distributionally similar across domains. While the $\\mathcal{H}$-divergence initially seems like the proper quantity to measure distributional distance, the quantity is extremely computationally expensive to estimate. Instead, the authors use think entropically-regularized Sinkhorn divergence to measure the difference between extracted feature distributions. They package these insights into an training algorithm in Section 3.3 that alternately updates the parameters of different component networks.\n\nThe authors provide some theoretical analysis of the algorithm's properties, without including no new optimization or generalization bounds. Lemma 3.4 bounds the Lipschitz smoothness of each stack's feature extraction map, while Theorem 3.6 bounds the extracted features' maximum of the Sinkhorn divergences by the maximum of the divergence between domain input distributions, which depends on the aforementioned Lipschitz bound.\n\nThey empirically evaluate the performance of the algorithm in Section 4 by comparing the performance of these algorithms with alternatives on time series datasets like FRED and NCEI. They show marked improvements in outside domain generalization over N-BEATS algorithms without feature alignment and post comparable performance among in-domain generalization. They also compare several levels of entropic regularization and demonstrate a sharp improvement in runtime over computing Wasserstein-2 divergences, while maintaining similar performance.", "questions": "Are the empirical time series tasks and the formulations of ODG, CDG, and IDG standard and comparable to other works? I noticed that the original N-BEATS paper was evaluated on M3 and M4 competition datasets, as well as a Tourism dataset. To the authors and other reviewers, are there other standard datasets that should be considered for this method to be empirically compelling? \n\nIs there any intuition or experimental results that comment on the smoothness of $\\sigma \\circ g^m$? While the Lipschitzness bound in Lemma 3.4 suggests it could be highly non-smooth, I'd be interested to know more about the gradients were largely well-behaved.\n\nWhat should I take away from Figure 1? It appears that the structure of the embeddings changes little between stacks and blocks, and it's unclear to me whether that should be the case or not. The structural differences between $\\lambda = 1$ and $\\lambda = 3$ are almost undetectable.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "As far as the reviewer (who is not an expert in either domain shift or time series forecasting) is aware, the paper is a novel and interesting contribution to both fields with compelling empirical results and strong application of optimal transport methods. The crafting of the feature alignment regularization term appears to be original for time series forecasting, as is the application of Sinkhorn divergence. Given the challenges of applying neural networks to time series data and the particularly pressing domain shift challenges posed there, the problem seems relevant, and the paper's introduction does a good job making that clear.\n\nThe paper is easy to read and includes clear explanations of the purposes of each component of the algorithm. Figure 2 is a particularly helpful illustration of the neural network architecture. Notation is clear overall with few mistakes, and the reviewer detected no major mathematical errors.\n\nThe experiments are thorough and compare the method with a wide range of alternatives, considering both runtime and accuracies.", "weaknesses": "While I am overall satisfied with the contributions of the papers, there are several issues that I would like to see addressed by the authors.\n\nFor ease of understanding, I think that either the Contributions paragraph or a different subsection before Section 2 could be used to more clearly signpost how the algorithm will be presented in Sections 2 and 3. Including something like equation (3.7) earlier to demonstrate the kind of training objective being constructed could be helpful for readers to identify the role played by feature alignment throughout. A large number of parameters are introduced in these sections, such as $\\alpha$, $\\beta$, $\\gamma$, $K$, $M$, and $L$; I think some of these could be better contextualized, so that the reader can know which are expected to be large and which small. (For instance, we should probably think of $L$ and $M$ as relatively small constants, due to their impact on Lipschitzness.)\n\nWhile Table 2 of the empirical results compares the runtime of computing different divergences, I'd be interested in seeing comparisons between the end-to-end runtimes of the methods in Table 1, in order to evaluate whether the computing the regularization term for feature alignment adds a substantial computational cost.  \n\nAs the authors state, the theoretical contributions of the paper are relatively light and focus on bounding the Sinkhorn divergence of extracted features. Future proofs of convergence (or experimental analyses thereof) and generalization bounds for Sinkhorn divergence in the vein of Proposition 2.1 would be valuable, but I do not consider these essential inclusions in this work for publication. \n\n# Minor comments\n- The clause on pg 1 \"the proposed model is built a purely deep learning-based model which is N-BEATS proposed by [42,43]\" parses a little awkwardly.\n- $\\epsilon$ is overloaded as both the entropic regularization parameter and the expected risk in Proposition 2.1, which is never explicitly defined.\n- In Definition 3.1, should $m=1, \\dots, d$ instead end with $M$?\n- The forecasting algorithms N-BEATS-I and N-BEATS-G are seen for the first time in Section 4 without context; a pointer to the first page of the appendix---where they are helpfully disambiguated---would be appreciated.\n- Given the number of entries and small fonts in the tables (like Table 1), bolding the values of the best losses in each row may improve readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)