SutraNets: Sub-series Autoregressive Networks for Long-Sequence, Probabilistic Forecasting
Authors: Shane Bergsma
Tim Zeyl
Lei Guo
Venue: NeurIPS 2023 poster (2023)
Keywords: time series
probabilistic forecasting
autoregressive generative models
neural networks
TL;DR: We present SutraNets, a general method for converting probabilistic forecasting of long-sequence time series into multivariate prediction over lower-frequency sub-series, via an autoregressive (across time and sub-series) generative model.

Abstract: We propose SutraNets, a novel method for neural probabilistic forecasting of long-sequence time series. SutraNets use an autoregressive generative model to factorize the likelihood of long sequences into products of conditional probabilities. When generating long sequences, most autoregressive approaches suffer from harmful error accumulation, as well as challenges in modeling long-distance dependencies. SutraNets treat long, univariate prediction as multivariate prediction over lower-frequency sub-series. Autoregression proceeds across time and across sub-series in order to ensure coherent multivariate (and, hence, high-frequency univariate) outputs. Since sub-series can be generated using fewer steps, SutraNets effectively reduce error accumulation and signal path distances. We find SutraNets to significantly improve forecasting accuracy over competitive alternatives on six real-world datasets, including when we vary the number of sub-series and scale up the depth and width of the underlying sequence models.

SutraNets: Sub-series Autoregressive Networks for Long-Sequent Probabilistic Forecasting
Shane Bergsma
Timothy Zeyl Lei Guo
Huawei Cloud, Alkaid Lab Canada (shane. berggsm,, timothy. seetthyy.yyyy.......... htw.g///ewwwwwwww..coom
Abstract
We propose SutraNets, a novel method for neural probabilistic forecasting of long- sequence time series. SutraNets use an autoregressive generative model to factorize the likelihood of long sequences into products of conditional probabilities. When generating long sequences, most autoregressive approaches suffer from harmful error accumulation, as well as challenges modeling long-distance dependencies. SutraNets treat long, univariate prediction as multivariate prediction over lower- frequency sub-series. Autoregression proceeds across time and across sub-series in order ensure coherent multivariate (and, hence, high-freeg freh-freuuency univariate) outputs. Since sub-series can be generated using fewer steps, SutraNets effectively reduce accumulation and signal path distances. We find SutraNets to signifi- cantly improve forecasting accuracy over competitive alternatives on six world datasets, including when vary the number of sub-series and scale up the depth and width of the underlying sequence models.
Introduction
As a cloud provider, we rely on long-range forecasts to allocate resources and plan capacity. In particular, need predict the fine-grained pattern of demand (e.g., hourly) for large number of products, over a long time horizon (c.g., months). Since purchase decisions are large-scale (e.g.. supplying regions with 100K machines), and made weeks-to-months before delivery, cloud providers have a profound need for methods that "capture the uncertainty of the future" [43]. Fine granularity, long histories, long forecast horizons, and quantification of uncertainty can be formalized as the long-sequence (many-step) probabilistic forecasting problem.
Recent work applies Transformers [82] long segueequence forecasting. In order circumvent the Transformer's ' meadratically-scaliin attention mechanism, long sequences necessitate trading off increased signal path for reduced computational complexity [44, 92, 87, 47]. Unfortunately, the effectiveness and even validity of these approaches have been questioned [90, 32). Most of these approaches do not provide probabilistic outputs (rather best-guess point predictions), and all condition on fixed-size windows, typically of very short duration. Indeed, many recent papers on "long-term" forecasting [87, 86, 70, 93] condition on maximum of 96 inputs only 4 days of history at 1-hour granularity. Such restricted context is serious limitation given that many time series (including in standard evaluation datasets based on electricity and traffic) have strong weekly seasonality.
Our production forecasting system, like many in industrial settings [65, 52, 72] and cloud services [69], based RNNs. The RNN's 's recurrent architecture forces the compression and consolidation of long-range information, and RNNs work well on smaller datasets as they are not as "data hungry" Transformers [16]. RNNs theoretically condition context of any length; in this paper, condition RNN forecasts on length-2016 contexts without difficulties in scaling.
RNNs are often used to generate probabilistic forecasts via autoregressive approach [21, 65, 60, 7], whereby the probability over future timesteps factorized into a product of one-step-ahead
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Multivariate subseiies
Autoregressive prediction
Long, univariate history
Long, univariate output
Figure 1: SutraNets: a long, univariate history is converted to fower-frequency sub-series. Multivariate autoregressive prediction, across time and sub-series, ensures coherent univariate output samples.
conditionals [6, 28). When generating many steps, however, two key problems arise: (1) a discrepancy grows between training, where we condition on true previous values, versus inference, where we condition sampled values (the discrepancy problem), and (2) long-term dependencies may be challenging to exploit (the signal path problem). The discrepancy problem leads to error accumulation during inference [5], but also a kind of informational asymmetry: because highly-informative true values are available during training, the model may ignore other useful info, e.g., seasonally- -laggdd inputs provided extra features. The signal path problem, meanwhile, is especially acute for RNNs, where distant information must be preserved over many steps in the RNN hidden state.
We propose SutraNets.' general method for probabilistic prediction of long time series. We focus on the application of SutraNets to RNN-based forecasting, where they offer major benefits. SutraNets transform a length-N -V series into K sub-series of length N/K. Sub-series forecasts are generated autoregressively, scquentially conditional on each other, enabling coherent outputs (Fig. 1). SutraNets address both the discrepancy problem (by forcing the network to take larger generative strides, i..., predicting without access to immediately- pereeeddily--eeedinggg true values) and the signal path problem (by reducing the distance between historical and output values by factor of K). Training of SutraNets decomposes over sub-series; i.e., sub-series RNNs can be trained in parallel, enabling a K-fold improvement in training parallelism over standard RNNs.
Experimentally, SutraNets demonstrate superior forecasting across six datasets, improving accuracy by an average of 15% over C2FAR [7] and over strong seasonality-aware baselines. We evaluate variety of other strategies for improving long-sequence forecasting, including lags, input dropout, deeper and wider networks, and multi-rate hierarchical approaches. Through these evaluations, gain insights into how to succeed at long-sequence generation, in time series and beyond.
2 Background and related work
Probabilistic autoregressive forecasting Autoregressive forecasting models typically combine backbone sequence model with technique to estimate an output probability distribution at each step. In this paper, we apply SutraNets to C2FAR [7], a state-of-the-we distribution estimator that was shown to improve over DeepAR [65], DeepAR- binned [60], SQF-RNN [21], and IQN-RNN [27]. SutraNets can serve as the sequential backbone for any of these estimators, as well as for other recent approaches such as the denoising diffusion models [62], or conditioned normalizing flows [63].
Multi-rate forecasting is sometimes acceptable to aggregate high-frequene series or event data to lower frequency prior to sequence modeling, e.g., aggregating patient events over 6-month intervals [48], or network traffic over 10 subframes [79]. When high-frequency predictions are required, fwwer-frequency forecasts can provide guidance. Techniques for temporal disaggregatton convert frequency series to higher frequencies via related high- figh-freeuency indicator series [13, 67]. Athanasopoulos et al. [2] simultancously forecast multiple frequencies and then reconcile the predictions. Such methods do not provide uncertainty estimates nor are they based on neural networks.
Scaleformer [70] aggregates series, forecasts at the lower frequency, and upsamples the lower-freq predictions order to provide additional inputs for the original forecast model. We evaluate a similar approach, Low2HighFreq ($3), as a baseline. Other neural approaches (in forecasting and beyond) implicitly encourage learning of lower- frequecyy dynamics through hierarchical attention [20, 47, 16),
In Sanskrit, a sutra is thread or "that which like thread runs through or holds everything together" [85]. A SutraNet prediction woven from threads of sub- subsseries predictions; Fig, 2 shows different stitching patterns.

---

pooling blocks [92. 87, 49, 93], and deeply-stacked [29, 81] and multi-rate [68, 18, 38, 73, 14, 11] recurrence layers. Unlike these approaches, SutraNets explicitly generate low-ureyyency sub-series conditional on other owwfrrequency sub-series, without directly forecasting at high frequency at all.
Reducing training/inference discrepancy we know a priori how far to forecast, it is possible to avoid error accumulation by directly forecasting all horizons in one step [76], or via non-autoregrrssive
decoding [20]. Such systems usually generate point predictions [92. 87, 91, although predicting certain fixed forecast quantiles is also possible [84, 20]. ProTran [77] is probabilistic but non- autoregressive approach. Without modeling local dependencies among outputs, such models prevent generation of coherent samples, and preclude computing robust likelihoods for given output sequences - -where error accumulation is not an issue (e.g., s. for anomaly detection, missing value interpolation, compression, etc.). SutraNets are more in-step with flexible autoregressive models like GPT3 [8].
For autoregressive models, scheduled sampling [5] aims to handle errors better during inference by sampling (incorrect) inputs during training. This approach was effective in tasks such as human motion synthesis [45], but results have been mixed autoregressive forecasting [65, 66). Noting that scheduled sampling produces biased estimator, Lamb et al. [40] instead used adversarial techniques to reduce training/inferenee discrepancy. Wu et al. [88] applied this method to forecasting, using a GAN [26] with autoregressive sparse transformer. However, adversarial training showed insignificant gains over the sparse transformer alone on 7-day electricity and traffic predictions.
Like scheduled sampling, dropout [74] is kind of stochastic regularization that prevents overfitting by injecting noise into networks. Dropout rates of around 20% are commonly used in input layers of autoencoders and feed-forward nets [3, 74]. Dropout is also common in autoregressive models [82, 61], where, in theory, it may encourage networks to rely less on (now noisy) previous values [56].
Modeling long-term dependencies When forecasting long series, seasonally lagged values- historical values from, e.g., one year ago - can be used as extra features in the sequence model, acting a kind of residual input [31]. A recent competition featured long time series (e.g., 700 days of history) [1]; the winning model found lag to be more effective than fixed attention weights [75]. Some researchers have even declared that, when it comes to attention, "lag is all you need" [19].
Learned attention can capture long-range dependencies [82], but infeasible for very long sequences due to memory and time complexity quadratic in sequence length. Complexity also scales quadrat- layers [57. 9]. Recent work has explored sparsified or
ically for forecasting via hierarchical attention in forecasting [44, 92, 47, 87]. TimesNet [86] transforms univariate time series into 2D tensors of multiple periods, which are then processed via Inception-style 2D kernels to generate point predictions. As mentioned earlier, most of these systems fail to condition on more than very short windows - a few days of hourly data. Ultimately, we are interested in probabilistic forecasting using years of fine-grat fine-grained historical data. It is unclear whether lag is really all need, or even whether seasonal-naive baselines [34] are already more effective than Transformers [32].
Unlike Transformers, RNNs can be applied to long sequences out-of-the-bux. However, since RNN signal path is linear in sequence length, mechanisms are required to preserve long-range info, such as the special gated units of LSTMs [33]. Multi-dimension via row and column connections across images [78, 81J. Didolkar et al. [16] combine RNNs for long-range signals with Transformers that attend over shorter chunks. SutraNets. in contrast, do not require architectural changes within RNNs; indeed, they work with RNNs or Transformers. In SutraNets, now-frequency information is both implicitly captured in network state and explicitly and autoregressively communicated through generated lowffrquency forecasts.
RNNs [30] propagate distant info
Comparison to other RNNs Table 1 compares SutraNets prior RNNs (cf. Table in [82]). PatchTST [55] groups consecutive values into input patches. While patching was proposed as method to reduce the attentional complexity of Transformers, patching was also recently used with RNNs in SegRNN [46], where it effectively reduces the signal path by a factor of K (taking in K inputs each step like SutraNets - but with K times fewer overall input steps). However, neither PatchTST nor SegRNN provides a mechanism to probabilisticall generate patches, and thus generate coherent probabilistic outputs. Dilated LSTMs [83] and Dilated RNNs [10] with minimum dilation amounts > do facilitate error reduction and parallel training, but at the cost of sacrificing coherency, as the model is then "equivalent to multiple shared- shared-weight networks, each working on partial inputs"[10 [10, $4.4]. With such DilatedRNNs, if the time series were to spike to a high level at
3

---

Table 1: Comparison of SutraNets to prior RNNs. Like prior methods, SutraNets reduce signal path, but also improve error accumulation (via a larger generative stride), while enabling greater training parallelism (via fewer sequential operations), without sacrificing coherent probabilistic output.
Signal path
Generative max stride
Sequential operations
Coherent
RNN method
Meaning of K
outputs
Standard RNN, e.g., LSTM [33]
N/A
O(N)
O(1)
((N)
Yes
Skips [91], lags/residuals [31]
Skip/lag amount
(NN/K)
O(1)
O(N)
Yes
PatchTST [55]. SegRNN [46]
Patch size/stride
(NN/K)
N/A
((N/K)
N/A
DilatedRNN [10]. dilations C.
Largest dilation
ONNKK)
O(C)
O(N/C)
If C=
SutraNets
Num. sub-scries
O (N/KK
O(K)
O(N/K)
Yes
the very final conditioning input, only subset of the outputs would be generated conditional on this spike, resulting in highly incoherent output. This can be mitigated for point predictions by adding a final fusion layer [10]. but such an approach is not compatible with probabilistic forecasts.
Another related approach is the Subscale WaveRNN model for audio generation [35] (conditioned on input text). For Subscale WaveRNN, the motivation is not to reduce signal path nor improve error accumulation, but to have K-fold increase in inference parallelism when generating very, very long audio sequences. Generation proceeds somewhat similar to our regular, non non-alternating version ($3) but conditioning on past and some future values, and with a shared neural network for all sub-series.
3 SutraNets: Sub-series autoregressive networks
Let Ur E R be the value of time series at time I, and x be a vector of time-varying features or covariates. Let Yay denote sequence y1 ....y We seek model of the distribution of N future values of yt (the prediction range) given T historical values (the conditioning range). We can formulate this conditional distribution as an autoregressive (AR) generative model in [28]:
Uj-
T+N II
p XITT
ply 11 E=
Following DeepAR [65]. forecasting approaches have modeled the one-step-ahead distributions using RNNs [21, 65, 60, 7] or Transformers [44]. In these approaches, a global sequence model trained by slicing many training series into many windows, i.e., conditioning-preddie prediction ranges at different start points, and normalizing the windows using their conditioning ranges. Model parameters are fit via gradient descent, minimizing NLL of prediction-range outputs. At inference time, forecasts for given conditioning range are generated by sampling ÚT+1.... 4T sequentially, autoregressively conditioning on samples generated at previous timesteps. By repeating this procedure many times, Monte Carlo estimate Eq. (1) is obtained, from which desired forecast quantiles can be derived.
SutraNets consider each univariate window to be comprised of an ordered collection of K lower- frequency sub-series, each 1/K of the original length (Fig. 1). The kth sub-series is obtained by selecting every Kith value in the original window, each sub-series starting at unique offset in 1... K.² Let y² be the kth sub-series. Let Ñ N/K and T = T/K be prediction and conditioning lengths corresponding each sub-series, and let < and > h denote indices from K that are less than and greater than k. Through another application of the chain rule, now across sub-series,
Eq. (1) becomes:
T+N K
II
II
plu 11 :
7+1k=1
where i iterates over timesteps in each sub-series. Log-likelihoods of sub-series prediction ranges can be summed compute overall NLL, which is minimized during SutraNet training. Rather than modeling these conditionals using a single RNN, SutraNets use separate RNN for each sub-series.
Since the kth sub-series is assigned after slicing training/testing windows, it will vary in terms of which offset it corresponds in the original time series, e.g., it may start 1pm, 2pm, 3pm, etc., in diffeernnt windows.

---

RNNo
RNN
RNNO RNNI RNNS
KNNN
200
x
5
3N
80
(a) Standard RNN, e.g., C2FAR
(b) Low2HighFree
(c) Regular-all alt SutraNet
RNNO
RNN RNNI RNNO
RNN1
RNNN
96
@
30
(d) Regular-nor non SutraNet
(e) Backfill non SutraNet
(f) Backfill alt SutraNet
Figure 2: Ordering of generative steps in SutraNets. One output value (bold border) is generated in each row, conditional on (I) feature nodes (dashed border) in that row, and on (2) state from previous steps ( arrow connections). SutraNet RNÑs (c-f) use features generated in earlier steps by both themselves and other RNNs; these values can even be from future timesteps (backfill cases).
Each RNN has its own parameters and hidden state, but SutraNets are trained and tuned collectively single network. The kth RNN determines the kth conditional probability each timestep:
P-N
II T- k=1
II
plu = ((b)))
where hy mn* (h parameters of parametric output distribution, m(u/105) So while prior models condition yr only on Vt-1- SutraNets autoregressively condition the generation of y on:
and f(-) a function mapping the kth RNN state to
Current values (at i) of sub-series with indices <k (always)
2. v/_1
The previous value (at & 1) of the current sub-series, (always)
3. M >k. Previous values (at f 1) of sub-series with indices k (optional)
Conditioning b y. makes the model autoregressive across sub-series; this ensures sub-series are generated conditional on each other, and thus (recombined) samples of the original high- sigh-frequency series will be coherent. Conditioning on y is optional; when using y SutraNets must generate in alternating manner, generating one value for each sub-series at cach timestep, i (Fig, 20). Without - SutraNets can generate the complete prediction range of the kth sub-series before generating any predictions for k ones (Fig. 2d); in other words, we can exchange ordering of the >k from the equation, and proceed in non-alternating manner.
products in Eq. (2), remove y
"Note we can input any subset of the covariates to the RNN at each timestep. Here we include those specific to the given timestep of the given sub-series. x as in practice most covariates are either static across the prediction range (e.g.. product ID), or provide timestep-specific info (e.g.. the of of-r-o---h--aaa the--ayeyyy [65].

---

SutraNets must also specify how the sub-series index. k. relates to the offset in the original series. Figs. 2c and 2d illustrate regular ordering, where sub-series l begins at offset in the original series. An alternative is backfill ordering, where sub-series k starts at offset K-k+1 (Figs. 2e, 2f). We must also specify the number of sub-series, K. We piek K so it divides into the primary seasonal period (e.g., K=6 for 24-hour seasons), as each sub-series then exhibits seasonality; signal processing techniques can reveal seasonality when it is not known a priori. Sub-series with seasonality may be more predictable, which will be especially useful for non-alternating models, as these models generate initial sub-series while conditioning on only 1/K of historical values. Success for these models depends on whether there is redundancy in the series, or whether values from other sub-series are sufficiently important that non-altemating generation creates a missing information problem.
Reducing training/inference discrepancy, signal path Non-alternating and backfill approaches force the network to predict K steps ahead, without access to true previous values. Non-alternating also generates far horizons in fewer steps (e.g., for the æ=1 sub-series), reducing error accumulation. In contrast, Regular-al alt generates in the same order, and conditional on the same values, as standard RNN models (Fig. 2a), and so should suffer the same effects from training/inference discrepancy.
All SutraNets reduce RNN signal path by a factor of K, as can be seen by tracing paths between features and outputs in Fig. 2. Note Regular-alt has low signal path but standard discrepancy (as noted above). The relative effectiveness of Regular-alt can thereby provide a diagnostic for whether discrepancy or signal path affects a given forecasting task (54).
Form of output distribution and input encoding Unlike sampling text from a softmax, forecasting requires parametric output distribution, p(vş105 (05) (Eq. (3)), that can account for mixed discrete and continuous outputs, potentially of unbounded dynamic range. C2FAR [7] proposes such an output distribution, via an autoregressive generative model over a hierarchical coarse-to-fing discretization of time series amplitudes (with Pareto-distriutted tails). We adopt C2FAR-LSTMs for our sub-series sequence models. We use 3-level C2F model with 12 bins at each level, discretizing normalized values to a precision of 123 but requiring only 12 x 3 output dimensions at each timestep. Previous values from each sub-series and covariate features from other sub-series provided as inputs at each step. Here the efficiency of C2FAR proves helpful; at each step we can encode K covariate values (i.e., from K other sub-series) using only 12 x K input dimensions, as opposed to, e.g., the 123 x K dimensions that would be required with equivalent flat binnings [60].
also C2F-encoded and
Low2HighFreq Instead of generating all lower-frequency sub-series, experiment with generat- ing a single low-freq sub-series and conditioning on it within a high-freq model. This Low2HighFreo approach (Fig. 2b) is similar to multi-rate approaches ($2), except here low-freq samples provide not only features, but hard constraints on every corresponding K'th value in the high-freq output. During inference, when we use Monte Carlo sampling to estimate p(yr+17T+ I....... ...... high-freq values conditional on low-free ones. As such, techmiques such as rejection sampling [53] are applicable; we experimented with importance sampling via likelihood weighting (fixing the low-freq values and sampling the rest), but ultimately found the likelihood weights had such high variance that it was more effective to use uniform weights and accept inconsistent estimation.
we can actually sample
Complexity Let H be the number of RNN hidden units and K the number of sub-series. The complexity-per pepplexxty--er--imetteee per-timestep of each SutraNet RNN scales with the sum of the RNN's recurrence, H H. and projection of (at most) K inputs (from other sub-series) to the RNN hidden state, K x H. Even though SutraNets have K separate RNNs, only one runs at each original timestep (i.e.. each SutraNet RNN runs 1/K of total timesteps). In consequence, SutraNet complexity-per-tieesst H x H + K x H, whereas standard RNNs scale X H. In practice, >> K, and we find empirically SutraNets also scale H x H. resulting in similar training/inference speeds and memory requirements compared to standard models, given the same individual LSTM depth and width ($-4)
scales with
One key advantage of SutraNets is that training decomposes over sub-series. Sub-series RNNs do not condition on hidden states from other sub-series RNNs, rather only on generated values. During training, when we have access to the true values of other sub-series, all inputs are known in advance and sub-series RNNs can be trained in parallel. The only sequential action is evolution of each RNN's hidden state over 1/K fewer steps than standard or multidimenssinal RNNs [78, 81]. In this way, SutraNet-RNNN provide middlee-gouud between RNNs and standard Transformers (with no sequential computation in training). Inference in SutraNets, like all AR models, is sequential [36].

---

Application to Transformers Rather than using K RNNs to parameterize the conditional proba- bilities as in Eq. (3), we could instead use K autoregressive Transformers. Compared to a standard Transformer, backfill and non-alternating sub-series Transformers would have the advantage of reducing error accumulation (by increasing the generative stride), but not of signal path (already ((1) for Transformers [82]). Moreover, at each timestep, a sub-series Transformer could attend to essentially all prior values, limited only by the generative order. In a naive implementation. SutraNet Transformers would therefore have an asymptotic complexity of ( - the same as standard Transformers. However, we can attain O(Nz/K) complexity by restricting each sub-series Transformer only attend to values from its sub-series, plus few proximal values from other sub-series (similar to strided or banded self-attention [12. 8)). Although asymptotically larger than the O(N (1og 2 of LogSparse attention [44], it merits empirical investigation to determine which approach achieves the most favorable accuracy complexity trade-off.
Limitations and broader impact SutraNets are more complex than standard sequence models in a few ways. First, they require more decisions, such as specifying the number of sub-series. While superior accuracy can be obtained when making these choices heuristically, we evaluate the extent to which tuning can achieve further gains ($4). Secondly, having K separate RNNs results in roughly K times as many total parameters. Although, as noted above, this does not translate into requiring more memory or training/inference time, SutraNet models are larger store and transmit.
As a flexible predictor of both long and short horizons (and without needing a priori specification of confidence quantiles), SutraNets could also serve as the basis for seneral-purpose large forecasting model (LFM), along the lines of LLMs like GPT3 [8]. Like modern LLMs, LFMs could potentially be used with little task-specific data and no fine-tuning, and may have similarly-large environmental and financial costs [4]. As with L.Mss, new techniques will be needed to detect [64, 71, 54), document [51, 22), and mitigate [25, 58) unsafe usage and other failure cases.
4 Experiments
Training. We implemented SutraNets in PyTorch [59]. We use Adam [37] (default 31=0.9, 32=0.999. e)0--), early stopping, and a decaying learning rate (7=0.999). The default RNN is I-layer LSTM with 64 hidden units; when using >1 layers (Table 3), inter-layer dropout [50] with =0.0p=0.001 is applied. For every model/dataset pair, we tune weight decay and initial learning rate over a 4x 4 grid.
Systems. We evaluate the following systems. Naïve and SeasNaive are non-probabilistic, untrained baselines, while the C2FAR variants follow the standard RNN ordering (Fig. 2a):
Naïve: At each horizon, repeats last-observed historical value [34].
SeasNaive: At each horizon, repeat previous value at same season as horizon [34], e.g., same hour-of-day (SeasNaiveld) Id) or cour-of-week (SeasNaivelw).
C2FAR: LSTM-based forecasting model [7], configured as described above ($3).
C2FAR+lags: C2FAR, but with extra features for previous hour-of-day values (C2F-encoded).
C2FAR+dropout: C2FAR, but with =0.2 (and re-scaling (74)) applied to inputs during training.
Low2HighFreq: High-frequency forecast generated conditional low-frequency values ($3).
SutraNets: Variations Regular-alt, Regular-non, Backfill-alt, Backfill-non ($3).
Datasets. We evaluate on the following datasets, where we note the number of SutraNet sub-series, K, as well as lengths of conditioning, C. and prediction, P, ranges:
azure: K=6, C=2016, P=288:
5-minutely usage of 2085 VM flavors, groupings (by tenant, etc.) in
Azure cloud. Originally from [15]. aggregated here instead to 5-minute intervals.
elec: K=6, C=168, P=168: Hourly electricity usage of 321 customers [17], version from [39].
traff: K=6, C=168, P=168: Hourly occupancy for 862 car lanes [17], version from [89].
wiki: K=7, C=91, P=91: Daily hit-count for 9535 pages, first used in [21], version from [27].
mnist: K=4, C=392, P=392: row-wise sequential pixel values for 70,000 28 28x28 digits [42].
mnist": K=4, C=392, P=392: mnist sequences, but with fixed, random permutation applied.
7

---

Table 2: ND%, wQL% across datasets, best results in bold. For mnist/mnistf, previous row value used for SeasNaive Poor results for C2FAR relative to Regular-alt indicate signal path problem (green cells). Poor results for both C2FAR and Regular-alt indicate discrepancy problem (pink). Non-alternating weakness indicates missing info problem (yellow). Backfill-ai alt reduces signal path and discrepancy without missing info, therefore offering best or near-best solution on each dataset.
azure
elec
traff
wiki
monist
mnist"
Naive
3.2, 3.2
43.0, 43.0
75.5, 75.5
44.0, 44.0
100.0, 100.0
117.9, 117,9 163.5, 163.5 N/A
ScasNaiveld
2.8, 2.8
10.5, 10.5
31.1, 31.1
44.0, 44.0
145.2, 145.2 N/A
ScasNaive
5.2,
11.1, 11.1
17.5, 17.5
41.8,
C2FAR
3.5,
10.6, 8.4
19.3, 16.0
31.1,
67.9, 52.3
100.0, 95.7
C2FAR+lags
2.7
10.4, 8.3
18.0, 15.0
27.2
68.1,
99.3, 89.9
CdFAR+dropout
2.6
10.9, 8.7
18.5, 15.3
31.3,
68.6,
99.7, 90.4
Low2HighFreq
2.6.
10.3, 8.2
20.2, 16.6
30.9,
62.4, 47.6
98.1, 88.3
Regular-alt
3.6
7.9
15.5, 13.0
30.4, 26.4
67.9,
48.7, 38.3
Regular-non
2.6,
9.7, 7.8
15.6, 13.1
31.2, 27.3
59.4,45.4
90.1, 69.7
Backfill-alt
2.5.
9.3, 7.4
15.3, 12.8
30.1, 26.1
64.4, 49.7
72.0, 54.6
Backfill-non
2.3,
9.3,7.4
15.7, 13.1
30.5, 26.8
58.9, 44.9
78.2, 58.3
Sequential (binarized) MNIST has been used for NLL [40] and classification [41] evaluations. Le et al. [41] used a fixed permutation "to make the task even harder." Here, we forecast a distribution over real-valued pixel intensities in the second half of an image, based values in the first. Unlike the other datasets, train/dev/test splits are not based on time, rather we use the standard test digits [42].
Metrics. We evaluate the median of our forecast distribution by computing normalized deviation (ND) from true values. We evaluate the full distribution using weighted quantile loss (wQL), evaluated at forecast quantiles f0.1, 0.2 ... 0.9), in [60, 7]. For non-probabillittc Naïve and SeasNaive baselines, note wQL reduces to ND, similarly to how CRPS reduces to absolute error [24, $4.2].
Results. We summarize the most important experimental findings as follows:
Finding 1:
Backfill-alt and Backfill-non improve over C2FAR and other baselines on every dataset, with both obtaining an average relative reduction in ND of around 15% over the best non-SutraNet.
Table 2 has the main results; note the differences in ND% are very stable: we repeated grid search tun- ing and testing multiple times with different random seeds and found SutraNets superior to standard models across all repeats (supplemental $B.2). Backfill-non does best when the training/inference discrepancy problem predominates (azure & mnist, Table 2). Adverse effects of discrepancy seem to be magnified by high data redundancy; in these cases, non-alternating generation reduces discrepancy while having sufficient info to forecast accurately. We also hypothesize that Regular- -nn performs poorer because it generates less-coherent output: by construction, it only conditions on past values, even when subsequent values have already been generated (e.g., row 7 in Fig. 2d). In contrast, when generating a given point (e.g., row 7 in Fig. 2e), Backfill-non has always seen the previous and future proximal values, which were input at the previous and current timesteps, respectively. When signal path problems predominate (traff & mnist"), alternating generation, with access to more info, performs better. On mnist", differences between regular and backfill depend solely on the random permutation; the superiority of Regular-alt here motivates tuning over multiple orderings of sub-series, perhaps averaging over them, as is done in some prior non-sequential models [23, 80).
Accuracy gains seem derive from solving both discrepancy and signal path problems together. When discrepancy predominates, Regular-alt performs poorly, even though signal path is improved. For the opposite case, where error accumulation alone is improved, we implemented an RNN that processes values in backfill order, but otherwise acts as a standard RNN. This model performs much poorer than SutraNet- based Backfill-alt (supplemental $B.4). The lack of a holistic solution may also explain the inconsistency of lags and dropout, which sometimes help, sometimes hurt compared to vanilla C2FAR. Low2HighFreq works best on datasests where discrepancy predominates, while performing much poorer when signal path is important. Note that weekly-seasons baselines have not previously been evaluated on these datasets. Observing particular the relatively strong accuracy of ScasNaive Iw on traff, we suggest they be included in future evaluations.
8

---

Backfill-ait
NeasNalleeww
C2FAR
forecast
80% C.I.
Figure 3: Forecasts and 80% intervals for elec series. Backfill-alt performs well with a changepoint (row 1, where SeasNaivel w fails) and with weekly seasonality (row 2, where C2FAR fails).
Backfill-non
C2FAR
Backfill-non
C2FAR
Actual p25% p50% p75%
Actual p25
550%
p75%
Actual
p25%
p50%
p75%
Actual p25%
p50%
p75%
5
1
1
I
0
0
2. 7
2
2
7 7
7
7
7
0
7
0
7
0
0
3
7
3
2
3
7
3
0
8
3
0
0
8
8
0
0
3
0
4 4
Figure 4: Percentiles of forecast distributions for first occurrence of ....
9 in mnist test data, Pixels
before yellow line (in row-order) are history. Backfill-non more confident in bottom part of digits.
Finding 2: SutraNets are able to effectively model both long and short-term dependencies.
On elec data, where Neavel is more effective than longer-term SeasNaivel w (Table 2), model- ing long-term dependencies remains important (Fig. 3). Backfill-alt is able to capture both short-term changes and long-term seasonality, the latter of which is neglected by C2FAR. In plots of the mnist forecasts (Fig. 4), we see that compared C2FAR, Backfill-non generates more non-zero values in the bottom parts of digits 0, 2, 5, etc., e.g., at p50%, implying the networks have more certainty or confidence (i.e., tighter prediction intervals) on the digit being generated. Also, Table results indicate that discrepancy is the predominant problem on mnist; this may help explain why autoregres- sive models like PixcIRNN [81] score very high in likelihood-basee evaluation (where there is no sampling). but produce less-rcalistic samples compared to GANs. SutraNets may therefore provide useful tool for reducing training/inference discrepancy in a variety of image generators.
Finding 3: SutraNets are more accurate than C2FAR across all forecast horizons.
Autoregressive models are normally thought to accumulate and compound [56] errors "along the generated sequence" [5]. We therefore expect the accuracy gap between SutraNets and C2FAR to grow across horizons. Instead, on elec and traff' we already see a large gap over the first 24 hours (supplemental BB.3). Improvements in both informational asymmetry ($1) and signal path evidently enable gains across all steps. Low2HighFreq, in contrast, is only accurate at horizons corresponding its low-freq forecast, while Backfill-alt is slightly better at horizons corresponding to its first sub-series. This suggests errors do "accumulate" to a small extent across sub-series autoregression.
Finding 4: Deeper models enable improved long-sequence forecastingg for SutraNets and C2FAR.
In particular, going from 1 to 2 layers greatly improves accuracy on traff (Table 3), dataset where signal path is the predominant problem. Smaller improvements are also seen on elec. Since both LSTM stacking and use of SutraNets increase complexity, a natural question is which approach offers the best computational tradeoff for desired level of accuracy. Fig. 5 shows that for given number of parameters and especially for given inference time SutraNets are able to obtain better

---

Table 3: ND% decreases for SutraNets & C2FAR
Table 4: ND% lowest when #sub-series di- vides into seasonal period (24), but alternat- ing models less sensitive on elec (cf. C2FAR with 10.6% elec, 19.3% traff.)
number of RNN layers/hidden units increases.
Dataset
traff
#layers
2
3
3
#hidden
64
128
256
256
64
128
256
256
Dataset #subseries
elec
traff
7
12
12
C2FAR
10.6
10.2
9.9
9.7
19.3
14.7
14.8
14.4
Regular-alt
9.9
9.3
9.0
9.2
15.5
14.3
14.3
14.2
Regular-alt
9.9 10.0
10.0
15.5
17.2
15.0
Backfill-alt
9.3
9.3
9.0
9.3
15.3
14.2
14.1
14.0
Backfill-alt
9.3
9.4
9.4
15.3
16.9
14.9
Backfill-non
9.1
9.5
9.3
15.7
14.4
14.4
Backfill-non
10.0
17.3
15.4
Inf. time (8) per forecast
Inf time (8) per forecast
9.7
9.3
14.4
0.4
9.0
14.1
[4.8
0.2
0.2
10.2
14.7
9.3
C2FAR Backfill-alt
14.2
C2FAR Backfill-aft
10.6
9.3
19.3
15.3
clec: Total number of parameters -107
traff: Total numher of parameters
Figure 5: Total parameters (x-axis) and average inference time per forecast (y-axis, lower better) for C2FAR and Backfill-alt systems of Table 3, for elec (left) and traff (right). ND% labeled at each point in boxes Backfill-alt enables more parameters and better ND% at equivalent speeds.
ND% (plots of memory usage and training time are similar, see supplemental). SutraNet speed and memory usage scale with the complexity of each RNN rather than the number of sub-series RNNs ($3); they can consequently make use of x more parameters without Kx the cost.
Finding 5: Backfill-ali is robust across different configurations, including when we vary K.
We also experimented with varying the number of sub-series. For all K>1, SutraNets work much better than standard C2FAR (Table 4). Interestingly, K=12 led to the highest accuracy on traff, where the signal path problem predominates. When K does not divide evenly into the seasonal period, non-alternating models have a missing info problem (e.g., they miss predictive preceding values at the same hour-of-day). Both Backfill-alt and Regular-alt can access these values through covariate features (cf. Fig. 2f), and hence perform relatively better when K=7 on elec. Backfill-alt is relatively weaker at K=7 when there is less redundancy (traff), although still superior to standard C2FAR.
See the supplemental for further experimental details and results. Code for SutraNets is available at https: / / thub. com/huaweicloud c2a.r
Conclusion
Probabilistic forecasting of long time series an under-studied problem, with most recent work focusing point prediction. We presented SutraNets, novel autoregressive approach to probabilistic forecasting of long-sequence time series. SutraNets convert a univariate series into a K K-dimmesioonal multivariate series, each dimension comprising distinct every- KKth- sub-series of the original sequence. The SutraNet autoregressive model generates each sub-series conditional on both its own prior values, and on other sub-series, ensuring coherent output samples. From these samples, an estimate of the full joint probability of future values is made. SutraNets leverage C2FAR under-the- hood, allowing for efficient representation of time series amplitudes, and low-overhead input encoding of covariate sub-series. SutraNets provide holistic solution to challenges in long-sequence sampling, reducing training/inference discrepancy (by generating conditional on longer-term information) and signal path distances (by factor of K). Experimentally, SutraNets achieve state-of thie--of-hhhartt results on a variety of long- sequeeeee forecasting data sets, while demonstrating similar speed and memory requirements standard sequence models, even when using K x more parameters.
10

---

References
[1] Oren Anava and Vitaly Kuznetsov. 2017. Web Traffic Time Series Forecasting. https: Seemm--erres--ooeece
//www. wttt:////////to///oiiiie
[2] George Athanasopoulos, Rob Hyndman, Nikolaos Kourentzes, and Fotios Petropoulos. 2017. Forecasting with temporal hierarchies. European Journal of Operational Reseurch 262, (2017), 60-74.
[3] Pierre Baldi and Peter J Sadowski. 2013. Understanding Dropout. Advances in neural informa- lion processing systems 26 (2013).
[4] Emily M Bender, Timnit Gebru, Angelina McMillan- MMjior,,-MMaor, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. In ACM Conference on Fairness, Accountability, and Transparency. 610-623
[5] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. Advances in Neural Information Processing Systems 28 (2015).
[6] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research 3 (2003), 1137-1155.
[7] Shane Bergsma, Tim Zeyl, Javad Rahimipour Anaraki, and Lei Guo. 2022. C2FAR: Coarse- to-Fine Autoregressive Networks for Precise Probabilistic Forecasting. In Advances in Neural Information Processing Systems, Vol. 35.
[8] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems 33 (2020), 1877-1901.
[9] Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza, Max Mergenthaler-Canseen and Artur Dubrawski. 2022. N-HiTS: Neural Hierarchical Interpolation for Time Series Forecasting. arXiv preprint arXiv:2201 12886 (2022).
[10] Shiyu Chang, Yang Zhang. Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark A Hasegawa-Johnson, and Thomas Huang. 2017. Dilated recurrent neural networks. Advances Neural Information Processing Systems 30 (2017).
[11] Zipeng Chen, Qianli Ma, and Zhenxi Lin. 2021. Time-Aware Multi-Scale RNNs for Time Series Modeling.. In International Joint Conference On Artificial Intelligence. 2285-2291.
[12] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. arXiv preprint arXiv: /904. 10509 (2019).
[13] Gregory C Chow and An-loh Lin. 1971. Best linear unbiased interpolation, distribution, and extrapolation of time series by related series. The Review of Economics and Statistics (1971), 372-375.
[14] Junyoung Chung. Sungjin Ahn, and Yoshua Bengio. 2017. Hierarchical Multiscale Recurrent Neural Networks. International Conference on Learning Representations (2017).
[15] Eli Cortez, Anand Bonde, Alexandre Muzio, Mark Russinovich, Marcus Fontoura, and Ricardo Bianchini. 2017. Resource Central: Understanding and Predicting Workloads for Improved Re- source Management in Large Cloud Platforms. In Symposium on Operating Systems Principles. 153-167.
[16] Aniket Didolkar, Kshitij Gupta, Anirudh Goyal, Nitesh Bharadwaj Gundavarapu, Alex M Lamb, Nan Rosemary Ke, and Yoshua Bengio. 2022. Temporal Latent Bottleneck: Synthesis of Fast and Slow Processing Mechanisms in Sequence Learning. Advances in Neural Information Processing Systems 35 (2022), 10505-10520.
[17] Dheeru Dua and Casey Graff. 2017. UCI Machine Learning Repository. http://archive. ics. uci, edu/ml
11

---

[18] Salah EI Hihi and Yoshua Bengio. 1995, Hierarchical Recurrent Neural Networks for Long-Term Dependencies. Advances in Neural Information Processing Systems (1995).
[19] Christos Faloutsos, Valentin Flunkert, Jan Gasthaus, Tim Januschowski, and Yuyang Wang. 2019, Forecasting Big Time Series: Theory and Practice. In ACM SIGKDD International Conference on Knowledge Discovery & Duta Mining. 3209-3210.
[20] Chenyou Fan, Yuze Zhang, Yi Pan, Xiaoyue Li, Chi Zhang, Rong Yuan, Di Wu, Wensheng Wang, Jian Pei, and Heng Huang. 2019. Multi-Horizon Time Series Forecasting with Temporal Attention Learning. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2527-2535.
[21] Jan Gasthaus, Konstantinos Benidis, Yuyang Wang. Syama Sundar Rangapuram, David Salinas, Valentin Flunkert, and Tim Januschowski. 2019. Probabilistic Forecasting with Spline Quantile Function RNNs. In International Conference on Artificial Intelligence and Statistics. 1901- 1910.
[22] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2021. Datasheets for Datasets. Commun, ACM 64, 12 (2021), 86-92.
[23] Mathieu Germain, Karol Gregor, lain Murray, and Hugo Larochelle. 2015. MADE: Masked Autoencoder for Distribution Estimation. International Conference on Machine Learning. 881-889.
[24] Tilmann Gneiting and Adrian E Raftery. 2007. Strictly Proper Scoring Rules, Prediction, and Estimation. J. Amer. Statist. Assoc. 102, 477 (2007), 359-378.
[25] Hila Gonen and Yoav Goldberg, 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. arXiv preprint arXiv: (2019).
[26] Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets, Advances in Neural Information Processing Systems (2014), 2672-2680.
[27] Adèle Gouttes, Kashif Rasul, Mateusz Koren, Johannes Stephan, and Tofigh Naghibi. 2021. Probabilistic Time Series Forecasting with Implicit Quantile Networks. arXiv preprint arXiv:2107.03743 (2021).
[28] Alex Graves. 2013. Generating Sequences with Recurrent Neural Networks. arXiv preprint 1308. 0850 (2013).
[29] Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech Recognition with Deep Recurrent Neural Networks. In 2013 IEFE International Conference Acoustics, Speech and Signal Processing. 6645-6649.
[30] Alex Graves and Jürgen Schmidhuber. 2008. Offline handwriting recognition with multidi- mensional recurrent neural networks. Advances in Neural Information Processing Systems 21 (2008).
[31] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In IEEE Conference on Computer Vision and Putlern Recognition. 770-778.
[32] Hansika Hewamalage, Klaus Ackermann, and Christoph Bergmeir. 2023. Forecast evaluation for data scientists: common pitfalls and best practices. Data Mining and Knowledge Discovery
37, 2 (2023), 788-832.
[33] Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural computation 9, (1997), 1735-1780.
[34] Rob Hyndman and George Athanasopoulos. 2018. Forecasting: Principles and Practice. OTexts.
12

---

[35] Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron van den Oord, Sander Dieleman, and Koray Kavukcuoglu. 2018. Efficient Neural Audio Synthesis. In International Conference on Machine Learning. 2410-2419.
[36] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. 2020. Trans- formers are RNNs: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning. 5156-5165.
[37] Diederik P Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. arXiv preprint arXiv: 1et...12.... 6980 (2014).
[38] Jan Koutnik, Klaus Greff, Faustino Gomez, and Juergen Schmidhuber. 2014. A clockwork RNN. In International conference on machine learning. 1863-1871.
[39] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. 2018. Modeling Long-and Bhrrt-Term Temporal Patterns with Deep Neural Networks. In ACM SIGIR Conference on Research & Development in Information Retrieval. 95-104.
[40] Alex M Lamb, Anirudh Goyal Alias Parth Goyal, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio. 2016. Professor forcing: A new algorithm for training recurrent networks. Advances in Neural Information Processing Systems 29 (2016).
[41] Quoe V Le, Navdeep Jaitly, and Geoffrey E Hinton. 2015, A Simple Way to Initialize Recurrent Networks of Rectified Linear Units. arXiv preprint arXiv::5044 1504.0094 (2015).
[42] Yann LeCun, Corinna Cortes, and Christopher JC Burges. 1998. The MNIST database of handwritten digits. attp://yann.. http://aaaanneeunnnnnmmm com/ oxdb/mnist//
[43] Ji You Li, Jiachi Zhang, Wenchao Zhou, Yuhang Liu, Shuai Zhang, Zhuoming Xue, Ding Xu, Hua Fan, Fangyuan Zhou, and Feifei Li. 2023. Eigen: End-to-end Resource Optimization for Large-Scale Databases on the Cloud. Proceedings of the VLDB Endowment 16, 12 (2023),
[44] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. 2019. Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting. Advances in Neural Information Processing Systems 32 (2019).
[45] Zimo Li, Yi Zhou, Shuangjiu Xiao, Chong He, Zeng Huang, and Hao Li. 2017. Auto- conditioned recurrent networks for extended complex human motion synthesis. arXiv preprint arXiv: 1707 (2017).
[46] Shengsheng Lin, Weiwei Lin, Wentai Wu, Feiyu Zhao, Ruichao Mo, and Haotong Zhang. 2023. SegRNN: Segment Recurrent Neural Network for Long-Term Time Series Forecasting. arXiv preprint ar0iv: / 1200 (2023).
[47] Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. 2021. Pyraformer: Low Low-complexity pyramidal attention for longrange time series modeling and forecasting. In International Conference on Learning Representations.
[48] Xing Han Lu, Aihua Liu, Shih-Chieh Fuh, Yi Lian, Liming Guo, Yi Yang, Ariane Marelli, and Yue Li. 2021. Recurrent disease progression networks for modelling risk trajectory of heart failure. PloS one 16, (2021).
[49] Kiran Madhusudhanan, Johannes Burchert, Nghia Duong-Trune Stefan Born, and Lars Schmidt- Thieme. 2021. Yformer: U-net inspired transformer architecture for far horizon time series forecasting, arXiv preprint arXiv:2110.0825 (2021).
[50] Gábor Melis, Chris Dyer, and Phil Blunsom. 2017. On the State of the Art of Evaluation in Neural Language Models. arXiv preprint arXiv: (tt.:ii:7777.5..0 170700558 (2017).
[51] Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchin- son, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model Cards for Model Reporting. In Conference Fairness, Accountability, and Transparency. 220-229.
13

---

[52] Srayanta Mukherjee, Devashish Shankar, Atin Ghosh, Nilam Tathawadekar, Pramod Kompalli, Sunita Sarawagi. and Krishnendu Chaudhury. 2018. AR-MDN: Associative and Recurrent Mixture Density Networks for eRetail Demand Forecasting, arXiv preprint arXiv: (2018).
[53] Kevin P Murphy. 2012. Machine Learning: A Probabilistic Perspective. MIT press.
[54] Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. StereoSet: Measuring stereotypical bias in pretrained language models. arXiv preprini arXiv:2004.09455 (2020).
[55] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. 2022. A time series worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.14730 (2022).
[56] James O' and Danushka Bollegala. 2018. Analysing Dropout and Compounding Errors in Neural Language Models. arXiv preprint arXiv: 1.0099 (2018).
[57] Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. 2019. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint 104477 (2019).
[58] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instructions with human feedback. Advances Neural Information Processing
Systems 35 (2022), 27730-27744
[59] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach Vito, Martin Raison, Tejani Alykhan, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems 32 (2019).
[60] Stephan Rabanser, Tim Januschowski, Valentin Flunkert, David Salinas, and Jan Gasthaus. 2020. The Effectiveness of Discretization in Forecasting: An Empirical Study on Neural Time Series Models. arXiv preprint 10X1 (2020).
[61] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. (2018).
[62] Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. 2021. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In International Conference Machine Learning. PMLR, 1857-8868.
[63] Kashif Rasul, Abdul-Saboon Sheikh, Ingmar Schuster, Urs Bergmann, and Roland Vollgraf. 2020. Multi-Variate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows. arXiv preprint arXiv:2002.06100 (2020).
[64] Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. arXiv preprint
(2018).
[65] David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. 2020. DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks. International Journal of Forecasting 36, 3 (2020), 1181-1191.
[66] Matteo Sangiorgio and Fabio Dercole. 2020. Robustness of LSTM neural networks for multi- step forecasting of chaotic time series. Chaos, Solitons & Fractals 139 (2020), 110045.
[67] Christoph Sax and Peter Steiner. 2013. Temporal disaggregation of time series. The R Journal 5, (2013), 80-
[68] Jürgen Sehmidhuber. 1992. Learning Complex, Extended Sequences Using the Principle of History Compression. Neural Computation 4, 2 (1992), 234-242.
14

---

[69] Amazon Web Services. 2023. DeepAR+ Algorithm Amazon Forecast. https://docs. aws. amazon. cttp///rreettt/aaatt//////aaaaaa hatpt//sss////asssooooottttt-tee recipe www.-reeeepippppplie.e.ee html Ac- cessed: 2023-05-01.
[70] Amin Shabani, Amir Abdi, Lili Meng, and Tristan Sylvain. 2022. Scaleformer: Iterative Multi- Scale Refining Transformers for Time Series Forecasting. arXiv preprint arXiv:2206.04033 (2022).
[71] Emily Sheng, Kai- Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The woman worked babysitter: On biases in language generation. arXiv preprint urXiv:190.001322 (2019).
[72] Slawek Smyl. 2020. A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting. International Journal of Forecasting 36, (2020), 75-85.
[73] Alessandro Sordoni, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simonsen, and Juan-Yun Nie. 2015. Hierarchical Recurrent Aware Query Suggestion. In Conference on Information and Knowledge Management. 553-562.
Recooerr for Generative Context-
[74] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research 15, (2014), 1929-1958.
[75] Arthur Suilin. 2017. Kaggle Web Traffic Time Series Forecasting: How it works. https: //github. wtt.//o/////./ coom//rttuuu//kagglll www-wwewwtt-ee-eee www-rrfffttett-tttooeee-------ee ttt.///atter/oowtttttooooo itw_wwkks___... md
[76] Souhaib Ben Taieb and Amir F Atiya. 2015. Bias and Variance Analysis for Multistep-Ahead Time Series Forecasting. IEEE Transactions Neural Networks and Learning Systems 27, 1 (2015), 62-76.
[77] Binh Tang and David S Matteson. 2021. Probabilistic Transformer for Time Series Analysis. Advances in Neural Information Processing Systems 34 (2021), 23592-23608.
[78] Lucas Theis and Matthias Bethge. 2015. Generative image modeling using spatial LSTMs. Advances in Neural Information Processing Systems 28 (2015).
[79] Hoang Duy Trinh, Lorenza Giupponi, and Paolo Dini. 2018. Mobile traffic prediction from raw data using LSTM networks. In 2018 IEEE 29th Annual International Symposium Personal, Indoor and Mobile Radio Communications (PIMRC). 1827-1832.
[80] Benigno Uria, Marc-Alexandre Côté, Karol Gregor, lain Murray, and Hugo Larochelle. 2016. Neural Autoregressive Distribution Estimation. Journal of Machine Learning Research 17, (2016), 7184-7220.
[81] Aaron den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. 2016. Pixel Recurrent Neural Networks. In International Conference on Machine Learning. 1747-1756.
[82] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. Advances in Neural Information Processing Systems 30 (2017).
[83] Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. 2017. Feudal networks for hierarchical reinforcement learning. In International Conference on Machine Learning. PMLR, 3540-3549.
[84] Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. 2017. A Multi-Horizon Quantile Recurrent Forecaster. arXiv preprint arXiv: (2017).
[85] Monier Williams. 1963. Sanskrit- Engllsh Dictionary. Oxford: Clarendon Press.
[86] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. 2022. TimesNet: Temporal 2D-variation modeling for general time series analysis. arXiv preprint 210.02221 (2022).
15

---

[87] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. 2021. Autoformer: Decomposition Transformers with Auto-Correlatio for Long-Term Series Forecasting. Advances in Neural Information Processing Systems 34 (2021).
[88] Sifan Wu, Xi Xiao, Qianggang Ding, Peilin Zhao, Ying Wei, and Junzhou Huang. 2020. Adversarial Sparse Transformer for Time Series Forecasting, Advances in Neural Information Processing Systems 33 (2020), 17105-17115.
[89] Hsiang-Fu Fu Yu, Nikhil Rao, and Inderjit S Dhillon. 2016. Temporal Regularized Matrix Fac- torization for Digh--Dimesiionll Time Series Prediction. Advances in Neural Information Processing Systems 29 (2016).
[90] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. 2022. Are Transformers effective for time series forecasting? arXiv preprint arXiv:2205. /3504 (2022).
[91] Saizheng Zhang, Yuhuai Wu, Tong Che, Zhouhan Lin, Roland Memisevic, Russ R Salakhutdi- nov, and Yoshua Bengio. 2016. Architectural complexity measures of recurrent neural networks. Advances in Neural Information Processing Systems 29 (2016).
[92] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li. Hui Xiong, and Wan- cai Zhang. 2021. Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting. In AAAI Conference Artificial Intelligence. 1110--11155
[93] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. 2022. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. International Conference on Machine Learning. 27268-27286.
16


Reviews:
[{"review_id": "FKqdz88cyM", "reviewer": "Reviewer_iCD8", "summary": "This paper proposes a novel method for probabilistic forecasting of long-sequence time series. It uses an autoregressive generative model to factorize the likelihood of long sequences into products of conditional probabilities.  The proposed model SutraNets treat long and univariate prediction as multivariate prediction over lower-frequency sub-series to effectively reduce error accumulation. Experimental results show improved forecasting accuracy on a variety of datasets.", "questions": "Could this approach be applied to Transformer based models?", "limitations": "The limitations are mentioned in Section 3. However, it doesn't show the limitations to extend it to Transformer based models which are very popular in time-series forecasting recently. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. It's novel to convert a univariate series into a multivariate series, each dimension comprising sub-series of the original sequence.\n\n2. It's interesting that SutraNet model generates each sub-series conditional on both its own prior values and on other sub-series.", "weaknesses": "The proposed model is only applied to LSTM based model, while many of the state-of-the-art time series forecasting model is Transformer based. Not sure whether it can be integrated into Transformer based models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful feedback, and your support for the paper's core idea.\n\nYou raise an important point regarding Transformers: we did not spend enough time describing how SutraNets could be applied to Transformers.  While we did mention in line 45 that SutraNets could be applied to Transformers, in the methods section we only briefly noted (lines 162-163) how attending to SutraNet sub-series could lower Transformer complexity.\n\nWe will add a new sub-section to Section 3:\n\n> **Application to Transformers**\n\n> SutraNets can also be applied with autoregressive Transformer models ([Li et al., 2009](https://arxiv.org/abs/1907.00235)). Rather than using $K$ separate RNNs to parameterize the conditional probabilities in Eq. (3), we can use $K$ separate Transformers.  Compared to a single standard Transformer, using Backfill and non-alternating SutraNet Transformers would have the advantage of reducing *error accumulation* (by forcing the network to predict without immediately-preceding values), but not of *signal path*, which is already $\\\\mathcal{O}(1)$ for standard Transformers ([Vaswani et al., 2017](https://arxiv.org/abs/1706.03762)).  Moreover, at each timestep, a SutraNet Transformer can attend over all prior values, limited only by the generative order.  In a naive implementation, alternating SutraNet models could therefore attend to essentially all prior historical values, resulting in an asymptotic complexity of $O(N^2)$ - the same as standard Transformers.  However, we can achieve $\\\\mathcal{O}(N^2/K)$ attentional complexity by restricting each sub-series Transformer to only attend to values from its own sub-series, plus a small number of proximal values from other sub-series (similar to strided or banded self-attention ([Child et al., 2019](https://arxiv.org/abs/1904.10509), [Brown et al., 2020](https://arxiv.org/abs/2005.14165))).  Although asymptotically larger than the $\\\\mathcal{O}(N^2/K^2)$ complexity of patching (Nie et al., 2023), or the $\\\\mathcal{O}(N (\\\\log N)^2)$ of LogSparse attention ([Li et al., 2009](https://arxiv.org/abs/1907.00235)), it merits further investigation to determine which approach results in the most favorable accuracy vs. complexity trade-off in practice.\n\nIn the limitations section, we will also mention that SutraNets will not improve (nor harm) the parallelization of training in Transformers, since Transformers are already fully parallelizable across the sequence.\n\nFinally, we will do a better job of motivating why we evaluated SutraNets in the context of RNN-based forecasting in the first place.  Notably, our production forecasting system, like many others in industrial settings and cloud services, (e.g., via Amazon Forecast), is based on RNNs.  As our paper shows, SutraNets offer major benefits to such RNN-based systems, as SutraNets are the first approach to advance all three key dimensions of signal path, error accumulation, and parallelization of RNN training."}, {"review_id": "FyOSdoeZlK", "reviewer": "Reviewer_6VCF", "summary": "This manuscript proposes SutraNets for long-range probabilistic forecasting on time series data and pixel sequences. \nSutraNets is a type of recurrent neural network that transforms a long series into a collection of shorter sub-series. Sub-series forecasts are generated autoregressively, sequentially conditional on each other, enabling coherent outputs.\n", "questions": "see weaknesses", "limitations": "Limitations and broader impacts have been discussed in Section 3.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Figure 2 presents the generation ordering in SutraNets well, making it easy to follow.\n\n", "weaknesses": "Since this work focuses on probabilistic forecasting, it should include related work and baselines on time series probabilistic forecasting models. \n\nHowever, most related baselines are missing. For example:\n- NeurIPS’21 Probabilistic Transformer for Time Series Analysis\n- ICLR’21 Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows\n- ICML’21 Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting\n\n“long-sequence” in the title means to predict a long sequence, it would be better to compare recent long-range time series prediction models. This is because the main work in this paper is the designed recurrent structure, rather than a new way to estimate the uncertainty. In other words, the connection between the proposed model and the probabilistic modeling is weak.\n\nMoreover, I believe that the structure of SutraNets is quite similar to recurrent neural networks with skip or residual connections in the temporal direction. And it does not depend on the K. It is unclear what advantages do SutraNets have over RNNs with skip connections. The main differences between SutraNets and the following works should be discussed. \n- NIPS’17 Dilated Recurrent Neural Networks\n- NIPS’16 Architectural Complexity Measures of Recurrent Neural Networks", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your helpful and insightful review. Your pointers to related work will enable us to substantially improve the paper. It is also gratifying to know that Figure 2 was valuable and contributed to the overall good presentation.\n\n### Regarding prior probabilistic forecasting models\n\n> Since this work focuses on probabilistic forecasting, it should include related work and baselines on time series probabilistic forecasting\n\nPlease refer to the section \"Regarding prior probabilistic forecasting models\" in our rebuttal to Reviewer BQ6D for full details on how we will use your suggestions to address this.\n\n### Regarding [Tang & Matteson, 2021](https://proceedings.neurips.cc/paper/2021/hash/c68bd9055776bf38d8fc43c0ed283678-Abstract.html): Probabilistic Transformer for Time Series Analysis\n\nThis approach is worth mentioning in our Section 2 sub-section \"Reducing training/inference discrepancy\", as it is non-autoregressive, but still probabilistic. However, since the predicted outputs do not condition on each other (see their Eq. 2), the output values would not be generated coherently. Without modeling local dependencies among outputs, the model can only provide a very weak assessment of whether a given sequence is likely, e.g., for anomaly detection, interpolating missing values, etc. As mentioned, our goals are more aligned with flexible autoregressive models like GPT3, where not modeling local dependencies would result in, e.g., ChatGPT generating incoherent responses.\n\n### Regarding the connection to probabilistic modeling\n\n> the main work in this paper is the designed recurrent structure, rather than a new way to estimate the uncertainty... the connection between the proposed model and probabilistic modeling is weak\n\nWe will clarify in our intro that, in our use case, quantifying the variance in a predicted output is not a \"nice-to-have\", but essential for downstream decision makers that use our predictions to make risk-aware resource allocation and capacity planning decisions. Computing a full likelihood of a long time series also facilitates other important use cases such as anomaly detection, missing value interpolation, etc. We thus view SutraNets as fundamentally a novel way to compute a joint probability of very long time series forecasts, as opposed to an RNN-specific enhancement. Indeed, you can apply the SutraNet factorization to both LSTMs and Transformers (see our response to reviewer iCD8). While better capturing long-term dependencies *is* one of the advantages of SutraNets in RNNs (and could be applied more broadly), it goes along with reducing error accumulation in autoregression, enabling a $K$-fold improvement in training parallelism, and enabling coherent outputs - advantages that are simultaneously lacking in other proposed probabilistic models, as we shall discuss next.\n\n### Regarding advantages of SutraNets over other RNNs\n\n> It is unclear what advantages do SutraNets have over RNNs with skip connections\n\nThis is valuable feedback. We will unify the discussion of lagged/residual inputs (lines 84-85), and skip connections between LSTM states (categorized as multi-rate recurrence layers in lines 64-65). We will also add a summary section to the related work, which will include the following new table:\n\n| RNN method | Meaning of K | RNN signal path | Max generative stride without feedback | RNN training parallelism | Coherent outputs |\n|-|-|-|-|-|-|\n| Standard RNNs, e.g. LSTMs ([Hochreiter & Schmidhuber, 1997](https://dl.acm.org/doi/10.1162/neco.1997.9.8.1735)) | N/A | $\\\\mathcal{O}(N)$ | $\\\\mathcal{O}(1)$ | $\\\\mathcal{O}(1)$ | Yes |\n| Skip connections ([Zhang et al., 2016](https://proceedings.neurips.cc/paper/2016/hash/860320be12a1c050cd7731794e231bd3-Abstract.html)), lagged/residual inputs ([He et al., 2016](https://arxiv.org/abs/1512.03385)) | Skip/lag amount | $\\\\mathcal{O}(N/K)$ | $\\\\mathcal{O}(1)$ | $\\\\mathcal{O}(1)$ | Yes |\n| PatchTST (as RNN) ([Nie et al., 2023](https://arxiv.org/abs/2211.14730)) | Size/stride of patches | $\\\\mathcal{O}(N/K)$ | N/A | $\\\\mathcal{O}(1)$ | No |\n| DilatedRNN ([Chang et al., 2017](https://arxiv.org/abs/1710.02224)) (dilations from $C \\\\ldots K$) | Highest dilation amount | $\\mathcal{O}(N/K)$ | $\\\\mathcal{O}(C)$ | $\\\\mathcal{O}(C)$ | Only if $C=1$ |\n| SutraNets | Number of sub-series | $\\\\mathcal{O}(N/K)$ | $\\\\mathcal{O}(K)$ | $\\\\mathcal{O}(K)$ | Yes |\n\n> Table 0: Advantages of SutraNets over prior RNNs: prior methods can reduce signal path, but not all methods improve error feedback nor facilitate training parallelism. Patching, as used in PatchTST, can reduce signal path by grouping consecutive elements into \"patches\", but it does not provide probabilistic outputs. Dilated RNNs with minimum dilation amounts $>1$ do facilitate error reduction and parallel training, but at the cost of sacrificing coherency, as the model is then \"equivalent to multiple shared-weight networks, each working on partial inputs\" ([Chang et al., 2017, Section 4.4](https://arxiv.org/abs/1710.02224)). SutraNets reduce signal path distances while simultaneously reducing error accumulation, enabling training parallelism, and generating coherent outputs. Experimentally, SutraNets perform better than standard LSTMs and LSTMs with lagged/residual inputs.\n\nWe will also add a discussion of Dilated RNN to sub-section \"Modeling long-term dependencies\", where we will provide the example of a time series that spikes to a high value at the very final historical (conditioning) input. In a Dilated RNN (with $C>1$), some of the outputs will be generated conditional on this spike, while others will be generated completely independently of the spike, and independent of other outputs that were aware of the spike, resulting in highly-incoherent output. The Dilated RNN paper mitigates this for point predictions by adding a final \"fusion layer\", but such an approach is not compatible with probabilistic forecasts."}, {"review_id": "Ob6FNO5auu", "reviewer": "Reviewer_MDp2", "summary": "The author proposed SutraNets, a novel method for neural probabilistic forecasting of long-sequence time series. It addresses challenges faced by previous autoregressive approaches, such as error accumulation and modeling long-distance dependencies. SutraNets treat long predictions as multivariate predictions over lower-frequency sub-series, reducing errors and signal path distances. The authors conduct extensive experimental results on real-world datasets, and they show significant improvements.", "questions": "1. Could the author discuss more about this work compared with pathTST (2023)?\n2. See above.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The sub-series in long-sequence analysis have seldom been investigated, especially for RNN-type networks.\n2. The paper is well-organized, and the discussion is very clear.", "weaknesses": "1. What is the deviation point of high and low frequency? What is the connection between traditional signal analysis?\n2. The performance improvement seems moderate. Could the author include std for complete?\n3. Finding 4 could include more recent SOTA methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your very useful comments, and for your kind words regarding the paper's key idea, organization, and clarity.\n\n### Regarding selection of high and low frequencies\n\n> What is the deviation point of high and low frequency? What is the connection between traditional signal analysis?\n\nThis is a good point: we did not describe how we chose the frequencies.  While the high frequency is implicitly the sampling frequency of the time-series, the low frequency (both for SutraNets and for the Low2HighFreq baseline) follows from the setting of $K$, which we basically viewed as a kind of hyperparameter to be set \"heuristically\" (line 201).  We will clarify in the methods section that $K$ is typically chosen such that it divides evenly into the primary seasonal period (i.e., the low frequency is twice or four times per day for series with primarily daily seasonality).  We will also gratefully use your idea and note in the paper that techniques such as autocorrelation and Fourier analysis can reveal the primary seasonality, and hence inform the choice of $K$, in cases where such information is not known *a priori*.\n\nNote also we still found a strong benefit of SutraNets over baselines even when $K$ does not divide evenly into the seasonal period, particularly for the alternating generative order.  See Finding 5 (line 283) and Table 3 for more details.\n\n### Regarding error bars for experimental results\n\n> The performance improvement seems moderate. Could the author include std for complete?\n\nYes, note we do study the (very consistent) stability of improvement in supplemental section B.2, in particular supplemental Figures 3 and 4, and we noted the overall finding in easy-to-miss Footnote 2 of the main paper.  We should promote Footnote 2 to be part of the main paper discussion and expand on our findings there.\n\n### Regarding recent \"deeper\" SOTA methods\n\n> Finding 4 [Deeper models enable improved long-sequence forecasting] could include more recent SOTA methods.\n\nThis is a very good point.  Beyond strengthening our finding, it would be of significant practical and theoretical importance to know if handling of long-term information can be improved in recent SOTA methods by simply increasing the depth of the underlying neural network.  We will definitely pursue this direction, thanks!\n\n### Regarding relationship to PatchTST\n\n> Could the author discuss more about this work compared with pathTST (2023)?\n\nYes, this paper is definitely worth adding to our sub-section \"Modeling long-term dependencies\" (and to a proposed new summary table, see Table 0 in our response to Reviewer 6VCF).  We will discuss PatchTST as follows:\n\n> In contrast to SutraNets, which operate over sub-series of the original input (i.e., points spaced $K$ apart), PatchTST groups $K$ *consecutive* values into input *patches*.  While patching was proposed as a method to reduce the attentional complexity of *Transformers*, patching could also be used with RNNs, where it would effectively reduce the signal path by a factor of $K$ (taking in $K$ inputs each step, with $K$ times fewer inputs overall).  However, PatchTST does not provide a mechanism to probabilistically *generate* patches, rather it directly outputs non-patched point predictions at all timesteps in one shot."}, {"review_id": "oV6KltJtGm", "reviewer": "Reviewer_7ywN", "summary": "The authors propose new ways execute the recurrence in auto-regressive  forecasting models. The approach is practical and shouldn't be too complicated to implement for a forecasting practitioner. Experiments show performance uplifts on common real and toy datasets.\n", "questions": "* The equation (2) is incorrect:  the right hand side terms must condition on $\\mathbf x_{1:T+N}$ not $\\mathbf x^k_{1:\\tilde T+\\tilde N}$ in order for both sides to be equal. That has implications for the RNN formulation in l131 where in the current form the RNN state $h^k_\\tilde t$ does not depend on any $\\mathbf x_t$ with $t>K\\tilde t$ effectively, while in equation (1) it does.\n\n* Equation (2) and the inline formula in l131 describe the Regular-alt case. It would be helpful to have these explicitly formulated also for the other 3 cases, so please add them. That could save space between l135 and l146\n\n* l143-145: *\"When not conditioning on > k values, SutraNets can instead generate the complete prediction range of the kth sub-series before generating any predictions for > k ones\"*. I think that's not correct. When sampling according to xxx-non schema, the RNN has to be conditioned on $h_{\\tilde T+\\tilde N}^{<k}$ in order to get (wrapped up) information about the whole past sub-series $y_{\\tilde T:\\tilde T+\\tilde N}^{<k}$ and not just $y_{\\tilde t}^{<k}$. This may have implications to the experiments and currently impose an unfair disadvantage to the xxx-non methods.", "limitations": "limitations are properly addressed", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The approach is interesting and practical, although the motivation appear a bit heuristically. One important advantage that is mentioned but not emphasized much, is that due to the lagged recurrence, roughly $K$-fold parallel processing can be achieved during training, which can be rather significant for RNN models on long sequences.\n\nThe paper is well organized, clearly written and easy to follow. The background section is nice and informative and provides a good overview over related work.", "weaknesses": "The authors explore different interleaved recurrence orders, but there is more to the topic that could strengthen the paper. For example, how can the performance be attributed  to different recurrence order on the one hand, and different hidden state dynamics on the other hand? As an example Fig 2(a) and (c) have the same recurrence order, but different hidden state dynamics. Accordingly  Backfill-alt Fig 2(f) could perhaps perform even better with the straight hidden state dynamics of a standard RNN.\n\nAnother analysis I'd love to see is how the conditional distributions $p(y_t| ...)$ look like for the different methods. Is there a recurrence order where these are mostly Gaussian or have another preferable simple shape? Take for example $y_1\\sim\\mathcal N(0,\\sigma_1)$ and $y_2\\sim\\mathcal N(y_1,\\sigma_2)$, then both $p(y_1)$ and $p(y_2|y_1)$ are Gaussian, but for $p(y_2)$ and $p(y_1|y_2)$  the later has a complicated bimodal form. Similar effects may impact the performance of auto regressive models in general, and it would be interesting to see that analyzed for the proposed models. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your detailed and constructive review, and your kind words regarding the advantages of the approach and the paper's organization, clarity, and overview of related work.\n\n### Regarding parallelization of training\n\n>  One important advantage that is mentioned but not emphasized much, is ... roughly $K$-fold parallel processing can be achieved during training, which can be rather significant for RNN models on long sequences\n\nThis is a great point: we will definitely emphasize this advantage in the intro, and contrast SutraNets to prior work along these lines. For example, see the proposed new summary table (Table 0) in our response to Reviewer 6VCF.\n\n### Regarding disentangling the effects of generative order vs. hidden state dynamics\n\n> Backfill-alt Fig 2(f) could perhaps perform even better with the straight hidden state dynamics of a standard RNN\n\nSo basically have a single evolving RNN hidden state, but step through the time series in backfill order in sub-sequences of $K$ consecutive values (akin to reading a document downwards but from right-to-left on each line). Essentially, a version of 2(f) where the state arrows always connect the output in row $i$ to the output in row $i+1$, rather than connecting states within sub-series. Since signal path would be high, and error accumulation lower, we agree this would help disentangle the effects of generative order vs. hidden state dynamics. Also, what is great is that this simply requires a kind of re-shuffling of the values in the conditioning and generation windows (i.e., a simple pre-processing step) before applying any standard sequence model. We will definitely pursue this further - great idea!\n\n### Regarding visualizing the conditional distributions for the different methods\n\n> Another analysis I'd love to see is how the conditional distributions look like for the different methods\n\nThis is also a good suggestion, as it may help shed light on *why* certain orderings work better. We have the machinery to plot the distributions, we just need to think further about how to systematically and objectively analyze these plots.\n\n### Regarding conditioning on covariates\n\n> The equation (2) is incorrect: the right hand side terms must condition on $x_{1:T+N}$ not $x^k_{1:\\\\tilde{T}+\\\\tilde{N}}$ in order for both sides to be equal. That has implications for the RNN formulation in l131\n\nYes, good catch: here we break strict equality. While mathematically there is no reason not to condition on *all* the covariates, in prior forecasting models such as DeepAR ([Salinas et al., 2020](https://www.sciencedirect.com/science/article/pii/S0169207019301888), see Eq. (2)), it is common for the RNN to consume the current covariates alongside the prior observation at each step, so the output is implicitly only conditioned on covariates at $\\\\le t$. We will fix this by conditioning on all the covariates in Eq. (2) (with strict equality), and then using approximate equality in Eq. (3) when we introduce the RNN, which consumes sub-series covariates. We will also provide more explanation.\n\nRegarding the implications for the covariates input to each sub-series RNN: note that in practice, ignoring future covariates is not really a limitation, as we can always introduce new covariates at time $t$ that identify future covariates, e.g., they may say something like, \"there is a sale on this product beginning at time $t+1$\". Moreover, in prior work, most covariates are either static across the whole prediction range (e.g., a product ID), or provide timestep-specific information (e.g., the hour-of-the-day or day-of-the-week of that particular timestep).\n\n### Regarding full equations for all generative orderings\n\n> Equation (2) and the inline formula in l131 describe the Regular-alt case. It would be helpful to have these explicitly formulated also for the other 3 cases\n\nYes, we can do this. For the benefit of other reviewers, we note the non-alternating formulas arise when removing the purple $y^{>k}_{1:\\\\tilde{t}-1}$ as a conditioning value.\n\n### Regarding non-alternating models\n\n> \"When not conditioning on $>k$ values, SutraNets can instead generate the complete prediction range of the $k$th sub-series before generating any predictions for $>k$ ones\". I think that's not correct. When sampling according to xxx-non schema, the RNN has to be conditioned on $h_{\\\\tilde{T}+\\\\tilde{N}}^{<k}$ in order to get (wrapped up) information about the whole past sub-series ... This may have implications to the experiments and currently impose an unfair disadvantage to the xxx-non methods.\n\nWe should clarify that the $k$th sub-series RNN does not condition on *hidden states* from other sub-series RNNs, rather only on *values* from other sub-series, which the $k$th sub-series RNN consumes at each timestep along with its own prior values, as illustrated in Figure 2. Each sub-series RNN thus gets \"wrapped up\" information about the other sub-series in its own hidden state, but only *up to the current timestep* - and, for non-alternating models, only for sub-series with index $<k$. Furthermore, in Figure 2(d), note that when generating y5 in row 7, we *could* theoretically condition on y6 and y7 at this point, but we do not. In practice, this seems to impair the performance of Regular-non (as noted in Footnote 3 on page 7), but not Backfill-non, where all the proximal values (before and after the value being generated) *are* conditioned on (input either at the current or at the previous timestep) - we will expand Footnote 3 to explicitly contrast the regular-non and backfill-non situations.\n\nNow, it is not clear that ignoring distant future values is a particular disadvantage to the non-alternating models, since the alternating models (and standard RNNs) also only condition on values generated up to and including the current timestep, however it would be informative to us if you see it differently."}, {"review_id": "CzrBglXhPJ", "reviewer": "Reviewer_BQ6D", "summary": "The paper \"SutraNets: Sub-series Autoregressive Networks for Long-Sequence, Probabilistic Forecasting\" propose to model uni-variate time-series with two interleaved networks, that model 'fine-grained' and 'coarse-grained' time-steps. This has the advantage of reducing signal paths, reducing inference error accumulation, enable parallel training, and sharing parameters potentially more efficiently. The paper investigates 5 additional architectures on 6 standard datasets (e.g. electricity, but also MNIST), and demonstrates superior performance over the baselines.", "questions": "- in eq.2 the conditioning is on x^k_{1:T+N}, i.e. y_{\\tile t} is conditioned on co-variates from the future. I assume this should be x^k_{1:\\tile t} instead?\n", "limitations": "+ The paper has a good limitations section, which covers the limitations. E.g. the method being more complex, and requiring additional tuning for better performance. Also the societal impacts are considered in this section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ the paper is well and very clearly written (including the literature review, methods, Figures etc.)\n+ the core idea of the paper makes intuitive sense, and the paper proposes several good ideas how to extend the standard autoregressive architecture, and investigates their impact on the final performance.\n+ the method is shown to improve performance over baselines", "weaknesses": "mainly minor things:\n- Figure 4: it's a bit unclear what is meant with 'more confident' - I think it is probably meant that it is less blurry? I'd be great to mention this explicitly if this is the case. Also, it'd be great to somehow quantify these findings: what are the likelihoods on the hold-out test data in the different settings? Maybe in this case (MNIST), also image-based evaluations could be performed (e.g. FID).\n- Figure 6: \"per 100\" in the figure itself. I think it'd be better to re-do the Figure with \"inference time per forecast\" and then put it there directly - otherwise it lacks a unit. \n- There is some prior work on multi-variate forecasting that has some similarity to this paper, but hasn't been mentioned. E.g. [1] (please also check reference & citations of this work), though focussing on the full multi-variate time-series problem, do also use an 'inner' architecture to generate data, and then have an outer autoregressive model. These should be mentioned in the related literature.\n\n[1] Rasul et al 2021. Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your helpful and thoughtful review, and your positive comments regarding the paper's core idea, clarity of presentation, and thorough evaluation.\n\n### Regarding concept of \"confidence\" in Figure 4\n\n> it's a bit unclear what is meant with 'more confident' - I think it is probably meant that it is less blurry? I'd be great to mention this explicitly if this is the case. Also, it'd be great to somehow quantify these findings: what are the likelihoods on the hold-out test data in the different settings? Maybe in this case (MNIST), also image-based evaluations could be performed (e.g. FID).\n\nYes, this is a good point.  We should have clarified that by \"more confident\", we meant that at a given percentile of the forecast distribution (e.g., p50% = 50% confidence), the bottom parts of the digit have more non-zero values, implying the networks have more certainty in the digit being generated.  While we compare model performance on MNIST using standard forecasting metrics in Table 1, we like the idea of computing likelihood and image-based evaluations for this dataset, and will include these in the final paper.\n\n### Regarding Figure 6\n\n> \"per 100\" in the figure itself. I think it'd be better to re-do the Figure with \"inference time per forecast\" and then put it there directly - otherwise it lacks a unit.\n\nAgreed.  We will change this.\n\n### Regarding prior probabilistic forecasting models\n\n> There is some prior work on multi-variate forecasting that has some similarity to this paper, but hasn't been mentioned. E.g. [1] (please also check reference & citations of this work), though focussing on the full multi-variate time-series problem, do also use an 'inner' architecture to generate data, and then have an outer autoregressive model. These should be mentioned in the related literature.\n\nAlso a good point.  In response to your comment and also suggestions from Reviewer 6VCF, we will revise the related work to add an initial subsection on \"Probabilistic autoregressive forecasting\", and we will note that SutraNets are fully *compatible* with prior techniques to estimate/sample an output probability distribution at each step. For example, SutraNets can serve as the RNN model both with the AR denoising diffusion model - as box RNN in Figure 1 in [Rasul et al., 2021a](https://arxiv.org/abs/2101.12072), or with the AR conditioned normalizing flows - as box RNN in Figure 1 in [Rasul et al., 2021b](https://arxiv.org/abs/2002.06103). While these papers focus on short-term multivariate forecasting, there is no reason these estimators could not also be applied to long-term forecasting, where SutraNets would likely prove very helpful as the sequential backbone.\n\nMoreover, while our paper shows that SutraNets improve the accuracy of C2FAR ([Bergsma et al., 2022](https://openreview.net/forum?id=lHuPdoHBxbg)), we should definitely have clarified that C2FAR is a state-of-the-art distribution estimator, and was previously shown to itself improve over DeepAR ([Salinas et al., 2020](https://www.sciencedirect.com/science/article/pii/S0169207019301888)), DeepAR-binned ([Rabanser et al., 2020](https://arxiv.org/abs/2005.10111)), SQF-RNN ([Gasthaus et al., 2019](https://proceedings.mlr.press/v89/gasthaus19a.html)), and IQN-RNN ([Gouttes et al., 2021](https://arxiv.org/abs/2107.03743)) - all essentially different methods for distribution estimation that operate on top of a standard sequence model. In our paper's evaluation, we went deep in testing C2FAR alongside various enhancements to the core sequence model, such as adding lags, dropout, and frequency hierarchies. However, it would certainly help our case to go broader and show that SutraNets also enhance other methods for distribution estimation, and we will pursue such experiments in advance of the camera-ready deadline.\n\n### Regarding conditioning on covariates\n\n> in eq.2 the conditioning is on $x^k_{1:T+N}$, i.e. $y_{\\\\tilde{t}}$ is conditioned on co-variates from the future. I assume this should be $x^k_{1:\\\\tilde{t}}$ instead?\n\nYes, we should clarify this.  Since covariates are known *a priori*, mathematically there is no reason not to condition on covariates over all timesteps.  However, in prior forecasting models such as DeepAR ([Salinas et al., 2020](https://www.sciencedirect.com/science/article/pii/S0169207019301888), see Eq. (2)), it is common to define the RNN as consuming the current covariates alongside the prior observation at each step, so the output is implicitly only conditioned on covariates at $\\\\le t$, which, as you have noted, is also what we do (see our Eq. (3) and definition of RNN in line 131). \n\nFYI, in practice, ignoring future covariates is not really a limitation, as we can always introduce new covariates at time $t$ that identify future covariates, e.g., they may say something like, \"there is a sale on this product beginning at time $t+1$\".  In prior work, most covariates are either static across the whole prediction range (e.g., a product ID), or provide timestep-specific information (e.g., the hour-of-the-day or day-of-the-week of that particular timestep).\n\nWe will revise the paper to make it clear when we are dropping conditioning on all covariates, to condition on only those up to time $\\\\tilde{t}$, and explain the rationale for this as noted above.  See also our similar response to Reviewer 7ywN on this topic."}]

Author rebuttal:

Meta-review: This paper studies the problem of long and univariate prediction by treating it as multivariate prediction over lower-frequency sub-series. The proposed method reduces error accumulation. Experiments show that the proposed method improves forecasting accuracy on real datasets.

The paper received five reviews, most of which are positive, except one negative review by Reviewer 6VCF. This reviewer raised concerns about previous baselines based on RNN and Transformer, as well as connection to probabilistic modeling. These concerns were adequately addressed in the rebuttal and subsequent discussion. Another reviewer had similar concerns, but raised his or her rating to 5 after rebuttal and discussion.

Overall, the final reviews are positive about the novelty and contribution of the paper.
Decision: Accept (poster)