SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shane Bergsma Timothy Zeyl Lei Guo
Huawei Cloud, Alkaid Lab Canada
{shane.bergsma, timothy.zeyl, leiguo}@huawei.com
We propose SutraNets, a novel method for neural probabilistic forecasting of long-sequence time series. SutraNets use an autoregressive generative model to factorize the likelihood of long sequences into products of conditional probabilities. When generating long sequences, most autoregressive approaches suffer from harmful error accumulation, as well as challenges in modeling long-distance dependencies. SutraNets treat long, univariate prediction as multivariate prediction over lower-frequency sub-series. Autoregression proceeds across time and across sub-series in order to ensure coherent multivariate (and, hence, high-frequency univariate) outputs. Since sub-series can be generated using fewer steps, SutraNets effectively reduce error accumulation and signal path distances. We find SutraNets to significantly improve forecasting accuracy over competitive alternatives on six real-world datasets, including when we vary the number of sub-series and scale up the depth and width of the underlying sequence models.
As a cloud provider, we rely on long-range forecasts to allocate resources and plan capacity. In particular, we need to predict the fine-grained pattern of demand (e.g., hourly) for a large number of products, over a long time horizon (e.g., months). Since purchase decisions are large-scale (e.g., supplying regions with 100K machines), and made weeks-to-months before delivery, cloud providers have a profound need for methods that "capture the uncertainty of the future" [43]. Fine granularity, long histories, long forecast horizons, and quantification of uncertainty can be formalized as the long-sequence (many-step) probabilistic forecasting problem.
Recent work applies Transformers [82] to long-sequence forecasting. In order to circumvent the Transformer's quadratically-scaling attention mechanism, long sequences necessitate trading off increased signal path for reduced computational complexity [44, 92, 87, 47]. Unfortunately, the effectiveness and even validity of these approaches have been questioned [90, 32]. Most of these approaches do not provide probabilistic outputs (rather best-guess point predictions), and all condition on fixed-size windows, typically of very short duration. Indeed, many recent papers on "long-term" forecasting [87, 86, 70, 93] condition on a maximum of 96 inputs — only 4 days of history at 1-hour granularity. Such restricted context is a serious limitation given that many time series (including in standard evaluation datasets based on electricity and traffic) have strong weekly seasonality.
Our production forecasting system, like many in industrial settings [65, 52, 72] and cloud services [69], is based on RNNs. The RNN's recurrent architecture forces the compression and consolidation of long-range information, and RNNs work well on smaller datasets as they are not as "data hungry" as Transformers [16]. RNNs can theoretically condition on context of any length; in this paper, we condition RNN forecasts on length-2016 contexts without difficulties in scaling.
RNNs are often used to generate probabilistic forecasts via an autoregressive approach [21, 65, 60, 7], whereby the probability over future timesteps is factorized into a product of one-step-ahead
Figure 1: SutraNets: a long, univariate history is converted to lower-frequency sub-series. Multivariate autoregressive prediction, across time and sub-series, ensures coherent univariate output samples.
conditionals [6, 28]. When generating many steps, however, two key problems arise: (1) a discrepancy grows between training, where we condition on true previous values, versus inference, where we condition on sampled values (the discrepancy problem), and (2) long-term dependencies may be challenging to exploit (the signal path problem). The discrepancy problem leads to error accumulation during inference [5], but also a kind of informational asymmetry: because highly-informative true values are available during training, the model may ignore other useful info, e.g., seasonally-lagged inputs provided as extra features. The signal path problem, meanwhile, is especially acute for RNNs, where distant information must be preserved over many steps in the RNN hidden state.
We propose SutraNets, a general method for probabilistic prediction of long time series. We focus on the application of SutraNets to RNN-based forecasting, where they offer major benefits. SutraNets transform a length- $N$  series into  $K$  sub-series of length  $N / K$ . Sub-series forecasts are generated autoregressively, sequentially conditional on each other, enabling coherent outputs (Fig. 1). SutraNets address both the discrepancy problem (by forcing the network to take larger generative strides, i.e., predicting without access to immediately-preceding true values) and the signal path problem (by reducing the distance between historical and output values by a factor of  $K$ ). Training of SutraNets decomposes over sub-series; i.e., sub-series RNNs can be trained in parallel, enabling a  $K$ -fold improvement in training parallelism over standard RNNs.
Experimentally, SutraNets demonstrate superior forecasting across six datasets, improving accuracy by an average of  $15\%$  over C2FAR [7] and over strong seasonality-aware baselines. We evaluate a variety of other strategies for improving long-sequence forecasting, including lags, input dropout, deeper and wider networks, and multi-rate hierarchical approaches. Through these evaluations, we gain insights into how to succeed at long-sequence generation, in time series and beyond.
Probabilistic autoregressive forecasting Autoregressive forecasting models typically combine a backbone sequence model with a technique to estimate an output probability distribution at each step. In this paper, we apply SutraNets to C2FAR [7], a state-of-the-art distribution estimator that was shown to improve over DeepAR [65], DeepAR-binned [60], SQF-RNN [21], and IQN-RNN [27]. SutraNets can serve as the sequential backbone for any of these estimators, as well as for other recent approaches such as the denoising diffusion models [62], or conditioned normalizing flows [63].
Multi-rate forecasting It is sometimes acceptable to aggregate high-frequency series or event data to a lower frequency prior to sequence modeling, e.g., aggregating patient events over 6-month intervals [48], or network traffic over 10 subframes [79]. When high-frequency predictions are required, lower-frequency forecasts can provide guidance. Techniques for temporal disaggregation convert low-frequency series to higher frequencies via related high-frequency indicator series [13, 67]. Athanasopoulos et al. [2] simultaneously forecast at multiple frequencies and then reconcile the predictions. Such methods do not provide uncertainty estimates nor are they based on neural networks.
Scaleformer [70] aggregates series, forecasts at the lower frequency, and upsamples the lower-freq predictions in order to provide additional inputs for the original forecast model. We evaluate a similar approach, Low2HighFreq (§3), as a baseline. Other neural approaches (in forecasting and beyond) implicitly encourage learning of lower-frequency dynamics through hierarchical attention [20, 47, 16],
<sup>1</sup>In Sanskrit, a sutra is a thread or "that which like a thread runs through or holds everything together" [85]. A SutraNet prediction is woven from threads of sub-series predictions; Fig. 2 shows different stitching patterns.
pooling blocks [92, 87, 49, 93], and deeply-stacked [29, 81] and multi-rate [68, 18, 38, 73, 14, 11] recurrence layers. Unlike these approaches, SutraNets explicitly generate low-frequency sub-series conditional on other low-frequency sub-series, without directly forecasting at a high frequency at all.
Reducing training/inference discrepancy If we know a priori how far to forecast, it is possible to avoid error accumulation by directly forecasting all horizons in one step [76], or via non-autoregressive decoding [20]. Such systems usually generate point predictions [92, 87, 9], although predicting certain fixed forecast quantiles is also possible [84, 20]. ProTran [77] is a probabilistic but non-autoregressive approach. Without modeling local dependencies among outputs, such models prevent generation of coherent samples, and preclude computing robust likelihoods for given output sequences — where error accumulation is not an issue (e.g., for anomaly detection, missing value interpolation, compression, etc.). SutraNets are more in-step with flexible autoregressive models like GPT3 [8].
For autoregressive models, scheduled sampling [5] aims to handle errors better during inference by sampling (incorrect) inputs during training. This approach was effective in tasks such as human motion synthesis [45], but results have been mixed in autoregressive forecasting [65, 66]. Noting that scheduled sampling produces a biased estimator, Lamb et al. [40] instead used adversarial techniques to reduce training/inference discrepancy. Wu et al. [88] applied this method to forecasting, using a GAN [26] with an autoregressive sparse transformer. However, adversarial training showed insignificant gains over the sparse transformer alone on 7-day electricity and traffic predictions.
Like scheduled sampling, dropout [74] is a kind of stochastic regularization that prevents overfitting by injecting noise into networks. Dropout rates of around  $20\%$  are commonly used in input layers of autoencoders and feed-forward nets [3, 74]. Dropout is also common in autoregressive models [82, 61], where, in theory, it may encourage networks to rely less on (now noisy) previous values [56].
Modeling long-term dependencies When forecasting long series, seasonally lagged values — historical values from, e.g., one year ago — can be used as extra features in the sequence model, acting as a kind of residual input [31]. A recent competition featured long time series (e.g., 700 days of history) [1]; the winning model found lag to be more effective than fixed attention weights [75]. Some researchers have even declared that, when it comes to attention, "lag is all you need" [19].
Learned attention can capture long-range dependencies [82], but is infeasible for very long sequences due to memory and time complexity quadratic in sequence length. Complexity also scales quadratically for forecasting via fully-connected layers [57, 9]. Recent work has explored sparsified or hierarchical attention in forecasting [44, 92, 47, 87]. TimesNet [86] transforms univariate time series into 2D tensors of multiple periods, which are then processed via Inception-style 2D kernels to generate point predictions. As mentioned earlier, most of these systems fail to condition on more than very short windows — a few days of hourly data. Ultimately, we are interested in probabilistic forecasting using years of fine-grained historical data. It is unclear whether lag is really all we need, or even whether seasonal-naive baselines [34] are already more effective than Transformers [32].
Unlike Transformers, RNNs can be applied to long sequences out-of-the-box. However, since RNN signal path is linear in sequence length, mechanisms are required to preserve long-range info, such as the special gated units of LSTMs [33]. Multi-dimensional RNNs [30] propagate distant info via row and column connections across images [78, 81]. Didolkar et al. [16] combine RNNs for long-range signals with Transformers that attend over shorter chunks. SutraNets, in contrast, do not require architectural changes within RNNs; indeed, they work with RNNs or Transformers. In SutraNets, low-frequency information is both implicitly captured in network state and explicitly and autoregressively communicated through generated low-frequency forecasts.
Comparison to other RNNs Table 1 compares SutraNets to prior RNNs (cf. Table 1 in [82]). PatchTST [55] groups consecutive values into input patches. While patching was proposed as a method to reduce the attentional complexity of Transformers, patching was also recently used with RNNs in SegRNN [46], where it effectively reduces the signal path by a factor of  $K$  (taking in  $K$  inputs each step — like SutraNets — but with  $K$  times fewer overall input steps). However, neither PatchTST nor SegRNN provides a mechanism to probabilistically generate patches, and thus generate coherent probabilistic outputs. Dilated LSTMs [83] and Dilated RNNs [10] with minimum dilation amounts  $>1$  do facilitate error reduction and parallel training, but at the cost of sacrificing coherency, as the model is then "equivalent to multiple shared-weight networks, each working on partial inputs"[10, §4.4]. With such DilatedRNNs, if the time series were to spike to a high level at
Table 1: Comparison of SutraNets to prior RNNs. Like prior methods, SutraNets reduce signal path, but also improve error accumulation (via a larger generative stride), while enabling greater training parallelism (via fewer sequential operations), without sacrificing coherent probabilistic output.
RNN method	Meaning of K	Signal path	Generative max stride	Sequential operations	Coherent outputs
Standard RNN, e.g., LSTM [33]	N/A	O(N)	O(1)	O(N)	Yes
Skipping [91], lags/residuals [31]	Skip/lag amount	O(N/K)	O(1)	O(N)	Yes
PatchTST [55], SegRNN [46]	Patch size/stride	O(N/K)	N/A	O(N/K)	N/A
DilatedRNN [10], dilations C...K	Largest dilation	O(N/K)	O(C)	O(N/C)	If C=1
SutraNets	Num. sub-series	O(N/K)	O(K)	O(N/K)	Yes
the very final conditioning input, only a subset of the outputs would be generated conditional on this spike, resulting in highly incoherent output. This can be mitigated for point predictions by adding a final fusion layer [10], but such an approach is not compatible with probabilistic forecasts.
Another related approach is the Subscale WaveRNN model for audio generation [35] (conditioned on input text). For Subscale WaveRNN, the motivation is not to reduce signal path nor improve error accumulation, but to have a  $K$ -fold increase in inference parallelism when generating very, very long audio sequences. Generation proceeds somewhat similar to our regular, non-alternating version ( $\S 3$ ) but conditioning on past and some future values, and with a shared neural network for all sub-series.
3 SutraNets: Sub-series autoregressive networks
Let  $y_{t} \in \mathbb{R}$  be the value of a time series at time  $t$ , and  $\mathbf{x}_t$  be a vector of time-varying features or covariates. Let  $y_{i:j}$  denote sequence  $y_{i} \ldots y_{j}$ . We seek a model of the distribution of  $N$  future values of  $y_{t}$  (the prediction range) given  $T$  historical values (the conditioning range). We can formulate this conditional distribution as an autoregressive (AR) generative model as in [28]:
$$
p \left(y _ {T + 1: T + N} \mid y _ {1: T}, \mathbf {x} _ {1: T + N}\right) = \prod_ {t = T + 1} ^ {T + N} p \left(y _ {t} \mid y _ {1: t - 1}, \mathbf {x} _ {1: T + N}\right) \tag {1}
$$
Following DeepAR [65], forecasting approaches have modeled the one-step-ahead distributions using RNNs [21, 65, 60, 7] or Transformers [44]. In these approaches, a global sequence model is trained by slicing many training series into many windows, i.e., conditioning+prediction ranges at different start points, and normalizing the windows using their conditioning ranges. Model parameters are fit via gradient descent, minimizing NLL of prediction-range outputs. At inference time, forecasts for a given conditioning range are generated by sampling  $\hat{y}_{T + 1}\dots \hat{y}_{T + N}$  sequentially, autoregressively conditioning on samples generated at previous timesteps. By repeating this procedure many times, a Monte Carlo estimate of Eq. (1) is obtained, from which desired forecast quantiles can be derived.
SutraNets consider each univariate window to be comprised of an ordered collection of  $K$  lower-frequency sub-series, each  $1 / K$  of the original length (Fig. 1). The  $k$ th sub-series is obtained by selecting every  $K$ th value in the original window, each sub-series starting at a unique offset in  $1 \ldots K$ . Let  $y^{k}$  be the  $k$ th sub-series. Let  $\tilde{N} = N / K$  and  $\tilde{T} = T / K$  be prediction and conditioning lengths corresponding to each sub-series, and let  $< k$  and  $> k$  denote indices from  $1 \ldots K$  that are less than and greater than  $k$ . Through another application of the chain rule, now across sub-series, Eq. (1) becomes:
$$
p \left(y _ {T + 1: T + N} \mid y _ {1: T}, \mathbf {x} _ {1: T + N}\right) = \prod_ {\tilde {t} = \tilde {T} + 1} ^ {\tilde {T} + \tilde {N}} \prod_ {k = 1} ^ {K} p \left(y _ {\tilde {t}} ^ {k} \mid y _ {1: \tilde {t}} ^ {<   k}, y _ {1: \tilde {t} - 1} ^ {k}, y _ {1: \tilde {t} - 1} ^ {> k}, \mathbf {x} _ {1: T + N}\right) \tag {2}
$$
where  $\tilde{t}$  iterates over timesteps in each sub-series. Log-likelihoods of sub-series prediction ranges can be summed to compute overall NLL, which is minimized during SutraNet training. Rather than modeling these conditionals using a single RNN, SutraNets use a separate RNN for each sub-series.
<sup>2</sup>Since the  $k$ th sub-series is assigned after slicing training/testing windows, it will vary in terms of which offset it corresponds to in the original time series, e.g., it may start at 1pm, 2pm, 3pm, etc., in different windows.
(a) Standard RNN, e.g., C2FAR
(b) Low2HighFreq
(c) Regular-alt SutraNet
(d) Regular-non SutraNet
(e) Backfill-non SutraNet
(f) Backfill-alt SutraNet
Figure 2: Ordering of generative steps in SutraNets. One output value (bold border) is generated in each row, conditional on (1) feature nodes (dashed border) in that row, and on (2) state from previous steps  $(\Rightarrow$  arrow connections). SutraNet RNNs (c-f) use features generated in earlier steps by both themselves and other RNNs; these values can even be from future timesteps (backfill cases).
Each RNN has its own parameters and hidden state, but SutraNets are trained and tuned collectively as a single network. The  $k$ th RNN determines the  $k$ th conditional probability at each timestep:
$$
p \left(y _ {T + 1: T + N} \mid y _ {1: T}, \mathbf {x} _ {1: T + N}\right) = \prod_ {\tilde {t} = \tilde {T} + 1} ^ {\tilde {T} + \tilde {N}} \prod_ {k = 1} ^ {K} p \left(y _ {\tilde {t}} ^ {k} \mid \theta_ {t} ^ {k} = f \left(h _ {\tilde {t}} ^ {k}\right)\right) \tag {3}
$$
where  $h_{\tilde{t}}^{k} = \mathrm{rn}\mathrm{n}^{k}(h_{\tilde{t} -1}^{k},y_{\tilde{t}}^{< k},y_{\tilde{t} -1}^{k},y_{\tilde{t} -1}^{>k},\mathbf{x}_{\tilde{t}}^{k})$  and  $f(\cdot)$  is a function mapping the  $k$ th RNN state to parameters of a parametric output distribution,  $p(y_{\tilde{t}}^{k}|\theta_{t}^{k})$ . So while prior models condition  $y_{t}$  only on  $y_{t - 1}$ , SutraNets autoregressively condition the generation of  $y_{\tilde{t}}^{k}$  on:
1.  $y_{\tilde{t}}^{< k}$ : Current values (at  $\tilde{t}$ ) of sub-series with indices  $< k$  (always)
2.  $y_{\tilde{t}_{-1}}^k$  : The previous value (at  $\tilde{t} - 1$ ) of the current sub-series,  $k$  (always)
3.  $y_{\tilde{t} -1}^{>k}$  : Previous values (at  $\tilde{t} -1$  ) of sub-series with indices  $>k$  (optional)
Conditioning  $y_{\tilde{t}}^{k}$  on  $y_{\tilde{t}}^{< k}$  makes the model autoregressive across sub-series; this ensures sub-series are generated conditional on each other, and thus (recombined) samples of the original high-frequency series will be coherent. Conditioning on  $y_{\tilde{t}-1}^{>k}$  is optional; when using  $y_{\tilde{t}-1}^{>k}$ , SutraNets must generate in an alternating manner, generating one value for each sub-series at each timestep,  $\tilde{t}$  (Fig. 2c). Without  $y_{\tilde{t}-1}^{>k}$ , SutraNets can generate the complete prediction range of the  $k$ th sub-series before generating any predictions for  $>k$  ones (Fig. 2d); in other words, we can exchange ordering of the products in Eq. (2), remove  $y_{\tilde{t}-1}^{>k}$  from the equation, and proceed in a non-alternating manner.
Note we can input any subset of the covariates to the RNN at each timestep. Here we include those specific to the given timestep of the given sub-series,  $\mathbf{x}_{\hat{t}}^{k}$ , as in practice most covariates are either static across the prediction range (e.g., a product ID), or provide timestep-specific info (e.g., the hour-of-the-day) [65].
SutraNets must also specify how the sub-series index,  $k$ , relates to the offset in the original series. Figs. 2c and 2d illustrate regular ordering, where sub-series  $k$  begins at offset  $k$  in the original series. An alternative is backfill ordering, where sub-series  $k$  starts at offset  $K - k + 1$  (Figs. 2e, 2f). We must also specify the number of sub-series,  $K$ . We pick  $K$  so it divides into the primary seasonal period (e.g.,  $K = 6$  for 24-hour seasons), as each sub-series then exhibits seasonality; signal processing techniques can reveal seasonality when it is not known a priori. Sub-series with seasonality may be more predictable, which will be especially useful for non-alternating models, as these models generate initial sub-series while conditioning on only  $1 / K$  of historical values. Success for these models depends on whether there is redundancy in the series, or whether values from other sub-series are sufficiently important that non-alternating generation creates a missing information problem.
Reducing training/inference discrepancy, signal path Non-alternating and backfill approaches force the network to predict  $K$  steps ahead, without access to true previous values. Non-alternating also generates far horizons in fewer steps (e.g., for the  $k = 1$  sub-series), reducing error accumulation. In contrast, Regular-alt generates in the same order, and conditional on the same values, as standard RNN models (Fig. 2a), and so should suffer the same effects from training/inference discrepancy.
All SutraNets reduce RNN signal path by a factor of  $K$ , as can be seen by tracing paths between features and outputs in Fig. 2. Note Regular-alt has low signal path but standard discrepancy (as noted above). The relative effectiveness of Regular-alt can thereby provide a diagnostic for whether discrepancy or signal path affects a given forecasting task (\$4).
Form of output distribution and input encoding Unlike sampling text from a softmax, forecasting requires a parametric output distribution,  $p(y_{\tilde{t}}^{k}|\theta_{t}^{k})$  (Eq. (3)), that can account for mixed discrete and continuous outputs, potentially of unbounded dynamic range. C2FAR [7] proposes such an output distribution, via an autoregressive generative model over a hierarchical coarse-to-fine discretization of time series amplitudes (with Pareto-distributed tails). We adopt C2FAR-LSTMs for our sub-series sequence models. We use a 3-level C2F model with 12 bins at each level, discretizing normalized values to a precision of  $12^3$  but requiring only  $12 \times 3$  output dimensions at each timestep. Previous values from each sub-series and covariate features from other sub-series are also C2F-encoded and provided as inputs at each step. Here the efficiency of C2FAR proves helpful; at each step we can encode  $K$  covariate values (i.e., from  $K$  other sub-series) using only  $12 \times 3 \times K$  input dimensions, as opposed to, e.g., the  $12^3 \times K$  dimensions that would be required with equivalent flat binnings [60].