SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Low2HighFreq Instead of generating all lower-frequency sub-series, we experiment with generating a single low-freq sub-series and conditioning on it within a high-freq model. This Low2HighFreq approach (Fig. 2b) is similar to multi-rate approaches ( $\S 2$ ), except here low-freq samples provide not only features, but hard constraints on every corresponding  $K$ th value in the high-freq output. During inference, when we use Monte Carlo sampling to estimate  $p(y_{T+1:T+N}|\ldots)$ , we can actually sample high-freq values conditional on low-freq ones. As such, techniques such as rejection sampling [53] are applicable; we experimented with importance sampling via likelihood weighting (fixing the low-freq values and sampling the rest), but ultimately found the likelihood weights had such high variance that it was more effective to use uniform weights and accept inconsistent estimation.
Complexity Let  $H$  be the number of RNN hidden units and  $K$  the number of sub-series. The complexity-per-timestep of each SutraNet RNN scales with the sum of the RNN's recurrence,  $H \times H$ , and projection of (at most)  $K$  inputs (from other sub-series) to the RNN hidden state,  $K \times H$ . Even though SutraNets have  $K$  separate RNNs, only one runs at each original timestep (i.e., each SutraNet RNN runs  $1 / K$  of total timesteps). In consequence, SutraNet complexity-per-timestep scales with  $H \times H + K \times H$ , whereas standard RNNs scale  $H \times H$ . In practice,  $H >> K$ , and we find empirically SutraNets also scale  $H \times H$ , resulting in similar training/inference speeds and memory requirements compared to standard models, given the same individual LSTM depth and width (\$4).
One key advantage of SutraNets is that training decomposes over sub-series. Sub-series RNNs do not condition on hidden states from other sub-series RNNs, rather only on generated values. During training, when we have access to the true values of other sub-series, all inputs are known in advance and sub-series RNNs can be trained in parallel. The only sequential action is evolution of each RNN's hidden state — over  $1 / \kappa$  fewer steps than standard or multidimensional RNNs [78, 81]. In this way, SutraNet-RNNs provide middle-ground between RNNs and standard Transformers (with no sequential computation in training). Inference in SutraNets, like all AR models, is sequential [36].
Application to Transformers Rather than using  $K$  RNNs to parameterize the conditional probabilities as in Eq. (3), we could instead use  $K$  autoregressive Transformers. Compared to a standard Transformer, backfill and non-alternating sub-series Transformers would have the advantage of reducing error accumulation (by increasing the generative stride), but not of signal path (already  $\mathcal{O}(1)$  for Transformers [82]). Moreover, at each timestep, a sub-series Transformer could attend to essentially all prior values, limited only by the generative order. In a naive implementation, SutraNet Transformers would therefore have an asymptotic complexity of  $\mathcal{O}(N^2)$  — the same as standard Transformers. However, we can attain  $\mathcal{O}(N^2 / K)$  complexity by restricting each sub-series Transformer to only attend to values from its own sub-series, plus a few proximal values from other sub-series (similar to stripped or banded self-attention [12, 8]). Although asymptotically larger than the  $\mathcal{O}(N(\log N)^2)$  of LogSparse attention [44], it merits empirical investigation to determine which approach achieves the most favorable accuracy vs. complexity trade-off.
Limitations and broader impact SutraNets are more complex than standard sequence models in a few ways. First, they require more decisions, such as specifying the number of sub-series. While superior accuracy can be obtained when making these choices heuristically, we evaluate the extent to which tuning can achieve further gains (§4). Secondly, having  $K$  separate RNNs results in roughly  $K$  times as many total parameters. Although, as noted above, this does not translate into requiring more memory or training/inference time, SutraNet models are larger to store and transmit.
As a flexible predictor of both long and short horizons (and without needing a priori specification of confidence quantiles), SutraNets could also serve as the basis for a general-purpose large forecasting model (LFM), along the lines of LLMs like GPT3 [8]. Like modern LLMs, LFMs could potentially be used with little task-specific data and no fine-tuning, and may have similarly-large environmental and financial costs [4]. As with LLMs, new techniques will be needed to detect [64, 71, 54], document [51, 22], and mitigate [25, 58] unsafe usage and other failure cases.
Training. We implemented SutraNets in PyTorch [59]. We use Adam [37] (default  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$ ,  $\epsilon = 10^{-8}$ ), early stopping, and a decaying learning rate ( $\gamma = 0.99$ ). The default RNN is a 1-layer LSTM with 64 hidden units; when using  $>1$  layers (Table 3), inter-layer dropout [50] with  $p_{drop} = 0.001$  is applied. For every model/dataset pair, we tune weight decay and initial learning rate over a  $4\times 4$  grid.
Systems. We evaluate the following systems. Naïve and SeasNaïve are non-probabilistic, untrained baselines, while the C2FAR variants follow the standard RNN ordering (Fig. 2a):
Naive: At each horizon, repeats last-observed historical value [34].
SeasNaive: At each horizon, repeat previous value at same season as horizon [34], e.g., same hour-of-day (SeasNaive1d) or hour-of-week (SeasNaive1w).
C2FAR: LSTM-based forecasting model [7], configured as described above (§3).
C2FAR+lags: C2FAR, but with extra features for previous hour-of-day values (C2F-encoded).
C2FAR+dropout: C2FAR, but with  $p_{drop} = 0.2$  (and re-scaling [74]) applied to inputs during training.
Low2HighFreq: High-frequency forecast generated conditional on low-frequency values (§3).
SutraNets: Variations Regular-alt, Regular-non, Backfill-alt, Backfill-non (§3).
Datasets. We evaluate on the following datasets, where we note the number of SutraNet sub-series,  $K$ , as well as lengths of conditioning,  $C$ , and prediction,  $P$ , ranges:
azure:  $K = 6$ ,  $C = 2016$ ,  $P = 288$ : 5-minutely usage of 2085 VM flavors, groupings (by tenant, etc.) in Azure cloud. Originally from [15], aggregated here instead to 5-minute intervals.
elec:  $K = 6$ ,  $C = 168$ ,  $P = 168$ : Hourly electricity usage of 321 customers [17], version from [39].
traff:  $K = 6$ ,  $C = 168$ ,  $P = 168$ : Hourly occupancy for 862 car lanes [17], version from [89].
wiki:  $K = 7$ ,  $C = 91$ ,  $P = 91$ : Daily hit-count for 9535 pages, first used in [21], version from [27].
mnist:  $K = 4$ ,  $C = 392$ ,  $P = 392$ : row-wise sequential pixel values for 70,000 28×28 digits [42].
mnist:  $K = 4$ ,  $C = 392$ ,  $P = 392$ : mnist sequences, but with a fixed, random permutation applied.
Table 2: ND%, wQL% across datasets, best results in bold. For mnist/mnist $\pi$ , previous row value used for SeasNaive1d. Poor results for C2FAR relative to Regular-alt indicate signal path problem (green cells). Poor results for both C2FAR and Regular-alt indicate discrepancy problem (pink). Non-alternating weakness indicates missing info problem (yellow). Backfill-alt reduces signal path and discrepancy without missing info, therefore offering best or near-best solution on each dataset.
azure	elec	traff	wiki	mnist	mnistπ
Naïve	3.2, 3.2	43.0, 43.0	75.5, 75.5	44.0, 44.0	100.0, 100.0	117.9, 117.9
SeasNaïve1d	2.8, 2.8	10.5, 10.5	31.1, 31.1	44.0, 44.0	145.2, 145.2	163.5, 163.5
SeasNaïve1w	5.2, 5.2	11.1, 11.1	17.5, 17.5	41.8, 41.8	N/A	N/A
C2FAR	3.2, 2.5	10.6, 8.4	19.3, 16.0	31.1, 27.2	67.9, 52.3	100.0, 95.7
C2FAR+lags	2.7, 2.2	10.4, 8.3	18.0, 15.0	31.2, 27.2	68.1, 52.8	99.3, 89.9
C2FAR+dropout	3.2, 2.6	10.9, 8.7	18.5, 15.3	31.3, 27.0	68.6, 53.3	99.7, 90.4
Low2HighFreq	2.6, 2.1	10.3, 8.2	20.2, 16.6	30.9, 27.2	62.4, 47.6	98.1, 88.3
Regular-alt	3.3, 2.6	9.9, 7.9	15.5, 13.0	30.4, 26.4	67.9, 52.4	48.7, 38.3
Regular-non	2.6, 2.1	9.7, 7.8	15.6, 13.1	31.2, 27.3	59.4, 45.4	90.1, 69.7
Backfill-alt	2.5, 2.0	9.3, 7.4	15.3, 12.8	30.1, 26.1	64.4, 49.7	72.0, 54.6
Backfill-non	2.3, 1.9	9.3, 7.4	15.7, 13.1	30.5, 26.8	58.9, 44.9	78.2, 58.3
Sequential (binarized) MNIST has been used for NLL [40] and classification [41] evaluations. Le et al. [41] used a fixed permutation "to make the task even harder." Here, we forecast a distribution over real-valued pixel intensities in the second half of an image, based on values in the first. Unlike the other datasets, train/dev/test splits are not based on time, rather we use the standard test digits [42].
Metrics. We evaluate the median of our forecast distribution by computing normalized deviation (ND) from true values. We evaluate the full distribution using weighted quantile loss (wQL), evaluated at forecast quantiles  $\{0.1, 0.2 \ldots 0.9\}$ , as in [60, 7]. For non-probabilistic Naïve and SeasNaïve baselines, note wQL reduces to ND, similarly to how CRPS reduces to absolute error [24, §4.2].
Results. We summarize the most important experimental findings as follows:
Finding 1: Backfill-alt and Backfill-non improve over C2FAR and other baselines on every dataset, with both obtaining an average relative reduction in ND of around  $15\%$  over the best non-SutraNet.
Table 2 has the main results; note the differences in  $\mathrm{ND\%}$  are very stable: we repeated grid search tuning and testing multiple times with different random seeds and found SutraNets superior to standard models across all repeats (supplemental §B.2). Backfill-non does best when the training/inference discrepancy problem predominates (azure & mnist, Table 2). Adverse effects of discrepancy seem to be magnified by high data redundancy; in these cases, non-alternating generation reduces discrepancy while having sufficient info to forecast accurately. We also hypothesize that Regular-non performs poorer because it generates less-coherent output: by construction, it only conditions on past values, even when subsequent values have already been generated (e.g., row 7 in Fig. 2d). In contrast, when generating a given point (e.g., row 7 in Fig. 2e), Backfill-non has always seen the previous and future proximal values, which were input at the previous and current timesteps, respectively. When signal path problems predominate (traff & mnist $^{\pi}$ ), alternating generation, with access to more info, performs better. On mnist $^{\pi}$ , differences between regular and backfill depend solely on the random permutation; the superiority of Regular-alt here motivates tuning over multiple orderings of sub-series, perhaps averaging over them, as is done in some prior non-sequential models [23, 80].
Accuracy gains seem to derive from solving both discrepancy and signal path problems together. When discrepancy predominates, Regular-alt performs poorly, even though signal path is improved. For the opposite case, where error accumulation alone is improved, we implemented an RNN that processes values in backfill order, but otherwise acts as a standard RNN. This model performs much poorer than SutraNet-based Backfill-alt (supplemental §B.4). The lack of a holistic solution may also explain the inconsistency of lags and dropout, which sometimes help, sometimes hurt compared to vanilla C2FAR. Low2HighFreq works best on datasets where discrepancy predominates, while performing much poorer when signal path is important. Note that weekly-seasonal baselines have not previously been evaluated on these datasets. Observing in particular the relatively strong accuracy of SeasNaive1w on traffic, we suggest they be included in future evaluations.
Figure 3: Forecasts and  $80\%$  intervals for elec series. Backfill-alt performs well with a changepoint (row 1, where SeasNaive1w fails) and with weekly seasonality (row 2, where C2FAR fails).
Figure 4: Percentiles of forecast distributions for first occurrence of  $0\ldots 9$  in mnist test data. Pixels before yellow line (in row-order) are history. Backfill-non more confident in bottom part of digits.
Finding 2: SutraNets are able to effectively model both long and short-term dependencies.
On elec data, where SeasNaive1d is more effective than longer-term SeasNaive1w (Table 2), modeling long-term dependencies remains important (Fig. 3). Backfill-alt is able to capture both short-term changes and long-term seasonality, the latter of which is neglected by C2FAR. In plots of the mnist forecasts (Fig. 4), we see that compared to C2FAR, Backfill-non generates more non-zero values in the bottom parts of digits 0, 2, 5, etc., e.g., at  $p50\%$ , implying the networks have more certainty or confidence (i.e., tighter prediction intervals) on the digit being generated. Also, Table 2 results indicate that discrepancy is the predominant problem on mnist; this may help explain why autoregressive models like PixelRNN [81] score very high in likelihood-based evaluation (where there is no sampling), but produce less-realistic samples compared to GANs. SutraNets may therefore provide a useful tool for reducing training/inference discrepancy in a variety of image generators.
Finding 3: SutraNets are more accurate than C2FAR across all forecast horizons.
Autoregressive models are normally thought to accumulate and compound [56] errors "along the generated sequence" [5]. We therefore expect the accuracy gap between SutraNets and C2FAR to grow across horizons. Instead, on elec and traffic, we already see a large gap over the first 24 hours (supplemental §B.3). Improvements in both informational asymmetry (\$1) and signal path evidently enable gains across all steps. Low2HighFreq, in contrast, is only accurate at horizons corresponding to its low-freq forecast, while Backfill-alt is slightly better at horizons corresponding to its first sub-series. This suggests errors do "accumulate" to a small extent across sub-series autoregression.
Finding 4: Deeper models enable improved long-sequence forecasting, for SutraNets and C2FAR.
In particular, going from 1 to 2 layers greatly improves accuracy on traffic (Table 3), a dataset where signal path is the predominant problem. Smaller improvements are also seen on elec. Since both LSTM stacking and use of SutraNets increase complexity, a natural question is which approach offers the best computational tradeoff for a desired level of accuracy. Fig. 5 shows that for a given number of parameters — and especially for a given inference time — SutraNets are able to obtain better
Table 3: ND% decreases for SutraNets & C2FAR as number of RNN layers/hidden units increases.
Dataset	elec	traff
#layers	1	2	3	4	1	2	3	4
#hidden	64	128	256	256	64	128	256	256
C2FAR	10.6	10.2	9.9	9.7	19.3	14.7	14.8	14.4
Regular-alt	9.9	9.3	9.0	9.2	15.5	14.3	14.3	14.2
Backfill-alt	9.3	9.3	9.0	9.3	15.3	14.2	14.1	14.0
Backfill-non	9.3	9.1	9.5	9.3	15.7	14.4	14.4	14.4
Table 4: ND% lowest when #sub-series divides into seasonal period (24), but alternating models less sensitive on elec (cf. C2FAR with  $10.6\%$  on elec,  $19.3\%$  on traffic.)
Dataset #subseries	elec	traff
6	7	12	6	7	12
Regular-alt	9.9	10.0	10.0	15.5	17.2	15.0
Backfill-alt	9.3	9.4	9.4	15.3	16.9	14.9
Backfill-non	9.3	10.0	9.5	15.7	17.3	15.4
Figure 5: Total parameters (x-axis) and average inference time per forecast (y-axis, lower better) for C2FAR and Backfill-alt systems of Table 3, for elec (left) and traffic (right). ND% labeled at each point in boxes. Backfill-alt enables more parameters and better ND% at equivalent speeds.
ND% (plots of memory usage and of training time are similar, see supplemental). SutraNet speed and memory usage scale with the complexity of each RNN rather than the number of sub-series RNNs ( $\S 3$ ); they can consequently make use of  $K \times$  more parameters without  $K \times$  the cost.
Finding 5: Backfill-alt is robust across different configurations, including when we vary  $K$ .
We also experimented with varying the number of sub-series. For all  $K > 1$ , SutraNets work much better than standard C2FAR (Table 4). Interestingly,  $K = 12$  led to the highest accuracy on traffic, where the signal path problem predominates. When  $K$  does not divide evenly into the seasonal period, non-alternating models have a missing info problem (e.g., they miss predictive preceding values at the same hour-of-day). Both Backfill-alt and Regular-alt can access these values through covariate features (cf. Fig. 2f), and hence perform relatively better when  $K = 7$  on elec. Backfill-alt is relatively weaker at  $K = 7$  when there is less redundancy (traffic), although still superior to standard C2FAR.
See the supplemental for further experimental details and results. Code for SutraNets is available at https://github.com/huaweicloud/c2far_forecasting/wiki/SutraNets.
Probabilistic forecasting of long time series is an under-studied problem, with most recent work focusing on point prediction. We presented SutraNets, a novel autoregressive approach to probabilistic forecasting of long-sequence time series. SutraNets convert a univariate series into a  $K$ -dimensional multivariate series, each dimension comprising a distinct every- $K$ th-value sub-series of the original sequence. The SutraNet autoregressive model generates each sub-series conditional on both its own prior values, and on other sub-series, ensuring coherent output samples. From these samples, an estimate of the full joint probability of future values is made. SutraNets leverage C2FAR under-thehood, allowing for efficient representation of time series amplitudes, and low-overhead input encoding of covariate sub-series. SutraNets provide a holistic solution to challenges in long-sequence sampling, reducing training/inference discrepancy (by generating conditional on longer-term information) and signal path distances (by a factor of  $K$ ). Experimentally, SutraNets achieve state-of-the-art results on a variety of long-sequence forecasting data sets, while demonstrating similar speed and memory requirements as standard sequence models, even when using  $K \times$  more parameters.
[87] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. 2021. Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting. Advances in Neural Information Processing Systems 34 (2021).
[88] Sifan Wu, Xi Xiao, Qianggang Ding, Peilin Zhao, Ying Wei, and Junzhou Huang. 2020. Adversarial Sparse Transformer for Time Series Forecasting. Advances in Neural Information Processing Systems 33 (2020), 17105-17115.
[89] Hsiang-Fu Yu, Nikhil Rao, and Inderjit S Dhillon. 2016. Temporal Regularized Matrix Factorization for High-Dimensional Time Series Prediction. Advances in Neural Information Processing Systems 29 (2016).
[90] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. 2022. Are Transformers effective for time series forecasting? arXiv preprint arXiv:2205.13504 (2022).
[91] Saizheng Zhang, Yuhuai Wu, Tong Che, Zhouhan Lin, Roland Memisevic, Russ R Salakhutdinov, and Yoshua Bengio. 2016. Architectural complexity measures of recurrent neural networks. Advances in Neural Information Processing Systems 29 (2016).
[92] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021. Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting. In AAAI Conference on Artificial Intelligence. 11106-11115.
[93] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. 2022. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International Conference on Machine Learning. 27268-27286.