C2FAR: Coarse-to-Fine Autoregressive Networks for Precise Probabilistic Forecasting
Authors: Shane Bergsma
Tim Zeyl
Javad Rahimipour Anaraki
Lei Guo
Venue: NeurIPS 2022 Accept (2022)
Keywords: time series
probabilistic forecasting
autoregressive generative models
neural networks

Abstract: We present coarse-to-fine autoregressive networks (C2FAR), a method for modeling the probability distribution of univariate, numeric random variables.  C2FAR generates a hierarchical, coarse-to-fine discretization of a variable autoregressively; progressively finer intervals of support are generated from a sequence of binned distributions, where each distribution is conditioned on previously-generated coarser intervals.  Unlike prior (flat) binned distributions, C2FAR can represent values with exponentially higher precision, for only a linear increase in complexity.  We use C2FAR for probabilistic forecasting via a recurrent neural network, thus modeling time series autoregressively in both space and time.  C2FAR is the first method to simultaneously handle discrete and continuous series of arbitrary scale and distribution shape.  This flexibility enables a variety of time series use cases, including anomaly detection, interpolation, and compression.  C2FAR achieves improvements over the state-of-the-art on several benchmark forecasting datasets.

C2FAR: Coarse-to-Fine Autoregressive Networks for Precise Probabilistic Forecasting
Shane Bergsma
Timothy Zeyl Javad Rahimipour Anaraki Lei Guo
Huawei Cloud, Alkaid Lab Canada (shane. bergssma, timothy. zeyl, javad. anaraki, 1 Chuawei. com
Abstract
We present coarse-to-fine autoregressive networks (C2FAR), method for model- ing the probability distribution of univariate, numeric random variables. C2FAR generates hierarchical, coarse-to-fin discretization of variable autoregressively; progressively finer intervals of support are generated from a sequence of binned dis- tributions, where each distribution is conditioned on previously-generace coarser intervals. Unlike prior (flat) binned distributions, C2FAR can represent values with exponentially higher precision, for only a linear increase in complexity. We use C2FAR for probabilistic forecasting via a recurrent neural network, thus modeling time series autoregressively in both space and time. C2FAR is the first method simultaneously handle discrete and continuous series of arbitrary scale and distri- bution shape. This flexibility enables variety of time series use cases, including anomaly detection, interpolation, and compression. C2FAR achieves improvements over the state-of-the-art on several benchmark forecasting dataastss
1 Introduction
Probabilistic forecasting the task of estimating a joint distribution over future values of a time series, given sequence of historical values. As an important problem with many applications and an abundance of real-world data, probabilistic forecasting has unsurprisingly come to be dominated by deep learning methods 77. Such methods typically fit a global sequence model to dataset of related time series. To forecast given series, future values are iteratively generated from one-step-aheee univariate output distributions. In this paper, we propose coarse-to-fing autoregressive networks (C2FAR), general method for modeling distributions of univariate numeric data, and we show how C2FAR enables improved output distributions for probabilistic forecasting.
Such improvements are needed because time series offer challenges not present in applications like text and image modeling. First, dynamic range scale of real-world time series can vary widely within a single dataset (55). The typical solution is to normalize the data based on input historical values, but many time series have heavy tails [19, meaning even after normalization, future values must be modeled via distributions with unbounded support. Second, even within one dataset, time series may be discrete, continuous, or mixed, and each of these may be distributed in arbitrary, multi-modal ways, which vary over time. All of this makes it difficult for practitioners to encode input values and select appropriate parametric forms for output distributions.
C2FAR allows efficient modeling of any univariate numeric output ($3) C2FAR extends prior work
using categorical output distributions over binned representations 49 19 While the precision of
prior methods is limited by the number of bins in the discretization. C2FAR achieves an exponential increase in precision through only linear increase in the number of bin parameters. Like digits in the decimal number system (and unlike, say, roman numerals or simple tally marks), C2FAR represents scalar quantities hierarchically, treating numbers as multidimensional objects, with each successive dimension or level representing the original value at a progressively finer resolution (Fig. T).
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

12 hins
6 12 bins
4 12 hins 4 intervals
intervals
62 36 intervals
(a) Flat binning
(b) 2-1evel C2F binning
(c) 3-1evel C2F binning
Figure I: Flat vs. coarse-to-fine binnings for a traff time series (55 C2F discretizations specify bin indices from coarse-to-fine, e.g. 3, 2, For the same total number of bins, the number of unique intervals a C2F binning is much higher (and thus reconstruction error is much lower).
C2FAR proposes autoregressive generative model for such representations. Levels are generated hierarchically from coarse to fine. Finer and finer intervals of support are generated from a sequence of categorical (binned) distributions, each distribution conditioned on the previously-genenann intervals. The conditional distributions are parameterized by a neural network, and the model is fit by maximizing log- log-likelihood of observed data. Compared to flat binnings, C2FAR models are more efficient for a given level of precision, and better enable learning of order and distance.
coarser
We use C2FAR to model output distributions in deep forecasting model based on DeepAR 55 In our experiments, 12FAR-based forecasting models better recover synthetic distributions compared to recent state-of state-of-the of-the-art methods ($5.1). Evaluation on benchmark forecasting datasets shows that, contrast to prior work, discretization improves accuracy ($5.)). Multi-level C2FAR models further improve over flat binnings across all datasets, including evaluation on a large new dataset of public cloud demand, which we release as a paper supplement. Code and data for C2FAR are available at https: //g::://iiii thtb..tt....... com/huaweicloud c2far forecasting
2 Background and related work
2.1 Probabilistic neural forecasting, use of binned output distributions
Probabilistic forecasting architectures include LSTMs 71 555, temporal convolutions
58 14 and
transformers 337 77
While C2FAR can be used with any such architecture, for simplicity we
restrict discussion in this paper to models that autoregressive in time [37 (55) While non-AR
models typically output best-guess point forecasts [1 (8.777 77] or specific quantiles of interest [7] [7]200. they can be made to output C2FAR parameters all horizons (as was done with Gaussians
Autoregressive networks must define step-wise wise input and output. To handle series of very different
scales, input values are often normalized (e.g., dividing values by their mean [37 55 49
output
values are scaled back after prediction. density over outputs achieved by mapping network states to location and spread parameters of specific distributions. Practitioners must choose a distribution
"to match the statistical properties of the data" (55) In practice, Gaussian [55] 37 Student's s-t 2) and Gaussian mixture distributions 46 have been used for real-valued data, while negative binomial
distribution [55] has been employed for discrete. Since output uncertainty is often not well captured by standard parametrics, recent work has investigated more flexible outputs, including spline quantile functions Ø and implicit quantile networks 28), which we compare to below ($5)
Rabanser et al. [49] use categorical output distribution over a binned representation. Categoricals are flexible, but do not encode any underlying concept of bin order distance (which must be learned). In prior work on pixel modeling, categorical distributions sometimes work better [69]. sometimes worse 53 than mixture densities. In [49, benefits of binning were mixed, and harmed DeepAR accuracy. Ehrlich et al. [9 "splice" continuous Pareto distribution into categorical in order to model extreme values; we show how Pareto distributions can also be used when samples are generated autoregressively using C2FAR

---

2.2 Challenges in forecasting mixed data, the density spike issue
Many time series are either fully discrete (e.g., counts), or continuous but with "clumps" at particular values. Dollar values are often specified to two decimal places. Even intrinsically continuous series are always processed or serialized to certain precision; for example, one version of elec has been quantized to integers, another to six decimal places ($ As cloud provider, we observe many semicontinuous [4] series, with significant probability of being either 0% or 100%, while otherwise being continuous (zero-inflation is also common in fully discrete demand data 13 57
Modeling discrete data can be challenging. The inherent limits of discrete series are rarely known at prediction time, so unbounded distributions such as negative binomials must be used 61]. Un- fortunately, is "not possible" for such distributions to output values in a normalized domain (55), precluding valuable nonallooomaliztionn. But since real-valued forecasts are usually acceptable for discrete data, practitioners often simply normalize discrete data and train continuous models. Gouttes et al. (28) report better results on wiki with Student's s-t distribution than with negative binomial. Rabanser et al. [49 normalize wiki prior to "discretizing" again for their categorical. We have also encountered practitioners with the (erroneous) belief that normalizing makes discrete data continuous.
Unfortunately, fitting powerful continuous models to discrete/mixed data "can lead to arbitrarily high density values, by locating a narrow high density spike on each of the possible discrete values" [68). While this is known issue in image modeling [66). it is apparently not well appreciated in forecasting. Density models based on splines 224 or flows (22 should therefore not be trained directly to maximize likelihood, as this may guide parameters (e.g.. spline knot positions) solely to enable density spikes. A potential solution is dequantizing, e.g., adding Uniform [o, 1] noise as in [52]. But this assumes: (1) we know the discrete series a priori, and (2) the resulting loss precision does not outweigh the benefits of continuous models. For C2FAR, the ability to locate high-densin spikes on discrete values feature, not a bug. C2FAR does not generate arbitrarily narrow spikes, rather it is restricted by the precision of the hierarchical binning. C2FAR can therefore train for likelihood on the actual discrete data, and it can generate realistic samples reflecting the true data distribution.
2.3 Coarse-to-fine representations, hierarchical softmaxes
Coarse-to- tousse-o--fie binnings have been used previously in machine learning. A coarse-to- tosse-ttofinee fine representa- tion is induced by algorithms that recursively discretize continuous-valuee attributes for decision tree classifiers (21] On the output side, SSR-Net (74 classifies human age in a coarse-to-fine (but not
autoregressive) manner. iSAX [59] uses coarse-to-fine representation for indexing time series.
Prior autoregressive models of discrete data can be viewed as instances of C2FAR. WaveRNN [3 models 16-bit audio samples by generating first the coarse (high) 8-bits, then the fine (low) 8-bits, conditioned on the sampled coarse. Vipperla et al. (70) found a different split between coarse and fine bits resulted in lower cross-entropy. These approaches can be regarded two-level C2FAR models with classifier conditionals. The multi-stage likelihood from [57] can be viewed as a three-level C2FAR model, with binary classification at level 1 and 2, and Poisson regression generate final counts. C2FAR unifies these approaches and generalizes them to unbounded, mixed/continuous data.
Beyond numeric data, hierarchical decompositions of categorical output distributions have been used previously in language modeling [45 44 As with C2FAR, the key benefit exponentially fewer computations than when predicting words with a flat softmax. Whether further efficiencies such as variable-length representations (based on Huffman coding) 41 could prove effective when modeling continuous and ordinal data with C2FAR merits further investigation.
3 C2FAR networks
The process of discretization first partitions the support of a continuous variable into a finite number of distinct bins. A value is discretized by mapping it to the index of its corresponding bin. With C2FAR, partitioning hierarchical: the support first divided into a number of coarse bins, each of these coarse bins is further divided into a number of finer bins, and so on (recall Fig. D in I C2FAR thereby discretizes a data point, z, into a vector of B indices for each of the B coarse-to-line levels, 2 = (2) ..... 2 BB). More formally, C2FAR assumes discretization function d : IR + (B),... .n)) where each B, has one of [B,| distinct values. Function d' is a kind of inverse of d, mapping a set of bin indices to an interval in the original real-valued domain, d' (BI.... .B.) + (IR, RR.
3

---

Observed z
B
Sampled ~ Pr(Z)
2
Network
h2
Network
Inputs
x, 22,200
Inputs
(a) Training
(b) Prediction
C2FAR network for regression. We train to minimize NLL of a set of examples; log loss on each example decomposes into the sum of NLL of each 2' under its categorical distribution. In prediction, we sample from the categoricals, and autoregressively use the sampled values as inputs.
Figure 2:
Let z - d(2) be the B-dimensional discretization of z. Let < denote the first - 1 bin indices, 2² , ='=1, Let be an optional set of features that may inform z's distribution (e.g.. for mixed data, x could flag which output bins contain integers). We follow prior work in modeling multivariate conditional distributions p(z|x) as autoregressive generative models via the chain rule [2 67
B
p(z|x)
II
p(z x, 25)
In C2FAR, each of the mie-dimesioonal distributions, p(="|x, z² is a categorical over the set of bin indices the ith granularity. B, Intuitively, the ith categorical is distribution over the support within the generated (i 11-1)p 1) bin. We parameterize these distributions using neural networks with a softmax output layer (specific architectures are discussed below), and fit the parameters to minimize negative log-likelihood (NLL) of training data (Fig. 2a). C2FAR is agnostic toward the encoding of the inputs 2"; they may, for example, be represented with I-hot-encodings or embedding layers. As generative model, we synthesize new data by first sampling coarse bin (# given then iteratively sampling finer and finer bins given the sampled (and encoded) coarser values (Fig. 2b).
C2FAR supports variety of neural architectures: following prior work in multivariate distribution modeling, we may use a separate (feedforward) network for each categorical (6) (as in Fig. 2 or
one global network with masking to enforce the autoregressive property [26). Separate networks may share parameters across the levels (36 or with classifications the same level across time (
Having separate networks for each level means the modeling problem at given level need not change we add more levels to the hierarchy; we adopt this approach in our forecasting experiments ($4.2).
While not strictly required, we use linear (evenly-spaced) binnings at each level. We hypothesize this enables the finer networks better generalize learned concepts of order and distance (i.e., regardless of the coarse bins that they are conditioned on). We refer to the span from the first interval to the last interval as the extent of the binning. The extent, the number of levels, and the number of bins at each level, are C2FAR hyperparameters. We also consider the extreme high/low intervals to be open-ended, terminating at łoo; we discuss the implications of this below.
Level + parametric distributions. Let (a, b) be the interval defined by ba (z). To complete generative story, we must generate value within (a, b) (rather than choosing single "reconstruction value" for each bin as in (49). Since our extreme intervals are open-ended, we cannot interpret our model piecewise uniform distribution (69), as uniforms are only defined on finite intervals. C2FAR solves this by allowing different distributions to be used depending on the interval (a, b). Essentially, we assume final conditional in (T) (implicitly level B 1) that generates from a (differentiable) parametric distribution. We use distributions of the form:
Uniform(u [a,
xo < a and b < 00
p(zB+1|a, b)
Paretofa, o,] b b=a
Pareto o2)
a = -0
Here Pareto indicates Type Pareto distribution with fixed scale parameter (a or b, defined a priori from the extent of the binning in the observed space) and dynamic shape parameter ag, Parameters

---

Z1
7it+1
2
2
2
p(z 1(71-100-11 10-1)
p(zs(0))
p(
(170-10662))
p(zz)0e)
plz (0++1)
14-1
h1
le41
h--1
he
het1
21-2,
X(-1, 1,24-11
Be-1, Xx, Zi
B
Ze, X
<B
21-2, 1.2
25B
Nx, 2,
21. x 2
(a) Training
(b) Prediction
Figure 3: High-level C2FAR-RNN for forecasting. Unlike DeepAR, C2FAR-RNN uses discretized input/output vectors, 24, rather than scalars, and has additional (autoregressive) inputs z,
a1j and 02 are generated by the hB+1 neural network in a manner analogous to how DeepAR outputs the mean and variance parameters of Gaussian distribution [55]. We are effectively defining piccewise-unifore density with Pareto-distributed tails. We use this model for both continuous and discrete data. If data are truly discrete and precision is useful, bins/levels. If one knows a priori the true discrete support, one may instead use discrete distributions at level B + 1, such negative binomials, Poisson regression [57] discrete uniforms.
tuming procedure will choose more
Complexity. Consider C2FAR discretization with B levels and an unvarying cardinality of K bins at each level; the original support of z is effectively partitioned into K total intervals, but modeled using only K B categorical outputs in total (B softmaxes with K values each). We may regard C2FAR as K K-ary tree of height B. At each node in the tree, we determine which of the K bins we fall into at that level, and follow the chosen branch to the sub-tree at the next level. We need only evaluate the softmax probabilities (and backpropagate gradients) for the B nodes on the path from the root to the leaf of the tree. Since the height of the tree, B, is logarithmic (in base K) over the total number of intervals, we compute exponentially fewer outputs with C2FAR compared to flat binnings. The final likelihood additionally requires computing the probability at the B + level according to the corresponding parametric distribution (Uniform Pareto) given in Eqn. ( In this way, full C2FAR densities are never explicitly manifested when training or predicting: for plotting (e.g., Fig. 6). explicitly compute likelihood at tiny increments a predefined range.
4 Forecasting with C2FAR
We now explain how C2FAR be used for N-step-aheaad probabilistic forecasting, We base the forecasting framework on DeepAR [55], which has served as the basis for other recent improvements in forecast distribution modeling [4 28) and therefore facilitates experimental comparison ($5).
4.1 RespAR- probabilistic forecasting
Let 2, be the value of a time series at time t, and x; be vector of time-varying features or covariates. Probabilistic forecasting aims to model the conditional distribution of N future values of 2, (the prediction range) given the T N covariates, and T historical values (the conditioning range);
p(=T+1.... 2T N 2T, NT+N) def p(ZT41TTHN X1:++)
IIn this way, "weighted" Pareto tails are "spliced" into the distribution fixed user-defined quantiles, they are in [9] In C2FAR, Pareto tails provide the probability density function of values in the extreme bin, given the values are in the extreme bin. Each Pareto density itself integrates to 1. but is dynamically "weighted" by the probability of being in the corresponding extreme bin. Note also that if data does not have heavy tails, alternative distributions may be used in the extreme bins instead, e.g., left and right- trunccardd Gaussians.
quantile function with linear splines. SQF- RNN 24 actually enforces piecewise uniformity, over a finite range. Neural spline flows [18] require derivatives of the spline functions to match at knots, to avoid "numerical issues." By parameterizing the PDF directly using classifiers, and only manifesting the portion needed, C2FAR has none of the above restrictions, while being efficient and well- well-behaved.
"By parameterizing

---

21+1
p p 1 18+1)
h2
h4
h4+1
h2
1-1
21
Xt,
2E1 27
B
2, X(+1 2+41
(a) Training
(b) Prediction
Figure 4: C2FAR-RNN model detailing high-level unit, he, from Fig. 3 train to minimize sum of NLL at each under categorical with parameters given by level-specific RNN, hy. We predict by sequentially sampling a discretization at each time step, 2g, which is re-input at step +
DeepAR [55] formulates this conditional distribution as an autoregressive generative model:
T+N M = II [=+11
T+N
Plalzua- = II [=++1
p(z, 0, = y(h)))
where hr =
is the output of LSTM 33) recurrent neural network, and y(-) function that maps the output of the LSTM to the parameters of parametric distribution p(z (t). For example, ((2)(0.) could be a Gaussian and y(-) output the Gaussian's 's mean and variance as 9,. Fig. 3a illluutttates the overall architecture (with additions for C2FAR noted). Information about the conditioning range 2y is conveyed through the state of the network at time T (i.e., hr). From encoder-decoder perspective [63] 3. DeepAR uses the same network to encode and decode.
For training, each series is sliced into multiple windows, i.e., conditioning-pprdiitioon ranges at different start points. Windows are normalized using their conditioning ranges, and parameters are fit minimize NLL of prediction-range outputs. To generate forecast for a given (normalized) conditioning range, DeepAR draws samples êr+1... êT+N in sequence from p(z, (0,) (Fig. 3b). Sampled roll-outs are unnormalized, and by repeating this procedure many times, a Monte Carlo estimate of 3) is obtained, from which desired forecast quantiles can be derived (see, c.g., Fig. 7
4.2 Forecasting with C2FAR: C2FAR-RNN
We augment DeepAR by converting time series their C2FAR discretization z, d(z)). Rather than generating Og in one shot at each time step, we generate C2FAR categoricals autoregressively within each time step (Fig. 4). Each generated bin index, = thus informs the distribution of the next bin index at that time step 2f. We also leverage information about discretized values at earlier time steps by replacing C2FAR's classifier-based conditionals ($3) Fig. 2 with LSTMs, one for each level in the C2FAR hierarchy. Intuitively, when generating bin index z our inputs comprise both current position in higher-level bins (e.g. 2 and the ith-level index at the previous time step (=f_1), along with the previous L.STM state, hi-1-
We call the resulting system C2FAR-RNN, As with DeepAR, C2FAR-RNN is trained to minimize NLL of observed outputs in (normalized) prediction ranges (Fig. 4ah. During training (and when evaluating NLL of test sequences), at each time step, all values are known and all distributions can be computed in parallel. When predicting (Fig. 4b). we must sample each bin index 2 sequentially.
Recall that C2FAR has an implicit B + level, where a real value is generated from parametric distribution (Eqn. 2 We do not use an RNN for this level; the uniform distributions are fully- specified by the interval endpoints, while we generate Pareto parameters (a1, 02) via a simple feed-forward neural network (with a single hidden layer and softplus output transformation). To help inform the Pareto networks, we also encode real-valued 24-1 additional input/covariate.

---

Complexity. Let H be the number of RNN hidden units, K a constant number of bins per level, and B the number of levels (I K total intervals). The mamplexit--per-tieenn essentially the sum of the RNN's recurrence operation, H x H, and the projection of the recurrence to output bins, K x H. For flat binnings, overall complexity is typically dominated by K x H, while B-level C2FAR-RNN models are dominated by B x H x H, i.e., the cost of running B RNNS in parallel. Measurements of timing and memory consumption (supplemental Table S 9 are well explained by these observations and corresponding values for H, K, and B (supplemental Table 6).
of each RNN is
Tuning. While C2FAR trained for NLL, is tuned for a given target metric. For forecasting.
the metric is multi-step-aheed error. Thus we evaluate our forecasts during training by periodically running Monte Carlo sampling on our validation set and computing such as anomaly detection, denoising, etc., we would train for NLL and tune for an application-speccin metric. While there is no one-size-fit-aal metric for evaluating generative models, oogliikeldoodd itself is sometimes regarded as the de facto standard [66]. Tronically, this is the one loss function C2FAR can not tune for, at least not on discrete data, because of the density spike issue (2.2) we tune directly for NLL, the tuner chooses more and more bins and levels, leading to narrower and higher spikes. If we must tune for logllikelihood, we could use discrete distributions in the B + I level. We could also add uniform noise, other approaches do (but note in prior work this is required to enable training, not tuning, on discrete data). NLL evaluations are explored in supplemental E
cuulii-sep-ahheas error. For tasks
4.3 Limitations and broader impact of C2FAR forecasting
Every B-level C2FAR model has an equally expressive [50] flat counterpart with single categorical over all the fine-grained intervals; given unlimited training data, C2FAR may therefore not offer modeling benefits over flat binning. Conversely, when there is limited data, simple parametric distributions, with fewer parameters, may generalize better than C2FAR. Also, given it has extra hyperparameters (number of levels/bins-per-leeee with added hyperparameters, C2FAR may not discover optimum settings as quickly. We investigate experimentally whether the benefits of C2FAR outweigh these drawbacks on real-world data ($5.2).
and given tuning search space grows exponentially
C2FAR is more complex than flat binning, but whether is less efficient depends on implementa- tion. Compared to their 2-level coarse-to-fine model, WaveRNN (33] found flat binning required "significantly more parameters, memory and compute." We find C2FAR to run slower, but with less memory (Tables g and 9 in the supplemental). If flat binning was truly more efficient, we could always generate large volume of data from a C2FAR model and uptrain 10 a flat model on
Broader impact. By enabling simultaneous modeling of discrete and continuous series, without human involvement, C2FAR step toward universal forecast model. Large C2FAR models could be trained on vast quantities of diverse series (similar to efforts in text [16. 39 75 and vision (60, 64 ) Such models could be fine-tuned for new domains, and help make highly- accurate forecasting systems more widely-used, improving decision making and resource allocation.
There are also risks to this approach. Very large models have environmental and financial costs 4) which may be unnecessary when smaller models suffice. A universal model may be used more easily by those with less expertise and this may lead to misuse; for example, automation bias has been shown to disproportionately affect those with less domain expertise [9]. Recommended usage and possible misuse should be documented through artifacts such model cards 43) and datasheets 25
5 Experiments
C2FAR is implemented in PyTorch [47] using a 2-layer LSTM 33 with intra-layer dropout 62 40
trained via Adam (34] Notation C2FAR-RNN, n refers to B-level C2FAR model. We evaluate:
C2FAR-RNN;; essentially the standard flat-binning approach
44 but with Pareto tails 19
C2FAR-RNN,; a two-level C2FAR-RNN model
C2FAR-RNNY a threc-level C2FAR-RNV model
'From discretization perspective, the optimal 2, as rather than a softmax, we may use a logistic function thresholded at 0.5 to select high vs. low bins, only computing log, total outputs. However, complexity also depends on the neural architecture. As C2FAR 12FAR-RNN RNN operates B separate RNNs, it is more efficient to use a higher value of K (but not so high that computing output probabilities dominates) and lower value of B.
7

---

1.0
1.0
True distribution
True distribution
Sampled distribution
Sampled distribution
0.6
0.6
0.0
10
Figure 5: Distribution recovery: CDFs for a Gaussian mixture (left) and discrete uniform over 1... 10
(right). After training C2FAR-RNNs each, sample paths and plot the sampled CDF, C2FAR
recovers the true CDF much better than SQF-RNN or IQN-RNN (cf. Fig. 3 in (24). Fig. 2 in (28
5.1 Distribution recovery from synthetic data
We first evaluate C2FAR-RNN's ability to recover the distribution of synthetic data, following 22 28 We generate data exactly as [24], creating 500 series with a year's worth of hourly observations, each value sampled independently from a 3-component Gaussian mixture with weights [0.3, 0.4, 0.3). mcans [-3, 0, 3] and standard deviation of 0.4. To illustrate C2FAR-
ability to model discrete data, we create equivalent dataset where every element is drawn from discrete uniform, ( 10).
On each dataset, we trained C2FAR-RNN, model with 20 bins per level and hyperparameters based on elec experiments (below).
Results. In prior work, SQF-RNN 24 and IQN-RNN 28 struggled recover the Gaussian mixture, but C2FAR can fit both distributions perfectly (Fig- 5). C2FAR is evidently the state-of-the-art in capturing complex, multi-modal densities without any prior information.
5.2 Empirical study on real-world data
Datasets. We evaluate C2FAR forecasting models on the following datasets:
elec: hourly electricity usage of 321 customers [17, using discretized version from
raff: vallev hourly occupancy from 0 to for 862 car lanes 17 using version from [76
wiki, daily integer number of hits for 9535 pages, first used in (24, using same version as 28
azure; (discrete) hourly usage of virtual machine (VM) flavors and flavor groupings (by tenant, deployment type, etc., in units of VCPUs or GB of memory) in a large public cloud, based on data
from [15). We release this dataset publicly as a paper supplement (see supplementary $ for further details).
For azure, we use 20 days as training, 3 days for validation, and 3 final days for testing. Other splits are as [54]. We forecast 24 hours for hourly series, 30 days for wiki, and use rolling evals in 28
Metries. To evaluate point forecasts, we output the median of our forecast distribution and compute normalized deviation (ND) from true values. We evaluate probabilistic forecasts using weighted quantile loss (wQL), evaluated at forecast quantiles (0.1, .........00... 0.9}. as in (4). For non-probabilistic baselines Naïve and Geasonal-naive (described below), note wQL reduces ND, similarly to how CRPS reduces to absolute error (27] $4.2]. We also test the calibration and sharpness [71] of each system, measuring coverage of true values within particular percentiles (e.g. from 10% 90%. coverage closer to 80% is better) and normalized width of this interval (lower is better). For target coverage X%, we refer to these metrics together as CovX (e.g. Cov80). Scores at extreme percentiles (e.g., Cov99%) help evaluate modeling of distribution tails.
Baselines. We compare our model to the following baselines:
Naive 32 outputs the last-observed historical value all forecast horizons
Saave [32 : at each horizon, outputs the value at the most-recently-ooseeeed season
matching the season at that horizon. We thus output the value at the most-recently-obseeee matching hour-of-day for hourly series, and matching lay-of-weck. for the daily dataset (wiki).

---

Table 1: ND, wQL.. Cov80, and Cov99 for implementations (top), results from 49 (middle, denoted f) and [28 (bottom, denoted f), where available. In all cases, flat binned C2FAR-RNN, while deeper C2FAR-RNN, likewise improves C2FAR-RNN; Results are generally superior to prior state-of-the-art output distributions in RNN-based forecasting.
improves
ND%
wQL%
Cov80% azure
Cov99%
elec
traff
wiki
azure
elec
traff
wiki
azure
azure
Naïve
40.8
73.6
35.7
3.49
40.8
73.6
35.7
3.49
Seasonal-naive
6.97
25.1
33.2
3.67
6.97
25.1
33.2
3.67
ETS
8,61
33.3
34.3
3,46
8.40
31.5
32.5
2.97
85,5/10.5
96.3/20.3
DeepAR- AaGassaan
7.05
16.1
43.8
3.60
5.60
13.7
54.7
3,06
89.9/16.9
98.0/37.7
CZFAR-RNN,
6.14
13.0
24.6
2.95
4.87
10.7
21.3
2.41
83.6/8.3
98.5/32.2
C2FAR-RNI
6.09
12.9
24.2
2.86
4.83
10.6
21.0
2.31
79.0/8.5
98.4/29.1
C2FAR-RNN:
6.00
13.3
24.1
2.77
4.76
10.9
21.0
2.27
86.0/8.9
98.6/32.7
DeepAR-Binng R-Binned
8.21
23.2
94.6
6.47
18.8
84.7
DeepAR StudentTt
6.95
14,6
26.9
5.71
12.2
23.8
IQN-RNN
7.40
16.8
24.1
SQF-RNN?
9,70
18.6
32.8
7.80
21.6
27.0
probabilistic state space models based on exponential smoothing, implemented in statsmodels [56). For each dataset, we tune (on validation data) whether to include seasonality and trend (damped undamped) terms, and whether to use estimated or heuristic initialization.
ETS B1:
DeepAR- DeepAR-Gaussingg Gaussian: our implementation of DeepAR with Gaussian output distribution; Gaussian outputs are common practice, even on discrete data [37 55
Tuning. Deeper C2FAR models include shallower models as special cases. We therefore address the question: does C2FAR's improved modeling outweigh the wider tuning required? For fair comparison, we establish a parameter budget (40) of 1M parameters; after sampling other hyperparameters, we restrict the possible number of LSTM hidden units so that this cap is enforced. We also restrict the number of tuning trials (on validation data) to 100 for each system.* 4 We tune directly for normalized deviation via Optuna M. with TPESampler
g using MedianPruning and early stopping.
Results. Table I confirms that deeper C2FAR models improve over flat binnings. In only one case (traff) did C2FAR model not improve over flat binning. In this case, C2FAR-RNNs, with larger tuning search space, also failed to find a superior setting of hyperparameters on validation data; going forward, using the same number of bins each level (i.e., a constant K) could enable deeper C2FAR models without additional tuning. Contrary to prior work, we also find flat binning much more effective than standard parametric distributions. We attribute this difference our of systematic tuning; prior work used a fixed 1024 bins [49]. while our tuner often selected quite fewer bins for C2FAR-RNN, (supplemental Table 6). In terms of both standard and extreme percentiles (Cov80% and Cov99% in Table D we find C2FAR models are both better calibrated than DeepAR-Gaussing (being twice as close to the desired coverage) while also having a sharper prediction interval width.
Seasonal-naive is surprisingly competitive with DeepAR DeepAR-Gaussian Gaussian (except on traff', which has both daily and weekly seasonality). This has been observed previously (e.g., Table 1 in (2);; indeed, one of the authors of DeepAR and Gluon'SS has remarked that "a seasonal naive model performs almost well as DeepAR other deep models [elec and traff), and without additional covariates, I suspect it's almost impossible to perform significantly better in terms of point predictions" (33. But our results show that C2FAR can significantly improve over DeepAR and Seasonal-naive without using additional covariates, but rather with a better output model.
Analyzing the distributions, we find C2FAR models are able to achieve good precision (having many intervals), while simultaneously learning smooth distributions (Fig. 6). Flat binnings must first learn "that value of 128 is close to a value of 127 or 129" [53), while for C2FAR models, intervals will implicitly be close in probability because they are in the same coarser bins.
*In the prior work that mentions hyperparameter tuning, is common to compare systems tuned with fixed number of tuning trials, whether as part of grid search 77 78 51 54 or other procedure (e.g.. hyperopt in [12]. random search in (8) Training times the same order for all our trained systems (supplemental Table 7

---

DecpAR-GGussinn (azure)
C2FAR-RNNI (azre))
C2FAR-RNN, (azwe)
DeepAR Gaussian (wikk)
C2FAR-RNN, (wik))
C2FAR-RNN (wiki)
Figure 6: Distributions possible next values. To cover likely values, Gaussians (left) also cover many unlikely ones. Flat binnings (middle) are optimized to either use fewer bins, but suffer in precision (top), many bins but suffer in noise (bottom). 22FAR-RNNY (right) is able place high probability on a repeat of the previous value (top) or generate smooth distributions (bottom).
history
real
MA
real
MA
90%
MYVS:
p50
p50
MM
MM
DUN
UUM
DeepAR-Gaussian Gaussiia (elec)
C2FAR RNNg (elec)
Figure 7: Forecast for an elec time series. DeepAR- DeesAR-Gaussian (left) yields reasonable p50, but its lower percentiles span too low. C2FAR- R yields better lower quantiles. C2FAR percentiles also suggest high-fidelity roll-outs, i.e., samples that closely mimic the discretized dynamics of the series.
Autoregressive models have known disconnect between training, when true values used as autoregressive inputs, and generation, when samples are used [5]. Our results suggest that such error accumulation may be especially problematic when model is trained discrete or mixed data, but uses standard continuous parametric outputs (as in DeepAR-Gaassiann only provide better forecast quantiles, they enable figher-fidlliyy samples to be recursively fed back in as autoregressive inputs (Fig. 7).. Analyzing error by forecast horizon (supplemental (C.8). we find that shorter horizons, SeepARR-Gausae horizons, the gap widens. So while other solutions to error accumulation exist 65 71 73 one effective approach is evidently to generate and recurse on higher-fidelity outputs using C2FAR.
Precise C2FAR models
often performs similarly to C2FAR, but later
6 Conclusion
We presented C2FAR, new method for density modeling. C2FAR reduces the generative process to a sequence of classifications over a hicrarchical, discretized representation, with special handling of data outside the binning range. C2FAR can be applied to variety of neural architectures, including RNN-based probabilistic forecasting, where achieves state-of-the-at results when recovering synthetic distributions and forecasting real-world data. We show binned models (whether flat or, especially, coarse-to- caare-too-fine) fine) are superior to standard distributions - binning precision is tuned.
Analysis shows that C2FAR can successfully model complex, multi-modd densities, in real or discrete data, without any prior information. This flexibility enables improved modeling for a variety time series use cases, including forecasting, anomaly detection, interpolation, compression, denoising, and generating high-fidelity samples. also enables development of large-scale forecasting models trained on diverse datasets, i.e., is step toward universal neural forecaster.
10

---

References
[1] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A Next-generation Hyperparameter Optimization Framework. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2623-2631.
[2] Alexander Alexandrov, Konstantinos Benidis, Michael Sohlke-Sccheidee Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Danielle Maddix, Syama Rangapuram, David Salinas, Jasper Schulz, Stella Lorenzo, Ali Caner Türkmen, and Yuyang Wang. 2020. GluonTS: Probabilistic and Neural Time Series Modeling Python. Journal of Machine Learning Research 21, 116 (2020), 1-6.
[3] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. International Conference on Learning Representation (2015).
[4] Emily Bender, Timnit Gebru, Angelina SoMMillanMMaoor and Shmargaret Shmitchell. 2021.
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. In ACM Conference on Fairness, Accountability. and Transparency. 610-623.
[5] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. Advances Neural Information Processing Systems 28 (2015).
[6] Yoshua Bengio and Samy Bengio. 2000. Modeling High-Dimensional Diserete Data with Multi-Layer Neural Networks. In Advances in Neural Information Processing Systems.
[7] Konstantinos Benidis, Syama Sundar Rangapuram, Valentin Flunkert, Yuyang Wang, Danielle David Salinas, Lorenzo Stella, Xravie Aubet, Laurent Callot, and Tim Januschowski. 2022. Deep Learning for Time Series Forecasting: Tutorial and Literature Survey. Comput. Surveys (2022).
Maddix, Caner Turkmen, Jan Gasthaus, Michael
[8] James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for Hyper-Parameter Optimization. Advances in Neural Information Processing Systems 24 (2011).
[9] Raymond R Bond, Tomas Novotny, Irena Andrsova, Lumir Koc, Martina Sisakova. Dewar Finlay, Daniel Guldenring. James McLaughlin, Aaron Peace, Victoria McGilligan. Stephen J. Leslie, Hui Wang, and Marek Malik. 2018. Automation bias in medicine: The influence of automated diagnoses interpreter accuracy and uncertainty when reading electrocardiograms. Journal of Electrocardiology 51, 6 (2018), S6-S11
[10] Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil 2006. Model Compression. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 535-541.
[11] Defu Cao, Yujing Wang, Juanyong Duan, Ce Zhang, Xia Zhu, Congrui Huang, Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong, and Qi Zhang. 2020. Spectral Temporal Graph Neural Network for Multivariate Time-series Forecasting. Advances in Neural Information Processing Systems
33 (2020).
[12] Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza, Max Mergenthaler, and Artur Dubrawski. 2022. N-HiTS; Neural Hierarchical Interpolation for Time Series Forecasting. arXiv preprint Xixi /2886 (2022).
[13] Nicolas Chapados. 2014. Effective Bayesian Modeling of Groups of Related Count Time Series. In International Conference on Machine Learning. 1395-1403.
[14] Yitian Chen, Yanfei Kang, Yixiong Chen, and Zizhuo Wang. 2020. Probabilistic Forecasting with Temporal Convolutional Neural Network. Neurocomputing 399 (2020), 491-501.
[15] Eli Cortez, Anand Bonde, Alexandre Muzio, Mark Russinovich, Marcus Fontoura, and Ricardo Bianchini. 2017. Resource Central: Understanding and Predicting Workloads for Improved Re- source Management in Large Cloud Platforms. In Symposium on Operating Systems Principles. 153-167.

---

[16] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre- training of Deep Bidirectional Transformers for Language Understanding, arXiv preprint arXiv: 1810:08800 (2018).
[17] Dheeru Dua and Casey Graff. 2017. UCI Machine Learning Repository. http://archive ics.uci. edumml
[18] Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. 2019. Neural Spline
Flows. Advances in Neural Information Processing Systems 32 (2019).
[19] Elena Ehrlich, Laurent Callot, and Trançois-Xavier Aubet. 2021. Spliced Binned-Pareto Distribution for Robust Modeling of Heavy-tailed Time Series. arKiv preprint (0t::2000./090 (2021).
[20] Chenyou Fan, Yuze Zhang, Yi Pan, Xiaoyue Li, Chi Zhang, Rong Yuan, Di Wu, Wensheng Wang, Jian Pei, and Heng Huang. 2019. Multi-Horizon Time Series Forecasting with Temporal Attention Learning. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2527-2535.
[21] Usama Fayyad and Keki Irani. 1993, Multi-Interval Discretization of Continuous- Valued Attributes for Classification Learning. International Joint Conference On Artificial Intelligence 2 (1993), 1022-1027.
[22] Brendan J Frey, Geoffrey E Hinton, and Peter Dayan. 1995. Does the Wake-sleep Algorithm Produce Good Density Estimators? Advances in Neural Information Processing Systems (1995).
[23] Jan Gasthaus. 2020. Re: A Worrying Analysis of Probabilistic Time-series Models for Sales Forecasting [discussion comment]. https://gitthb. com/ watpm//wsaab/gguoostetoo discussions/ http////////oooootooooorrrrr orw. 169433
Posted: 2020-
11-30, Accessed: 2022-09-13.
[24] Jan Gasthaus, Konstantinos Benidis, Yuyang Wang. Syama Sundar Rangapuram, David Salinas, Valentin Flunkert, and Tim Januschowski. 2019. Probabilistic Forecasting with Spline Quantile Function RNNs. In International Conference on Artificial Intelligence and Statistics. 1901- 1910.
[25] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2021. Datasheets for Datasets. Commun, ACM 64, 12 (2021), 86-92.
[26] Mathieu Germain, Karol Gregor, Iain Murray, and Hugo Larochelle. 2015. MADE: Masked Autoencoder for Distribution Estimation. In International Conference on Machine Learning. 881-889.
[27] Tilmann Gneiting and Adrian E Raftery. 2007. Strictly Proper Scoring Rules, Prediction, and Estimation. J. Amer. Statist. Assoc. 102, 477 (2007), 359-378.
[28] Adèle Gouttes, Kashif Rasul, Mateusz Koren, Johannes Stephan, and Tofigh Naghibi. 2021. Probabilistic Time Series Forecasting with Implicit Quantile Networks. arXiv preprint arXiv:2107.03743 (2021).
[29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In IEEE Conference on Computer Vision and Pattern Recognition. 770-778.
[30] Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory, Neural computation
9, 8 (1997), 1780-1780.
[31] Rob Hyndman, Anne B Koehler, J Keith Ord, and Ralph D Snyder. 2008 Forecasting with Exponential Smoothing: The State Space Approach. Springer Science & Business Media.
[32] Rob Hyndman and George Athanasopoulos. 2018. Forecasting: Principles and Practice. OTexts.
12

---

[33] Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron van den Oord, Sander Dieleman, and Koray Kavukcuoglu. 2018. Efficient Neural Audio Synthesis. In International Conference on Machine Learning. 2410-2419.
[34] Diederik P Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. arXiv preprint arXiv: /412.6988 (2014).
[35] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. 2018. Modeling Long and Short-Tern Term Temporal Patterns with Deep Neural Networks. ACM SIGIR Conference on Research & Development in Information Retrieval. 95-104.
[36] Hugo Larochelle and lain Murray. 2011. The Neural Autoregressive Distribution Estimator. In International Conference on Artificial Intelligence and Statistics. JMLR Workshop and Conference Proceedings, 29-37.
[37] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. 2019. Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting. Advances in Neural Information Processing Systems 32 (2019).
[38] Bryan Lim, Sercan Ô Arık, Nicolas Loeff, and Tomas Pfister. 2021. Temporal Fusion Trans- formers for Interpretable Multi-Horizon Time Series Forecasting. International Journal of Forecasting 37, 4 (2021), 1748-1764
[39] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv: 1907. (2019).
[40] Gábor Melis, Chris Dyer, and Phil Blunsom. 2017. On the State of the Art of Evaluation in Neural Language Models. arXiv preprint 1:X..:/70..05588 (2017).
[41] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. arXiv preprint arXiv: 1301.3781 (2013).
[42] Yongyi Min and Alan Agresti. 2002. Modeling Nonnegative Data with Clumping at Zero: Survey. Journal of the Iranian Statistical Society (JIRSS) (2002).
[43] Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchin- son, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model Cards for Model Reporting. In Conference on Fairness, Accountability and Transparency. 220-229.
[44] Andriy Mnih and Geoffrey Hinton. 2008. A Scalable Hierarchical Distributed Language Model. Advances Neural Information Processing Systems 21 (2008).
[45] Frederic Morin and Yoshua Bengio. 2005. Hierarchical Probabilistic Neural Network Language Model. In International Workshop on Artificial Intelligence and Statistics. 246-252.
[46] Srayanta Mukherjee, Devashish Shankar, Atin Ghosh, Nilam Tathawadekar, Pramod Kompalli, Sunita Sarawagi. and Krishnendu Chaudhury. 2018. AR-MDN: Associative and Recurrent Mixture Density Networks for eRetail Demand Forecasting, arXiv preprint 1003.0000003000 (2018).
[47] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach Vito,, Martin Raison, Tejani Alykhan, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, Tiigh-Pefformace Deep Learning Library. Advances in Neural Information Processing Systems 32 (2019).
[48] Slav Petrov, Pi-Chuan Chang, Michael Ringgaard, and Hiyan Alshawi. 2010. Uptraining for Accurate Deterministic Question Parsing. In Empirical Methods Natural Language Processing. 705-713.
13

---

[49] Stephan Rabanser, Tim Januschowski, Valentin Flunkert, David Salinas, and Jan Gasthaus. 2020. The Effectiveness of Discretization in Forecasting: An Empirical Study on Neural Time Series Models. arXiv preprint arXiv:2005. 10111 (2020).
[50] Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. 2017.
On the Expressive Power of Deep Neural Networks. In International Conference on Machine Leurning. 2847-2854.
[51] Syama Sundar Rangapuram, Lucien D Werner, Konstantinos Benidis, Pedro Mercado, Jan Gasthaus, and Tim Januschowski. 2021. End-to-End Learning of Coherent Probabílistic Forecasts for Hierarchical Time Series. In International Conference Machine Learning.
[52] Kashif Rasul, Abdul-Saboor Sheikh, Ingmar Schuster, Urs Bergmann, and Roland Vollgraf. 2020. Multi-Variate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows. arXiv preprini arXiv:2002.06100 (2020).
[53] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. 2017. PixeICNN++: Improving the PixeICNN with Discretized Logistic Mixture Likelihood and other Modifications. arXiv preprint arXiv: 1 701.05517 (2017).
[54] David Salinas, Michael Schnee-S Laurent Callot, Roberto Medico, and Jan Gasthaus. 2019. High-Dimensional Multivariate Forecasting with Low-Rank Gaussian Copula Processes. Advances in Neural Information Processing Systems 32 (2019).
[55] David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. 2020. DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks. International Journal of Forecasting 36, 3 (2020). 1181-1191.
[56] Skipper Seabold and Josef Perktold. 2010. Statsmodels: Econometric and Statistical Modeling with Python. In 9th Python in Science Conference. 92-96.
[57] Matthias W Seeger, David Salinas, and Valentin Flunkert. 2016. Bayesian Intermittent Demand Forecasting for Large Inventories. Advances in Neural Information Processing Systems 29 (2016).
[58] Rajat Sen, Hsiang-Fu Yu, and Inderjit Dhillon. 2019. Think Globally, Act Locally: A Deep Neural Network Approach High-Dimensional Time Series Forecasting. arXiv preprint (2019).
[59] Jin Shieh and Eamonn Keogh, 2008. iSAX:
Indexing and Mining Terabyte Sized Time Series. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 623-631.
[60] Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large- Scale Image Recognition. arXiv preprint arXiv: 1556. (2014).
[61] Ralph D Snyder, J Keith Ord, and Adrian Beaumont. 2012. Forecasting the intermittent demand for slow-moving inventories: A modelling approach. International Journal of Forecasting 28, (2012),
[62] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research 15, 1 (2014), 1929-1958.
[63] llya Sutskever, Oriol Vinyals, and Quoe Le. 2014. Sequence to Sequence Learning with Neural Networks. In Advances in Neural Information Processing Systems, Vol. 27.
[64] Christian Szegedy, Wei Liu, Yangging Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going Deeper with Convolutions. In IEEE Conference on Computer Vision and Pattern Recognition. 1-9.
[65] Souhaib Ben Taieb and Amir F Atiya. 2015. A Bias and Variance Analysis for Multistep-Ahead Time Series Forecasting. IEEE Transactions on Neural Networks and Learning Systems 27, (2015), 62-76.
14

---

[66] Lucas Theis, Aäron van den Oord, and Matthias Bethge. 2015. A note on the evaluation of generative models. arXiv preprint arXiv:1511.0184+ (2015).
[67] Benigno Uria, Marc-Alexandre Côté, Karol Gregor, lain Murray, and Hugo Larochelle. 2016. Neural Autoregressive Distribution Estimation. Journal of Machine Learning Research 17, 1 (2016), 7184-7220.
[68] Benigno Uria, lain Murray, and Hugo Larochelle. 2013. RNADE: The real-valued neural autoregressive density-estimatee Advances in Neural Information Processing Systems 26 (2013).
[69] Aaron den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. 2016. Pixel Recurrent Neural Networks. In International Conference on Machine Learning. 1747-1756.
[70] Ravichander Vipperla, Sangjun Park, Kihyun Choo, Samin Ishtiaq, Kyoungbo Min, Sourav Bhat- tacharya, Abhinav Mehrotra, Alberto Gil CP Ramos, and Nicholas Lane. 2020. Bunched LPC- Net: Vocoder for Low-Cost Neural Text-To-Speech Systems. arXiv preprint #rXiv:2008.04574 (2020).
[71] Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. 2017. A Multi-Horizon Quantile Recurrent Forecaster. arXiv preprint arXiv:1711.11055 (2017).
[72] Haixu Wu, Jiehui Xu, Jiammin Wang, and Mingsheng Long. 2021. Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting. Advances in Neural Information Processing Systems 34 (2021).
[73] Sifan Wu, Xi Xiao, Qianggang Ding, Peilin Zhao, Ying Wei, and Junzhou Huang. 2020. Adversarial Sparse Transformer for Time Series Forecasting. Advances in Neural Information Processing Systems 33 (2020), 17105-17115.
[74] Tsun-Ye Yi Yang, Yi-Hsuan Huang. Yen Yu Lin, Pi-Cheng Hsiu, and Yung-Yu Yu Chuang. 2018. SSR-Net: A Compact Soft Stagewise Regression Network for Age Estimation. In International Joint Conference On Artificial Intelligence. 1078-1084.
[75] Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Russ R Salakhutdinov. and Quoc V Le. 2019. XLNet: Generalized Autoregressive Pretraining for Language Understanding. Advances Neural Information Processing Systems 32 (2019).
[76] Hsiang-Fu Yu, Nikhil Rao, and Inderjit S Dhillon, 2016. Temporal Regularized Matrix Fac- torization for Nigh-Dimensional Time Series Prediction. Advances in Neural Information Processing Systems 29 (2016).
[77] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wan- cai Zhang. 2021. Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting. In AAAI Conference on Artificial Intelligence. 1 11106-11115.
[78) Zhibo Zhu, Ziqi Liu, Ge Jin, Zhiqiang Zhang, Lei Chen, Jun Zhou, and Jianyong Zhou. 2021. MixSeq: Connecting Macroscopic Time Series Forecasting with Microscopic Time Series Data. Advances in Neural Information Processing Systems 34 (2021).
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] In partieular, $4.3
(c) Did you discuss any potential negative societal impacts of your work? Yes]] $4.3
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
15

---

(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proof' of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material full codebase is proprietary, however the core code for C2FAR is shared at httpss //githu.. cwm/huwweeclood//eeeccooooo ffoecassooo All datasets are either shared as paper contribution, or already public.
URL)? [No] The
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See $5.2 paper supplement.
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] See paper supplement.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See paper supplement.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] $5.2
(b) Did you mention the license of the assets? [Yes] For shared data, see supplement
(c) Did you include any new assets either in the supplemental material or URL? [Yes] See paper supplement.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] See paper supplement.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes] See paper supplement.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "xxs1uIuq7gZ", "reviewer": "Reviewer_BP7i", "summary": "This paper introduces a mechanism, C2FAR, to parametrize a conditional distribution via successive hierarchical binnings parametrized by an autoregressive function (here, a neural network). This parametrization can reportedly maintain expressibility in high dimensional binned distributions with logarithmically many parameters required. The authors present a thorough empirical study of their method in the backdrop of time series forecasting, where C2FAR-RNN (their hierarchical--coarse-to-fine--binned distribution on top of DeepAR) performs favorably both in forecast accuracy and calibration. ", "questions": "- The authors allude to limited precision in most forecasting data. I am not sure if this is really a justification for the method? Note that this is different than significant \"atoms\" of probability observed on 0% or 100%--reminiscent of zero-inflated distributions which are more similar to the authors' construction.\n- In Eq (1), should the lhs be $p(\\mathbf{z} | \\mathbf{x})$ instead?\n- On L129: \"Inputs may be represented with 1-hot-encodings or embedding layers\" Which inputs are these? $\\mathbf{x}$? If not, what is the difference implied between one-hot encoding and an embedding layer?\n- L192.  During training, and when evaluating NLL of test sequences, all values are known and all distributions can be computed in parallel. I am not sure of this, as the backprop would have to be sequential in any case with an RNN? \n- Why restrict the total number of hyperparameter optimization runs instead of total time? Are the training times on the same order? If so, please mention. Seeing as the sequential evaluation is the bottleneck in GPU implementations, then it may be that computation really takes B-times more time than DeepAR in training, which is unfair. I would urge the authors to report wall-clock training times of their method against DeepAR in the same number of bins. \n", "limitations": "Other than the last question mentioned above, I do not believe there are significant limitations of the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-written, well-organized and gives a good review of comparable methods in forecasting. To the best of my knowledge, the introduced techniques are novel, although I couldn't help comparing them to hierarchical softmax/categorical distributions used in NLP and wondered if any analogies could be drawn from there. In the forecasting domain, I believe the results posted by the authors are significant and advance the state of the art. Despite the simplicity of the ideas proposed, I believe the paper is insightfully written and benefits from a well-thought-out experimental set-up. Moreover, I believe C2F can have impact in other related areas of probabilistic deep learning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8c44MGMMfvJ", "reviewer": "Reviewer_sLp7", "summary": "The authors propose a new parametrize of the predictive distribution of probabilistic forecasting models. The build on prior work proposing binning or mixtures of uniform distributions (with parametric tails). They propose to avoid the pitfall of flat binning, where increasing the precision of the distribution requires to drastically increase the number of bins and so the number of output parameters of the neural network. They compare their method to existing work and show improvements on standard datasets.\n\n", "questions": "- How does one obtain a wQL% with the naive baseline, as the naive baseline does not provide a predictive distribution?\n- You mention using your implementation of DeepAR. Why not use the open source implementation of DeepAR in GluonTS?", "limitations": "The author discuss exhaustively the limitations of their work, which is to be called out. They discuss the complexity and the efficiency of the method, I can see indeed that the efficiency depends a lot on the implementation and I would have liked to get to see their implementation.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The core idea of the paper is very good. It is a simple and efficient mechanism to solve the niche but real problem of the increase in the bins requiring an increase in the number of output parameters. Having a finer grid of bins through auto-regressive additions, is very good idea.\n- I want to call out the honesty of the authors in the presentation of their work, their are openly vocal about the limitations of the work, which is something that should be praised.\n- The author make a very valid point on the discretization of time series in general in the section 2.2.\n\n\nWeaknesses:\nA few things are not explained or laid out very clearly in the current version of the paper, and I think that the paper would benefit a lot from having these written out more clearly:\n- I am not very clear on the parametrization of the neural network for p(z_i|x,z_{<i}). What kind of neural network is being used?\n- How is the NLL computed at training time? (I wonder if one could optimize the code since only one value is required, that of the bin where the data point falls) I think that a pseudo code would be helpful to understand how this is being done.\n- In the evaluation, please make clear what C2FAR-RNN1 is, and what the others are. I only saw in the appendix that if corresponds to K = 1.\n\n- I would like to point out that I could verify all these points by looking at the code, but the authors do not provide the code. I see this as a major issue and would want the code to be open sourced if the paper is to be accepted.\n\nI have difficulties understanding how the Pareto tails work exactly. In general, and compared to [17].\n- alpha_1 and alpha_2 are outputs of the last layer, are they values on the observed space or quantiles?\n- If they are values on the observed space, how is the density corresponding to the Pareto tail estimated?\n- Or, does the maximum bin value move with alpha_1? If so, how does the network cope with the changing bin positions?\n- Does having these tails force to compute the full densities?\n- Are any of the experiments done with the tails? Have you evaluated the calibration of the tails?\n\nWeaknesses in the evaluation:\n- I would like to point out that DeepAR-Binned is not uniform bins but simply discretized values. And with that, the paper does not compare to the main comparison partner [17] which has uniform bins (like the flat tiling of C2FAR) and Pareto tails.\n- It would be interesting to see the evaluation on a wider range of probabilistic forecasting datasets, and potentially regression datasets.\n- The is a natural trade-off between the number of bins K in each level and the number of levels B. It would be interesting to lay it out clearly and to evaluate what the optimal trade-off is.\n- It would be interesting to see the comparison to more flexible distributions like normalizing flows?\n- It would be good to show the results with standard deviation.\n- Overall, given the small methodological contribution of the paper it would have been valuable to provide a more exhaustive evaluation.\n\n\n\n[17] Elena Ehrlich, Laurent Callot, and François-Xavier Aubet. 2021. Spliced Binned-Pareto Distribution for Robust Modeling of Heavy-tailed Time Series", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4SERAX2tKhX", "reviewer": "Reviewer_coLS", "summary": "The paper proposes C2FAR, a new method for probabilistic univariate forecasting. C2FAR sequentially discretizes the time-series in bins, from coarse to fine resolution, using an RNN to learn temporal relations similar to the DeepAR model. At each layer, a shared RNN model further divides each bin into finer intervals based on covariates and values of different layers. Forecasts are produced by drawing samples conditional on the learned parameters. The paper compares the proposed method against other RNN probabilistic architectures, such as DeepAR, IQN-RNN, and SQF-RNN, on several benchmark datasets.", "questions": "- Inference (prediction) times are not reported or mentioned in the paper. Can you provide average inference times for different datasets?\n- Have you evaluated C2FAR on longer horizons? Since forecasts are produced recursively, errors will accumulate, degrading the performance considerably.", "limitations": "Limitations are discussed, and I do not identify additional potential negative social impacts.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Novel method for probabilistic univariate forecasting, based on sequential hierarchical binning.\n- SoTA performance on 4 benchmark datasets, compared to other RNN-based models.\n- The shared model between bins of each layer allows for exponentially expressiveness with linear cost.\n- The last layer can handle extreme values thanks to the Pareto-distributed tails.\n- Comprehensive ablation and stability experiments. The experimental setting is very clear and thoroughly explained.\n\nWeaknesses:\n- Only RNN architectures are used as baselines. While RNN is widely used, other architectures have shown superior performance in multiple forecasting tasks, especially for multi-step forecasting. Authors should include other models in the experiments, including simple baselines such as seasonal-naive, ETS, or ARIMA.\n- C2FAR requires more training time and resources than baselines. Training time should be also compared against other baselines.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept