The Time-Energy Model: Selective Time-Series Forecasting Using Energy-Based Models
Authors: Jonas Brusokas
Seshu Tirupathi
Dalin Zhang
Torben Bach Pedersen
Venue: Accepted by TMLR ()

Abstract: Time-series forecasting is an important task in many domains, including finance, weather prediction, and energy consumption forecasting, and deep learning methods have emerged as the best-performing time-series forecasting methods over the last few years. However, most proposed time-series forecasting models are deterministic and are prone to errors when deployed in production, potentially causing significant losses and penalties when making predictions with low confidence. In this paper, we propose the Time-Energy Model (TEM), a framework that introduces so-called selective time-series forecasting using energy-based models. Selective forecasting estimates model confidence and allows the end-user to selectively reject forecasts while maintaining a desired target coverage. TEM is model-agnostic and can be used to improve forecasting accuracy of any encoder-decoder deterministic time-series forecasting model. TEM is trained using a combination of supervised and self-supervised learning, leveraging excellent single-point prediction accuracy while maintaining the ability to reject forecasts based on model confidence. Experimental results indicate that TEM generalizes well across 5 state-of-the-art deterministic time-series forecasting models and 5 benchmark time-series forecasting datasets. Using selective forecasting, TEM reduces prediction error by up to $49.1\%$ over 5 state-of-the-art deterministic models. Furthermore, TEM has up to $87.0\%$ lower error than selected baseline EBM models, and achieves significantly better performance than state-of-the-art selective deep learning models. Code for the proposed TEM framework is available at https://github.com/JonasBrusokas/Time-Energy-Model

Published in Transactions on Machine Learning Research (February/0225)
The Time-Energy Model: Selective Time-Series Forecasting Using Energy-Based Models
Jonas Brusokas
jonasb
Department of Computer Science
Aolborg University
Seshu Tirupathi
seshutir Bie.ibm..mmm.......m com
IBM Research, Ireland
Dalin Zhang
dalin=@cs. dk
Department of Computer Science Aolborg University
Torben Bach Pedersen
tbp Dcs, aappeesaa..cooo aou. dk
Department of Computer Science
Aalborg University
Reviewed on OpenReview: https: // openrevie.. net/ forum? id= FHYC&TA0ęF
Abstract
Time-series forecasting is an important task in many domains, including finance, weather prediction, and energy consumption forecasting, and deep learning methods have emerged as the best-performing time-series forecasting methods over the last few years. However, most proposed time-series forecasting models are deterministic and are prone to errors when deployed in production, potentially causing significant losses and penalties when making predictions with low confidence. In this paper, we propose the Timo-Energy Model (TEM), framework that introduces so-called selective lime-series forecasting us- ing energy-based models (EBMs). Selective forecasting estimates model confidence and allows the end-user to selectively reject forecasts while maintaining dcsired target cov- erage. TEM is model-agnostic and can be used improve forecasting accuracy of any encoder-decoder deterministic time-series forecasting model. TEM is trained using combi- nation of supervised and supervipd learning, leveraging excellent single-point prediction accuracy while maintaining the ability to reject forecasts based on model confidence. Ex- perimental results indicate that TEM generalizes well across 5 stale-of-the-arl deterministic time-series forecasting models and benchmark time-series forecasting datasets. Using selective forecasting, TEM reduces prediction error by up to 49.1% over 5 state-of-the- art deterministic models. Furthermore, TEM has up to 87.0% lower error than selected baseline EBM models, and achieves significantly better performance than state-of-the-art selective deep learning models. Code for the proposed TEM framework is available at https://github com//onasBrusokas///oa. Energy-Model
1 Introduction
Time-series forecasting plays a pivotal role various domains, enabling informed decisions such as smart building control, adjusting the operation of heating systems, and buying and selling financial assets (Jin) et al. 2021 Affonso 2021). Recent advancements time-series forecasting using deep learning have significantly improved prediction accuracy and efficiency while addressing key limitations of previous models.

---

Published in Transactions on Machine Learning Research (2ebruay/20255)
such high computational costs and inability to capture global time-series patterns (Zhou et al. 2021 Wu
ett al.] 2021: Zhou et. al., 2022 Wu et al. 2023 Nie et al. 2023). Most current time-series iorecasting models are deterministic, producing a single prediction for an observed target process using historical data
as imput. Howover, single predictions are often insufficient for real-world applications, as they do not estimate model confidence that would enable decision makers to avoid using inaccurate predictions (Wen el al. 2017 Gneiting 2011).
Selective prediction, also known as prediction with reject option, addresses these limitations by allowing models to abstain from making predictions when model confidence is low. Enabling predictive models to reject potentially inaccurate predictions provides significant utility in domains where prediction errors carry significant costs [Lathe& & Saeys 2024 Hasan et al. 2023). Recent applications of selective prediction include healthcare diagnostics, autonomous driving systems, and financial markets (Zhang et al. 2023 Mohri et al, 2024 Cao et al.) 2024). Despite their benefits, selective prediction has not been explored in the context of time-scrics forccasting, and although there are similar neural network architoctures cnabling selective prediction for Lime-series classification, there are no known selective time-series forecasting methods (Nam)
et al. 2022 Zhang et al. 2023).
The only applicable selective prediction framework for time-scries forecasting is SelectiveNet, which was developed for classification and regression tasks (Geifman & El-Yaniv] 2019). SelectiveNet enables selective prediction based on user-defined target coverage, which describes the minimum number of predictions the model should perform. This framework provides a specialized loss function based on the interior point op- timization method and defines neural network architectures for both classification and regression. However, due to SelectiveNet's ' loss function, the models suffer from degraded prediction accuracy compared to deter- ministic models. SelectiveNet also requires training separate model for each user-defined target coverage and does not allow for rejecting predictions based on other criteria, such as estimated prediction error.
Energy-based models (EBMs) have been extensively studied and have recently seen a resurgence in the machine learning community. EBMs are unnormalized probabilistic models that provide a scalar measure called eneryy estimating the compatibility between a given input and output. EBMs provide an unnor- malized density over all configurations of input and output with lower energy being assigned to more likely configurations. EBMs make no prior assumptions about the output and are capable of capturing highly com- plex output distributions (Gustafsson el al. 2022). Recently, EBMs parameterized by deep neural norwsrks
have been successfully applied to various machine learning tasks Gustafsson et al.) 2020 Hendriks et al.
2021
Gustafsson et al.] 2021; Castillo-Navarro et al. 2022 Tu et al. 2020b a Li et al. 2021 Zhu et al, 2024 Singh et al.] 2024 Li et al. 2023). Time-series EBMs could therefore be used for selective time-series
forecasting, where the energy for given input and output could be used to estimate model confidence and selectively reject inaccurate predictions without having to retrain the entire model.
However, applying EBMs for time-series forecasting has several open challenges: series forecasting, an EBM architecture and training method must be capable of capturing and learning sequential dependencies and provide accurate predictions. Many recent time-scries forecasting papers pro- pose more accurale and efficient architectures for time-series forecasting (Zhou et al.) 2021 Wu et al.) 2021 Zhou et al. 2022 Wu et al. 2023 Nie et al. 2023). However, there are no currently known EBM architectures for timo-scrics. Generative mothods, such as EBMs, tend to quantitatively underperform against tailor-made discrininative methods in downstream tasks, such as classification, regression, and forecasting (Grathwohl key to providing utility in decision making, thus an FBM
(i) To be applied for time-
2021 Zheng et al., 2023). Accurate predictions for time-series forecasting must have comparable performance to deterministic forecasting models. (ii) Infer- ence on time-series using an EBM must be scalable for arbitrary time-scries forecasting horizons. Many EBM inference methods rely on deterministic sampling techniques, generating outputs autoregressively by explor- ing a sufficient subset of the output space to produce an accurate prediction. However, these techniques are generally computationally expensive for time-series as both the input and output spaces scale exponentially with series length, making the solution space too large to traverse in a reasonable time (Gustafsson et al, 2020 Tu et al. 2020b).
This paper addresses these challenges by proposing energy-based model framework for time-series fore- casting called the Time-Eneryy Model (TEM). This paper makes the following contributions: (1) Proposes
2

---

Published in Transactions on Machine Learning Research (February/202)
the Time- Time-Energy Model (TEM), energy-based model framework for time-series forecasting parameterized by deep neural networks addressing challenge ( (2) Proposes joint parameterization and training tech- niques of state-of-the-art plug-in deterministic models and energy-based based models for time-series forecasting, combining high single prodiction accuracy with estimating prediction error addressing challenge | (3) Proposes a scalable selective prediction procedure: selective Jorecusting with inference methods that use the energy-based model to estimate prediction error and enable rejecting predictions above a selected error bound, addressing challenges ii (4) Provides an experimental quantitative and qualitative evaluation of TEM on 5 benchmark time-series datasets, 1 baseline energy-based model for regression, 1 state-of- srate-of-the-art selective prediction deep learning model, and 5 state-of-the-art deterministic forecasting models. Using selec- tive forecasting, TEM reduces prediction error by up to 49.1% over 5 state-of-the-at deterministic models. The evaluation shows that. TEM has up to 87% lower prediction error than applicable EBM models for time-series and over 4244.3% lower error than baseline SelectiveNet selective prediction models. The exper- iments demonstrate that TEM generalizes improve the accuracy of all selected plug-in encoder-decodor deterministic time-series forecasting models over 5 datasets.
The remainder of this paper is structured as follows: Section 2 provides the problem definition and back- ground on encrgy-based models and selective forccasting, Section 3 presents the proposed TEM framework, including its architecture, training procedure, and selective forecasting methods. Section 4 describes the experimental setup, including baseline forecasting models, benchmark datasets, and evaluation metries. Sec- tion presents and analyzes the experimental results. Finally, Section 6 concludes the paper and discusses future work directions.
2 Problem Definition
TEM Architocture
TEM Training method
TEM Selective foroaasing
Energy--assen based model Eg
Pradicl Ae(Xi
State-ofthheart madels:
cetorministo model Av
Plug-in
Y
1. Train or ro-use traindd determinesic model Ap
Calcsdii Energy B)
TimesNat
PatchTST
Auloformer
2. Train EBM encuder
informer
B. Find Energy interval ExiX
prediction entor
Repe Hy
3. Train EBM deceder P.
Check if antisies colected tarpet
MLP
6xp
Empirical coverage
Select Reject prediction
Selected
130%
target coverage
Figure 1: Overview of the TEM framework. Starting from the left: TEM architecture, TEM training method, and TEM Selective Forecasting. Colors indicate which TEM components are trained at which step.
This section formally describes the preliminaries and provides a problem definition for the paper.
Deterministic time series forecasting. X a regular multivariate time series containing observed data needed for prediction with sequence length m. X is composed of vectors 2t, 24 Rd representing d observed features at time step 6. X, where X' = R 1xd X X, is the space containing all possible input time-series.
X = (
(1)
y= [
D44h-1
(2)
Y is a regular time series containing values for a single observed feature ahead time step Prediction horizon defines how many time steps ahead will be predicted. is composod of real numbers yt. V1 representing the observed feature. this paper Y, y y single best-guess prediction. D, where y R Y )" is the space containing all possible output time-series, such that mapping X y exists.
3

---

Published in Transactions on Machine Learning Research (obruary/2025)
Deterministic time-series forecasting models. A deterministic time-series forecasting model Ay is predictive model with parameters V that takes X as an argument and produces a single Y prediction as output. The prediction error is defined as the absolute difference between observed output Y and prediction Y.
Energy-based models. An energy- based model Eg is a predictive model with parameters that takes X and Y as arguments and produces energy as output. E R a measure of compatibility between given and Y, where lower energy means higher compatibility (LeCun et al., 2006).
Av : X )
Eg X x - lR
(3)
Eg can defined as an unnormalized probabilistic model, defining a conditional distribution (o for possible output Y, given X.
exp( Exp(-Eg(X,Y))
= arg min (x
(5)
(Y Jy exp(- Ea(X Z(0)
(4)
The normalization constant (() is intractable in a general case. However, calculating ( is not strictly necessary for energy-basod model training or inference. Unlike deterministic models, inference using an EBM is done by finding predicted output Y that minimizes energy w.r.t. Y E Y, given X.
Selective forecasting. Selective prediction for time-series (selective forecasting) can be defined as pair of deterministic forecasting model Ag and selection function 9. The deterministic forecasting model Ay provides predictions Y for given input X (as defined in Equation where the selection function is a decision function for selecting or rejecting the prediction Y.
Av(X)=8
gf
(Av,g)(X)
(6)
None, if g
Selective prediction performance can be quantified using selective coverage and selective risk. Selective coverage o(g) quantifies the proportion of selected predictions using selection function g. Coverage can also be viewed as the probability of prediction being selected using function g-
Selective risk R(Av,g, 9,l) defines prediction error for predictive model Av for selected predictions using selection function Prediction error is calculated with distance metric L, such as Mean Squared Error (MSE).
<(g) Eig())
(7) R(Av, g. E((((Av,)))) g)),,, 9(X)
(8)
x(g)
Problem definittion. Given lime series X, Y, find an energy-based model Eg with parameterization 0 jointly trained with a deterministic model Ag and selection function such that selective risk R R(Ar, g. () is mini- mizod while controlling selective coverage Φ(g).
3 TIME-ENERGY MODEL (TEM)
In this paper, we propose TEM, a deep-learning framework for time-series forecasting using energy-based models that enables selective prediction based on user-defined target coverage.
3.1 TEM overview
TEM a novel ensegy-based time-series forecasting framework that combines the accuracy and low latency of deterministic forecasting models with selective forecasting capabilitics using an energy-basod model. As shown in Figure 1 TEM consists of three key components: 1) TEM architecture 2) TEM joint training method 3) TEM selective forecasting.
4

---

Published in Transactions on Machine Learning Research (February20255
The TEM architecture consists of an encoder-decoder-basee plug-in deterministic forecasting model Ay that provides accurate low-latency forecasts Y and an energy-based model Eg that reuses the parameters of Av for estimating energy LXX,Y) Both Ay and Eg learn to capture sequential dependencies in time-series X.Y. The TEM framework allows "plugging in" any deterministic encoder-docoder forecasting model Ay (Section 3.2 TEM provides a training method that utilizes both supervised and self-supervised learning to jointly train Ag and Eg (Section 3.3 TEM introduces selective forecasting which uses the energy-basod model Eg achieve user-defined target coverage while minimizing selective risk (Section 3.4).
3.2 TEM architecture
fz
8xY MLP 8xY
MLP Py
concat fes fz)
Fully Connected Layer
Fully Connected Layer
Layers
Layers
ReLU
ReLU
Figure 2: Proposed architectures for TEM EBM encoder By and decoder Gry. Starting from the left: the MLP-based architecture for TEM decoder Ory, the MLP-based architecture for TEM encoder fy.
As shown in Figure 1 TEM consists of two interoperating deep neural network time-series models: the encoder-decoder plug-in deterministic forecasting model Ag and the energy-based model Eg.
Plug-in deterministic model Av- A deterministic parameterized by and is composed of an input encoder Va and an input decoder Vy.
seader-decoder-basoe plug-in forecasting model Ay is
A V (V)( +
(9)
Eg Oxy(
+ + (10)
As shown in Equation ! the input encoder T is trained to produce hidden input representation fz from an input X. fa is then used by the imput decoder 4, to produce an accurate forecast Ý. Most recent state-of- the-art transformer- based deterministic forecasting models use encoder-decoder architectures (Wen
et al. 2023 Zhou et al. 2021 Wu et al. 2021: Zhou et al. 2022. Nie al. 2023 Wu et al. 2023).
Energy-based model Eg. The energy-based model Eg is parameterized by 0. Eg re-uses the encoder V from the deterministic forecasting model Aş to calculate the hidden input representation fz. Eg consists of output encoder Oy and output decoder Ory, using the joint representation of fx and fy to calculate energy. As shown in Equation 10 the output encoder Gy produces a hidden output representation fy from an arbilrary given output. Y. The output decoder Ory then uses representations fz and fy to produce energy E = EX,
Like the deterministic model Ay, the EBM encoder and docoder fy, Ary are paramcterized using deep neural networks. As shown in Figure 2 we propose using multi-layer perceptron (MLP) architecture for Oy to caleulate the hidden output representation fy. MLPs provide very low computational latency, enabling fast energy calculation for different Y values and faster inference. For Ozy, propose architecture similar to by that concatenates the two representations fæ, fy and uses an MLP decoder to produce energy E.
3.3
TEM training
TEM uses a joint training method that combines supervised and self-supervised learning techniques to train both the deterministic forecasting model Av and the EBM Eg.
5

---

Published in Transactions on Machine Learning Research (2e2raay/2025))
Training method overview. We propose using traditional supervised learning to train the state-of-the-art forecasting model parameters v and then using sell-supervised learning to train the Eg parameters. As shown in Figure three components are trained in the following sequence:
1. Training Ay. The deterministic encoder-decoder forecasting model trained with supervised learning using the loss function and hyperparameterss as described known literature. After training, Av parameters y Vx are frozen. Alternatively, if Av trained apriori, we can directly reuse the model parameters v.
2. Training Eg parameters By and 8ry The EBM encoder 8y and decoder dzy trained using Contrastive Divergence self-supervised learning (Hinton 2002). Although the EBM Eg uses the encoder Vz from the deterministic model Av (as shown in Equation 10 the parameters of the deterministic model y remain frozen during this step.
This training method preserves the state-of-the-art forecasting accuracy deterministic forecasting models Ay while enabling energy calculation Eg 7xXX Y) for forecasts using the EBM Eg.
EBM training with Contrastive Divergence. Contrastive Divergence (CD) is parameter estimation method for learning EBMs (Hinton 2002 Song & Kingma. 2021).
CD learns the EBM parameters by contrasting "positive" output sample y from the training set for given X against a single "negative" sample y())
yo y()) - V Ea t
(1I)
As shown Equation 11 the negative sample (()) is obtained by refining a randomly generated point using Langevin dynamics (initialized from N((0,21)) with step size r and step count NCD-
LCD (E+ E (
(12)
As shown in Equation 12 the loss LCD is calculated as the difference between positive and negative sample energies (X, (o)) Ex(X, (()), with regularization term multiplied by coefficient aCD- A detailed description of the CD training method is provided in Appendix Algorithm
3.4 Selective forecasting with TEM
Calculate prediction error for each energy interval Esg
2. Rank Eøi intervals and calculate cumulative coverages
Select reject
3. intervals based on target coverage
Mean
Mean prediction error
Selected intorva
prediction
prediction error
Rejected intoval Ey
error
Energy intervals Eg
Cumulative empirical
Cumulative
empirical
coverage
coverage
Selected target coverage
Figure 3: Overview of selective forecasting with TEM: (1) Energy values are calculated for training set predictions and partitioned into intervals. (2) Mean forecast error and empirical coverage are calculated for each interval. (3) Intervals are ranked by error and selected to achieve desired target coverage.

---

Published in Transactions on Machine Learning Research (2ebruary20255)
TEM proposes a novel method to perform selective forecasting by using energy values to maintain user- defined target coverage while minimizing selective risk. It uses the EBM Eg to evaluate forecasts Y made by the deterministic model Ay and selectively reject forecasts.
In traditional (normalized) probabilistic forecasting, achieving user-defined target coverage can be done using parameters of the estimated output probability distribution, quantiles, or generated samples (Salinas et al. 2019; Wen et al.) 2017). However, an EBM Eg is an unnormalized probabilistic model and the computed energy E (x Y) can only be interpreted as relative compatibility score for given imput and forecast pair Y. Thus, energy scores can only be directly compared for the same input X, with exact score values varying between different X values. To circumvent this, other litcrature proposes deterministically sampling energy values around Y to evaluate forecasts (Gustafsson et al. 2020). For time-series, the output space grows exponentially with forecast horizon, making it infeasible to deterministically sample the output space in reasonable time.
TEM proposes to use both the energy of the single forecast Ex Y) and the energy values for y + = γ around the forecast Y to rank forecasts and achieve target coverage. If the initial forecast Y is accurate, the energy values around the forecast (X Ý + 6) should be low. Alternatively, if the initial forecast y is not accurate or the output distribution has high variance or is highly multimodal, the energy around the forecast should be higher. We propose two inference methods for achieving target coverage with TEM: 1) Aggregated energy inference - which ranks forecasts by adding noise to the forecasts Y to sample energy and 2) Energy optimization inference which ranks forecasts by minimizing the energy for forecast Y.
Aggregated eneryy inference. The Aggregated energy inference method directly samples the energy values around the prediction Y. These energy values are used to calculate aggregated energy which is then related to prediction error e. Aggregated energy Es Ex(X, 8) can be defined as:
Ex E. Eo( + +.) Ex(
(13)
72
where Ex Y) is the the mean of energy values Ex Ý - 5,) calculated on and around Ý by adding noise 8,. Samples &, are drawn from a noise distribution. In this paper, we will use the multivariate normal distribution (000 I²1), where is an identity matrix and covariance coefficient o² is selected according to the model and data.
Energy oplimization inference. We also propose an alternative inference method called Energy optimization inference. This inference method is based traditional EBM inference methods (as shown in Equation 5) and directly minimizes the energy Ex(X, P) on the prediction Y and then relates the energy to prediction error e. Energy optimization inference can be defined as:
B(X, = Ex(X,Y), where Y = arg min Ex(X, Y)
(14)
where É(X, Y) is the minimized energy Y) w.r.t. Y. Y made by the deterministic model Ag. We use gradient descent to minimize Ex(X, with step count T and step size n, Unlike the Aggregated energy inference method, the Energy optimization method relates a single energy value E) to prediction error c. Notably, TEM weights are not updated while minimizing Ex(X.
The initial value for Y is the single prediction
Using eneryy for selective forecasting. As shown in Figure g we propose to calibrate TEM using energy to estimate model confidence and select forecasts with the lowest error e, while still achieving the desired user-defined target coverage o(g). To achieve this, we partition the energy range into finite mumber R of disjoint energy intervals Eps and calculate the mean forccast error and empirical coverage ((g) of all training X, Y samples for which Ex, E Eai, as shown in step 1 in Figure 3 As shown in step 2 in Figure g we rank all energy intervals in ascending order of forecast error e and calculate the cumulative empirical coverage for each interval starting from the interval with the lowest forccast error. As shown step 3 in Figure 3, we then calculate which energy intervals should be selected or rejected for a desired target coverage ((g).
7

---

Published in Transactions on Machine Learning Research (2ebruay/2025))
After TEM calibration, the model forecasts can be selected or rejected by calculating the energy value E Y) of the prediction y and checking which energy interval the energy belongs to Ex Y) E Fos- Notably, the target coverage o(g) can be dynamically selected based on the utility of the prediction and the requirements of the application without the need to retrain or recalibrate the model Eg. Furthermore, this approach enables forecasts to be selected or rejected based on other criteria, such as prediction error 6.
4 Experimental Setup
4.1 Baseline models
Baseline Emergy-based models. In this paper, we use EB-NARX (Hendriks et al. 2021 a baseline energy- based model to evaluate the performance of the TEM framework. EB-NARX is an energy-based model parameterized by deep neural networks that was initially developed for time-series regression. While EB- NARX uses a combination of solution space sampling and energy minimization w.r.t. Y perform inference, this method not scalable for multi-stop time-scrics forccasting as the output space grows exponentially with forecast horizon. We perform multi-step forecasting with EB-NARX by making predictions one time-step time (from V/+1 to Yt+h), propagating each single best-guess prediction until reaching the desired forecast horizon.
Stale-of-lhe-art delerminislic forecasling models. In this paper, we use 5 stale-of-the-art transformer-based deterministic time-series forecasting models to evaluate the performance of the TEM framework.
Informer (Zhou et al. 2021 was one of the first Transformer models for deterministic timo-serics forocasting, Informer uses direct multi-step inference avoiding error accumulation in the auloregressive forecasting setting. It also was one of the first such models to utilize learnable positional encodings for input sequence and max- pooling to down-sample intermediate hidden representations (Wen et al. 2023
Autoformer (Wu et al. 2021 built. on Informer by introducing seasonal trend decomposition and a novel autocorrelation block instead of a traditional attention module reducing inference complexity while providing
higher prediction accuracy (Wen et al. 2023
FEDformer (Zhou et al. 2022) further built on Informer and Autoformer by introducing Fourier and Wavelet transformations in addition to scasonal decomposition. It achieves a higher prediction accuracy with signif-
icantly lower inference and memory complexity (Wen et al. 2023
TimesNet [wu et al. 2023) proposes a novel approach that treats time series forecasting as an aanageto-imagee translation problem. It introduces a lcarnable timo-froquency transformation to capture both temporal and frequency patterns in time series data. TimesNet also utilizes a series of inception blocks with different kernel sizes to capture multi-scale temporal dependencies, allowing to adapt to various seasonal patterns and trends in time series data.
PStchTST Nie et al. 2023) is patch-based time series transformer that addresses the limitations of previous transformer models in capturing long-range dependencies. PStchTST divides the input time series into non- overlapping patches and applies self-attention mechanisms to efficiently process longer input sequences and capture both local and global temporal patterns. These techniques result in significantly improved forecasting performance for long-term predictions.
SelectiveNet-like time-series forecasting models. SelectiveNNt (Geifman & EI- Yaniv 2019) is deep learning framework proposed for enabling selective prediction with neural networks. It uses a specialized selective loss function and a neural network architecture that produces three outputs: selection, prediction, and auxiliary prediction. These outputs are used to train the model and to perform selective prediction reject a proportion of predictions to achieve the desired larget coverage Φ(g). Each SelectiveNet model is trained for a specific target coverage d(g), that is set before training the model. To the best our knowledge there is no SclcetiveNet implementation for selective timo-scrics forocasting. As such, we adapt the framework use selective loss based on MSE (which is used as a loss function by stato-of-tho-as deterministic forecasting models described in 4.1 SelectiveNet was selected as baseline it is the only end-to-end deep learning
8

---

Published in Transactions on Machine Learning Research (February/0225)
framework enabling coverage-basse based selective prediction that could be applicable for selective time-series forecasting.
4.2 Datasets
To evaluate TEM performance, we use five open benchmark time-series datasets. The Electricity Trans- former Temperature datascts: ETTh1, ETTh2 (Zhou et al.) 2021) contain 2 years of hourly temperature measurements from two electricity transformers in separate Chinese counties, each with 7 sensor features. The Exchange Rate dataset contains the daily exchange rates between 8 different currencies against USD from 1990 to 2016, with XRP/USD as the target variable for forecasting. The Weather datasot contains years of daily weather mcasurements from able being the temperature readings from specific station. The National Illness dataset, contains weekly inftuenza-like illness ratios reported by the US Centers for Disease Control (CDC), containing data from 2002 to 2021 across multiplc US regions. These datascts were selected as they are commonly used to benchmark time-series forecasting models and we re-use the data preprocessing and splitting procedures, as found in re- cent state-of-the-art deterministic forecasting model literature (Zhou et al. 2021 Wu et al. 2021 Zhou et al.
monitoring stations across Canada, with the target vari-
2022
Wu et al. 2023
Nie et al. 2023). Additional statistics for the datasets are provided in Appendix 6
4.3
Metrics
We use the Mean Square Error (MSE) metric to evaluate prediction error for all forecasting models. To evaluate selective forecasting performance for both TEM and SelectiveNel, we use selective coverage ((g) and selective risk R(Av,ge g.l) with MSE as the distance metric (as defined in Equations
4.4 Implementation details
For the deterministic forecasting models Informer, Autoformer, FEDformer, PatchTST, and TimesNet Ag we re-use known hyperparameterss from their respective experiments (Zhou al.) 2021 Wu et al. 2021 Zhou et al., 2022 Wu et al. 2023; Nie et al.) 2023). For the MLP-based encoder and decoder fy, Gry parameterizations, we use 4 layers for each with 128 hidden units in each of the fully connected layers.
For TEM selective forecasting using Aggregated energy inference, as described in Section 3.4 we select the covariance coefficient o² for the multivariate normal distribution (0, o²1) from which we will draw noise samples 5,. We select one a² E (0.0, 0.02, 0.05, 0.1, 0.2, 0.3, 0.5} for each trained TEM model. For each prediction Y, we draw 32 samples 5; generate aggregated energy Ex(X, Y). For TEM selective forccasting using Energy optimization inference, we perform gradient descent using the Adam optimizer using step sizes 7) {0.1, 0.01, 0.001} and step counts T {5, 10, 25}. We provide additional implementation details in Appendix Scction A.6 Changing TEM selective forecasting parameters docs not require changing retraining any components of TEM.
5 Results
5.1
Quantitative TEM Selective forecasting performance
We evaluate TEM models in terms selective risk and coverage performance with target coverage ((g) against plug-in deterministic models Av. We quantitatively evaluate TEM using all combinations of: 5 datasets (Section 4.2g 5 plug-in deterministic forecasting models Av (Section 4.1 and TEM inference
methods (Section 3.4
For each configuration, we conducted 3 experiments with different random number seeds to reduce the likelihood of non-representative results. Selected target coverages p(g) E (10%, 20%,50% 70%, 90%} were chosen to include those used in the SelectiveNet paper [Geifman & EI- Yaniv 2019) as well as 10% and 30%.
Overall TEM performance. As shown in Table [ TEM roduces prediction error across all configurations of five deterministic models Ag and five benchmark datasets. Tables highlight the best performing models for each target coverage, by choosing the model with the lowest prediction error for each target coverage, if the
9

---

Published in Transactions on Machine Learning Research (February/202)
Table 1: TEM performance comparison across different models and datasets. Results show selective risk and empirical coverage (in parentheses) for target coverages (gg E 10%, 30%, 50%, 70%, 90%. Best performing models for specific target coverages are marked bold.
Base Model
Coverage
Method
Dataset
ETTh1
ETTh2
Weather
Exchange Rate
National Iiluess
Original
0.0876
0.1577
0.0079
0.0899
1.1758
TEM
0.0906 (16.80)
0.1591
(28.39)
0.0076 (57.13)
0.0712 (17.60)
0.5634 (2.78)
10%
SclectiveNet
0.7862 (55.92)
0.7103 (40.13)
0,2903 (45.41)
1.1609 (60.56)
3.3422 (51.91)
TEM
0.0889 (51.79)
0.1591 (28.39)
0.0076 (57.13)
0.0708 (26.73)
1.0141 (20.14)
30%
SelectivoNet
0.6865 (65.46)
0.9926 (64. 14)
0.3479 (64.13)
1.4727 (52.55)
3.6843 (56.43)
Antoforner
TEM 0.0889 (S1.79)
0.1568 (59.22)
0.0076 (57.13)
0.0902 (61.93)
1.0128 (42.13)
50%
SelectiveNet
(54.08)
1.0288 (55.82)
0.3276 (51.06)
1.7034 (38.36)
4.1228 (57.36)
70%
TEM
0.0871 (76.64)
0.1568
(59.22)
0.0075 (73.91)
0,0897 (69.35)
1.0484 (65.74)
SelectiveNet
0.9668 (79.58)
1,5178 (81.07)
0.6144 (72.30)
1,3532 (76,79)
5.1191 (75.45)
TEM
0.0881 (87.45)
0.1568
(76.78)
0.0075 (73.91) )
0.0890 (87.96)
1.0974 (83.10)
90%
SelectiveNet
1.5981 (91.95)
1.5979
(84.55)
0.6439 (96.85)
3.1883 (93.54)
5.6797 (84.31)
Original
0,0772
0.1184
0.011
0,0653
1,0503
10%
TEM
0.0782 (45.86)
0.0958
(17.75)
0.0104 (61.27)
0.0354 (3.19)
0.6754 (1.40)
SelectiveNet
0.5129 (25.05)
0.6745 (19.00)
0.3382 (49.98)
0,9054 (48.13) 13)
4.1730 (97.74)
TEM
0,0782 (45.86)
0.1192 (75.68)
0.0104 (61.27)
0,0616 (34.74)
0.8927 (13.66)
30%
SelectiveNet
1.1763 (63,
3,0803 (86.68)
0.3297 (19.42)
0.3131 (19.02)
3,4399 (80.37)
FEDiormer
50%
TEM
0.0770 (94.77)
0.1192 (75.68)
0.0104 (61.27)
0.0616 (34.74)
0.8470 (26.39)
SelectiveNet
0.5833 (30.61)
0,6300 (18:90)
0.1021 (59.81)
1.3906 (76.61)
1,9532 (41.76)
TEM
0.0770 (94.77)
0.1192 (75.68)
0,0105 (67.49)
0.0634 (72.38)
0.8748 (43.75)
70%
SelectiveNet
1.1240 (70.69)
3,3039 (91.68)
0.5702 (85.10)
1.4646 (75.63)
2.5128 (61.66)
TEM
0.0770 (91.77)
0.1190 (85.77)
0.0105 (67.49)
0.0655 (91.33)
0.9229 (62.73)
90%
SelectivoNet
1. 1.8596 (97.29)
3.7828 (96. 10)
0.6025 (98.34)
1.6270 (93.52)
3.8254 (96.50)
Original
0.6461
118777
0.3313
0.73
4.6609
TEM
0.6008 (37.08)
1,2057 (13:05)
0.0046 (54.75)
0.1828 (7.81)
4.0733 (9.00)
10%
SelectiveNet
0.7258 (43.89)
0.4411
(13.14)
0,2537 (38.93)
0.7313 (38. 84)
2.4625 (51.77)
30%
TEM
0.6008 (37.08)
1.1751 (27.66)
0.0016 (54.755
0.2000 (8.11)
1.1375 (17.92)
SelectiveNet
0.5989 (37.10)
0.4349 (13.82)
0.1355 (21.43)
1.6538 (91.89)
1.8485 (45.88)
Informer
TEM
0.6460 (89.32)
1.1845 (49.45)
0.0046 (54.75)
0.1921 (10.47)
4.3605 (66.20)
50%
SelectiveNet
0.8827 (61.93)
0.1902 (14.88)
0,3819 (56.73)
0.0723 (3.91)
2,1334 (39.48)
TEM
0.6460 (89.32)
1.1814 (70.75)
0.0046 (59.67)
0,28188 (12.98)
4.5612 (80.32)
70%
SelectiveNet
1,5353 (93.71)
2.3146 (81.68)
0.6279 (95.77)
0.0774 (3.85)
3.3572 (82.48)
TEM
0.6438 (93.87)
1,1850 (85.72)
0.0046 (59.67)
0.4741 (34.02)
4.5576 (90.74)
90%
SclectiveNet
1.5991 (93.58)
2.7387 (80.36)
0.6709 (99.76)
2.5347 (97.99)
4.7834 (96.82)
Original
0.0416
0.1079
0:0011
0.0617
0.7324
TEM
0.0340 (15.32)
0.0735
(11.98)
0.0008 (21.02)
0.0488 (6.33)
0.4686 (6.25)
10%
SelectiveNet
0.7245 (46.66)
1.1741 (43.79)
0.2904 (44.02)
1,3008 (43.17)
3.0678 (41.82)
TEM
0.0413 (30.05)
0.1077 (62.31)
0.0008 (21.02)
0.0526 (7.60)
0.4209 (12.96)
30%
SelectiveNet
0.7245 (16.66)
1.1741 (13.79)
0,2901 (14.02)
1.3008 (13.17)
3.0678 (11.82)
PatelTST
TEM
0.0418 (55.94)
0.1077 (62.31)
0.0008 (26.25)
0.0500 (15.60)
0.4210 (15.51)
50%
SelectiveNet
0.9246 (57.67)
1.5540 (58.10)
0.6266 197.14)
2,0810 (67.66)
4.2466 (57.52)
TEM
0.0417 (60.38)
0.1076 (75.24)
0.0008 (33.02)
0,0505 (25.61)
0.5040 (32.18)
70%
SelectiveNet
1.3013 (73.65)
2.1454 (75.39)
0.6088 (93.70)
2.1041 (68.53)
5.5219 (78.86)
90%
TEM
0.0417 (88.96)
0.1079 (93.97)
0.0008 (5.1.18)
0.0558 (16.40)
0.6313 (66.90)
SelectiveNet
1 1.5963 (90.38)
2. 2.6392 (94.30)
0.6394 (99.95)
2.6975 (89.78)
6.9872 (87.01)
Original
0.0438
0.1273
0.0016
0.0540
0.8391
10%
TEM
0.0308 (16.80)
0.1069 (29.04)
0.0014 (61.17)
0.0416 (6.03)
0.5222 (5.79)
SelectiveNet
0.9161 (53.09)
1.2297 (45.05)
0.3200 (19.72)
2,0732 (S5.11) 11)
3,2588 (62.76)
TEM
0.0447 (34.79)
0.1069 (29.04)
0.0014 (61.17)
0.0434 (16.12)
0.4281 (9.03)
30%
SelectiveNet
1.0185 (57.71)
1,2813 (47.15)
0.3517 (51.15)
199370 (52.73)
3,2699 (52.94)
Net
TEM
0.0455 (71.70)
0.1281 (60.91)
0.0014 (61.17)
0.0151 (32.45)
0.4413 (13.66)
50%
SelectiveNet
0.8006 (64.48)
0.5844 (25.75)
0.3500 (54.01)
1.8190 (49.63)
3,5008 (57.96)
70%
TEM
0.0455 (71.70)
0.1281 (82.69)
0.0014 (65.83)
0.0485 (46.81)
0.4270 (16.90)
SolectiveNet
1.2080 160
1.7400 (79.76)
0.5239 (78.78)
3.4055 (85.40)
4.7244 (78.43)
TEM
0.0459 (84.58)
0.1276 (94.16)
0.0014 (65.83)
0.0502 (64.89)
0,6104 (37.96)
90%
SelectiveNet
1.5088 (90.83)
2.8587 (92.41)
0.6379 (97.73)
3.3017 (90.90)
5.8539 (93.66)
10

---

Published in Transactions on Machine Learning Research (February/0055)
model achieves target coverage. If no model achieves target coverage, then calculate the percentage error reduction and difference between target coverage and actual coverage. The model with the lowest sum of prediction error and coverage difference is considered the best performing model for that target coverage, datasct, model configuration.
TEM reduces prediction error across all five models by between 11.1 39.0% on average for larget coverages ((g) < 50%. The highest prediction error reduction was achieved using target coverages ((g) E {10%, 30%), where TEM achieves average 21.0% reduction in prediction error across all models and datasets using the Aggregated energy inference method.
The largest error reduction was achieved with the Informer model. For target coverages Φ(g) E {10%, 30%), TEM reduces Informer's prediction error by over 34.1%. TEM selective forecasting also significantly reduces error for the best performing baseline deterministic models, achieving up to 45.5% reduction for PatchTST and 49.1% for TimosNet for target coverages ((g) E (10%, 30%].
The average actual coverage ((g) recorded for target coverages x(g) E {10%, 30%] is 21.4% and 32.7% respectively across all models and datasets. This result is expected, Informer has the lowest deterministic prediction accuracy among all 5 tested deterministic bascline models and has the most room for error reduction.
TEM does not always achieve the target coverage, with actual coverage being on average 11.0% and 17.4% lower than the target coverage for target coverages ((g) E (70%, 90%} respectively. Most of this difference comes from the PST and TimesNet models, which have the lowest deterministic prediction error among all tested models. We have also noted that the Energy optimization inference method achieves average 3.5% lower actual coverage than the Aggregated energy inference method across all tested target coverages, but provides up to 30.4% lower prediction error (sce Appendix A.1.1 for more detailed comparison).
5.2 Comparison against selected EBM baselines
Configurations of TEM were compared against bascline EBM models. As seen in Table 1 TEM compares favorably against the EBM model baselines. The baseline EB-NARX model has the overall second highest delerministic prediction error of all the lesled models and TEM configuralions. EB-NARX only outperforms the Informer model on all datasets, having on average 51.6% lower error. EB-NARX on average has a 407. higher error than the state-of- tho-art deterministic TimesNet and PatchTST models across all datascts How- ever, EB-NARX has the lowest deterministic prediction error for the Weather dataset, outperforming both of the state-of-the-art TimesNet and PatchTST models by 50.0% and 27.3 27.3% respectively. TEM outper- forms EB-NARX all datasets using selective forecasting, having up to 87.0% lower error for lower target coverages <(g)) E (10%,30%}
The observed EB-NARX performance is expected and can be attributed to the fact that EB-NARX is originally a regression model which tends to suffer from error accumulation when predicting for longer forecast horizons. Furthermore, EB-NARX generates over 2000 samples and performs energy minimization with gradient descent. for each time-step i, causing inference to be over times slower than TFM.
5.3 Comparison against SelectiveNet baselines
Configurations of TEM were compared against baseline SelectiveNet models, which are based on the only other end-to-end deep learning framework that enables selectively rejecting predictions based user-defindd target coverage. We adapted SelectiveNet for the time-series forecasting task described in Section [.1 Three architectures of the adapted SelectiveNet were trained for each of the statc-of-the-art forccasting models. 6 coverages E (10% 30%, 50%, 70%, 90%, 100%) were selected to train SelectiveNet models, which contain the coverages used in the original SelectiveNet paper (Geifman 30% evaluate performance for lower target coverages. As with prior experiments, SelectiveNet models were trained per architecture, initialized with random seeds, to reduce the likelihood for non-representative results.
EI- Yaniv 2019 and 10% and
11

---

Published in Transactions on Machine Learning Research (February/20) 2025)
Autoformer
FEDformer
Informer
PatchTST
TimesNet
Actual Coverage (86) Eneer Reduction (50
100
-106
-102
10 20
30
40 50 60
70 80
90
10
20
30
40
50
60
70 80
90
10
20
30
40
50
60
70 80 00
10
20
30
40
50
60
70 80 90
10
20
30
40
50
60
70 80 90
100
000
30
80
30
70
70
70
70
50
50
80 Target Coverage (%)
30
20
30
20 50
Target Coverage (5))
Target Coverage (%)
Target Coverage (%)
Target Coverage (56)
TEM using Aggregated energy inference
TEM using Energy optimization inference
Selectivehee models
Figure 4: Prediction error reduction (figures the top), target and actual coverage percentages (figures at the bottom) for TEM and SelectiveNet, across selected target coverages {10%, 30%, 50%, 70%, 90%} models Autoformer, FF.former, Informer, PatchTST, and TimesNet. The top figures' Y-axes use log scale to visualize the several orders of magnitude difference in performance between selective forecasting with TEM and SelectiveNet. For bottom figures, the dotted line represents the ideal case, where actual coverage is equal to target coverage.
As seen in Table [ SelectiveNet failed to consistently decrease prediction error, across all target coverages, for all tested models across all datasets. TEM using the Aggregated energy inference method outperformed SelectiveNet in every case in terms of selective risk, where TEM reduces crror by on average 16.5% across models and dalasets and SelectiveNet increases error by 4244.3% However, SelectiveNet did achieve target coverage more consistently and had on average 14.0% higher coverage than TEM across all tested target coverages, which can be explained by the way SelectiveNet models trained (sce more details in Appendix A.2.2
Furthermore, SelectiveNet did manage to reduce prediction error for the Informer models, achieving on average 18.4% lower error for target coverages Φbg) E (10%, 30%) and 30.2% lower егror for target coverages ((g) E [50%]. Notably, unlike TEM, Selective actual coverage does not rise monotonically as target coverage increases, but instead has a saw-tooth pattern, where actual coverage for lower larget coverages higher than for higher larget coverages. For example, SelectiveNet with Informer achieves on average 42.0% actual coverage for target coverages Φ(g) E (30%]. but only 35.4% actual coverage for target coverage ((g) E [50%). This is expected, as SelectiveNet models are trained for each target coverage independently, meaning that the coverage achieved for target coverage is not indicative of the coverage that would be achieved for different chosen target coverage. This makes SelectiveNet less predictable than TEM in terms of how changing target coverage will affect actual selective coverage and selective risk.
5.4 Ablation study and additional experiments
To evaluate the impact that different components of TEM have on selective forecasting performance, we conducted an ablation study and additional experiments. We conducted an ablation study on the two proposed TEM inference methods, Aggregated energy and Energy optimization, comparing them to each other well as to a naive baseline where energy Eg (X,Y) was directly used to estimate uncertainty (see more details in Appendix A.1.Ip. Results show that the Aggregated energy inference method achieves average 10.9% lower selective risk than the Energy optimization inference method, but has 3.5% lower coverage. However, both methods outperform the naive baseline mot by 461.9% and 462.2% respectively, showing that both proposed inference methods are superior to the naive baseline. We also conducted an ablation study on the proposed joint training mothod, comparing it to a configuration where only self-supervised Contrastive Divergence learning was used to train TEM models (see more details in Appendix A.1.2 Results show that TEM without joint training increases forecasting error for selective forecasting by up to 2798.6% across
12

---

Published in Transactions on Machine Learning Research (2erruay/20255)
all coverages, making it unsuitable for practical use. Furthermore, conducted additional experiments to evaluate TEM performance for univariate time-series forecasting (see more details in Appendix A TEM manages to reduce forecast eтror to a similar degree as for the multivariate forecasting case suggesting that the effectiveness of TEM is not significantly impacted by the dimensionality of the forecasting task and demonstratting that TEM can be used for selective univariate Lime-series lorecasting. Finally, we conducted additional experiments to further analyze the performance of SelcctiveNet and identify potential reasons for its poor performance compared to TEM (see more details in Appendix A.222).. Experiments show that SelectiveNet models tend to converge to stable coverage level, close to the target coverage. Furthermore, SelectiveNet. models are overly conservative during training, selecting too many forecasts and achieving higher coverage than desired, resulting in higher prediction error.
6 Conclusions and Future Work
This paper proposes the Time-Energy Model, an energy-based model framework for time-scrics forecast- ing. TEM addresses challenges of applying energy-based models to time-series forecasting by providing a framework to parameterize, train, and perform inference with EBMs on time-series data using deep neural networks. TEM introduces selective forecasting that enables the EBM to estimate model confidence, allow- ing the end-user to selectively reject predictions based on potential forecast error. TEM is parameterized and trained using a proposed joint training method that improves existing baseline EBM models by having a lower inference latency and siguificantly higher prediction accuracy. As shown in the experiments on state- of-thc-art forocasting models TEM can improve the prediction accuracy of encoder-docoder deterministic time-series forecasting models. Experiments show that TEM increases the prediction accuracy over known state-of-the-art forecasting models by up 49.1% 5 benchmark datasets. Also, TEM has 87.0% lower error than baseline EBM models and 4244.3% lower error than SelectiveNet models, while also providing significantly faster inference than the former.
In future work, we will extend the TEM framework and develop improved architectures and inference methods that provide better error reduction while maintaining higher coverage. Furthermore, we will apply TEM for time-series outlier and anomaly detection. Finally, we will explore the use of TEM with time-series foundation models.
Acknowledgements
This research was partially funded by the 6G-XCEL project under the Horizon Europe programme and the H2020 FEVER project (Grant Agrocment No. 864537) within the Europcan Union's research and innovation initiatives.
References
Felipe Affonso, Thiago Magela Rodrigues Dias, and Adilson Luiz Pinto. Financial Times Series Forecasting of Clustered Stocks. Mobile Networks and Applications, 26(1): ):256) 265, February 2021. ISSN 1572-8153. doi: 10.10077/11036-00000011777
https://doi.org
00007//110336000-0166//8
Sam Bond-Taylor, Adam Leach, Yang Long, and Chris G. Willcocks. Deep generative modelling: compar- alive review of VAEs, GANs, normalizing flows, energy-based and autoregressive models. IEEE Transac- tions on Pattern Analysis and Machine Irtelligence, 44(11):7377-7347, 2022. ISSN 0162-8828, 2160-9292, 1939-3539. doi:
Zheng Cao, Raymond Guo, Wenyu Du, Jiayi Gao, and Kirill V. Golubnichiy. Optimizing stock option forecasting with the assembly of machine learning models and improved trading strategics. In Kohei Arai (ed.), Advances in Information and Communication, pp. 610 620, Cham, 2024. Springer Nature Switzerland. ISBN 978-3-081-53363-3.
Javiera Castillo-Navaroo, Bertrand Le Saux, Alexandre Boulch, and Sébastien Lefevre. Energy-based models in earth observation: From generation to semisupervised learning. IEEE Transactions on Geoscience and Remote Sensing, 60:1-11, 2022. doi: 10. 20.1009/TRSS.021..33
13

---

Published in Transactions on Machine Learning Research (Februar/20255)
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In H. Wallach, H. Larochelle, A. Beygekzimer F. d' Alché-Bue, F. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL htttps:/pproceedinggg cwwpppperccppppr.... http:/aprrffee/////eeeeeaa http:://001iii11aaaaaaatttttooo.
Paper Paper.pdt. pdf
Yonatan Geifiaa and Ran Yaniv. Selectivenet: A deep neural network with an integrated reject option. In International Conference on Machine Learning, 2019.
Tilmann Gneiting. Quantiles as optimal point forecasts. International Journal of Forecasting, 27(2):197- 207, 2011. ISSN 0169-2070. doi:
URL https: //wpp:://www...
Will Sussman Grathwohl. Applications and Methods for Energy-based Models at Scale. Phd thesis, University of Toronto, November 2021. URL htttps://tspace. 1ibrary...b.rr.. www.ontoooooooooooo httpaa/...ttrrrrooo.aaaaaa////// aa/haannde/18071101155
Fredrik K. Gustafsson, Martin Danelljan, Goutam Bhat, and Thomas B. Schön. Energy-based models for deep probabilistic regression. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020, pp. 325 343, Cham, 2020. Springer International Publishing. ISBN 078-3-030-58566--
Fredrik K. Gustafsson, Martin Danelljan, and Thomas B. Schön. Accurate 3d object detection using energy- based models. In 2021 IEEE/CVE Conference on Computer Vision and Pattern Recogition Workshops (CVPRW 2021)= :, IEEE Computer Sociely Conference on Computer Vision and Pattern Recognition Workshops, pp. 2849 2858. Institute of Electrical and Electronics Engineers (TEEE), 2021. ISBN 978-1- 6654-4899-4. doi: 10.1109 10.1109/CVPPI /CWPR/V300..202222222220 20w.:553388222...000 200.000.0220.0000000 00320.
Fredrik K. Gustafsson, Martin Danelljan, and Thomas B. Schön. Learning Proposals for Practical Energy- Based Regression. In International Conference on Artificial Intelligence and Statistics, Vol 151 ;, volume 151 of Proceedings of Machine Learning Research, pp. 4685-4704. JMLR-JOURNAL MACHINE LEARN- ING RESEARCH, 2022.
M. Hasan, Moloud Abdar, Abbas Khosravi, Uwe Aickelin, Pietro LiA?, Tbrahim Hossain, Ashikur Rah- man, and Saeid Nahavandi. Survey on leveraging uncertainty estimation towards trustworthy deep neu- ral networks: The case of reject option and post-training procossing. arXiv preprint Stl:2304.04900 (bss/230.004900 2023. doi: 00.48550/arXiv.23044
Johannes N. Hendriks, Fredrik K. Gustafsson, Antônio H. Ribeiro, Adrian G. Wills, and Thomas B. Schön. Deep energy-based NARX models. IFAC-PapersOnlánee, 54(7):505 510, 2021. ISSN 2405-8963. doi:
10.1016/)..facoo..222 (.00:)..faco1.2221.88..1000 400.000..
Geoffrey E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Computation, 14($):1711-1800, 2002. doi: 00.116200899660027000
Yuan Jin, Da Yan, Xuyuan Kang, Adrian Chong, Hongsan Sun, and Sicheng Zhan. Forecasting building oc- cupancy: A temporal-sequentill analysis and machine learning integrated approach. Energy and Buildings, 252:111362, December 2021. ISSN 0378-7788.
doi: (0001//)enbuild.2211...1
Umi Lathe and Yvan Sacys. Uncertainty-aware single-cell annotation with a hierarchical reject option. Bioinformatics, 40(3))btac128, 2024. doi: 1..100//bbonformmaici /btacc11.
Yann LeCun, Sumit Chopra, Raia Hadsell, Marc' Aurelio Ranzato, and Fu Jie Huang. A tutorial on energy- based learning. Technical report, Courant Institute of Mathematical Sciences, New York University, 2006. pdf Accessed on November 20, 2024.
URL hittps://wwe ca. wwtpontt.edd//mmmaii/iirrrr////e
Shuang Li, Yilun Du, Gido Martijn van de Ven, Antonio Torralba, and Igor Mordatch. Energy-based models for continual learning. In International Conference on Learning Representations, 2021.
Zengyi Li, Yubei Chen, and Friedrich T. Sommer. Learning energy-based models in high-dimensional spaces with multiscale denoising matching, Entropy, 250) 2023. ISSN 1099-4300. doi: 10.3390/ (25001225113.7. URL https::///// http://wwaaalaccm///000092/////0
14

---

Published in Transactions on Machine Learning Research (2ebruar/2025))
Christopher Mohri, Daniel Andor, Eunsol Choi, Michael Collins, Anqi Mao, and Yutao Zhong. Learning to reject with fixed predictor: Application to decontextualization. In International Conference on Learning Representations, 2024.
Borun Nam, Joo Young Kim, In Young Kim, and Baek Hwan Cho. Selective Prediction With Long Short- term Memory Using Unit-Wise Batch Standardization for Time Scries Health Data Sets: Algorithm Development and Validation. JMIR medical informalics, 10(3)e30587, March 2022. ISSN 2291-9694. doi:
10.2196/30587.
Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformors, 2023. URL
Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autorogressive denoising diffusion models for multivariate probabilistic time series forecasting. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8857- 8868. PMLR, 2021.
David Salinas, Valentin Flunkert, and Jan Gasthaus. DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks. arXiv:1704.0411 fcs, stat], February 2019. URL http://arxiv. 04110 arviv: 1704.04110.
http:://axxi.org//bbb////// www./ogg//////oooo.oo abs//144/
Vaibhav Singh, Anna Choromanska, Shuang Li, and Yilun Du Wake-sleep energy based models for continual learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognation (CVPR) Workshops, pp. 1118-4127, June 2024.
Yang Song and Diederik P. Kingma. How to Train Your Bhergy-BBasee Models. arXtc:2101.08288 fcs, stat), February 2021. http://and http://arxiv.orgg wwwg/bbs//////// /t///22110002222 032288.
Lifu Tu, Richard Yuanzhe Pang, and Kevin Gimpel. Improving joint training of inference networks and structured prediction energy networks. In Proceedings of the Fourth Workshop on Structured Prediction for NLP, pp. 13.73. Association for Computational Linguistics, 2020a. doi: 10.18653/ /2000//s112000.plppppppp
Lifu Tu, Richard Yuanzhe Pang, Sam Wiseman, and Kevin Gimpel. ENGINE: Energy-based inference networks for non-autoregressive machine translation. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (ods.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguislies, pp. 2819 2826, Online, July 2020b. Association for Computational Linguistics. doi: 10.18653/ 2020 acl acl-main.... .51
v1/2020.acl-mmn aww.p/a..lmmmi...m 251. URL https://aclantthoogg
Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Trans- formers in time series: A survey. In Proceedings of the 32nd International Joint Conference on Artificial Intelligence, pp. 6778 6786, 2023.
Ruofeng Wen, kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. A multi-horizon quantile recurrent forecaster. In NIPS Time Series Workshop, 2017.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long, Autoformer: Decomposition Transformers with Auto-Correlation for Long- Term Series Forecasting. In Advances in Neural Information Processing Systems, volume 34, pp. 22419-2241 22430. Curran Associates, Inc., 2021.
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In International Conference on Learning Represen- tations, 2023. URL ptttps://arxss org/abg/
02286
Xu-Yao Zhang, Guo-Sen Xie, Xiuli Li, Tao Mei, and Cheng-Lin Liu. A survey on learning to reject. Proceed- ings of the IEEE, (11(2):185-2144 February 2023. ISSN 1558-2256. doi: 10.109//PROC..203333333333 3238004..
Chenyu Zheng, Yufeng Liu, and Quanquan Gu. Revisiting discriminative vs. generative classifiers: Theory and implications. arXio preprint or Xiw:2300.02334, 2023.
15

---

Published in Transactions on Machine Learning Research (February/0055)
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. In- former: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 35, pp. 11106 11115, 2021. doi: 10.1609 10.1609/aaai x33.2.3311...3355.
URL https://aaaa //rs:i///xxiv.// org/ org/abs/2012..7433 20122
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency En- hanced Decomposed Transformer for Long-lerm Series Forecasting. In Proceedings of the 39th International Conference on Machine Learning, pp. 27268 27286. PMLR, June 2022.
Yaxuan Zhu, Jianwen Xie, Ying Nian Wu, and Ruiqi Gao. Learning energy-based models by cooperative diffusion recovery likelihood. arXiv preprint 2xxi:2402.00077 February 2024. URL https://arrxe org/abs/2402.029
16

---

Published in Transactions on Machine Learning Research (February
A Appendix
A.1
Ablation study details
In this section, we provide detailed results for the ablation study conducted to evaluate the impact that proposed TEM inference methods and TEM joint training have on selective forecasting performance.
A.1.1 Comparison of TEM inference method performance
In this section, we provide detailed results for selective forecasting using both TEM inference methods: Energy optimization inference and Aggregatod energy inference. As shown in Table 3 experiments indicate that both Aggregated energy and Energy optimization are effective and reduce prediction error by more than 16.4%. For target coverages ((g) E {10%, 30%), the Energy optimization method had around 11.0% lower selective coverage but achieved 8.2% lower prediction error than the Aggregated energy method. However, for target coverages ((g) (50%, 70%]. the Energy optimization method had 9.2% lower selective coverage while achieving 7.9% higher prediction error. On average, across all target coverages, the Energy optimization method yielded around 6.2% lower selective coverage and 1.9% lower prediction error than the Aggregated energy method. However, this means that Energy optimization tends not to achieve target coverages as often as the Aggregated energy method. Energy optimization therefore recommended for opplications where lower prediction error prioritized over higher selective coverage. Notably, it also possible to use combination of both inference methods, depending on end-user requirements or desired error bounds, the use of either method does not require retraining the TEM model.
Autoformer
FEDtormer
Informer
PatchTST
TimesNet
Erer Reduction (8)
268
-100 -10
-100
-108
-100 -106
-100
-108
30
50
Actual Elveernge [N]
100
1.00
100
100
100
80
80
60
60
40
40
20
10 20 30 40 50 60 70 80 90
10 20 30 40 50 60 70 80 90
10 20 30 40 50 60 70 80 90
10 20 30 40
30 60 70 00 90
10
20 30 40 50 60 70 80 90
Target Coverage (1))
Target Coverage (%)
Target Coverage (%)
Target Coverage 136
Target Coverage (56)
TEM with no inference method
TEM using Energy optmizatoonnieeeeee
Selectivehee models
TEM Aggregated energy
Figure 5: Prediction error reduction (figures at the top), target and actual coverage percentages (figures at the bottom) for TEM models using Aggregated energy and Energy optimization inference methods as well as the naive baseline using EX(X, Y) directly for estimating uncertainty and selecting forecasts across selected target coverages ((g)= E {10%, 30%, 50%, 70%, 90%, 100%} on models Autoformer, FEDformer, Informer, PST and TimesNet. The top figures' Y-axes log scale to visualize the several orders of magnitude difference in performance between selective forecasting with TEM and SelectiveNet. For bottom figures, the dotted line represents the ideal case, where actual coverage is equal to target coverage.
We also evaluate the performance of TEM without using either Aggregated energy or Energy optimization inference methods. In this case, use the energy value at the model's output, Ex, Y), directly for estimating model uncertainty and selecting forecasts. As shown in Figure 5 the experiments show without using either of the proposed inference methods TEM fails to reduce forccast error and instcad increases it by on average 445.6% across all coverages. This is a 461.9% difference when compared to the Aggregated energy inference method and a 462.2% difference when compared to the Energy optimization inference mothod. However, the results also show that using Ex( Y) directly for selecting forecasts yields slightly higher coverage than using either Aggregated energy or Energy optimization inference methods, on average 7.8% higher coverage than Aggregated energy inference method and 14.9% higher coverage than Energy
17

---

Published in Transactions on Machine Learning Research (February/0255)
Table 2: TEM performance comparison across Aggregated energy and Energy optimization inference meth- ods. Results show selective risk and empirical coverage (in parentheses) for larget coverages x(g) e
10%, 30%, 50%, 70%. 90%. Best performing models for specific target coverages are marked bold.
Dataset
Model
ETTh1
ETTh2
Weather
Exchange Rate
National Illness
Energy-based model EB-NARX
0.2151
0.3003
0.0008
0.697
4.0931
Original
0.0876
0.1577
0.0079
0,0899
1.1758
10 %
0.0906 (16.80)
0.1591 (28.39)
0.0076 (57.18)
0.0712
(17.60)
0.5634 (2.78)
TEM Autoformer with
3D
0.0889 (51.79)
0.1591 (28.39)
0.9076 (37.13)
0.0708 (26.73)
1.0141 (20.14)
Aggregated Energy
0.0889 (51.79)
0.1568 (59.22)
0.0076 (37.18)
0.0902 (61.93)
1.0128 (42.13)
Inference
70 %
0.0871 (78.64)
0.1568 (59.22)
0.0075 (73.91)
0.0897 (69.35)
1.0464 (85.74)
90 %
0.0881 (87.45)
0.1568 (76.78)
0.0075 (73.91)
0.0890 (87.96)
1.0974 (83.10)
Original
0.0772
0.1184
0.011
0.0653
1.0503
10 %
0.0782 (45.86)
0.0958
(17.75)
0.0104 (61.27)
0.0354 13.19)
0.6754 (4.40)
TEM FEDformer with
30 %
0.0782 (45.86)
0.1192
(75.68)
0.0104 (61.27)
0.0616 (34.74)
0.8927 (13.66)
Aggregated Energy Inference
0.0770 (94.77)
0.1192 (75.68)
0.0101 (61.27)
0.0616 (34.74)
0.8470 (26.39)
70
0.0770 (9-1.77)
0.1192
(75.68)
0.0105 (67.49)
0.0631 (72.38)
0.8748 (13.75)
90 %
0.0770 (94.77)
0.1190 (85.77)
0.0105 (67.49)
0.0655 (91.33)
0.9229 (62.73)
Original
0.6461
1.1877
0.3313
0.73
4.6609
10 %
0.6008 (37.08)
1.2037 (13.05)
0.0046 (54.75)
0,1828 (7.81)
4.0733 (9.03)
TEM Informer with Aggregated Energy Inference
3D %
0.6008 (37.08)
1.1754 (27.66)
0.0046 (54.75)
0,2000 (8.11)
4.4375 (47.92)
50 %
0.6460 (89.32)
1.1845 (49.45)
0.0048 (54.75)
0.1921 (10.47)
4.3605 (66.20)
0.6460 (89.32)
1,1814 (70.75)
0.0046 (59.67)
0,2818 (12.98)
4.5612 (80.32)
90
0.6438 (93.87)
1,1850 (85.72)
0.0046 (59.67)
0.4741 (34.02)
4.5576 (90.74)
Original
0.0416
0.1079
0.0011
0.0617
0.7324
10
0.0340 (15.32)
0.0735 (11.98)
0.0008 (21.02)
0.0488 (6.33)
0.4686 (6.25)
TEM PStchTST with
30 %
0.0413 (30.05)
0,1077 (62.31)
0.0008 (21.02)
0.0526 (7.60)
0.42009 (12.96)
Aggregated Energy Inference
50 %
0,0418 (35.94)
0,1077 (62.31)
0.0008 (26.25)
0.0500 (15.00 60)
0.1210 (15.51)
70
0.0417 (00.38)
0,1076 (75.24)
0.0008 (33.02)
0,0505 (25.61)
0.5040 (32.18)
0.0417 (88.96)
0.1079 (93.97)
0.0008 (54.18)
0.0558 (46.40)
0.6313 (66.90)
Original
0.0138
0,1273
0.0016
0.05419
0.8391
0.0308 (16.80)
0,1069 (29.04)
0.0014 (61.17)
0.0416 (6.03)
0.5222 (5.79)
TEM TimesNet with
30 %
0.0447 (34.79)
0.1069 (29.04)
0.0014 (61.17)
0.0434 (16.12)
0.4281 (9.03)
Aggregated Energy Inference
50 %
0.0455 (71.70)
0.1281 (60.91)
0.0014 (61.17)
0.0451 (32.45)
0.4413 (13.66)
70 %
0.0455 (71.70)
0.1281 (82.69) 69)
0.0014 (65.83)
0.0485 (46.81)
0.4270 (16.90)
9D %
0.0459 (84.58)
0.1276 (94.16)
0.0014 (65.83)
0.0502 (64.89)
0.6104 (37.96)
Original
0.0876
0.1577
0.0079
0.0899
1.1758
10
0.0870 (23.03)
0.1295 (6.77)
0.0072 (49.13)
0.0770 (14.91)
0,8180 (7.41)
TEM Autoformer with
0.0877 (36.83)
0.1438 (33.00)
0.0072 (19.13)
0.0823 (27.69)
0,9259 (17.59)
Energy Optimization Inference
50 %
0.0868 (50.32)
0,1560 (55.28)
0,0072 (53.79)
0,0848 (14.15)
1.0024 (26.85)
70 %
0.0877 (70.49)
0.1523 (66.87)
0.0072 (72.44)
0.0881 (66.12)
1.0443 (41.44)
90 %
0.0874 (89.62)
0.1582 (93.41)
0.0073 (92.71)
0.0890 (80.91)
1.0252 (56.18)
Original
0,0772
0,1184
0.011
0.0653
1.0503
10 %
0.0837 (19.13)
0.1195 (1.33)
0.0104 (59.76)
0.0590 (14.18)
0.6012 (7.64)
TEM FEDformer with
0.0829 (31.89)
0.1191 (16.16)
0.0104 (59.76)
0.0624 (26.84)
0.6012 (7.64)
Energy Optimization
50 %
0.0802 (53.28) 28)
0.1172 (46.53)
0.0104 (6)
0.0644 (44.40)
0.6421 (12.50)
Inference
70 %
0.0721 (63.05)
0.1172 (46.53)
0.0104 (72.11)
0.0616 (65.46)
0.6245 (16.44)
90 %
0.0721 (63.05)
0.1172 (46.53)
0.0106 188
0.0632 (69.51)
0.6732 (29.86)
Original
0.6461
1.1877
0.3313
0.73
4.6609
10 %
0.3781 (1.70)
1.0603
(15.52)
0.0277 (21.73)
0.3151 (3.87)
1.0905 (12.50)
TEM Informer with
3D
0.6071 (28.61)
1.0449 (27.14)
0.0277 (24.73)
0.2251 (7.81)
4.0831 (30.79)
Energy Optimization Inference
0.6502 (48.61)
1.2784
(47.18)
0.0277 (24.73)
0.8804 (32.40)
4.9617 (53.70)
70
0.6382 (77.61)
1.1949
(71.68)
0.0262 (25.95)
0.5810 (42.69)
1.3389 (06.67)
90 %
0.6463 (91.31)
1.1938 (93.81)
0,1971 (42.59)
0,6350 (55.94)
4.4865 (80.56)
Original
0.0416
0.1079
0.0011
0.0617
0.7324
10 %
0.0423 (12.14)
0.0928 (23.16)
0.0007 (10.86)
0.0444 (5.69)
0.4118 (11.11)
TEM PStchTST with
30 %
0.0415 (28.87)
0.0916 (48.95)
0.0007 (10.86)
0.0448 (9.18)
0.4084 (16.67)
Energy Optimization Inference
50 %
0.0419 (53.24)
0.0918 (55.68)
0.0007 (10.86)
0.0533 (22.40)
0.4692 (20.83)
0.0415 (90.83)
0.1103 (74.94)
0.0007 (10.86)
0.0574 (44.31)
0.4503 (25.23)
90
0.0416 (98.28)
0. 1.12 (93.90)
0.0006 (15.13)
0.0584 (65.96)
0.4554 (35.65)
Original
0.0138
0.1273
0.0016
0.0549
0.8391
10 %
0.0459 (81.58)
0.0797 (9.60)
0.0012 (56.03)
0.0156 (13.68)
0.5387 (1.40)
TEM TimesNet with
30 %
0.0459 (84.58)
0.0745 (31.04)
0.0012 (56.03)
0.0627 (22.34)
0.4703 (12.73)
Energy Optimization
50 %
0.0459 (81.58)
0.1290 (73.26)
0.0012 (56.03)
0.0525 (32.01)
0.1489 (15.51)
Inference
70
0.0459 (84.58)
0.1290 (73.26)
0.0013 (71.81)
0.0526 (11.14)
0.1722 (22.69)
0.0459 (84.58)
0.1278 (90.36)
0.0015 (93.78)
0.0530 (54.6)
0.5636 (34.49)
18

---

Published in Transactions on Machine Learning Research (February/200)
Table 3: TEM performance comparison across Aggregated energy (Agg. Energy) and Energy optimization (Energy Opt.) inference methods. Results show selective risk and empirical coverage (in parentheses) for target coverages o(g) E {10%, 110%,00% 50%, 90%). Best performing models for specific target coverages are marked bold.
Dataset
Base Model
Coverage
Method
ETTh1
ETTH2
Weather
Exchange Rate
National Illness
EB-NARX
Original
0.2154
0.3003
0.0008
0.697
4.0934
Original
0,0876
0.1577
0.0079
0.0899
1.1758
Agg. Energy
0.0906 (16.80)
0.1591 (28.39)
0.0076 (57.19) 13)
0.0712 (17.60)
0.5634 (2.78)
10%
Energy Opt.
0.0870 (23.03)
0.1299 (6.77)
0.0072 (49.13)
0.0770 (14.91)
0.8180 (7.41]
30%
Agg. Energy
0.0869 (51.79)
0.1591 (28.39)
0.0076 13)
0,0708 (26.73)
1.0141 (20.14)
Energy Opt.
0.0877 (36.83)
0.1438 (33.00)
0.0072 (49.13)
0.0823 (27.89)
0.9259 (17.59)
Autoformer
Agg. Energy
0.0889 (51.79)
0.1568 (59.22)
0.0076 (57.13)
0.0902 (61.93)
1.0188 (42.13)
50%
Energy Opt.
0.0868 (50.32)
0.1560 (85.28)
0.0072 (58.79)
0,0818 (41.15)
1.0021 (26.85)
Agg. Energy
0.0871 (78.64)
0.1568 (59.22)
0.0075 (73.91)
0.0897 (69.35)
1.0464 (65.74)
70%
Energy Opt.
0.0877 (70.49)
0.1523 (66.87)
0.0072 (72.44)
0.0881 (66.12)
1.0441 (41.44)
Age. Energy
0.0881 (87.45)
0.1568 (76.78)
0.0075 (73.91)
0,0890 (87.96)
1.0974 (83.10)
90%
Energy Opt.
0.0874 (89.62)
0.1582 (93.41)
0.0073 (92.71)
0.0890 (80194)
1.0252 (56.48)
Original
0.0772
0.1188
0.011
0.0653
1,5503
10%
Age. Energy
0.0782 (45.86)
0.0958 (17-75)
0.0104 (61.27)
0.0354 (3.19)
0.6754 (4.40)
Energy Opt.
0.0837 (19.13)
0.1195 (1.33)
0.0104 (59.76)
0.0590 (14.18)
0.6012 (7.64)
30% Agg. Energy
0.0782 (45.86)
0.1192 (75.68)
0.0101 (61.27)
0.0616 (34.74)
0,8927 (133666
Energy Opt.
0.0829 (31.89)
0.1191 (16.16)
0.0104 (59.76)
0.0624 (26.84)
0.6012 (7.6.))
FEDformer
Agg. Eneryy
0.0770 (94.77)
0.1192 (75.68)
0.0104 (61.27)
0.0616 (34.74)
0.8470 (26.39)
50%
Energy Opt.
0,0802 (53.28)
0,1172 (46.53)
0.0104 (59.76)
0.0644 (44.40)
0,6421 (12.50)
Agg Energy
0.0770 (91.77)
0.1192 (75.68)
0.0105 (67.49)
0.0634 (72.38)
0.8748 (13.75)
70%
Energy Opt.
0,0721 (63.05)
0,1172 (16.33)
0.0104 (72.11)
0,0616 (65.46)
0,6245 (16.44)
90%
Agg. Energy
0.0770 (94.77)
0.1190 (85.77)
0.0105 (67.49)
0.0655 (91.33)
0.9229 (62.73)
Energy Opt.
0.0721 (63.05)
0.1172 (46.53)
0.0106 (88.18)
0.0632 (69-51)
0.6732 (20.86)
Original
0.6461
1.1877
0.3313
0.73
4.6609
10%
Agg. Energy
0.6008 (37.08)
1.2055
(13.05)
0.0016 (51.75)
0.1828 (7.81)
4.0733 (9.03)
Energy Opt.
0.3781 (4.70)
1.0603
(15.52)
0.0277 (24.73)
0.3151 (3.87)
4.0905 (12.50)
30% Agg. Energy
0.6008 (37.08)
1.1754 (27.66)
0.0046 (54.75)
0,2000 (8.11)
4.4375 (47.92)
Energy Opt.
0,6071 (28.61)
1.0449 (27.14)
0,0277 (24.78)
0:2251 (7.81)
4.0831 (30.79)
Informer
Agg, Energy
0.6460 (89.32)
1.1845
(49.15)
0.0046 (54.75)
0.1921 (10.44)
4.3605 (66.20)
50%
Energy Opt.
0.6502 (18.61)
1,2781 (17.18)
0.0277 (24.78)
0,8801 (32. 10)
1.9517 (53.70)
70%
ARR. Energy
0.6400 (89.32)
1.1814 (70.75)
0.0046 (59.67)
0,2818 (12.98)
4.5612 (80.32)
Energy Opt.
0,6382 (77.61)
1.1949 (74.68)
0.0262 (25.95)
0.5810 (42.69)
4.5380 (66.67)
Agg Energy
0.6438 (93.87)
1.1850 (85.72)
0.0016 (59.67)
0.4741 (34.02)
4.5576 (90.74)
90%
Energy Opt.
0.6463 (91.31)
1.1938 (93.81)
0.1971 (42.39)
0.6350 (55.94)
4.4863 (90.56)
Original
0.0416
0.1079
0.0011
0.0617
0.7324
Agg. Energy
0,0340 (15.32)
0.0735 (11.98)
0.0005 (21.02)
0.0488 (6.33)
0.4686 (6.25)
10%
Energy Opt.
0,0123 (12.14)
0,0928 (23. 16)
0.0007 (10.86)
0.0111 (5.69)
0.4118 (11.11)
Age. Energy
0.0413 (30.05)
0.1077 (62.31)
0.0008 (21.02)
0.0526 (7.60)
0.4209 (12.96)
30%
Energy Opt.
0.0115 (28.87)
0.0916 (48.95)
0.0007 (10.86)
0.0448 (9.18)
0.4084 (16.67)
PatchTST
Agg. Energy
0.0418 (55.94)
0.1077 (62.31)
0.0008 (26.25)
0.0500 (15.60)
0.4210 (15.51)
50%
Energy Opt.
0.0419 (53.24)
0.0918 (55.66)
0.0007 (10.86)
0.0533 (22.40)
0.4692 (20.83)
Agg. Energy
0.0117 (60.38)
0.1076 (75.24)
0.0008 (33.02)
0.0505 (25.61)
0.5040 (32.18)
70%
Energy Opt.
0.0415 (90.83)
0.1103 (74.94)
0.0007 (10.86)
0.0574 (44.31)
0.4503 (25:23)
Agg. Energy
0.0417 (58.96)
0.1079 (93.97)
0.0008 (84.18)
0.0558 (46.40)
0.6313 (66.90)
90%
Energy Opt.
0.0416 (98.28)
0.1082 (83.90)
0.0006 (15.18)
0.0584 (65.96)
0.1554 (35.65)
Original
0.0138
0.1273
0.0016
0.05-19
0.8391
Agg. Energy
0.0308 (16.80)
0.1069
(29.04)
0.0014 (61.17)
0.0416 (6.03)
0.5222 (5.79)
10%
Energy Opt.
0.0459 (84.58)
0.0797 (9.60)
0.0012 (58.03)
0.0458 (13.68)
0.5387 (4.40)
Agg. Energy
0.0447 (34.79)
0.1069 (29.04)
0.0014 (61.17)
0.0434 (16.12)
0.4281 [9.03]
30% Energy Opt.
0.0459 (84.58)
0.0745 (31.04)
0.0012 (56.03)
0.0527 (22.34)
0.4703 (12.73)
TimesNet
Agg. Energy
0.0455 (71.70)
0.1281 (60.91)
0.0014 (61.17)
0.0451 (32.45)
0.4413 (13.66)
50% Energy Opt
0.0459 (84.58)
0.1290 (73.26)
0.0012 (56.03)
0.0525 (32.01)
0.4489 (15.51)
Agg. Energy
0,0455 (T1.70)
0.1281 (82.69)
0.0014 (65.83)
0,0485 (46.81)
0.4270 (16.90)
70%
Energy Opt.
0.0159 (81.58)
0.1290 (73.26)
0.0013 (71.81)
0.0526 (11-11)
0.1722 (22.69)
90%
Agg. Energy
0.0459 (84.58)
0.1276 (94.16)
0.00114 (65.83)
0.0502 (64.89)
0.6104 (37.96)
Energy Opt.
0.0159 (84.5))
0.1278 (90.36)
0.0015 (98.78)
0.0530 (51.966
0.5636 (31.19)
19

---

Published in Transactions on Machine Learning Research (February/0255)
Table 4: Relative change in forecasting error for deterministic models using joint training and using Con- strastive Divergence only. Positive percentages indicate the relative forecasting error increase when models using only Contrastive Divergence (without joint training), negative percentages indicate the relative forecasting error reduction.
Deterministic model
ETTh1
ETTh2
Weather
Exchange Rate
National IIlness
FEDformer
+364.51%
+306.25%
+3085.459.
+-99.10%
+118.19%
Autoformer
+501.37%
+273.18%
+5396.20%
+379.64%
+88.07%
Informer
+212.24%
+81.71%
-1.51%
+-276
+116.32%
PatchTST
+53.37%
+11..86%
118.18%
+71.47%
+172.57%
TimesNet
+55.48%
+85.55%
-18.75%
+101.28%
+ +119.22%
optimization inference method. These results show that sampling a single energy value at the model's 's output (x, does not provide enough information for estimating model uncertainty and selecting forecosts.
Sampling energy values from around the model's output (as done when using Aggregated energy and Energy optimization inference methods) provides more accurate model uncertainty estimates. Intuitively, if the model is confident. in its prediction, the energy value is low, then the energy values around the model's output should also be low - model should be confident in forecasting very similar values. If the energy values on and around the forecast are high, then that indicates that the energy surface around the forecast is not low and the model might have not seen many similar examples in its training data.
However, aggregating the energy values can disrupt the natural ordering of the energy values (as generally, low energy values indicate high probability and high model confidence). We have observed that it when performing the calibration and ranking of energy values, the natural ordering of energy values is generally not disrupted, but. instead it improves the accuracy of the estimation of mean prediction error for each energy interval (as shown in Figure 3.. More accurate prediction error estimates enable TEM selective forecasting to more consistently select forecasts with lower prediction error, reducing prediction error more for the same larget coverages.
A.1.2 TEM performance when trained only using self-supervised learning
In this section, we provide the results of experiments training deterministic and TEM models without joint training, only using Contrastive Divergence self-supervised learning. The key difference between TEM joint training and training using only Contrastive Divergence is in how the model parameters are optimized. With joint training (Section 3.3), the deterministic model Ay is first trained using supervised learning to directly minimize forecasting crror, and then the cnergy-basod model paramcters 0 are trained using Contrastive Di- vergence while keeping Av parameters frozen. In contrast, when training using only Contrastive Divergence, all model parameters (both As and Fi)) are trained simultaneously using self-supervised learning to learn the entire data distribution, without directly optimizing for forecasting accuracy.
As shown in Table 4 the experiments indicate that without joint training, deterministic models have average 498.4% higher forecasting error across all 5 models and 5 datasets. However, on the Weather dataset, using Contrastive Divergence yielded a slight increase in forecasting accuracy for models TimesNet and Informer, 18.8% and 1.5% respectively. The performance reduction is particularly significant for Autoformer and FEDformer models, where training using only Contrastive Divergence results in up to 5396.2% and 3085.5% higher error on average across all datascts. These results show that the proposed TEM joint training method is essential for mainluining high determinislic forecasting accuracy.
As result of poor deterministic performance, TEM models without joint training do not yield any relative error reduction when compared to TEM trained with joint training. As seen in Figure D. on average, TEM without joint training increases forecasting error when using selective forecasting by 2798.6% across all coverages. Notably, TEM without joint training yields 89.5% lower forecasting error than SelectiveNet models. However, neither TEM without joint training nor SelectiveNet models yield sufficient deterministic performance to be used effectively in practice. Poor deterministic performance of TEM models trained without joint training can be attributed to the models not being trained using loss function that is directly
20

---

Published in Transactions on Machine Learning Research (Februar/20255)
optimized for forecasting error. EBMs are trained using Contrastive Divergence, which is a loss function designed to learn the entire data distribution, not directly optimize forecasting error. Without the utilization of supervised learning, as proposed in this work, TEM models (or generative models in general) often cannot show comparable forecasting performance to conventional discriminative deterministic models (Bond- Taylor
et al.
2022 Zheng et al. 2023
Autoformer
FEDtormer
Informer
PatchTST
TimesNet
Artual Doverage (86 Errar Reduction (%)
180
-100
-168
10 20
30
40 30
60
70 80
90
10
20
30
40
50
60
70 80
90
10
20
30
40
50
60
70 80 90
10
20
30
40
50
60
70 80 90
10
20
30
40
50
60
70
80 90
100
09
900
00
010
70
70
70
70
5:0
50
50
50
30
30
20 5b 60 Coverage
30 40 70
20 30 50 60
20 30 40 so 70 80
20 30 40 so 60 70
"Durget Coverage
Target Coverage
Target Coverage
Target Coverage
TEM without joint training
SelectiveNit models
TEM
Figure 6: Prediction error reduction (figures the top), target and actual coverage percentages (figures at the bottom) for TEM models trained with and without joint training, and SelectiveNet for multivariate selective forecasting, across selected target coverages o(g) E {10%, 30%, 50%, 70%, 90%, 100%} on models Autoformer, FEDformer, Informer, PatchTST, and TimesNet. The top figures' Y-axes use log scale to visualize the several orders of magnitude difference in performance between selective forecasting with TEM and SelectiveNet. For bottom figures, the dotted line represents the ideal case, where actual coverage is equal to target coverage.
A.2 Additional experiments
In this section, we provide detailed results for additional experiments evaluating TEM performance for selective univariate time-series forecasting and further analysis of SelectiveNet performance.
A.2.1 TEM performance for univariate selective forecasting
In this section, we evaluate TEM performance for univariate selective forecasting. In these experiments, TEM was trained to forecast using only observed data for one feature (the larget. variable) TEM achieves similar performance in the univariate forecasting scenario to multivariate forecasting. As seen in Table 5 across all models and datasets, TEM achieves an average prediction error reduction of 23.6% for target coverages (()) E {10%, 30%}. which is within 2% of the multivariate case. This suggests that the effectiveness of TEM is not significantly impacted by the dimensionality of the forecasting task.
As can be seen in Figure 7 the performance gap between TEM Aggregated energy and Energy optimization inference methods notably smaller for univariate selective forecasting. The Energy optimization method achieves on average 10.9% lower prediction error than Aggregated energy, while having 3.5% lower coverage. SelectiveNet shows improved performance in the univariate case, achieving 80.7% lower prediction error compared to SelectiveNet applied for multivariate forecasting across all models and datasets. This could be attributed to the fact that univariate forecasting is comparatively easier task, the forecasting model does not need to consider the interactions between covariate features. SelectiveNet, like in the multivariato selective forecasting case, also performs well with the Informer architecture, in some cases reducing error by up to 95% and outperforming TEM with Aggregated energy in most scenarios across all five datasets. However, SelectiveNet still on average increases forecasting error by 820.0% compared to deterministic mod- cls, and performs 4944.5% worse than TEM across all target coverages and across all models and datasets, showing that it does not generalize across different model architectures and datasets well, unlike TEM.
21

---

Published in Transactions on Machine Learning Research (February/202)
Table 5: TEM performance comparison for univariale selective forecasting. Results show selective risk and empirical coverage (in parentheses) for target coverages ((g) E {10%, 30%, 50,, 70%, 90%}. Best performing models for specific target coverages are marked bold.
Base Model
Coverage
Method
Dataset
ETTh1
ETTh2
Weather
Exchange Rate
National Illness
EB-NARX
0.2151
0.3008
0.0008
0.697
4..931
Original
0.0876
0.1578
0.0078
0:0897
1.1766
TEM
0.0869 (88.18)
0.1471
(12.09)
0.0076 (47.58)
0.0535 (7.17)
0.7639 (7.41)
10%
SelectiveNet
0.1667 (53.82)
2.8711 (51.16)
0.00/1 (56.58)
1.8717 (49.61)
3,4556 (50.35)
TEM
0.0869 (58.18)
0.1537 (52.55)
0.0076 (17.58)
0.0866 (67.01)
0.9514 (25.23)
30%
SelectivoNet
0.1116 (55.25)
1,2875 (52.96)
0.0412 (40.22)
18006 (58.53)
3.8514 (50.45)
Autoformer
TEM
0.0869 (88.18)
0.1537 (52.55)
0.0075 (53.51)
0.0868 (67.01)
1.1224 (72.45)
50%
SelectiveNet
0,1392 (51.56)
1,5872 (18.32)
0,0163 (54.23)
1.6051 (50.08)
1.0139 (58.30)
TEM
0.0864 (90.90)
0.1564 (72.45)
0.0075 (70.76)
0.0866 (67.01)
1.1224 (72.45)
70%
SelectiveNet
0.1384 (64.39)
3.7973 (70.33)
0.1271 (80.08)
1.9077 (78.
4.1406 (70.55)
TEM
0.0864 (90.90)
0.1578 (87.71)
0.0075 (92.00)
0.0889 (91.73)
1.1416 (86.57)
90%
SelectiveNet
1.4219 (93.44)
4.2827 (96.40)
0.0404 (98.96)
3.1112 (89:03)
4.7521 (84.38)
Original
0,0772
0.1185
0.011
0.0648
1,0198
10%
TEM
0.0594 (18.63)
0.0960
(20.39)
0.0102 (62.40)
0.0444 (8.24)
0.5466 (3.70)
SclectiveNet
0.1176 (49.65)
0.0007 (0.41)
0.1456 (55.01)
0,1360 (50.32)
1,5580 (65.82)
TEM
0.0773 (63.75)
0.1209 (64.70)
0.0102 (62.40)
0,0551 (42.14)
0.7346 (22.22)
30%
SelectiveNet
0.1002 (53.32)
0.1260 (75.43)
0.0066 (19.75)
0,1068 (16.51)
1.3187 (55.67)
FE.fformer
TEM
0.0773 (63.75)
0.1209 (64.70)
0.0102 (62.40)
0.0619 (74.50)
0.7089 (29.86)
50%
SelectiveNet
0.1084 (63.59)
0.1410 (24
0.0071 (62.80)
0.1029 (G3.64)
1.2956 (53.68)
TEM
0.0769 (69.77)
0.1200 (74.55)
0.0103 (68.51)
0.0619 (74.50)
0.7250 (38.66)
70%
SelectiveNet
0,2000 (58,23)
0,1698 (97.25)
0.0108 (63.73)
0,1538 (68.20)
2.7279 (69.05)
TEM
0.0775 (91.93)
0.1184 (92.26)
0.0103 (68.51)
0.0654 (93.60)
0.9505 (66.90)
90%
SelectiveNet
0,2103 (91.04)
0,1320
(82.71)
0.0076 198
0,2258 (87.39)
3.8030 (88.10)
Original
0.6459
1,1883
0.331
0.7290
4.6562
TEM
0.6140 (14.03)
1.1228
(16.51)
0.0079 (58.72)
0.2263 (6.63)
3.8635 (24.54)
10%
SelectiveNet
0,2162 (75.14)
0.0791
(34.24)
0.0026 (22.32)
0.1179 (00.32)
1.6542 (45.53)
TEM
0.5962 (38.37)
1,1450
(32.89)
0.0079 (58.72)
0.2263 (6.63)
3.8635 (24.54)
30%
SelectiveNet
0.1080 (27.82)
0.0882 (41.66)
0.0064 (46.22)
0.1346 (55.71)
2,6800 (63. 18)
Informer
TEM
0,6437 (90.94)
1.1450 (32.89)
0.0079 (58.72)
0.4293 (18.12)
4.3947 (39.12)
50%
SelectiveNet
0,2090 (83.82)
0.0581 (29.83)
0.0120 (61.19)
0,1201 (32.81)
1.5793 (12.86)
TEM
0.6437 (90.94)
1.1749 (54.26)
0.0069 (64.92)
0.4847 (35.66)
4.7776 (67.36)
70%
SclectiveNet
0.3951 (95.74)
0.1736 (99.35)
0.0077 (97.83)
0.1917 (63.12)
0.4581 (12:22)
TEM
0.6437 (90.94)
1,1766 (79.21)
0.0069 (64.92)
0.5828 (59.65)
4.5910 (90.51)
90%
SelectiveNet
0.1277 (92.33)
0.1537 (61.71)
0.0179 (98.68)
2.0231 (51.53)
Original
0.0416
0.1075
0:0011
0.0617
0.7324
TEM
0.0309 (9.13)
0.0817 (30.45)
0.0006 (38.06)
0.0383 (2.87)
0.5478 (6.25)
10%
SelectiveNet
0.9696 (51.24)
0.5151 (61.69)
0.0031 (51.76)
1.5066 (15.68)
3,3755 (50.57)
TEM
0.0408 (39.34)
0.0817 (30.45)
0.0006 (36.06)
0.0440 (11.98)
0.6111 (11.34)
30%
SelectiveNet
0.9696 (51.21)
0.5451 (61. 69)
0.0034 (51.76)
1.5066 (15.68)
3,3755 (50.37)
PatchTST
TEM
0.0405 (56.43)
0.1085 (63.04)
0.0007 (47.94)
0.0529 (34.11)
0.5964 (20.60)
50%
SelectiveNet
0.9696 (51.24)
0.6768 (70.70)
0.0070 (58.87)
1.5066 (45.68)
3.3755 (50.57)
TEM
0.0401 (63.89)
0.1088 (71.10)
0.0009 (81.80)
0.0536 (11.87)
0.6068 (38.19)
70%
SelectiveNet
1.3973 (80.32)
0.7841 (78.96)
0.0009 (86.33)
2.7152 (74.00)
4.5747 (72.65)
90%
TEM
0.0412 (87.25)
0.1079 (92.44)
0.0009 (89.11)
0.0570 (78.25)
0.6580 (62.78)
SelectiveNet
1,5713 (92.21)
1.1102 (90.96)
0.0059 (95. 16)
3.0774 (80.21)
6.2271 (80.40)
Original
0.0138
0.1273
0.0016
0.05-19
0.8391
10%
TEM
0.0308 (16.80)
0.1069
(29.04)
0.0014 (61.17)
0.0406 (7.97)
0.5222 (5.79)
SelectiveNet
0.9464 (53.09)
1:2297 (45.05)
0.3200 (49.72)
2.0732 (15.11)
3,2588 (62.76)
TEM
0.0447 (34.79)
0.1069 (29.04)
0.0014 (61.17)
0.0436 (25.59)
0.4281 (9.03)
30%
SelectiveNet
1.0185 (57.71)
1.2813 (47.15)
0.3517 (51.15)
1.9870 (32.73)
3,2699 (52.94)
TimesNet
TEM
0.0455 (71.70)
0.1281 (60.91)
0.0014 (61.17)
0.0451 (32.45)
0.4413 (13.66)
50%
SeleetiveNet
0.8006 (54.48)
0.5844 (25.75)
0.3560 (54.01)
1.8190 (49-63)
3,5668 (57.96)
TEM
0.0455 (71.70)
0.1281 (82.69)
0.0014 (65.83)
0.0463 (49.70)
0.5914 (31.48)
70%
SolectiveNet
1.2080 170.
1.7400 (79.76)
0.5239 (78.78)
3.4055 (85.40)
4.7244 (78.43)
TEM
0.0459 (84.58)
0.1276 (94.16)
0.0014 (65.83)
0.0513 (72.50)
0.6939 (61.57)
90%
SelectiveNet
1.5088 (90.83)
2.8587 (92.41)
0.6379 73)
3.3017 (90.90)
5.8530 (93.06)
22

---

Published in Transactions on Machine Learning Research (February/20) 2022)
Autoformer
FEDformer
Informer
PatchTST
TimesNet
Actual Coverage (96) Brerr Reduction (50
60
40
20
-100-
-0kK
10 20
30
50
60
70 90
-10k
10
20
30
40
50
70 80 90
10
20
30
40
50
60
70 00
-10k
10
20
30
40
50
70 80 90
10
20
30
40
50
60
70 80 00
80
00
90
90
70
70
70
70
50
50
30
20 50
Target Coverage (5))
Target Coverage (%)
Target Coverage (%)
Target Coverage
Target Coverage (56)
TEM using Aggregated energy inference
TEM using Energy optimization inference
SelectiveNet models
Figure 7: Prediction error reduction (figures at the top), target and actual coverage percentages (figures the bottom) for TEM and SelectiveNet for univariate selective forecasting, across selected target coverages ((g) E (10%, 30%, 50%, 70%, 90%, 100%) on models Autoformer, FEDformer, Informer, PatchTST, and TimesNet. The top figures' Y-axes use log scale to visualize the several orders of magnitude difference in performance between selective forecasting with TEM and SelectiveNet. For bottom figures, the dotted line represents the ideal case, where actual coverage is equal to target coverage.
A.2.2 SelectiveNet performance analysis
SelectiveNet Autofomer
SelectiveNet Informer
reduction (9))
-250
-300
50
11000
1120
10
15
20
25
5
10
15
20
25
30
100
100
90
90
80
70
60
60
50
50
30
30
20
20
10
10
10
25
20
Epochs
Target Coverage
10%
50%
-d-- 50%
---
70%
90%
100%
Figure Error reduction (figures at the top) and actual coverage percentages (figures the bottom) for SelectiveNet. Autoformer and Informer models the ETTh2 dataset. for epochs E [1,30] and target coverages d(g) E (10%, 30%, 50%, 70%, 90%, 100%).
In this section provide additional analysis on the performance of SelectiveNet to identify the reasons for its poor performance compared to TEM. We trained 3 SelectiveNet models using different seeds for each of the Autoformer and Informer variants of SelectiveNet on the ETTH2 dataset for epochs E [1,30] and target coverages ((g) E (10%,30% 30%, 50%, 70%, 90%, 100%].
SelectiveNet models tend to converge a stable coverage, the actual coverage percentages reach close the target coverage after a few epochs. As the coverage converges, the forecasting performance of SelectiveNet. also stops improving despite further training. This indicates that the loss
As shown in Figure X
23

---

Published in Transactions on Machine Learning Research (262raay/20255)
function used by SelectiveNet prioritizes achieving the target coverage during optimization, rather than minimizing prediction error, which is consistent with all prior experimental results. And since the models converge, even if training were extended with more epochs, the performance would not meaningfully improve.
Furthermore, we notice that for lower coverages, SelectiveNet consistently achieves significantly higher ac- tual coverage than target coverage. On average, SelectiveNet achieves 35.0% and 15.5% higher coverage across both models for coverages ((g) (10%, 30%), respectively. This shows that SelectiveNet is overly conservative during training, selecting too many forecasts and achieving higher coverage than desired and, result, higher prediction error. These patterns consistent across all tested models and datasets.
We conducted additional experiments to evaluate the performance of SelectiveNet when trained using a traditional loss function Mean Squared Error, without using the selection head. This SelectiveNet without using the selection head achieved the same deterministic forecasting performance as the base model Ap.
A.3 Additional information on datasets
In this section, we provide more information for the five datasets used in experiments for evaluating TEM performance. The Features column in Table G represents the number features the dataset, including the larget variable. The Dataset Size column in Table shows the number of data points each of the training, validation, and test subsets.
Table 6: Statistics for Time Series datasets used in experiments
Name
Features
Dataset Size
Frequency
Domain
ETThl
(8545, 2881, 2881)
Hourly
Electricity
ETTh2
(8545, 2881, 2881)
Hourly
Electricity
Exchange
8
(5120, 665, 1422)
Daily
Exchange Rate
Weather
21
(36792, 5271, 10540)
Daily
Weather
National Illness
(617, 74, 170)
Weekly
Illness
A.4 Contrastive Divergence training
In this section, provide the joint training algorithm for training TEM with Contrastive Divergence enabling selective forecasting.
Algorithm Calculating Contrastive Divergence (CD) loss
INPUT:
Eg Energy-based based model (EBM)
X
Ground-truth input
(O) Ground-truth output given input X
CD step size
ØCD CD regularizer coefficient
NED CD step count
Output:
LCD Contrastive Divergence (CD) loss
()) + o0(,0211)
for + to NeD do
w NNO0 021) y()) y()) Vos (()) +
Eq. 11
end for
E+ + (x(XY
E- t Fx(X,Y'o y(1)
LCD + (E+ E (E))
Eq.
return LOD
24

---

Published in Transactions on Machine Learning Research (ebruary/2025]
A.5 Additional implementation details on SelectiveNet
In this work, we adapt the SelectiveNet framework for time-series forecasling by modifying its architecture and loss function to work with state-offthe-art deterministic forecasting models. Similar to TEM, our SelectiveNet implementation reuses architecture from the deterministic forecasting model Ae to maintain the model's ability to capture temporal dependencies and patterns the input time series (as described in Section 3.2) The adapted SelectiveNet architecture adds three heads on top of the deterministic model Ay: 1) a selection head added on top of the encoder of Ay, that outputs selection scores (and is used to implement the selective function g shown in Equation 6), 2) a prediction head is appended to the decoder of the Ay and produces deterministic forecasts, and 3) auxiliary prediction head is added to the decoder of the Ag and also produces predictions, as described in the original SelectiveNet paper (Geifman & El- Yaniv 2019).
A.6 Additional implementation details on TEM
In this section, provide additional implementation details. The base implementation of TEM models presented in this paper is available at https://github. www/ /ot.aa/rrrsskkkss/ieeeeeeeeeeeee
The
deterministic forecasting models (Informer, Autoformer, FEDformer, PatchTST, and TimosNet, were trained according to the peprraaraeetess specified in their respective papers (Zhou et al. 2021 Wu et al. 2021 Zhou
et al. 2022 Wu et al., 2023 Nie et al. 2023). The model implementations were adapted from an open-source time-series forecasting model repository that implements state-of- thearrt models and runs experiments with the same hyperparamcters, ensuring consistent and reproducible model architectures and training procedures. This repository available at hhttps:// wtt. http:////ao.oocoimiiaeeeeeeeeeee
For all experiments use the same training, validation, and test data splits the original papers. Sequence length m was set to 96 for all models, with prediction horizon h set to 48.
For deterministic forecasting models Ag, we recreate the model architectures and training procedures as described the original papers, using the same hyperparameters. All models were trained with 2-layer en- coders, 1-layer decoders. All deterministic models were trained using the Mean Squared Error loss function, using Adam optimizer with learning rate 0.0001 and dropout rate of 0.05. Deterministic models were trained for up 30 epochs, using carly stopping with patience parameter 3. Transformer-basee models (Informer, Autoformer, FEDformer, and PatchTST) were trained with 8 attention heads and 512 dimensionality emn bedding, attention, and feed-forwer layers.
A.7 Standard deviations across experiments
As mentioned in Section 5.1 the results reported are the average of runs. We provide the standard deviations across the 3 runs for model and dataset combination in Table
Table 7: Standard deviations for selective risk for the tested models and Selective models across runs.
Dataset
Model
ETThh
ETTh2
Weather
Exchange Rate
National Illness
TEM Autoformer
0.0817
0.1421
0.0074
0.0929
0.4474
TEM FEDformer
0.0833
0.1041
0.0078
0.0708
0.4442
TEM Informer
0.3570
0.6497
0.6855
0.6503
2.4439
TEM PatchTST
0.0479
0.1021
0.0016
0.0657
0,4128
TEM TimesNet
0.0536
0.1255
0.0025
0.0582
0.4065
SelectiveNet Autoformer
0.0977
0.2146
0.0106
0.2769
0.5654
SelectiveNet FEDformer
0.1738
0.6707
0.0467
0.1086
0.2989
SelectiveNet Informer
0.4178
0.7700
0.0243
0.4219
0.4342
SelectiveNet PatchTST
0.2400
0.5423
0.0334
0.8309
0.5265
SelectiveNet TimesNet
0.1276
0.4620
0.2089
0.7244
0.7938
25

---

Published in Transactions on Machine Learning Research (February/002) 2005)
A.8 Additional experiments with LSTM
Table 8:
Comparison of TEM performance with Informer, TimesNet, and LSTM models. Results show selective risk and empirical coverage (in parentheses) for target coverages x(g) E 10% 30%, 50%, 70%, 90%. Best performing models for specific target coverages are marked bold, second- best performers are underlined.
Dataset
Model
ETTh1
ETTh2
Weather
Exchange Rate
National Illness
Original
0.6161
1,1877
0,3313
0.73
4,6609
10
0.6008 (37.08)
1.2057 (13.05)
0.0046 (54.75)
0.1828 (7.81)
4.0733 (9.03)
TEM Informer with Aggregated Energy Inference
0.6008 (37.08)
1.1754 (27.66)
0.0046 (54.75)
0.2000 (8.11)
4.4375 (47.92)
50 %
0.6460 (89.32)
1.1845 (49.45)
0.0046 (54.75)
0.1921 (10.47)
4.3605 (66.20)
70 %
0.6460 (89.32)
1,1814 (70.75)
0,0046 (39.67)
0,2818 (12.98)
4.5612 (80.32)
90 %
0.6438 (93.87)
1.1850 (85.72)
0.0046 (39.67)
0.4741 (34.02)
4.5576 (90.74)
Original
0.0438
0.1273
0.0016
0.0549
0.8391
10 %
0.0308 (16.80)
0.1069
(29.04)
0.0014 (61.17)
0.0416 (6.03)
0.5222 (5.79)
TEM TimesNat with
3D
0.0141 (34.79)
0.1069 (29.04)
0.0014 (61.17)
0.0434 (16.12)
0.4281 (9.03)
Aggregated Energy Inference
0.0455 (71.70)
0.1281 (60.91)
0.0014 (61.17)
0.0451 (32.45)
0.4413 (13.66)
0.0455 (71.70)
0.1281 (82.69)
0.0014 (65.83)
0.0485 (46.81)
0.4270 (16.90)
0.0159 (841.5)
0.1276 (91.16)
0.0014 (65.83)
0.0502 (61.89)
0.6104 (37.96)
Original
0.1514
0.1272
0.0033
0.085
4.196
10 %
0.0443 (1.91)
0.1263 (4.96)
0.0026 (45.27)
0.0596 (0.72)
(0.00)
TEM LSTM with Aggregated Energy Inference
30
0.1407 (25.35)
0.1191 (20.02)
0.0026 (45.27)
0.0793 (4.92)
3.6716 (33.68)
0.1582 (30.00)
0.1264 (51.92)
0.0020 (60. 16)
0.0551 (17.25)
3,2841 (11.11)
0.1406 (77.44)
0.1264 (51.92)
0.0021 (65.77)
0,0199 (39.28)
3.97 (63.89)
0.1406 (77.44)
0.1264 (51.92)
0.0034 (98.03)
0,0667 (64.96)
4,2693 (85.07)
In this section, we provide additional experiments with LSTM deterministic models for lime-series forecast- ing. As with the rest of the models, we use the same training, validation, and test data splits as the original papers. Soquence length m was set to 96 for all models, with prodiction horizon h set to 48. The architecture selected used 1 layer of LSTM cells with 128 hidden units for the encoder and fully connected layer with 128 hidden units for the decoder. The LSTM was trained using the Mean Squared Error loss function, using Adam optimizer with learning rate 0.0001 and dropout rate of 0.05. The LSTM was trained for up to 30 epochs, using early stopping with patience parameter of 3. We used the same training procedure to train TEM with LSTM as the deterministic model, as with the rest of the deterministic models.
We provide the results for TEM with LSTM in Table S where we compare TEM with LSTM to TEM with Informer and TimesNet, the worst and best performing models in the deterministic forecasting experiments. The deterministic LSTM model numerically underperforms against the top-performing TimesNet model, but outperforms the Informer model in all observed cascs. Results indicate that TEM with LSTM achieves, average, 71.8% lower prediction error than TEM with Informer across all targel coverages. TEM is able to recuce the prediction error of the LSTM model by up to 30% across all datasets. However, we observe that TEM with LSTM often yiclds lower than the target coverage. When target coverages are low (d(g) E {10%, 30%}), the TEM with LSTM model yields up to 10% lower coverage than the target coverage. It important note, however, that the encoder of the LSTM model is only 1 layer deep and the model itself is significantly smaller than all the other models tested, which results in a worse performing encoder and less accurate representations of the input time scries.
A.9 Error-bounded selective forecasting with TEM
In this paper we propose TEM for selective forecasting based on target coverage, but the framework can be extended to use other selection criteria. This section describes how TEM can be used to reject predictions based on estimated prediction error bounds.
The key difference from the original coverage-based approach is that instead of selecting energy intervals to achieve a target coverage, we can use the relationship between energy values Ex, Y) and prediction errors (XEg(X, ()) to reject predictions that are likely to have errors above a specified threshold. Specifically, after partitioning the energy range into intervals Eg, and calculating mean prediction error e for each interval (as described in Section 3.4 we can define selective forecasting with error bounds as:
26

---

Published in Transactions on Machine Learning Research (February/0255)
[x( (f ss. None,
(Av, E)) -
(15)
otherwise,
where 8 is the maximum acceplable prediction error and c(Eo(X, P)) is the estimated prediction error for energy Y))
This approach allows end-users to specify the maximum error bound for their use-case. Like with the coverage-based approach, this approach also allows dynamic adjustment of selection criteria based the utility of the prediction and potential penalty for errors.
Like the original coverage-based approach, the error bound-based selection can use either the Aggregated energy or Energy optimization inference methods described in Section 3.4 Both the coverage-based and error bound-based approaches can be used interchangeably without retraining the modol, as they both utilize the same underlying relationship between energy values and prediction errors.
A.10 Comparison of selective forecasting with probabilistic forecasting models
In this section, we provide a comparison of TEM with a probabilistic forecasting model TimeGrad.
The TimeGrad model is autoregressive denoising diffusion model designed for multivariate probabilistic time series forecasting (Rasul et al. 2021). Uses recurrent neural networks (RNNs) to encode past sequences and temporal dependencies using the hidden state. Inference with TimeGrad is performed using annealed Langevin dynamics, which generates multiple samples to obtain empirical quantiles of uncertainty for each prediction.
To enable fair comparison between TimeGrad (a probabilistic model) and TEM, we adapt TimeGrad for selective forecasting follows: First, we generate multiple samples from TimeGrad and use the variance between these samples uncertainty measure, (rather than using energy values in TEM). We then apply similar selective forecasting procedure described in Section 3.4.
1. Divide predictions into intervals based on their variance
Calculate prediction error and coverage for each interval
3. Select forocasts that meet the target coverage critcria
This adaptation allows us to directly compare TimeGrad's selective forecasting capabilities with TEM. Note, that we do not perform any type of aggregation or additional sampling (beyond generating 10 forecasts) as we do in TEM selective forecasting. For TimeGrad, we also do not rank the intervals by prediction error, instead assuming that the variance between samples is good proxy for prediction error.
Experiments are conducted using the openly available GitHub repository from the authors TimeGrad [Rsuu et al. 2021). TimeGrad is trained for 20 epochs, with batch size of 64 and GRU is used to model the hidden state. Other hyperparameters are the same hyperparameters as in the original paper or the default values used in the repository. Inference is performed autoregressively, by predicting one time-step at a time. We use anncaled Langevin dynamics generate S forecasts from TimeGrad. The same datascts are used in experiments and same preprocessing is applied to the data in prior experiments. For selective forecasting, use the same target coverages d(g) {10% 30%, 50%, 70%, 90%} in prior experiments.
As can be seen in Table 9 TEM with TimesNet outperforms TimeGrad in all cases. The best performing TEM TimesNet model has over 91% lower error across all datasets for the deterministic forecasting case and can consistently further reduce error by performing selective forecasting. TimeGrad is more competitive against Informer, outperforming it on the ETTh1, ETTh2, and Exchange Rate datascts. However, Informer has, on average, 42% lower error than TimeGrad, because of the very high errors recorded on the National Illness and Weather datasets.
27

---

Published in Transactions on Machine Learning Research (February/022)
Table 9: Comparison of TEM performance with Informer, TimesNet, and TimeGrad models. Results show selective risk and empirical coverage (in parentheses) for target coverages Φ(g) E 10%, 30%, 50% 70%, 90%. Best performing models for specific target coverages marked bold, second-best performers are underlined.
Dataset
Model
ETTh1
ETTh2
Weather
Exchange Rate
National Illness
Original
0.6461
1.1877
0.3313
0.73
4.6609
10 %
0.6008 (37.08)
1,2057 (13.05)
0,0046 (31.70)
0,1828 (7.81)
4,0733 (9.03)
TEM Informer with Aggregated Energy Inference
30
0.6008 (37.08)
1.1754 (27.66)
0.0046 (54.75)
0.2000 (8.11)
4.4375 (47.92)
50
0.6460 (89.32)
1,1845 (49.45)
0.0046 (54.75)
0.1921 (10.47)
4.3605 (66.20)
70
0.6460 (89.32)
1.1814 (70.75)
0.0046 (39.67)
0.2818 (12.98)
4.5612 (80.32)
0.6438 (93.87)
1,1850 (85.72)
0.0046 (39.67)
0.4741 (34.02)
4.5576 (90.74)
Original
0.0438
0.1273
0.0016
0.0549
0.8391
0.0308 (16.80)
0.1069
(29.04)
(61.17)
(6.03)
0.5222 (5.79)
TEM TimesNet with
0.0447 (34.79)
0.1069 (29.04)
0.0014 (61.17)
0.0434 (16.12)
0.4281 (9.03)
Aggregated Energy
50 %
0.0155 (71.70)
0.1281 (60.91)
0.0014 (61.17)
0.0451 (32.45)
0.4413 (13.66)
Inference
70 %
0.0155 (71.70)
0.1281
(82.69)
0.0014 (65.83)
0.0485 (16.81)
0.4270 (16.90)
90
0.0459 (84.58)
0.1276 (94.16)
0.0014 (65.83)
0.0502 (64.89)
0.6104 (37.96)
Original
0-4235
1.1541
0.7733
0.1459
9.1967
0.3030 (61.95)
1.3082 (86.67)
0.8338 (72.53)
(0.00)
5.5429 (7.92)
TimeGrad
30
0.4197 (99.13)
1.3082 (86.67)
0.7929 (86.70)
0.1147 (15.35)
5.5429 (7.92)
50
0.4224 (99.70) 76)
1.3082 (86.67)
0,7929 (86.70)
0.1201 (32.35)
5.5429 (7.92)
70 %
0.4235 (100 00)
1,3082 (86.67)
0,7929 (86.70)
0.1283 (17 06)
5.5429 (7.92)
90 %
0.4235 (100.00)
1.3082 (86.67)
0,7929 (86.70)
0.1254 (12.16)
5.5429 (7.92)
These results can be explained by the fact that TimeGrad is probabilistic model and is not trained or tuned to perform very accurate deterministic forecasting. As result, even though TimeGrad can be used for selective forecasting to reduce forecasting error by trading off coverage, it is still not as accurate as TEM. Furthermore, the TimeGrad model is significantly slower during inference because of its autoregressive forccasting and sampling process. Finally, TimeGrad often cannot effectively perform seleetive forecasting, because the variance between samples is not a consistently good proxy for prediction error. As a result, selective forecasting sometimes increases the forecasting error (as in the ETTh2 and Weather dataset cases).
A.11 Discussion on applications of TEM to forecasting of multiple time-series
Currently, TEM has been evaluated only for the case of multivariate time-series forecasting where the fore- casted time-scries is univariate. TEM in principle allows for selective forecasting of multivariate timo-series, but this has not been investigated in the context of multivariate time-series forecasting. In particular, when forecasting multiple target variables simultaneously, the relationship between energy values and prediction errors may become more complex. It is possible that for high-dimensional multivariate forecasting problems, the output search space can grow too large to sample energy values and provide accurate prediction error estimates. Energy-based models have been shown to work well even in very high-dimennsioall spaces (e.g. with image video data) (Du & Mordatch| 2019), but this has not been investigated in the context of multivariate time-series forecasting and will be part of future work.
28


Reviews:
[{"review_id": "qACgvC8gde", "reviewer": "Reviewer_3U5f", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strenghts**  \n- The proposed methodology is novel and tries to address shortcomings of current EBM inference strategies with approaches that reduce the computational burden of sampling energy values in the context of a large output space like that of multi-step ahead time series forecasting.\n- The provided results show that the approach is suitable as a plug-in component for different state-of-the-art time series forecasting models. Moreover, it compares favorably with respect to SelectiveNet which, as per authors claim, it is the only existing selective prediction approach suitable for adaptation to time series forecasting settings.\n\n**Weaknesses**\n- Details on how the different models are trained are not complete and, in absence of code for reproducibility, it is difficult to ensure that the bad performance of SelectiveNet is not due to implementation choices.\n- The proposed approach is proposed in the context of univariate forecasting, i.e., the target variable is univariate. However, there is no discussion on the suitability of this approach for multivariate forecasting, where the target variable is multivariate. This is a common setting and it is not clear which is the applicability of the proposed approach for this scenarios in terms of performance and computational complexity. This discussion would be very useful for readers as parts of model training and inference involve iterative optimization or repeated sampling Eqs. (11, 13, 14).\n- The experimental results mostly involve transformer based architectures, however architectures based on temporal convolutions networks and recurrent neural networks are still relevant approaches in time series forecasting. It would be nice to have an idea of the performance of the proposed approach when applied to such architectures.\n\n**Questions**\n- How is too low real coverage penalized when comparing performance of two selection models? Intuitively having an higher real coverage than the target one is not problematic and it is eventually implicitly penalized if the higher coverage is used to let bad predictions slip through. However, having a lower coverage should be penalized in some way as, in the extreme case, a selection method could result in a 0% coverage which technically results in a selective prediction error of 0, if my understanding is correct.\n- Could the bad performance of SelectiveNet stem from the fact that its loss function results in a worse overall predictor? It is not clear if SelectiveNet is added on top of a pre-trained predictor, however, from my understanding, I imagine that SelectiveNet is trained from scratch using a weighted loss comprising an MSE term and a term related to the coverage. In this case the method could be very sensitive to the relative weight of the different components. Could the authors report the performance of the SelectiveNet predictor when the selection output is ignored and all predictions are considered ok? How does it compare to the performance of the corresponding standard supervisedly trained point-predictor?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a framework, called Time-Energy Model (TEM), for selective time series forecasting based on coupling possibly pre-trained point predictors, based on encoder-decoder architectures, with Energy Based Models (EBMs). In this framework the EBM acts as a selector model which selects whether a prediction is trustworthy or not based on a target coverage. To achieve this the authors propose an unsupervised training strategy for the EBM and two inference strategies to sample energy values for a prediction. Moreover, they propose a methodology to achieve different target coverages without need for additional model training.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I would like to see a more precise discussion on the models implementation and training (hyperparameter selection for instance). Moreover, if possible, code used to train the models should be released for reproducibility purposes.\n- A discussion on the suitability of the approach for multivariate forecasting, with experimental results (see Weaknesses).\n- Experiments for non-transformer based architectures would broaden the scope of the proposed approach. Even simple baseline architectures would suffice in my opinion.\n- I would like to see some information on the performance of the SelectiveNet predictor when the selection output is ignored (see Questions), to better understand if the bad performance could be partly explained through an ineherently worse predictor with respect to the standard supervisedly trained baseline.\n- Table captions should clarify if the reported results are the best run of 3 or the mean, and also the standard deviation across runs should be reported.\n- Readability of Table 1 and similars could be improved, for instance by putting the results for each target coverage close together. The caption states that bold results denote best for each target coverage, however it seems more to denote the best selective approach for each model **and** target coverage. If the purpose of the table is to show that TEM is better than others, model and coverage wise, a table in which lines are grouped by model and then by coverage could drastically improve readability and convey better the message. Moreover, it seems that only the selective error is considered, to decide if a model is better than another. This should be stated more clearly in the caption.\n- There were minor problems I found while reading the paper, I summarize them in the following:\n\t- Sec 5.1, Par. 7: It is stated that the average error reduction is between 11.1 and 39% across models, but then that the best average prediction error reduction was 21%. This seems counterintuitive. Does the first phrase refer to average reduction per dataset across models while the second is the overall average? Also in the same paragraph, you both use \"coverage < 50%\" and \"converage in [10%, 30%]\". Given the values used in the experiments, isn't this the same thing? I think, in general, the phrasing for results discussion could be improved to convey them more clearly.\n\t- In the abstract the acronym EBM is used but the term was not introduced yet.\n\t- Sec. 1, Par. 5: There is a question mark, possibly in place of a missing reference.\n\t- Sec. 1, Par. 6: Before (3) either it should be \"challenge (i)\" or there is a missing challenge, maybe the authors meant \"challenges (i) and …\".\n\t- Sec. 3.4, Par. 6: Referencing figure 3 when mentioning \"step 1\" and \"step 2\" would make it clearer to which steps the authors are referring to.\n\t- Sec. 5.1, Par. 7. The last phrase should be \"provides up to\", or \"provides on average\", regarding the reduction error of optimization based selection.\n\t- Sec. 5.3, Par. 1 and Figure 4: target coverage 100% is mentioned, however I do not see it in the figure. Is that a typo? I imagine target coverage 100% to be meaningless in this setting.\n\t- Sec. 5.1, Par. 7 and Sec. 5.4 seem to make conflicting statements about which energy inference method performs best.\n\t- A.1.1: Here it is stated that the energy optimization method achieves better prediction error of 1.9%, on average, over the aggregation method, while in Sec. 5.4, 10.9% is reported. I imagine this to be a typo, however, many average percentages are reported throughout the paper, focusing on different groupings of coverages and models. This might be confusing and I encourage the authors to find wordings that make it clear on which groupings of models and datasets these averages are computed.", "broader_impact_concerns": "I don't have concerns regarding the broader impact of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nnlCi9rb8i", "reviewer": "Reviewer_WaXh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n1. Important Problem\nI think this paper proposes an important problem that has not received much attention but has great meaning.\nWhile most research in time-series forecasting focuses on deterministic prediction, this paper aims to address probabilistic forecasting and selective forecasting. In the real world, a probabilistic problem is also important because it requires measuring the confidence of predictions, which helps users determine if they believe the predictions of models. Also, selective forecasting makes the decision process more sysmatically. I believe that although various works for probabilistic forecasting have already been introduced, selective forecasting is a somewhat newly proposed problem in time-series forecasting.\n\n2. Model-Agnostic Methods\nIt is not a standalone model but one that can be applied agnostically to other existing models. In spite of the fact that their method can be applied to forecasting models with encoder-decoder structures, it is still meaningful in that many forecasting methods employ encoder-decoder structures. \n\n# Weaknesses\n\n## Major Issue\n\n1. As stated in the manuscript, energy $E_\\theta$ cannot be directly used for comparison of different $X$, because it is not normalized. To overcome this, the authors propose Aggregated energy inference or Energy optimization inference. I think the detailed descriptions of why these inference techniques can overcome unnormalized characteristics. Also, can you provide the reason for experimental results in the ablation studies where a naive energy method and their proposed energy methods are applied? In other words, I'm wondering why the proposed inference methods outperform the case where energy values are used naively.\n\n2. In general, a small  $E_\\theta$  indicates high probability, while a large  $E_\\theta$  suggests low probability. In other words, the order of energy also carries important meaning. However, in step 2 of selective forecasting with TEM, all energy intervals are ranked based on forecasting error, which may disrupt the natural ordering of energy values and potentially affect the interpretability of the model’s energy-based predictions.\n\n3. Joint training is one of the key contributions highlighted by the authors. However, the lack of detailed explanations makes it difficult to understand the specific characteristics of their joint training approach. Could you clarify the differences between using joint training and not using it, as well as its application when leveraging pre-trained deterministic forecasting models?\n\n4. The authors highlight that existing SelectiveNet models do not allow predictions to be rejected based on criteria such as estimated prediction error, whereas their approach enables selection or rejection of forecasts based on such criteria. To support this claim, I believe two points need to be addressed: first, a detailed explanation of the procedure when using other criteria, and second, an outline of the unique features or novel modules that make this capability possible.\n\n5. Selective forecasting is inherently a form of probabilistic forecasting. Therefore, I believe this paper should incorporate recent probabilistic forecasting models that use a straightforward inference technique, where multiple predictions are sampled, and their variance is utilized as a measure of confidence (or as the criterion for rejection). [1,2,3]\n\n## Minor Issue\n\n1. Using Equation (11), negative samples are generated. Due to lack of my knowledge in training energy-based models with negative samples, please provide more details of justification for this method. Specifically, Equation 11 results in $Y^{(1)}$ with lower $E$ which means high probabilities. Then, can $Y^{(1)}$ from Langevin dynamics be regarded as positive samples?\n\n2. There are a few typos in the manuscript, such as the use of ? in place of citations and the incorrect notation $\\theta xy$, which should be corrected to $\\theta_{xy}$.\n\n3. There is no paragraph which refers to Figure 4.\n\n4. I believe it would be beneficial to further propose a method for refining rejected predictions, enabling them to be corrected and ultimately accepted. Although the problem area defined in this paper is constrained to rejecting well. However, I think the extension to correction is necessary for real-world scenarios.\n\n5. Are there any advanced evaluation metrics for selective forecasting? I think the evaluation metrics should measure not only how well models forecast but also the alignment between test forecasting errors and the rank of energy intervals, which means as the rank changes, test forecasting errors monotonically increase or decrease. Table 1 and Figure 4 indirectly show this alignment, but a more direct way to measure the degree of alignment or misalignment is required. Based on the existing literature on selective predictions, concerns 4 and 5 in **Minor Issue** may not necessarily need to be addressed. However, I believe these concerns become significant when considering real-world applications.\n\n[1] Tashiro et al., CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation, 2021  \n[2] Kollovieh et al., Predict, Refine, Synthesize: Self-Guiding Diffusion Models for Probabilistic Time Series Forecasting, 2023  \n[3] Rasul et al., Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting, 2021", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Due to the importance of time-series forecasting in real-world applications, various methods have been developed. However, most of recent methods are constrained to deterministic forecasting, which means that they cannot measure the confidence of their prediction, leading to low utilization in real-world scenarios. To overcome this limitation, this paper proposes selective time-series forecasting with a Time-Energy Model (TEM). Compared to existing not only deterministic models but also models for selective prediction, their method has several distinct characteristics that contribute to improved performance. First, they combine time series forecasting models and energy-based models into a single framework with joint parameterization and training. Second, their method can be applied to any time-series forecasting model with encoder-decoder structures. Finally, they introduce a scalable selective forecasting procedure that narrows the search space of time-series data, the size of which extremely increases according to time length. In the experiments, through selective forecasting, TEM achieves up to a 49.1% reduction in prediction error compared to five leading deterministic models. Additionally, TEM demonstrates up to 87.0% lower error than baseline EBM models and significantly outperforms state-of-the-art selective deep learning models.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Addressing the **Major Issue** outlined in the Strengths and Weaknesses section is essential for this paper to be considered acceptable.\n\nAs for the **Minor Issue**, they do not critically weaken the paper but addressing them would enhance its quality. In particular, resolving concerns 4 and 5 could significantly improve the overall impact of the paper.", "broader_impact_concerns": "Can you include the broader impact of this paper in a separate section?", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lb6zWERtVs", "reviewer": "Reviewer_58WR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1. The framework proposed in the paper does a good job of extending selective prediction to time-series data\n2. The problem setup, architecture, and forecasting procedure are all clearly explained (Figure 1 does a particularly good job of summarizing all this information)\n3. Unlike SelectiveNet, the target coverage can be changed without re-training the network allowing for flexible risk-coverage trade-off during run-time\n\n\n**Weaknesses:**\n\nUnfortunately, the experimental results don't appear to be very convincing. As I understand from Table 1, there doesn't appear to be too much of a lift using TEM over using the deterministic baseline models thereby negating the need to do selective forecasting in the first place. To make the results more compact and easier to read, I feel that it might be a good idea to compare area under the risk-coverage curve against baseline risk instead of the individual risk for every coverage level. This would give the reader a quick and easy way to compare the selective prediction models against the deterministic baselines. In addition, it might be helpful for the readers to have some more information on how SelectiveNet was adapted to accommodate time-series data.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose a selective prediction framework called the Time Energy Model (TEM) for time-series forecasting that uses an energy-based model (EBM) to model the conditional probability of $Y$ given $X$ where $Y$ is the predicted forecast and $X$ is the time series. Section 2 introduces some basic preliminaries and sets up the problem formulation. Section 3 dives into the details of the framework and has separate sub-sections explaining the architecture, training, and forecasting procedures. Section 4 details a set of experiments on five benchmark datasets to test the efficacy of $TEM$ against $SelectiveNet$ - a state of the art selective prediction framework for classification/regression.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Although the framework is both interesting and novel, the experiments section doesn't seem to make a convincing case of using TEM over just using the deterministic baselines. It could be the case that forecasting on the datasets used in the paper do not benefit much from selective prediction and thus do not do a good job of highlighting the strengths of TEM. With that in mind, it would be difficult to accept this paper unless TEM's utility can be empirically demonstrated where there is a clear advantage of using a selective forecasting framework like TEM over a deterministic model.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a time-series forecasting framework based on energy-based models, capable of estimating prediction error and rejecting predictions using energy functions. While one reviewer remains negative after the discussion, the majority of reviewers, along with the Action Editor (AE), agree that the claims in the submission are well-supported by clear evidence. The AE believes that researchers will find this work interesting, and practitioners will see value in applying this approach to real-world scenarios. Therefore, the AE recommends accepting the paper.
Decision: Accept (poster)