Towards Measuring Predictability: To which extent data-driven approaches can extract deterministic relations from data exemplified with time series prediction and classification
Authors: Saleh GHOLAM ZADEH
Vaisakh Shaj
Patrick Jahnke
Gerhard Neumann
Tim Breitenbach
Venue: Accepted by TMLR ()

Abstract: Minimizing loss functions is one important ingredient for machine learning to fit parameters such that the machine learning models extract relations hidden in the data. The smaller the loss function value on various splittings of a dataset, the better the machine learning model is assumed to perform. However, datasets are usually generated by dynamics consisting of deterministic components, where relations are clearly defined and consequently learnable, as well as stochastic parts where outcomes are random and thus not predictable. Depending on the amplitude of the deterministic and stochastic processes, the best achievable loss function value varies and is usually not known in real data science scenarios.
In this research, a statistical framework is developed that provides measures to address the predictability of a target given the available input data and, after training a machine learning model, how much of the deterministic relations have been missed by the model. Consequently, the presented framework allows to differentiate model errors into unpredictable parts regarding the given input and a systematic miss of deterministic relations. The work extends the definition of model success or failure as well as the convergence of a training process. Moreover, it is demonstrated how such measures can enrich the procedure of model training.
The framework is showcased with time series data on different synthetic and real-world datasets.
The code is available at https://github.com/Saleh-Gholam-Zadeh/predictability_measure.

Published in Transactions on Machine Learning Research (02/2024)
Towards Measuring Predictability: To which extent data- driven approaches can extract deterministic relations from data exemplified with time series prediction and classifica- tion
Saleh Gholam Zadeh
Autonomous Learning Robots, Karlsruhe Institute of Technology
SAP SE
Vaisakh Shaj
wiiskkss.shajj@gmakk com
University Of Edinburgh
Patrick Jahnke Turba Al
pj@turbo.c ai
Gerhard Neumann
gerhard.neumannn Bkit edu
Autonomous Learning Robots, Karlsruhe Institute of Technology
Tim Breitenbach
tim. hin.bbreiennccckkkkkuvueerrarr
SAP SE
Universsitt Würzbury
Reviewed on OpenReview: https: // openreview. /ttpee///peereveeeu net/ forum? id= (.B.AAAAUUUU VFCUUO
Abstract
Minimizing loss functions is one important ingredient for machine learning parameters such (hat the machine learning models extract relations hidden in the dala. The smaller the loss function value various splittings of dataset, the better the machine learning model is assumed to perform. However, datasets are usually generated by dynamics consisting of deterministic components, where relations are clearly defined and consequently learnable, as well as stochastic parts where outcomes are random and thus not predictable. Depending on the amplitude of the deterministic and stochastic processes, the best achievable loss function value varios and is usually not known in real data science scenarios. In this research, statistical framework is developed that provides measures to address the predictability of target given the available input data and, after training machine learning model, how much of the deterministic relations have been missed by the model. Consequently, the presented framework allows to differentiate model errors into umpredictable parts regarding the given input and a systematic miss of deterministic relations. The work extends the definition of model succoss or failure as well as the convergence of a training process. Morcover, is demonstrated how such measures can enrich the procedure of model training. The framework is showcased with time series data on different synthetic and real-world datasets. Tadch//cedictabillty measure.
The code is available at https://gittab..co//acchhhhCihh

---

Published in Transactions on Machine Learning Research (02/2024)
1 Introduction
Data analysis and the application of the corresponding insights work if there reliable and stable relations belween the measured quantilies and their model. However, due lo the fact that any measurement may not be error free or the dynamics that determine the values of the considered quantities may be inherently noisy to a certain extent, measurement values do not only purely reflect the relations to be investigated. Furthermore, the input data could miss relevant information, e.g., relevant features are not provided or even measured, to model the output data such that regarding the given input data some parts of the output are unpredictable due to the lack of information. One example is that past stock prices may not always provide clear indication of future prices due to the complexity of market dynamics and the influence of external factors such as economic events and investor sentiments. Therefore, accurately predicting future stock prices might not be directly evident from historical observations alone. Without having criterion for the amount of predictable relations between both the input and output or model deviations, respectively, in such scenarios, one could spend a huge amount of time on learning dynamics which either don't exist (such as pure noise) or it is impossible to learn more than a certain level because the given history is sharing no or low information with the output. Usually, real-world datasets are not binary in terms of being predictable, meaning that there are some deterministic patterns plus patterns which can't be inferred from the provided historic data, as might be the case in the example from the financial markets above, and therefore there is no chance for any model to prodict them totally accurately. Depending on the relative magnitude of these elements, the prediction error can vary even for a successful model. In such cases, it is beneficial to know the upper bound of the model's performance to avoid trying to train it at all where there is no deterministic relation or to improve it while further improvement is not possible. We extend the term for model success by its ability to extract or learn, respectively, all the available deterministic relations. Consequently, we argue that the magnitude of prediction errors alone does not unequivocally signify the model success failure because high inaccuracy might arise from the umpredictability of the data, rather than from deficiencies in the model or flaws in the learning process.
One key issue before any data analysis, machine learning (MI.) model training or information extraction from a datasct is testing whether such reliable relations exist between the quantitics in the datasct of interest, as outlined in the example above. Such tests deliver valuable insights to evaluate efforts for further analysis, and if reasonable all. A second key aspect after an iteration of data analysis or information extraction, such as the training of an MI. model, is testing if there is information left to extract or all the reliable information is extracted in terms of deterministic relations between input and output data. If all relations are extracted or learned, respectively, the deviations between the predictions given the input data and the target (ground truth) are supposed to be stochastically independent of the input data. Thus, given that input, additional analysis on this input-target relation might not reveal further insights or improve accuracy in terms of extracting more deterministic relations. Consequently, further training might not improve a model in terms of learning these deterministic relations.
Following the outline above, in the presented work, we provide framework to address the following, which is illustrated in Figure] [
We utilize measures to evaluate the stochastic dependence and information content between random variables modeling the data to evaluate to what extent the data is interconnected and to quantify extractable information based on defined input and output (target) variables assembled from the dataset.
After fitting a model, we use these measures to estimate if there is still information left to extract. This evaluation is done by testing the stochastic independence and information content between the input and the deviations of the model output regarding the ground truth. For this purpose, we investigate the relation of random variables that model corresponding quantities.
These bullet points provide the foundation of the presented framework for charactorizing failure cases in prediction modeling. To provide an implementation of practical framework and to give evidence with some showcases for its usability is the main purpose of this work. In our work, failure case is defined as
2

---

Published in Transactions on Machine Learning Research (02/2024)
Investigate the data
Is the target predictable from the input?
Yes
No
Refine data set or
Train ML model
STOP
Adapt model
Yes
Determine the deviation between model prediction and target
Done!
Are the deviations predictable from the input?
No
Model extracts
all information
Figure 1: Graphical abstract representing the main concept of the presented work to analyze for information content and information extracted or learned, respectively, by model.
scenario in which a prediction model fails to capture the essential, meaning deterministic, dynamics of the target variable(s), given the input information. This definition of failure allows us decide on whether there is potential for improvement the prediction model has already achieved the best possible scenario, where inaccuracies are not further predictable and thus not improvable, given the input data. Our mathematical framework provides an additional insight for model evaluation and extends the current performance measures of ML models considering not only the ground truth and the model output but also the input and thus to what degree the ground truth is predictable given the input. This extension aligns with the perception in the area of generative models such as GAN Yoon et al. (2019); Jeha et al. (2022) or diffusion models Kollovieh et al. (2024): Tashiro et al.] (2021): ;: Li et al. (2022); Yuan Qiao (2024) where the discriminative score, the generative score, the Frechet Inception Distance (FID), the Maximum Mean Discrepency (MMD), the Negative Log Likelihood (NLL) or the Contimuuus Ranked Probability Score (CRPS) is also reported besides the standard evaluation framework of mean squared or absolute error (MSE/MAE) to have a more probabilistic view by taking distributions of predictions into account. One contribution of the presented work is to extend the term of model assessment beyond the model accuracy based on its output and ground truth to also laking the predictability of the deviation between model output and ground truth into account given the input. Moreover, we provide also numerical implementations of such measures and show its applicability on synthetic and real-worid data sets. Through the numerical experiments, we exemplify determining if the deviation is improvable at all given the input data.
One benefit of our metries, which are based on the stochastic independence of input and model deviations, is that they can provide an explanation for poor model accuracy. The explanation is given by analyzing the deviations if there is information left or deviations consist only of unpredictable parts from the perspective of the input data. The ability to differentiate what data points are predictable and what are not given the input data is not provided by accuracy measures or loss functions that only consider the model output and ground truth, for instance mean squared or absolute errors. Instead, we additionally consider the information the model deviations share with the input data. The framework is not limited to specific model architecture and thus is model-agnostic since we only inspect the dependence of input and the model errors, which does not require the knowledge of the inner function of the model.
The implementation of this general framework provided with this work is the mutual information, the chi- square test of independence and the Pearson correlation investigate the existence of relations between random variables. However, our framework is not limited these tests. Consequently, other methods that test for information content stochastic dependence can be used and easily included into our modularized
3

---

Published in Transactions on Machine Learning Research (02/2024)
git repository. The choice of the measures and their implementations themselves are not the focus of this work.
Evaluating the deviations for dependence on the input can be applied to facilitate model training particularly as follows. The first three bullet points are elaborated in Sections 4.1 bullet points are addressed in Section 4.5
4.2
4.3 and 4.4
while the last two
Success criterion: We provide loss function agnostie framework to evaluate the performance of ML accuracy defined by the quantity of extracted deterministic relations, which can be used model selection criterion. Furthermore, we can decide if a loss function value is a corresponding lower bound for the given dataset since no further deterministic relations remain within the dataset.
Further convergence criterion: Apart from utilizing the loss function for convergence, we can stop training whenever there is no information or dependence left between the input and the deviation of a model prediction from the ground truth.
Hyperparameter tuning: We can stop a grid search whenever we have identified a hyperparameter set with which the corresponding model has extracted all the information on the training or validation set.
Distribution shift detection: A data splitting into training and validation set or historical and present can result in different stochastic properties, therefore resulting in different learnable relations in the scts.
Data efficiency: In case of a closed system where distribution shift can be excluded, e.g., the dynamics are known and fixod, the total amount of training data can be used for fitting model parameters since no data is required to be utilized as an additional set for early stopping.
In Section present examples demonstrating the utility of our framework for issues described in the bullet points above. Among different data modalities such as text, image, and time series, our insight is that in time series data, the existence of a clear relationship between input and output is often not directly noticeable. Therefore, in this work, we focus specifically on time series analysis. For instance, in natural language processing (NLP), we typically do not encounter a sequence of random words or tokens. Similarly, in image datasets, depending on the task (e.g., classification, segmentation), the relationship between input and output usually exists a priori, and one would not typically spend time proving such existence, instead would directly train a model to solve the task at hand. However, in time series analysis, the underlying dynamics of a process need to be learned from the measurements, For this purpose, we showcase our general framework with an application to time series forecasting. The basic procedure is the following. After quantifying the level of predictability by inspecting the input and ground truth target, we can perform an exactly similar test on the input and the residuals, which define the deviations as ground truth minus prediction, to see whether they are still predictable given the input. We use supervised ML techniques to predict future parts of the time scries. Another use case is the application for time scries classification with nominal target data.
This use case differs from the ordinal output data above as model deviations cannot be defined by differences due to the lack of order. A corresponding concept is presented and its usability to quantify left information is showcased in this work. One further example of unpredictable time series might be the prediction of workloads, mainly network traffic in data centers, where time points of the triggering of the workload by users might be unpredictable. In Greenberg et al. (2009), we find: "measurement studies found two key results with implications for the network design. First, the traffic patterns inside a data center are highly divergent..., and they change rapidly and unpredictably." " A more recent manifesto by Buyya et al. (2018) re-iterated the brittleness of existing "demand estimation and workload prediction methods" ", leaving it an open question if "Machine Learning (ML) and Artificial IntelligenceAA)) methods could fully address this shortcoming". With our present work, we can even provide a quantified answer in each case.
Paper outline The work is organized as follows: The main theoretical concepts used in the proposed franework are explained in Section 2 This includes a precise definition of the mcasure of mutual information and the chi-square test of independence, as well as an equinumeric discretization scheme for features with a continuous co-domain.
4

---

Published in Transactions on Machine Learning Research (02/2024)
A section of related methods follows Section 2 and addresses among others how our work extends related works using mutual information, ensemble learning and time series prediction.
Applications of the information extraction evaluation of a model are showcased in Section 4 with different experiments in the area of time series prediction and classification. These experiments include a basic proof of concept, an analysis of the influence of the loss function on the information extraction depending on the structure of the noise, suggest additional convergence criteria based on the presented framework, as well detect distribution shifts of the data.
In the Discussion, we provide assumptions and limitations of our current approach. Furthermore, we sketch further applications of our framework in the area of unstructured data. Moreover, efficient model size reduction is also discussed by ranking subparts of a model with stochastic measures since the presented framework is general and can be applied to any function generating output from input data.
2
Theoretical background and Methods
In this section, we provide the necessary background and mathematically establish our framework.
2.1 Basic concept for unpredictability in a nutshell
Definition of unpredictability
The random variable Y is deemed unpredictable with respect to information set given by the random variable X if the conditional distribution P(Y = y|X x) aligns with the unconditional distribution
P(Y y) according to
P(Y y/x = z)) P =)
(1)
for all x and y. Specifically, if X comprises the past realization of Y, then (1) suggest that having knowledge about these past realizations does not enhance the predictive accuracy Y. Tt is important to note that this form of unpredictability in Y is an inherent attribute, unrelated to any prediction algorithm (eerrithmBeebochinn [Bezbochia) let al. (2023). One famous example of unpredictable time series is white noise where samples are identically but independently distributed (iid), where the independence of future samples from the past samples makes it essentially unpredictable and therefore training an ML model pointless. In this case, the best predictor in terms of Lo=loss is a mean predictor, suggesting further investing on improving the prediction model is fruitless. Please note that while umpredictable data and noisy data are related, they are not equivalent. Noise represents specific subset of unpredictability.
On the other hand, if I doesn't hold true, it suggests that given X, it is reasonable to train an ML model to predict Y. The more the distributions of the two sides of ( deviate from each other, the more chance we have to train a model with a high accuracy to predict Y based on X. In the scope of this work, we call X the context/ /input variable and Y the target target/output /output that we want to predict. In this work, we use the terms input and context interchangeably, analogously the terms output and target. In the following part., explain how to measure and quantify the concept we have introduced so far.
Mcasuring predictability
Assuming is satisfied, we can derive the joint distribution by
P(Y
= P
X x) 9)
(2)
for all x and y by utilizing the definition of conditional probability. However, in practical data science scenarios (2) barely holds entirely true even in case of independence of X and Y due to noise numerical errors. Therefore, it is crucial to quantify the degree to which the independence assumption is met and to introduce statistical concepts to decide based on a level of significance if the hypothesis of independence cannot be rejected.
Our approach to quantifying the independence of these random variables grounded in (2), where we measure the deviation between its left and right-hand side. Although in general any measure of deviation can be used, in this work, mainly focus on Kullback-Leiin divergence and chi-square test of independence
5

---

Published in Transactions on Machine Learning Research (02/2024)
measure of this deviation. In the former case, such deviation can be calculated based the mutual information formula given by
T(X:Y) PRI (xx,(Pxx @ Py)
where DKL is the Kullback-Leible divergence, Px @ Py denotes the outer product distribution, and P(x,y) is the joint distribution (see Murphy (2022) Section 2.2.4 Figure 2.3). A higher value mutual information represents higher predictability of Y based on X which aligns with the definition of mutual information, measuring the information gained about Y through observation of X.
Quantifying measure of success
We define a successful prediction when the deviations between model output and ground truth, e.g., the resid- ual defined as the ground truth minus the prediction, are stochastically independent of the context/input information. In case of independence, the deviations contain no information shared with the context, ren- dering them effectively unpredictable based on the provided context information. Consequently, no further improvements are possible, marking the prediction as successful. It is crucial to note that as long as some mu- tual information remains, there is potential for enhancements, whether through selecting a different model, loss function, adjusting various parameters. For the sake of assessment, lower values of mutual information suggest a better prediction quality.
We remark that the scope of this work does not include the development of new methods to calculate, estimate or approximate mutual information but the application of such methods to improve ML training. In the Related Work section, we provide references to such methods. For the demonstration of the application, we provide one implementation that worked for our experiments. Our framework is flexible and does not depend on a specific method to quantify stochastic independence, allowing for various measures such mutual information. The following sections provide a rigorous mathematical formulation and details on the framework's implementation
2.2 Foundations and implementation details for the predictability framework
In this section, explain our framework in detail.
Measures for stochastic independence and information content
We are given n E N input features in the format x E R" and m E N output features in the format y E R". Each feature is modeled as random variable taking values upon measurement. Consequently, the set of input random variable given by X (I1, Fn), F3 : R Zz., t + x; (t), E {1,..., n} where t is a time point of measurement and Zz. = k=......... la E N, set of discrete events. In our case, such event is defined through the random variable 2; taking a value at time point t between two predefined boundaries (see Figure and 10 in the Appendix). In this work, these boundaries are calculated by an adaptive scheme presented in Algorithm [
Analogously, the set of output random variables is defined by Y (y1...., Vm ) Bj : R R- Zyy, yj (t), j E {I....mm} and Zy = {2t)) J k=I....,,, Ly E N, a set of discrete events. Our framework not only holds for random variables with a discrete co-domain but also for random variables with a contimuous co- domain. In our case, we choose to discrctize continuous domains and we will later explain Algorithm [ that discretizes random variables with a continuous co-domain equinumerically. Furthermore, independent of the topology of the co-domain, we define information or dependence measure by $ X x Y R, (I1, In. V1,,,,,)) B..) + & (41, V1.... Vm) that describes how much information, resp., stochastic dependence exists between the input and the output variables. An example for such measure can be the stochastic independence of multiple real-valued random variables, see, e.g., (Gallager 2013 1.3.4) or mutual
information as outlined in Subsection 2.1
In this work, we focus on a specific structure of 4, which a pairwise test between input and output random variables providing corresponding information stochastic dependence summing up each value of
6

---

Published in Transactions on Machine Learning Research (02/2024)
the pairwise measure. The measure & is given by
m
t (#1:--, ons y1, by vm) =
@ (3i, Uj)
f=1
where Φ: X X +R, (1), +> Φ (2i Uj) for each E (I.....m) and j E (m....,m.
We are aware that pairwise test might be an approximation of the actual value of the mcasure for the deterministic relations, e.g., as in the case of the stochastic independence of multiple real-valued random variables. However, this approximation provides the advantage of much less computational costs, in particu- lar for large n and m as provided in the discussion section of Breitenbach et al. (2022). It is one outcome of this work that this approximation is a useful measure to estimate the deterministic relations between input and output as well as model deviations between predictions and ground truth, moreover to estimate the learning success of a model which the reduction of deterministic relations between the input dataset and the model deviations from the ground truth. A similar consideration holds for the mutual information where precise calculation of the joint probability can be very costly in case of many input and output variables and where other mutual information estimators cxist as well to circumvent this issue, plcase see the related work and the discussion section for references and further details about this issue.
In the present work, we focus on the mutual information and the chi-square test of independence between two random variables as measures, which are both explained later in detail. However, the presented work is generic and can also be executed with different measures for independence, like Pearson's correlation coefficient defined in, e.g., [Breitenbach testing hypothesis if input and model deviations are independent of each other, it is beneficial to rely on several tests to be sure about the consistency of the results and not having a wrong decision because requirements of test are not fulfilled, which is often challenging to verify.
al. (2023) for random variables. We remark that in terms of
In the following part, we explain the main ingredients of the present framework to analyze for deterministic relations. Although these concepts might be well-known, we repeat them here for the convenience of the reader since they are central for this work and precise definition consistent with this work facilitates its understanding.
Mutual information The probability P (xz = =b)) describes the likelihood that the outcome of x; equals the event z for any E {1,...,]) n) and any k E {1,....,a } Analogously for P Uj = = for any j E {1,...,m) m} and k2 E (1,.... b
The probability P [2 2 Ag = =53)) describes the likelihood that the outcome of I, equals the event 2h and the outcome of y5 equals the event =/ for any E (I,...,n, j E (1,...,m, kn (2, and 2z E (1,. Based on this definition, we can define the mutual information for a pair of random variables 2, and yj as one example for & as follows
P
231 It
A Uj
= 2
T (I: 11) ==
J; = As
= Aリ
Azz
loga
k2=1
P
I,
=
P
12 243
for any i e {I...., nn and j E {L,..., m) with the basis a E N \ describes how much information we gain about the outcome of yj given the outcome of x4. If the outcome of iu independent of yj, namely
{1}
of the logarithm. Mutual information
P
Fi
ki Ex
A Ui
Av =
P =11) P P(. b
1y =/0)
P
yj
kg Zyz
for all K1 E {1,,..,aa)} and b E (I,,..,, Les where P
p(xx (x;= kk by =
is the conditional probability that
x; = zh under the condition that y 25; we expect zero mutual information since log, 1 = 0.
7

---

Published in Transactions on Machine Learning Research (02/2024)
The mutual information is bounded from below by O. The upper bound depends on the number of events of Vj. In order to normalize the mutual information such that it is bounded from above by for any Vj, we define the corresponding log base a = 4 We remark that in case where yj is replaced by the corresponding model deviation, the basis defined accordingly to the number different events of the model deviation from the ground truth. The normalization in particular important to compare the information content between z; and yj with the left information content between I; and the corresponding deviation between model output and ground truth after the training of the ML model to estimate the information extraction of the model from the dataset.
As next example, we introduce the chi-square test of independence of two random variables.
Chi-square test of independence
In case the chi-square test of independence is taken as the measure for stochastic independence, then we define this work that returns if the corresponding random variables of the pair are not independent of each other, else 0.
Let us have N E N measurements where at each measurement the values of all random variables deter- mined. For any fixed i E [1.....n] n} and j {1...., m), we define
P( p ki A = = k2
Ok N
where Okika E N is the number of observed measurements where xi =x, and yj z for the corresponding k E {l,..., 1,:) and 22 E (1,....p, J. Then, we have
Ing Oka N
Ok and P (リ3 #/ N
P ((:: k1)
with N -by Eni =1 O2g Under the hypothesis that the random variables x; and Mjs i E (l and j {I,... 1 are stochastically independent, the number of expected measurements Ex, E where 24 2x and yj 2j
k2 is given by
ERIK2 = P (xx; =t:) P yjj =3 N
for the corresponding k E {1,.... and 22 E {1...., b,, } as we can take the corresponding proportions from N independent. of the outcome of the other random variable. Consequently, m; and Yo are independent, it necessary that the observed and the expected number of measurements for all kj E (1....,a:)) and 2 E (1,... ] each equal. The chi-square statistic given by
by
(Oks E
(4)
E
A 1y=1ks= 1s=1
equals zero if Okik, and kx, equal each other for all k1 E {L, I and k2 E {1,....(y, deviation from not being equal. However, due the presence of noiso, even under independence of x; and y/- it might hold that (Ox,h² )² > 0 for some k E (1,...,1z.) and hy E (1....,w,,} Consequently, we need to estimate from a distribution how likely the observed chi-square value under the hypothesis of independence is. the observed value is too unlikely, we rather assume that the opposite our hypothesis of independence is the case, meaning the variables depend on each other and there exists a dependence between the random variables in taking their values. It. can be proven that x² is chi-square distributed with (hr, 1) (hy, - 1) degrees of freedom, see, e.g., [Rao 1973 6d.2) or (Georgii 2015 11.3). One important assumption is that the term Ox, - Feyt2 is approximately normally distributed, which is usually sufficiently the case if E k2 2 5 /EA, for all ly (I,..., l and l2 E (1....,y,, J. see, e.g., McHugh (0033) or (Greenwood & Nikulin 1996 page 21). Based the distribution, we calculate p-value for the observed x value. The p-value is the
b, } and quantifies the
8

---

Published in Transactions on Machine Learning Research (02/2024)
probability get higher chi-square value than the observed one under the hypothesis of independence. If the p-value is too small, e.g., for this work we use lower than the level of significance of 0.01, we reject the hypothesis of independence and assume that the random variables take their values not independent of each other. Our mcasure of dependence is the number of chi-square tests that indicate dependence of the tested pairs for the input and output variables (1)) for each i E (1....,)) nn and j e (1,....m. However, since the number of chi-square tests is given by nm, which can scale large numbers, we use an adapted p-value lower the risk of wrongly rejected hypothesis, which would result in assuming too often dependence. We use the Bonferroni-correction dividing our level of significance 0.01 by the number of chi-square tests mn.
Discretization scheme for co-domains of random variables
In the next part, we describe how to discretize real-valued continuous random variables, meaning that the co-domain is continuous. We take Algorithm 4 from Breitenbach et al. (2022). We remark that our algo- rithm also works for real-valued discrete random variables without any change. Consequently, no separation between discretized and contimuous random variables is necessary. The algorithm provides an adaptive dis- crelization scheme for each random variable, meaning that boundaries of the bins, in which the co-domain of random variable is divided and define the events denoted with 2 or 25g are not set equidistant but rather equinumeric. With equinumeric, we mean that cach bin has if possible the same number of data points which balances the likelihood of each event.
Algorithm Discretize the co-domain of random variables 1. Set p E N number of minimum data points per bin.
2. For any random variable o:
(a) Determine the minimum value m and the maximum value M of all measured values of v.
(b) If M S m. Skip v
(c) If M > m:
i. Sort the measured data points of e in ascending order.
ii Go through the data points in ascending order. Determine the range of a bin such that there are at least p data points within the current bin and that the value of the last data point of the current bin is smaller than the first one of the next bin.
ini If there are less than data points left: Join these data points with the last bin that has at least P data points.
iv. Output: Upper and lower bound of cach bin defining the events a for l E 41.
Algorithm works as follows. The parameter p E N determines the minimum number of data points per bin in which the co-domain of random variable, such as i; or yj, is discretized. These upper and lower bounds of each bin define the event 2 meaning that the random variable x; takes value within the corresponding Eth bin. Analogously, it holds for the events 2 of the random variable yj- Thus,
the parameter p inffuences the marginal probability P P(x; = and P ((y) = of the joint probability
P (m,=>h yリ = k2 and consequently the corresponding expected frequency Fx, ka as well as the number of bins for each random variable. Increasing p will increase the quantity E which might be useful if E < 5 for one kj E (1,..., 1,,) and one k E (1...., Consequently, this discretization scheme is beneficial for the implemented chi-square test and can be used without any restriction for the calculation of the mutual information as well. We need to keep in mind that too big p might lead to a too coarse discretization deleting information from the continuous random variable. In case of the chi-square test, one strategy might be start with a small p where E < for one by E (1,...,lz,) and one k2 (I..... } and increase p until Es 2 5 for all kj E {l....,4,, I. and all h/ E {,,...,y,,) A further advantage of that scheme that by cnsuring always a minimum number of data points in cach bin of cach marginal distribution, the denominator in the mutual information and chi-square formula is never zero which could happen with an equidistant discretization strategy.
9

---

Published in Transactions on Machine Learning Research (02/2024)
In step 2 a), we determine the minimum and the maximum of the available data points of the corresponding random variable to filter out constant random variables in 2 b). Without any variation, constant random variables do not provide any information for predicting something for what at least two different kind of events are necessary. For non-constant random variables, we sort the data points of the random variable in ascending order. Once this is done, we can go through the points in ascending order and determine the boundaries of the bins such that at least p data points are included in a bin. If there are several data points with equal values, we include all these points into the current bin such that the first data point in the next bin is larger than all data points in the bin before. If the remaining data points, which have not yet been associated to a bin, are less than p data points, we include them into the bin with the largest upper bound.
The binning generated by Algorithm | can be used for the chi-square test and the mutual information to calculate corresponding marginal and joint probabilities as we do in our implementation provided with this work. The marginal and joint probabilities of (3] and 4 are calculated as follows. We take the froquency of each event 2/1 ktz and 2x1 2 and divide by N to obtain the corresponding probabilities
Zyz
P(z,- = = P U/ ((1 and jE
2/ d2) and P
k1 22,
yu = > for all l E {1...., 4. J. k2 E {{,.... Lys J.ie [l....n}}
24
In Appendix A the process is illustrated with time series data.
Stochastic independence as a hypothesis test
We conclude this section with a remark about the distribution of test statistics. Even under the hypothesis of independent data/random variables, there are some variations by coincident that lead to distribution of the test statistic. Consequently, random variables (no deterministic relation between input and output) to get a test statistic value bigger than the observed one, and thus exclude that the relation in the measured data is just by coincidence. Then, we can decide if a certain observed test statistic is too unlikely under the assumption of independent random variables, and we should rather assume that the opposite is true, meaning that there are some deterministic relations by whose effect random variables do not take their values independent of each other. Based on such a statistical view, all models where it cannot be rejected that their deviations from the ground truth are independent of the imput are equally good in terms of extracting the deterministic relations.
need probability how likely it is under the assumption of independent
The chi-square value x² is chi-square distributed with corresponding degrees of freedom under certain as- sumptions. Consequently, we can evaluate the likelihood for the observed chi-square value of being generated by two independent random variables. This likelihood can be calculated for each pair of random variables separately as in ( which results in mn decisions. However, when considering cach term of (4) as a random variable where (4.-1) 1) (by - 1) of them take their value freely for each {1.....n] n and j E {I..... m]. rosulting in the corresponding degree of freedom under which we evaluate the likelihood of the observed x² we can add the chi-square value of all random variables together as follows
(OKika Exg )²
x2
(5)
ERI
k2=1
The likelihood of x² defined in 5 under the hypothesis of independent input and output variables can be evaluated as sum of Σ ΣØ= (Le 1) (ly, 1) freely varying normalized random variables, defining the degree of freedom for the corresponding chi-square distribution. The value of x² defined in (5) can be used as a measure for stochastic independence between input and output as an alternative to the number of stochastic independent. input and output pairs as described in the context of The difference is in defining the function Φb
We are not aware of theoretical distribution for the mutual information defined in 3 under the hypothesis of independent variables. However, to get threshold for mutual information, such that we can decide based on a level of significance if an observed mutual information value is too unlikely under the assumption of independent input and output, variables, we can nunerically determine a distribution of mutual information with the following permutation procedure. The idea is to shuffle the association of value pairs between imput and output based on the available data according to (a; (t) My (α (t))) for all E (1...., n) and j E (1,..., .m.. where π. : T + T, t + x (t) is a bijective map, called permutation, and T is the set of all time points of measurements of the data points. Intuitively, the permutation test destroys the relation between input and
10

---

Published in Transactions on Machine Learning Research (02/2024)
output by shuffling them. The reason is the sampling of the data from the marginal distributions, instead of from the joint distribution, thus, enforcing independence by design. Ideally, the mutual information computed from samples drawn according to the marginal distributions is zero. However, in practice, due to possible bias in any finite sampling, it might not be exactly zero. Therefore, the mutual information computed from this test can serve as an experimental minimum or baseline. If the computed mutual information value based on the dataset samples, which are distributed according to the joint probability distribution, is unlikely according to the numerically determined distribution from the permutation test based on independent variables, we assume that the computed mutual information value is determined by deterministic mechanisms relating input and output. This applies to both model output and deviations from the ground truth. The observed mutual information value is considered unlikely under the hypothesis of independent input and output variables if less than a predefined percentage of mutual information values (level of significance; in this work 5%) generated with the randomly shuffled data is bigger than the observed mutual information value. The distribution is generated by calculating mutual information for several random permutations (100 times in this work). The same procedure holds to calculate a distribution for the chi-square value defined in or (5) in case assumptions are violated to justify the application of the chi-square distribution for x². However, we remark that the theoretically available distributions for the test statistic defined in 44p and 5 are an advantage in terms of computation time as the above described permutation method requires repeated computations to generate the distribution under the assumption of independent. input, and output variables. Finally, with such a stochastic framework in place, we are able to make concrete decisions if a further improvement is possible given the data or if all the learnable relations are already extracted.
3 Related Work
Relation to feature selection methods In analogy to the idea of using information and stochastic dependence measures for feature selection, we apply the same methods to the input and the deviations between the model prediction and the ground truth. Our approach for measuring the predictability of output based on input variables is similar to feature selection methods, such as the minimize redundancy maximize relevance (mRMR) method Peng el al. (2005), which is based on mutual information.
Mutual information estimation and applications
Besides classical methods to estimate mutual information such as Darbellay & Vajda (1999):
Kraskov et al..
(2004; Gretton et al. (2005): Moon et al. (1995) : Kozachenko & Leonenko (1987): Gao et al. (2017), a more
recent neural network based cstimation of mutual information [Belghazi et al. (2018); ; Oord et al. (2018 Song
& Ermon (2020) ; Franzese et al. (2024) is proposed. Such methods provide an alternative to Algorithm for
the calculation of mutual information. However, since it is the focus of this work to use such implementations to test model convergence in terms of independent input and output, we use framework in which we can directly implement further stochastic measures such as the chi-square test, not being just specific for mutual information estimation. Our work may further motivate the development of such methods as we give another application of such measures and estimators for model evaluation. Furthermore, in different scenarios it may be that different requirements of certain estimation methods are more less fulfilled, which justifics considering several stochastic measures and their implementations which is facilitated by our modularized approach. Moreover, by using different statistical tests, we can demonstrate how to include hypotheses testing to decide for independent input and output, like the target data, residuals or model deviations, which allows us to make a stochastic decision for model convergence based on the given data. Nevertheless, in Cuyá et al. (2023) several mutual information estimators are benchmarked regarding their accuracy. This work provides evidence that in low-dimensional scttings, such as the pairwise case, histogram based mothods, e.g. Algorithm D can provide accurate results. Under model convergence with independent input and model deviations, it is required that the pairwise test confirms independent input and model deviations as well, which represents a necessary condition for convergence. On the other hand, if we have to calculate the full mutual information according to Czyž et al. (2023), neural estimators may better handle high dimensional settings, making it suitable to compute the joint mutual information.
The work Xu et al.] (2019) introduces an information-theordti mcasure called Determinant- Mutual Information (DMI) to train deep networks that are robust to label noise. However, the key conceptual differ- ence from our approach is that they do not account for the input in the same way we do. Specifically, once
11

---

Published in Transactions on Machine Learning Research (02/2024)
the prediction and the ground truth are available, their method does not consider the input that generated the prediction anymore. In contrast, our approach consistently evaluates whether given prediction can be improved based on the provided imput, taking into account the entire triple of input, prediction, and ground truth, whereas Xu et al. (2019) focus solely on a tuple consisting of the prediction and ground truth.
DeepInfoMax (DIM) Hjelm et al. (2019) as well as contrastive predictive coding (CPC) Oord et al. (2018) maximizes mutual information between raw data and its compressed representation to find a better repre- sentation of raw images. Furthermore, Chen et al. (2016) employs mutual information to find a disentangled and interpretable representation of images. The work of Brakel & Bengio 2017) also focused on finding a hindppeenll/ epprecntatioonfeaauuee
of images and uses mutual information as mcasure of independence between these features and thus the degree of disentanglement. Our framework can be directly applied to the corresponding generated representations to test if the model accuracy is only caused by unpredictable parts like noise. For more details, please see the Discussion about the case of unstructured input data.
Innovation method
In time series analysis and signal processing, the Innovation method Reid & Term (2001): Houts et al. (2013) is a widely recognized metric. It is often used as an indirect measure of the consistency of the prediction of the model (typically Kalman Filter) when the true (ground truth) latent states of a dynamical system are inaccessible. The Innovation method works by comparing the observed data with the model's predictions with the goal that the differences known as innovations or residuals have a zero mean and a specific covariance where residuals from different time points are uncorrelated. For Kalman Filters, this covariance can be determined in closed form. A range of tests Reid & Term (2001) can be conducted to ensure these criteria are met, thereby confirming the filter is consistent and reliable. While the Innovation method uses stochastic means to investigate properties of the residuals to evaluate the model performance, our framework additionally includes the imput data to evaluate the stochastic independence of input and residuals to evaluate the potential of a model to be improved due to remaining deterministic relations between input and residuals. Due to computational challenges the calculation of stochastic measures might. have and thus due to the usage of approximations, the Innovation method and our presented framework can be used as necessary tests that a successful model has to pass. Furthermore, our framework also suggests an option how to extend to nominal ground truth data where it is not possible to define differences and thus residuals.
Ensemble learning
Our proposed framework can be considered as an extension of ensemble learning methods such as Wortsman et al. 2002), in particular any kind of boosting algorithms Schapire & Freund (2012). We refrain from the indiscriminate combination of multiple models. Our framework triggers a combination only if it confirms the presence of potential for further improvement in terms of further learnable/ betraable/xxttctabblee eextractable deterministic relations. Thus, we can define a convergence criterion for the number of combined models.
Information bottleneck
Another application of our framework is to extend the information bottleneck concept Saxe et al. (2019) Kawaguchi et al. (202)). The basic framework of the information bottleneck is to maximize the mutual information between a data representation sought and the corresponding output data and at the same time to minimize the mutual information between this representation and the input data, There is a parameter to balance both contradicting requirements. With our framework, we can extend the information bottleneck method by providing a procedure how to choose this balance parameter the input data. Starting with configuration where the model extracts all the information between input and output data, we tune the balance parameter such that the compression is weighted higher until the model cannot extract all the information between input and out assessed by having dependent input and residuals or model deviations. With that procedure, we find the threshold of the balance parameter for a lossless compression.
find lossless compression of
Time series prediction
Besides the above related methods, the showcase of this work is in particular associated with time series prediction. In the past few years many of time series prediction models have been developed to improve the prediction performance, such as RNN, LSTM Memory (2010), GRU Cho et al. (2014)), transformer and
12

---

Published in Transactions on Machine Learning Research (02/2024)
its variants Vaswani et al. 2017): |: Kitacv et al 2019 Zhou et al. (2021 Liu et al. 2022]: Zhang & Yan [2022] ; Wu et al. 20211: :: Zhang et al. (2022) and state space models such as RKN Becker et al. (2019). S4 Gu et al. p MTS3 Shaj et al. (2023) as well as simple yet effective linear models such as Zeng et al. 2202). The improvement is measured in terms of corresponding loss function values of the best weight conliguration of a model. We would like to extend these evaluation by our framework where we introduce another quantity that evaluates how much information left to extract from the time series given the input length of the historic data. Furthermore, our approach would like to establish level before model training, that allows for evaluation if time series is predictable given historic data. This is in particular important for challenging time series and to evaluate if already simple models are sufficient in a prediction scenario Zeng et al. 2202)) Li et al. (2023). Nevertheless, there is question in all scenarios: How can we make sure that the current method has already achieved the lowest possible bound of the error for each time series or can it still be improved by fitting a better model for each case?
4 Appiications/Numrriall Numerical experiments
In this section, showcase our framework to analyze time series for their predictability, measure the dependence between input/ /context and output/targe data for the purpose specified in the bullet points on page 4 and demonstrate how to measure learning success of models with our framework. Moreover, we show how our framework extends the performance evaluation of a model.
Apart from several real world datasets, we use a synthetic dataset to purely demonstrate the core coucept of the paper described in the bullet points on page 2 where we can control properties of the noise, such as the ratio between information and noise, which we do not know in real dataset. The synthetic dataset consists of the time series that is composed of the sinus function, sin IR - R, + sin(t) and the random variable 0: R + R, L O(),, which generates noise by random values according to y(() == sin(l) + aQ(1) where >> 0 scales the amplitude of the noise. In order to evaluate performance in an already established metric, we use the normalized root mean squared error (RMSE) defined by 1 2V E of measurement points, zt is the model prediction the discrete time point E N, yi is the ground truth and a is the standard deviation the data points of the training data.
VVE (ū) w)² where N E N is the number
E (g- y calculated on the training dataset where y is the mean of
The parameter P of Algorithm dataset on which the algorithm is performed, unless otherwise stated.
! is set to 5% of the number of measurement points of the corresponding
We remark that the term residuals is usually used instead of model deviations to name the difference between model output and the ground truth in case of ordered target data. In the nominal case, we further use the term model deviation, which is defined in Subsection 4.4 We show in both settings the stochastic dependence decays over training epochs on training and validation datasets.
We provide a short overview about the intention of each subsection this section and what it is supposed to demonstrate:
Subsection
4.1
The noise-independent evaluation of model performance shows the success criterion.
Subsection 4.2
The fossffuncio agnostic evaluation of model convergence demonstrates the success
criterion as well.
4.3 Differentiating model inaccuracies into systematic model failures or those due to unpredictability and thus show stochastic independence of input and residuals or model deviations as stopping criterion. Consequently, shows the success criterion, convergence criterion and thus the usability for determining hyperparameters.
Subsection
Subsection The application of our framework to nominal data. demonstrates the first three bullet points of page 4 for the classification problem.
13

---

Published in Transactions on Machine Learning Research (02/2024)
The detection of distribution shift in the sense that relations learned by model do not. hold on data set different from the training data set. Consequently, it shows the distribution shift detection and data efficiency bullet point of page 4
Subsection 1.5
While these items demonstrate the numerical application of our proposed framework outlined in the bullet points of the Introduction, we have an additional section in the Appendix, Section [ where we stack models top of each other to correct the output of the former model. The numerical experiments with the stacking architecture provide further evidence that any remaining stochastic dependence between input and model deviations, identified with our implementation, can be used to correct former predictions. These dependencies are not detectable with accuracy metrics that only consider model output and ground truth.
4.1 Splitting noise off from model inaccuracy
In practical scenarios where data is obscured by noise or cases that some components of the future samples are not predictable based on the given features, like the the history, evaluating prediction models becomes challenging- Traditional metrics such as T,2-loss on validation set may not suffice due to the uncertainty surrounding the ratio of unpredictable to predictable parts. This uncertainty complicates determining the lower bound of prediction error beforchand. Therefore, solely relying on loss metrics for model assessment is insufficient, as the source of error could be either model For example, when the power of noise (as an unpredictable component) equals to half of the data power, the lowest possible normalized mean squared error (MSE) on validation is 0.5. However, by employing our framework that identifies when the model has learned the primary data component effectively, we can stop training and attribute the remaining residual to initial data noise, although the 1.2-error is still high. Motivated by this discussion, we start our experiments with such a data where the first component is fully predictable such as the sinusoid function plus some independent, identical distributed Gaussian noise that is completely umpredictable, however, the portion of each is controllable for us in this synthetic case.
lalqquuuccy/flurrrru or inherent data umpredictabiitty
For the numerical implementation, we consider a window of size 50 and the first 49 samples are considered as context/inpit to the model and the 50th sample as the larget/output. To generate the dataset, the window is slid over the time series that was split into training and gest/validation set before the experiment according to ratio of 0.75/0.25.
For the experiments depicted in Table D the random variable 0 adds Gaussian noise. According to the table, the chi-square test and the mutual information indicate that the MLP model, trained with I,2-loss, has extracted the sinus function. We see that all essential information is extracted because the chi-square test indicates that all residuals are independent of the input. In the case of mutual information where the p-values are greater than 0.01, it is indicating that based on that level of signíficance, we cannot reject the hypothesis that the input and the residual are independent of each other and thus the input is unlikely to carry information about the outcome of the residual. Therefore, in these cases, even a much more sophisticated model is not able to decrease the error further and might lead to overfitting to the noise. While the RMSE increases with the amplitude of the noise, indicating that the model performance would become worse, our stochastic measures indicate that the model has extracted the sinus function as the main deterministic driver of the time series. This can be seen since the - between model prediction and the pure sinus value, shown as Actual Err in Table [ is (almost) independent of the noise amplitude, Furthermore, the fact that the normalized test RSME always close to the relative noise standard deviation indicates that the deviation between model and data results from the noise and not from systematic deviation from the sinus function. We remark that the first and third column can only be presented because we exactly know the signal and the noise component separately. In real world data, these numbers usually cannot be computed, however, our method can provide the valuable insight if there is some relation left to extract or the deviation between model and data is rather due to errors coming from unpredictable relations given the input data.
4.2 Information extraction influenced by loss function and noise properties
In this part, we show that small values of stochastic measures coincide with a model that learned the under- lying dynamics of the deterministic part of the data. Moreover, the choice of loss function is an important
14

---

Published in Transactions on Machine Learning Research (02/2024)
Exp.
Err
Chi-square Analysis
Mutual
Information
Analysis
Rel. Boise std
RMSE
Actual Err
Init. dep.
Res. dep. var
Lui.. MI
Init. Perm
Res. MI
Res. Perm
2.2 x 10
2.2 10
49
49*
27.497
0.7385 + 0.0272
27.2781
0.4039 + 0.236
0.2715
0.2731
0.0616
49
8.3351
0.7076 + 0.0136
0.7301
0.7096 # 0.0114
0.05
0.4927
0.4914
0.1073
49
4.193
0. 0.7337
0.0119
0.7643
0.7332
0.0108
0.6465
0.6568
0.1476
47
2.5647
0. 1.7022 0.0104
0.7472
0.7049
0.0095
0.7482
0.7461
0.1682
38
1.6659
0. 0.7322 0.0032
0.7503
0.7054
0.0089
0.8166
0.8184
0.1789
37
1,2591
0. 1.7293 + 0.0095
0.7862
0.7296
= 0.0085
0.8611
0.8595
0.1831
30
1.053
0.711 0.0077
0.7562
0.7102
0.0069
0.8921
0.9068
0.1893
28
0.9725
0.7275 0.0086
0.7931
0.7278
0.0075
0,9148
0.9134
0.1846
19
0.8825
0.7109
0.0079
0.7605
0.7384
0.0082
0.01
0,9306
0,9415
0.1934
11
0.8751
0. 0.7341 0.0075
0.7521
0.7070
0.0073
0,9425
0,9410
0.1952
0,8153
0.7053 +: 0.007
0.7652
0.7042
0.0085
Table 1: Impact of varying amplitudes of white noise on a simusoidal signal and assessing its influence on the performance of a simple MLP. The relative power of the noise component, which is the root of the variance of noise divided by the variance of the total time series, is shown in the first column, which is the theoretical lower bound of lest RMSE. In the second columm, the normalized RMSE on the test data is shown. The third columns shows the Iossoss between the prediction and actual clean sinusoid. The fourth column illustrates the dependency of the target on the history evaluated by chi-square test, which is initially complete until almost half of the power is taken by noise and decreases to small numbers when noise becomes the dominant (94 percent) part of the time series. The fifth column evaluates the dependency of residual target on the corresponding context to show how successful the model is to reduce this dependencies. The last five columns show the same concept in terms of mutual information. Similar to chi-square, we have initial and residual values as well as two more columns to report their corresponding lower bound which obtained by random permutation of imput and the corresponding residual, see Subsection [.2 for details. The last column shows the p-value of the observed mutual information under the hypothesis that residuals are independent of the input given the dataset calculated by the mentioned random permutation. Note: * The model fits the data up to numerical errors occurring in the residuals around the peak of the sinus, which represents a predictable pattern due to the periodicity, see Figure 15 Such cases can be handled by introducing threshold indicating a very close fit between model output and data in some norm. An alternative could be to impose a minimal bin width in Algorithm which might impact the equinumeric property.
key factor to extract the deterministic relations from the data. To demonstrate it, we use our synthetic dataset where the random variable f is defined by O(t) = 1n (t) 100g(t), 0 is Gaussian distributed with mean zero and standard deviation 1 and 02 is a Poisson distributed random variable over the number of peaks (high values of the time series) within a time interval. If at a time point t there is such a random peak, the variable (2())= = 1 and 0 otherwise. In this numerical experiment, the rate of arrivals of a peak is 200 peaks per sample (or expected time betweon two peaks), meaning that we expect one peak after 200 data points on average.
Since peaks only increase the values of the time series randomly, the mean of the noise is greater than zero, indicating asymmetry. In this case, we see in Figure that the initial dependence between input and output is lotally reduced only by the MLP that is trained with the T.*llsss function while it does not extract the total deterministic relations when using an - function. By "extracting deterministic relations", we refer to tearn/approseee approximate the sinus-function with the model. This extraction has taken place when only noise is left that is independent of the input, as shown in Figure 2 where we see that in the L'-norm the model output trained with I, -los function is much closer to the corresponding funustnnnttion, evaluated on the validation set. In other words, the results of Figure 2 depict that the model trained with an L function is prone to the properties of unpredictable patterns, specifically irregular peaks, in this experiment. For illustration, plots of predictions and residuals for - and - functions are given in Figure [6] We remark that also here the synthetic set is useful since we know the exact functional formula that generates the data apart from noise in order to confirm that our utilized metrics can better reveal which prediction is more in line with the underlying relations in the data, in particular where these metrics are solely applied on measured and possibly noisy data. Consoquently, this example demonstrates that when a stochastic measure reports independence between model deviations and input, indicates that the model has successfully extracted deterministic relationships, excluding noise or unpredictable patterns. This aligns with being close to the
15

---

Published in Transactions on Machine Learning Research (02/2024)
Results for Chi-square Analysis and Residual MI
L2 Distance Between Prediction and Sinusoid
tradrrined Model
0.12
12-trained Model
Li-trained Model
Li-trained Model
40
Number Dependent Variables
0.10
Mean RMSE # Variance
0.8
30-
0.08
Resibial MI
0.6
0.06
20-
0.04
10:
0.0
0.00
Chi-square Analysis
Analysis
L2-trained Model
L1-trained Model
Figure In the left sub-figure, the bars represent the stochastic measures of independence between the input and residuals, along with their corresponding standard deviations obtained from several training runs with different seeds. The initial number of dependent variables 49, The value of initial MI is 13.342 + 0.071. The p-values are shown in Figure 3 For the L model, the p-value always remains zero. The right sub-figure shows the RMSE values between the model predictions and the clean (pure) sinusoid for models trained with T and loss functions. The lower bound on the error is O, the predictions are compared to the pure sinusoidal function. This results is completely in line with the result of the left sub-figure as being closer to the clean data coincides with a lower value the stochastic measures. Notably, these metrics require no prior knowledge of the clean data, underlying relationships, or model architecture, as they rely solely on the input, output, or residual data. All models were trainod on noisy data. More information is provided in Figure 3 All values are reported on the validation set.
ground truth, not in terms of fitting the training data (which includes noise), but in capturing the underlying deterministic dynamics. In our case, this corresponds to the pure sinusoidal signal derived from the input data. further example is given in the Appendix, Section C showing that independence of input and model deviations indicates being close to the clean data.
An explanation for the observations that the minimization of a loss function under the constraints of the model does not necessarily coincide with extracting the real dynamic that underlies dataset extracting the most information from the dataset, respectively. For example, the minimum of the L function subject to the constraints of the model is more distracted from the real dynamics (sinus-function) by the specific noise than the L -corrrepooddnce. Another reason for the difference of the loss functions in taking optima might be that the L''loss weights any deviation between model and ground truth equally regardless if the corresponding data point is predictable given the input while mutual information, like any other measure based on stochastic independence, takes only predictable data points into account. Depending on the properties of the unpredictable data part, this difference might lead different optima regarding the model's parameter values. The key point is that these stochastic dependence mcasures determine which optimum is closer to the underlying dynamics, something loss functions cannot always achieve.
4.3 Stochastic measures convergence criteria
Next, we show how the mutual information and the chi-square test evolve during the training after each epoch to demonstrate its capability to work convergence criteria both for model training and hyperparameter scarch. The procedure is as follows. If input and output are not independent of cach other, training of a model is started. If after some epoch, input and residuals or model deviations, respectively, are independent regarding the training or validation set, more detailed we cannot reject the hypothesis that input and residuals, resp., model deviations are stochastically independent, then there is no information left to extract and we can stop the training. The importance of the stochastic measure is that we can evaluate if flattening of the loss function after some epochs is due to whether the training is done in terms of information extraction
16

---

Published in Transactions on Machine Learning Research (02/2024)
or if the convergence speed meanwhile has just slowed down. In the latter case, it is worth further patience since further information may be extracted. Moreover, during later epochs, the convergence speed might increase again once the optimal weight updates are identified. The advantage of a stochastic measure taking the relation between input and model deviations into account is that there is a lower bound of dependence known in advance. This bound theoretically always achievable by a model, namely when all the deterministic relations are extracted, which is by definition independent of the unprodictable parts given the input data. In contrast, for loss functions that do not take this relation into account, there is no lower bound known in advance that is achievable on the concrete dataset.
In this experiment, we use the data from Section 4.2 and plot relevant metrics in Figure 3 We see that when the loss function doesn't improve any more, the corresponding chi-square test is zero and the p-value of the mutual information becomes non-zero in a magnitude of order such that we cannot reject the independence of imput and residuals. Although we only check pairwise instead of all possible combinations of variables (joint distribution), see the limitations in the Discussion for details, this experiment shows that our framework provides valuable convergence criteria as a stagnating loss function coinciding with stochastic independence can be rather interpreted that the model has extracted all the deterministic relations between imput and output. The rationale is that if input and residuals or model deviations are independent of cach other, it is necessary that also a pairwise test indicates independence. Furthermore, the turning point when stochastic measures increase marks the condition for early stopping without relying on a validation set. This turning point indicates that, while the loss function continues to decrease, the model may start overfitting to the training data by adapting to spurious correlations or noise. The overfitted relations do not seem to be relevant for the validation set as here the stochastic measures do not decrease any more after some epochs.
We provide further example based on the dataset ETTh2 Zhou et al. 2021 with a similar result depicted in Figure [ We see based on the chi-square test that, after some epochs, there is already a set of weights based on which the residuals are clearly stochastically independent of the input. Similarly, we see that the mutual information is close to the value generated by the permutation test, supporting the chi-square results. Furthermore, the increase of the chi-square and the mul ual information on the validation set for later epochs might indicate an overfitting since the minimun based on the -ooss function, which further decreases over epochs, does not necossarily coincide with the maximum of extracted deterministic relations defined by stochastic measure, see also Section
4.4 Time series classification
In this section, we apply our framework on time series classification (TSC) problems. We remark that for ordered classes, we can apply the framework where the differences between the discrete output values and the ground truth model the residuals. The deviations from the ground truth in the nominal case are defined as follows. The random variable f modeling the deviation of the model classification from the ground truth, takes the value of the correct class in case the output. of the model is not correct and -1 otherwise. This definition allows us to measure the information content between the correction of the model (representing the model deviation) and the input. We apply our framework on two subsets of the well-known UCR datasct Dau et al. (2019). More specifically, the dataset AastalPhalanxOutlnnCCCrrre classification problem from time series data and the dataset ElectricDevices shown in Figure 0 consists of seven classes. We relabeled the classes always starting from O until all classes are labeled accordingly. The model architectures used are the Nonstationary Transformer (NSTs) Liu et al. (0022) and the MLP.
shown in Figure 5 is binary
In this case, the parameter p of Algorithm 1 set to 10 to cope with the high imbalance of the classes and the fact that over epochs this imbalance increases since most of the cases are classified correctly and thus the size of the class labeled with -1 for 0 increases. Results including cross-entropy loss, accuracy and the mutual information per epoch are shown in Figure and Figure 0 In these figures, we observe that while the accuracy is increasing, the dependence of the variable 0 with the input x decrcases over epochs as it is supposed to since less and less cases are not predicted correctly. In other words, the random variable 0 tends to become a constant function where information gain becomes small or zero, regardless of any input time scries. The difference in both cases is the following. From the results of Figure 5. we can conclude that over training epochs we approach a parameter configuration where we cannot reject the hypothesis, at significance level of 1 1%, that the input and the model deviations are independent based on our mutual
17

---

Published in Transactions on Machine Learning Research (02/2024)
Chi-square Analysis Per Epoch Train
aeilssuuaeennallsii Per Epoch Validation
m
40
40
20
20
10
10
100
140
Remaining Experiimental Ninimum |Permutation Train
Remaining Minimum
Validation
Remutiice
1iM PCD) Ferruration
P1:0 Hil PiPmuutrio 1:M)
MIPLI
20
60
100
120
140
20
00
80
100
120
140
P-value Associated with Current Nutual Information Train
P-value Associated with Current Nutual Information Per Epoch Validation
HUnL HLPU2
-UP12 -NP.1
P-palue
0.8
0.8
0.6
M 11 M
M ypha /yw
0.2
0.2
Epoch
Spoch
LA Train LoGS
La Validation Loss
0.6
0.6
0.4
0.4
0.3
0.3
Epoch
0.1
20
60
80
100
120
140
20
40
50
100
120
140
MLF12
MLPL2
0.9
0.9
A //
M
Epoch
Epoch
0.6
0.6
20
60
80
120
140
20
80
1.00
120
140
12 Norm between Prediction and Sinusold Train
L2 Norm Between Prediction and Sinusoid Vallidation
HUPU HLPU
HUPL N.P.2
0.4
0.5
0.4
0.3
0.2
0.2
0.1
Eaoch
Figure 3: Curves showing stochastic measures of independence and loss function history.
18

---

Published in Transactions on Machine Learning Research (02/2024)
Chi-square Analysis Per Epach- -Train
Chi-square Analysis Per Epoch- Aepoch-Valideee Validatiio
80
MA AuM pp In
M0
80
80
40
20
Epoch
Epoc)
50
100
Remaining vs. Experimental Minimum (Pennutation Test) - Train
Remaining MI Experimeentl Minimum (Permutation Test). Validation
Perristrr
HRP.L Permutation
20
15
15
5
Epoch
Epoch
150
Loss Train
L1 Loss Validation
HPPL
-IPLL
0
0.6
0.6
0.2
0.2
Spoch
50
100
150
0
50
100
150
Figure 4: Curves showing values of stochastic measures of independence between input and residuals as well as the loss function history based on the ETTh2 dataset.
information measure. In this case, we do not expect a further improvement regarding the model performance since it potentially captures all deterministic relations. In contrast, in the experiment depicted in Figure G our mutual information test allows us to reject this hypothesis, indicating that there are still deterministic relationships to be extracted, which may improve the model's performance.
4.5 Detecting distribution shifts
One prevalent issue hindering the advancement of ML models towards higher accuracies distribution shift, meaning that relations that hold within the training set do not hold on the validation set. A reason could be that the validation set contains newer data and that the underlying dynamics have changed change over time. Especially several existing works such as (Zeng et al. 2023 Figure 5) and Kim et al. (2021). in particular (Kim et al. 2021 Figure 3) have confirmed this phenomenon, e.g., on ETTI and ETT2 data sets. This section presents a novel insight into this phenomenon, illustrating how our proposed framework can detect and distinguish such cases from mere overfitting to noise. In a typical training scenario, after some epochs while training loss continues decrease, validation loss may gradually start to increase. Without prior assurance of the absence of distribution shift, pure loss function based approach without including stochastic measures struggles to differentiate between overfitting to the noise in training data and (partial) distribution shift due to different deterministic relations between the training and the validation set, as both can lead to similar observations of an increasing loss function on the validation set.
Our framework provides a concise solution. Instead of solely monitoring the loss function, tracking mutual information enables us to determine the types of relationships the model is learning. A decrease mutual information across epochs indicates successful extraction of information, suggesting that the model is learning deterministic relationships within the training set, and not already overfitting to noise. If it does so as well
19

---

Published in Transactions on Machine Learning Research (02/2024)
Cross Entropy Loss Train
Cross Entropy Loss Volidation
HPP
0.8
0.0
Cross Entigg)
Cross
0.4
0.4
Epoch
Epoch
0
120
100
140
Accuracy- Teryy-Tre Train
Accuracy Validation
Aocurec
0.6
0.6
0.2
0.5
Epoch
Epoch
Remaining MI Between Input and Deviation Per Epoch Train
Remaining Between and Deviation Epoch Validation
Notual Informatoo
15
35
Epoch
Epoch
ofputt Epoch
Input-Devistion Epoch
HLP
HLP
Actual Infor Information
Luualinfommttion
15
5
Epoch
20
40
90
100
120
140
20
50
100
120
140
P-valuu Associated with Current Information-
P-value Associated Current Mutual Information Validation
0.05
P-yalus
0.05
0.04
0.03
0.02
0.02
Epoch
40
80
80
1000
120
140
20
40
80
80
300
120
140
Figure 5: Results on a classification problem showcasing our framework, in particular the definition of the model deviation for nominal target data. Although validation accuracy remains below 0.8, mutual information analysis shows the model deviations already independent of the input.
20

---

Published in Transactions on Machine Learning Research (02/2024)
Cross Entropy Loss Train
Cross Entropy Loss Validation
Spoch
Epoch
Accuracy- Train
Accuracy Validation
0.9
0.0
0.7
0.6
0.6
0.5
0.3
Epoch
0.3
Epoch
Remaining MI Between Input and Deviation Train
Remaining Between Deviation Epoch Validation
XST
NIP
35
30
20
15
15
20
60
40
Input Epoch Train
Input: Deviation Epoch
MLD NST
NLP
NST
Informo mirrmation
40
35
30
30
20
15
15
Epoch
10
0
20
40
60
20
40
60
Figure Results on a classification problem showcasing our framework for nominal target data. The p- value for mutual information always remains zero in this experiment for both the training and validation sets indicating that the achieved accuracy is still improvable.
on the validation set until input and deviations between model and ground truth are independent, then we can stop the training process since the model might have learned all deterministic relations on the training set that hold true for the validation set, too. In that case, further training might cause an overfitting relations not present the validation set. Similarly, if there is no significant reduction mutual information despite decreasing training loss, may indicate overfitting the noise in the training data, as the model is fitting to the unpredictable elements of the target which share mutual information with the input.
In our synthetic datasct, the deterministic relations are identical in training and test set by construction. In Figure 3 we observe that during the training based on the I function that the mutual information increases on training and validation set after reaching their minimum within a few epochs simultaneously. It may mean that the model fits noise rather than the actual simus function since the norm of the difference between model prediction and sinus function increases simultancously while the loss on the training set further decreases.
21

---

Published in Transactions on Machine Learning Research (02/2024)
Remaininn MI between Input and Residual Per Epoch Train NUPLLLon wtt.hhhsttthhhhaess _9
Remaining between Input and Residual Per Epoch- Validation http_p_s__stiis__eeihi..
190
Mu Inloerstoo SrOtto
Mutual Information
354
185
352
350
Epoch
348
Epoch
10
20
0
30
Loss Train
Loss Validation
0.292
0.236
0.29
0.237
0.288
0.236
Epoch
0.235
Epoch
5
10
30
35
15
20
25
Figure 7: Distribution shift experiment on the ETTh1 dataset. The experiment is repeated for six different random seeds. The shaded area depicts the minimum and maximum over seeds and the solid line shows the average.
case the deterministic relations in the training data may (partially) not hold true for the validation set, it may lead an increase of the loss function value and potentially mutual information over epochs on the validation set, while mutual information the deterministic relations learned the training data do not cause a fitting output of the model the validation set. In contrast, please see Figure 3 where mutual informat ion between inpul and residuals on training and validation set demonstrate the same behavior. The results in Figure shoow the learning curves when fitting an MLP with an L'lloss function on the residuals a PatchTST model Nie et al. (2022 with the best setting they proposed on the ETThl data set where the input length 336 lags and we predict a target length 96 lags. Details about the combination of both models can be found Section of the Appendix. We remark that this experiment also showcases the application of our proposed framework to multi-dimenssonal output.
the training set decreases, we in Figure 7 In this figure,
5 Discussion
This section discusses the assumptions and limitations of our approach, followed by potential applications.
Assumptions and Limitations: The implemented approach tests the pairwise relationship between input and output features. We acknowledge that there is a difference between pairwise stochastic independence and mutual stochastic independence when dealing with more than two random variables (Gallager 2013 Section 1.3.4). This implies that more information might be obtained by considering multiple input features simultaneously and testing their relationship with the output, rather than testing pairwise relations between individual input and output features. However, the full consideration, instead of pairwise testing, scales exponentially in terms of the computational costs. Consequently, we aware that the current pairwise approach, which is computationally cheap compared to the full approach, cannot provide in general full statement that no information is left to learn. In this regard, the current approach can only be used as an additional metric to evaluate if training complete and further iterations might not provide a further improvement. This could be the case if low pairwise measure of the relation between input and output coincides with little or no improvement in the loss function. Pairwise test is a specific case of a full consideration. Consequently, it holds: pairwise test indicating deterministic relations between inputs and
22

---

Published in Transactions on Machine Learning Research (02/2024)
residuals/model deviations implics that there is still information left that the model can learn. An analogous framework involves using gradient descent as a convergence criterion in optimization. Gradient descent is widely used in deep learning and has been successful despite in general lacking sufficient conditions for global minima convergence. The gradient provides necessary conditions for convergence, and only under certain circumstances, it can be a sufficient condition. Nonetheless, it produces useful optimization results and keeps computational costs managcable, similar to our pairwise stochastic measures. In this regard, all deterministic relations are learned, it is necessary that the measures for independence based on a pairwise test indicate independence of the input and the residuals the model deviations.
It is left to investigate under which conditions a pairwise consideration is sufficient to test for total stochastic independence of input and output. Furthermore, we remark that our framework is not limited to a specific seeeenddence//ffrmaattee
choice of stochastic measures can be quickly included in a modularized manner.
measures and also our git repository is designed that new
Due to the fact that any approximation or estimation of measure, such as mutual information, may not be sufficiently accurate Czyž et al. (2023), it is prudent not to rely only on single estimator but rather to use multiple ones. We achieve this by considering both an approximation for mutual information and the chi-square test. The generality of our framework allows for the use of different measures for independence where we can also consider other features like computational runtime apart from accuracy. For example, the test statistic of the chi-square test of independence provides a theoretically known distribution enabling quick hypotheses testing to determine if based on the data corresponding variables are independent.
For time series, we would like to remark, even if a method shows (full) mutual independence between the input (history of the time series) and the output (prediction target), that this result does not imply that the time series is not predictable. It just indicates that with the given input/provided information, the time series is not predictable. Maybe with other features that are related to the quantity measured as a time series, there is a deterministic relation that can be used for predicting the time series.
Alternative of model deviation for nominal data: One further option to model deviations of a model from the ground truth in the case of nominal target data could be a multi-dimensional random variable that models the difference between the actual probability distribution (c.g., 1 for the correct class, 0 otherwise) and the predicted distribution from the model over the classes to be predicted. If the input variables and the difference of the distributions, which serves as the model deviation, are deterministically related, then the model deviation could be represented as a multi-dimensional variable (one variable for each class) to mcasure the predictability. In other words, the classification problem could be seen as a multi-dimensional ordinal data scenario.
Extension to unstructured data: For models that extract information from an input like videos, images, sound, or text, the input data needs to be transformed into a mumerical representation that shares information with the ground truth. An example are the pixels in figure where small object of interest moves within a big blank picture or tokenized text, where the position of a word can vary while not changing the meaning. We could optimize first layer(s) to have the highest mutual information with the ground truth that is to be predicted analogously to Chen et al. (2016) or Brakel & Bengio (0117) to optimize the encoding of the unstructured input into numerical vector representations.
It is the lask of an encoder to find numerical representation (vector embedding) from an input signal that maximizes information content, e.g., in the form of mutual information stochastic dependence, with the ground truth to be predicted. Thus, the encoder is supposed to extract the relevant information from the (unstructured) input regarding the prediction target. This extraction can be seen as a definition for "relevant information". In that sense, training a model can be two-staged. First, we could train an encoder part to maximize the mutual information or stochastic dependence between the vector embedding (output of an encoder) and the ground truth. Using a gradient-based method for this purpose requires differentiable estimators of mutual information (e.g., Franzese et al. (2024)) or any other differentiable definition of stochastic independence. To consider the example above, the role of the encoder may be to extract the moving object from the figure, independent of its position. In the second stage, we train the decoder part, which is supposed to make the prediction based on the vector embedding, with some loss function to minimize errors between model output and ground truth. Another option is to optimize
23

---

Published in Transactions on Machine Learning Research (02/2024)
First training process (Er crodddr traiiing: Optima ze paramiters of the Er Brocder sueh that giver information consont smeuth approsimation it ofher Lansten mecmmninn atscheeds depprttento
Nev pannorugee 21044
Ground but
More ourputy
Encoder
Decodder
Bcoond training procoso (Docodor training: gi: Optimoo panemetees of Dectztee such that glown betowen mihimized.
Figure 8: Shnoder-Decoddee architecture for a separated training of the encoder and the decoder based on stochastic dependence.
the decoder for minimizing mutual information or the stochastic dependence between the model deviation (decoder's output and ground truth) and the vector embedding (encoder's output). However, this approach also requires a diffcrentiable function for defining stochastic independence or mutual information. We remark use
that separating the training of the encoder from the decoder's training might be important in loss function based on reducing stochastic dependence between the encoder's output and the decoder's deviation from the ground truth because if the encoder's representation does not contain much information about the ground truth, the decoder's deviation from the ground truth is not predictable given the encoder's representation ion and would thus indicate training success. Given the fixed dimension of the vector embedding, we can compare different encoder architectures and define the best according to, e.g., the highest (normalized) mutual information between vector embodding and prediction target. Analogously, the best decoder can be defined as the one having the lowest (normalized) mutual information between the encoder's vector embedding and the model deviation of the decoder's output and the ground truth. The mutual information measure can be replaced by any measure for stochastic dependence. The corresponding encoder-decoder architecture is illustrated in Figure g
With the definition of model deviation, see Subscction 4.4 subparts of the model, such as layers, in par- ticular those associated with representing the unstructured input as a numerical vector representation, can be analyzed regarding their contribution to find the next token in the context. of large language models. Moreover, investigating the dependence of the model's ayverssubsstructues with the model deviation can help to answer the question how small model can be and where most information is learned/extractee to make large language even multimodal models more efficient. For more details, see the next paragraph about cutting down models. Since distributions over classes represent the probability for corresponding outputs, it might also be an option to work on distributions directly to investigate the dependence between the layers and the model deviation as described above under 'Alternative of model deviation for nominal data', although the dimensionality of the problem increases accordingly.
Another advantage of calculating stochastic measures between representations within (pre-)trained model and the ground truth belonging specific prediction task such as classifying input predicting numbers, is to find out which representations have the highest dependence (information content) on the corresponding task, which might make the inference more efficient by only utilizing the most contributing parts.
Cutting down models to structures extracting most information: we train a model with usual loss functions without any additional optimization for mutual information specific layers, we could identify the best structures within trained model that extract most information and which subparts only contribute in minor way to, e.g., an embedding. For this purpose, we zoom in model's architecture to analyze
24

---

Published in Transactions on Machine Learning Research (02/2024)
the model itself regarding its subunits (e.g. layers, embeddings, attention mechanisms, etc.). The approach is to apply mutual information similar to the scenario of multi-dlmennionall input and output. The output can either be filled with the data to predict or the output of some other subunit, such as an embedding, upstream towards the output of the total model where the output of several subunits is combined. If there is no contribution or only a minor one, we could cut the corresponding subunit out.
By ranking subunits with such measures, we have clear procedure what subunits to prune make a model more efficient, instead of randomly selecting some for cutting out before retraining the model. The training could start from the current model parameters to just fine-tune the remaining layers. Training from the scratch is also possible but could delete a lot of parameter values that are still valid. The procedure can be iteratively repeated and even to that point where model size is balanced against a (small) drop of accuracy. Our framework can additionally help to find a lossless pruning similar to the Related Work section about the information bottleneck by testing the stochastic measures between input or any representation of it and the model deviations get worse after a pruning. Such investigations might facilitate an understanding about the problem-specific relevant, structures and keeping model sizes efficient without lowering their accuracy. As example, we could test how large, e.g., large language or multimodal models need to be and which structures extract the most information, similar to Liu et al. (2024). Further, we remark that with a mcasure for seff-similarity, like the Pearson correlation, we could probably identify identities, resp., structures that behave similarly, e.g., in a sequential arrangement of layers that have the same output behavior, where all the layers except the last one should be considered as less relevant since they only direct information through. A similar work done in Gromov et al. (2024) to identify layers with a similar behavior that can be pruned. However, with mutual information, besides considering similarly behaving units, we can also investigate the single subunits' contribution within branch structure to the part where the branches come together.
If a training fails, we could also use such methods to test which structures fail. An example is cascade of layers with one layer where input and output do not share any information anymore. The disruption of routing information could mcan that one layer might not be well trained and is like a constant function, which may delete important information.
6 Conclusion and Future Work
In this work, a framework for measuring predictability of input-output relation was developed. Furthermore, it was shown how the information extraction of an ML model from this input-output-relation can be mea- sured. Moreover, it was demonstrated how the corresponding stochastic measures for predictability can be used to exlend the current definition of model convergence and training success. The total framework was showcased with time series prediction and classification on synthetic and real world datasets.
The presented framework provides measures to evaluate the existence of deterministic relations that a model can extract and how successful a model has been in extracting them. All these measures consider the triple of input, model output and ground truth to determine whether the model deviations from the ground truth are independent of the input. Promising further rescarch could involve developing a differentiable loss function based on such stochastic measures to fit the model to the deterministic relations directly, sorting out umpredictable parts like noise. In contrast, there are loss functions that only consider a tuple of ground truth and model output, such as 2 L2, cross entropy or KL-Divergence that minimize the deviation between the model's prediction and the ground truth without differentiating if a data point is mainly influenced by, e.g., noise or the deterministic relations.
According to the presented framework the mutual information between input and the model deviations might serve such loss function sought. However, for the implementation, there are some challenges left that need further research to overcome them. One challenge might be that the current implementation is not differentiable since the discretization procodure of Algorithm 1 is not differentiable when boundary of a bin is crossed by a data point. To overcome the challenge of the non-smooth issue, smooth approximations of mutual information, such as Belghazi et al. (2018) and Franzese et al. (0244) could be utilized. However, these approximations can bocome computationally too costly since the mutual information estimator model might have to be retrained after every epoch of the prodiction model to approximate the mutual information between input and model deviations. Another challenge is the potential instability in estimating mutual information,
25

---

Published in Transactions on Machine Learning Research (02/2024)
reported in Choi & Lee (202)). Such an inaccuracy in the estimation of the mutual information utilized as loss function could cause divergence of the optimization procedure and hinder the model's capability to extract more deterministic relations differentiating them from unpredictable parts like noise given the input data. Further research for a smooth and computational cheap approximation of mutual information is promising to focus the model training on the deterministic relations encoded in the data to the degree the corresponding architecture able to extract.
Such framework can also benefit applications outside machine learning where parameters of function are optimized to map imput to ground truth data as closely possible. One example is in the field of optimization and data-driven modeling, such as best paramcter fitting Raue et al. (2015); Crouch et al. (2024) since such a loss function might improve the parameter finding with regard to making their value choice more robust against noise. In this regard, the input variable is the time in case the modeling is done with ordinary differential equations (ODEs) or the space a and optionally the time & if the modeling is done with partial differential equations (PDEs). Additionally, we could have some external stimuli as further input variables, such the effect of drugs in biology or the heating of work piece. The corresponding quantities y that are to be described by the model are the model output (ground truth). Thus, we ideally have that, e.g., ftt, u) approximates y as well as possible where f solves the underlying system of differential equations. With such an approach, we can test if is predictable given the input data at all. After that, we can see if the underlying model (consisting of, e.g., ODFs or PDFs) fits the data, meaning the model is able to capture all the deterministic relations within the data or if the model should be extended, egg., with further interactions between the system states, to capture the information hidden in the data. The successful extraction is the case if cannot reject the hypothesis that the input variables and the residuals are independent.
References
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. arXiv:1607.066 06450,064,,0000 2016.
Layer normalization. arXiv preprint
Philipp Becker, Harit Pandya, Gregor Gebhardt, Cheng Zhao, C. James Taylor, and Gerhard Neumann. Recurrent kalman networks: Factorized inference in nigh-dimensional deep feature spaces. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 544 552. PMLR, 09 15 Jun 2019. URL h mlr. grress/v97/beckert 1atmer199.htm1
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference on machine learning, pp. 531 540. PMLR, 2018.
Alexandra Bezbochina, Elizaveta Stavinova, Anton Kovantsev, and Petr Chunaev. Enhancing predictability assessment: An overview and analysis of predictability measures for time series and network links. Entropy, 25(11):1542, 2023.
Philemon Brakel and Yoshua Bengio. Learning independent features with adversarial nets for non-linear ica. arXiv preprint 1rXic:1710.0000 2017.
Tim Breitenbach. On the SQH method for solving optimal control problems with non-smooth state cost functionals or constraints. Journal of Computational and Applied Mathematics, 415:114515, 2022.
Tim Breitenbach, Lauritz Rasbach, Chinguang Liang, and Patrick Jahnke. A principal feature analysis.
Journal of Computational Science, 58: 58:101502, 2022.
Tim Breitenbach, Bartosz Wilkusz, Lauritz Rasbach, and Patrick Jahnke. On a method for detecting periods and repeating patterns in time series data with autocorrelation and function approximation. Pattern Recognition, 138:109355, 2023.
Rajkumar Buyya, Satish Narayana. Srirama, Giuliano Casale, Rodrigo Calheiros, Yogesh Simmhan, Blesson Varghese, Erol Gelenbe, Bahman Javadi, Luis Miguel Vaquero, Marco A. S. Netto, Adel Nadjaran Toosi,
26

---

Published in Transactions on Machine Learning Research (02/2024)
Maria Alejandra Rodriguez, Ignacio M. Llorente, Sabrina De Capitani Di Vimercati, Pierangela Samarati, Dejan Milojicie, Carlos Varela, Rami Bahsoon, Marcos Dias De Assuncao, Omer Rana, Wanlei Zhou, Hai Jin, Wolfgang Gentzsch, Albert Y. Zomaya, and Haiying Shen. A manifesto for future generation cloud computing: Research directions for the next decade. ACM Comput. Suru., 51(6):100:11100::38 November 2018. ISSN 0360-0300. doi: 10.1145//241133. URL http:://diic..// acc. org/ 1otg/10.100 1144/3343332477
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: In- terpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rmn encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 1724. Association for Computational Linguistics, 2014.
Kwanghee Choi and Siyoong Lee. Combating the instability of mutual information-basod losses via regular- ization. In Uncertainly in Artificial Intelligence, pp. 411 421. PMLR, 2022.
Samantha AW Crouch, Jan Krause, Thomas Dandekar, and Tim Breitenbach. Dataxflow: Synergizing data- driven modeling with best parameter fit and optimal control an efficient data analysis for cancer research. Computational and Structural Biotechnology Journal, 23:1755- 1772, 2024.
Pawel Czyz, Frederic Grabowski, Julia E Vogt, Niko Beerenwinkel, and Alexander Marx. Beyond normal: On the evaluation of mutual information estimators. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:/ httpp:///ppaareiiivvvee//roootte www..ee/foormnddd2b5555o5o
Georges A Darbellay and Igor Vajda. Estimation of the information by an adaptive partitioning of the observation space. IEEE Transaclions on Information Theory, 45(4):1315 1321, 1999.
Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn Keogh. The uer time series archive. LEEE/CAA Journal of Automatica Sinica, 6(6):1293-1300) 2019.
Giulio Franzeso, Mustapha BOUNOUA, and Pietro Michiardi. MINDE: Mutual information neural diffusion
cstimation. In The Twelfth International Conference on Learning Representations, 2024. URL https:
//openreview Setenrrviee.nnt/Eeee httt//oorummii-ttdddsss...
Robert G. Gallager. Stochastic Processes: Theory for Applications. Cambridge University Press, 2013. doi: 10.1017/C0097813666614
Weihao Gao, Sewoong Oh, and Pramod Viswanath. Density functional estimators with k-nearest neighbor bandwidths. In 2017 IEEE International Symposium on Information Theory (ISIT), pp. 1351 1355, IEEE, 2017.
Hans-Otte Georgii. Stochastik: Einfüührug in die Wahsscheinlichheestthhriie und Statistik. Walter de Gruyter GmbH & Co KG, 2015.
Albert Greenberg, James R Hamilton, Navendu Jain, Srikanth Kandula, Changhoon Kim, Parantap Lahiri, David A Maltz, Parveen Patel, and Sudipta Sengupta. V12: scalable and flexible data center network. In Proceedings of the ACM SIGCOMM 2009 conference Data communication, pp. 51 62, 2009.
Priscilla Greenwood and Michael S Nikulin. A guide to chi-squared testing, volume 280, John Wiley & Sons, 1996.
Arthur Gretton, Ralf Herbrich, Alexander Smola, Olivier Bousquet, Bernhard Schölkopf, et al. Kernel methods for measuring independence. 2005.
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A Roberts. The unrea- sonable ineffectiveness of the deeper layers. ar preprint arXiv:2403. 17887, 2024.
27

---

Published in Transactions on Machine Learning Research (02/2024)
Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Represenlations, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778 2016.
R Devon Hjelm, Alex Fedorov, Samuel Mavoie-Marchildon Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019. URL bttpp:://ppenreviee.nt.t// id=Bklr3jOcKX
Sarah E Houts, Shandor G Dektor, and Stephen M Rock. A robust framework for failure detection and recovery for terrain-relative navigation. Unmanned Untethered Submersible Technology, 2013.
Paul Jeha, Michael Schneee-e Pedro Mercado, Shubham Kapoor, Rajbir Singh Nirwan, Valentin Flunkert, Jan Gasthaus, and Tim Jamuschowski. Psa-gan: Progressive self attention gans for synthetic time series. In The Tenth International Conference on Learning Representations, 2022.
Kenji Kawaguchi, Zhun Deng, Xu Ji, and Jiaoyang Huang. How does information bottleneck help deep learning? In International Conference on Machine Learning, pp. 16049 16096. PMLR, 2023.
Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations, 2021.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Leurning Representations, 2019.
Marcel Kollovich, Abdul Fatir Ansari, Michael Bohlke-Schneider, Jasper Zschiegner, Hao Wang, and Yuyang Bernie Wang. Predict, refine, synthesize: Self-guiding diffusion models for probabilistic time series forecasting. Advances in Neural Information Processing Systems, 36, 2024.
Lyudmyla F Kozachenko and Nikolai N Leonenko. Sample estimate of the entropy of a random vector. Problemy Peredachi Informatsit, 23(2):9 1987.
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical review E, 69(6):066138, 2004.
Yan Li, Xinjiang Lu, Yaging Wang, and Dejing Dou. Generative time series forecasting with diffusion, denoise, and disentanglement. Advances in Newral Information Processing Systems, 35:23009 23022, 2022.
Zhe Li, Shiyi Qi, Yiduo Li, and Zenglin Xu. Revisiting long-term time series forecasting: An investigation on linear mapping. ar preprint 2Xx::3305.1072. 2023.
Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Exploring the stationarity in time scries forecasting. Advonces in Neural Information Processing Systems, 35:9881-9893, 2022.
Zechun Liu, Changsheng Zhao, Forrest Iandola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, et al. Mobilellm: Optimizing sub-billion parameter language models for on-device use cases. arXiv preprint arXiv:2402.14900 2024.
Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The m4 competition: 100,000 time scrics and 61 forccasting methods. International Journal of Forecasting, 36(1):54-74, 2020.
Mary I. McHugh. The chi-square test of independence. Biochemia medica, 23(2))143-1444 2013.
Long Short- Memory. Long short-term memory. Neural computation, 98) 10001-188088,,
1780, 2010.
Young-Il Moon, Balaji Rajagopalan, and Upmanu Lall. Estimation of mutual information using kernel density estimators. Physical Review F, 52(3):2318, 1995.
28

---

Published in Transactions on Machine Learning Research (02/2024)
Kevin P Murphy, Probabilistic machine learning: an introduction. MIT Press, 2022. Available at https:
Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In The Fleventh International Conference Learning Repre- sentations, 2022.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXie:1807.037733 0374,, 2018.
Hanchuan Peng, Fuhui Long, and Chris Ding. Feature selection based on mutual information criteria of max- dependency, max-relevance, and min-redundaucy. IEEE Transuctions on poltern analysis and machine intelligence, 27(())1226- 1238, 2005.
Calyampudi Radhakrishna Rao. Linear statistical inference and ils applicalions, volume 2. Wiley New York, 1973.
Andreas Raue, Bernhard Steiert, Max Schelker, Clemens Kreutz, Tim Maiwald, Helge Hass, Joep Vanlier, Christian Tönsing, Lorenz Adlung, Raphael Engesser, et al. Data2dynamics: a modeling environment tailored to paramcter estimation in dynamical systems. Bioinformatics, 31(21):3558-35 35600, 2015.
lan Reid and Hilary Term. Estimation ii University of Oaford, Lecture Notes, 2001.
Andrew M Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan Tracey, and David D Cox. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124020, 2019.
Robert E. Schapire and Yoav Freund. Boosting: Foundations and Algoritims. The MIT Press, 2012. ISBN 0262017180.
Vaisakh Shaj, Saleh GHOLAM ZADEH, Ozan Demir, Luiz Ricardo Douat, and Gerhard Neumann. Multi time scale world models. In Thirty-seventh seventh Conference on Neural Information Processing Systems, 2023. URL https::/ http://www...eeeeeoooooeeeettaaa
Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estima- tors. In International Conference on Learning Representations, 2020. URL https://openrevlee net/
Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. Csdi: Conditional score- based diffusion models for probabilistic time series imputation. Advances in Newral Information Processing Systems, 34: 14804-24811 2021.
Michael Ulbrich. Semismooth Newlon methods for varialional inequalilies and constrained oplimization problems in function spaces. SIAM, 2011.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukaszz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Mor- cos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Interna- tional Conference on Machine Learning, pp. 23965-23998. PMLR, 2022.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34: 00:199224439 2021.
Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. L dmi A novel information- loss function for training deep nets robust to label noise. Advances in neural information processing systems, 32, 2019.
29

---

Published in Transactions on Machine Learning Research (02/2024)
Jinsung Yoon, Daniel Jarrett, and Mihacla Van der Schaar. Time-scries generative adversarial networks. Aduances in neural informalion processing systems, 32, 2019.
Xinyu Yuan and Yan Qiao. Diffusion-TTS -S:: Interpretable diffusion for general time series generation. In The Twelfth International Conference Learning Representations, 2024. URL htttps:/
4orum?id=4h1app0099
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 11121-11128, 2023.
Ray Zhang, Nicholas F Lahens, Heather Ballance, Michael E Hughes, and John B Hogenesch. A circadian gene expresion atlas in mammals: implications for biology and medicine. Proceedings of the National Academy of Sciences, 111(45):16219114) 162244, 2014.
Xiyuan Zhang, Xiaoyong Jin, Karthick Gopalswany, Gaurav Gupta, Youngsuk Park, Xingjian Shi, Hao Wang, Danielle C. Maddix, and Bernie Wang. First de-trend then attend: Rethinking attention for time- series forecasting. In NewrIPS '22 Workshop on All Things Attention: Bridging Different Perspectives on Attention, 2022. URL https: https://opennreviot http://pearrrii.i...taoooooaaaaa
Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for mul- tivariate time series forecasting, In The Eleventh International Conference on Learning Representations, 2022.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. In- former: Beyond efficient transformer for long sequence time-series forecasting, In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 11106-11115, 2021.
30

---

Published in Transactions on Machine Learning Research (02/2024)
A Further explanation on discretization and empirical probability estimation
This section aims at providing further explanation on the discretization schema and the details of the calculation of the empirical marginal and joint probability functions. We exemplify the procedure how we compute these entities based on Algorithm I with our time series data.
We first consider sufficient number of input-output windows depicted in Figure Da each having the same total length, which is the sum of the input length and the output length. In each window, the ith point in the input is considered as the random variable I;, while the jth point in the output is treated the random variaale Øj. This is illustrated in Figure 9b For each window, we get one realization of each random variable 24 and yy-
(a) Several input-output windows randomly selected from dataset.
Input Context
Oulput Target
Vi
(b) Details of a selected window including input and output modeled as random variables.
Figure 9: Preparation of the data for the statistical tests.
Once we have sufficient number of realizations for each of the random variables, apply Algorithm [ to discretize the co-domain of each random variable. Each bin of the discretized co-domain of random variable defines the event that the value of this random variable is between the boundaries of that bin. Algorithm [ can be viewed as performing adaptive binning and histogram construction for each of the random variables. zk In this context, each event zn; represents the random variable v taking a value within the interval defined by the upper and lower bounds of the 4tth bin after discretization. Consequently, the output of this algorithm inchudes the upper and lower bounds of the bins as well as the frequency with which the random variable i; falls into each bin.
To compute the joint probability distribution of random variables, modeling the input of a model, the output of model, the residuals or the model deviations, we calculate the frequency of the joint events where cach random variable is within a bin. For this purpose, each joint realization is placed into the corresponding 2D cell (or cuboids in higher dimensions where more than pairwise combinations of random variables are considered), whose bin boundarics are predeterminod by the outer product of the marginal bins. This process is illustrated in Figure 1O The probability for marginal and joint distributions obtain by dividing the corresponding frequency by the number of realizations (in this the number of windows).
31

---

Published in Transactions on Machine Learning Research (02/2024)
Boundries eventintervalz:
P1
10
P1
PL
loss
Range variable 22
Boundries of eventiinerval
Range Variable Yj
P2 or leos
Figure Computation of the joint histogram for pairs of random variables. Each joint realization of d and Yj observed across all input-output windows is represented by green square and is placed in the corresponding 2D bin. The boundaries of the 2D bins are determined by the outer product of the boundaries for each random variable. Discretization of each random variable follows the procedure outlined in Algorithm
Note that the marginal distributions are discretized to have equal numbers of data points per bin (where there might be an exception for the last bin according to our realization or if there are many points with the same value), the joint distribution can deviate arbitrarily much from a uniform distribution. In this figure, P1 and P2 are illustrated with different values (p1 10, P2 15) for the sake of generality. Consoquently, each bin of the marginals here contains 10 15 green squares, respectively. However, in our experiments, these values are set to be equal.
B Convergence and bounds on estimation of mutual information
In this section, discuss the convergence property of our approximation of mutual information based Algorithm [ For this purpose, we focus on Darbellay & Vajda (999) which describes convergence result for mutual information based on adaptive discretization scheme of the co-domains of random variables. We show that our method fulfills the same conditions as used in Darbellay & Vajda (1999). These conditions are that the discretization scheme decouples the process of decreasing bin width and approximation of the true density with the empirical one by increasing the samples in each bin. Once we have shown that our approach fulfills the same conditions, we can follow the reasoning Darbellay convergence of the mutual information based on our framework to the true mutual information between random variables. For this rigorous analysis, have to equip algorithm by further threshold keeping bin at a minimal width. In this case of reaching the minimum bin width, the condition of the minimum number of points per bin is suspended for this bin. These two further items are used for the theoretical consideration to apply it several times for the convergence process, however, do not come into action for its application.
Vajda (1999) prove
First, we show how we can ensure the decreasing bin width by including more samples. The data, prediction and therefore residuas/moode deviations from the data finite. The reason for the boundedness is that the
32

---

Published in Transactions on Machine Learning Research (02/2024)
model is contimuous function and thus bounded input is mapped to bounded output. Consequently, these entities constitute our random variables, the marginal and joint distributions are of bounded support. Therefore, adding more data points would lead to an arbitrarily smaller bin width when a fixed amount of data points per bin is considered, unless the marginal probability is exactly zero. However, in this case the joint probability also zero resulting in the fact that. these events do not contribute to the total mutual information. Consequently, we can discard these bins and without loss of generality, we require non-zero probability density within the domain. By iteratively including more data points, our discretization scheme provides decreasing bin width until the minimum bin width is reached.
The next requirement for applying the reasoning of Darbellay & Vajda 19999) is to show that putting more samples increases the data points per bin to approximate the true probabilities by the empirical ones to any arbitrarily small error. Due to the action of the minimum bin width, putting more data points solely increases the number of data points per bin since we have a non-zero probability. Consequently, the minimum bin width ensures a docoupling of the limit process of getting finer discrctization of the domains of the probability functions and more data points per bin to make empirical distributions converge to the true ones.
We remark that the main focus of our work is to show that measures quantifying the stochastic independence between input and output, and thus residuals or model deviations, are useful to monitor model convergence. Examples are the chi-square test. of independence or the mutual information. In particular, for the mutual information there are many different implementations that have their pros and cons. For some overview, please have look into (Czyż et al. 2023 Appendix), including convergence results for the estimators.
C Extra notes on Subsection 4.2
During the training of models in experiment 4.2 we observed an intriguing phenomenon. Specifically, there exists a model trained with an L function that not only performs better in the Le-metric (MSE) but also achieves comparable performance in the -merri (MAE) to model trained with an L function on the validation set. Under normal circumstances, one would naturally prefer the 1.2-los--triined model in such a scenario.
However, our analysis using the proposed framework reveals that, even in this situation, the tross-nd model is superior. To validate this finding, we examined the distance to the true underlying relationship, which is sinusoidal function in this experiment. As shown in Figure 11 our framework effectively highlights the importance of model selection and demonstrates the advantages of our proposed metric compared to traditional loss functions.
D Data description
Datasets Here is description of the datasets used in our experiments:
(1) ETT Zhou et al. (2021 contains seven features including the oil temperature and six power load feature. ETTh indicates the ETT data with a granularity of 1-hour-level and ETTm indicates the ETT data with a gramularity of 15-minutes-level.
(2) Weather is recorded every 10 minutes for the whole year 2020, and contains meteorological indicators such humidity and air temperature.
(3) Nasdaq dataset consists of 82 variables, including important indices of markets around the world, the price of major companies in the U.S. market, treasury bill rates, etc. It is measured daily, having total of 1984 data samples for each variable. We set the corresponding input length as 60 similar to Kim et al. 2201). In this work, we conduct our experiments on DE1 variable.
(4) FlectricDevice is a subset of the UCR time series classification dataset. It consists of seven different classes, each representing a specific type of electric device which is to be predicted from the time series.
https://wwx.bge jenaaa--jjnanpeeeeee mpg....pp.... de/vetter/
33

---

Published in Transactions on Machine Learning Research (02/2024)
Validation
La Validation
0.9
My M/M yon Ma Mama Mmm
M
0.85
0.19
0.8
0.75
/n Epoch
0.36
MM Epbch
0.7
Remaining Mi Between Input Residuall Validation
Norm Between Prediction Sinusold Validation
YLPL2 MLPL3
MUML MUPL
0.38
0.36
W M
0.8
0.08
0.06
Epoch.
Epoch
0.04
80
Figure 11: Curves showing mutual information as a stochastic measures of independence between input and residuals, along with the loss function history. Traditional metrics such as L'- and Lř-distance, when measured on noisy validation data, are not as indicative as mutual information (MT) in identifying which model closer to the underlying clean relationships in the data (e.g,, the sinusoidal pattern).
(5) DistalPhalonnsOuuiineCrrrr a subset of the UCR time series classification dataset. This dataset focuses on the classification of outlines of distal phalanx bones from time series.
E Model stacking architecture and numerical experiments
With the stacking of models, we provide further evidence that information left between input and model deviations be used to correct the corresponding predictions.
E.1 Stacking architecture
Different properties of a model may influence its capability of extracting deterministic relations. Conse- quently, this part, we provide an architecture combine different. models with different properties systematically extract deterministic relations. Roughly, models are stacked together where all models get the same input, however, try to learn only what the models in the stack of models far have not extracted regarding deterministic relations.
We need to differentiate two cases. The first case is where the target is ordinal data. Here the random variables modeling the deviations the model output from the ground truth are defined by the difference between the model output and the ground truth. Tn this case the model deviations are termed as residuals. In case where the target nominal data, the random variables that model the deviations of the model are defined follows. In case the prediction of the model is correct, the random variable modeling the deviation of the model from the ground truth takes the value of the ground truth class label. In case the model is right, the corresponding random variable takes value that does not represent model class, e.g. a negative integer.
Next, we explain the stacking procedure in detail for both cases. We are given the data ((1,, n., i E I} where v, is the voctor-valued input, Yi the corresponding rectorvalled output and finite subset of the natural numbers N. With and y, we denote the corresponding vectors of random variables that take the corresponding values for given i. First, check if and y are stochastically dependent each other. If
34

---

Published in Transactions on Machine Learning Research (02/2024)
g
Model 3
Ay²
g + Ay
Model 2
Ay
Model
27
Input
Figure 12: Stacking architecture for problems with ordinal target data.
yes, we train the first model/part of the model stack best fit the prediction y to y given z. With y, we denote a (vector-valued) random variable taking the values Tt for the corresponding T.
For ordinal larget data, the procedure looks as follows. If Ayy y - is not independent of x, there is still some information left that can be extracted. Consequently, we fit model that may have properties different to the current model to learn these relations between x and the residuals Ay In other words, the purpose of the second model on top of first model is to correct the prediction of the first model. The output of the two models is then the prediction of the first model 7 plus the correction Ayy In the next iteration, we test the residuals Ay² = y - y Ay for stochastic dependence on æ. This procedure can be repeated until the corresponding residuals are stochastically independent of z. We call this procedure stacking of models and can be generalized as follows such that a new model top of stack learns to correct the prediction of the previous stack. The procedure is illustrated in Figure 12
To generalize, we define
Ay = Ay
where Ay" = for any j E N. In a purely deterministic scenario, we would extend the stacking until Ay' = 0. In a real scenario where there are unpredictable parts in the data, our definition of no further improvement possible that Ay' is stochastically independent of x based on statistical test or measure. The random variable representing the output of the total stack is denoted with
A K=U
taking the corresponding values V. upon the input I, for E 1, where j is the smallest number such that Ay" is stochastically independent of z. Consequently, the stacking stops y ÿ is independent of x. This definition also works for discrete ordinal data, where the differences, resp., residuals take only discrete values. We remark that this architecture does not require more inference time since each model of the stack does not depend on the output of other layers but all perform their inference the same input data and thus can be run in parallel.
For nominal target data (the difference between class labels has no meaning), the stacking architecture works analogously except the definition of the deviation of the model from the ground truth is different, please compare with Figure [3] The deviation of the model output from the ground truth is defined by a random variable 03, j E N that take the value of the correct class in case below and a value that does not represent a discrete class of the ground truth in case the prediction of the model/stack below is correct. If 01 is independent of the input & for one j, we can stop the stacking of more
wrong prediction from the model/stack
35

---

Published in Transactions on Machine Learning Research (02/2024)
Decision module
Model 3
g
Decision module
Model 2
0
y
Model
20
Input
Figure 13: Stacking architecture for problems with nominal target data.
models. As long as 0 is dependent on the input H, where 0² is based on the predictions of the first model, there is a deterministic relation that another model can learn to predict øú, which models correction to the prediction of the model/stack below. In such case, the stack is extended by another model predicting 65, Based on the prediction & of the model/stack below and the prediction for the corresponding 63 we can decide during inference which prediction to take. In case of @ equaling a class label, we take the value of () predicted by the corresponding model subsequent upstream in the stacking architecture as the value for g Otherwisc, i.c. A equals a number not representing class label, the value of g'+ equals the one of g'. The final prediction of the stack is denoted with y, which represents the vector-valued case well. We remark that the co-domain of the random variable @1 is not one-hot encoded but the models output j should be due to the nominal character. However, in terms of the random variable 0, since there is a bijection between the co-domain of this random variable and the corresponding one-hot encoding, the information content or statistical independence between the random variable 6 with its one-dimensional co-domain consisting of integer numbers and the input also exists and is supposed to be equal to the one-hot encoded version of 02,
One greedy implementation of both stacking concepts above to combine models randomly and check that after each new model Ay' or l has less information with the input than Ay²-1 0 has with the input to ensure benefits of the new model top of the stack. This is test that the new model successfully extracts remaining information and doesn't do guessing rather than really filling in what is missing in terms of deterministic relations and thus provides a well-working correction.
E.2 Stacking of models systematically extracts information and improves prediction
We have seen in this work that the loss function has an influence on the information extraction depending on the noise, see in particular Section 4.2 We investigate in this section how different model properties, like the architecture or hyperparameters, contribute to capturing different kind of information and how such differences could be systematically combined to extract all available deterministic relations in a dataset. We present a general architecture that is supposed to combine the capabilities of different models as described the Soction E.1 Further, this scction showcasing the function of the stacking architecture provides evidence that dependent input and model deviations indeed contain information to correct the output of the former model.
We showcase the efficacy of our framework with real-world time-serics datasets. The datasct is Nasdaq datasets taken from the UCI repository and the M4 competition dataset Makridakis et. al. (2020). More specifically, we take the variable DE1. We train models according to MLP and Nonstationary Transformer (NSTs) Liu al. (2022) architecture on the dataset that is split according to the ratio of 0.7/0.3 into a training and test set. The input of the models are the past 60 time lags and the output is the next value in the time series (singlestep prediction). To this end, an MLP model trained with T.-lloss is chosen as the
36

---

Published in Transactions on Machine Learning Research (02/2024)
Metries
Init MI
Init Perm
Init diff
Res MT
Res Perm
pv
Res diff
Init Chi-square
Res Chi-sauare
MLP L1-0.5
23.038
4.167
18.871
5,256
4.617
0,6392
60
MLP L.-0.5
23.038
4.167
18.871
5.347
4.579
0.7679
60
stacked-0.5
23.038
4.167
18.871
4.843
4.564
0.06
0.2798
60
2nd stacked model-0.5
5,256
4.617
0.6392
4.483
4.564
0.06
0,2798
2
MI.P LI
23.038
4.167
18.871
5.262
4.586
0
0.6758
60
MLP L2
23.038
4.167
18.871
5.414
4.533
0,8814
60
Stacked MLP
23.038
4.167
18.871
4.955
4.579
0.01
0.3765
60
2nd stacked model
5.262
4.543
0.7182
4.956
4.579
0.01
0.3765
4
NST L1
23.038
4.167
18.871
5.267
4.577
0
0.6901
60
NST L2
23.038
4.167
18.871
5.468
4.564
0.9038
60
Table 2: Comparison of performance metrics for various standalone and stacked models. Initial mutual information (MJ), permutation analysis values, and their differences are presented, providing insights into the starting states. Residual metrics, including mutual information, permutation values, and the corresponding p-values in the mutual information framework assessing the independence of input and residual output, are also reported. Additionally, chi-square values for both initial and residual states are included indicating the number of correlated input lags of the time series.
first model and another MLP trained with L is chosen as a second model in the stack. The weights of the last layer of the models in the stack, except the first model, are initially set such that the output of each model is close the zero. The rationale is that the prediction from the stack below correct, only minor corrections are necessary building on the previous predictions. Furthermore, the last layers in models are chosen for weight rescaling since the first. layers are usually intended for feature extraction. Additionally, the layer norm operation (dividing by standard deviation) would cancel the scaling to small values, for details about this part please see the Appendix, Section L.4
As illustrated in Table 2 none of the individual models successfully render the residuals independent of the input. Remarkably, it was only through the combination in stacked models that an increase of p-values was observed, enhancing the overall performance. To provide a more comprehensive comparison, results for NST are also included. We see that in this case, the MLP stack not only extracts more information than the NST but is computationally even cheaper. In order to exclude that the effect. is result of more free parameters, we have included MLPs with about the half of free parameters each indicated by the '0.5'. Moreover, we see that only the stacked models provide non-zero p-value such that only in this case, we cannot reject the hypothesis that imput and residuals are independent based on a level of significance of 1%. Beyond mutual information and chi-square test, considcrations such as L and learning curves in Figure further support the empirical evidence that stacking models outperforms their individual counterparts by having a smaller T,2-loss function (comparison only valid if the last layer of the stack is trained with the same loss functions, which in this case L function, as the corresponding single model). This evidence showcases the capacity of multiple models to learn diverse aspects. However, remark that comparison in terms of loss functions and information extraction is tricky as shown in Section
4.2
We remark that a linear combination of loss functions according to A1ll Al eters Å1. 2 > could be an alternative approach to stacking in terms of loss functions. However, the success of that formulation might depend on the right choice of the hyperparameters A and d2. Our framework provides an option to choose the hyperparameters accordingly such that most information is extracted from the data, e.g., the mutual information is minimized between the input and the residuals. Please note that finding the best combination or any hyperparameter optimization is not the focus of this work. In the present work, the focus is on showcasing our framework in terms of its potential for applications in various MI. use cases.
with the hyperparam-
Next, we show that our stacking framework also works for classification problems. For architectural details, please see Scction E.T In Table 3 we provide numbers based on the ElectricDevices dataset. To further improve the effect of stacking, we see potential when including a corresponding loss function into the training process for optimizing parameters such that the corresponding MT. architecture extracts most deterministic
37

---

Published in Transactions on Machine Learning Research (02/2024)
L2 Train Loss
on MLPP-L1 MLP+
0.8
0.5
0.4
0.2
Epoch
200
L2 Validation Loss
NS
NST L1
MLP + L2
MLP L2 stacked on MLP - MLP
0.8
0.5
0.4
0.2
Epoch
0
100
200
300
Figure 14: The 1,2-loss comparison for stacked models and single models.
relations possible. Regarding this topic, please see the Conclusion and Future Work scction in combination with the Discussion part 'Allernative of model deviation for nominal data".
Metrics
Init MI
Init Perm
Init diff
Res MI
Res Perm
Res diff
SVM-rbf kernel
41.65
16.00
25.65
23.72
15.50
8.22
SVM-sigmoid kernel
41.65
16.00
25.65
34.13
16.62
17.51
SVM-sigmoid SVM-rbf
41.65 (23.72)
16.00 (15.50) 25.65 (8.22)
23.23
14.41
7.82
Numbers parentheses show the starting point of the last stacked model. The abbreviation SVM refers to the standard Python sklearn implementation of a support vector machine. The p-values assessing the independence of the input and the model deviation from ground truth alsways remains zero in all experiments. For description of the meaning of the columns, please refer to Table 4
Table 3:
The results the next experiment demonstrating 1-step ahead prediction with the stacking procedure is provided in Table for the Weather dataset, see Subsoction D The split setting from Nie al. (2022) used here. Except the prediction length, we use the same training parameters and architecture for PatchTST used in Nie et al. (2022).
38

---

Published in Transactions on Machine Learning Research (02/2024)
Metrics
Init MI
Init Porm
Init diff
Res MI
Res Perm
Res diff
Palc (0.4MM)
3213
153
3060
482
MLPLI (0.59M)
3213
153
3060
514
174
340
NSTL1 8.8
3213
153
3060
608
182
426
NSTLI(L.05M NSTL2(0
3213
153
3060
641
183
458
3213
153
3060
182
518
(. P
3213(659)
153(177)
482)(482)
506
181
325
(. PaTTIILIBB
3213(659)
153(177)
(060(482)
183
333
3213(514)
153(171)
3060(343)
400
181
219
(STLII0.6M))+MPPP
3213(514)
153(171)
3060(343)
441
182
259
3213(514)
173(171)
3060(343)
442
260
NSTT:122.66//-MMP111000000 (STTLLLLL. PatchTSTL2 (0.4IM)
3213(514)
153(171)
3060(343)
448
182
266
3213(516)
183(153)
3060(363)
317
Comparison of performance metries for various standalone and stacked models on weather dataset. Initial mutual information (MI), permutation analysis values, and their differences presented, providing insights into the starting states. Residual metrics, including mutual information, values from the permutation test of mutual information are also reported. The corresponding p-values assessing the independence of input and residual output is always 0 in all experiments. Additionally, in the first column, number of parameters for the models is shown in parentheses in millions. In the other columns, the values of the metrics only for the last model of the stack is depicted in the parentheses. In the first column, the model on the left is the first one and the one on the right is the last model in the stack.
Table 4:
In the next experiment, demonstrate framework for multistep prediction. We take the Nasdaq dataset from Kim et al. (2021). We choose the input of length 60 to predict target length of 30. The result is provided in Table and shows that also in this case, with stacking of models, we can systematically extract the information and gradually make the input independent of the residuals in contrast to the single models. The evidence is provided by the fact that a stack of MLP and NST models provides the smallest mutual information between input and residuals when subtracting the mutual information generated by randomly shuffling the data (column "Res diff' of Table 5
Metries
Init MI
Iuit Permi
Init diff
Res MI
Res Perm
Res didf
Inis Cli square
Res Cli
MLP LHZIIM
617.48
120.22
$94.19
100.45
196.11
30.34
181M
3532
017.48
135.30
MLF L2 (46TM)
617.48
123.20 29
494.19
162 82
135.47
27.35
1800
177
NST L1(2-67M)
617.48
183.20
136.15
NST L2/2,6MM)
617.48
123. 29
494.19
179.47
136.28
44.19
1881
729
MIPP L2N0677M)
LIZZOM)
617.48(116945)
494.19134.00
157.91
135.28
22.63
3800)332)
SILPL20GMM) -NGGMAAASSSTLAAA XSTLI26M
(189.22))
120 2812 140, 10)
156.84
196.22
21.02
1800811)
85
MLP L2(0.LM))NST 2N5T222001)
(17938)1)9.37)
28)
194.19
135.25
18001729
PL....00000) (MIPIMM 6777)))
617.48(169.17)
104.10(17.89)
136.27
16.53
1800(30)
Aug Ensembie (3.0M)
617.48
191.19
170.50
196.00
Table 5: Comparison of performance metrics for various standalone and stacked models. Initial mutual information (MT), permutation analysis values, and their differences are presented, providing insights into the starting states. Residual metrics, including mutual information, values from the permutation test of mutual information. Additionally, the number of dependent input lags lested by the chi-square test of independence for both initial and residual states are included. In the first column, the number of parameters for the models is shown in parentheses in millions. In the other columns, the values of the metrics only for the last model of the stack is depicted in the parentheses. In the first column, the model on the left is the first one and the one on the right is the last model in the stack. In the last row, we take the average prediction of the three models of the penultimate row when each of those models is separately trained to predict the original ground truth. The p-values assessing the independence of input and residual output based on mutual information remains always zero in all experiments this table
Relation to ensemble learning: The stacking approach is characterized by progressiveness: We don't assign each model the task of learning the ground truth Wortsman et al. (2022) but rather focus on capturing
39

---

Published in Transactions on Machine Learning Research (02/2024)
what remains unlearned by the stack of previous models. This progressiveness not only contributes to the efficacy of our stacking strategy but also enables each model to focus on specific tasks that are not covered by other models, similar to the concept of gradient boosting. We stack models in a way that the input is given to all models in a stack, and each model attempts to correct the errors left by the preceding models in the stack. In order to enable models to extract different information that the preceding models could not so far, it could be helpful that the models vary in their properties, like the model parameters, the loss function they are trained with or the architecture itself. The similarity to gradient boosting, where the models are trained to predict residuals defined based on the gradient of the loss function with respect to the model output, is that the proposed stacking framework directly uses the model residuals between model stack and ground truth as a reference to train a new model. While for ordinal data, the stacking procedure is very related to gradient boosting with L2~loss, the difference for nominal data is that the stacking procedure runs on the correction of the classification while gradient boosting uses the gradient of the corresponding loss function, like cross-entropy loss. The presented stacking architecture is supposed to showcase our proposed framework to provide further evidence that model output. can be further corrected visualized by measuring stochastic dependence. By considering the stochastic independence of the input and the model deviation, we can set up further convergence criterion to stop adding further models. This is the case if input and model deviations are independent, as adding new models to the ensemble would not extract more deterministic relations as the input does not contain any information about model deviation.
Stacking software pipeline: Apart from finding relevant properties of a model to vary, like size, depth, loss function etc., another aspect is that we assume the stack to be constant once trained and parameters are kept constant while only the new model top of the stack is trained. It is to investigate in future roscarch if, c.g., training all the parameters of the whole stack after training the new model on top of the stack might benefit the accuracy before testing if another model for the top of the stack is needed. One important application of a pipeline is to facilitate precise time series prediction related to a concrete single time series and provide this capabilitics to a broad audience even outside the ML community that apply the predictions of time series, like weather forecasts. Another use case is improving therapies where their effect depends on time-varying patient-specific parameters. Thus, by laking, e-g.; the daily rhythm of gene expression of humans into account, as argued in the concept of chronotherapy ZZhang et al. (2014), a precise prediction of the expression levels may improve the effect of therapics and can be one brick for personalized medicine.
F A remark on the sensitivity of loss functions regarding distracting the model from the ground truth
In Section 4.2 we see the T.'sloss function is less prone to the asymmetric noise than the I.2-loss function. Since - function weights all deviations from the real data with the same penalty, we formulate the hypothesis that losso function might perform even better than L -losos function. However, since the -º_ loss function is discontiiuous and thus not differentiable at all, a lack of numerical efficient optimization algorithm capable to deal with the discontinuity of the loss function might hinder its broader application. Consequently, a starting point for further research might be to analyze the performance of loss functions with regard to information extraction that consist of parts cutting off bigger deviations with, e.g., min (min(max( V, -.) > 0, which can be solved with semi-smooth methods, soe, e-g., [ubrich] (2011) or as shown in Breitenbach (2022) by transforming the corresponding optimization problem into a higher dimensional one to resolve the min- and max-function to differentiable functions. Such loss function, e.g., taking the absolute value or the square of the projection min(max(ū (oanjjj-.. -)), could be tradeoff between numerical efficiency and robustness against asymmetric noise or outliers paramcterized by the parameter т. Starting the learning with a big T and restarting the optimization with a smaller 7 with the result from the last optimization procedure or decreasing 7 within one optimization run could also accelerate the convergence speed. This procedure could make the prodiction more precise with regard to extracting the deterministic relations assuming that bigger determinations come (mostly) from noise given the input data. Our framework can monitor the effect of with regard to extracting the deterministic relations.
40

---

Published in Transactions on Machine Learning Research (02/2024)
G
Models' settings for numerical experiments
This section of the appendix is allocated to the architecture of the utilized neural networks (NNs) in the experiments. Through this appendix we show the architecture of the MLPs with the number of nodes per each layer inside list. The number of layers is the same the length of the list. Unless specified differently, all activation functions are Relu and initial learning rates are 1e-4.
Section 4.1
All NNs MLPs with Relu activation functions. MLP Layers: (99,490,980,1]
Activation functions: Relu
Initial learning rate: le-4
0.506M parameters
Section 4.2
All NNs MLPs with Relu activation functions, Number of nodes in each layer is written in the list. MLP Layers: 149,490,700,490, Activation function: Relu
Iniiial learning rate: le-4 0.712M parameters
Figure Timesreise Classification
MLP CrossEntropy (0.84M)
MIP Layers: [80, 720, 720, 360, 2]
Figure Timesreise Classification
NST CrossEntropy (0.69M)
Number of encoder layers: 2
Number of decoder layers: 1
Number of heads: 8
d model: 128
Dropout: 0.1
MLP CrossEntropy (0.85M) MLP Layers: [96, 720, 720, 360, 7]
Table 2
MLPL1-0.5 (0.025M) & MLPL2-0.5 (0.025M):
Layers ob both models: [60,80,120,80,1] Activation function: Relu
Initial. learning rate: 1c-4
MLPLL(0.05M) & (LPL2(0.05M)) Layers for both models: [120,80, 180
NSTLI (0.05M) NSTL2 (0.05M)
Number of encoder layers:
Number of decoder layers: 1
Number of heads: 2
41

---

Published in Transactions on Machine Learning Research (02/2024)
d model: 40 Dropout: 0.1
Table 4
Here the details of the pool the used models -MLPs and Transformers- one step ahead prediction experiment on weather dataset. For all non-stationary transformers (NSTs) dropout to 0.1.
The initial learning rate for all models the first stack is 1c-4 and for the second and the third stack is 1e-5.
The dropout for NSTs is 0.1 and for PatchTsT is 0.2, and there is no dropout for MLPs.
The size of subsequent hidden layer after the attention head (d ff) in transformers are set to the provided default numbers, i.e. #=d_modee for NSTs and for @rd_mode for PatchTST.
Please note that PStchTST uses the vanilla Transformer encoder as its core architecture Nie al. (2022) and therefore the number of decoder layer is zero.
PStchTST L2(00.MMM
PatchTST L2 architecture
Number of encoder layers: 3
Number of decoder layers: 0 Number heads: 16
model: 128
Dropout: 0.2
MLP (1.599)
MLP Layers: [96, 720, 720, 1]
NST L1(8888M)
NST L1 architecture
Number of encoder layers: 2
Number of decoder layers: 2
Number of heads: 8
d_ model: 256
Dropout: 0.1
NST (1(1005M)
NST architecture
Number of encoder layers: 2
Number of decoder layers: 2
Number of heads: 8
d model: 128
Dropout: 0.1
NST (0.86)) on PatchTST L2(0.41MM
NST architecture
Number of encoder layers: 2
Number of decoder layers: 1
Number of heads: 8
d model: 128
Dropout: 0.1
PST architecture
Number of encoder layers: 3
Number of decoder layers: 0
Number of heads: 16
d model: 128
42

---

Published in Transactions on Machine Learning Research (02/2024)
Dropout: 0.2
NST LI(1.13M) on PatchTST L2(0.41M)
NST architecture
Number of encoder layers: 2
Number of decoder layers: 2
Number of heads: 8
d model: 128
Dropout: 0.1
PStchTST architecture
Number of encoder layers: 3
Number of decoder layers: 0
Number of heads: 16
d model: 128
Dropout: 0.2
NST (.2(0.) on MLP L.1 (0.59M) MLP Layers: [96, 720, 720, 1]
NST architecture:
Number of encoder layers: 2
Number of decoder layers: 1
Number of heads: 6
d model: 96
Dropout: 0.1
NST (0.86)) on MLP I, (0.59M)
MLP Layers: [96, 720, 720, 1]
NST architecture:
Number of encoder layers: 2
Number of decoder layers: 1
Number of heads: 8
d model: 128
Dropout: 0.1
NST L1(205) on MLP L1(0.59M) MLP Layers: [96, 720, 720, 1]
NST architecture:
Number of encoder layers: 4
Number of decoder layers: 4
Number of heads: 8
d model: 128
Dropout: 0.1
MLP L1(0.59M) on NST L1(1.13M) on PatchTST (2(0041M) MLP Layers: [96, 720, 720, 1]
NST architecture
Number of encoder layers: 2
Number of decoder layers: 2
Number of heads: 8
d model: 128
Dropout: 0.1
43

---

Published in Transactions on Machine Learning Research (02/2024)
PStchTST architecture
Number of encoder layers: 3
Number of decoder layers: 0
Number heads: 16
model: 128
Dropout: 0.2
Table 3
In this experiment, two simple SVM models are used: SVM with rbf kernel SVM with with sigmoid kernel
Table 5
Here is the details of the pool of the used models -MLPs and Transformers- in multistep ahead prediction experiment on Nasdaq-DE1 dataset.
MLP L1 (2.09M):
MLP Layers: [60,360,3340;244033)
MLP L2 (3.77M):
MLP Layers: (60.720,3440,300.300
MLP L2 (4.61M): MLP Layers: (60,900,3340,,22,330)
NST L1 & NST L2 (2.67M):
Number of encoder layers: 2
Number of decoder layers: 1
Number of heads: 8
d model: 256
Dropout: 0.1
MLP L2 (0.67M) MLPL1 (2.09M)
MLP T.2 Layers: 108,0,0,080,24000,
MIP L1 Layers: 00.360,3440,240,,0
MLP L2 (0.64M) on NSTLI (2.67M):
MLP L2 layers: 600,360,1000,240.0
NST L1 architecture:
Number of encoder layers: 2
Number of decoder layers: 1
Number of heads: 8
d model: 256
Dropout: 0.1
MLP T.2 (0.09M) MLP L.2 (0.64M) NST L.2 (2.67M)
NST L2 architecture:
Number of encoder layers: 2
Number of decoder layers: 1
44

---

Published in Transactions on Machine Learning Research (02/2024)
Number of heads: 8
d_ model: 256
Dropout: 0.1
MLP L2 (0.64M) layers:
MLP 1.2 (0.09M)
layers: (60,720,60,3))
Section 4.5
MLP L1 (0.658IM) on PatchTST
MLP L1 layers: [96 ,7720, 720, 96]
details PatchTsT can be found in Nie et al. 2022
45

---

Published in Transactions on Machine Learning Research (02/2024)
H Further tables and figures
Without Noise: residuals are magnified by 10000
Amplitude of Noise 0.2
Amplitude of Noise 0.6
Amplitude of Noise 0.8
Figure 15: Plots of model outputs (prediction), residuals and the data (ground truth) of the experiments of
Section 4.1
46

---

Published in Transactions on Machine Learning Research (02/2024)
MLP-L1
MLP-L2
trd
Teet
pricusion
Trahn
predictor rilia!
pradicionn
Figure 16: Plots of model outputs (prediction), residuals and the data (ground truth) of the experiments of Section 4.2
M Muuf
Figure 17: Plots of model outputs (prediction), residuals and the data (ground truth) of the E1TTh dataset.
47

---

Published in Transactions on Machine Learning Research (02/2024)
Thoorctical bound
Experimental Error
Pearsonr Analysis
Relative noise sld
Normalized Test RMSE
Initial Rt
Residual
0
0.000023
30.764
0.2715
0.2737
28.538
0.4919
0.5043
23. 129
0.6465
0.6687
17.494
0.9446
0. 0.7499
0.7683
13.197 197
0.8161
0.8287
10.455
0.8614
0.8718
7.868
0.8923
0.901
5.561
0.9149
0.9206
4.664
0.9308
0.9427
3.723
0.9429
0.9373
2.882
Table 6: Pearson correlation results for Section 4.1 The test consists of the sum of the absolute value of the Pearson correlation between each input and output features. However, the correlation measure is only considered if the p-value is smaller than 0.01. Otherwise, the corresponding input and output pair is considered as uncorrelated. In the first column, there is the relative noise, which is the root of the variance of the noise divided by the total signal. The second column provides the normalized RMSE defined in Sention 4 The third column is the sun of the absolute value of the Pearson correlation between input and the single step forecast as output and the forth column is analogous to the third column where the single step forecast is replacod by the corresponding residual (model prodiction minus data). Since the rosiduals are less correlated or even decorrelated with the input, the model extracted the deterministic relations measured in the Pearson correlation test.
Initial pearsonr
Residual Pearsonr + L2
Residual Pearsonr Ll
Trial 0
15.33
0
Trial 1
16.925
0
Trial 2
15.25
0
Trial 3
17.104
0
Trial 4 15.449
O
Table 7: The effect of the choice of the loss function in mitigating asymmetric noise effect in terms of distracting a model from extracting/learning the deterministic relations. The experiments are conducted Section 4.2 The Pearson test is described in Table D While the chi-square test and the mutual information test depicted in Figure reflect the better fitting of the model trained with L*-loss function to the real data, see Figure 2 the Pearson test does not, which could be traced back to the limitation of testing only linear correlations where chi-square and mutual information are generalizations in terms of mcasuring stochastic independence.
Transformer architecture explanation
In this section, we explain the transformer architecture in more detail. Specifically, we explain the building blocks as depicted in (Vaswani al. 2017 Figure 1).
1.1 Token embedding
In the transformer model, each token of the input sequence first represented a dense vector called a token embedding. This embedding captures the semantic meaning of the token in the context of the task being performed.
48

---

Published in Transactions on Machine Learning Research (02/2024)
Each token is encoded by vector Rd one-hot encoding the corresponding token where dN N is the number of different tokens generated from the vocabulary. The token embedding is obtained by applying a linear transformation A E RCxd, to the one-hot encoded representation of tokens & where C E N is the number of dimensions of the transformer's internal reprosentation of the embeddings (C sometimes also denoted with embedding size). The entries of A are learnable weights and optimized during the training process. The mapping Rd + RC, 2 H> Az can be implemented as:
liken_embeddingg nn. Li..inaarcoonig.vooo vocab size, config.n_embel)
where config. vocab size = d₂ is the size of the vocabulary (number of unique tokens), and config.n_cmmiee embed is the desired embedding dimension C. Consequently, this mapping turns the input shape (B, d,,) to the out- put shape of the token embedding (B, T, C), where B represents the batch size and T represents the sequence length of the input (number of tokens). The transformation A is applied to each token and element of a batch by 2(b :.:) + :(), for each E {i..., B} where 2 E Rxxxxd, 2(6, :,:) = (2oak) (e J..... ( provides the one-hot encoded representation for each token and each element of the batch. By using batches, several inputs can be considered at once.
For the specific case of time series modeling, the token embedding is replaced by the following. A convolu- tional neural network (CNN) is used for generating the embedding. The input channels of the CNN equal I in an autoregressive sccnario (only historic parts of the time serics itself are use to prodict future parts of the time series) but can be set to any feature number F E N, which measured at each time point, in case, e.g., an output is predicted from several input time series. Each time window for each channel, which cuts out each part of the F time scries of length T and which is used as the input for the prediction, is represented by vector of size T. These vectors from the sliding window
transformed by onc-dimensional CNN into the space CRC The padding is set such that the input length T equals the output length of the CNN. We remark that the shift one by one time points is not necessary and can be increased such that the token number in the embedding is smaller than the number of time points used as the input for the transformer. In our case, each filter (convolution) of the CNN is applied to each output, controoledd by the parameter groups=1 and a common choice for the kernel size of 3. To achieve this transformation, we employ a ono-dimensional convolutional layer in our implementation. Specifically, a ID convolutional layer with an input channel size of F and an output channel size of C (the embedding dimension of the transformer) can be utilized:
nn. Connid((__channels
channels = F, yout_chanmeel = =C)
This convolutional layer applies a set of learnable filters across the temporal dimension T of the in- put data, extracting relevant patterns and features. It's important to note that the kernel size, padding, and stride parameters of the convolutional layer can be adjusted to ensure that the out- put length matches the input length T. For more details, see the PyTorch documentation, c.g. waw.rraeeeeeeoroo..ooooolllll Conv
hllps://pylocch.org/eeee
In general, the transformation (token embedding by a CNN) is applied as follows CNN : RXF RTxC, z(b,:,:) + CNN (z(b,;,:)) to each each clement of a batch =(b,;.:) E BxTXX 2(b.;,:) = (E (I....,),,,,((....... FY, numerated by be (I.... B} with the batch size B E N, (in parallel) generaling a tensor of dimension (B,T,)).
For more details about the general framework of token embedding, see, e.g., Zhou et al. 2021
1.2 Positional embedding
The purpose of positional embedding is to include information about the position of a token relative to other tokens from the input into the total embedding of each token. The positional embedding is a vector of dimension C and is added to the token embeddings to provide information about the relative positions of tokens.
There are several methods available to code for positional information, see, e.g. Vaswani et al. 2017 7
49

---

Published in Transactions on Machine Learning Research (02/2024)
1.3 Attention head operation
The attention head is the essential building block of a transformer. Each atlention head forms one layer where the output of one layer is processed by a subsequent layer until the output of the final layer is processed by a linear layer to obtain a corresponding output, sec, c.g., (Vaswani et al. 2017 Figure 1).
Let's consider a single allention head operation a transformer model. In this operation, we have the input tensor E [Bxxxx The tensor can be the one after the token embedding (inclusive positional encoding) or the output of a previous layer. We remark that each layer in this presentation preserves the format (B.T.C).
The input tensor is next transformed into different representations via linear transformation matrices. These matrices contain the learnable weights and are given by W E RCxe WQ E RCxC and WV E RCxx We define a tensor for key (K), query (Q), and value (V) by the linear mappings
k(b.;,)) = , E TXC
200 E TXC V(b, =0(;)))((b....) WWV E TR
for each b {L, B} where x(6,,,-) E RTxC such that æ(, xbbii,k = Task for all b E {l, B}. i {l, TH and h E (1,..,C}, C}. Each of the matrices WK, Wº and WV is an instance of a linear layer and can be implemented with PyTorch as follows:
nn. Ln..an C, bias = False).
We remark that C may be called embedding size.
In order to quantify the attention of token i E {1,..., T3 represented in its key representation (K) with regard to token j E {1,...,T) T} of the imput represented in its query representation (Q), the dot product is calculated for each E {1,..., B} between the query (Q) and key (K) tensors over the vector embedding for each token pair i (1...., T). This operation can be represented as
Σ + (1,..., B} x {1,..., T3 x {I. R, (b,i,)) = VC k=1
Qbiike Kojjk
(6)
where b represents the batch index, i and j represent the positions in the sequence (input), and h represents the embedding dimension. The sum is scaled by Cr-0.5, One reason behind dividing the sum by the square root of the embedding dimension is given in the section about the Softmax function below, see Section I.3.1
We implement the mapping O using the key (K) and query (Q) tensors and the Einstein summation con-
vention as follows:
(b,,,,)) torch.cinsum bkj- > bik, Q K) VC
1.3.1 Softmax
After calculating the similarities between keys and queries, the purpose of the Softmax function is to nor- malize the scores similarity. A high similarity between the key of token and the quory of token j is a proxy for a high association or attention the token has to token j, meaning the connection is important for predicting the corresponding output. Due to the monotonicity of the Softmax function, a bigger similarity score between the corresponding key and query will result in bigger value, called attention between the corresponding tokens, compared to smaller ones.
The Softmax function in our case is defined by
(b..i.)
Softmax (1...., B3 (1,..., T} x (1...., T) + R, (b,i,)) + Softmax((,,,))
ΣT (bb,..))
50

---

Published in Transactions on Machine Learning Research (02/2024)
Here, the Softmax function is applied along the last dimension, ensuring that the attention weights sum up to I along this dimension. This normalization means, fixing batch number and a token number of the imput provides us a normalized attention score about all the other token numbers j E {I,... The implementation is done by applying the corresponding Softmax function along dim=-1 1 to the tensor br = i,j) for all E B) and i,j E ().
Next, we explain the normalization by C-0.5 of (6). For large numbers, the Softmax function is approximately constant function and changes in the weights of the transformer model do not result in significant change of the attention. Depending on the embedding dimension C, the corresponding sec, c.g., (Vaswani et al. 2017 footnote 4). The scaling of the sum by C-0.5 ensures to stay in a range where the Softmax is in an area of larger steepness and changes in the weights of the transformer result in significant changes the attention values. Similarly, see (Vaswani et al. 2017 Section 3.2.1).
scales. For an illustration,
1.3.2 Weighted aggregation
We apply the attention scores to the value tensor (V) to obtain weighted sum. The attention tensor is defined by
Ans = Softmax(b,,,,)
for all b E (1,..., B} and i,j E (1,..., T) such that A BxTxT, The weighted of attention scores is given by
(V E Txx
for each b (1,..., B} and turns the output of attention head into the tensor format. (B, T, C). The output tensor of attention head H e BxTXxC is defined by
Hb.ik = Σ
Ao.i,
1=1
for all E (1,..., B}, i E (I....?? T} and C}.
1.3.3 Multi-head attention
Optionally, in order to calculate attention on different subspaces of keys and queries for the same input in each layer, there is a multi-head attention taking only projected parts of keys and queries.
In the multi-head attention formalism, the output of cach head is concatenated along the last dimension, which is the embedding dimension
Concat((B,T, C/n heads), (B,T, heads)) = (B,T,C)
where n heads is the number attention heads. Specifically, the output of cach head is calculated with the following weight matrices W E RCCC/n_heads W E @gCxC//_heads W E RCxC/n_heaad and wo RCxC where {1...., n m__hedds]. Furthermore, n. heads and C are chosen such that the quotient C/n heads is an integer. The matrices calculate the corresponding projections of keys, queries and values follows (hbb..,)) k(b TRCX//n_ headd (b.,,)) Q(I W TTTC//n_ heads (h(b,,,) V(b.,-)W TxC/n heads for all [L....,B..
The weight matrices W w and WX are each implemented with a linear layer according to
Ln.err head size, bias False)
and head size is calculated as
head size embedding size
n heads
where for WO the number head size is replaced by C.
51

---

Published in Transactions on Machine Learning Research (02/2024)
Applying the procedure for the single-head attention for each h "..,,,__haddss by replacing each K by the corresponding K", each Q by the corresponding Qh and each V by the corresponding V provides the tensor of each attention head Hh E /B,TC//n heads where in 6 the index in the sum is only over I to C/n_heade cach.
After processing all attention heads, the outputs are concatenated along the last dimension, resulting in a tensor of shape (B,T, C). The output of the multi-head attention is given by
H(b,: -.)) = Concat(H' (b, (,,;;,),.. )),,, H hr__heasspp (b, (b,,,))wwO E RxC
for each b E {I,...,
1.4 Adding and layer normalization
After the attention head operation, residual connections and layer normalization are applied.
The adding of the input and the output of a layer (residual learning He et al. (2016) is crucial for maintaining information flow and is easing the training of deep networks. The residual connection involves adding the input tensor x to the output of the multi-head attention operation H according to + H where + denotes an element-wise addition. The addition helps to loop through the original information from the input to all the layers sequential layer architecture while also incorporating the information learned by the attention mechanism. A prerequisite is that the attention head preserves the input format.
Following the residual learning operation, layer normalization applicd to stabilize the learning process Ba et al. 2016 according to
LayerNorm 2- ((2)
Var(=)
where & is the output the previous layer, E(2) is the mean value of all the values of the output of the previous layer (mean over the elements of =) and Var(z) is the corresponding variance. Since the square root is not differentiable at a small constant >> 0 keeps the numerical implementation stable in case of a small variance of z. Morcover, the constant avoids division by zero errors. More details about the implementation can be found under httpss//pyttrcch. org/ docs/ https///ttblllggggeeatar//rrrroo http///eeeeeeeeee....eo/ mn. LayerNorm, stmllllmmmmll Due to its construction, the normalization is done for each element of the batch.
We remark that for our implementation, the normalization is applied over the embedding dimension (C) separately for each token of the imput (T dimension), meaning that 2 E R This is reasonable since the feed forward networks (explained Subscction [.5] are applied over the d model (C) on each element of the batch (B) and input length (T).
1.5 Feed forward network (FFN)
After the normalization step of the attention head's residual learning, each token's representation is passed through a feed-forward neural werk (FFN). (FFN).Such Such an FFN consists of two linear transformations separated by a non-linear activation function Rmxn -> RRx,, m, n E N, 2 + g(z), such as ReLU (Rectified Lincar Unit; g = max) according to
FFN (x(b,t.)) (x(b,t, + b,) W2 + b2 R
for each b E (1,..., B) and t fl.... .,. T} where E BxTxC the output from the previous operation the architecture, W1 E ROxd1, d3z E N, in our implementation d 4C, W2 E & and by Rdsr, Furthermore, x(b,t, W E Rdi, which is the same bias for each & in contrast to b E RTxd1 where
52

---

Published in Transactions on Machine Learning Research (02/2024)
for each there is another bias, and by RC are learnable parameters. These operations are applied by Pytorch's linear layer https://pytorch.orrg
nn www.Liinar.rhtt html The activation function g introduces non-linearity to the model, enabling it to learn non-linear patterns from the data. In the transformer architecture, an FFN is also followed by residual learning and layer normalization as described above Subsection
1.6 Masking
Looking at (Vaswani et al. 2017 Figure 1), the masking is one of the essential building blocks located within the decoder (explained in Subsection
The masking of values of the attention weights O has the purposes of forcing altention between tokens to zero, meaning not allowing them to interact or to extract information from the interaction. Due to the iterative application of the transformer for text generation, we would like to force the attention mechanism that a token only considers tokens backwards in time (that come earlier in a sentence). This backward orientation helps to generate representations of tokens that collect information from tokens that are already there and prevents generating representations that make use of tokens that come after that token in a sentence. By the procedure of masking, the representations of tokens are more unified independent of the input length. As example: 'I am hungry and thus go a restaurant." Although probably "hungry" should get lot of attention with 'restaurant" without masking, in iterative application of the transformer, the word "hungry" in 'I am hungry and thus I go' would not be useful since its most attention was on restaurant that is not there yet. However, with masking we foree the transformer to find embeddings and representations such that the word "hungry" gets a useful representation to predict the next token during learning, independent if the input is am hungry and thus I go to a restaurant. or 'I am hungry and thus go'.
A different interpretation of the masking can be causality in a use case where the sequence of events is of importance forcing attention only to historic events.
We implement the masking effecting only backwards interaction by a lower triangular mask M E RTXT, This matrix is applied to the attention weights θ generating masked attention weights. The tokens of the input are counted in the second dimension of the tensor θ where the third dimension accounts for the dimension of the current vector representation (T or C depending on the current representation). Consoquently, the lower triangular matrix (1 on the diagonal and below and 0 above), allows token to have a non-zero attention only with token 1, token 2 can interact with token 2 and token and so on until token T.
Subsequently, for each b {1,...,),, B), the entries of the attention weight tensor O(b,:.)) are replaced by -00 where M equals wero. The attention tensor θ is thus transformed into the masked attention weight tensor @M, The -o forces the corresponding Softmax calculation to zero in the corresponding positions, meaning that the corresponding token does not pay attention to the corresponding other tokens.
1.7 Encoder and decoder
In a transformer architecture, there are typically two main components. This is the encoder and the decoder. Next, we explain both components according to /Vaswani et al. 2017 Figure 1).
Usually each layer in the encoder consists of an attention head followed by residual learning and normaliza- tion, which is the input into two linear transformation separated by a non-linear activation function, also followed by residual learning and normalization. For the decoder, a layer consists of masked attention unit (self-attention), followed by residual lcarning and layer normalization, followed by an attention unit where key and values are calculated from the corresponding encoder layer (cross-attention). The rest of the layer is according to an encoder layer. The repetition of layers of encoder and decoder is the main building block for the transformer.
The encoder processes the input sequence. These are the text input or for time series prediction the historic time series (the time series itself or other time series of features) generating vector representation that
53

---

Published in Transactions on Machine Learning Research (02/2024)
captures the contextual information of each token, which is meant also for the time series case discussed in Subsection L1
The decoder, on the other hand, takes the encoded representations and generates an output sequence. It also consists of multiple layers, each containing self-attention mechanisms and cross-attention mechanisms. The self-attention mechanisms help the decoder focus on different parts of its input sequence, while the cross-attention mechanisms allow it to incorporate information from the encoder's output.
For the case of generating iteratively the next token for text generation, the prediction target is typically the next token in a sequence. Since for the text generation, several predicted tokens are required, the input of the decoder grows by the predicted token after each iteration. For inference the next token is predicted. Also during the training, the model is trained to predict the next token given the previous tokens in the input sequence. The number of imput tokens for the decoder is given by L E N. The input to the encoder can be passod to the input of the decoder. If tokens are itcratively generated, the number is supposod to be bigger Chan T in such cases, where T is the input length of the encoder. If the iterative output of the decoder becomes longer than maximum size L E N for the decoder's imput, which can exist due to limitation the hardware to calculate attention for such imput length between each token, then only the latest L tokens are used as an input for the decoder. If the sequence is shorter, corresponding positions L6 For translating, input language of the encoder's input can
are masked out as explained in Subsection be different to the language of the decoder's s output/impil /input. In such a case, the encoder's input may not be passed to the decoder's input and the input of the decoder is iteratively generated by several applications of the transformer.
In any case, the output of the decoder BXXXC is transformed by linear map W C Rd with bias b E Rd= such that the last dimension fits the number of available tokens from dictionary according to
i.,,, ib,,,:)W + DERd.
for each b E {l,..., B} and E (I,..,,). LJ. Then, the Softmax function is applied to the last slice L of the tensor according to
Softmax : (L,..., B} x {I, d + R, (b, Softmax(b, 8)
Σ Bo,.
to obtain a probability over all possible tokens choose the most likely token as the following token for each b E [1....,, To include some variety on choosing the next token, we can disturb this distribution for the next token (e.g,, introducing a temperature parameter) a bit such that also tokens become the most likely one that are close to the most likely token according to the undisturbed distribution over the tokens.
For time series forccasting tasks, the prediction target may vary deponding on the application. It could be the next value in the time series sequence, multiple future values, or even a binary classification indicating whether certain conditions will be met in the future. The basic concept is that a linear transformation W C + E, E E N, with a bias b e g transforms the output of the decoder the output format that corresponds to what is to predict, like the number of features or the numbers of classes that is then turned into a probability over classes by a corresponding Softmax function.
In this work, we focus on time series prediction. As discussed in Zhou et al. 2002)). it is advantageous to generate a multistep prediction (which includes a singlestep prediction) not by an iterative application of the transformer, like explained above, but provide the prediction at once, meaning to provide the prediction of length L N with single application the transformer. As a consequence, the training is done with a direct multistep loss. The rationale behind generating the prediction once is to avoid error accunulation within the multistep ahead time series prediction task. Considering (Liu et al. 2022 Algorithm 4), we define the input for the decoder by z E Bx & +LXF where L E N is the number of steps within the multistep prediction or prediction length, respectively, and F N the number of features, analogously to the token embedding for time series prediction tasks described in Subsection [.] While for the encoder the initialization is the input sequence, the initialization values for the decoder are as follows. The first T slices of the decoder input æ are filled with the last. T slices of the input of the encoder ã BBTIxF xF according to Ib,i.f F for all
2
54

---

Published in Transactions on Machine Learning Research (02/2024)
E B), i (1.... f fI,...,F F}. The last slices of the decoder are initialized with zeros according Øb.i.f for all b [), i (2 + ,,., I L}, [L,..,FF This representation is embedded, see Subsection L. and processed as shown in (Vaswani et al. 2017 Figure 1) by number of layers within the transformer. The output of the decoder, again denoted with BB is transformed by a linear mapping according to
P(b,i,;) =(b,),))((( +DERE
i.),
for each E (1,... B} and E (1,..., I L} where W E RCXE, b R and P E BB In our application, where we predict the time series from its history, F = =. The output after the linear transformation represents the L-step prediction and is given by
T i(),ú,e) for all i ( +1,... T
L}
for each element of the batch b E {1...., B} and dimension E {1,..., E}. Based on the output, loss functions are calculated with respect to the corresponding ground truth.
J Architecture for multilayer perceptrons for time series prediction
In this section, we describe the multilayer perceptron (MLP) architecture that we use for the time series prediction in this work. There is evidence that also MLPs are very powerful model to predict time series Zeng et al. (2023
Iteratively, an input tensor x E BxFT with B E N as the batch size, T E N as the length of the historic input of the time series for the prediction and F E N as the nunber of features is transformed to the output tensor E R Bx Px1. where L E N is the length of the multistep prediction, which includes singlestep prediction where L = 1. In between there can be several hidden layers. All layers have the following structure taking an input tensor zd-1 E Bx Faxa with certain number of nodes ("ncurons') Na N where d (I,...,") n), n E N is the number of layers, Ni = TT, Nu+1 L and zq x. The layer d is defined by the function given as follows
Ma No I Ne+1, zd- 2d-1 + Ma (za-1 f.:))) ga (za (b. ba)
for each b E {1,..., B) and f [I..... F} where 9r Pa is a pointwise applied non-linear activation function for each d E ni)..., like the ReLu function where gd max, 2d-1 E BBxFxN is the output from layer d - 1 and the input for layer d, Wa Nax and b E Ne+1, The operation zd-1(b, (b, f,:) is the common matrix-vector multiplication for any d (I....,n. n), b [1, B). E (1,.., F). We remark that ga can be but does not have to be a different function for each layer. For each b E {1...., B} and f {1,..., F}. we have that y(b, f::)= == (zn features. This the implementation we provide and is used in Zeng et al. (0023). However, in the examples
1 (b, f::). In this formulation, all weights in each layer are the same for all
within the present work, we have F =
To implement a version that has different weights for each feature in each layer d, we just need to reformulate the input of the layers by (b.:, 2b, f.:) R for all {1...., F}. Accordingly, the definition of the layers looks like
Ma Na RR Net1,
(b,1, + Ma. (zd-1, = g (2..,,, ba.f)
where applying the definitions separately to each feature leads to F different MLPs where the weight matrices and bias can differ per feature.
In order to introduce cross learning where information from one feature can influence the prediction of other features, we need to reshape the three dimensional tensor F,N,,N) to (B, FN,) for some layers where a corresponding weight matrix W FN, + FN can mix information from different features.
In a multi layer architecture, we can combine cross learning and lcarning per fcature in different layers assembling them in one model by reshaping outputs in the corresponding formats from F. Na) to (B, FN)) (B, FNa) to (B, F, N.) after layer before the next one depending on the learning type to change.
55

---

Published in Transactions on Machine Learning Research (02/2024)
With Pytorch such layers are implemented with
nn..Linerr(n m, bias True)
where n E N is the dimension of the input and E N is the dimension of the output. The bias parameter Truc adds bias with non-zero values and the parameter False fixes the values of the bias to zero.
56


Reviews:
[{"review_id": "MrHIeK7WTw", "reviewer": "Reviewer_HzAU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The paper introduces a novel framework that goes beyond traditional loss function minimization by focusing on the predictability of deterministic relationships, which is a valuable addition to model evaluation techniques.\n- The paper has a clear practical application - by providing a method to assess when a model has extracted all available information, the framework can help prevent unnecessary computational effort and guide efficient model training and hyperparameter tuning.\n- The framework is not limited to specific machine learning models, making it versatile and applicable across various domains and model architectures.\n\n\n### Weaknesses\n- The paper is not well drafted, particularly the experiment section is hard to follow with last of empty spaces and unclear plots. I would ask the authors to concisely present the important results in the main paper and move the rest to Appendix. \n- I would like to see more experiments on real world datasets and comparisons with SOTA time series approaches. With the claims mentioned in contributions, it would be possible to improve the SOTA on time series forecasting and provide analysis. Can the authors also report the common metrics such as MSE/MAE for time series forecasting.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a statistical framework designed to assess the predictability of a target variable given input data and to evaluate how well a machine learning model captures deterministic relationships within the data. The framework distinguishes between model errors that are due to unpredictable stochastic components and those that result from a failure to learn deterministic patterns. The proposed approach extends traditional measures of model success by focusing not just on prediction error magnitude but also on the model's ability to extract all available deterministic information. The framework provides tools to evaluate the extent to which a target variable is predictable from input data, using measures like mutual information, and chi-square tests. After training a model, the framework assesses whether there is remaining information to be extracted by testing the independence between input data and the residuals (deviations between model predictions and ground truth). Overall, the paper provides an interesting perspective on model evaluation, emphasizing the importance of understanding the limits of predictability and the potential for further model improvement.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Mentioned in the weaknesses section", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uhzC9ijZmO", "reviewer": "Reviewer_oj2b", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1. The framework introduces a unique approach to measuring predictability and using this measure to guide the training process of machine learning models.\n2. The paper provides concrete examples and applications, particularly in time series prediction and classification, showcasing the framework's effectiveness.\n3. The framework is designed to be independent of specific model architectures, making it versatile and broadly applicable.\n4. The differentiation of model errors into predictable and unpredictable parts offers valuable insights into model performance and potential for improvement.\n\n**Weaknesses:**\n1. The approximation used for measuring mutual information, due to computational complexity considerations, may affect the framework's effectiveness. A thorough theoretical analysis is missing, raising concerns about the reliability of the measures.\n2. There is a need for more in-depth theoretical analysis to support the framework. Experimental details are provided, but a rigorous theoretical foundation is lacking.\n3. The paper could benefit from exploring and comparing other measures of predictability, such as innovations in signal processing, to strengthen its claims and provide a more comprehensive evaluation.\n4. The manuscript suffers from organizational issues, including excessive cross-referencing of sections, repetitive statements, and numerous typos, which can distract readers from the main contributions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a novel framework that aims to measure predictability in machine learning models by using mutual information. The proposed method assesses the predictability of a target given the available input data and utilizes this information to refine the training process. By differentiating model errors into unpredictable components and systematic misses of deterministic relations, the framework provides a new perspective on model success and convergence. The primary contribution lies in the stacking architecture, which enhances model training by combining different models to extract remaining information. The framework's application is demonstrated on both synthetic and real-world time series datasets. Overall, I find the ideas presented in this paper very promising. With some additional theoretical analysis, comparative studies, and a bit of polishing in writing, this work could make a significant impact in the field.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Provide a more detailed theoretical analysis of the mutual information measure and its approximations. Discuss the potential impact of the measure's discrepancy on the framework's effectiveness.\n\n2. Elaborate on why mutual information was selected and compare it with other potential measures, such as those used in signal processing for time series modeling. Discuss the advantages and limitations of each.\n\n3. Include additional experimental details and results that explore how the discrepancy in the measure affects the training paradigm. This could involve sensitivity analysis or robustness checks.\n\n4. Writing and presentation improvements:\n   - Reduce excessive cross-referencing and repetitive statements to improve readability.\n   - Correct typos and improve the overall organization and formatting of the manuscript.\n   - Provide clearer, more concise explanations of key concepts and findings.", "broader_impact_concerns": "The paper currently doesn’t address the broader ethical implications of the proposed framework. Given the increasing focus on transparency and fairness in machine learning, it would be beneficial to include a Broader Impact Statement. Specifically, \n1. How does the framework handle potential biases in the input data? Ensuring fair model training and predictions is crucial.\n2. How transparent and interpretable are the predictability measures for stakeholders who may not be machine learning experts?\n3. Are there considerations for data privacy, especially when dealing with sensitive datasets?", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "56zW3voVZE", "reviewer": "Reviewer_Zume", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- The list of related works seems to be quite comprehensive, and the paper is generally well written, however, there are some points that remain unclear (see the Weaknesses section).\n- The work presents the general statistical framework for improving ML training with an application to time series forecasting, demonstrating its practical relevance and real-world applicability. This general framework is not reliant on a specific implementation of mutual information, allowing for the use of any method to calculate mutual information.\n\nWeaknesses:\n- The parts concerning the calculation of margin probabilities and joint distributions for estimating mutual information lack detail. For example, the section \"Discretization scheme for co-domains or random variables\" can be revised to include exact formulas for implementing the chi-square test and calculating mutual information.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work presents a statistical framework to measure how well a machine learning model captures deterministic relationships in data, distinguishing these from unpredictable components. It also proposes a stacking architecture to systematically extract information where individual models fall short and demonstrates its effectiveness on application to time series forecasting. This approach paves the way for developing new loss functions that focus on fitting models to deterministic relations, potentially improving model accuracy by accounting for noise.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The requested changes are detailed in the Weaknesses section above.", "broader_impact_concerns": "No Broader Impact Statement provided, and I do not think one is required.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: First, the reviewers were split in their recommendation. Some of their suggested improvements had been addressed and some not. The remaining concerns include extended evaluation on real-world problems and further theoretical analysis of the success and failure of the proposed approach (this question is only currently addressed in the discussion, and only informally). Beyond this, several reviewers had requested that the authors further edit the manuscript to improve readability. This is my main concern with the current version. I therefore recommend that the paper is revised before publication. 

Second, the "model stacking" procedure proposed in 2.3 is closely related to the long-established idea of boosting [1] (e.g., AdaBoost, gradient boosting, etc), in the sense of sequentially growing models through repeated prediction of residuals. The similarity is high enough to warrant comment at the very least. Section 3 comments on the relationship to ensemble methods but does not compare to boosting. 

Third, the results section should be further curated and edited to help the reader. For example, Tables 2 and 3 present results for multiple trials that could be presented as aggregates. Several of the subplots in Figures 4 and 5 are completely flat lines and several of them are not commented on. The font in the legend in Figure 8 is small enough to be barely legible. 

Finally, the paper is in its current form *substantially* longer than the average TMLR paper, which, while permitted by the journal format, should be "justified by its content", as stated in the author guidelines. It is my view, and mentioned by reviewers, that this criteria is not met: two of the reviewers asked for a more concise presentation of key results. Still, the revised submission is longer than the original submission, despite 2/3 reviewers asking for concisenesss. 

I recommend that the paper is revised with the primary goals to: 
* Clearly expose the claimed contributions of the work by editing the introduction. 
* Make it easier for a reader to assess whether those claims are supported by experiments by editing the results section, highlighting key results.
* Reduce repetition and discussion of tangential results and potential applications of the proposed idea beyond what is tested in the paper

Additionally, I would like the authors to include a comment on the relationship between their stacking procedure and boosting methods [1]. 

[1] Shapire & Freund. Boosting: Foundations and Algorithms, 2012
Decision: Accept (poster)