Transfer Entropy Bottleneck: Learning Sequence to Sequence Information Transfer
Authors: Damjan Kalajdzievski
Ximeng Mao
Pascal Fortier-Poisson
Guillaume Lajoie
Blake Aaron Richards
Venue: Accepted by TMLR ()

Abstract: When presented with a data stream of two statistically dependent variables, predicting the future of one of the variables (the target stream) can benefit from information about both its history and the history of the other variable (the source stream). For example, fluctuations in temperature at a weather station can be predicted using both temperatures and barometric readings. However, a challenge when modelling such data is that it is easy for a neural network to rely on the greatest joint correlations within the target stream, which may ignore a crucial but small information transfer from the source to the target stream. As well, there are often situations where the target stream may have previously been modelled independently and it would be useful to use that model to inform a new joint model. Here, we develop an information bottleneck approach for conditional learning on two dependent streams of data. Our method, which we call Transfer Entropy Bottleneck (TEB), allows one to learn a model that bottlenecks the directed information transferred from the source variable to the target variable, while quantifying this information transfer within the model. As such, TEB provides a useful new information bottleneck approach for modelling two statistically dependent streams of data in order to make predictions about one of them.

Published in Transactions on Machine Learning Research (02/2023)
Transfer Entropy Bottleneck: Learning Sequence to Sequence Information Transfer
Damjan Kalajdzievski--2
damjank7354 @gmail com
Ximong Maol.3. 1.3,-
mimeng.maa @vrcila, qucbec
Pascal Pertier-Poisson
pascal
Guillaume L.88
15.8.
g. Ialjjoo
Blake A. Richards1267.8t
rike..ichhadds www.la..damaaaoooomommo quebee
Mila Quebec AI institule, Montreal, QC, Canada
Department of Neurology and Neurosurgery, McGill University, Montréal, QC, Canada Department of Computer Science and Operations Research, Université de Montréal, Montréal, QC, Canada BIOS Health Ltd., Combridge, UK
Department of Mathmatics and Statistics, Université de Montréol, Montréal, QC, Canada School of Computer Science, McGill University, Montréal, QC, Canada Montreal Neurological Institute, McGill University, Montréal, QC, Canada CIFAR, Toronto, ON, Canada
Equal Contribution
Equol Advising
Reviewed on OpenReview: https: openreview. net/ nor/fo id= P7BRs
Abstract
When presented with data stream of two statistically dependent variables, predicting the future of one of the variables (the target stream) can benefit from information about both its history and the history of the other variable (the source stream). For example, fluctuations in temperature at a weather station can be predicted using both temperatures and barometric readings. However, a challenge when modelling such data that it is casy for neural network to rely the greatest joint correlations within the targel stream, which may ignore crucial but small information transfer from the source to the target stream. As well, there are often situations where the target stream may have previously been modelled independently and it would be useful to use that model to inform new joint model. Here, we develop an information bottleneck approach for conditional learning on two dependent streams of data. Our method, which we call Transfer Entropy Bottleneck (TEB), allows one learn a model that bottlenecks the dirccted information transferred from the source variable to the target variable, while quantifying this information transfer within the model. As such, TEB provides useful new information bottleneck approach for modelling two statistically dependent streams of data in order to make predictions about one of them.

---

Published in Transactions on Machine Learning Research (02/2023)
Y
4
ZJY
Y1
Y
Y
Y1
DKL
X
Xa
ZJX,Y
(a)
(b)
Figure 1: (a): Probabilistic graph representing directed information transfer between stochastic processes. X is the source stream, Y is the target stream. (b): Architecture diagram of the TEB model as implemented.
1 Introduction
Scientists are often presented with two streams of statistically coupled data, a source stream, which provides information, and a target stream, which they wish to predict the future of (See Figure [p). Obviously, one can use the history of an individual variable to predict its future values, but at the same time, it is desirable to use the statistical dependency in joint model, rather than treating each variable independent. However, using typical joint input stream model ignores the directed nature of the information transfer from the source stream to the target stream. Put another way, one would ideally use a method that can differentiate the directionality of information in order to better modulate and interpret the role of different input streams on prediction. To this end, we propose to leverage transfer entropy (Schreiber) 2000). a measure of directed information flow between stochastic processes, to help with the modelling. Specifically, if we have a source stream, X and a target stream, Yr-e...., Yo, the transfer entropy from X to Y (with horizon 0 S 00) is:
T x4y = -XX)
where I(::1-)) is the conditional mutual information.
Estimating transfer entropy with a deep learning approach has been extensively explored (Zhang et al. 2019) (see Appendix C for discussion contrasting with TEB). But, leveraging the transfer entropy for prediction is often challenging problem and one that has yet to be resolved for deep neural networks (Schreiber) 2000 Gençaga 2018). One reason for this difficulyy is that the transfer entropy the overall entropy in the data streams, and such, many prediction models will ignore it. For example, in weather forecasting, periodicity of average monthly temperatures could dominate a prediction model of future temperatures, which would prevent a simple predictive model from detecting important anomalies indicated other sources of information such as air pressure, wind, greenhouse gas emission and other human activitios. Indeed, we shall show that a neural network trained on joint input stream can struggle to learn or generalize when the statistical coupling between the two processes is strongly correlated, but with conditionally small and crucial information transfer from the source stream.
be small relative to
We use an information bottleneck (IB) approach Tishby et al. 2000) to derive method, which we term Transfer Entropy Bottleneck (TEB), that learns a compressed conditional information representation. This allows the benefits of IB to be applied to the new domain of double stream conditional processing. As in other information bottleneck architoctures, TEB works by lcarning a latent reprosentation of the data, Z, such that Z contains only the relevant information required capture the relationship between the random processes X and Y (Tishby et al. 2000). But, in TEB, this is done using information that is conditional
2

---

Published in Transactions on Machine Learning Research (02/2023)
the history of Y, allowing to capture both the statistical dependencies over time on Y, as well as the transfer entropy. As well, we design TEB in manner that allows one to use pre-trained predictive model for the target stream. That is, if one has an existing model that has been trained to predict Yt from the history Y (i.c. (Vi)e-ss<ee), TEB allows one to plug this cxisting model into a new model that includes in order to capture the transfer entropy and use it to improve the prediction Y, This provides a flexible solution when one might have reasons to keep previously trained model fixed. In practice, the reasons can be the desirable performance and interpretability of given model, retraining being computationally expensive and time-consuming, and/or compatibility to other components in larger system.
More preciscly, the contributions of this paper are the following:
We design loss function that promotes representations obeying Conditional Minimum Nec- essary Information (CMNI) principle, which ensures that the latent represenlation of the data contains only the relevant information transferred from the source to the target stream. We use this loss to develop TEB, an information bottleneck algorithm that learns representations following the CMNI, such that:
1. The learnt representation captures information embedded in the transfer entropy.
2. The larget st.eam can have its latent representation pre-trained separately from the joint latent representation, which can provide numerous computational and other practical benefits.
We introduce three synthetic tasks on dual stream modeling problems. We provide experiments on these tasks to show that TEB allows one to improve the predictions of the target stream, and that is applicable various data modalities including images and time-series signals.
Altogether, TEB is mathematically principled new IB method for learning to predict from two streams of statistically dependent data, with a wide array of potential applications.
1.1 Related work
The concept of an IB, as first defined in Tishby et al. (2000), was introduced way of quantifying the "quality" of a compressed reprosentation, Z, of a signal, X, for making inferences about some other signal, Y. This work defined high quality latent representations as being those that minimize the following Lagrangian:
IB = I(X; - 3Y;
(IB)
In Tishby & Zaslavsky (2015). the IB concept is explored in deep neural networks as a possible approach for studying the quality of representations in the hidden layers of network, and as possible explanation for generalization. The analysis of IB so as to understand the learned representations of neural networks was further studied in Saxe et al.] (2019).
Traditionally, the IB literature seeks to find a model with the most compressed representation, 2, that preserves the relevant information about Y. This approach seeks out a representation where the following equality holds:
((x;))
(MNI)
We use the terminology of Fischer 2020 and say that such a representation captures the Minimum Necessary Information (MNT). In the case of IB variable Z minimizing the information about X relevant to Y, this is the same as Z being a minimal sufficient statistic of X for Y. Note that it might not be possible to exactly reach MNT (Wu et. al. 2019 Fischer 2020 however, it proves to be a valid training objective in
practice Fischer 2020).
The previously developed IB methods most relevant to ours, are the Variational Information Bottleneck (VIB) AAlmi al. 2017) and the Conditional Entropy Bottleneck (CEB) (Fischer 2020
3

---

Published in Transactions on Machine Learning Research (02/2023)
Deep VIB Alemi et al. 2017) is method using latent variable encoder-decoder model to derive variational IB procedure, with a training approach analogous to variational autoencoders (Kingma & Welling 2014). They show this procedure can be used to learn an IB representation in neural networks. Put more preciscly, with the computation arranged in a feod-forward structure + Z + Y, via an encoder g(2 and a decoder d(y)z), VIB optimizes q, d with a variational bound for the IB loss:
VIB = Ep(x)
(VIB)
where p(2) is a pre-specified fixed prior. In this loss, the KL divergence term bounds the information that Z carries about X:
X); < Ep(e) (+=]))(()))
and so minimizing this term compresses the representation, while the other term:
Ep(u.:) (oog(d())))))))))))) s 1(Y
is maximized to increase the information that Z carries about Y. Alemi et al. 2017 show that their method has increased generalization and adversarial robustness compared other forms of regularization like dropout (Srivastava et al. 2014), and confidence penalty with label smoothing (Pereyra et al. 2017).
The CEB method (Fischer 2020), proceeds similarly to VIB, but instead of using the KL divergence to prior as a bound for T(Z; X), it minimizes an upper bound of the difference X(Z I(Y; ; Z) using a backwards encoder b. The loss minimized for d, b is then:
CEB = E Ep(u.)) [DLL (g(2\=)]((<(1))) - p(e (log [log(d())2)) (lg(dy))))
(CEB)
TEB shares many conceptual links with both VIB and CEB. However, unlike VIB and CEB, TEB is specif- ically designed to work with sationio where one has two sequences of data, one serving as a source seam and one as a target stream. Thus, TEB can be thought of as a means to bring similar principles of the IB and MNI in order to obtain a high quality representation that captures directed information transfer.
In Skatchkovsky et al. 2002),, variational information bottleneck method is derived for directed information [Kamer 1998). which is mcasure similar to transfer entropy. In contrast to ours, their method uses non-conditional prior, and imposes stronger Markov assumptions. In implementation, the prior is a fixed distribution, they limit their approach to simple spiking network encoders, and only process non-sequential data. Thus, though related, TEB solves a very different problem. Note that throughout the paper we use the term "directed information transfer" to denote transfer entropy, not to be confused with "directed information" above which is a separate measure.
Another related concept to transfer entropy is Granger causality (Granger 1969). and many previous works have focused on the relationship between the two (Barnett et al. 2009: Hlavackova-Schindl. 2011). For example. Barnett et al. (2009) showed that the two concepts are equivalent under Gaussian assumptions. Originally developed in econometrics, Granger causality has gained attention in other domains, including neuroscience (Seth et al.] 2015) and industrial processes (Lindner et al.) 2019). The calculation of Granger causality is traditionally via linear vector autoregressive models (VAR), which is generally considered cheaper than transfer entropy. To estimate Granger causality of to Y in practice, VAR trains two linear regression models predict Y with or without X as input, and Granger causality can be calculated as measure how much the latter reduces the error of the former. Throughout the years, many works extended this traditional approach in terms of both causality analysis from the data and time-series forecasting, for the lincar model (Lozano et al.] 2009). as well as non-lincar models including kernel regression (Gregorová et al. 2017), radial basis functions neural network @Wismüller et al. 2021), multilayer perceptrons Talebi et al. 2017 Tank et al.) 2022) and recurrent neural networks (Tank et al.) 2022). In general, these methods aim to learn the causal relationships among different timo-serics and or among different timo-lagged variables regression problem, and that is very different from the objective of TEB, and other IB methods described above, which is to learn compressed latent. representation of the data.
4

---

Published in Transactions on Machine Learning Research (02/2023)
2 Methods
In this section we fully specify TEB for learning representations which compress the transfer entropy via a bottleneck. Note that conditional mutual information is a special case of transfer entropy, and so TEB is also an IB method for creating a conditional information bottleneck. We encourage the interested reader to read the Appendix for thorough exposition of the mathematical details and proofs.
Our situation is analogous wish to learn latent encoding Zi of (X,)-essic via map q, which is conditional (X))-ssie from which one is able to decode Y, conditionally on (Y:)--egier with the decoder d. Put another way, we want to be able to predict Y at time using the values of Y and X up to time t- 1, but using our latent encoding of X conditioned on the history of Y. Namely, denoting 7 = Zz, X = (Xi)e-esict, Y (Vi)e-egict, and Yr, we have p, 9 such that:
latent variable encoder-decoder model, but with conditional structure; We
ply 2, v. p(u v)((2(,,)) ply', (2,))((((,,)111
(1)
as illustrated Figure 2a Note that g==|a, y) is actually the true distribution 2 y), since we artificially introduced the variable defined by 9. Note that Equation equivalent to the assumption on the conditional independence y' Y.
Furthermore, require the distribution to be learned by feed-forward encoder-decoder structure, which assumes the following decomposition of joint probability distribution (as illustrated in Figure 2b where d(y' | y) is variational approximation to p()/1z,y)
ply' 2,,,,, = p(z, y)p('' v) y) xe p(a, vddyy/ ()() (r,
(2)
The first part of Equation is equivalent to the conditional independence Y' 1x|Z, Y.
(a)
(b)
(c)
(d)
Figure 2: Probabilistic graph representing the random variables and their relationships as assumed by
TEB. (a): The actual random variables in the dataset. Independencies include y" 1. Z|X,Y, Y. (b): The random variables as computed by our food-forward computation. Independencies include Y' 1 XIZ,Y. (c), (d): The actual and feed-forward graphs, respectively, when utilising a pre-trained context encoding C. Independencies for (c) include y' L,,,C,, y' 1'1 C|Y, and Y' 1 CCX,Y, Y, and for (d) include Y' 1 X and y T CI
As is standard in IB methods, we want our optimization procedure to target a MNI principle of optimal compression; ie. we wish to learn our latent representation, Z, such that it captures exactly the necessary conditional information (conditioned on Y) between X and Y". Specifically we would like to target the following analogous MNI point, the CMNI point:
1(Y': 1(Y' 1(;:X(y).
(CMNI)
Due the more complex probabilistic structure of our setting, we do not have I(Y': Z|Y) < X|Y) and (YY;X|V) x T(Z; X|Y) directly from the data processing inequalities, opposed to the situation in
5

---

Published in Transactions on Machine Learning Research (02/2023)
the standard IB setting, where the joint probability follows simpler Markov chain Z - ++ Y. We thus prove that this is the case given our set-up (see Proposition 3 in Appendix [A), which implies that we can arrive at the CMNI point by maximizing I(Y'; 2|Y) and minimizing (Z; We remark here that, as in reaching the MNI point for IB mothods, it is potentially not possible practice to reach the CMNI point exactly on a given dataset. However, precisely reaching the CMNT point does not appear to be a necessity for effective compression practice as long as one approaches it.
However, estimating directly I(Y"; Z and XY) can be challenging, derive variational bounds to optimize the two conditional mutual information terms.
First, to maximize (YY; we decompose this information as
T(Y'; Z H(") +
(3)
Importantly, the term H(Y"(Y) is fully determined by the true data distribution, and thus, can be dropped from the optimization procedure, which is why the proportionality holds
We can readily bound ZH from below with:
-H(YY Bp(v.e. (oo(((s//e (z, w))l
Ep(x,)) [Der (p(y/ [z, m)lld(y/|=, + Eptna (og((d//)))))
(4)
Eptyar [lg (dly/1 / |2, y))].
Thus, in order to maximize I(Y"; we can simply maximize Bp(n..)) log(((y//2 v))] Moreover, since maximizing this bound also minimizes the expected KL divergence, i.c.
E Ep [z, v))l (/p.....// (z, y))l
we will simultaneously also learn to approximate p(y/|z,y) with d(v'lz, v), and the bound will become tight.
To minimize I(Z; X X by a tractable upper bound one needs to do more work, and use more distributions ional where p(=) is
a approximations. In VTB, one bounds T(Z: X) above with [ZZ X) S Epp()) [DEL (g(=(=))) pre-specified fixed prior. We shall see that I(; X|Y) can be bounded above similarly for TEB, however our prior cannot be hand-set fixed distribution, since it must be relevant to Y. Introducing the distribution do (zly) as a learnable approximation to p(zly), and recalling that ((2/x,)) = p(=[2, y) is the true distribution, we get that:
q22/22 v) p(z1z)
]
I(Z = Ep(v.2.r)
log
<(>/21, v)
Ep(y.2,))
log
Epti [DKL (=lp)
(5)
()(()))
g(2/2,, v)
< Ep(y.2.r)
log
Eptr,s) (KI..((22 ()l,,,
(yy(=l))
Training (2(y)) by minimizing this bound, also gives (y(y))) p(zly), since it squeezes the KL divergence of the two, and separate learning of dy to approximate p(>ly) is not necessary.
Equation 4 with Equation 5 is used to define Lagrangian loss for the TEB algorithm, which can be shown to bring us to the CMNI point when minimized:
Theorem 1. Under the independence assumptions implied by Equation and the minimum of the fol- lowing objective with respect to q, qy d will be at the CMNI point for Z, X, Y,Y. for some hyperparameter 3 >>0
TEB Ep(z.)) (22(1, v)ll9, 3..p(,... [log(d d(y/ |z, v))].
(TEB)
'However, this also means that the information I(Y": may not be possible explicitly calculate from existing procedure.

---

Published in Transactions on Machine Learning Research (02/2023)
Proof. Directly from Proposition 3 in Appendix A.1 the CMNI point can be arrived at by maximizing T(Y'; ZIY) and minimizing T(Z; X()). The proof of Theorem [ is immediate from the proposition, by replacing Z|Y) with its lower bound in Equation | and [(; X with its upper bound in Equation
Please refer to Appendix A.1 and A.2 for the full derivations of Theorem
Moreover, at convergence, the value of the upper bound is given by:
Bp(a.a) (DLL(922) 1)19 (zlw)) = I(2:
(6)
and so it can be used as a metric for X()). Thus in the case that the TEB loss optimum is at the CMNI point, we also have the fact that this bound measures the transfer entropy ;(Y"; Note that if the decoder does not converge to the true decoder, the bounds we use for optimization are still valid, and all we stand to losc the ability to get to the CMNI point exactly. No matter how bad the decoder is, for whatever output space we are able to represent, we still do bottleneck I(Z; and our bound of X)) becomes tight if converged.
The loss function of TEB shares structural similarity with those of VIB and CEB but differs in the choice of prior distribution in the Kullback-Leible (KL) divergence term. The VIB method uses a fixed distribution as the prior (a Gaussian in the original implementation), while for CEB, the prior is learned from the image of a target Y', mapped back to the latent via a backwards encoder. In contrast, TEB learns the prior only from the history of Y, making the role of the KL term to regularize the jointly inferred latent Z, so that it relies on as little information from the X stream as possible.
Now, suppose one has already learned an encoding Eu (-lv) for Y that provides latent representation of Y. We can use this as a fixed prior in TEB. This could be any existing model for encoding Y to variable C, e.g. a pre-trained self-supervised model, which will not be optimised with the derived TEB optimisation procedure (see Figure 2c and 2d to see how we integrate this latent in the TEB computation). One can then use this encoding place of Y the TEB algorithm. This could come with many practical advan- tages like computational efficiency, training efficiency, reuse of existing models, and rapid prototyping when experimenting with multiple configurations or hyperparameter scttings for the full TEB model.
To successfully bottleneck the transfer entropy with respect to conditioning on Y instead of C, it sulfices to require that the representation C' is relevant to Y' in that [(; Y") is maximized (see Lemma 5 in Appendix
AJ so that Y(ICY"IC) z (X;Y'I)) With this in mind we get the following TEB loss for CMNI learning from a pre-existing contextual encoding:
Theorem 2. Under the independence assumplions Y' 1 C, Y' Y Y' L CIX,Y, Y' LIZ,C Y' L'M Y|X,CC C, and C ~ (-v-) such that I Y") = I(Y,Y",, the minimum of the following objective with respect to q, d will be at the CMNI point for Z, X, Y,Y", for some hyperparameter 0:
TEB Ep(z.) (g() (x, (14l - 1/
(TEB*)
Proof. Directly from Lemma Appendix A.3 with ((CYY' = YY:; we can replace Y everywhere with C and still arrive the CMNI point. Then the proof is immediate by following a similar procedure as Theorem D Please refer to Appendix A.3 for the full derivations of Theorem 2
3 Experiments
To implement TEB we used architecture as shown in Figure Dp. It should be noted that TEB could be applied to other architectures, but here we will describe the architecture used in experiments. The model uses two initial encoders for the two streams by and qr, both of which feed into two separate LSTMs [Hochreiter & Schmidhuber 1997) lo aggregate information across the sequence. The hidden state from the Y pathway LSTM is then linearly projected to calculate the means and log variances for the dimensions of the prior 9, (xly), which are treated as being independent Gaussians in each dimension. The mean and log variance of this prior are then combined with the output from the X pathway LSTM, via a multi- layer perceptron (MLP) that perturbs the prior gy, to arrive at means and log variances parameterizing
7

---

Published in Transactions on Machine Learning Research (02/2023)
Target (Y): Retating MNIST images
Predict
f-2
t-1
3
3
i
Real
Predicted by TEB
t-4
t-3
t-2
2
1
t
6
i
3
3
3
2
Source (X): Category of image next time-step
(a)
(b)
Figure 3: (a): Sample sequence of rotating MNIST, which switches example digit at the fourth frame. (b): This is a sample testing generation of a TEB model using a fixed pre-trained context encoding, reconstructing the next step of sequence which always switches digits the prediction step. Images are arranged from left to right as: two frames of input video, next step, generated prediction. As an added demonstration of flexibility of TEB in this figure, the pre-trained context encoding was trained to reconstruct the next step of a rotating digit. The datasct the pro-trained context encoding was trained on is a separate one without switching of digits, hence the context encoding model had never encountered on video where the digit example switched. Note that the blur in the prediction is expected since the optimal prediction is the least squares minimum distance to all examples of the corresponding class.
the independent Gaussians in each dimension of q(2|z, y). Then the reparameterizztion trick (Kingma & Welling 2014) is used to sample latent representation Z from q=2/,; y)-
When the expected decoder d output is an image, the decoder arranges the sampled latent representation Z into a spatial representation using positional encoding, and this version of the latent is then transformed via space preserving convolutions to arrive at the output image, Y'. When the decoder is used to generate a time- scries signal, it is implemented as a ncuralODE (Chen et al., 2018) with its dynamics parameterized by an ML.. In both cases, we assume the decoder out.put is the mean of Gaussian distribution with fixed variance, independently in each pixel or at each time-step. See Appendix B for exact specifications of model architec- ture, more detailed descriptions on the task datasots, well more sample generations in every experiment. Code of our implementations can be found at https://github. com wtw.m///erraaafffoooooooooooooo
3.1 Generating Rotated MNIST
To experimentally test Theorems & 2 we create a dataset of videos of rotating MNIST (Lecun et al. 1998) digits. In given video sequence, a digit starts rotated at some random angle that is a multiple of i and rotates counter-clockwise 1 at every frame, but with chance that the example digit in the video may change to another random digit at given frame with some specified probability. We can interpret this data as a target stream Y which is the image video, and source stream X consisting of class labels for the next step in the video. See Figure
3 for an illustration of the rotating MNIST task.
To test the properties of TEB near the CMNT point, we train several models with different values of 3 for 10 epochs. On this test we set next step digit switching probability of 0.5, which gives that the true expected transfer entropy from X to Y is 1.67689 nats. See Appendix B.2 for the caleulation of the true transfer entropy for this task.
From Figure 4a we can see that there is a "bifurcation point" for 8 around 0.2, such that for any 8 smaller than that the oulput loglikelihood sharply drops off. We refer to such a point as 31, and the sharp drop-off for 3 < 31 is likely due to that essential information transfer from X being left out of the latent representation Z of the model, as in X|Y) < I(Y'; XJY). On the other hand, as in other IB methods (Wu et al. 2019). there is another value of 8, which we denote By as in Wu et al. (2019), such that for 8 < Bu the model is not learning and Z becomes trivial representation, which in our case means that compressed too much on
8

---

Published in Transactions on Machine Learning Research (02/2023)
3.0
30
distance to true transfer entropy
2.5
20
2.0
output loglikelyyhoo
1.25
IIZ; X11) (nats)
1.5
10-
1.00
(mats)
1.0
0.75
0.5
-10-
0.25
0.0
true
true TE @ S
-2
-0.5
11
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
0
B
(a) TEB testing
-10
1.50
distance to true transfer entr opy
3.0
output loglikelyhood
-15
11.5
2.5
N2 X(C)
(nats)
-20
1.00
(nats)
2.0
0.75
-25
1.5
1.0
-30-
0.25
true TE @
0.5
-35
0.00
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
B
(b) TEB from pre-trained C' in testing
Figure 4: Plots the information metric and reconstruction loglikelihood vs incremented by tenths, with extra hundredths increments the chaotic regime [.1, for both TEB and TEB' testing. In the information metric subfigure (the left subfigure both (a) and (b)), the green dashed horizontal line indicates the true transfer entropy for this dataset. From this obtain 3 the specific value of B achieving the true transfer entropy, calculated as the interpolated value of for which 1(Z; I(Y'; X (Y) average from repeated trials. In the reconstruction loglikelihood subfigure (the right subfigure), the output loglikelihood of the model trained with the obtained 3 is marked using the red dashed vertical line. Note that in (a) the variance of runs highly increases for 3 E (.1,22), but the models in this interval that found a solution with ((; X|Y) &e (YY;;|XY) are the same models that achieved higher reconstruction performance.
the information in the bottleneck, so that the information from X to recover Y' not available to the model any more. From Figure we can put 30 roughly in the range from 0.1 to 0.14, corresponding nearly zero X|Y). Note that $o. 31 are "bifurcation points" for the task performance both in training and testing (see Appendix B.2 for training plots), as for values of 3 E (So. 31) we quickly begin to learn representations that perform better until 3 2 31 and the model does not compress lower than the true transfer entropy T(Y'; XJY) < T(Z; X()). In Figure 5 we show the image generation samples when varying B.
We choose to mark 8" in Figure as the interpolated value of & for which I(Z; X = X|Y) on average. At 8* the true expected transfer entropy is approximately identifiable by the value of our upper bound for T(Z; X in Equation 6 and we find that 3 lies close to 8.. We that the models learned

---

Published in Transactions on Machine Learning Research (02/2023)
at 81 very close to the CMNI point with I(2; = X(Y) x 1 Consistent with IB principles, we can see that at values of 3 which result in the representation closest to the CMNT point, TEB obtains approximately the best testing score. Moreover, on this dataset the TEB model testing performance decreases monotonically with distance I(Z:X|) from I(Y"; ())), outside a small ncighbourhood of 31.
Target
Target
Input
(a) TEB
(b) TEB
the same input for varying B equal to 0.1,0.12,0.14. 0.14, 0.18, 0.2, 0.3, 0.4, 0.5, from top to bottom respectively. For B = 0.1 both models learn the trivial representation which does not incorporate class information from the X stream, i.e. change of digit class from 1 to (a): The TEB model's representation remains trivial until the sharp increase in reconstruction loglikelihood at = 0.18, as seen in Figure (b): The TEB model's representation appears transmit class information for 3 as low as 0.12: This consistent with the appearance (see Figure of the reconstruction loglikelihood dropping off more gradually than TEB for 3 2 0.13, with the transmitted information very near (distance 0.2 nats) to the true expected transfer entropy for $ .12 some seeds.
Figure 5: Sample testing generation
For the TEB models, we pre-trained the context embedding module, which we will call the "Y module", represent C' with next step prediction of Y' using CEB with 7 = on the same dataset. In this case for maximum training efficiency and difficulty for TEB', also fix the decoder from the Y module and use it for the reconstruction from Z in the full TEB model. We find in Figure 4b that the bifurcation point 81 for TEB" is similar to that recovered for TEB trained end to end. However, we do find that unlike the situation for TEB, here the CMNI point does not appear to be reachable since our bound for I(Y"; Z YY continues to increase for 3 > Br. On the other hand, it difficult to find So directly from Figure 4b but as shown in Figure TEB fails predict the change of digit class B = 0.1, which puts its 3o approximately 0.1.
One reason the CMNI point not reachable could be due to our representation of the output loglikelihood: As is the case for any gradients proportional to l or fl, the output probability space assumes independence of each pixel, and therefore does not accurately represent the true image probability space for the data. This cause is evidenced by the fact that the generation quality does not perceptibly vary much between models for 3 > 31 even though the loglikelihood does vary significantly. Another cause could be that T(Y'; C) was
2One can Figure that reconstruction lnglikelihood bound for I(Y'; (which differs from I(Y' by constant) is approximately maximized 81, and since T(Z; X T(Y'; X|Y) 81, we are close the CMNI point.
10

---

Published in Transactions on Machine Learning Research (02/2023)
f-3
t-2
1-1
Real
Predicted by TEB
Figure 6: Sample TEB testing sequence of the needle in a haystack task with 20 distractors, with a color switch at the prediction step. Images are arranged from left to right as: three frames of imput source stream, next step, generated prodiction. For visual clarity, we show only the needle pixel channel group of the source stream, with the first 10 distractor pixels added and randomly placed on the perimeter (note that needle pixel always at top left). The actual streams in the experiments had stacked RGB channel groups, where each group has either a needle or distractor pixel at top left.
not maximized enough due to regularisation. Also, since the Y module does not have any information to reconstruct randomly swapped in digit, and is never able to represent this its own, it may not have learned a representation of angle which is disentangled from the identity of the digit. This would result in requiring a higher 3 for learning a TEB model which adjusts the latent by sending extra information from X on sequences which switch, in addition to the class of the digit. To partially test these causes, we check the loglikelihood performance of TEB with a higher 3 = 10, from a pre-trained Y module such that it is trained with a higher y = 100 to prioritise maximizing I(Y"; C). This results in a significantly improved testing reconstruction loglikelihood of 19.9317 = 0.3105 (as compared Another way to find a better C is to let Y be pre-trained on rotating digits that never switch, because then the training of Y can focus on learning just the rotating of digits. See Figure Bp for example testing generation in this case with 3 == 1. This also demonstrates that TEB is flexible in modulating pre-trained model for a new task modality.
16.0291 + 0.6041 B = 0.2).
3.2 Detecting a needle in haystack
Here we design challenging task to test performance of TEB, and to test its added ability to generalize when the statistical coupling between the X and Y streams is high, but with a conditionally small and crucial directed information transfer from X to Y. The task is next step prediction for colored bouncing balls (Sutskever et al. 2008) video, where the color of the balls can change to one of seven colors at the moment of prediction with probability 0.5. The twist is that there are some number of randomly colored noisy pixels in the video, of which one pixel, which we shall call the "needle pixel", noisily corresponds to the color of the balls in the next step; To predict the next step color correctly, a model has to decide on the three previous frames of video which pixel corresponds to the needle pixel. The sequences for both X and Y are defined as video sequences consisting of the images of the balls and the distractor pixels, but importantly, only X contains the added needle pixel. To make the task of predicting color more difficult still, we introduce baseline intensity in all color channels (red, green, and blue), at each pixel of each ball. This means that the output reconstruction component of the loss, which is common to all models (for deterministic models it is the only component of the loss), depends only mildly on predicting the correct color. As such, X and Y are very strongly correlated, both with each other and over time. Moreover, the transfer entropy provided by the needle pixel is crucial to this color prediction task, though this information is relatively very small. This ensures that our task fits the description of a large correlation of the sequences, but small and crucial directed information transfer from X to Y. See Appendix B.3 for more detailed descriptions on the utilized X and Y video sequences with the distractor pixels.
In our experiments, we also trained three basclines for comparison. The first two are deterministic basclines. Between them, the first is simply a deterministic version of the TEB model, with the same architect and latent size, but we dropped the sampling from q(2[1; y) and excluded the KL divergence term in the objective, which is the first term in the TEB objective function. It uses directly the mean of q2|x,, y) as the input for the decoder. The second which we refer to as the joint stream deterministic model or "Deterministic joint" in Table and is 18 (He et al. 2016 + LSTM which takes all the X and Y information
11

---

Published in Transactions on Machine Learning Research (02/2023)
unified input, followed by the same deterministic decoder architecture TEB. Note that particularly in this task, every image of Y is included in X, thus X itself is valid unified input, and we adopted it in our implementations. The third baseline is CEB, with the same architecture as the Y module of TEB, with added backward encoder. As with "Deterministic joint", CEB takes a unified input as well. The reasons behind including joint stream models, i.e. "Deterministic joint" and CEB, as comparisons are to showcase the advantage when bottlenecking transfer entropy in this task.
To evaluate performance of the directed information transfer from X to Y, we want to determine whether the predicted image at time t has the correct color for the balls. To quantify the color accuracy use an additional color classification network. This classification network is trained to classify color into one of seven color classes, on images of the balls from the training set. Since this is an easy task, the classification achieves 100% accuracy on the test set as well. As such, any errors in color classification reflect errors in the color of the predicted frame, and so, the color classification serves as a metric of color prediction, i.c. the transferred information.
Table I shows that the color prediction task for 20 distractors, the deterministic version of the TEB architecture does not perform much better than what one achieves by simply predicting a constant color as previous frames, which gives an accuracy of approximately 57%. In contrast, TEB model picks up a transfer entropy signal via information bottleneck, which allows it to predict the color at a much better accuracy. TEB outperforms the deterministic baseline consistently across different number of distractors, with an increasing gap for the harder task. In comparison, both the joint stream deterministic and stochastic models (CEB) do not even learn to generalize better than the constant color baseline on 5 distractors. One explanation is that they are likely dealing with a harder task than TEB and "Deterministic", because they cannot make use of the difference from X to Y in the latent space, which is essentially the information denoting the needle pixel. Another explanation lies in the joint stream formulation. For example for CEB, it has to compress the joint stream input as a whole, including the bouncing balls images, needle pixel and all the distractors, so it might ignore the needle pixel which is relatively small.
Table 1: Testing color prediction accuracy on the needle in a haystack task (8 seeds), best performance is shown in bold.
Model
5 distractors
10 distractors
15 distractors
20 distractors
TEB
94.64 + 2.21 %
88.39 3.63 %
81.99 + 3.04 %
68.37 + 3.92 %
Deterministic
93.87 + 0.94 %
88.33 + 2.57 %
73.27 + 7.50 %
59.97 + 7.26 %
Deterministic joint
38.35 + 0.75 %
CEB
40.78 + 0.67 %
Testing reconstruction log-likelihood on the needle in a haystack lask (8 seeds), best performance is shown in bold.
Table 2:
Model
5 distractors 10 distractors
15 distractors 20 distractors
TEB
-551 + 25
-568 + 26
-562 + 17
-566 + 17
Deterministic
-515 + 26
-542 + 21
-561 + 25
-558 + 28
Deterministic joint
-573 30
CEB
-823 + 66
However, Table 2 shows that unlike the case of color prediction accuracy, the aggregate statistics of recon- struction loglikelihood do not differentiate between any model except for CEB, which trades loglikelihood for better identifying the transfer entropy than "Deterministic joint", no matter the number of distractors. This due to the added whitening on cach of the color channels at each ball pixel. As such, the recon- struction can still be largely accurate even when the system is not getting the color correct. In other words, the majority of the reconstruction quality is dependent the localisation of the balls and not the color of
12

---

Published in Transactions on Machine Learning Research (02/2023)
Input_ x
Target
Predicted
Input X
Target
Predicted
(transformed with shifted pixels)
(transformed with shifted pixels)
Figure 7: Sample TEB testing generation for 5 distractors
the balls, which highlights the high difficulty disentangling the needle pixel signal, and obtaining a quality representation which allows for the prediction of the true color.
Lastly, we show batch sample generations of TEB for 5 distractors in Figure We believe that it can even be challenging task for people, without knowing that the needle pixels are fixed always the top left corner, and the models would have to infer the needle pixels based on the history. See Appendix B.3 for more sample generations in both training and lesting and the common generation error made by TEB.
3.3 Extrapolating time-series
We also evaluate the performance of TEB on synthetic time-series data, where each time-series signal is generated as the average of five sinusoidal waves of distinct frequencies E {0.2, 0.4, 0.8, 1.0) with starting point randomly in [-1, Each sinusoidal wave is generated by unrolling a 2-dimensional continuous time dynamical system with purely imaginary eigenvalues, and recording the coordinates of the resulting trajectories. Note that with initial conditions lying on the unil. circle, the trajectories of the said dynamical system are rotations on the unit circle. See Appendix [.4 for more detailed descriptions on the implemented dynamical system. Note that this is done to facilitate the sudden change of frequency at any given time- step. We refer to this task multi-comppnent sinusoids. As with other tasks discussed above, switch with probability 0.5 is also implemented in this dataset, where the frequency of one random component is resampled every time switch occurs. The task is to extrapolate the future sequence of length 20 for a given time-series signal of length 100. Here Y the averaged signal and X the noisy frequencies for all the sub-components for the next time-step. The difficulty of this task comes in largely from the averaging nature of Y, as there might be multiple ways to decompose the same signal. Hence it requires the model to capture the information embedded in the transfer entropy in order understand the true decomposition based the frequencies.
For this task, we compare TEB model with three other basclines. The first is "Deterministic"" defined above, that is simply a deterministic version of the TEB model architecture. The other two are both joint stream models, taking the concatenation of X and Y as a unified input. Between them, the second is simply deterministic LSTM, which outputs the next sequence as vector, whereas the third is similar the latentODE model introduced in Chen al. (2018), with an LSTM encoder and neuralODE decoder, but here only ask the decoder predict the extrapolation points, instead of reconstructing the inputs as the
13

---

Published in Transactions on Machine Learning Research (02/2023)
Target
-0.2
-0.4
Predicted by TEB
20
80
100
120
Figure 8: Sample TEB testing sequence of the multi-component sinusoids task when there is a switch at prediction time-step 100, indicated by the yellow vertical line. Note that the sharp change curve at the prediction point due to a larger span in the change of frequency of the fourth sub-component, from 0.8 0.2.
Table 3: Testing reconstruction logelikeli the multi-component sinusoids task (3 seeds), best perfor-
mance is shown in bold.
Model
on the entire set
only when switching
TEB
4.3497 + 0.0761
3.6839 + 0.1973
Deterministic
4.2279 - 0.0499
3.4123 + 0.1264
LSTM
3.9941 = 0.0046
2.8577 + 0.0168
AdlentODE-VE
3.7902 + 0.0020
2.5475 + 0.0146
original latentODE. As with the original version, uses a fixed Gaussian as a prior, so is functionally close to VIB, and refer to it IDEEnI
Table 3 shows the reconstruction logelikel both the entire testing set and only when switching to new froquency. Note that since we allow switching to the same frequency, the second scenario comprises around 40% of the entire testing set. Judging from the results, TEB outperforms all the baselines in both cases. In addition, although all the algorithms suffer from a reduction in performance when evaluating only when there is change of frequency, TEB able to achieve the smallest reduction among the four. This indicates that TEB does a better job capturing the transfer entropy in this task than all other basclines. Furthermore, show batch sample generations of TEB in Figure where right show common extrapolating error made by TEB. See Appendix B.4 for of large deviation in TEB's testing generation, which is more rarely seen in our experiments.
in the left and 2 in the
4 Discussion and future work
In this paper we developed TEB, information bottleneck approach designed to capture the transfer entropy between a source and target stream of data. In doing so, TEB brings the benefits of IB approaches to dual stream modeling problems where disentangling statistical dependencies can be very challenging. As well, TEB allows one to use pre-existing encoding models for the target stream. We showed experimentally that TEB allows one to arrive at high quality latent representations that obey the CMNT principle, and use these make more accurate predictions about the target stream even when the total information transferred is small relative to the joint information.
A limitation of TEB, transpiring from Theorems and is that TEB enables exact estimation of the true transfer entropy only at convergence to the optimum of the loss for a particular B, which is the / and optimum reaching the CMNI point. Recall that 3 is a weighting coefficient in TEB's loss function, where a larger 3 focuses more on the generations of the outputs and a smaller B focuses more on reducing the KL divergence. In practice depending on the architecture, learning, and dataset, the optimum may not be reached and CMNI learning may not be possible. Additionally, CMNI learning is possible, the identification of this particular may not be possible. However, as shown in the rotating MNIST experiment, identify
14

---

Published in Transactions on Machine Learning Research (02/2023)
Input_y Y
Target
Predicted
0.0
0.0
-0.5
-0.9
0.5
0.0
0.0
0.5
0.0
0.0
0.5
-0.5
20
60
80
100
120
20
40
60
100
110
Figure 9: Sample TEB testing time-serics sequence generation when there is switch prediction time-step 100, indicated by the yellow verlical line. The samples with frequency switch are marked with green frames.
phenomenon similar to the one observed for the usual IB case (Wu et al. 2019 which can provide for the approximate identification of this B. We expect this to occur in some form for most datasets when learning can reach CMNI point optimum. But, more importantly, estimating the transfer entropy is not the primary purpose of TEB, so even when one cannot get hold of an estimate on this quantity, TEB can still use the transfer entropy to improve its predictions.
TEB has many potential applications. principle, TEB could be used any situation where one pos- sesses multimodal sequential data with statistical dependencies. This could include videos, natural language captioning, financial modeling, medical device time-series, and more. In all cases, appropriate architec- ture would have to be built, but TEB can be applied generally. In fact, TEB can potentially be applied to traditionally deterministic architectures for dual data streams tasks by adding some stochasticity to the representations like the technique of Achille & Soatto (2018). As well, TEB could be modified to work in situations with more than two streams of data. Future work will examine the utility of TEB with real data in these potential applications and new architectures, and potential theoretical guarantees of learnability as done in Wu al. (2019) for the IB loss. This current work provides the analytical foundations of TEB and shows with synthetic data that works well in principle.
Broader Impact Statement
This work provides method for prediction from dual-stream data sources. As with other machine learning approaches to prediction, there may be questionable uses that such system could be applied to. To avoid such problematic situations, use of TEB should be carefully considered by researchers in advance, and any potential harms should be considered before applying TEB to a new dataset.
Acknowledgments
This work was done in partnership with BIOS Health Ltd. https::/www.bbos.hallttee and supported by MEDTEQ+ (Impact Grant 13-F Neuromodulation) Hcalthy Brains, Healthy Lives (Neuro-Partnerships Program: 3d-13), Mitacs (Accelerate Program: IT19551), NSERC (Discovery Grant: RGPIN-2020-051005
Discovery Accelerator Supplement: CGPAS-2020-0000 and CIFAR (Canada Al Chairs to BAR and GL; Learning Machine and Brains Fellowship BAR). GL further acknowledges the Canada Rescarch Chair in Neural Computations and Interfacing (CIHR). This research was also enabled in part by support provided and Compute Canada www.computecnnada.c.)
by (Calcul Québec) htttps://www.caleuqquee
15

---

Published in Transactions on Machine Learning Research (02/2023)
References
Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2897-2905, 2018. doi: 10.109//PAMM.221.2274444
Alex Alemi, Ian Fischer, Josh Dillon, and Kevin Murphy. Deep variational information bottleneck. In ICLR, 2017. URL
Lionel Barnett, Adam B. Barrett, and Anil K. Seth. Granger causality and transfer entropy are equivalent for gaussian variables. Phys. Rev. Lett., 103:238701, Dec 2009. doi: 1.1100//PyyReeeeett.1033.. URL https:///inkkk.... aps http://ww...pp....oo////ooo/.... http:///a........../ooeeeeeeeeee 203.23870111
Mohamed Tshmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In Proceedings of the 35th International Confer- ence on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 331-540. PMLR, 10 15 Jul 2018. URL htttps://proceedinggs mlr www.i..pres/////eee//e htt./////eeeleiieeeeeeee html
Ricky Q. Chen, Yulia Rubanova, Jesse Rettencourt, and David Duvenaud. Neural ordinary differential equations. Advances in Neural Information Processing Systems, 2018.
Tvan De La Pava Panche, Andres M. Alvarez-Meza, and Alvaro Gutierrr A data-driven measure of effective connoctivity based on renyi's a-entropy. Frontiers in Newroscience, 13, 2019. doi: 10.3389/fnil /fnins.
2019.01277. URL //www/ www.::ffoooorrriiiommmmmm https://www.frrtiierrsii.co wtwp://aceeaaarrrtroooooooo 3389/ffin... wtt.:9/////nnnn.111.011.117 2019.01277
Ian Fischer. The conditional entropy bottleneck. Entropy, 22(9). 2020. ISSN 1099-4300.
doi: 10.3390/
222090999. URL
Deniz Gençağa. Transfer entropy. Entropy, 20(4), 2018. ISSN 1099-4300. doi: 10.3390/ URL
https: mdpi. wtt.c///-0-994400/////00000
C. W. Granger. Investigating causal relations by econometric models and cross-spectral methods. Econo- metrica, 37(3):444-438, 1969. ISSN 00129682, 14680262. URL http:://www..sstr.orrr//tttttttt
Magda Gregorová, Alexandros Kalousis, and Stéphane Marchand-Maillet Forecasting and granger modelling with non-linear dynamical dependencics. In Michelangelo Ceci, Jaakko Hollmén, Ljupčo Todorovski, Celine Vens, and Sašo Džcroski (eds.). Machine Learning and Knowledge Discovery in Databases, pp. 544-558, Cham, 2017. Springer International Publishing.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance on imagenet classification. In 2015 IEFE International Conference on Computer Vision (ICCV), pp. 1026 1034, 2015. doi: 10.1109/ICCV 2015.123.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770 778, 2016. doi: 10.1109/CVV CCVP..016.......
Katerina Hlavackova-Schindae Equivalence of granger causality and transfer entropy: A generalization. Applied Mathemationl Sciences, 5:3637 -3648, January 2011. ISSN 1314-7552. URL http: http:://ppintts..om
univie. ac. at..4996.. at/4968/
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Newral Comput. 98): 17800 nov 1997. ISSN 0899-7667. doi: 00.1162/neco..997.9888 9. 9.8.1734
URL
1997.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. URL
http:///rxii: org/ awspaa//..111111
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
URL
http:
7/axxvv htt//bbs/11311 http:org///b///...11111 6114
16

---

Published in Transactions on Machine Learning Research (02/2023)
Gerhard Kramer. Directed information for channels with feedback. PhD thesis, ETH Zurich, Zurich, 1998.
Diss. Techn. Wiss. ETH Zürich, Nr. 12656, 1998. Rel.: J. L. Massey Korrel.: A. J. Han Vinck.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-basee learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998. doi: 10.1109/55 00000995.7779..
Brian Lindner, Lidia Auret, Margret Bauer, and J.W.D. Groenewald. Comparative analysis of granger causality and transfer entropy to present a decision flow for the application of oscillation diagnosis. Journal 21....0199.00000
of Process Control, 79:72-84 84, 2019. ISSN 0959-1524. doi:
URL http://www..ooooie........cosaaa
Aurélie C. Lozano, Naoki Abe, Yan Liu, and Saharon Rosset. Grouped graphical Granger modeling for gene expression regulatory networks discovery. Bioinformatics, 25 (1)):11100 i118, 05 2009. ISSN 1367-4503. doi: 10.1093, hit.0////biiiooooaatiee//// btp199. URL https://doi. /ttp:///0///iiiioooooota////////
1093
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, pp. 807-814, 2010.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-perffrmance deep learning library. In Advonces in Newral Information Processing Systems 32, pp. 1034-005.. 2019.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions, 2017. URL htttt:://arxiiiii wwtpesiii.orr//bie argg/bas///1///00
11/. 06548.
Andrew M Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D) Tracey, and David D Cox. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2014(12) dec 2019. doi: ab3985
/bb088///b./... URL https://doi.
Thomas Schreiber. Measuring information transfer. Phys. Rev. Lett., 85:461-466 Jul 2000. doi: 10.1103/
PhysRevI.ett.5.4661. URL
aps. httpp//orrr///..../////eeeeooee/ 1tt..11s///yyeseeett...... .tw.lltttttt........ 88..6651.1..
Anil K. Seth, Adam B. Barrett, and Lionel Barnett. Granger causality analysis in neuroscience and neu- roimaging. Journal of Neuroscience, 35(8):3293 32977 2015. ISSN 0270-6474. doi: 10.152//INUROOCC..
4399-14.2015. URL httt:://ww.jeeurroc..ooooooeeeee
Nicolas Skatchkovsky, Osvaldo Simeone, and Hyeryung Jang. Learning to time-decode in spiking neural networks through the information bottleneck. In Advances in Neural Information Processing Systems, 2021.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, 1lya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56): 1958, 2014. URL http://jmlr.. oww.oogg//..e.oee wwwpgg/apers/aeee saapppersvv1//// http//11//srrrreee wtw.ssaaaaaaaaaaaaaaaaaaaaaaaaaa 1a. html
Ilya Sutskever, Geoffrey E Hinton, and Graham W Taylor. The recurrent temporal restricted boltzmann machine. In Advances in Neural Information Processing Systems, volume 21, 2008.
Nasibeh Talebi, Ali Motie Nasrabadi, and Iman Mohammaad-Rezvadee Estimation effective connectivity using multi-layer perceptron artificial neural network. Cognitive Neurodynamics, 12:21 42, 2017.
A. Tank, 1. Covert, N. Foti, A. Shojaie, and E. B. Fox. Neural granger causality. IEEE Transactions Pattern Analysis and Machine Intelligence, 44608)44267-4279, aug 2022. ISSN 1939-3533 doi: 10.1109/ TPAAM22021 2021.3065601.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pp. 1-5, 2015. doi: 10.1109/TTWV 00110//TT..01..77133 70w.2015...3316..
17

---

Published in Transactions on Machine Learning Research (02/2023)
Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method, 2000. URL
https://arxiv. atg/abss/physccs/0004a/
Mauro Ursino, Giulia Ricci, and Elisa Magosso. Transfer entropy a measure of brain connectivity: A critical analysis with the help of neural mass models. Frontiers in Compudational Newroscience, 14, 2020. 00040. URL
doi:
2020.
00045
Axel Wismüller, Adora M. DSouza, Ali Vosoughi, and Anas Zainul Abidin. Large-scale nonlinear granger causality for inferring directed dependence from short multivariate time-series data. Scientific Reports, 11, 2021.
Tailin Wu, lan Fischer, Isaac Chuang, and Max Tegmark. Learnability for the information bottleneck. In UAI, 2019. URL
.73311
Jingjing Zhang, Osvaldo Simeone, Zoran Cvetković, Eugenio Abela, and Mark P Richardson. Itene: Intrinsic transfer entropy neural estimator. ArXiv, abs/1912 119/1992.07277, 2019.
18

---

Published in Transactions on Machine Learning Research (02/2023)
A Mathematical formulation
This section is written in a standalone succinet format for maximum clarity on the mathematical details. We presume the reader has some familiarity with information bottleneck methods.
A.1 Setup
First we state the definition of transfer entropy:
Definition 1. Given sequences of random variables (X,), (Y), the transfer entropy from (X,) to (Yi at > 0 (with horizon 0 < l S x)) is the conditional mutual information:
T T x = I(Y: (X)aasses
= =SSel - HYY)l())sser (Xi)esset)
where 4 = max(1- 2,
Note that the conditional mutual information between two random variables conditioned another, is just special case of transfer entropy, and all the methods developed apply to that casc.
Latent variable encoder-decoder models seek to learn a conditional density p(ylæ) by learning an encoded latent representation Z of X, with an encoder g(2(z) which factorises the true density as a Markov chain with independent of Y given X (i.c. Z x
ply, 2, ply x)p(2/=)- 2(2l)))(2(z).
(7)
where p(=|=) <(=lær) since we introduce z. As well, use decoder d(yl=) that approximates p(y|z). The functional Markov structure that comes from feed-forward computing z from and y from implicitly assumos that the distribution also factorisable as:
2, = p(a)p))p(pp))))pppee x (z))))(())))p())
Our situation is analogous this, but in setting actually wish to learn a latent encoding Z, of (X,)ssi via a map q. which is conditional on (Yi)ise, from which one is able to decode Y, conditionally on (Y))is: with the decoder d. Put another way, we want to be able to predict Y at time t using the values of and X up to time - 1, but using our encoding of X and Y via Z. Namely, denoting:
z X (Xi)esisns Y (Y)usisn Y' Yv.
(9)
Assuming the conditional independence Y Z,X,Y,, we have 2,d, are such that:
,l/.2, 2, 2. = ply x v)pe(l)))))) = ply/ x, x.)))(2(2.1))
(10)
as illustrated in Figure 10a
Furthermore, require the distribution to be learned by feed-forward encoder-decoder structure, which assumee the following decomposition of joint probability distribution (as illustrated in Figure 10b] equivalent to the conditional independence Y' 1 X Z.),, where d(s/|2, y) is a variational approximation to p(y/12.00 v):
ply,,,, 2,,,,, = p(z, v)p(s' [2,))p(=/z,, v) x p(a, v)dy/// v)q(z) (x,
(11)
We call the method develop using this structure the Transfer Entropy Bottleneck (TEB) algorithm.
19

---

Published in Transactions on Machine Learning Research (02/2023)
Remark 1. As aside, note that for the TEB method will also want to consider the case where there is an existing model for encoding Y, (clv) which could be any existing model for encoding Y with a variable C' (e.g. & pre-trained self-supervised model). Thus, Y is first encoded into its own latent C, which will not be optimised with the derived optimisation procedure (sec Figures 10c and 10d So in reading, keep in mind that in the text one can replace "Y" with "C", and every result still holds.
For the rest of the text assume /. is fixed and we will use the notations defined in Equation above.
y
a
(b)
(c)
(d)
Figure Probabilistic graph representing the random variables and their relationships as assumed by TEB. (a): The actual random variables in the dataset. Independencies include Y' 1 Z|X, Y. (b): The random variables computed by our feod-forward computation. Independencies include Y' 1 x|Z,Y. (c); (d): The actual and feed-forward graphs, respectively, when utilising pre-trained context encoding C. Independencies for (c) include Y' - C, Y' 1 CIY, and Y' 1 C|X,Y, and for (d) include Y' T. LXZZ and Y 1 CI
As is standard in information bottleneck methods, we follow a minimum necessary information principle, in which we wish to learn our latent representation Z such that it captures exactly the necessary conditional information (conditioned on Y) between X and Y'.
Specifically we would like the following to hold, which, in analogy to the usual minimum necessary information principle, we call the Conditional Minimum Necessary Information point CMNT point:
I(Y 1I' ((Y; I(Z;
(CMNI)
Due to the probabilistic structure of our setting, we do not have I(Y'; Z I(Y;; and I(Y"; S I(Z; X directly from the data processing inequalities, opposed to the standard IB setting (where T(Y; Z) < X) and X) < I(Z; Thus, we first need to prove that we can arrive at the CMNI
point by the following procedure:
1. maximizing I(Y':
(CMNI procedure)
minimizing I(Z:
To prove this, first we remind the reader of the chain rule for conditional mutual information here since we will use it throughout the text: Recall, The chain rule for conditional mutual information
B[A; = T(A; B, C) I(A;
T(A; = T(A; T
We then prove the following proposition:
20

---

Published in Transactions on Machine Learning Research (02/2023)
Proposition 3. The following inequalities hold under the independencies implied by Equation 10 and
11
T(Y': I(Y'; (
Proof. Inequality 1 (
< ((Y): By the chain rule
XYY 2 =
+ 1
=I(Y';X,)) =(X""
however, Equation
implies Y 1 Y and so I(Y: () which implics
(YY; & x,Y)
Since
I(Y'; = -1":2)) X(Y"; = (17;xX))-((YY3)
we get that 1(X) (
Incquality 2 XI(Y": - (2;) By the chain rule:
T(x;Y', 2, ((x
= ([Xx22+)(((xx)111)))
however, Equation
implies X Y, and so (( = which implies
I(X:Y' ( ());2)))
Since
[[X,21))1((x;,)) = ((x;)) I(X,Y'))= = -X
we get that ;(Y'; < T(Z;
Remark: 2. Note that in the proof above, if had used Equation 11 and X 1 Y'IZY in the first part in addition to 10 we would have ((Y'; I(Y'; Z|Y), and therefore X(Y) Z(Y"; This shows that our feed-forward generation of Y' cannot represent the true distribution unless it maxinizes I(Y
A.2 transfer entropy bottleneck objective for CMNI learning
We will now prove that can use these structures derive a tractable loss function for optimizing a model in accordance with the CMNI procedure in order to reach the CMNI point. This method is the TEB algorithm.
First, in-line with item of the CMNI procedure we want to maximize 1(Y"; One can decompose the information I(Y'; Z|Y) as
1(Y'; -H + -H(Y'
(12)
Importantly, the term H(Y"(Y) is fully determined by the true data distribution, and thus, can be dropped from the optimization ion procedure, which is why the proportionality holds. However, this will lead to the problem that the information 1(Y'; Z|Y) will not be tractable to calculate from our existing procedure. We will address this in the later scction A.3 with another form of the TEB algorithm.
We readily bound - -,YY from below with:
21

---

Published in Transactions on Machine Learning Research (02/2023)
Z,YY ( (2))
Bp(z.)) [DLL (p(v'l 12, v)1ld("(l,,))) + Bxtnaar) [og(d()'(22,)))
(13)
2 w/ (loll//////////////l (d(y//z,)))) [),
Thus, order to maximize T(Y:; V we can simply maximize (ogdd///) Moreover, since maximizing this bound also minimizes the expected KL divergence
E Piy. (ogg://) v))] Py v))l
we will simulltanoossy also learn to approximate (//ll,,)) with ((,||,, and the bound will become light.
Now, per the second item of the CMNI procedure we want to minimize (Z; To obtain tractable upper bound for T(Z; X|)) one needs to do more work, and use more distributional approximations. In VIB, one bounds [(Z; X) above with
I( S Ep(x) [PkL((())))l(())))
where p(z) is pre-specified fixed prior. We shall that T(Z; X be bounded above similarly, however our "prior" cannot be a hand-set fixed distribution, since it must be relevant to Y. We introduce an additional encoder (y(2/1) for learning a conditional prior on the latent Z that is informative for the full TEB computation. This gives a clear bound of T(Z; X(V), which becomes tighter over training and thus can also serve as an estimato of I(2; X\Y) at convergence (and therefore an estimate of ;
((>|1r, y)
I(2:
E Bp(y.2.0)
log
p(=ly)
<(2100, y)
E Ep(
log
Eptin
(14)
((2|22 v)
]
Ep(v.s..)
log
Eptz, [DKI.(u(2/a)
(y((=l))
Training (=ly) by minimizing this bound, also gives (z(2))) p(zly), since it squeezes the KL divergence of the two.
Equation 13 with Equation 14 gives the Lagrangian loss to minimize for the transfer entropy bottlenock algorithrn:
Theorem 4. Under the independence assumptions implied by Equation 10 and 11 the minimum of the following objective with respect to 4y, d will be at the CMNT point for 7, X, Y,Y" for some haperyorameter
3 >
TEB- Ep(z,)) [DKI(((2/0,
- 3Epu,. Dog (d(y' |>,)))].
(TEB)
A.3 Transfer entropy bottleneck from learned context encoding
Suppose one has already learned an encoding (-(-l)) for Y that provides a latent representation Y. This section will show one can use this encoding in place of Y the TEB algorithm. This can come with many practical advantages like computational efficiency, training efficiency, reuse of existing models, and rapid protolyping when experimenting with multiple configurations hyperparameter settings for the full TEB model.
To avoid confusion with the latent space Z of the TEB model, we will refer to this latent encoding for Y as C. We integrate the computation of C into the TEB model according to Figures 10c and 1od] As promised by remark I here we invoke the fact that C is fixed when following the procedure (CMN procedure), so that Equation 12 holds when replacing "Y" with "C", then all of the results of the previous soctions hold when replacing " with "C", Since we actually care about bottlenecking conditioned on Y rather than C, and measuring the information T(Y'; X rather than T(Y'; X(C), need the following lemma
22

---

Published in Transactions on Machine Learning Research (02/2023)
Lemma 5. If 1 C, Y' CIX,Y, Y' 1 CYY, and qy(cly) is such that I(C; Y') is marimized, then - (Y))
In particular, one has I(C;Y") x 1();)) gives [Xx;Y')) ~ 1(X;Y1)).
Proof. Since Y' L'LC L|Y, the data processing inequality gives [(C; YY) - <(YYY") and so maximizing I(C; Y") takes us to I() = I(YY)) Y"). Now, expanding ;YY;,, Y,C) two different ways gives
= -V
=
and
I(Y'XXYY,)) (Y";X,Y,0)-1((1':> -1X")
Using the facts that I(C:Y") I(Y:Y), XX, by Y 1 YIX,C, and C(X' Y) by Y Y we gel
XIX + I(Y =
=
and therefore 1(Y"; = 1(X;x(I))
The condition that [(C; Y x (Y) via qy (cly), means that we (approximately) do not lose information about Y' using C in Y's place. As such, 4, (2(g)) is a perfectly reasonable prior for the latent Z|Y of the TEB model (which implies that Z is the same vector space as C). Thus, we design the following loss such that we assume p(zly)= = ( by definition. Noting that we required C and thus by to be fixed, this allows us use a new version of the loss function TEE with C, 4y as:
Theorem 6. Under the independence assumptions Y L ZXXC,, Y' Y'Y,,, Y' T CIX,Y, Y' T LXZ, C Y' L Y|X, C, and C ~ 4y(-lu) such that I(C,Y") I(Y,Y", the minimum of the following objective with respect to d will be at the CMNI point for Z,X,Y,Y', for some hyperparameter B > 0:
TEB Ep(() [DKI () BEpua [og(ddy')zzc)) 1/
(TEB))
Let us also suppose now that we have a decoder dy(s' lc.. This will allow us to estimate I(Y': Z(Y) separately, at convergence of the decoders d, dy, even B or 7 are such that we do not compress Z enough to converge to the CMNI point, which can be seen from the following proposition:
Proposition 7. If the distributions d) dy(v/c) are such that d(y'lz,c) p( and dyly' (c) p(y' (c), then
I(Y'; Z Re Ep(y(..e) [log (d(y'Iz, c)) log (d [b]]c)]
Proof. The proof is immediate from the assumptions, and the fact that
IYY; 2(C' Ep(r'...)) [log (p(s/1z, c)) log (o(s/(e))))
The above approximation is readily interpretable, since the expected difference between the log- likelihoods of y", as estimated by d and dy- In practice, the above estimates are good when one knows the form of the output distribution and thus is able correctly parameterize it with our approximate distributions. For example, when the outputs are categorical as in classification task.
23

---

Published in Transactions on Machine Learning Research (02/2023)
Remark 3. Finally, although are assuming the representation C is not trained by the TEB loss and is fixed, this assumption is only needed for the proportionality in Equation 12 If we let By be optimized by the TEB loss, but at the same time continue to also optimize dy such that ((Y; O) stays maximized during the optimization process, then H(Y = -IYY' + H(Y)) remains fixed, and so Equation 12 remains valid.
We find in practice that if we let dy be optimized by the TEB loss, but continue to train Qy to maximize I(Y"; C) in the same way that it was pretrained, T(Y'; C)) does indeed stay maximized and the TEB procedure works as well.
24

---

Published in Transactions on Machine Learning Research (02/2023)
B Additional implementation and experimental details
B.1 Additional model implementation details
All of the implementations were done using PyTorch Paszkk et al. 2019 were initialized using the default He initialization using normal distribution (He et al. 2015). We used Adam optimizer (Kingma & Ba, 2015) with the default parameters; an initial learning rate of 10 with no weight decay. All the experiments were conducted on a single GPU with 48G memory. For all models the latent dimension of 128 was used in all non convolutional hidden layers, including the hidden dimension size of the LSTMs.
The paramcters of the models
The implementation of the TEB model uses two initial encoders for the two streams 4y and Qz both of which feed into two separate LSTMs to aggregate information across the sequence. The encoder depends on the input modality; on images the initial encoders are ResNet18, on a discrete modality like class integers, we used an embedding vector lookup table (i.c. a torch. nn. Embedding). and on time-serics it is just the identity mapping. The hidden state from the Y pathway LSTM is then linearly projected into a 128 3 dimensional vector. The first two 128 dimensional chunks correspond to the means My and log variances log(o}) for the 128 dimensions of the prior 4y (=|y), which are treated being independent Gaussians in each dimension. The last 128 dimensional chunk is fed, along with the output of the initial X stream encoding (r. into an MLP with one hidden layer with ReLU activation function (Nair & Hinton 2010 3 The outputs of this MLP are means Ae and log variances log(o?). These lead to the parametrization of the 128 independent Gaussian dimensions of q(2 x. y) as having mcans jt = My + Ha, and log variances log(a?), and the latent representation Z then sampled from this distribution. We note here that we initialise the last lincar layer of the MLP computing μıx and log(a?) so that μs the zero vector, and log(o?) ) is log(10 -?) every coordinate.
For rotating MNIST and needle in the haystack tasks, the decoder d is positional encoding followed by convolutional network, which results in an image with dimensions (c, h, w), where for rotating MNIST and c == 3 for the needle in a hayslack task. To create the positional encoding, we first arrange a spatial grid of shape (4, h, w) where each channel is the distance to each image border for a given coordinate, normalized into [0,1]. This grid is passed through a learnable linear layer which maps every coordinate into 128 dimensional space, and results a positional encoding of shape (128, h, w). The sampled Z is copied and repeated in each spatial dimension into tensor with same shape as the positional encoding, and is then added to the positional encoding. The output image is then obtained by passing this through space preserving convolutional component of the decoder, where all convolutions have stride and are padded (with reflection padding) to preserve spatial dimensions. For example for rotating MNIST the decoder has the following architecture:
(1): Conv2l(teatuues=-188, kernel_size-(7, 7)), BatchNorm, ReLU
(2): Conv2l(featuuee=66,, kernel_size=(5, 5)), BatchNorm, ReLU
(3): Conv2lffeature==32, kernel_size- 5)), BatchNorm, ReLU
(4) Conv2((featuuee=116, kernel_size= 5)), Batchlorr,, ReLU
(5) Conv2((feature==8, kermel_aize= 3))
(6) Conv2((feature=44, kernel_size- (3, 3)), BatchNorm, ReLU
(7) Conv2l(featuresse, kernel_size=(3, 3))
whereas for the needle in the haystack (ask, the decoder has all the layers above except. for 5 and 6.
Foor the datasets in our experiments, found that using MLP as such not necossary all for performance, and could just use de output Pe, log(a?) directly. However, the MLP may be necessary when using dataset where the processing of the X stream depends on the particular Y example. Thus we kept the MLP the architecture such for full generality.
25

---

Published in Transactions on Machine Learning Research (02/2023)
For multi-comppoent sinusoids task, the decoder d is neuralODE, implemented using the torchdiffeq package. The dynamics of the ODE is parameterized by a MLP with one hidden layer with 50 hidden units and a ReLU activation function, and another MLP with the same hidden units is utilized to map each latent state to the scalar output at each time-step. The decoder uses default dopri5 ODE solver for The ODE solver
lorward propagation, representing Runge-Kulta of order of Sretmaadd-Prinn-Shaappe is tasked to solve an initial value problem for a sequence of time-steps from to 1 second with an interval of 0.05 second. To calculate the gradient, since the memory cost not our concern, we chose to directly backpropagate through the operations of the solver instead of using the adjoint method, for reduce in the running time. Lastly, the output sequence of the decoder starts with the last point of the input, as we found it to be beneficial for the overall performance.
To optimize the TEB loss on these architectures we interpret our output as probabilistic, so that we can calculate the output loglikelihood term in the TEB loss. In our interpretation we assume each pixel each channel is independently distributod with means equal to the output image, and when a variance parameter is required for the distribution we fix the variance to 0.1. This interpretation of the oulput give reconstruction loss gradients proportional to common usual loss functions like mean squared error, el, or various others, with the particular loss depending on the chosen output distribution. For rotating MNIST and multi-component simusoids we choose Gaussian output distribution, giving reconstruction gradients proportional to mean squared error. For the needle in a haystack task we could have used the same interpretation, however training times were siguificantly longer than for rotating MNIST, so instead we interpreted each pixel as a binary intensity, as we found this speed up training time without compromising performance. This gives reconstruction loss proportional to binary cross entropy with logits across pixels.
Also for the needle in haystack experiments, to test the accuracy of the models in reconstructing the correct color out of the 7 color classes, we trained color classifier on the training set. The color classifier is a convolutional network with the following architecture:
(1): Conv2l( kernel_size-dee (5, 5), stride=1, padding-2), ReLU
(2): MaxPool2@terrno_sszze 2), stride=2)
(3): Conv2llfeaturee=32, pernel_size-r (3, 3), stride=1, padding-1), ReLU
(4): MarPool2a(kermee__size 2), stride=2)
(5): Canv2 harnel_size-s (3, 3), stride=1, padding=1), ReLU
(6): (aaeoll2(keerne]_ssioor22 2), stride=2)
(7): : Linear (otepptoutut_eezee
B.2 Rotating MNIST details and samples
The rotating MNIST task was generated from a base dataset of vidoos of 500 examples of rotating MNIST digits (and an additional 500 validation and 500 lesting base videos). In given base video sequence, digit starting upright rotates clunter-e I at every frame, for 6 frames. The dataset was generated from this base dataset by sampling 300 possible random digit changes in the last frame of every possible triple of consecutive frames in a base video, with a given switching probability of 0.5. Note that when a digit is switched, the new digit has the same angle as the digit it is replacing. Also, although a new switched digit is from a different example digit, the class of the switched digit may be the same as the previous one.
Note that we can calculate the ground truth transfer entropy for rotating MNIST task as follows. Firstly, note that digits' class labels the only mutual information captured in T(Y'; XJV), without loss of generality,
26

---

Published in Transactions on Machine Learning Research (02/2023)
use Y', X and Y to denote their classes respectively. Note that the task is such that we have ply' |x, 3)
1, if x
p(zlv.0')
0, otherwise
and we have equal probability for cach digit class
1
ply)
size(Y) 10"
We can decompose transfer entropy for rotating MNIST as follows:
pll. 2, v) log [pl//1, v)
}
I(X);
E
p(u' 1)
2.9.3
Σ ply)((''l)) v) log ((ul(1,,)) ()),
p(v'lv)
Σ p(y)p(y (w) [log (p(w/lw))
v y
(15)
Σ p(u)(('ll)) lv) [log (p(v'lv)) v'v sY
- k) [og (p(s/lv k))l
sYze(V)
p(uly k)) [log (p(v'ly k))]
where & is any of the digit classes. Following the equation, the ground truth transfer entropy is 1.67689 nats, with
0.55, i ply = kly = k)) 0.05, otherwise
We arrange each input sample for the Y module by sequence of 2 input frames of rotating digit preceding a possible switch. The input for the X module da is length 2 sequence of the class of the digit in the next frame. The target Y' is the next frame of the video where a possible digit switch may have occurred.
For these experiments we train the TEB models on 3 seeds, for every B from 0.1 to incremented by tenths, with extra hundredths increments in the chaotic regime in [.1, End end TEB models were trained for 10 epochs whereas TEB' models were trained for 5 epochs. The TEB models used Y module trained for 20 epochs to represent C' with next step prediction of Y' using CEB with 1. In this case for maximum training efficiency and difficulty for TEB', we also fixed the decoder from the Y module and use it for the reconstruction from Z in the full TEB model.
27

---

Published in Transactions on Machine Learning Research (02/2023)
70
2.5
distance to truc transfer ooy
60
2.0
output teglikelyhood
50
1.25
112, x (nats)
1.5
40
1.00
(nats)
1.0
30
0.75
0.5
20
0.50
-.25
TE
tru TE @
-0.5
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
B
(a) TEB training
3.0
30
distance true trensfer trensfer entropy
2.5
-1.50
2.0
output loglikeyhhood
20
1.25
(nats)
1.5
10
1.00
(mats)
1.0
0.75
0.50
-10
0.25
0.0
true
true TE @ 8
-20
-0.5
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
B
(b) TEB testing
Figure 11: Plots the information metric and reconstruction loglikelihood vs 3 incremented by tenths, with extra hundredths increments in the chaotic regime [.1, for TEB testing. The green dashed horizontal line in the left subfigure indicates the true transfer entropy for this dataset, from which we obtain the interpolated value of ((Y"; on average from repeated trials. In the right subfigure, the output loglikelihood of the model trained with the obtain 3* is marked using the red dashed vertical line. Note that the variance of runs highly increases for 3 E (21, but the models in this interval that found a solution with T(Z; X\Y) R T(Y'; XYY are the same models that achieved higher reconstruction performance.
3 the specific value of achieving the true transfer entropy, calculated B for which I(Z; X(Y)
28

---

Published in Transactions on Machine Learning Research (02/2023)
3.5
transfer ooy (nats)
1.50
output loglikelyhaod
1.25
2.5
12 X(C)
1.00
20
0.75
1.5
10
distance
1.0
0-
0.25
true TE @ p
0.5
0.00
10
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
B
(a) TEB training
distance to tru transfer entropy
-10-
1.50
3.0
output loglikelyhood
-15
1.25
2.5
IIZ X (nats)
1.00
(nats)
-20
2.0
0.75
-25
1.5
0.50
1.0
-30-
0.25
true
true TE @ B"
0.5
-35 m
0.00
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
B
(b) TEB in testing
Figure 12: Same plots as Figure
11
but for TEB
29

---

Published in Transactions on Machine Learning Research (02/2023)
Imput
Predicted
S
5
5
2
554
5
2
/
2
2
g 9 3
84
Figure 13: Sample TEB lesting generation with B = 0.2.
30

---

Published in Transactions on Machine Learning Research (02/2023)
Imput
Predicted
4
4
V
5
5
5
8
-
/
1
9.
9
0
3
4 4
Figure 14: Sample TEB' testing generation with 3 = 0.2. Quality is somewhat. worse in comparison to TEB.
For discussion relevant to this, we refer the reader to the end section 3.1 and also Figure 15
31

---

Published in Transactions on Machine Learning Research (02/2023)
Imput
Predicted
4
4
V
5
5
5
S
-
9. /
/
3
4 4
Figure 15: Sample TEB lesting generation with 3 = 10, which helps speed up learning, and with a pre- trained Y module which maximizes C(Y';C) more than in Figure [4 and [2 Y module is trained with the CEB method with y = 100, as opposod to ? = 1 for Figure [4] Note the cleaner generation comparison to Figure 14 especially when the Y module does the majority of the "work" in the case where digits do not switch.
32

---

Published in Transactions on Machine Learning Research (02/2023)
Target
Predicted
6
8
4
&
6
Figure 16: Sample TEB' testing generation for 8 = 1, trained on digits which always switch, and where the Y module is pre-trained on rolating digits that never switch. This demonstrates the flexibility of TEB in modulating pre-trained model with modality. This TEB model was trained for 30 epochs opposed to 5 for all other TEB experiments (or 10 for all other TEB experiments), which improves reconstruction quality and consistency.
B.3 Needle in haystack details and samples
The needle in haystack task base dataset consists of 142 videos of two colored bouncing balls consisting of 6 frames, in each of distinct color classes (and an additional 75 validation, and 75 testing, base videos in each color class). The chosen color classes are all possible RGB triples with values in the alphabet (85, 255), except (85, 85, 85). The balls all have a minimum intensity as bound the distance between images of different colors with the same ball configuration, thus adding difficulty.
in all red, green, blue color channels so
The dataset generated from this base dataset by sampling possible random ball color changes in the last frame of every possible 4-tuple of consecutive frames in a base video, where a new ball color class in the last frame was sampled with probability 0.5. In addition to this, D distractor pixels (we generated datasets for all of D E (5, 10, 15, 20} distractors), and one true or "needle" pixel, was generated and incorporated for each such 4 frame video section according to the described procedure as follows.
First the color of each distractor pixel v E {0. 255) 3 is randomly (uniformly) chosen each frame from the available 7 color classes. Second, the needle pixel's color in each frame is set to the color class of the balls
33

---

Published in Transactions on Machine Learning Research (02/2023)
in the next frame. Note that both distractor and needle pixels have no minimum baseline brightening (so 1.d intensity 85 is replaced with 0). Third, for each pixel including the needle pixel, U1, U2 us ~ U 102) noise is sampled, and that pixel's color (r))ga is replaced with (1v, - 44 w//ss-
To incorporate these D + extra pixels, each frame of the 3 initial bouncing ball frames is repeated D + 1 times, the repetitions are concatenated in the channel dimension, and the D individual colored pixels are added to the top left of the frame; One pixel is added in each of the D + repeated RGB channel groups in random order.
Stacking channets
RG8
Distracton N-1
Needlé
k: random integer in [0, N]
Distractor
Input X single time-step
Input single time-step: Input without channel group
Figure 17: Illustration of repetitions and stackings of the RGB channel groups for input X and Y in the needle in the haystack task.
We arrange each input sample for the X module 2a such a sequence of 3 input frames. The input for the Y module is this same sequence, but with the RGB channel group containing the needle pixel removed. The target Y' is the next frame of the video of the bouncing balls, where a possible color class switch may have occurred.
We trained all models on seeds and, in each seed, tested the model with the best color prediction accuracy of the color classifier on the validation set. The TEB model was trained for 450 epochs on each dataset. We see, the deterministic models are overfitting found the deterministic models stabilised early (Though Validation and training metrics stabilised well to the distractor pixels for larger numbers of distractors); within 300 epochs for 5 distractors (usually within 200 epochs), and well within 400 for all other number of distractors (usually well within 300 epochs), and so we stopped training for deterministic models at 300 epochs for 5 distractors, and 400 epochs for all other distractor numbers.
For the TEB model, we conducted a hyperparameter search for values of B between and 10 and found that since the task difficulty varied greatly with the number of distractors, diffeernt values for B could be best for the different datasets. Also, training difficulty meant that the best validating values of 3 were quite high relative to the rotating MNIST experiment. For example, for 20 distractors we found that values of B between 103h and 10 achieve the best validation performance. To learn a more bottlenecked representation we also gradually decreased 3 during training from the selected hyperparameter.
To generate the results of Table D and 2 with the number of distractors D {5, 15, 20}, we used an initial 8 = 105, decreasing to 10a at epoch 300, and further decreasing to 10 at epoch 400. For D = 10 we used initial 103, decrcased to 102 at epoch 350, and decreased further to 10 at epoch 400.
For the CEB model on 5 distractors task, We also conducted a hyperparameter search for between 0.5 and 100 and found that y = 10 achieved the best validating performance
34

---

Published in Transactions on Machine Learning Research (02/2023)
Input_X (transformed with shifted pixels)
Target
Predicted
Figure 18: Sample TEB training generation for 20 distractors
35

---

Published in Transactions on Machine Learning Research (02/2023)
Input_X (transformed with shifted pixels)
Target
Predicted
Figure 19: Sample TEB testing generation for 20 distractors
36

---

Published in Transactions on Machine Learning Research (02/2023)
Input_X (transformed with shifted pixels)
Target
Predicted
Figure 20: Sample TEB testing generation for 20 distractors, showing prediction errors. At times the color prediction is missed, or balls generaled are uneven uneven/irregule irregular.
Lastly, to demonstrate the performance of TEB on this challenging dataset, we trained CEB with r on separate dataset of 5 distractors with the bouncing balls colors never change for 1000 epochs, and used it the pre-trained Y module to be incorporated into the TEB formulation. This is in essence
37

---

Published in Transactions on Machine Learning Research (02/2023)
simpler task than the normal needle in the haystack task, and the objective of the pre-trained Y module is to predict the localisation of the balls in the target frame while keeping the same color as in the histories. We trained both TEB with 3 = 10 and "deterministic" with the same pre-trained Y module fixed, on the 5 distractors dataset with 0.5 probability of switching the color, but both models suffered severe overfitting quickly. Therefore we increased the size of the dataset by 20 times, and trained both models on it instead for scods. We found the testing color accuracy of TEB is 93.48 3.93 %, while that of the deterministic baseline is 86.13 + 4.67 %. Therefore, introducing an information bottleneck is even more advantageous with a fixed pre-trained Y module, compared to those without as reported in Table | One potential reason for the deterministic baseline's s poorer performance is that the pre-trained Y module might just not be good enough for it. Alternative reason could be that it is still easier to overfit in this scenario, since here the deterministic model has all the information in the X stream just to find the color changes, without any form of regularizations. This demonstrates that TEB has the capability of incorporating a pre-trained model in this challenging task. However, training TEB is considered as a more difficult task than TEB.
B.4 Multi-component sinusoids details and samples
The multi-component sinusoids task dataset consists of 30k, 5k, and 5k time-series signal of 6 sec- onds, for training, validation, and testing, respectively. The sampling rate is 20Hz, corresponding to in total 120 points per signal. Each time-series signal has sinusoidal waves with distinct frequencies f E (0.2, 0.4, 0.6, 0.6,0.8, H. and starting point randomly sampled in [-1, as sub-components, and the order of the 5 components is also random. At the end of the 5th secouds, with 0.5 probability, the frequency of a randomly chosen component will be randomly resampled into one of the 5 frequencies. The objective of the task is to capture the change in frequency and accurately predict the last 20 points of extrapolations given the past.
Each sinusoidal wave of the time-series signal is generated by unrolling a 2-dimensional continuous time linear autonomous dynamical system with purely imaginary eigenvalues.
dy 0 2T f
di - f
0
y
(16)
With initial condition yo lying on the unit circle, the trajectory of y(t) is clockwise rotation on the unit circle, and we used the the second dimension of the trajectory's Cartesian coordinates as the generated sinusoidal wave for frequency f. When there is switch in frequency, the future trajectory will be unrolled according to the updated dynamical system. The dataset was generated by unrolling the dynamical system using dopri5 ODE solver of package torchdiffeq. The frequencies of all sinusoidal waves were recorded during the dataset generation, and to add difficulty, we added U'(-0.15,0. 0.15) noise to the recorded frequencies post-generation.
We arranged each input sample for the Y module by as a length 100 averaged signals of the 5 sinusoidal waves preceding a possible switch. The input for the X module qu as a length 100 sequence of the frequencies of all waves in the next time-step. The target Y' is the length 21 averaged signal, including the length 20 sequence after possible frequency switch may have occurred, as well the last time-step of Y.
We trained all models on 3 seeds and, in each seed, tested the model with the best reconstruction loglikelihood on the validation set. All models were trained for 2000 epochs. We found that most of the runs reached best validation performance within 500 epochs, after which, the validation performance of TEB and deterministic model stabilisod well, while that of latent ODE and deterministic LSTM degraded due to more severe overfitting.
For the TEB and latentODE-VIB, we conducted hyperparameter search for values of B between 10 and 1000 and between 10 and 100, respectively, and we found that 1000 for TEB and 50 for latent VIB achieved the best validating performance. In addition, similar as in the needle in the haystack task, with the initial B = 1000, we adopted the schedule of gradually decreasing 3 by 100 every 100 epochs starting epoch 500, until it reached 3 = 200 at epoch 1200. However, unlike the cases in the neodle in the haystack task, the schedule was less effective, and we only found that it yielded an improving validating performance (only by small margin) in one run. Therefore, the schedule might not be necessary for this task.
38

---

Published in Transactions on Machine Learning Research (02/2023)
Averaged Time Series Signal
0.
0.5
1st Sub compcommenent
2nd sub-component
sub-compponntt
4th cob-oommenentt
5th sub-component
Figure 21: Sample data in multi-compooent sinusoids dataset. The first row shows the signal used for training in this task, made from averaging all sub-components from the other rows. Shown in the subsequent rows, the sub-components are sinusoidal waves with distinct frequencies, with 0.5 probability to switch frequency time-step 100. This example demonstrates switch in the second sub-component from 0.2Hz 1Hz.
39

---

Published in Transactions on Machine Learning Research (02/2023)
Input_y
Target
Predicted
-0.5
0.5
0.0
-0.5
0.5
0.0
-0.5
0.5
0.0
-0.5
0.3
0.0
0.0
0.5
0.5
0.5
0.0
0.0
0.5
0.0
60 30 100 120
Figure 22: Sample TEB testing time-series sequence generation when there is switch at prediction time-step 100, in which the switches are shown in green frame, showing prediction error. Note the large prediction deviation from the target in the first row. This is more rarely seen case when there is error in predictions, more frequently, the error occurs similar to the 4th row, with accumulated error at later time-steps.
40

---

Published in Transactions on Machine Learning Research (02/2023)
C Discussion on TE estimation vs TEB
Here we provide a discussion contrasting methods which estimate transfer entropy with the goals of our TEB method.
The main goal of our paper is learning a transfer entropy bottlenecked representation for prediction, and we provide an estimate of the information transfer of the model only when converged CMNI point rotating MNIST task. On the other hand, there are existing methods which focus on the estimation of conditional mutual information and transfer entropy (Ursino al.) 2020; De La Pava Panche al.) 2019; Zhang et al. 2019). Generally, these methods are algorithms which estimate by: kernel density estimation methods, targeting a Donsker Varadhan bound, or by a mutual information estimation approach to separately estimate each component B, C), (AAC), of I [AA; BJC) = I(A; C) - (AAC').
In principle the bounds for [[Z or I(Y'; Z|Y) we derive may be substituted our algorithm by any estimator for TE, though as the goal of this paper to introduce a variational learning technique for learning a transfer entropy bottlenecked representation, examining the combination of these methods is beyond the scope of this paper. In practice the price paid for using a separate Iearned information estimator to bottleneck the representation are the extra parameters needed for the neural estimation of the information. and a potentially higher variance or less tractable optimisation procedure. For example, Zhhagg et al. (2019) showed that utilizing directly MINE (Belghazi et al. 2018) estimator for transfer entropy would introduce more variance to the loss. In addition, ITENE algorithm proposed in (Zhang et al. 2019) maximizes and minimizes different MINE estimators for each of the chain-ruleddecomposeed parts of the conditional mutual information, and one of these parts will not coincide with the direction of optimization of the TEB, and more generally, conditional IB procedure. For example for I(Z: estimates I [[: via maximizing MINE estimator of I(Z;Y) and minimizing that of I(Z;X,Y) Y), but IB would require the resulted T(Z; X estimator to be minimized as a whole. Therefore one would follow a more complicated optimization procedure which alternates maximizing the statistics network parameters with respect to the information estimate, and the minimizing the bottleneck parameters with respect to the information estimate. This could increase the variance of learning, and may require more steps of optimization for the ITENE phase early in learning, ensure the information estimate provides a meaningful learning signal for the bottleneck.
I(Z;X,Y) I[Z; Y); ITENE
41


Reviews:
[{"review_id": "NxY6dmft36", "reviewer": "Reviewer_k9TR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is great as-is. It is very well written and clear in its exposition. Its literature review introduces CEB and VIB nicely. \n\nThe experiments are clear and show that the TEB works as intended in the presented settings. I especially like the introductory experiment §3.1 and Fig 4.a) which clearly shows the $\\beta$ trade-off and takes time to explain the considerations.\n\nAs such, I think the paper will be both interesting to readers, and its claims are clear and accurate as provided.\n\nThe appendix is also extensive and contains details for reproduction as well as detailed proofs.\n\n### Regarding the TEB:\nPlease correct me if I'm wrong, but as far as I understand, the data processing inequality also applies to the conditional quantities, as such, some of the proofs might not be necessary:\n\nIn particular, for any $y$, we can look at the probability space $p(y',x,z \\mid y)=p(y' \\mid x, y) \\, p(z \\mid x, y)$. Within this probability space, the DPI holds without change; that is, we have:\n$$I(Y'; Z \\mid y) \\le I(Y' ; X \\mid y) \\land I(Y';X \\mid y) \\le I(Z ; X \\mid y).$$\nTaking an expectation over $y$ (and continuity) provides the result.\n\nCan the TEB can also be found from a decomposition of a joint IB between $(Y,X)$ and $Y'$:\n$$\n\\underbrace{I(Y,X;Y')}\\_{\\ge I(C, Z; Y')} = \n\\underbrace{I(Y; Y')}\\_{\\ge I(C; Y')} + \n\\underbrace{I(X; Y' \\mid Y)}\\_{I(\\ge Z; Y' \\mid Y)}?\n$$\n\n### Haystack\n\nFrom my reading, the haystack experiment only uses $X$ as input for the deterministic joint and CEB baselines. Even though $X$ is considered to be unified in the sense that it contains all the information that $Y$ contains, isn't this a harder task? When both $X$ and $Y$ are provided, to find the needle, we only need to subtract $X-Y$. Could this explain the bad performance of the joint baselines? If this is obvious, it was not made clear, and if I misunderstood, the text might benefit from further clarification. \n\nIn that regard, could an example be provided of X and Y in figure 6?\n\n---\nLastly (but independently of the review), will example notebooks or similar be released/", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper derives an information bottleneck objective between separate data streams using the concept of transfer entropy (Transfer Entropy Bottleneck).\n\nIt validates its approach empirically in several settings and analyzes the behavior of its objective for different strengths of the bottlenecks ($\\beta$ hyperparameter).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In addition to the clarifications and suggestions in the previous section, the paper might be improved by slight re-layouting. Some of the figures might be rotated to save space (5, 7, , while other figures could be enlarged (e.g. 1ab, 3ab, 8, 9).\n\nPage 7 and 8 also seem to have a lot of unnecessary whitespace from formatting issues.", "broader_impact_concerns": "The broader impact statement might be removed (I do not think it is necessary for TMLR, and its current form is laudable but very unspecific---I'm not faulting that).", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4Wtft81ldu", "reviewer": "Reviewer_W7uf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths - information theoric view of deep learning provides useful insights and an alternative justification for the latent variable formulation of deep learning (as a representation learning problem with compression motivated objectives). This paper builds on this theoretical framework to justify their model formulation, extending the existing work to the problem of stream / time/series data with asymmetric information transfer. This in my view is both interesting theoretically and potentially useful for practice. As far as I know, the model formulation is also new.\n\nWeaknesses - a rather well-known concept related to asymmetric transfer of information in time-series forecasting is that of Granger causality. The concept and the possible relations to the proposed CMNI are not discussed in the paper at all. While I understand that the paper has to limit its scope somehow, I find that at least a brief discussion and comparison of the approaches is almost necessary in this context.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a 'Transfer Entropy Bottleneck' (TEB) method to train a latent variable model that can predict the future evolution of a target stream based on its own history and, at the same time, benefiting from information available in a coupled source stream. Importantly, the modelling is asymmetric - we only wish to predict the target stream, not the source. For this purpose the authors propose to use the transfer entropy (Schreiber, 2000) as a directional measure of information flow from source to target.\n\nThe TEB is formulated as a conditional latent variable model, building on previous work of Alemi et. al, 2017 (Variational Inforamtion Bottleneck, VIB) and mainly Fischer, 2020 (Conditional Entropy Bottleneck, CEB), the latter introducing the concept of Minimum Necessary Information (MNI) as an alternative to the classical Information Bottleneck (IB) principles (Tishby, 2020). The TEB extends MIN for the conditional setting (Conditional Minimum Necessary Information, CMNI) and in analog with CEB formulates the respective variational bound on TEB as the loss for the method.\n\nThe efficacy of the method is documented on a set of synthetic experiments, which illustrate the ability to transfer information from the source stream to inform the target stream prediction. The effect of the hyper-parameter choice in the loss on the transfer ability and the ability to reach the CMNI point is further discussed. \nReal data experiments as well as discussion of theoretical guarantees of learnability are left for future work. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Major: \n- Please add at a discussion about the similarities with and differences from the concept of Granger causality and mainly the large body of works that tries to learn this from the data.\n- Theorems 1 and 2 are stated without any proofs. The only \"proof\" is the empirical evidence studied in the experiments. I do not think such an \"anecdotal\" evidence is sufficient for calling these statements theorems. Please either provide real mathematical proofs or rephrase avoiding the term \"theorem\". \n- The discussion of Figure 4 is rather confusing. You mention $\\beta_1$ and $\\beta_0$ that as you say \"we can identify\" from the figure. I cannot and I do not see them anyhow identified in the figures directly.\n- Figure 4 itself is confusing - green and red dashed lines should both indicate the true TE but the green is a horizontal line while red is vertical. I think, I am missing something - please correct or explain in description better.\n- In section 3.2 under Figure 6 you explain the deterministic version of TEB. You say this has \"no sampling\". This is, however, the very first place in the paper where you mention some sampling. I can guess what sampling you mean (presumably z ~ q(z|x,y) using a standard reparametrization trick as in VAEs) but would be better not to guess. Please clarify.\n- At the same place you say the model has \"no KL\". What does that mean? I am guessing this boils down to a simple AE architecture. Is this what you mean? Please clarify.\n\nOther:\n- Figure 4 and mainly Figure 9 is barely readable. Please improve! (Size, fonts, etc. Eg. Fig 9 could be split to 2 columns, etc.) \n", "broader_impact_concerns": "I have no specific concerns here.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "L3XtOQcTjg", "reviewer": "Reviewer_7gXH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- This is an example of using an information theoretic objective function to translate domain knowledge of the data into model inductive biases. This is very appealing and could have wide applications.\n- Practically, I like how the same TEB objective is used to optimize all model parameters. The paper doesn't discuss this, but an alternative for learning q(z|x,y) is to use an adversarial model that attempts to predict y from z, such that the two models are optimizing different objectives, resulting in training instabilities. This approach seems to avoid those problems. \n\nWeaknesses:\n- Some of the writing could be more clear, including the motivation, assumptions, approximations, bounds, comparisons of the models in experiment 3.2, and the description of Figure 4.\n- The motivation behind this method is not completely clear. The paper provides two motivations: (1) Small amounts of crucial information might be ignored in the input; (2) we may want to improve an existing supervised model given extra inputs without retraining. Regarding (1), the synthetic datasets used in the experiments appear to exemplify the applications the authors have in mind, but it would be helpful to have a concise description of where this method should be applied. Reason (2) seems like an afterthought, and the TEB method isn't compared to simpler alternatives for this use case. \n- It's not clear to me why the focus is on sequence data. Can't this method be applied to any supervised learning task where the inputs can be divided into a set of primary variables Y and secondary variables X, where X and Y are correlated but X might contain some key information? This seems to be a more general case that would be simpler to consider (especially since the experiments are on synthetic data anyway). \n- All experiments are on synthetic data sets that exhibit particular properties that the TEB aims to exploit. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents an information bottleneck method for supervised learning when the input can be separated into two sets of variables X and Y (e.g. X and Y are both images, or image sequences), such that most of the useful information is contained in Y but some key information must be extracted from X. Ideas from previous work applying information bottleneck methods to deep neural networks are adapted to this particular setting. The result is a loss function (the TEB loss) for training a supervised learning model, and an approximation for the amount of information extracted from X. Experiments on synthetic data demonstrate the advantage of this approach over standard supervised learning. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical: \n- The logic is backwards in Equation 2, which makes the assumption that Y' is independent of X conditioned on Z,Y. This is an assumption in Eq. 2, rather than an implication of it.\n\nJust suggestions:\n- In Appendix B.1, the encoder models use ReLU output units which are initialized to zeros. I would suggest using Softmax output units to avoid problems where the gradients are multiplied by zero when backpropagated through the activation.\n- Appendix A replicates the main text, almost word-for-word in parts. Fig 2 and 10 and the captions are the same.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers have unanimously consented that the paper validates its claims and is of interest to the community.

In particular the reviewers feel as though their concerns have been addressed in the updated version.

I want to thank the reviewers and authors for taking the time and engaging in the process, seems the paper has benefited as a result.
Decision: Accept (poster)