The Variational Bandwidth Bottleneck: Stochastic Evaluation on an Information Budget
Authors: Anirudh Goyal
Yoshua Bengio
Matthew Botvinick
Sergey Levine
Venue:  (2020)
Keywords: Variational Information Bottleneck
Reinforcement learning

Abstract: In many applications, it is desirable to extract only the relevant information from complex input data, which involves making a decision about which input features are relevant.
The information bottleneck method formalizes this as an information-theoretic optimization problem by maintaining an optimal tradeoff between compression (throwing away irrelevant input information), and predicting the target. In many problem settings, including the reinforcement learning problems we consider in this work, we might prefer to compress only part of the input. This is typically the case when we have a standard conditioning input, such as a state observation, and a ``privileged'' input, which might correspond to the goal of a task, the output of a costly planning algorithm, or communication with another agent. In such cases, we might prefer to compress the privileged input, either to achieve better generalization (e.g., with respect to goals) or to minimize access to costly information (e.g., in the case of communication). Practical implementations of the information bottleneck based on variational inference require access to the privileged input in order to compute the bottleneck variable, so although they perform compression, this compression operation itself needs unrestricted, lossless access. In this work, we propose the variational bandwidth bottleneck, which decides for each example on the estimated value of the privileged information before seeing it, i.e., only based on the standard input, and then accordingly chooses stochastically, whether to access the privileged input or not. We formulate a tractable approximation to this framework and demonstrate in a series of reinforcement learning experiments that it can improve generalization and reduce access to computationally costly information.

Published as a conference paper at ICLR 2020
THE VARIATIONAL BANDWIDTH BOTTLENECK: STOCHASTIC EVALUATION ON AN INFORMATION BUDGET
Anirudh Goyal', Yoshua Bengio', Matthew Botvinick², Sergey Levine
ABSTRACT
In many applications, it is desirable to extract only the relevant information from complex input data, which involves making decision about which input features are relevant. The information bottleneck method formalizes this as an information- theoretic optimization problem by maintaining an optimal tradeoff between com- pression (throwing away irrelevant input information), and predicting the target. In many problem settings, including the reinforcement learning problems we consider in this work, we might prefer to compress only part of the input. This is typically the case when we have a standard conditioning input, such as a state observation, and a "privileged" input, which might correspond to the goal of a task, the output of a costly planning algorithm, or communication with another agent. In such cases, we might prefer to compress the privileged imput, either to achieve better gener- alization (e.g., with respect to goals) or to minimize access to costly information (e.g., in the case of communication). Practical implementations of the information bottleneck based on variational inference require access to the privileged input in order to compute the bottleneck variable, so although they perform compression, this compression operation itself needs unrestricted, lossless access. In this work, we propose the variational bandwidth bottleneck, which decides for each example on the estimated value of the privileged information before seeing it, i.e., only based on the standard input, and then accordingly chooses stochastically, whether to access the privileged input or not. We formulate tractable approximation to this framework and demonstrate in a series of reinforcement learning experiments that it can improve generalization and reduce access to computationally costly information.
INTRODUCTION
model that generalizes effectively should be able to pick up on relevant cues in the input while ignoring irrelevant distractors. For example, if one want to cross the street, one should only pay attention to the positions and velocities of the cars, disregarding their color. The information bottleneck (Tishby et al.) 2000) formalizes this in terms of minimizing the mutual information between the boutlenek representation layer with the input, while maximizing its mutual information with the correct output. This type of input compression 2000), and has recently been extended to deep parametric models, such as neural networks where it has been shown to improve generalization (Achille & Soatto 2016 Alemi et al. 2016).
improve generalization (Tishby et al.
The information bottleneck is generally intractable, but can be approximated using variational inference (Alemi al., 2016)). This variational approach parameterizes the information bottleneck model using neural network (i.e., an encoder). While the variational bound makes it feasible to train (approximate) information bottleneck layers with deep neural networks, the encoder these networks the layer that predicts the bottleneck variable distribution conditioned on the input must still process the full input, before is compressed and irrelevant information is removed. The encoder itself can therefore fail to generalize, and although the information bottleneck minimizes mutual information with the input on the training data, it might not compress successfully on new inputs. To
Mila, University of Montreal, Deepmind,
University of California, Berkeley,
cani rudhgoya1 911 90 com

---

Published as a conference paper at ICLR 2020
address this issue, we propose to divide our input into two categories: standard input and privileged imput, and then we aim to design a bottleneck that does not need to access the privileged input before deciding how much information about the input is necessary. The intuition behind not accessing the privileged input is twofold: (a) we might want to avoid accessing the privileged input because we want to generalize with respect to it (and therefore compress it) (b) we actually would prefer not to access it (as this input could be costly to obtain).
The objective is minimize the conditional mutual information between the bottleneck layer and the privileged input, given the standard input. This problem statement is more narrow than the standard information bottleneck, but encompasses many practical use cases. For example, in reinforcement learning, which is the primary subject of our experiments, the agent can be augmented with some privileged information in the form of model based planner, or information which is the result of communication with another agent. This "additional" information can be seen privileged input because it requires the agent to do something extra to obtain it.
Our work provides the following contributions. First, we propose a variational bandwidth bottleneck (VBB) that does not look at the privileged input before deciding whether to use it or not. At a high level, the network is trained first to examine the standard input, and then stochastically decide whether to access the privileged input or not. Second, we illustrate several applications of this approach to reinforcement learning, in order to construct agents that can stochastically determine when to evaluate costly model based computations, when to communicate with another agent, and when to access the memory. We experimentally show that the proposed model produces better generalization, it learns when to use (or not use) the privileged input. For example, in the case of maze navigation, the agent learns to access information about the goal location only near natural bottlenecks, such doorways.
2 PROBLEM FORMULATION
We aim to address the generalization issue described in the introduction for an important special case of the variational information bottleneck, which we refer to as the conditional bottleneck. The conditional bottleneck has two inputs, standard input, and a privileged input, that are represented by random variables and G. respectively. Hence, S, G, Y are three random variables with unknown distribution Pdist (S, G, Y).
The information bottleneck provides us with a mechanism to determine the correct output while accessing the minimal possible amount of information about the privileged input G. In particular, we formulate a conditional variant of the information bottleneck to minimize the mutual information between the bottleneck layer and the privileged input I GJS). given the standard input while avoiding unnecessary access to privileged input G. The proposed model consists of two networks (see Fig. Ty: The encoder network that takes in the privileged input G as well as the standard input S and outputs distribution over the latent variable z such that z ~ p S). The decoder network Pdec( (Y (Z, S) takes the standard input and the compressed representation and outputs the distribution over the target variable Y.
3 VARIATIONAL BOTTLENECK ON STANDARD INPUT AND PRIVILEGED INPUT
The information bottleneck (IB) objective Tishby et al. 2000 is formulated as the maximization of I(Z; Y) [I; X), where X refers to the input signal, Y refers the target signal, Z refers the compressed representation of X, and | controls the trade-off between compression and prediction. The IB has its roots in channel coding, where compression metric I X) represents the capacity of the communication channel between Z and X. Assuming a prior distribution r(Z) over the random variable Z, constraining the channel capacity corresponds to limiting the information by which the posterior p(Z X is permitted differ from the prior r ZZ). This difference can be measured using the Kullback-eeblee (KL) divergence, such that DKL (z (Z)))(((())) refers to the channel capacity.
Now, we write the equations for the variational information bottleneck, where the bottleneck is learnt on both the standard input S as well as a privileged input G. The Data Processing Inequality (DPI) (Cover& & Thomas 2006) for a Markov chain x + - y ensures that [(x; z) 2 I(x: Hence for a bottleneck where the input is comprised of both the standard input as well as privileged input, we have I(Z; G|S) > G|S). To obtain an upper bound on I(Z; G|S), we must first obtain an
2

---

Published as a conference paper at ICLR 2020
upper bound on [Z; G|S s), and then average over ps)). We get the following result: We ask the reader to refer to the section on the conditional bottleneck in the supplementary material for the full derivation.
IZZ G[S) I Σp(s) (p0ss
(I)
THE VARIATIONAL BANDWIDTH BOTTLENECK
We now introduce proposed method, the variational bandwidth bot- tleneck (VBB). The goal of the vari- ational bandwidth bottleneck is to avoid accessing the privileged input G if it is not required to make an in- formed decision about the output Y. This means that the decision about whether or not to access must be made only on the basis of the standard input S. The standard input is used to determine channel capacity, deap. which controls how much information about G is available compute Z.
det vedinge
B(S)
Xyris ~ 5400 1)
Standardipuu
Capudty Natwork
Sangle Vempler
Z
Buthooack
deap denotes the channel capacity, one way satisfy this channel capac- ity is to access the input losslessly with probability deap. and otherwise send no information about the input all. In this communication strategy. we have P(Z|S, G) = o(fenc (S. G)) we choose to access the privileged input (with probability deap), where
Dirged
Figure 1: The variational bandwidth bottleneck: Based the standard input S, the channel capacity network determines the capacity of the bottleneck 7. The channel capacity then determines the probability of accessing the privileged input. In the event that the privileged input is not accessed, no part of the model actually reads its value.
fene (S, G) is a deterministic encoder, and denotes the Dirac delta function. The full posterior distribution PZZ|S, G) over the compressed representation be written weighted mixture of (a) (deterministically) accessing the privileged input and standard input and (b) sampling from the prior (when channel capacity is low), such that is sampled using
Z ~ deap * (ō(fene (S, G))) + (1 deap) * r(z).
(2)
This modified distribution p sZ|S, G) allows us to dynamically adjusts how much information about G is transmitted through Z. As shown in the Figure if dleu set to zero, Z is simply sampled from the prior and contains no information about G. If it IS set to one, the privileged information in G is deterninistically transmitted. The amount of information about G that is transmitted is therefore determined by deap. which will depend only on the standard input S.
This means that the model must decide how much information about the privileged input is required be- fore accessing it. Optimizing the information bottleneck objective with this type of bottleneck requires computing gradients through the term (p G)]][(Z) (as in Eq. [T.. where
p(Z|S, G) is sampled as in Eq. 2 The non-differentiable binary event, whose probability is represented by desp precludes us from differentiating through the channel capacity directly. In the next sections, we will first show that this mixture can be used within variational approximation to the information bottleneck, and then describe practical approximation that allows us train the model with standard backpropagation.
4.1 TRACTABLE EVALUATION OF CHANNEL CAPACITY
In this section, we show how we can evaluate the channel capacity in a tractable way. We learn deterministic function B(S) of the standard input S which determines channel capacity. This
3

---

Published as a conference paper at ICLR 2020
function outputs scalar value for deap E (0, 1), which is treated the probability of accessing the information about the privileged input. This deterministic function B(S) is parameterized as neural network. We then access the privileged imput with probability deap = B()). Hence, the resulting distribution over Z is a weighted mixture of accessing the privileged input fene (S, G) with probability deap and sampling from the prior with probability 1 deap- At inference time, using deap. we sample from the Bernoulli distribution b ~ Bernoulli(dpro) to decide whether to access the privileged input or not.
4.2 OPTIMIZATION OF THE KL OBJECTIVE
Here, we derive the KL divergence objective that allows for tractable optimization of (KI(p(Z/S, G)][r(Z)) (as in Eq. 2
Proposition 1 Given the standard input 8, privileged input g, bottleneck variable 2, and a deter- ministic encoder (8,9), we can express the DKL between the weighed mixture and the prior as
(xs. -day log deap (1 deap) [ogg log(deap p(f desp))
(3)
The proof is given the Appendix, section B This equation is fully differentiable with respect to the parameters of f(g, s) and (ss) deap. making feasible to standard gradient-basee bbasee optimizers.
Summary: As in Eq. 2 we approximate pZ|S G) as a weighted mixture (Sn G) and the normal prior, such that deap (fenc G)) (1 deap) *N 1). Hence, the quantity DKL. (P( G)][r(Z)) can be seen as a bound on the information bottleneck objective. When we access the privileged input G, pay
G which is bounded by Hence, optimizing this objective causes the model to avoid accessing the privileged input when it is not necessary.
equal
DKr. (P(Z GZ)( as in Eq. [
5 VARIATIONAL BANDWIDTH BOTTLENECK WITH RL
In order to show how the proposed model can be implemented, we consider a sequential decision making setting. though our variational bandwidth bottleneck could also be applied to other learning problems. In reinforcement learning, the problem of sequential decision making is cast within the framework of MDPs (Sutton et al., 1998). Our proposed method depends two sources of input, standard input and privileged input. In reinforcement learning, privileged inputs could be the result of performing any upstream computation, such running model based planning. can also be the information from the environment, such as the goal or the result of active perception. In all these settings, the agent must decide whether access the privileged input or not. If the agent decides to access the privileged input, then the the agent pays an "information cost". The objective is to maximize the expected reward and reduce the cost associated with accessing privileged input, such that across all states on average, the information cost of using the privileged information is minimal.
We parameterize the agent's policy (A S, G) using an encoder Pene (Z|S, G) and a decoder Paec A [S, Z). parameterized as neural networks. Here, the channel capacity network B(S) would take the standard input that would be used determine channel capacity, depending on which we decide to access the privileged input as in Section 4.1 such that we would output the distribution over the actions. That is, is A, and To A (S, G) En Ppiv(z G, (dce (,,)). This would correspond to minimizing I(A; G) resulting in the objective
(()) = E., |r| 3I(A; G S) En,lr] 3I G S),
where Er,, denotes expectation over trajectories generated by the agent's policy. We mini- mize this objective with standard optimization methods, such as stochastic gradient descent with backpropagation.
4

---

Published as a conference paper at ICLR 2020
6 RELATED WORK
number of prior works have studied information-theoreti regularization in RL. For instance, van Dijk & Polani (201T) use information theoretic measures to define relevant goal-information, which then could be used to find subgoals. Our work is related in that our proposed method could be used to find relevant goal information, but without accessing the goal first. Information theoretic
measures have also been used for exploration (Still & Precup 2012 2 Mohamed & Rezende [0155] Houthooft et al., 2016 Gregor et al. (016). More recently Goyal et (2.19) proposed InfoBot, where "decision" states are identified by training goal conditioned policy with an information bottleneck. In InfoBot, the goal conditioned policy always accesses the goal information, while the proposed method conditionally access the goal information. The VBB is also related to work on conditional computation. Conditional computation aims to reduce computation costs by activating only a part of the entire network for each example (Bengio et al.) 2013). Our work is related in the sense that activate the entire network, but only conditionally access the privileged input.
Another point of comparison for our work is the research on attention models ((Bahdanau et al. 2014: Mnih al. 2014 Xu et al. 2.15)). These models typically learn policy, that allows them to selectively attend to parts of their input. However, these models still need to access the entire input in order to decide where attend. Our method dynamically decides whether to access privileged information or not. As shown in our experiments, our method performs better than the attention method of Mnih al. (2014).
Recently, many models have been shown to be effective at learning communication in multi-agent reinforcement learning (Foerster et al. 2016 Sukhbaatar al. 2016). (Sukhbaatar al.) 2016) learns a deep neural network that maps inputs of all the agents to their respective actions. In this particular architecture, each agent sends its state as the communication message to other agents. Thus, when each agent takes a decision, it takes information from all the other agents. In our proposed method, each agent communicates with other agents only when its necessary.
Our work is also related to work in behavioural research that deals with two modes of decision making (Dic [985 Kahneman 2003) [lloan] 1996; Botvinick & Braver 2015) Shenhav et al.) 2017): an automatic systems that relies on habits and a controlled system that uses making decision making. These systems embody different accuracy and demand tradc-offs. The habit based system (or the default system) has low computation cost but is often more accurate, whereas the controlled system (which uses some external information) achieves greater accuracy but often is more costly. The proposed model also has two parts of input processing, when the channel capacity is low, agent uses its standard input only, and when channel capacity is high, agent uses both the standard as well as privileged input. The habit based system analogous using only the standard input, while the controlled system could be analogous to accessing more costly privileged input.
extra information for
7 EXPERIMENTAL EVALUATION
In this section, we evaluate our proposed method and study the following questions:
Better generalization? Does the proposed method learn an effective bottleneck that generalizes better on test distributions, as compared to the standard conditional variational information bottleneck?
Learn when to access privileged input? Does the proposed method learn when to access the privileged input dynamically, minimizing unnecessary access?
7.1 BASELINES
We compare the proposed method the following methods and baselines:
Conditional Variational Information Bottleneck (VIB): The agent always access the privileged input, with a VIB using both the standard and the privileged input InfoBot (Goyal al. 2019).
Deterministically Accessing Privileged Input (UVFA): The agent can deterministically access both the state as well as the privileged input. This has been shown to improve generalization in RI. problems UVFA (Schaul al. 2015).
5

---

Published as a conference paper at ICLR 2020
Accessing Information at Cost (AIC): We compare the proposed method to simpler reinforcemenn-leerniin baselines, where accessing privileged information can be formalized as one of the available actions. This action reveals the privileged information, at the cost of small negative reward. This baseline evaluates whether the explicit VBB formulation provides a benefit over more conventional approach, where the MDP itself is reformulated account for the cost of information.
Randomly accessing goal (RAG) Here, we compared the proposed method to the scenario where randomly access the privileged input (e.g., 50% of the time). This baseline evaluates whether the VBB is selecting when to access the goal in an intentional and intelligent way.
7.2 DECIDING WHEN TO RUN AN EXPENSIVE MODEL BASED PLANNER
Model-based planning be computationally expensive, but beneficial in temporally extended decision making domains. In this setting, we evaluate whether the VBB can dynamically choose to invoke the planner as infrequently easy to plan using a planner (like model based planner, which learns the dynamics model of the environment), it not very cheap, as it involves running planner every step (which is expensive). So, here we try to answer whether the agent can decide based on the standard input when to access privileged input (the output of model based planner by running the planner).
possible, while still attaining good performance. While is
Experimental Setup: We consider maze world as shown in Figure Z(a). The agent is represented by blue dot, and the agent has to reach the goal (represented by green dot). The agent has access to a dynamics model of the environment (which is pretrained and represented using pa- rameterized neural network). In this task, the agent only gets a partial view of the surrounding i.c. the agent ob- serves small number of squares in front of it. The agent has to reach the goal position from the start position, and agent can use the pretrained dynamics model to sample multiple plausible trajectories, and the output of the dynamics model fed as a conditional input to agent's policy (similar to (Racanière al., 7)17), thus agent can use this dynamics model to predict possible futures, and then make informed decision based on its current state as well as the result the prediction from the dynamic model.
(a) Maze World
Figure 2: Figure on the left shows the environment, where the agent needs go from blue dot to green dot.
In this setup, the current state of the agent (i.c. the egocentric visual observation) acts as the standard input S. and the result of running the planner acts the privileged input G. In order to avoid running the model based planner, the agent needs decide when to access the more costly planner.
Expensive Inference algorithm
% of times
Near junction In the Hallway
72% 1: 5%
28% 4%
Table I: Running Expensive Model Based Planner
Here, we analyze when the agent access the output of the planner. We find that most of the times agent access the privileged information (output of model based planner) near the junctions as shown in Table
Results:
7.3 BETTER GENERALIZATION IN GOAL DRIVEN NAVIGATION
The goal of this experiment to show that, by selectively choosing when to access the privileged input, the agent can generalize better with respect to this input. We consider an agent navigating through a maze comprising sequences rooms separated by doors, as shown in Figure 7 We use partially observed formulation of the task, where the agent only observes small number of squares ahead of it. These tasks are difficult to solve with standard RL algorithms, not only due to the partial observability of the environment but also the sparsity of the reward, since the agent receives reward only upon reaching the goal (Chevalier- Boisverr et al. 2018). The low probability of reaching the goal randomly further exacerbates these issues. The privileged input in this case corresponds the

---

Published as a conference paper at ICLR 2020
agent's relative distance the goal G. At junctions, the agent needs to know where the goal is that it can make the right turn. While in a particular room, the agent doesn't t need much information about the goal. Hence, the agent needs to learn to access goal information when it is near door, where it is most valuable. The current visual inputs act as a standard input S, which is used to compute channel capacity deap.
FindObjSY
Room
Train
FindObjS7
FindObjS10
Train
RoomN6S6
RoomN12S10
FindObjS5 (UVFA))
40% 1 2%
24% = 3%
RoomN2S4 (UVFA))
66% 3%
49% + 3%
FindObjS5 (InfoBot)
46% 1 4%
22% 1 3%
RoomN2S4 (InfoBot)
72% 2%
55% + 3%
FindObjS5 (RAG)
38% + 3%
12% = 4%
RoomN2S4 (RAG)
60% + 5%
41% % + 3%
FindObjS5 (AIC)
39% + 2%
16% + 4%
RoomN2S4 (AIC)
57% + 10%
43% + 5%
FindObjS5 (VBB)
64% 3%
52% 2%
RoomN2S4 (VBR)
82% + 4%
60 %
+ 2%
(a)
(b)
Table 2: Generalization of the agent to larger grids RoomNXSY envs and FindObj envs. Success of an agent measured by the fraction of episodes where the agent was able to navigate to the goal in 500 steps. Results are averaged over 500 examples, and 5 different random seeds.
Experimental setup: To investi- gate if agents can generalize by se- lectively deciding when to access the goal information, we compare our method to InfoBot ((Goyal et al.) 2019)) (a conditional variant of VIB). We use different mazes for training, validation, and testing. We evaluate unseen distribu- tion of tasks (i.c., more rooms than were seen during training). We exper- iment on both RoomNXSY (X num- ber of rooms with atmost size Y, for more details, refer to the Appendix G) as well as the FindObjSY environ- ment. For RoomNXSY, we trained on RoomN2S4 (2 rooms of at most size 6). and evaluate on RoomN6S6 (6 rooms of at most size 6) and RoomN 12S10 (12 rooms, of at most size 10). We also evaluate on the FindObjSY environment, which consists of 9 connected rooms of size Y - 2 x Y arranged in grid. For FindObjSY, train on FindObjS5, and evaluate FindObjS7 and FindObjS10.
(a) FindObjS7
(b) FindObjS10
generalization
Figure 3: Partially Observable FindObjSX environments The agent is placed the central room. object is placed in one of the rooms and the agent must navigate to the object in a randomly chosen outer room to complete the mission. The agent again receives an egocentric observation (7 x 7 pixels), and the difficulty of the task increases with X. For more details refer to supplementary material.
Results: Tables 3a 3b compares an agent
trained with the proposed method to goal conditioned baseline (UVFA) (Schaul et al. 2015), conditional variant of the VIB (Goyal et al. 2019), as well as to the baseline where accessing goal information is formulated one of the actions (AIC). We also investigate how many times the agent accesses the goal information. We first train the agent on Mul- tiRoomN2S4, and then evaluate this policy on MultiRoomN12210 We sample 500 trajecto- ries MultiRoomN 10een... Ideally, if the agent has learned when to access goal informa- tion (i.e., near the doorways). the agent should only access the goal information when it is near door. We take sample rollouts from the pretrained policy this new environment and check if the agent is near the junction point (or doorway) when the agent access the goal information. Table 3 quantittatively compares the proposed method with
Method
Percentage times
VBB
76% + 6%
InfoBot (Goyal et al. 2019 AIC
60% 3%
62% +
Table 3: Goal Driven Navigation Percentage of time steps on which each method acsess the goal in- formation when the agent is near the junction point (or branching points the maze. We show that the proposed method learns to access the privileged input (in this case, the goal) only when necessary.
7

---

Published as a conference paper at ICLR 2020
different baselines, showing that the proposed method indeed learns to generalize with respect the privileged input (i.c., the goal).
7.4 LEARNING WHEN TO COMMUNICATE FOR MULTIAGENT COMMUNICATION
Next, we consider multiagent communication, where in order to solve task, agents must communi- cate with other agents. Here we show that selectively deciding when to communicate with another agent can result better learning.
We use the setup proposed by Mordatch & Abbeel (2017). The environment consists of N agents and M landmarks. Both the agents and landmarks exhibit different characteristics such as different color and shape type. Different agents can act to move in the environment. They can also be affected by the interactions with other agents, Asides from taking physical actions, agents communicate with other agents using verbal communication symbols. Each agent has private goal that is not observed by another agent, and the goal of the agent is grounded in the real physical environment, which might include moving to particular location. It could also involve other agents (like requiring particular agent to move somewhere) and hence communication between agents is required. We consider the cooperative setting, in which the problem is to find policy that maximizes expected return for all the agents. In this scenario, the current state of the agent is the standard input S, and the information which might be obtained as a result of communication with other agents is the privileged input G. For more details refer to the Appendix D
Experimental setup:
Model
6 Agents
10 agents
Emergent Communication (Mordatch & Abbeel 2017) Randomly Accessing (RAG)
4.85 (100%) + 0.1%
5.44 (100%) + 0.2%
4.95 (50%) + 0.2%
5.65 (50%) + 0.1%
InfoBot (Goyal et al. 2019)
4.81 (100%) ) 0.2%
5.32 (100%) 0.1%
VBB (ours)
4.72 (23%) + 0.1%
5.22 (34%) + 0.05%
Table 4: Multiagent communication: The VBB performs better, as compared to the baselines. In the baseline scenario, all of the agents communicate with all the other agents all the time. Averaged over 5 random seeds.
Here we consider two tasks: (a) 6 agents and 6 landmarks, (b) 10 agents and 10 landmarks. The goal is for the agents to coordinate with each other and reach their respective landmarks. We measure two metrics: (a) the distance of the agent from its destination landmark, and (b) the percentage of times the agent accesses the privileged input (i.e., information from the other agents). Table shows the relative distance as well as the percentage of times agents access information from other agents (in brackets).
Tasks:
Table 4 compares an agent trained with proposed method to (Mordatch & Abbeel 2017) and Infobot (Goyal et al. (019). We also study how many times an agent access the privileged input. As shown in Table (within brackets) the VBB can achieve better results, as compared to other methods, even when accessing the privileged input only less than 40% of the times.
Results:
7.5 INFORMATION CONTENT FOR VBB AND VIB
Task
Infobot
Bernoulli- Reinforce
VBB
Navigation Env
4.45 (100%)
5.34 (74%)
3.92 (20%)
Sequential MNIST
3.56 (100%)
3.63 (65%)
3.22 (46%) %
Model Based RL
7.12 (100%)
7.63 (65%)
6.94 (15%)
Table 5: The VBB performs better, as compared to the baselines. The VBB transmits a similar number of bits, while accessing privileged information fraction of the time (in brackets % of times access to privileged information). Using REINFORCE to learn the parameter of the Bernoulli, does not perform well the proposed method.
Channel Capacity: We can quantify the average information transmission through both the VBB and the VIB in bits. The average information is similar to the conventional VIB, while the input
8

---

Published as a conference paper at ICLR 2020
is accessed only fraction of the time (the VIB accesses it 100% of the time). In order show empirically that the VBB is minimizing information transmission (Eq. in main paper), we measure average channel capacity DIL. (p(z)s, g) [rzz)) numerically and compare the proposed method with the VIB, which must access the privileged input every time (See Table
DISCUSSION
We demonstrated how the proposed variational bandwidth bottleneck (VBB) helps in generalization over the standard variational information bottleneck, in the case where the input is divided into standard and privileged component. Unlike the VIB, the VBB does not actually access the privileged input before deciding how much information about it is needed. Our experiments show that the VBB improves gencralization and can achieve similar or better performance while accessing the privileged input less often. Hence, the VBB provides framework for adaptive computation in deep network models, and further study applying to domains where reasoning about access to data and computation is exciting direction for future work. Current limitation of the proposed method is that it assumes independence between standard input and the privileged input but we observe in practice assuming independence does not seem hurt the results, Future work would be investigate how we can remove this assumption.
ACKNOWLEDGEMENTS
The authors acknowledge the important role played by their colleagues at Mila and RAIL throughout the duration of this work. AG is grateful to Alexander Neitz for the code of the environment used for model based experiments Fig. (2) AG is also grateful to Rakesh R Menon for pointing out the error and giving very useful feedback. The authors would also like thank Alex Lamb, Nan Rosemary Ke, Olexa Bilaniuk, Jordan Hoffmann, Nasim Rahaman, Samarth Sinha, Shagun Sodhani, Devansh Arpit, Riashat Islam, Coline Devin, DJ Strousse, Jonathan Binas, Suriya Singh, Hugo Larochelle, Tom Bosc, Gautham Swaminathan, Salem Lahou for feedback on the draft. The authors are also grateful to the reviewers ICML, NeurlPs and ICLR for their feedback. The authors are grateful to NSERC, CIFAR, Google, Samsung, Nuance, IBM, Canada Research Chairs, Canada Graduate Scholarship Program, Nvidia for funding, and Compute Canada for computing resources. We are very grateful to Google for giving Google Cloud credits used in this project.
REFERENCES
Actions and habits: the development of behavioural autonomy. Philosophical Transactions of the Royal Society B: Biological Sciences, 308(1135),6778, 1985. ISSN 0080-4622. doi: 10. 1098/rstb, 1985,0010 htt.:///.ooo.ttaato../ooooooooon
308/1135/67
Alessandro Achille and Stefano Soatto. Information dropout: learning optimal representatioss reanaiinniiiiiilleeereeeeeaooooo through noise. CoRR, abs/1611..01322 1.01353, 2016.
01353
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy, Deep variational information bottleneck. CoRR, abs/1612.00410, 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint DXXiv:1409.0444 2014.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.348: 2013.
Matthew Botvinick and Todd Braver. Motivation and cognitive control: from behavior to neural mechanism. Annual review of psychology, 66, 2015.
Maxime Chevalier Boisvert and Lucas Willems. Minimalistic gridworld environment for openai gym. https:://itthb.. httppe////gtthhhooo///e/eeeieiie
2018.

---

Published as a conference paper at ICLR 2020
Maxime Chevalier-Boisse Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: First steps towards grounded language learning with human in the loop. arXiv preprint arXivv/8100088272, 2018.
Thomas M.
Cover and Joy A. Thomas, Elements of Information Theory (Wiley Series in Telecom- munications and Signal Processing). Wiley-Intersccence, New York, NY, USA, 2006. ISBN 0471141954..
Jakob Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems. pp. 2137-21 2016.
Anirudh Goyal, Riashat Islam, Daniel Strouse, Zafarali Ahmed, Matthew Botvinick, Hugo Larochelle, Sergey Levine, and Yoshua Bengio. Infobot: Transfer and exploration via the information bottleneck. arXiv preprint arXiv: 10000000 2019.
arXiv preprint Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv:1410.540,, 2014.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint a6 1.07507, 2016.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pp. 1109-1117, 2016.
Michael Janner, Karthik Narasimhan, and Regina Barzilay. Representation learning for grounded spatial reasoning. Transactions of the Association of Computational Linguistics, 1:49-61, 2018.
Daniel Kahneman. Maps of bounded rationality: Psychology for behavioral economics. American
Economic Review, 93(5):1449-1473, December 2003. doi: 10. 10112377/0022338833335555 URL
Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. In
Advances in neural information processing systems, pp- 2204 2212,2222,11 2014.
Shakir Mohamed and Danilo Jimenez. Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. In Advances in neural information processing systems, pp. 2125-2133, 2015.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations, arXiv preprint
04000, 2017.
Sébastien Racanière, Théophane Weber, David Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adria Puigdomenech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, et al. Imagination- augmented agents for deep reinforcement learning. In Advances in neural information processing systems, pp. 5690-5701, 2017.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International conference on machine learning, pp. 1312-1320, 2015.
Amitai Shenhav, Sebastian Musslick, Falk Lieder, Wouter Kool, Thomas L Griffiths, Jonathan D Cohen, and Matthew M Botvinick. Toward rational and mechanistic account of mental effort. Annual review of neuroscience, 40:99-124, 124, 2017.
Steven A Sloman. The empirical case for two systems of reasoning. Psychological bulletin, 119(1):3, 1996.
Susamne Still and Doina Precup. An information -termrrttoo--teoettii approach to curiosity-drive reinforcement learning. Theory in Biosciences, 131(3):13-144, 2012.
Sainbayar Sukhbaatar, Jason Weston. Rob Fergus, et al. End-to-end memory networks. In Advances in neural information processing systems, pp. 2440-2448, 2015.
10

---

Published as a conference paper at ICLR 2020
Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backpropagation. In Advances in Neural Information Processing Systems, pp. 2244-2252, 2016.
Richard S Sutton, Andrew Barto, al. Reinforcement learning: An introduction. MIT press, 1998.
Naftali Tishby, Fernando C. N. Pereira, and William Bialek. The information bottleneck method. CoRR, physics/0004057, 2000.
G. van Dijk and D. Polani. Grounding subgoals in information transitions. 201 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), pp. 105-111, April
2011. doi: 10. 1009/ADPLL.211..66676
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pp. 2048-2057, 2015.
A CONDITIONAL BOTTLENECK
In this section, we construct our objective function, such that minimizing this objective function minimizes T(Y; G) Recall that the IB objective (Tishby et al. (2000) formulated as the minimiza- tion of T(Z, X) (Z, Y), where X refers the input, refers to the model output, , 7 refers to compressed representation or the bottleneck. For the proposed method, we construct our objective as follows: we minimize the mutual information between privileged input and output given the standard input, I(Y, G)S), to encode the idea that the we should avoid unnecessary access to privileged input G. and maximize the I(Z, Y). Hence, for the VBB, using the data processing inequality (Cover & Thomas 2006), this implies that
12 G|S) 2 G|S;
(5)
To obtain an upper bound on I(; S)S), we must first obtain an upper bound on I(G; Z|S = x), and then average over p(s). We get the following result:
Z|S = s) Σ p(gls)p(21s,,0) log p(=(s.g)
p(z[s)
(6)
2.9
We assume that the privileged input G and the standard input S are independent of each other, and hence p(g[s) = p(g). we get the following upper bound:
T(G; 2/5 s) - Σ p(g) Σ Ep(zls, log p(=1s.9)
Pprior
Σ (ottlssaelllss
where the inequality in the last line because we replace p(2]s) with Pprior (2). We also drop the dependence of the prior the standard input While this loses some generality, recall that the predictive distribution p(y]s, ,, 2) is already conditioned on 8, so information about & itself does not need to be transmitted through Therefore, have that [p(2[S)[[r(z) 2 0 Σ. p(2(s) log p(=ls) 2 Σg p(z(s) log r(z). Marginalizing over the standard input therefore gives us
I(2 < Σ p(s) Σ
(8)
ΣE(s p(s) Σ Diotaaaiiallss
We approximate p(z g) as a weighted mixture of Peno g) and the normal prior such that deap (Penc(Fenel s,ols,9 9)) + (I doap) *N 1). Hence, the weighted mixture p(z|s, g) can be seen as a bound on the information bottleneck objective. Whenever we access the privileged input G. pay information cost (equal to [ZZ G) which is bounded by DKL (p(2p()) (. prrsor (2)). Hence, the objective is to avoid accessing the privileged input, such that on average, the information cost of using the privileged input is minimal.
11

---

Published as a conference paper at ICLR 2020
B TRACTABLE OPTIMIZATION OF KL OBJECTIVE
Here, we first show how the weighted mixture can be a bound on the information bottleneck objective. Recall,
P(x)
DKL(PI(Q) = Σ
P(x) log
(9)
Q(x)
DEA
Hence, (p(as, () ppploor (z)) where p(z|s, g) is expressed as mixture of direc delta and prior, and hence it can be written as
DKL (p)())s))))l((()) = DKL (deap * p(z) + (1 - deap) o(f(s,
(10)
Further expanding the RHS using eq. 9 we get
DkL(p(zls.g))lr(a) = deap Bp(z) log p(2)p()))))))))())) Log(deopP(2 + (1- - dosp(
+ - deap) log peg(((( (s, g)) lo g)) deap)
Here, assume the 6(f(s,)) g)) to be zero under the prior (as it is a Direc delta function). This can further be simplified to:
DKLlp(=ls.0))lr(s)) = deap Ep(x) log p(z) ) log(deap) - log(p(=))]
+ (1 deap) [log pl (s, g)) - log(deap?' (s, (8,g)) + (1 - deap)))
And hence, reducing the above term reduces 10 DRL. (p(lss s, 9) (p)rioo(())) our original objective.
C ANOTHER METHOD OF CALCULATING CHANNEL CAPACITY
In the main paper show how can evaluate channel capacity in tractable way. The way we do is to learn a function (SS) which determines channel capacity. Here's another way, which we (empirically) found that parameterizing the channel capacity network helps. In order to represent this function ((S) which satisfies these constraints, we use an encoder of the form (B(S) p(ZcaplS)) such that Zcap ~ N(ccup f (S)), where S refers the standard input, and f f" are learned functions (e.g., for the distribution over Zeap- Here, D ( (BS) 1)) refers to the channel capacity of the bottle- neck. In order to get probability prob out of (SS). we convert B(S) into scalar prob [0, such that the prob can treated as probability accessing the privileged input.
multi-layer perceptron) that outputs and a respectively
prob Sigmoid(Normalizztioon (Batitioonnn(SS)B)) (BS)))
(11)
We perform this transformation by normalizing B(S) such that B(S) E [- (in practice we perform this by clamping B(S) E [-2,2) and then we pass the normalized B(S) through sigmoid activation function, and treating the output probability, prob, we access the privileged input with probability prob. Hence, the resulting distribution over z is weighted mixture of accessing the privileged input fene(s, g) with probability prob and sampling from the prior with probability 1- -robb. Here we assume prior to be (O 1), but can also be learned. At test time, using prob, we can sample from the Bernouilli distribution b ~ Bernoulli(prot) to decide whether to access the privileged input or not.
D MULTIAGENT COMMUNICATION
We use the setup proposed by Mordatch & Abbeel (2017). The environment consists of N agents and M landmarks. Both the agents and landmarks exhibit different characteristics such as different color and shape type. Different agents can act to move the environment. They can also be affected by the interactions with other agents. Asides from taking physical actions, agents
Experimental Setup:
12

---

Published as a conference paper at ICLR 2020
agent
agent
agent
landmark
landmark
Figure 4: Multiagent Communciation: The environment consists of agents and M landmarks. Both the agents and landmarks exhibit different characteristics such as different color and shape type. Different agents can act move in the environment. They can also act be effected by the interactions with other agents. Asides from taking physical actions, agents communicate with other agents using verbal communication symbols
communicate with other agents using verbal communication symbols. Each agent has a private goal which is not observed by another agent, and the goal of the agent is grounded in the real physical environment, which might include moving to particular location, and could also involve other agents (like requiring particular agent to move somewhere) and hence communication between agents is required.
Each agent performs actions and communicates utterances according to policy, which is identically instantiated for all the agents in the environment, and also receive the same reward signal. This policy determines both the actions and communication protocols. We assume all agents have identical action and observation spaces and receive the same reward signal. We consider the cooperative setting, in which the problem is to find policy that maximizes expected return for all the agents.
E SPATIAL REASONING
In order to study generalization across wide variety of environmental conditions and linguistic inputs, (Janner et al. 2018) develop an extension of the puddle world reinforcement learning benchmark. grid are first filled with either grass or water cells, such that the grass forms one
States in a 10 connected component. We then populate the grass region with six unique objects which appear only once per map (triangle, star, diamond, circle, heart, and spade) and four non-unique objects (rock, tree, horse, and house) which appear any number of times a given map. We followed the same experimental setup and hyperparameters as in (Janner et al. 2018).
Here, agent is rewarded for reaching the location specified by the language instruction. Agent allowed to take actions in the world. Here the goal to be able to generalize the learned representation for given instruction such that even if the environment observations this representation is still useful. Hence, we want to learn such representations that ties observations from the environment and the language expressions. Here we consider the Puddle World Navigation map as introduced in (Janner et al. 2018). We followed the same experiment setup as (Janner et al.) 2018). Here, the current state of the agent acts as a standard input. Based this, agent decides to access the privileged input.
rearranged,
We start by converting the instruction text into a real valued vector using an LSTM. first convolves the map layout to a low-dimensional repesentation (as opposed to the MLP of the UVFA) and concatenates this to the L.ST'ss instruction embedding (as opposed to a dot product). These concate- nated representations are then input to two layered MLP. Generalization over both environment configurations and text instructions requires model that meets two desiderata. First, must have flexible representation of goals, one which can encode both the local structure and global spatial attributes inherent to natural language instructions. Second, it must be compositional, in order to learn generalizable representation of the language observed with a single map during training. Namely, the learned representation for a given instruction should still be useful even if the objects map rearranged the layout is changed entirely.
though each unique instruction will only be
13

---

Published as a conference paper at ICLR 2020
Spatial Reasoning
Loss
-3
VBB
CHNESTV
100
200 Epochs
300
400
Figure 5: Spatial Reasoning
Figure 6: Puddle world navigation
F RECURRENT VISUAL ATTENTION LEARNING BETTER FEATURES
The goal of this experiment is to study using the proposed method enables learning dynamic representation of image which can be then used accurately classify image. In order show this, we follow the setup of the Recurrent Attention Model (RAM) (Mnih et al. 2014). Here, the attention process modeled the visual image. A recurrent neural network is trained to process the input sequentially, attending different parts within the image one time and hence combining information from these different parts to build up dynamic representation of the image. The agent incrementally combines information because of attending to different parts and then chooses this integrated information to choose where next to attend to. In this casc, the information due to attending at particular part of the image acts as a standard input, and the information which is being integrated over time acts as privileged input, which is then used to select where the model should attend next. The entire process repeats for N steps (for our experiment = 6). FC denotes a fully connected network with two layers of rectifier units, each containing 256 hidden units.
sequential decision process of goal-directed agent interacting with
Model
MNIST
60 60 Cluttered MNIST
FC (2 layers)
1.69%
11.63%
RAM Model (6 locs)
1.55%
4.3%
VIB (6 locs)
1 1.58%
4.2%
VBB (6 locs) (Ours)
1.42%
3.8%
Table Classification error results Mnih et al. 2014).
Averaged over random seeds.
Quantitative Results: Table 6 shows the classification enror for the proposed model, as well as the baseline model, which is the standard RAM model. For both the proposed model, as well the RAM model, we fix the number locations to attend to equal to 6. The proposed method outperforms the standard R. RAM model.
14

---

Published as a conference paper at ICLR 2020
G ALGORITHM IMPLEMENTATION DETAILS
We evaluate the proposed framework using Advantage Actor-Critic (A2C) to learn policy T g)
conditioned on the goal. To evaluate the performance of proposed method, we use range of maze multi-room tasks from the gym-minigrid framework (Chevalier-Boisse Boisvert & Willems 2018) and the A2C implementation from (Chevalier-Boisss Boisvert & Willems 2018). For the maze tasks, we used agent's relative distance the absolute goal position "goal".
For the maze environments, we use A2C with 48 parallel workers. Our actor network and critic networks consist of two and three fully connected layers respectively, each of which have 128 hidden units. The encoder network is also parameterized as a neural network, which consists of fully connected layer. We use RMSProp with an initial learning rate of 0.0007 to train models, for both InfoBot and the baseline for a fair comparison. Due to the partially observable nature of the environment, we further use a LSTM to encode the state and summarize the past observations.
H MINIGRID ENVIRONMENTS FOR OPENAI GYM
(a)
Multi-
(b)
Multi-
RoomN5S4
RoomN6S8
Figure 7: Partially Observable MultiRommSNXSY environments
The MultiRoom environments used for this research are part of MiniGrid, which is open source gridworld packagd] This package includes a family of reinforcement learning environments compati- ble with the OpenAl Gym framework. Many of these environments are parameterizable that the difficulty of tasks can be adjusted (c.g., the size of rooms is often adjustable).
H.1 THE WORLD
In MiniGrid, the world a grid of size NxN. Each tile in the grid contains exactly zero or one object. The possible object types wall, door, key, ball, box and goal. Each object has associated discrete color, which can be one of red, green, blue, purple, yellow and grey. By default, walls are always grey and goal squares are always green.
H.2 REWARD FUNCTION
Rewards sparse for all MiniGrid environments. In the MultiRoom environment, episodes terminated with a positive reward when the agent reaches the green goal square. Otherwise, episodes terminated with zero reward when a time step limit is reached. In the FindObj environment, the agent receives a positive reward if it reaches the object to be found, otherwise zero reward if the time step limit is reached.
The formula for calculating positive sparse rewards is 0.9 * (step_count/ copeett/mmmsssppp) That is, rewards are always between zero and one, and the quicker the agent can successfully complete an episode, the closer to the reward will be. The max_ste fas__stess___ parameter is different for each environment, and varies depending on the size of each environment, with larger environments having higher time step limit.
15

---

Published as a conference paper at ICLR 2020
Traiiig Convergance
HTM
Figure 8: Copying Task
H.3 ACTION SPACE
There are seven actions in MiniGrid: turn left, turn right, move forward, pick up an object, drop an object, toggle and done. For the purpose of this paper, the pick up. drop and done actions are irrelevant. The agent can use the turn left and turn right action to rotate and face one of 4 possible directions (north, south, east, west). The move forward action makes the agent move from its current tile onto the tile in the direction it is currently facing, provided there is nothing on that tile, or that the tile contains an open door. The agent can open doors if they are right in front of by using the toggle action.
H.4 OBSERVATION SPACE
Observations MiniGrid are partial and egocentric. By default, the agent sees a square of 7x7 tiles in the direction it is facing. These include the tile the agent is standing on. The agent cannot see through walls or closed doors. The observations are provided as a tensor of shape 7x7x3. However, note that these not RGB images. Each tile is encoded using integer values: one describing the type of object contained the cell, one describing its color, and flag indicating whether doors are open closed. This compact encoding was chosen for space efficiency and to enable faster training. The fully observable RGB image view of the environments shown in this paper is provided for human viewing.
H.5 LEVEL GENERATION
The level generation in this task works as follows: (1) Generate the layout of the map (X number of rooms with different sizes (at most size Y) and green goal) (2) Add the agent the map random location in the first room. (3) Add the goal at random location in the last room. MultiRoomNXXS In this task, the agent gets an egocentric view of its surroundings, consisting of 3x3 pixels. neural network parameterized as MI. is used to process the visual observation.
I MEMORY ACCESS DECIDING WHEN TO ACCESS MEMORY
Here, the privileged input involves accessing information from the external memory like neural turing machines (NTM) (Sukhbaatar et al. 2015 Graves et al., 2014). Reading from external memory is usually an expensive operation,
hence we would like to minimize access to the external memory. For our experiments, we consider external memory in the form of neural turning machines. NTM processes inputs in sequences, much like normal LSTM but NTM can allow the network to learn by accessing information from the external memory. In this context, the state of controller (the NTM's controller which processes the input) becomes the standard input, and based on this (the standard input), we decide the channel capacity, and based on channel capacity from external memory or not. In order to test this, we evaluate our approach on copying task. This task tests whether NTMs can store and recall information from the past. We use the same problem setup as (Graves et al. 2014). As shown in fig we found that we can perform slightly better as compared to NTMs while accessing external memory only 32 % of the times.
decide whether to read
16

---

Published as a conference paper at ICLR 2020
J HYPERPARAMETERS
The only hyperparameter we introduce with the variational information bottlemeck is 3. For both the VIB baseline and the proposed method, we evaluated with values of 3: 0.01, 0.09, 0.001, 0.005, 0.009.
J.1 COMMON PARAMETERS
We use the following parameters for lower level policies throughout the experiments. Each training iteration consists of 5 environments time steps, and all the networks (value functions, policy and observation embedding network) are trained at every time step. Every training batch has a size of 64. The value function networks and the embedding network are all neural networks comprised of two hidden layers, with 128 ReLU units at each hidden layer.
All the network parameters updated using Adam optimizer with learning rate 3 10
Table 7
lists the common parameters used.
Parameter
Value
learning rate
3 10
batch size
64
discount
0.99
entropy coefficient
10
hidden layers (Q, V, embedding)
2
hidden units per layer (Q, V, embedding)
128
Bottleneck Size
64
RNN Hidden Size
128
3
0.001/0.0099/0.0100..99
Table 7: Shared parameters for benchmark tasks
K ARCHITECTURAL DETAILS
For our work, we made sure to keep the architecture detail similar to the baseline possible.
Goal Driven Navigation: Our code is based on open source gridworld package https
Multiagent Communication: Our code based on the following open source implementa-
tion. http:s://gittbb...cc////ksss///p
language
Access to External Memory: Our code is based on the following open source imple- mentation of Neural Turing Machines.
com
pytorch-ntm
Spatial Navigation: Our code is based on the following open source implementation of htt.aas///ttttuocc////aaatttttte
The only extra parameters which our model is introduce is related to the channel capacity network. which is parameterized as neural network consisting of 2 layers of 128 dimensions each (with ReLU
non-linearity).
17


Reviews:
[{"review_id": "SJe_pvITtH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary - The paper proposes an approach, called the Variational Bandwidth Bottleneck (VBB), capable of compressing only a part of the input and still learn representations that are informative of the output. The approach is motivated from the following perspective -- there might be situations where a ``part’’ of the input is privileged in the sense that it may be costly / wasteful to maintain access to all the time, thereby rendering the standard IB pipeline infeasible (as it requires unrestricted access to the entire input). By breaking down the input into standard (always available) and privileged (not always available) components, the paper proposes a module that decides whether to access the privileged input during compression based on the standard input. The goal is to be able to decide when to access privileged information and not how to break the overall input into standard and privileged components. The approach tackles a narrow subset of problems compared to the standard information bottleneck. The authors show the applicability of the proposed approach in reinforcement learning setups -- specifically, (1) when to access an expensive model-based planner for goal-driven navigation; (2) when to access goal-information in goal-driven navigation and (3) treating communication in a multi-agent cooperative setting as ``privileged’’ information. The experimental results demonstrate that VBB accesses privileged information in a feasible and minimal manner and results in better generalization performance.\n\nStrengths\n\n- Apart from the flaws (highlighted in weaknesses), the paper is well-written and generally easy to follow.\n\n- The problem statement is well-motivated. The authors did a good job of first identifying when the standard IB approach would be infeasible / costly -- even though representations are compressed and relevant, computing them still requires unrestricted access to input -- and then motivating the need for selective access to information while decision-making. The problem is well-grounded in the experimental settings the authors provide results for.\n\n- Apart from the concerns (highlighted in weaknesses), the experimental results generally support the claims of the paper. Results in Sec. 7.1 (loosely) demonstrate that VBB accesses privileged information states where degree of freedom in terms of possible trajectories is higher -- this result, although not explored in it’s entirety, also correlates with notions of decision-states (as pointed out by authors) and bottleneck states in the existing literature. Results in Sec. 7.2 demonstrate improved generalization performance in terms of transfer. Additionally, results in Table.3 suggest that VBB accesses privileged information the least number of times. Similarly, results in Sec. 7.3 indicate VBB results in improvements in performance in the multi-agent setting while resulting in minimal communication among the agents. The baselines being compared with in the paper are also reasonable.\n\n- The problem statement and the proposed approach have some degree of novelty. Most works along the lines of restricting access to relevant information still assume unrestricted access to the `entire’ information. \n\nWeaknesses\n\nThat being said, the paper does have some flaws / clarity issues that make several associated details confusing when judged in context of prior work. These concerns (in addition to the strengths highlighted above) mostly surrounding experiments form the basis of my rating and addressing these would definitely make the paper stronger.\n\n- One of the underlying motivations / intuitions behind restricting access to the privileged information is that fact that -- “..avoid accessing the privileged input because we want to generalize with respect to it..”. This statement in isolation is misleading and makes things unclear. From a complete generalization standpoint, preventing the encoder from overfitting to the input (‘privileged’ or ‘standard’) can be addressed by just tracking the relevance (predictive performance) based on the representations on a held-out set. Unless I am missing something, generalization of the learned representations in standard IB becomes major problem only when we restrict (costly or otherwise) access to the privileged input. The statement seems to suggest that learning representations completely and generally agnostic to the privileged input is a good idea. However, this is only true when at test-time, the privileged input may be significantly different than what was seen during training time -- a different model-based planner / entirely different goal-specifications, etc. Could the authors comment more on this and reframe the intuition wherever applicable?\n\n- Last 3 lines of the 1st paragraph in section 3 (page 2) are incomplete -- “...constraining the channel capacity .. is permitted to differ from the prior r(Z)....”. The difference in the posterior p(Z|X) and the prior r(Z) quantify the channel capacity only under expectation over the distribution over inputs -- p(X). Therefore, in practice, it is also governed by the empirical distribution of the data (see page 4 of https://arxiv.org/pdf/1612.00410.pdf). The authors should change the lines to reflect the same.\n\n- One of the key contributions of the paper is to develop a mechanism where one isn’t required to access the privileged input all the time -- depending on the standard input, one can decide whether to access the privileged information. This manifests in form of a mixture-distribution over a dirac-delta transformation of the deterministic encoder and a prior distribution. At inference-time, the channel capacity (d_cap, based on the standard input) can be used to decide whether to access the privileged information or not. There’s lack of clarity in terms of what happens at training time -- is it the case that (1) d_cap is computed, f(s, g) is also computed and based on the KL-term in Eq (3), B(S) is incentivized to generate d_cap that results in less frequent access the privileged information (d_cap < 0.5 for the most part) or (2) d_cap is computed, a sample is drawn from the bernoulli and z is sampled from either r(Z) or \\del_(f(S, G) and KL is either 0 (if z ~ r(Z)) or a finite value (if z ~  \\del_(f(S, G))? (1) involves an exact computation and always requires access to G (privileged input) during training whereas (2) is approximate and does not always require access to G. It’s unclear what the pipeline is from the paper. Can the authors clarify this? Are there specific reasons why (1) / (2) was chosen?\n\n- Experimental Results: Highlighting these below:\n     - In the experimental setting of Figure 2, do the authors notice any (1) qualitative differences in terms of where the agent accesses the output of the planner when InfoBot is used (adapted to this setting) and (2) quantitative differences (Table. 2) in terms of how often does InfoBot and the other proposed baselines access the planner output at junctions and hallways? Junctions in 2D mazes can potentially be identified as important decision states using simpler approaches. As a demonstrative experiment, the key point to be made using this experiment seems to be how often does VBB access the planner output compared to InfoBot and other baselines. \n\n    - In Sec. 7.2, it is unclear how VBB is being used to generalize to novel environments -- as in, is the pipeline same as InfoBot, where a frozen encoder is used to provide an exploration incentive? \n\n    - While the generalization results in Table. 2 are impressive in terms of success values, I think it lacks a few numbers (assuming the transfer pipeline to novel environments is same as InfoBot) -- given that the environments being tested on are different from InfoBot, how well do count-based exploration and goal-conditioned A2C baselines perform? This is to understand whether InfoBot is the right thing to compare with in these environments. \n\n    - Furthermore, for the goal-driven navigation set of experiments can the authors report comparisons in terms of sample-efficiency as well -- how do success-rates and average task-returns vary with time-steps of training? \n\n    - For results in Sec. 7.3, Table. 4, I would encourage the authors to compare with IC3Net (https://arxiv.org/pdf/1812.09755.pdf) which learns “when to communicate” in a multi-agent setting irrespective of whether it’s a cooperative situation or not.\n\nReasons for rating\n\nApart from the points mentioned above, I don’t have major weaknesses to point out. The paper is generally easy to follow and the proposed approach and problem statement is well-grounded and somewhat novel. However, the paper suffers from lack of experimental details and comparisons (and other weaknesses highlighted above) and therefore I am inclined towards my current rating. Addressing those would significantly benefit the paper and help me in  increasing my score.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1lnUmn7cH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a type of conditional Information Bottleneck (IB) that addresses the following problem: given that some features may be expensive to obtain for use in prediction, when should they be obtained such that the overall benefit outweighs the cost? A variant of the IB is proposed to model this question. However, optimization is intractable. The paper replaces a certain non-differentiable operation by a deterministic neural network which outputs the probability of seeking the expensive features. The main application here is reinforcement learning, where an agent could compute some plan or communicate with other agents at a cost, and the goal is to solve the task more efficiently while making use of this additional information. It is shown that the proposed method, VBB, makes judicious use of the limited number of costly feature acquisitions it makes, resulting in improved task performance across 3 tasks.\n\nI am not an expert on the topic, but I find that this paper is well-written and tackles a basic question with effective methods that work well in practice. \n\nHowever, since the problem is so basic, I wonder how it connects to active learning or more specifically \"active feature acquisition\":\n\n-Saar-Tsechansky, Maytal, Prem Melville, and Foster Provost. \"Active feature-value acquisition.\" Management Science 55.4 (2009): 664-684.\n-Shim, Hajin, Sung Ju Hwang, and Eunho Yang. \"Joint active feature acquisition and classification with variable-size set encoding.\" Advances in Neural Information Processing Systems. 2018.\n-Ma, Chao, et al. \"Eddi: Efficient dynamic discovery of high-value information with partial VAE.\" arXiv preprint arXiv:1809.11142 (2018). \n\nMinor:\n- \"((Bahdanau et al., 2014; Mnih et al., 2014; Xu et al., 2015))\": double parentheses\n- \"minimizing unnecessary access? . We compare\": drop the . after ?\n- \"to dynamically adjusts\" --> \"to dynamically adjust\"\n- \"The agent always access the\" --> \"The agent always accesses the\"\n- \"Tables 3a, 3b compares\": no such tables in the paper\n- \"each method acsess the\" --> \"each method accesses the\"", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Existing implementation of information bottleneck need access to privileged information which goes against the idea of compression. The authors propose variational bandwidth bottleneck which estimates the value of the privileged information and then stochastically decided whether to access this information or not. They provide a suitable approximation and show that their  method improves generalisation in RL while reducing access to expensive information.

These paper received only two reviews. However, both the reviews were favourable. During discussions with the AC the reviewers acknowledged that most of their concerns were addressed. R2 is still concerned that VBB does not result in improvement in terms of sample efficiency. I request the authors to adequately address this in the final version. Having said that, the paper does make other interesting contributions, hence I recommend that this paper should be accepted.
Decision: Accept (Poster)