Maximum Total Correlation Reinforcement Learning
Authors: Bang You
Puze Liu
Huaping Liu
Jan Peters
Oleg Arenz
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement Learning
Total Correlation
Primary area: reinforcement_learning
TL;DR: Maximizing trajectory total correlation for learning simple and robust RL policies.

Abstract: Simplicity is a powerful inductive bias. In reinforcement learning, regularization is used for simpler policies, data augmentation for simpler representations, and sparse reward functions for simpler objectives, all that, with the underlying motivation to increase generalizability and robustness by focusing on the essentials. Supplementary to these techniques, we investigate how to promote simple behavior throughout the episode. To that end, we introduce a modification of the reinforcement learning problem that additionally maximizes the total correlation within the induced trajectories. We propose a practical algorithm that optimizes all models, including policy and state representation, based on a lower-bound approximation. In simulated robot  environments, our method naturally generates policies that induce periodic and compressible trajectories, and that exhibit superior robustness to noise and changes in dynamics compared to baseline methods, while also improving performance in the original tasks.

Maximum Total Correlation Reinforcement Learning
Bang You
Puze Liu
Huaping Liu 2 Jan Peters 3456 Oleg Arenz
Action State
VVVVVVVVVVVVVVVVVVVVVVVVVVVVVV
MMM RPC M AA
WwW LZ-SAC AMIMMINN
WWW
MTC (ours)
SAC
Figure 1. Maximizing the total correlation within trajectories results in more consistent behavior. As shown in our experiments, this consistency increases robustness to noise and dynamies changes.
Abstract
1. Introduction
Reinforcement learning (RL) is currently the preferred ap- proach for many challenging, practical control problems, as it can learn complex neural network policies that ef- fectively tackle the given task. For example, in robotics, reinforcement learning widely used to learn visuomotor policies for quadrupedal and bipedal locomotion (Lee et al., 2020a; Radosavovic al., 2024). However, since RL is learning- based method, it is prone to picking up spurious correlations between high- high-dimensional sensory inputs and desired actions, which can lead to brittle policies, that fail under slight, natural variations the state. An important countermeasure involves training policies using domain ran- domization, in particular in the sim-to-real setting, where the policy is learned in several varying simulation environ- ments. Yet, even in such data-intense settings, it remains unclear whether we can obtain sufficiently diverse training distribution to learn policies that transfer to more complex real-world scenarios, such as those involving robot con- trollers that need interact with humans.
Simplicity is a powerful inductive bias. In re- inforcement learning, regularization is used for simpler policies, data augmentation for simpler representations, and sparse reward functions for simpler objectives, all that, with the underlying motivation to increase generalizability and robust- ness by focusing on the essentials. Supplemen- tary to these techniques, we investigate how to promote simple behavior throughout the episode. To that end, we introduce modification of the reinforcement learning problem that additionally maximizes the total correlation within the induced trajectories. We propose a practical algorithm that optimizes all models, including policy and state representation, based on a nower-bound approx- imation. In simulated robot environments, our method naturally generates policies that induce periodic and compressible trajectories, and that exhibit superior robustness to noise and changes in dynamics compared to baseline methods, while also improving performance in the original tasks.
Consequently, there is a growing interest in exploring addi- tional techniques that add inductive biases to obtain simpler, less brittle policies, for example, by limiting the amount of state information used by the policy (Goyal et al., 2018; Igl et al., 2019; Lu et al., 2020), or by maximizing the predic- tive information within learned representations (Lee et al., 2020b). Such information-theoot theoretic biases have already been extended to sequences, account for the sequential nature of reinforcement learning. Namely, RPC (Eysenbach et al., 2021) aims to learn better representations by limiting the information between state-sequences and embedding- sequences, and LZ-SAC (Saanum et al., 2023) improves the predictability of the next action given the history of ac- tions. However, these formulations only focus on specific
SShooo of Information Engineering, Wuhan University of Technology, Wuhan, China 'Deparmment Computer Science, Tsinghua University, Beijing, China 'Inteliigent Autonomous Sys- tems Lab, Technische Universität Darmstadt, Darmstadt, Germany "Deutsches Forschungszentrum für Künsttlihhe Intelligenz (DFKI), Germany Hesssin Centre for Artificial Intelligence (Hessian.Al) 'Centre for Cognitive Science (CogSci). Correspondence to: Huap- ing Liu <hpliu@ tsinghua.cdu.cn>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Maximum Total Correlation Reinforcement Learning
aspects of the behavior-either state-consistency action- consistency- considering the complete behavior.
Gao al., 2024), and generalization (Goyal al., 201 8). Motivated by the InfoMax principle (Bell & Sejnowski, 1995), some previous RL methods preserve mutual informa- tion to extract useful representations from observations, and have achieved improvement in terms of performance and robustness on downstream tasks (Kim et al., 2019; Laskin et al., 2020; Mazoure et al., 2020; Rakelly et al., 2021; Dunion et al., 2024). These methods usually maximize mutual information in single transitions. In contrast, our approach maximizes the total interdependencies within the trajectories of an agent. Moreover, instead of using separate objectives for policy and state encoder, we use a unified ob- jective to optimize policy and representations with respect to the consistency within the resulting trajectories.
Instead, we propose a novel inductive bias that operates on the level of trajectories. Specifically, we aim to learn poli- cies that produce simple, consistent, and therefore compress- ible trajectories, exhibiting a tendency towards open-loop behavior. We hypothesize that such behavior is not only more interpretable and predictable, but also more robust to slight variations in the state that the agent inevitably counters due to sensor noise or unmodeled dynamic effects. Following the spirit of Newton's 's first rule (Newton et al., 1934, Book III), we are to admit no more reactions to the state variations than such as are both relevant and sufficient to achieve high expected return. Intuitively, we expect a given behavior that performed well for previous variations to also perform well for future variations. We introduce this inductive bias by means of the additional objective of maxi- mizing the total correlation within the generated trajectories. This total correlation corresponds to the amount of informa- tion that we can save by using a joint encoding of all states and actions within trajectories, compared to compressing all time steps independently. By maximizing total correlation, the agent is encouraged to produce compressible and pre- dictable trajectories, and thereby biased towards open-loop behavior such as clean periodic gaits, without preventing it from performing adaptations when necessary.
Total correlation is a fundamental concept in information theory to qualify the statistical dependency among multiple random variables (Watanabe, 1960). Previous methods have shown that total correlation is an effective tool to enhance machine learning models in many tasks, such as disentan- gled representation learning (Steeg, 2017; Gao et al., 2019) or structure discovery (Ver Steeg & Galstyan, 2014). Our work extends these results the RL setting by observing that the agent can actively change its behavior to maxi- mize consistency within state and action sequences. Our method is also related to previous methods that endow RL agents with robust behavior (Tessler et al., 2019; Tanabe et al., 2022; Reddi et al., 2023; Shi & Chi, 2024). While these methods have proposed purpose-designed methods to achieve robustness benefits, we focus on demonstrating that maximizing the total correlation is simple and effective task lask-independent solution for improving robustness.
The main contributions of our work are as follows. We intro- duce the maximum total correlation reinforcement learning problem (MTC-RL), which extends the typical RL formula- tion with additional objective of maximizing trajectory total correlation. We derive a lower-bound approximation of the total correlation and use it to propose practical al- gorithm for MTC-RI,, based on soft-actor critic (Haarnoja et al., 2018). 1).. Our algorithm features an adaptation scheme to automatically adapt the coefficient of the total correlation objective by treating it as the Lagrangian multiplier of a constrained optimization problem. We empirically evaluate our algorithm on simulated robotic control tasks and show that the learned policies induce more periodic and better compressible trajectories than baseline methods (Eysenbach et al., 2021; Saanum et al., 2023), leading to an improve in performance, as well as robustness to observation noise, action noise, and changes in the system dynamics.'
The principle of simplicity has garnered substantial atten- tion in constructing learning agents (Chater & Vitányi, 2003; Tishby & Zaslavsky, 2015; Grau-Moya et al., 2018; Igl et al., 2019; Goyal et al., 2018; Tishby & Polani, 2010; Leibfried & Grau-Moya, 2020). Some recent works induce simple policies by imposing temporal consistency in actions. For example, Saanum et al. (2023) propose to capture the tem- poral consistency in action sequences and induce simple behaviors by incorporating the preference for consistent ac- tions into the reward function. Another class of methods enforces temporal consistency in latent representations of states to obtain policies that produce simple behaviors. For instance, RPC (Eysenbach et al., 2021) learns policies that visit states whose representations are temporally consistent in individual transitions, by minimizing the mutual infor- mation between a sequence of observations and a sequence of their representations. In contrast, our total correlation objective maximizes the consistency among sequences of state representations and actions. This difference, which corresponds to learning dynamic models that predict the fu- ture from history of actions and states, allows the agent to achieve consistent behavior throughout whole trajectories.
2. Related Work
Information theory provides effective tools to solve prob- lems in RL (Peters et al., 2010; Memmel et al., 2022; Tishby & Zaslavsky, 2015; Ma et al., 2023; Chakraborty et al., 2023), such as representation learning (Oord et al., 2018;
1 Our code is publicly available at https: htttps///itthue /ttp//gttbbb.//////// com/ BangYouU1/ 1/MIC.
2

---

Maximum Total Correlation Reinforcement Learning
Our approach is also related to previous approaches that extract temporally consistent representations from observa- tions by learning latent dynamics models (Guo et al., 2022; Hansen et al., 2022). Unlike these approaches that only consider temporal consistency in representations of states, our total correlation objective aims to maximize the consis- tency among the whole trajectories of state representations and actions. As shown in our experiments (Fig. 5), addi- tionally enforcing the consistency within action sequences by learning the action prediction model improves the per- formance in the presence of environmental perturbations. Some approaches learn representations that discard unneces- sary information in raw states via the Fourier transform (Li & Pathak, 2021; Ye et al., 2023). In contrast, our approach filters out irrelevant state information by maximizing total correlation within trajectories of representations and actions.
and T' is the time horizon. At each time step, the agent ob- serves the current state 8; and selects its actions a, based on its stochastic policy <(a (st) and then receives the reward r(st, at). The original reinforcement learning objective is to maximize the expected cumulative rewards E, -> Nt
where 7 = (81, a1, 82, d2,...) denotes the agent's trajec- tory. As typically not all state information is relevant for choosing the optimal action, we will assume, without loss of generality, that the policy chooses the action based on latent variable 2, [=x(se) using learned encoder f. We refer to the parameters of encoder and policy by 0 and Φ respectively, and we sometimes write Te and fo to make this dependency explicit, however, we typically omit the subscript for brevity.
While we use the finite horizon setting for formulating MTC- RL to ensure that the total correlation of trajectories takes finite values, we will transition to the infinite horizon setting in Section 4.3, by letting I go to infinity and introducing a discount factor 7y In the infinite horizon setting, which underlies the practical implementation used in our experi- ments, the agent aims maximize the expected discounted cumulative rewards E, E>t= rT:...
3. Preliminaries and Notations
In this section, we provide a brief overview of background for information theory reinforcement learning, and intro- duce the notation used throughout the paper.
3.1. Information Theory Background
4. Maximum Total Correlation Reinforcement Learning
Mutual information (MI) is a commonly used statistical dependency measurement in machine learning (Alemi et al., 2017). Given two random variables xj and z2, their mutual information is defined as:
In this section, we introduce the maximum total correla- tion reinforcement learning problem, derive variational lower bound on the total correlation, and use it to formulate an optimization problem that can be solved with existing reinforcement learning methods.
log p(z1, 22) p(x1) ((1))(()))
I(x1:22) Ex,
Total correlation, or multi-information, generalizes mutual information to more than two random variables (Watanabe, 1960; Studený & Vejnarová, 1998). The total correlation C(z1: :e......... ;xx) of n random variables 24, is defined as the Kullback-Leibler (KI. divergence between the joint distribution and the product of their marginals.
4.1. Problem Formulation and Motivation
We want to bias the policy towards producing simpler behav- ior in order to increase its robustness towards state-, action- or casammccs-peturratiin the behavior by the total correlation of the complete tra- jectories induced by the policy, which corresponds their iofommtiinntthorceee sense. More
We quantify the simplicity of
22, 22,
((11:12 = E Ezg......... log
I p(T.)
compressibility in specifically, we extend the vanilla reinforcement learning objective by introducing the additional objective of maxi- mizing the total correlation within the trajectory of latent state representations and actions,
This KL divergence corresponds to the expected amount of information (measured in nats), that we can save when transmitting the sequence (æ1, (21....,,,, zn) using a code that is optimized with respect to the complete sequence, compared to independently encoding each random variable Z4.
3.2. Markov Decision Process
Σ r(s, (()) an) +a(2sia.....00
max 0.6
Ex
We formulate the maximum total correlation reinforcement learning problem in a finite horizon Markov decision process (MDP), denoted by the tuple M = (S, A. p, r, T), where S is the state space, A is the action space, p( |t, a,) is the stochastic dynamic model, r(s, a) the reward function,
(1)
where the hyper-parameter a controls the trade-off between both objectives.
Using the latent representation z rather than the raw states 8
3

---

Maximum Total Correlation Reinforcement Learning
for the total correlation objective serves two main purposes. Firstly, by restricting our total correlation objective task- relevant state information, we focus on learning behavior that is consistent only with respect to aspects of the state that actually matter for the task. The second motivation for formulating the total correlation with respect to the learned state representation to not only learn more consistent behavior, but also more consistent representations 2. By penalizing unnecessary variations in the representation, we aim to learn representations that are more robust to irrelevant variations in the state.
information--eegularized reward function
r' r'(st, ar, 01+1)))) r(se, de, 81+11 + +a(10g log 2n(2 | @1.e) (13-11
(4)
fo(z (
The modified reward biases the policy towards states for which the latent representation can be well predicted based the history, relative to the uncertainty in the encoder pre- dictions, and towards actions that can be well predicted by the action prediction model, relative to the uncertainty of the policy. Although simplified in notation, this reward function also depends on the history and on the current parameters of the policy and encoder. The dependence on past states might suggest the need for a history- based policy; however, our ablations show that providing only the current state as input the policy achieve similar performance. Further- more, despite the non-stationarity introduced by the reward function's 's dependence on policy and encoder parameters, we did not observe learning instabilities.
4.2. A Variational Bound on Total Correlation
The total correlation objective in Eq. can not be decom- posed into sum of step-rewards and involves probability distributions that are typically not available in analytic form. Hence, we replace it with a variational lower bound, us- ing a parameterized history-based latent dynamics model 9,( [21:t, d1:c) and parameterized history-based action prediction model 4x (0.121a, (1:-1),
The latent history-based dynamics and action prediction models get trained using maximum likelihood, and the en- coder and policy get biased towards the history-based pre- dictions, due to the additional objectives of minimizing the KI. divergence towards history-based models.
C(2);11....; ;0r-1;;2) 2 E(21; ai:... ::r-;;;)))
-ET,/ Σ log gr(2c+1l2n 12,e 01.1)xx (ael (aol=13, a1a-1)
(1441/44111)111111111111111)1( (o (acls))
For the practical implementation, we switch to the infinite horizon problem setting by letting T - oo, and introducing the discount factor 7; that is, we optimize the final objective
(2)
Please refer to Appendix A.1 for the derivation. The con- tribution of a given time step to the lower bound is large when the next latent state and the next action can be well predicted based on the history, while accounting for the irreducible uncertainty due to the stochastic encoder f and the policy æ. Hence, this mechanism encourages consistent trajectories. As shown in our experiments, both state con- sistency and action consistency are significantly improved when using the lower bound C within the MTC-RL objective (see Figure 1), which demonstrates that the lower bound captures important aspects of the total correlation.
max E Exa.fo 0.0.7:X
Σ (st,
(5)
As clarified in Appendix A.4 this objective corresponds to maximizing a lower bound on a natural extension of total correlation to infinite sequences.
4.4. Maximum Total Correlation Soft Actor Critic
Our total correlation regularized reinforcement learning problem in Eq. 5 can be optimized straightforwardly with existing RL methods. For our experiments we implement MTC on top of soft actor-critic (SAC) (Haarnoja et al., 2018). As an actor-critic method, SAC alternates between estimating the Q function (policy evaluation) and improving the policy with respect to the Q function (policy improve- ment). SAC considers the maximum entropy RL setting, that is. has the additional objective of maximizing the entropy of the policy, and therefore, it computes the soft-Q function Q (s. a) during policy evaluation, which also accounts for the expected future entropy of the policy. For our policy evaluation, we do not need to make any modifications to SAC, besides replacing the original reward function r(s,, a,) with the regularized reward r (st, a, 8t+1). Hence, we also learn the soft-Q function and use common techniques such
4.3. A Tractable Optimization Problem
By plugging the lower bound Č in Eq. 2 into the objective function Eq. we obtain the tractable objective
T-1
EToio r(srr ar) +
Σ (((s at)
max 0,0.7X
+ a log Go (y( |Z1ee anz) + log Ix (ael=1 (1:21) To (a(()))))
fo ((641118442)
(3)
that we optimize with respect to the parameters of the policy, encoder, and latent dynamics model.
The policy is, thus, optimized with respect to the

---

Maximum Total Correlation Reinforcement Learning
as target nets (Mnih et al., 2015) and dual Q nets (Fujimoto et al., 2018; Haarnoja et al., 2018).
and fair comparison to RPC, we compare MTC to RPC implemented by its original code (referred to as RPC-Orig in Table. 1) and to our implementation of RPC built on top of the same SAC codebase as MTC and LZ-SAC (referred to as RPC). Please refer to Appendix B for details on the implementations of the different approaches.
For policy improvement, however, we also optimize the dy- namics model, the action prediction model and the encoder along with policy. While the prediction models are, thus, trained on the replay buffer instead of using on-policy sam- ples, which slightly deviates from the derived update and may increase the gap of our lower bound, this change allows for an easy integration of the total-correlation regularizer for off-policy optimization. Similar to RPC (Eysenbach et al., 2021), we express the soft-Q function in terms of the regularized reward and the soft Q-function of the next time step, to arrive at the following objective,
5.1. Performance
In our first set of experiments we evaluate the performance on the original reinforcement learning problem. Table. I shows the final performance of our method and baselines on eight continuous control tasks from the DeepMind Control (DMC) (Tassa et al., 2018), (), a commonly used open-source simulated benchmark in RL settings. Learning curves are shown in Fig. 7 in Appendix C.1. MTC achieves better average asymptotic performance than baselines on the ma- jority of the tasks. In particular, MTC outperforms SAC on five tasks, Hopper Stand, Finger Spin, Cheetah Run, Walker Run, and Quadruped Walk. These results suggest that in- ducing simple policies by maximizing the total correlation also benefits policy learning.
max ED. 0.0.0.x
(a [a+++) + (0)
+ a ( log 2z (2(+++1 Ela,, a1:1 a1:)) (xrl))x(ar2z11))) 21:t, (1s-1))
f0(2++1|54+1)
+ +7 (wa + (S411, 0411) - (a log( ((
where S1:++1 and ai are sampled from the replay buffer D. a++1 is sampled from the current policy, and all embeddings Z1:+1 are sampled from the current encoder. The coefficient 3 corresponds to the weight of the entropy regularizer of SAC.
5.2. Zero-shot Robustness
Our main motivation for learning consistent behavior and representations is to improve robustness by focusing on the essentials. Our policies are biased to produce trajectories that have fewer variations, so we expect that they are more robust to unseen disturbances. Hence, we evaluated our method and baselines in terms of zero-shot robustness to observation, action and dynamics perturbations.
Furthermore, instead of choosing the hyperparameter a directly, we optimize it with respect to a desired bound Ip. by minimizing the dual objective
L(a)= = a log (2(2111| /1,e, 01 )g (1..),...00.1))
1n).
Sa(ze41 1844 (a(((()))
Robustness to observation perturbations. We first inves- tigate how observation perturbations affect policy perfor- mance by injecting Gaussian noise into the observations, St + 9t + where noise is sampled from a Gaus- sian distribution, c ~ (O, diag(o?)) with standard de- viation a. Using the same tasks as before, we evaluate our method and baselines on a series of noise strength a [0.02, 0.04, 0.06, 0.08, 0.1]. To compare the robust- ness across all eight tasks, we normalized the scores by the score achieved by the best method on each task. The aggre- gated robustness to observation perturbations with different noise strengths is shown for different methods in Figure 2 (left). MTC achieves the best aggregated performance when observations are perturbed with Gaussian noise.
(7)
5. Experimental Evaluation
We performed experiments to investigate how our to- tal correlation objective compares to vanilla soft-actor critic (Haamooj et al., 2018) and the closely related al- ternative methods RPC (Eysenbach
al., 2021), I.Z- SAC (Saanum al., 2023) and SPAC (Saanum et al., 2023) in terms of performance on the original RL objective (Sec. 5.1 and Sec. 5.4), robustness to noise, dynamics mis- match and spurious correlation (Sec. 5.2), and consistency of the resulting trajectories (Sec. 5.3). Furthermore, we per- formed ablations to investigate the effects for components of our model and total correlation constraints Lp (Sec. 5.5).
Robustness to action perturbations. As our total cor- relation objective encourages consistent actions, we also expect an improvement in terms of robustness to action per- turbations. Hence, we add Gaussian noise to the actions, og < a, where ~ (V diag(o²)) with noise strength a. However, please note that we had to clip the values of the noisy actions to be within [-1, 1]. due to requirements of the simulator.
We build MTC top of the open source implementation of SAC by Yarats et al. (2021). Whereas the official implemen- tation of LZ-SAC provided by Saanum et al. (2023) also uses this SAC implementation, the original implementation of RPC provided by Eysenbach et al. (2021) is based on the SAC implementation from AF-Agents. To ensure a reliable
5

---

Maximum Total Correlation Reinforcement Learning
Table
Scores (means over 20 seeds with 90% confidence interval) achieved by our method and haselines on eight DMC tasks 1 million environment steps. MTC achieves better or at least comparable asymptotic performance than all baselines. In particular, MTC outperforms LZ- LZ-SAC, RPC, and SPAC by large margin on five tasks.
Scores
MTC
RPC
RPC-Orig
LZ-SAC
SPAC
SAC
Acrobot Swingup
184 + 24
132 + 31
2013
100122
110
154 + 29
Hopper Stand
933 12
568 + 96
476 101
593+ 88
213++69
683+ 114
Finger Spin
985
869 + 19
921 + 13
805 38
1361112
9551 18
Walker Walk
967
940 21
951 =
939 26
8834:76
962 7
Cheetah Run
8741 21
772 57
636 + 10
787 17
458+552
811 36
Quadruped Walk
944 5
842 77
598 + 108
595 = 110
505+1185
738 93
Walker Run
790 +
778 +: 25
604 29
732 22
347-99
767 13
Walker Stand
983= 2
980 +
971 +1
977 2
931138
985 2
Robustness Obs Noise
Robustness to Action Noise
Robustness to Mass
Normalized Reward
0.9-
0.9
0.9
0.8
0.8-
0.8
MTC
RPC
0.6
LZ-SAC
0.5
0.5
SPAC
0.5
0.4-
0.4
0.00
0.02
0.04
0.06
0.08
0.10
0.00
0.05
0.10
0.15
0.20
0.25
0.5
0.75
1.0
1.25
3.5
1.75
Noise Strength
Noise Strength
Mass Scale
Figure evaluated the robustness towards ohservation noise (left), action noise (middle) and mass changes (right) on eight tasks from DMC benchmarks. The plots show the normalized mean rewards averaged over 20 independent runs and tasks, with error bars representing 90% confidence interval. For each task we normalized the return by the mean return of the best method. Each run includes 30 evaluation trajectories. MTC achieves better aggregated performance than baselines the presence of perturbations to observations and actions, while also obtaining higher mean rewards when the body mass changed slightly.
Overall, MTC achieves higher average rewards than all baselines even in the presence of strong action perturbations, see Fig. 2 (middle). Notably, our approach outperforms SAC in robustness to action perturbations with different noise strengths, indicating that maximizing trajectory total correlation improves robustness action perturbations.
we introduced additional state dimensions that are not con- trollable by the actor, but instead follow fixed Gaussian transition model. These distractors are not correlated with the remaining states, the actions nor the reward that the agent receives, although by coincidence, it might appear that such correlations exist, resulting in spurious correla- tions. Fig. 11 in Appendix C.9 shows the performance of our method, RPC and SAC. The plot shows the mean over 10 seeds, with a 90% confidence interval. MTC significantly outperforms RPC and SAC in rewards, suggesting that MTC improves robustness to spurious correlations.
Robustness to dynamics mismatches. We also expect sim- ple behavior to be more robust towards deviations between the dynamics encountered during testing compared to the dynamics used for training. test the effects of dynamics mismatch by scaling the mass of each robot body during evaluation. We evaluate six different scaling factors in each environment, namely [0.25, 0.5, 0.75, 1.25, 1.5, 1.75), and present the aggregated results on eight tasks in Fig. 2 (right). Overall, MTC obtains higher averaged scores than all base- lines in the presence of small dynamics changes.
5.3. Trajectory Consistency
Arguably, the consistency of behavior can most straight- forwardly judged by visualizing Hence, we generated plots of the state and action trajectories for MTC and all baselines the Finger Spin task. We already showed the trajectories for the first joint in Fig. 1. The remaining joints are shown in Figure 12 and Figure 13 in Appendix C.
Robustness to spurious correlations. We further per- formed experiments to evaluate the robustness of MTC to spurious correlations the Walker Stand task. To that end,
6

---

Maximum Total Correlation Reinforcement Learning
pendix C.3). These results suggest that encouraging the agent to generate smooth and simple trajectories by maxi- mizing total correlation contributes to successful completion of the manipulation task.
Trajectory Compression
1.0
Normalizze Bytes
0.8
0.6-
We further evaluate our approach on six image-based DMC tasks from the Planet benchmark (Hafner et al., 2019). On these tasks, the policy selects actions based on raw high- dimensional images rather than compact states. To han- dle image observations, use a convolutional neural net- work encoder and random shift augmentation. To ensure fair comparison, each algorithm use the same encoder and image augmentation. We compare the aggregated per- formance across 6 tasks in Fig. 4 (left). MTC achieves better performance than leading baselines, including RPC. CURL (Laskin et al., 2020), SAC-AE (Yarats et al., 2021), and SAC. We refer to Appendix B.9 for more experimental details and Appendix C.4 for scores on individual tasks.
0.4
0.2
0.0
MTC
RPC
LZ-SAC
SAC
Figure 3. The compressed state-action trajectories obtained by MTC have smallest file size in expectation.
Based on these visualizations, we argue that MTC produces the simplest and most consistent trajectories, characterized by highly cyclical patterns.
To support this qualitative assessment, inspired by Saanum et al. (2023), we use lossless compression algorithms to quantify the compressibility of trajectories produced by learned policies. We round the collected state-action trajec- tories to one digit behind the decimal point, save them as .npy-files and compress them using bzip2. Rounding the floating point numbers was necessary to achieve meaningful results because otherwise the highly random insignificant bits would dominate, leading to high variance in the result- ing file sizes. Figure shows the normalized average file sizes in bytes among 30 trajectories of 1000 steps for each of the tasks, with error bars representing 90% confidence interval. The normalized file sizes are achieved by dividing the compressed trajectories by the largest compressed trajec- tory among all methods for each task. Trajectories collected by MTC can be more efficiently compressed than baselines, which suggests that the trajectories produced by our poli- cies show more repetitive, periodic structures to solve tasks. Furthermore, more directly evaluate the predictability of policies, we trained a l-step-ahead action prediction model for different time steps / in data sets that have been collected by policies learned with the different methods. Results in Appendix C.2 show that MTC achieves the smallest predic- tion errors among all methods for all time steps, indicating that actions produced by MTC are more easily predicted.
5.5. Ablation
Minimizing the KL divergence between the encoder and the dynamics model (the third term in Eq. 3) and the KL diver- gence between the policy and the action prediction model (the last term in Eq. 3) both help to induce predictable tra- jectories. To further investigate our regularizer, we evaluate the effect of these two KI. divergences. Specifically, we consider two ablations of MTC, namely MTC-NoA which removes the KL divergence between the policy and the ac- tion prediction model in Eq. 3, and SAC which learns policy without total correlation maximization. We evaluate MTC and its two ablations with respect to original task per- formance, robustness to state, action and dynamics changes on the Walker Stand task and the Cheetah Run task.
Fig, 5 shows the experimental results. Each subplot shows mean and 90% confidence interval from 30 episodes, av- eraged over 20 seeds. On the Walker Stand task, MTC achieves better performance than its two ablations in the presence of observation noise and strong dynamics pertur- bations. When action is perturbed by Gaussian noise, MTC achieves competitive performance compared to MTC-NoA but significantly outperforms SAC. On the Cheetah Run task, MTC achieves better robustness action noise and mass changes, while being comparable to its two ablations in terms of robustness to observation noise. The improvements over MTC-NoA suggest that learning the action prediction model improves robustness. In addition, we also observe that overall MTC-NoA outperforms SAC, indicating that learning the dynamics model achieves robustness benefits.
5.4. Non-periodic and High-dimensional Tasks
Our experiments on the DMC control tasks showed that total correlation regularization can improve the performance and robustness on locomotion tasks, which are characterized by periodic motions. As it is also interesting investigate the performance of MTC also on task that require non-periodic behavior, we further evaluate our method on eight robotic manipulation tasks from the Metaworld benchmark (Yu et al., 2020). MTC achieves higher average rewards than baselines on all tasks (see Fig. right and Fig. 8 in Ap-
We also evaluate the effect of the hyperparameter I, which is used for optimizing the weight of the total correlation objective. Increasing the value of 4 results in larger val- ues of a and therefore biases the agent to increase total correlation. We evaluate the effect of Ip with respect to
7

---

Maximum Total Correlation Reinforcement Learning
image-based DMC Tasks
Push-back-v2
Button-pressse
Daor-lock-v2
1.0
MTC
1.0
Normalized Reward
0.25
IZ5AC
RPC
0.9
0.0
0.20
SAC
0.0
0.6
0.0
0.4
0.10
0.4
0.4
0.2
0.00
0.0
0.0
MTC
Envinronment Step(x10*)
Envinronment Step(x10*)
Envinronment Step(x10")
Figure 4. Left: aggregated performance of MTC and baselines 500K environment steps six image-based DMC tasks. The plot shows the normalized average rewards over runs and tasks, with error bars representing 90% confidence interval. For cach run, we collect 10 evaluation episodes. MTC achieves better performance than baselines. Right: performance of method and baselines three manipulation tasks from Metaworld. The curves represent the average success rate over 10 different runs, with 90% confidence interval. Each run collects 10 evaluation episodes. MTC is competitive to baselines.
Walker Stand
Walker Stand
Walker Stand
1000
950
Episode Reward
960
850
700
800
600
900
MTC
500
MTC NoA
700
880
400
SAC
650
0.2
840
0.50
1.25
1.75
Cheetah Run
Cheetah Run
Cheetah Run
900
800
Episode Reward
800
600
700
600
400
200
0.2
0.4
0.6
0.50
1.25
1.75
Obs Noise Strength
Action Noise Strength
Mass Scale
Figure 5. We test the robustness of MTC and its two ablations, MTC-NoA and SAC, on the Walker Stand task and the Cheetah Run task. Overall, MTC achieves better or least comparable average rewards in the presence of observation noise (left column), action noise (middle column), and mass changes (right column) than its ablations.
original task performance, compressibility, and robustness to state, action, and dynamics perturbations the Walker Stand task. Fig. 9 in Appendix C shows the experimental results, Each subplot shows mean and 90% confidence inter- val from 30 episodes, averaged over 20 seeds. We observe that tightening the lower bound our total correlation ob- jective by increasing Ip doesn't 't hurt the final performance (rewards without perturbations) but significantly decreases the encoding size of trajectories. This suggests that maxi- mizing the lower bound of the total correlation helps induce compressible or structured behaviors. We also find that in-
creasing I effectively improves the robustness of learned policies to observation noise, action noise, and changes in dynamics (see Fig. 9). This supports our claim that biasing policies to focus on the essentials helps increase robustness to perturbations.
6. Discussions and Limitations
Our regularizer in Eq. 4 is related to the regularizer of RPC (Eysenbach et al., 2021), but generalizes it by con- sidering the previous trajectory instead of only using the
8

---

Maximum Total Correlation Reinforcement Learning
information of the current step I, and by also including an action prediction model. These differences enable us to improve temporal consistency within trajectories, which sig- nificantly improves the consistency and robustness of the resulting behavior, as shown in our experiments. Further- more, the regularizer in RPC was derived as the negative of an upper bound on the mutual information between raw and whereas we prove in Appendix A.2, that our objective can not be derived from that perspective. We can, however, derive RPC from our formulation, showing that maximizing total correlation pro- vides an important new perspective on regularization in reinforcement learning that not only results in more consis- tent and robust behavior, but also deepens our theoretical understanding of related works.
Acknowledgement
Funded by the National Key Research and Development Program of China Project number 2024YFB4708000. the German Research Foundation (DFG) Project number PE 2315/18-1, and the German Federal Ministry of Re- search, Technology and Space (BMBFTR) Project number 011S23057B. This project has been supported by hardware donation by NVIDIA through the Academic Grant Program.
latent state sequences, [($1:T)
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There many potential societal consequences of our work, none which we feel must be specifically highlighted here.
However, our lower bound of the total correlation corre- sponds to a sum of negated KI. divergences, and is therefore always negative. Hence, it is not useful for estimating the ac- tual total correlation, which we know to be positive. While a vacuous bound may not be useful for estimation, it can still be valuable for optimization, as in the case of subtracting a constant offset from the true objective, As demonstrated in our experiments, our lower bound is very effective for producing consistent behavior.
References
Alemi, A. A., Fischer, I., Dillon, J. V., and Murphy, K. Deep variational information bottleneck. In International Conference on Learning Representations, 2017.
Bai, C., Wang, L.,, Han, L.., Garg, A., Hao, J., Liu, P., and Wang, Z. Dynamic bottleneck for robust self-supervised exploration. Advances in Neural Information Processing Systems, 34, 2021.
7. Conclusion and Future Work
Bell, A. J. and Sejnowski, T. J. An information- maximization approach to blind separation and blind deconvolution. Neural computation, 119-1119, 1995.
Auxiliary objectives that create inductive biases towards simpler solutions (regularizers) are commonly, and very successfully, used in machine learning to learn more gener- the total tra- novel regularizer for reinforcement
alizable and robust solutions. We propose jectory correlation learning, which acts on the level of the behavior. By directly corresponding to the informmtiiontthoreete compressibility of the induced trajectories, the total correlation is arguably the most principled choice to quantify the simplicity of behavior. As directly maximizing the total correlation is intractable, we derived variational lower bound and used it to formulate a regularized reinforcement learning problem that can be solved with standard techniques. Compared to similar sequence-based regularizers, total correlation reg- ularization achieved very promising results by producing more consistent behavior that is more robust to state-, action- and dynamics perturbations. Hence, we believe that total trajectory correlation may serve an important goal post for future reinforcement learning methods. Developing alter- nate bounds or approximations that better capture the total correlation while maintaining tractability is a promising direction for future research.
Chakraborty, S., Bedi, A. S., Koppel, A., Wang, M., Huang, F., and Manocha, D. Steering: Stein information di- rected exploration for mosel-based reinforcement learn- ing. arXiv preprint 1.Xii:230111203:: 2023.
Chater, N. and Vitányi, P. Simplicity: unifying principle in cognitive science? Trends in cognitive sciences, 7(1): 19 19-22, 2003.
Dunion, M., McInroe, T., Luck, K. S., Hanna, J., and Al- brecht, S. Conditional mutual information for disentan- gled representations in reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Eysenbach. B., Salakhutdinov, R. R., and Levine, S. Robust predictable control. Advances in Neural Information Processing Systems, 34, 2021.
Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In Interna- tional conference on machine learning, pp. 1587-1596. PMLR, 2018
Gao, S., Brekelmans, R., Ver Steeg, G., and Galstyan, A. Auto-encoding total correlation explanation. The 22nd

---

Maximum Total Correlation Reinforcement Learning
international conference on artificial intelligence and statistics, pp. 1157-1166, PMLR, 2019.
abc5986.
URL https://ww..ste wwtp://wwwww.ii.ee///eeeee org/ abcc586.
Gao, Y. Zhang. R., Guo, Wu, F., Yi, Q. Peng, S., Lan, S., Chen, R., Du, Z., Hu, X., et al. Context shift reduction for offline meta-reiiforreement learning. Advances in Neural Information Processing Systems, 36, 2024.
Lee, K.-H., Fischer, I., Liu, A., Guo, Y., Lee, H., Canny, J., and Guadarrama, S. Predictive information accelerates learning in rl. Advances in Neural Information Processing Systems, pp. 11890-11901, 2020b.
Goyal, A., Islam, R., Strouse, D., Ahmed, Z., Larochelle, H., Botvinick, M., Bengio, Y., and Levine, S. Infobot: Transfer and exploration via the information bottleneck. In International Conference on Learning Reprexentations, 2018.
Leibfried, F. and Grau-Moya, J. Mutual-informatti regu-
larization in markov decision processes and actor-critic learning. In Conference Robot Learning, pp- 360-373. PMLR, 2020.
Li, A. and Pathak, D. Functional regularization for rein- forcement learning via learned fourier features. Advances Neural Information Processing Systems, 34:1 14:46 19055, 2021.
Grau-Moya, J., Leibfried, F., and Vrancx, P. Soft q-learning with -nforal-inooraaiion regularization, In International conference on learning representations, 2018.
Guo, Z., Thakoor, S., Pislar, M., Avila Pires, B., Altché, F., Tallec, C., Saade, A., Calandriello, D., Grill, J..B., Tang, Y., et Byol-explore: Exploration by bootstrapped prediction. Advances in neural information processing systems, 01558855-18770 2022.
Lu, X., Lee, K., Abbeel, P., and Tiomkin, S. Dynamics generalization via information bottleneck in deep rein- forcement learning, arXiv preprint arXiv:2008.006 00614, 2020.
Ma, X., Kang, B., Xu, Z. Lin, M., and Yan, Mutual information regularized offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2023.
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: fa/p:002290000 2018.
Mazoure, B., Tachet des Combes, R., Doan, T. L.., Bachman, P., and Hjelm, R. D. Deep reinforcement and infomax learning. Advances in Neural Information Processing Systems, 23:3686-3699 2020.
Hafner, D., Lillicrap. T., Fischer, I., Villegas, R., Ha, D., Lee, H., and Davidson, J. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pp. 2555-2566 PMLR, 2019.
Memmel, M., Liu, Tateo, D.,, and Peters, J. Dimensional- ity reduction and prioritized exploration for policy search. In International Conference on Artificial Intelligence and Statistics, pp. 2134-2157. PMLR, 2022.
Hansen, N., Wang, X., and Su, H. Temporal difference learning for model predictive control. In International Conference on Machine Learning, PMLR, 2022.
Igl, M., Ciosek, K., Li, Y., Tschiatschek, S., Zhang, C.,, Devlin, S., and Hofmann, K. Generalization in rein- forcement learning with selective noise injection and in- formation bottleneck. Advances in Neural Information Processing Systems, 32:13978-11 13990, 2019.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidje- land, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. nature, 51 518(7540): 529-532 533, 2015.
Kim, H., Kim, J., Jeong, Y., Levine, S., and Song. H. O. Emi: Exploration with mutual information. In Interna- tional Conference on Machine Learning, pp. 3360-3369. PMLR, 2019.
Newton, I. Motte, A., and CAJORI, F. RULES OF REASONING IN PHILOSOPHY. University of California Press, edition, 1934. ISBN
9780520364257. URL httt:://www.sstor.rrr. 0t92l//1j.844153..33
Laskin, M., Srinivas, A., and Abbeel, P. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, pp. 5639-5633 PMLR, 2020.
Oord, A. v. d., Li, Y., and Vinyals, O. Representation leam- ing with contrastive predictive coding. arXiv preprint arXiv: faln::/8703344 2018.
Lee, J., Hwangbo, J., Wellhausen. L., Koltun, V., and Hutter, M. Learning quadrupedal locomo- tion over challenging terrain. Science Robotics, 5 (47))eabe5986, 2020a. doi: 10..126/ssssoooties.
Peters, J., Mulling, K., and Altun, Y. Relative entropy policy search. In Twenty- Fourth AAAI Conference on Artificial Intelligence, 2010.
10

---

Maximum Total Correlation Reinforcement Learning
Radosavovic, I., Xiao, T., Zhang, B., Darrell, T., Ma- lik, J., and Sreenath, K. Real-world humanoid locomotion with reinforcement learning. Science Robotics, 9889).cad99579, 2024. doi: 10. 10.1ioo/scisbboticcs adi9579. URL abs/10 112.1112666 htt.://///ciooooiiissooo adi9579.
Tishby, N. and Zaslavsky, N. Deep learning and the infor-
mation bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pp. 1-5. IEEE, 2015.
Ver Steeg, G. and Galstyan, A. Discovering structure in high-dimensional data through correlation explanation. Advances in Neural Information Processing Systems, 27, 2014.
www.eee.......eeeeeeeeeeeeeeeeee org/doiiioooooooooooooooooooo.o. /
Rakelly, K., Gupta, A., Florensa, C., and Levine, S. Which inutual-informatioo representation learning objectives are sufficient for control? Advances Neural Information Processing Systems, 34:26345-263577 2021.
Watanabe, Information theoretical analysis of multivari- ate correlation. IBM Journal of research and development, 4(1):66-82, 1960.
Reddi, A., Tölle, M., Peters, J., Chalvatzaki, G., and D' D'Eramo, C. Robust adversarial reinforcement learn- ing via bounded rationality curricula. In The Twelfth International Conference on Leurning Representations, 2023.
Yarats, D. and Kostrikov, I. Soft actor-critic (sac) im- plementation in pytorch. htttps://githie com/ denisyarats/pytoree sac, 2020.
Yarats, D., Zhang, A., Kostrikov, I,, Amos, B., Pineau, J., and Fergus, R. Improving sample efficiency in model- free reinforcement learning from images. Proceedings of the AAAI Conference on Artificial Intelligence, num- ber 12, pp. 10674-10681, 2021.
Saanum, T., Elteto, N., Dayan, P., Binz, M., and Schulz, E. Reinforcement learning with simple sequence pri- ors. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Ye, M., Kuang, Y., Wang. J., Rui, Y., Zhou, W., Li, H., and Wu, F. State sequences prediction via fourier trans- form for representation learning. Advances in Neural Information Processing Systems, 36:67565-67583 2023.
Shi, L. and Chi, Y. Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity. Journal of Machine Learning Research, 25 (200):1-91, 2024.
Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, Meta-world: A benchmark and evalua- tion for multi-tast and meta reinforcement learning. In Conference on robot learning. pp. 1094-1100, PMLR, 2020.
Steeg, G. V. Unsupervised learning via total correlation planation. In Proceedings of the 26th International Joint Conference Artificial Intelligence, pp. 5151-5155, 2017.
Studeny, M. and Vejnarová, J. The multiinformation func- tion as tool for measuring stochastic dependence. Learn- ing in graphical models, pp. 261-297, 1998.
Tanabe, T., Sato, R., Fukuchi, K., Sakuma, J., and Akimoto, Y. Max-min off-policy actor-critic method focusing on worst-case robustness to model misspecification. Ad- vances in Neural Information Processing Systems, 35: 6967-6981, 2022.
Tassa, Y. Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq. A., al. Deepmind control suite. arXiv preprint arXiv: 1801.00600 2018.
Tessler, C., Efroni, Y., and Mannor, S. Action robust rein- forcement learning and applications in continuous control. In International Conference on Machine Learning, pp- 6224. PMLR, 2019.
Tishby, N. and Polani, D. Information theory of decisions and actions. In Perception- -cction cycle: Models, archi- tectures, and hardware, pp. 601-636. Springer, 2010.

---

Maximum Total Correlation Reinforcement Learning
A. Proofs
A.1. Derivation Details of the Lower Bound
In this section, we provide full details about how to derive the lower bound (Eq. 2) from the total correlation definition. We start from the definition of the total correlation and derive lower bound using variational approximation g(<1:r 01:7-1) of the trajectory distribution.
p(Z1.r. (1:T-1)
log
IT p(2)) II p(a))
((21:1 a1:T-1)
log
IT=1 I[ p(a))
(8)
+ DKL (ox(pp(ea (p( (ir-)r (1
q(21,T, a1,7, II= p(z)) I p(a,)
2 Ep(arar
log
We parameterize the variational distribution q(21:1 01:r 11) autoregressively:
T-1
d a1:r-1 = p(21))((a1))))) (x)
II
In (n w.. a1:),
(9)
1=1
where 1 a1:) is history-based dynamics model, |(4+41/2::41 01:e) a history--basse action model. We plug Eq. 9 into Eq. 8, and then obtain
((2))((a11) 21
I 4n (2 (1.)) 1 [14411 1 a1:))
C(21:013... ;:)-1;z))) 2 E E
log
p(=)) I p(as)
T 1=1 On (214112211 a1:)
log
T IT p(2++1)
4x (ax 11 a1a-1)
+ Ep(enr
log
(10)
11: p(ar)
(1e a1d) p(2)41)
-B
log
21 (1--1)
ax 21:t; p(ar)
+ -
log
The marginal distributions p(=(-1) and p(a,) are unknown. However, the conditional distributions fo f0(241188441 and (a(alls) (st) are known and can be substituted while maintaining a lower bound:
((a.:ar...;aa-1133 2
Σ log On (
Σ log Ix (a a1s--))
E Bp(trrair-))
+ E Ep(es.eeen
f0(2441|5441)
TT(a)(st)
T-1
(11)
E Ep(sa))) DEL (fol (fo(e441ls44)) p(24t1))
Σ E Bp(n)) DKL (To(acls))) ll p(aa)))
(n(241l21a, Σ log f0(24418441)
log (x(a (1=-1) TT(ay(s))
2 E E
+ Epter
where the inequality in the last line holds because of the non-negativity of the KL divergence. Here we obtain the lower bound in Eq. Notably, the first term and the second term in the last line Eq. 11 both are lower bounds on the total correlation.
12

---

Maximum Total Correlation Reinforcement Learning
A.2. Connections to 7 (81127 F1:T)
RPC (Eysenbach et al., 2021) aims minimize the following upper bound of the mutual information between the state sequence and the latent state sequence,
((LTT(/1:)) p(21:T)
log f(
[($1:1 (1:)) (il)
log
< E (p(ee
(12)
(
In contrast to our bound, this bound does not use the history for the dynamics model, and it does not explicitly account for action consistency. Furthermore, we argue that the lower bound (Eq. 12) does not always hold as it was derived by replacing p(zır1/81:1 ) by IT==1 p( p(2++18*++1) (Eysenbach et al., 2021, Appendix C1). These distributions are general not the same because information about future state observations can decrease uncertainty about the current latent state, and therefore
p(z++1/21 $1.)) * p(2)41
We will now show that the latter replacement may invalidate the upper- bouud- by analyzing the gap.
[ (82+1)
E log
p(21:))
-
E
log
at)
= Ep(au.))
DKL (p(zurlssu (
20
T-1
at)
DKL.
The second term, may in general be smaller than the first term, for example, when the variational distribution perfectly matches the encoder, and, thus, the second term does not upper-bound the mutual information I (S1.T, 21:T).
We can, however, derive RPC based on total correlation perspective by using the variational distribution
('(211,1TT-11 ((21)p(a1) II
on (244 [e,, (()p()))1)
(13)
instead of Eg 9.
A.3. Updating Q function
Following the standard recursive Bellman equation, the Q function with parameters can be optimized by minimizing the loss
L (v) = EDDJ.. [Qu(se: az) - u(se, az))²
(14)
where the target is given by
y( at) = (s;, ar, S1+1) + + d) Q2 (3141, (441) log(
(15)
with discounted factor y and termination flag and next action ax+1 sampled from the current policy. We employ the independent target Q function to computer the target and stop the gradient through the target Q function.
A.4. Total Correlation for Infinite Sequences
The standard definition of total correlation is only useful for finite sequences, as it would typically not converge in the limit of infinite sequences. We will now show that our practical implementation which is based on an infinite-horiza MDP, maximizes a natural extension of the total correlation to infinite sequences.
13

---

Maximum Total Correlation Reinforcement Learning
We start by introducing the per-step contribution of step i the lotal correlation,
(( = arg log
Rt
p(2))
and note that for any finite sequence, its total correlation corresponds to the sum of per-step-contribtionns
p(r1, 22, ,2,,.., Fnn
02111 = Ez, 22...
log
2n
I[
Σ C(m):...;;...)
Intuitively, C, corresponds the additional amount of information (measured in nats) that (compared to per-step encodings) when encoding sequence of length instead of sequence of length / - 1,
C,
C(
Using the per-step contribution of the total correlation C,, we define the discounted total correlation C (211..... (oo) of an infinite sequence (I1...., 2oo) as
(x)).... "C.(
(16)
Hence, the discounted total correlation corresponds to the total correlation, when geometrically discounting future per-step contributions.
Using this definition, it can be shown analogously to the derivations in Appendix A.1, that maximizing our discounted objective 5 maximizes lower bound of the discounted total correlation C* (313....5
14

---

Maximum Total Correlation Reinforcement Learning
B. Experimental Details
B.1. Task Specification
We test our algorithms on MuJoCo continuous control tasks from the Deepmind Control, which provides a standardized set of benchmarks for reinforcement learning agents. For each task, the episode length is set to 1000 steps, and the action vector is bounded into [-1, We refer to (Tassa al., 2018) for more descriptions of tasks.
B.2. Implementation Details
SAC codebase. We implement our algorithm on top of the common PyTorch implementation of the SAC algorithm (Yarats al., 2021). We used the default hyperparameters from that implementation unless specified otherwise. Detailed descriptions of the SAC implementation are available in (Yarats et al., 2021).
Encoder. The encoder fe(= is parametrized as a 3-layer neural network with FCN (units=256) - FCN (units=256) + FCN (units=60) architecture and ReLU hidden activations. Its output is divided into the mean and the standard deviation of a diagonal Gaussian distribution.
Prediction models. Our prediction models 9n (ft4 E (1:)) and qn (aelaia,, a1=-1) 1) are parameterized by an LSTM module followed by a 3-layer neural network. The LSTM module is implemented using the common nn.LSTM class provided by PyTorch. The hidden dimension is set 256, the output dimension is set to 30, and the number of recurrent layers is set to for the LSTM module. The 3-layer neural network has the same architecture and activation function the encoder. The output of the dynamic model is normalized and then divided into the mean and the standard deviation of a diagonal Gaussian distribution.
Dual multipliers. We treat the hyperparameter a as a dual multiplier and optimize it via dual gradient ascent. We initialize the value of a to 10-6 and parametrize it as log a to ensure that it remains positive during optimization. For optimizing the entropy coefficient, we take the contribution from a into account and directly optimize 3 a.
Bound optimization. We refer to the first term and the second term in the last line of Eq. 11 the state bound C. and the action bound Cu. respectively. In practice, we implement our lower bound Č in Eq. 2 as a weighted combination of the state and action bounds, (1 - m)C, + mCa with an additional coefficient m E [o. 1]. Since the state bound and the action bound both are lower bounds on total correlation (see Eq. 11), their weighted combination is still lower bound on total correlation.
B.3. Other Hyperparameters
We initialize the replay buffer with 5000 samples from the initial policy and train all agents for million steps. We evaluate the agent every 20000 steps. All learnable parameters are updated using the Adam optimizer with learning rate of 10-4, We determine information constraints Ip and history length by performing hyperparameter tuning. We provide an overview of our used hyperparameters in Table. 2. For other details, please refer to the provided code.
B.4. Extended Description of Baseline Implementations
SAC. We obtain the results for SAC by running the PyTorch implementations of SAC (Yarats et al., 2021). We use the same hyperparameters for SAC as our algorithm ensure fair comparison. We found that our obtained results for SAC are stronger than the results of SAC reported in previous work (Yarats & Kostrikov, 2020).
LZ-SAC. We use the official implementation provided by Saanum et al. (2023) to obtain the results for LZ-SAC, since the official implementation is based on the same codebase of SAC and the hyperparameters has been tuned to achieve good results on DMC tasks.
RPC. To obtain the results for RPC, we first use the original code provided by Eysenbach et al. (2021), which is built on top of the SAC implementation from TF-Agents. To achieve as good performance possible for RPC, we perform hyperparameter tuning to select the suitable information constraint for RPC. To ensure a fair comparison, we additionally implement RPC by ourselves, using the same codebase of SAC as MTC and L.Z.SAC. We use the same SAC hyperparameters for our implementation of RPC as our algorithm.
15

---

Maximum Total Correlation Reinforcement Learning
Table 2. Hyperparameters used in MTC.
Parameter
Value
Ip for Cheetah Run, Hopper and Walker Stand
-0.5
Ip for other tasks
-7.0
history length
8
Replay buffer capacity
1 000 000
Optimizer
Adam
Critic learning rate
10-4
Critic Q-function soft-update rate
0.01
Critic target update frequency
2
Actor learning rate
10-4
Actor update frequency
Actor log stddev bounds
[-10 2]
Temperature learning rate
10-4
Initial temperature
0.1
Initial steps
5000
Discount
0.99
Initial
10-6
learning rate Representation dimension
10-4
30
Number of training steps Coefficient m
106
10-6 6
Batch Size for Hopper and Acrobot Batch Size for other tasks
512
256
B.5. Robustness to Observation Noise
In all experiments of robustness and trajectory compression, for each agent, we evaluated the performance of policies saved after finishing the training for 1M steps. Gaussian noise is regarded as a strong state distractor for reinforcement learning algorithms prior work (Bai et al., 2021). We add the Gaussian noise to observations and the learned policies select the actions based on the noisy observations.
B.6. Robustness to Action Noise
Noise added to actions can viewed as a type environment perturbation. this experiment, we first use the saved policies to select the action based on the current state, and add the Gaussian noise the chosen action. We then clip the action into [-1, before passing the action signal the task.
B.7. Robustness Dynamics
We modify the mass of the robot body to the robustness of learned policies. In our experiment setup, we get the body of the robot via the env. www.hhyies.model..oo wwdee.oody__eeeeee.e..o body_mass attribute provided by the environment. Since the body mass varies across different tasks, change the mass by scaling it, rather than increasing decreasing constant. We then evaluate the performance of the learned policies on the environment with the changed mass.
B.8. Trajectory Consistency
In experiment, measure the compressibility of trajectories using the bzip2 algorithm, which is easily available by installing the common bz2 python package. For each seed, we collect 30 trajectories using learned policies. Since the collected trajectories have the same number of data points and these data points have the same numerical precision, the uncompressed trajectories collected by the different algorithms have the same file size. We compress individual trajectories by calling the bz2. comprss)))) method provided by the bz2 package. Smaller file sizes of compressed trajectories mean that trajectories can be better compressed.
16

---

Maximum Total Correlation Reinforcement Learning
B.9. Image-based DMC Tasks
(a) Hall-in-cup Catch
(b) Caripole Swingup
(c) Finger Spin
(d) Reacher Easy
(c) Walker Walk
(f) Cheetah Run
Figure We evaluate our method on eight image-base DMC tasks.
We evaluate the performance of our method on the commonly used PlaNet benchmark, which consists of of complex image-based continuous control tasks. Specifially, consider six tasks:Ball-in-cup Catch, Cartpole Swingup. Finger Spin, Reacher Easy, Walker Walk, and Cheetah Run (see Figure 6). The flexibility of MTC allows us to apply it to image-based control settings with minimal modifications. We employ the convolutional encoder architecture from SAC-AE (Yarats et al., 2021) for encoding raw images into representations. Following common practice, we perform data augmentation by randomly shifting the image by [-4, 4], before we feed images into the encoder. We obtain an individual observation by stacking 3 consecutive frames, where each frame is RGB rendering image with size 84 84 from the Oth camera. An overview of our used hyperparameters for maaged-based tasks is shown in Table. 3. We refer code for more implementation details.
Table 3. Hyperparameters used in Image-based tasks.
Parameter
Value
Ip for Finger and Ball-In-Cup
-10.0
1p for other tasks
-0.1
history length for Finger and Ball-In-Cup
history length for other tasks
Replay buffer capacity
00 000
Optimizer
Adam
Critic learning rate
10
Critic Q-function soft-update rate
0.01
Critic target update frequency
Actor learning rate
10-4
Actor update frequency
Actor log stddev bounds
[-10
Temperature learning rate
10
Initial temperature
0.1
Initial steps Discount
5000
0.99
Initial a a learning rate Representation dimension Coefficient Batch Size
10 10-6 6
10-1
50
0.0001
256
B.10. Compute Resources
We performed every experiment on an Intel(R) Xeon(R) E5-2620 CPU with GeForce GTX 2080 Ti graphics card and used approximately one day for training.
17

---

Maximum Total Correlation Reinforcement Learning
C. Additional Results
C.1. Learning Curves
show the learning curves of MTC and other approaches on eight DMC tasks in Fig. 7. MTC achieves higher rewards and faster learning speed than baselines on the majority of tasks. Besides, MTC has a good learning stability on all tasks.
Acrobot Swingup
Hopper Stand
Finger Spin
Walker Walk
1000
1000
Episode Reward
800
800
800
1.50
100
400
400
50
200
200
0.6
0.8
1.0
0.2
0.4
0.6
1.0
0.6
0.0
Cheetah Run
Quadruped Walk
Walker Run
Walker Stand
800
Episode Reward
800
800
800
600
400
400
RPC
400
400
LZ-SAC
200
SAC
200
SPAC
0-:
Envinronment Step(x10")
Envinronment Step(x10*)
Envinronment Step(x10*)
Envinronment Step(x10*)
Figure 7. Leamning curves our method baselines on DMC tasks. The plot shows the average rewards over 20 seeds with shading of 90% confidence interval. MTC achieves better least comparable performance and sample efficiency than baselines.
C.2. Predictability of Policies
To directly assess the correlation of learned policies, additionally trained t-step-ahead predictive model for t [3, 5, 8, 10] time steps data sets that have been collected with the policy that has been learned with the different methods for a locomotion task and manipulation task. The model was trained to predict the action t steps ahead, when given the current action. We train the predictive model using maximum likelihood estimation. Table 4 and Table compare the prediction errors of the learned predictive models on the Finger Spin and Drawer-open-v2, respectively. For all tested time differences, the prediction error of MTC was the smallest among all baselines.
Table We compare prediction errors (negative log- likeliiheooo) of the learned predictive model for different time steps on the Finger Spin task. MTC achieves smaller prediction errors than other methods on all time steps.
Prediction Error (negative log- likehood)
t=5
t=10
MTC
0.22
0.32
0.48
0.67
RPC
0.34
0.58
0.72
0.76
LZ-SAC
0.96
1.31
1.48
1.53
SAC
0.78
1.13
1.47
1.57
C.3. Metaworld Tasks
We compare MTC to strong baselines, RPC and SAC, on 5 manipulation tasks from Metaworld in Fig Overall, MTC achieves higher mean rewards than RPC and SAC on all manipulation tasks.
18

---

Maximum Total Correlation Reinforcement Learning
Table We compare the prediction errors (negative log-likelihodd) of the learned predictive model for 4 different time steps the Drawer-open-v2 from Metaworld, MTC achieves smaller prediction emrors than other methods on all time steps.
Prediction Error (negative log-likchood)
t=3
(=5
1=8
t=
MTC
.4.84
-4.40
-4.05
-3.35
RPC
-2.62
-3.05
0.14
0.17
SAC
-2.37
-2.07
-2,7
-1.7
Handie-puli pull-pide-li
D Drawer-open-v
Plate dlide back-v2
Peg-insert-siae
Swoop-v2
1.5
1.0
3.55
9.20
3.25
02
22
3.50
0.0
0.0
Environment Stappx105)
Envilromment Stapoci0f)
Environment Stap0x10F)
Environment Stap(x10f)
Environment Steptx109)
Figure 8. Performance of our method and baselines on 5 manipulation tasks from Metaworld. The plot shows the average success rate and 90% confidence interval over 10 seeds. MTC achieves comparable performance than basclines.
C.4. Image-based DMC Tasks
Table. 6 shows the performance of our approach and baselines on six image-based DMC tasks. Overall, MTC achieves higher average rewards than baselines on five of six tasks. On the Walker Walk task, for instance, MTC achieves a reward of 939, higher than 917 achieved by RPC and 897 achieved by CURL.
Table Scores (means and standard error over seeds) achieved by our method and baselines on six DMC tasks at 500K environment steps. Each includes 10 evaluation episodes. MTC achieves higher at least comparable average reward than all baselines.
Scores
MTC
RPC
CURL
SAC-AE
SAC
Cartpole Swingup
868
859 + 8
837-115
748147
4361194
Ball-in-cup Catch
956 3
964 3
957 6
831 25
3551:77
Finger Spin
979 2
695 43
854 48
8394 68
5301:24
Walker Walk
939 6
917 15
897 26
8364 24
97+62
Reacher Easy
968 6
938 22
891 + 30
678 61
191140
Cheetah Run
585 = 25
572 + 15
492 22
476 22
250+22
C.5. Ablations on History-based Policies
While our total correlation objective results in reward function that depends on the past in order to encourage the agent to perform actions that are consistent with its previous actions, we did not choose history-based policy our experiments for simpler and fairer comparison. We ran additional experiments to test the effect of history- -bsed observations for standard SAC and our method on the Finger Spin task. The results are consistent with non- con-history-baat policies (see Table. 7), indicating that the improvements are caused by our total correlation objective, whereas non-markovianity observations not sufficient.
Table Performance comparison between MTC and SAC and their ablations, whose policies use history-hased observations. The table shows the mean and 90% confidence interval over 10 seeds. For each run, collected episode rewards 500K environment steps and computed the mean over 10 evaluation episodes. MTC achieves consistent performance with its history-basee ablation, MTC- CTC--istory.
Finger Spin
MTC
MTC-history
SAC
SAC-
Score
985 +
986 =
955 + 18
951 #23
19

---

Maximum Total Correlation Reinforcement Learning
C.6. Ablations Hyperparameters
We evaluate the performance of MTC for three different constraints 1p. The robustness to mass changes and observation and action noise, and the compression of behaviors improved while increasing I..
Trajectory Compression
Robustness to Obs Notse
Robustness to Action Nobie
Robustress to Mass
1020
1.0
507
6:60-
Notmalled Byt85
0.
Roward
592
Epicado Roward
420
Coloode Reward
2440
820
0.6-
-000
600
4- 1.0
592
730
660-
6-03.
Constraint 6
Obs Noiso Strangth
Action Noise Strongth
Mass Noise Strength
Figure 9. We test the performance of MTC with different constraints T.. The robustness changes and observation noise, and the compression of behaviors are improved while increasing Ip.
C.7. Robustness Comparison on A Single Task
A constant increase in noise could result in moderate effects relative to the zero-noise level, larger effects when applied on top of some existing noise (due to compounding effects), and in negligible effects when applied to very large noise level where the policy is not able to perform reasonably anyway. Hence, it is difficult to compare the robustness to noise when the different policies already start at different reward levels. However, we noticed that MTC, RPC, and SAC perform very similarly on Walker-Staie -taner--tanddaa and investigated the robustness while focusing on this single environment. As shown in Fig. 10, the robustness of MTC increases with respect to observation noise, action noise well mass change.
Robustness Obs Noise
Robustness Action Noise
Robustness
1000
1000
950
950
Eplsode Reward
900
800
850
850
MTC
700
800
RPC
750
800
LZ-SAC
SAC
500
700
750
0.2
1.0
Noise Strength
Noise Strength
Mass Scale
Figure Robustness observation noise (left), action noise (middle)) mass changes (right) on Walker Stand. The plot shows average reward over 20 seeds, with error bars representing 90% confidence interval. MTC achieves higher rewards than baselines in the presence strong ohservation, action, and mass perturbations.
C.8. Robustness on Metaworld Tasks
We also evaluate the robustness to observation noise, action noise, and mass changes in eight manipulation tasks from Metaworld. Overall, MTC achieves better performance when actions and dynamics are perturbed, while being comparable baselines the presence of observation noise.
C.9. Robustness Spurious Correlation
We perform experiments to investigate MTC is robust to spurious correlations. In the experiment, we added additional state dimensions that are not controllable by the actor, but instead follow a fixed Gaussian transition model. These distractors
20

---

Maximum Total Correlation Reinforcement Learning
Table Scores (mean and 90% confidence interval for 20 runs) achieved by method and haselines at eight manipulation tasks from Metaworld with action noise, mass changes, and observation noise. Overall, MTC obtained higher scores in the presence of action and dynamics perturbations, while being comparable baselines on with observation noise.
500K step scores
MTC(OUrr)
RPC
SAC
Handle-pul-side--2
0.98 0.01
0.72 + 0.05
0.62- 0.12
Drawer-open-v2
0.60 0.13
0.21 = 0.13
0.19
Platesslide-back-22
0.71 0.06
0.67 + 0.08
0.58
Beg-insert-side-ie
0.01 0.01
0.00 #0.00
0.00=0.000
Action noise with strength 2.0
Sweep-v2
0.01 0.01
0.00 +0.00
0.00+00.00
Button-press-aall--2
0.24 0.04
0.26 = 0.05
0.31
Door-lock-v2
0.84 0.03
0.69 = 0.06
0.71
Push-back-v2
0.10+ 0.04
0.08 = 0.03
0.01 + 0.01
Handle-pull-side-V
0.98 0.02
0.71 = 0.07
0.65 0.12
Drawer-open-vy
0.62 0.13
0.22 = 0.16
0.21 0.15
Plate-slide-back-V2
0.98 0.01
0.87 + 0.08
0.93+ 0.03
Reg-insert-side-ve
0.07+ 0.06
0.00 =0.00
0.00=0.000
Mass changes with scale 1.75
Sweep-v2
0.29 0.10
0.18 + 0.09
0.18 0.12
Button-press-war
0.51 0.08
0.59 = 0.12
0.62 0.10
Door-lock-v2
0.96 0.02
0.84 = 0.05
0.86+00055
Push-back-v2
0.18+ 0.07
0.12 + 0.04
0.00 + 0.00
Handle-pull-side-
0.95+ 0.04
0.74 = 0.07
0.69 0.13
Drawer-open-v2
0.45+ 0.15
0.18 = 0.11
0.13+ 0.08
Plate-slide-back--22
0.29 0.09
0.29 + 0.08
0.46.L: 0.07
Beg-insert-side-ve
0.00 0.00
0.00 = 0.00
0.00 0.00
Obs noise with strength 0.05
Sweep-v2
0.02. 0.01
0.02 = 0.02
0.01 0.01
Button cutton-press--we
walll-2
0.27 0.04
0.24 = 0.08
0.16 0.07
Door-lock-v2
0.76 0.07
0.74 + 0.04
0.73+ 0.04
Push-back-v2
0.00- 0.00
0.00 = 0.00
0.00 0.00
not correlated with the remaining states, the actions nor the reward that the agent receives, although by coincidence, it might appear that such correlations exists, resulting in spurious correlations. We evaluate our method and basclines on the Walker Stand task with such observation distractors. Fig. 11 shows the performance of our method, RPC and SAC. The plot shows the mean over 10 seeds, with a 90% confidence interval. MTC significantly outperforms RPC and SAC in rewards, suggesting that MTC improves robustness to spurious correlations.
C.10. Visualizations of Trajectories
We visualize the trajectories produced by our method and baselines on the Finger Spin task. For the Finger Spin task, dimensions of the action and state space are and 9, respectively. Fig and Fig visualize the action and state trajectories produced by our method and basclines. We observed that MTC produces more consistent and periodic patterns in trajectories.
21

---

Maximum Total Correlation Reinforcement Learning
Walker Stand
1000
800
Episode
400
MTC
RPC
SAC
0.0
0.1
0.2
0.3
0.5
Envinronment Step(x109)
Figure 11. Robustness to spurious correlations Walker Stand. MTC achieves higher rewards than RPC and SAC, when states expanded with unrelated Gaussian noises.
Actuator
Actuator
Action
Action
25
50
75
100
325
150
1.75
200
50
100
150
175
200
Action
N
25
50
75
100
125
150
175
200
25
50
75
100
125
150
175
200
WWW
Action
25
50
75
100
125
150
175
200
50
75
100
125
150
175
200
TimeStep
TimeStep
MTC
RPC
LZ-SAC
SAC
Figure 12. Visualizations of action sequences generated by our method and baselines on the Finger Spin task. MTC produces more consistent and periodic behavior than baselines.
22

---

Maximum Total Correlation Reinforcement Learning
MTC
L2-SAC
SAC
WWW WWW Mm
State
0.0
State
100
0.2
16
Sta62
-0.3
102
Staee
W
50
100
150
200
350
305
150
300
State 5
50
100
150
200
50
1.00
3.50
200
50
100
150
200
50
300
150
200
20
-20
100
150
202
-40
50
130
200
150
103
150
200
2.5
State
2.5
2.5
350
11meStop
TimeStop
Figure 13. Visualizations of state sequences generated by our method and baselines on the Finger Spin task. State sequences of our method show more repeating and periodic patterns.
23


Reviews:
[{"review_id": "V38UTwH06R", "reviewer": "Reviewer_ts9N", "summary": "The paper proposes an extension to standard reinforcement learning by introducing a regularization objective that maximizes the total correlation across latent state representations and actions in an agent’s trajectory. It derives a variational lower bound on this total correlation, which is incorporated into the soft actor-critic framework to jointly optimize the policy, state encoder, and prediction models. Experimental evaluations on simulated robotic control tasks show that this approach yields more coherent, periodic, and compressible trajectories, leading to improved performance and enhanced robustness against observation noise, action perturbations, and dynamics changes. Overall, the paper argues that embedding total correlation as an inductive bias can foster simpler and more robust behaviors in reinforcement learning agents.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing valuable comments and acknowledging the novelty and empirical performance of our approach. \n\n> Method uses a lower bound for total correlation that is always negative, making it unclear how accurately it reflects the true value.\n\nAs stated under limitations, our lower bound is not meant for approximating the total correlation, but still effective for optimization, as demonstrated in our experiments that show simpler, more compressible, and more robust behavior. To further investigate the ability of our method to maximize the total correlation, we conducted additional experiments on the Finger spin task for this rebuttal. Namely, we collected data sets using the policies that have been learned with the different methods, and trained a model to predict the action t steps ahead, based on the current action. As indicated in the following table, our lower bound object significantly increases the predictability, further suggesting that MTC is effective in increasing the consistency, predictability and thereby total correlation of the emerging behavior.\n\n| test log-likelihood | t=3  | t=5 | t=8 | t=10 | \n|:--------------------------:|:---------------:|:---------------:|:---------------:|:-----------:|\n|       MTC        | **0.22** | **0.32** | **0.48**    |   **0.67**  |  \n|       RPC        | 0.34  |  0.58  |  0.72   |  0.76   |  \n|       LZ-SAC     |  0.96     |  1.31  |  1.48   | 1.53    |  \n|       SAC        |  0.78   | 1.13   |   1.47  |   1.57  |\n\n\n> Not fully clear if the same benefits would apply in real-world situations. \n\nOur experiments show that policies that are trained with our regularizer exhibit a simpler behavior that is more robust to mismatches in the system dynamics. We argue that these properties are highly desirable when attempting to transfer these policies to a real robot. However, we unfortunately were not able to perform experiments on a real robot. \n\n> Author is encouraged to apply their approach to a wider variety of environments.\n\nWe now additionally evaluated our approach on 5 more tasks from Metaworld. Also on these non-periodic\nmanipulation tasks, MTC outperforms SAC and RPC in terms of average success rate, demonstrating the benefits of maximizing the total correlation\non solving manipulation tasks. All in all, our experiments are quite diverse, span periodic and nonperiodic, vision-based and non-vision-based tasks, closely follow the testbeds of the most related works, and show strong performance in all of these settings.\n\n| Success rate at 500K steps (mean and 90% CI over 10 seeds) | Handle-pull-side-v2 | Drawer-open-v2  | Plate-slide-back-v2 | Peg-insert-side-v2 |    Sweep-v2     |\n|:----------------------------------------------------------:|:-------------------:|:---------------:|:-------------------:|:------------------:|:---------------:|\n|                            MTC                             |   **0.98 ± 0.02**   | **0.68 ± 0.18** |   **0.99 ± 0.02**   |  **0.06 ± 0.05**   | **0.28 ± 0.14** |\n|                            RPC                             |     0.79 ± 0.10     |   0.23 ± 0.21   |   **0.98 ± 0.03**   |    0.00 ± 0.00     | **0.23 ± 0.14** |\n|                            SAC                             |     0.76 ± 0.19     |   0.22 ± 0.21   |   **0.93 ± 0.07**   |    0.00 ± 0.00     | **0.19 ± 0.17** |\n\n> However, optimal policies in DMC environments exhibit periodic behavior regardless of additional priors. Unclear how total correlation affects this.\n\nWhile DMC tasks are characterized by periodic motions, our results in Fig.1, Fig.3, Fig.9 and Fig.10 clearly show that \nmaximizing the total correlation improves the consistency and periodicity of behaviors compared to leading baselines. Furthermore, we are primarily interested in learning ***simple*** behavior, which does not necessarily imply periodicity, for example, for manipulation tasks.\n\n\n> [Comparison to literature such as TD-MPC and BYOL]\n\nWe now provide the following discussion of these two approaches in our Related Work Section: Our approach is also related to \nprevious approaches that extract temporally consistent representations from observations by learning latent dynamics models, such as TD-MPC and BYOL. Different from these approaches only considering temporal consistency in representations of states, our total correlation objective aims to maximize the consistency \namong the whole trajectories of state representations and actions. As shown in our experiments (Fig.5), additionally enforcing the consistency within action sequences by learning the action prediction \nmodel improves the performance in the presence of environmental perturbations.", "claims_and_evidence": "Most of the claims are supported by the experiments in simulation, which show improvements in consistency, robustness, and performance. However, one issue is that the method uses a lower bound for total correlation that is always negative, making it unclear how accurately it reflects the true value. Additionally, since the evidence comes only from simulated environments, it's not fully clear if the same benefits would apply in real-world situations.", "methods_and_evaluation_criteria": "Most claims are based on DMControl results. Results should be evaluated across a wider range of benchmarks.", "theoretical_claims": "Did not check.", "experimental_designs_or_analyses": "Experiments are insufficient, as claims are entirely supported by DMControl experiments. Author is encouraged to apply their approach to a wider variety of environments.", "relation_to_broader_scientific_literature": "Method attempts to imbue priors in existing RL algorithms to encourage temporal consistency and periodicity of movements. However, optimal policies in DMC environments exhibit periodic behavior regardless of additional priors. Unclear how total correlation affects this.", "essential_references_not_discussed": "Algorithms such as TD-MPC use BYOL losses to learn latent representations of the environment. Comparison to this literature would be relevant.", "other_strengths_and_weaknesses": "Strengths:\n- Introduces a creative method that combines total correlation with reinforcement learning to improve policy robustness and consistency.\n\nWeaknesses:\n- Results limited to DMC.", "other_comments_or_suggestions": "None for now.", "questions_for_authors": "None for now.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qcuVyD7XPe", "reviewer": "Reviewer_moC5", "summary": "An RL framework is presented that encourages total correlation within trajectories, leading to smoother, more periodic looking, and more compressible trajectories. This leads to robustness to observation noise, action noise, and robustness to changes in dynamics. The proposed algorithm is employed for DeepMind environments where it surpasses related work in expected reward, robustness, and compression.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for carefully reviewing our submission, and appreciating our contributions, and thorough experimentations.\n\n> is MTC better because of total correlation, or because of it being non-Markovian?\n\nWhile our total correlation objective results in a reward function that depends on the past in order to encourage the agent to perform actions that are consistent with its previous actions, we did not choose a history-based policy in our experiments for a simpler and fairer comparison. We ran additional experiments to test the effect of history-based observations for standard SAC and our method on the Finger Spin task. The results are consistent with non-history-based policies, indicating that the improvements are caused by our total correlation objective, whereas non-markovianity is not sufficient.\n\n| Score (mean and 90% CI over 10 seeds) |     MTC     | MTC-History |    SAC     | SAC-History |\n|:-------------------------------------:|:-----------:|:-----------:|:----------:|:-----------:|\n|              Performance              | **985 ± 2** | **986 ± 3** |955 ± 18 |  951 ± 23   |\n|       Trajectores Size in Bytes       | **2993 ± 292** | **3031 ± 259** | 5156 ± 526 | 5259 ± 442  |\n\n> Robustness claims would be more solid if the other algorithms did not start worse off\n\nMaintaining high performance in the face of noise or dynamics mismatch is significantly harder than maintaining low performance. Indeed, the performance of the worst possible policy could not degrade when subjecting it to observation noise. Instead, we argue that it is more relevant to consider the absolute performance in such test settings, rather than considering the changes relative to the training performance.\n\n> The main hypothesis, a policy that exhibits more periodic behavior is less brittle, needs better substantiation.\n\nWe want to clarify that our primary interest is in learning ***simple*** behavior, which may, but does not need to manifest in periodic behavior. We challenged the hypothesis that simpler behavior tends to be less brittle on various periodic and non-periodic tasks, but did not encounter detrimental effects. We did not intend to claim that periodicity always increases robustness, and will carefully revise our submission to improve clarity. We will also revise the introduction to better motivate our hypothesis:\n\nDuring training, the reinforcement learning agent, for example a robot, typically observes many slight variations in the state, for example due to sensor noise or unmodelled dynamic effects. In some cases, it can be crucial for the agent to drastically adapt its behavior to react to these variations, but sometimes they can be safely ignored. We aim to bias the agent towards ignoring such variations, whenever doing so does not seriously decrease the expected return, and thereby follow Newton's first rule of his third book of principia: \"We are to admit no more causes of natural things than such as are both true and sufficient to explain their appearances\". Intuitively, we expect a given behavior that performed well for previous variations, to also perform well for future variations. We introduce this inductive bias by means of the additional objective of maximizing the total correlation within the trajectory produced by the agent. This total correlation corresponds to the amount of information that we can save by using a joint encoding of all (latent) states and actions within trajectories, compared to compressing all time steps independently. By maximizing total correlation, the agent is encouraged to produce compressible and predictable trajectories, and thereby biased towards open-loop behavior such as clean periodic gaits, without preventing it from performing adaptations when necessary.\n\n> why there is the term $r(s_T, a_T)$ in equation (3)?\n\nThank you for spotting this mistake. The term belongs outside of the summation.\n\n> I am not so sure about the use of the word \"Coherence\" for trajectories that show periodic behaviour or more total correlation\n\nThank you for this feedback. We used coherence mostly interchangeably with consistency, but will only refer to consistency in the revision. \n\n> [why rounding is necessary]  \n\nFrom the point of lossless compression, the least significant bits are just as relevant as the most significant ones. When directly writing the floating point numbers of the simulator into a file, most of the bits in that file are used to encode digits that are beyond the precision of the control. When compressing such a file, the resulting file size will largely depend on how effectively those essentially random bits can be compressed, which significantly increases the variance of the resulting file size.\n\n> Is periodicity -> robustness?  \n\nWe do not believe that periodicity implies robustness and did not intend to make such a claim. We hypothesize that an inductive bias towards simpler behavior that avoids ***unnecessary*** adaptions can often improve robustness.", "claims_and_evidence": "Claims:\n- MTC leads to robustness\n- MTC leads to less \"unnecessary variation\", and more periodic trajectories\n- MTC leads to compressibility\n\nGood Evidence:\n- Robustness evidence presented for one environment\n\t- Observation noise is added and MTC seems clearly more robust\n\t- Action noise is added and MTC seems clearly more robust\n\t- Dynamics are modified and MTC seems clearly more robust\n- MTC generally better than other algorithms in DMC tasks\n\nBut Questions Arise\n- is MTC better because of total correlation, or because of it being non-Markovian? While it is true that total correlation would induce a more periodic behaviour, it is harder for a Markovian policy to exhibit periodicity. We see this trend in this paper as well: SAC is Markovian, LZ-SAC uses a history of some states, RPC uses the complete history but only of states, and MTC uses the complete history of state-action pairs. And the the trend in performance is similar from worst to best: SAC, LZ-SAC, RPC, MTC. In other words my question is, is a good policy (regardless of total correlation) just more periodic and it will be learnt if it is non-Markovian, or does total correlation itself has a significant role to play.\n- Robustness claims would be more solid if the other algorithms did not start worse off. A counter point can be made that since other algorithms start worse-off, addition of noise in actions/observations/dynamics causes degradation relative to how good a policy was originally.\n- The main hypothesis: that a policy that exhibits more periodic behaviour is less brittle, I think needs better substantiation. It might be very true that some domains like such policies where is there is repetitive behaviour, but there might be others that don't . That does not take away anything from the core contribution of the paper, but the language I feel is a bit exaggerated if we don't have a strong theoretical proof that periodicity <-> robustness. (But I see that compressibility has been proposed as a good thing to have in other related works)", "methods_and_evaluation_criteria": "Yes. Popular benchmarks. Very thorough experimentation. Comparison to relevant work.", "theoretical_claims": "Not in depth. No issues from a surface level reading except one: I don't understand why there is the term $r(s_T, a_T)$ in equation (3). i.e. the reward of last state-action pair ?", "experimental_designs_or_analyses": "Did not run any code. The design description in paper and appendix seems sound.", "relation_to_broader_scientific_literature": "A good contribution where we desire policies that exhibit more periodic behaviour. Good contribution in continuous control domains. Possibly the next natural step after RPC etc that the work is compared to.", "essential_references_not_discussed": "none that I am aware of", "other_strengths_and_weaknesses": "Strengths:\n- Good, thorough experiments that empirical evidence of the claims in DMCS tasks.", "other_comments_or_suggestions": "I am not so sure about the use of the word \"Coherence\" for trajectories that show periodic behaviour or more total correlation. But maybe the authors are a better judge for that. But I highly suggest reviewing this term.\n\nI don't understand p.5 col 2 last paragraph where it is said that rounding is necessary otherwise it leads to high variance in the results. Since rounding ignores the least significant bits, shouldn't it be the case that variance is not much affected by rounding?", "questions_for_authors": "- Is periodicity -> robustness, or that periodicity in these domains = good policy and therefore high reward and more robustness?\n- How much of this is due to non-Markovian nature, and how much due to explicit introduction of total correlation term?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4srFUcAVvE", "reviewer": "Reviewer_8GSd", "summary": "This paper proposes a method that learns compressible policies via a lower bound on the total correlation over a trajectory.\nThis is motivated by aiming to increase robustness by learning more simplistic policies.\nIn practice, the method trains a recurrent latent state and action predictor and adds the predictability of actions to the reward function.\nIn experiments, the authors show that their method leads to more robust and compressible policies.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for carefully reviewing our submission, and acknowledging the soundness of the main experiments, the good presentation and the empirical results that show more robust and compressible policies.\n\n> the claim of reducing spurious correlation should be investigated more directly or removed\n\nWe evaluated the effect of spurious correlation in Appendix C4. We did not add Gaussian noise to the observation, but added additional state dimensions that are not controllable by the actor, but instead follow a fixed Gaussian transition model. These distractors are not correlated with the remaining states, the actions nor the reward that the agent receives, although by coincidence, it might appear that such correlations exists, resulting in spurious correlations.\nOur results show that MTC achieves higher rewards than RPC and SAC on the Walker Stand task, suggesting that MTC improves robustness to spurious correlations.\n\n> Some experiments related to the evaluation of correlation could be done more directly  \n\nThank you for suggesting evaluating the predictability to more directly assess the total correlation. We trained a t-step-ahead predictive model for t=[3,5,8,10] time steps on data sets that have been collected with the policy that has been learned with the different methods for a locomotion task and a manipulation task. The model was trained to predict the action t steps ahead, when given the current action. For all tested time differences, the prediction error of MTC was the smallest among all baselines.\n\n| test log-likelihood (Finger-spin) | t=3  | t=5 | t=8 | t=10 | \n|:--------------------------:|:---------------:|:---------------:|:---------------:|:-----------:|\n|       MTC        | **0.22** | **0.32** | **0.48**    |   **0.67**  |  \n|       RPC        | 0.34  |  0.58  |  0.72   |  0.76   |  \n|       LZ-SAC     |  0.96     |  1.31  |  1.48   | 1.53    |  \n|       SAC        |  0.78   | 1.13   |   1.47  |   1.57  |\n\n\n| test log-likelihood (Drawer-open-v2) |    t=3    |    t=5    |    t=8    |   t=10    | \n|:--------------------------:|:---------:|:---------:|:---------:|:---------:|\n|       MTC        | **-4.84** | **-4.40** | **-4.05** | **-3.35** |  \n|       RPC        |   -2.62   |   -3.05   |   0.14    |   0.17    |  \n|       SAC        |   -2.37   |   -2.07   |   -2.7    |   -1.7    |\n\n> non-markovianness of the reward as well as the non-stationarity of the reward\n\nAlthough we clearly state that our augmented reward depends on the history-based decoder, we agree that the notation should reflect this dependency, and that the implication for the optimization should be discussed, and we will revise our submission accordingly. \nOur reward function depends on past states and actions to encourage the agent to act consistently with its previous actions. For our experiments, however, we did not provide the past states and actions to the policy, which limits its ability to perform optimally due to the non-Markovianity of its state-action-space. We did not provide the history to the policy solely to improve the simplicity of our method and the fairness of the comparisons. However, we performed an additional experiment on the Finger Spin task for the rebuttal to evaluate our approach in a Markovian setting (by using a recurrent policy), which did not result in significant differences.\n\n| Score (mean and 90% CI over 10 seeds) |     MTC     | MTC-History |\n|:-------------------------------------:|:-----------:|:-----------:|\n|              Performance              | **985 ± 2** | **986 ± 3** |\n|       Trajectores Size in Bytes       | **2993 ± 292** | **3031 ± 259** |\n\nWe also agree that the non-stationarity of the reward should be discussed and we will revise the submission accordingly. Non-stationary rewards are quite common nowadays and are used effectively in different subfields, such as imitation learning, representation learning, and---most-relevant for us---by related works such as RPC and LZ-SAC. While establishing convergence guarantees for MTC (and the related works) would be desirable, we unfortunately have to leave this for future work.", "claims_and_evidence": "* The main claim of the paper is that policies with a compressibility are more robust to disturbances, and that their method creates more compressible policies.\n\nThe improved robustness is well supported by experiments with distribution shifts.\n\n * They also claim that it improves robustness to spurious correlations.\n\nThis claim seems reasonable, but experiments do not directly evaluate it. The authors write that Gaussian noise is a strong distractor, but that's qualitatively different from an actual spurious correlation/ an actual distrator.\n\n\n* The correlation or predictability of their learned policy is evaluated by compressing trajectories with an off-the-shelf lossless compression method and by visual inspection. \n\nA more interpretable way to measure it might be to train predictors for each and evaluate their accuracies, or to actually evaluate the lower bound of total correlation derived by the authors", "methods_and_evaluation_criteria": "As stated above, the main experiments and evaluation are reasonable, but especially the claim of reducing spurious correlation should be investigated more directly or removed.\n\nOverall it's reasonable.\n\nShowing 90% CIs instead of the usual 95% is surprising, but it is clearly stated in the figure captions so that's fine.", "theoretical_claims": "I checked the derivation of the proof for the lower bound and did not find any mistakes.\n\nMy main concern is that the reward formulation in (4) depends on all previous actions and states in an episode, as well as on the learned predictors.\n\nThis makes the reward formulation non-markovian and dependent on predictor parameters $\\phi$ and $\\theta$, and the notation as $r^*(s_t,a_t,s_{t+1})$ is questionable, it should be $r^*(s_{1:t},a_{1:t},\\theta,\\phi)$.\n\nThe paper claims that this can be \"optimized straightforwardly with existing RL methods\".\n \nThis is in no way obvious, it is not clear why we can simply ignore this dependency.", "experimental_designs_or_analyses": "The  main experiments are sound.\n\nSome experiments related to the evaluation of correlation could be done more directly and the evaluation of spurious correlation only with observation noise is insufficient.", "relation_to_broader_scientific_literature": "The relation to prior method is discussed appropriately in the paper.", "essential_references_not_discussed": "I am not aware of missing references.", "other_strengths_and_weaknesses": "The presentation/writing is very good.", "other_comments_or_suggestions": "L432 (right): \"goal post\" should be \"goalpost", "questions_for_authors": "My main concern is the described non-markovianness of the reward as well as the non-stationarity of the rward due to the changing  predictor parameters.\n\nHow was this addressed in practice, or why is this not a problem? Maybe I am misunderstanding something.\n\nIf this can be addressed I am willing to raise my score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AxXRy5eGi9", "reviewer": "Reviewer_vAa9", "summary": "This paper proposes an algorithm called MTC, which is a SAC-style approach that utilizes total correlation regularization. The basic idea behind MTC is that reducing unnecessary variations in states and actions increases robustness; accordingly, the authors propose an algorithm that maximizes the total correlation—a generalized mutual information among all states and actions within an episode. For the total correlation regularization term, a variational lower bound is derived, and MTC employs this lower bound as the regularization term. Experimental results demonstrate that MTC exhibits higher performance compared to other SAC-based algorithms and is particularly beneficial for enhancing robustness.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for thoroughly reviewing our work and appreciating the performance and robustness of our method, and the persuasive evidence of our main claims.\n\n> benefit of total correlation maximization in non-periodic tasks\n\nOur main hypothesis that simple behavior that does not overfit to slight variations is less brittle is not restricted to periodic tasks, but also well-motivated for non-periodic tasks, such as manipulation tasks. For example, consider two different policies for grasping an object. The first policy is able to achieve high reward by always using essentially the same grasping motion and no adaptation. The second policy achieves similar reward, but relies more strongly on feedback and adapts even to slight variations. We expect the first behavior, which would have higher total correlation and better predictability but no periodicity, to be more robust to perturbations that have not been observed during training.\n\n> more non-periodic experiments\n\nWe conducted additional experiments on five non-periodic MetaWorld tasks. On all tasks we achieved similar or better success rate than the baselines, supporting our hypothesis that regularizing towards simpler behavior is also beneficial for non-periodic tasks. A table with the results can be found in our reply to reviewer ts9N.\n\n> Derivation of eq (11)\n\nOur derivations do not assume that $\\log(p(z\\_t))$ is equal to $\\log(p(z\\_t|s\\_t))$. However, due to the non-negativity of the expected KL $E\\_{p(s\\_t)} [ \\text{KL}(p(z\\_t|s\\_t) || p(z_t))]$, substituting $p(z\\_t|s\\_t)$ for $p(z\\_t)$ or $\\pi(a\\_t|s\\_t)$ for $p(a\\_t)$, does not invalidate our lower bound.\n\n> magnitude of noise\n\nThank you for this constructive feedback. We use stronger state and action noises now. The following tables compare the performance (mean and 90% confidence interval over 20 seeds) on all eight tasks. Overall, MTC obtains better performance in the presence of strong noise. \n\n| Action noise with scale 0.5 | Acrobot Swingup | Hopper Stand | Finger Spin | Walker Walk | Cheetah Run | Quadruped Walk | Walker Run | Walker Stand | \n|:---------------------------:|:---------------:|:------------:|:-----------:|:-----------:|:-----------:|:--------------:|:----------:|:------------:|\n|             MTC             |    **159 ± 19**     |   **711 ± 36**   |  **643 ± 14**   |   **921 ± 9**   |   **637 ± 8**   |    **928 ± 8**     |  **396 ± 12**  |   **975 ± 3**    |\n|             RPC             |    111 ± 22     |  300 ± 115   |  **609 ± 24**   |  832 ± 54   |  539 ± 46   |    825 ± 80    |  **388 ± 25**  |   967 ± 4    |\n|           LZ-SAC            |     27 ± 7      |    19 ± 3    |  346 ± 26   |  715 ± 47   |  278 ± 12   |    430 ± 84    |  295 ± 9   |   905 ± 20   |\n|             SAC             |    108 ± 19     |  485 ± 132   |  **627 ± 27**   |  857 ± 30   |  502 ± 24   |   686 ± 111    |  356 ± 18  |   970 ± 6    |\n\n| Obs noise with scale 0.3 | Acrobot Swingup | Hopper Stand | Finger Spin  | Walker Walk |   Cheetah Run   | Quadruped Walk |  Walker Run  | Walker Stand | \n|:------------------------:|:---------------:|:------------:|:------------:|:-----------:|:---------------:|:--------------:|:------------:|:------------:|\n|           MTC            |    **8 ± 2**    | **190 ± 70** | **138 ± 12** |  **851 ± 10**   |      48±4       |  **901 ± 23**  | **351 ± 18** | **975 ± 2**  |\n|           RPC            |     **6 ± 1**     |   44 ± 30    |    28 ± 8    |  769 ± 35   |     54 ± 9      |    825 ± 82    | **341 ± 22** |   961 ± 8    |\n|          LZ-SAC          |      **3 ± 5**      |   **218 ± 22**   |   97 ± 11    |  755 ± 34   |   **101 ± 5**   |    567± 113    |   **334 ± 15**   |   955 ± 13   |\n|           SAC            |      **7 ± 1**      |   102 ± 44   |   104 ± 13   |  775 ± 29   |     48 ± 5      |   672 ± 110    |   **332 ± 26**   |   945 ± 21   |\n\n> Inconsistency between Figure 2 and Figure 7\n\nThe inconsistency is caused by using different $I_p$, which we did not tune for each task in our robustness experiments. \n\n> depicting behavior in non-periodic environments\n\nAs we are not able to show a graph in our reply, we evaluated the action-predictability to demonstrate that our regularizer also improves simplicity for non-periodic tasks. The table can be found in our reply to Reviewer 8GSd.\n\n> Related approaches using Fourier transform [1,2]\n\nThank you for suggesting these methods, which are related to MTC and will be discussed in our revision. However, they are not closely related, since they do not consider the simplicity of behavior.\n\n> Normalized scores and learning curves.\n\nWe can not show the learning curves here, but will add them to the revision with original rewards. MTC shows good learning stability.\n\n> Figure 7\n\nWe will put Figure 7 in the main in our revision. \n\n> Figure 4\n\nNo, Fig. 4 does not show  robustness in non-periodic tasks. We focused on DMC for our robustness experiments to follow the setup of closely related work (RPC and LZ-SAC).", "claims_and_evidence": "The main claim of this paper is that reducing unnecessary variations in states and actions enhances robustness. This claim is supported by various experiments. In particular, the paper presents graphs showing that maximizing the proposed total correlation increases the consistency of the state and action trajectories of the learned policy, and it provides results demonstrating robust performance against action noise, state noise, and mass variations, thereby offering persuasive evidence for the main claim.", "methods_and_evaluation_criteria": "The total correlation maximization in MTC enables more efficient prediction of states and actions, and it is intuitively expected to work well for periodic tasks like those in DMC. However, in non-periodic tasks, there may be inherent limitations to the benefits of increasing total correlation. For instance, in Figure 4, while MTC shows superior performance compared to other baselines in the button-press-wall environment, the differences in other environments appear less significant. A more detailed explanation of the role and effects of total correlation maximization in non-periodic tasks would be appreciated.", "theoretical_claims": "In Appendix A.1’s derivation of eq (2), it appears that eq (11) may be incorrect. Specifically, at L589, the terms $p(z\\_{t+1})$ and $p(a\\_t)$ should be defined as $\\int\\_{s\\_{t+1}} p(z\\_{t+1} | s\\_{t+1} ) p(s\\_{t+1}) ds\\_{t+1}$ and $\\int\\_{s\\_{t}} p(a\\_{t} | s\\_{t} ) p(s\\_{t}) ds\\_{t}$, respectively. Therefore, it does not seem appropriate to simply substitute them with $p(z\\_{t+1} | s\\_{t+1} )$ and $p(a\\_{t} | s\\_{t} )$ without including the integrals.", "experimental_designs_or_analyses": "1. In Figure 2, the magnitude of the observation/action noise appears to be too small. With such minimal noise, there is not enough significant change to observe robustness, and the performance of all baselines seems to drop linearly. It might be more informative to increase the magnitude of the observation/action noise, as shown in Figure 7, to better demonstrate robustness.\n2. The robustness experiment results regarding mass scale changes seem inconsistent between Figure 2 and Figure 7. This appears to be due to differences in the hyperparameter $I\\_p$. If $I\\_p$ is set to -1.0, as in Figure 7, the results might show improved robustness.\n3. The experiments in non-periodic environments, such as those in metaworld, appear to have been conducted on too few environments. Expanding the evaluation to a broader range of environments could more convincingly demonstrate the effectiveness of MTC in non-periodic settings.\n4. It would be beneficial to include a graph depicting behavior in non-periodic environments. Such a graph could help verify whether unnecessary state and action noise is effectively reduced in these cases as well.", "relation_to_broader_scientific_literature": "In the existing literature, algorithm performance and robustness are improved by removing unnecessary noise from either state or action sequences. This study also follows that research direction; however, while previous works have considered only the state sequence or the action sequence, this study takes into account the entire state-action trajectory. To achieve this, total correlation is introduced, and a lower bound is derived and utilized for the practical algorithm.", "essential_references_not_discussed": "An approach with a similar motivation is the use of Fourier transform, which has been explored in previous studies [1, 2]. These methods also aim to remove unnecessary information from state and action sequences to facilitate faster learning, which is aligned with the motivation of MTC. Explaining the differences between these studies and MTC, and incorporating additional experiments, could further enhance the novelty of the paper.\n\n[1] A. Li & D. Pathak, “Functional regularization for reinforcement learning via learned fourier features,” NeurIPS 2021.\n[2] M. Ye, et al., “State sequences prediction via fourier transform for representation learning,” NeurIPS 2023.", "other_strengths_and_weaknesses": "There are no additional strengths or weaknesses to mention.", "other_comments_or_suggestions": "1. Figure 2 is reported only in normalized scores; it would be preferable to also report episode rewards so that the results are consistent with the other evaluations.\n2. Including the $I\\_p$-related experiments from Figure 7 in the main paper could be beneficial, as the graph demonstrating that a stricter trajectory consistency constraint increases robustness serves as compelling evidence for the paper’s main claim.", "questions_for_authors": "1. Could you also show the learning curve for Table 1? While the final performance is impressive, observing the stability and learning speed during training might provide more detailed insights into MTC’s performance.\n2. Have you conducted experiments on other metaworld tasks with non-periodic environments (e.g., drawer-close, reach, etc.)? Presenting evaluations across a wider range of environments could further demonstrate the method’s high performance in non-periodic settings.\n3. In Figure 4, does the right-hand graph include experimental results regarding robustness in non-periodic environments?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper adds a regularization to SAC to maximize the total correlation between action trajectories and latent representations. The objective is meant to yield a simpler / more consistent policy. The method shows robustness to various types of noise, and the experimental results appear to be consistent with the central motivations and claims.

There were some concerns about the claims in the paper wrt periodicity of the policy, tested environments, Markovian behaviors, etc. Some of these appear to have been addressed sufficiently, some still are somewhat unresolved. I have my own questions as well: the total correlation objective in Equation 1, as it includes the policy, can encourage the policy to move towards trivial / easy-to-predict states. This was a problem in earlier literature that attempted to use mutual information as a regularizer, and a careful balance was needed to ensure there was sufficient signal from the reward to avoid this sort of collapse. But the bound used here only has the policy parameters in the marginal terms (denominator, the joint action terms use an estimator q which doesn't appear to effect the policy parameters), which appears to mitigate this problem. I feel like this was an important point missed in the paper as well as the discussion, as this represents a discrepancy between the original objective and the final algorithm that non-trivially rectifies a known problem with using TC regularization on policies.

Despite this, for the most part the paper appears to be a good enough contribution. If possible I'd like to see some sort of discussion about the above, but I'm inclined to accept the paper.
Decision: Accept (poster)