Distributionally robust free energy principle for decision-making

Domain: physics
Published: 2025-12-17
Editorial outcome: minor_revision

Abstract
Despite their groundbreaking performance, autonomous agents can misbehave when training and environmental conditions become inconsistent, with minor mismatches leading to undesirable behaviors or even catastrophic failures. Robustness towards these training-environment ambiguities is a core requirement for intelligent agents and its fulfillment is a long-standing challenge toward their real-world deployments. Here, we introduce a Distributionally Robust Free Energy model (DR-FREE) that instills this core property by design. Combining a robust extension of the free energy principle with a resolution engine, DR-FREE wires robustness into the agent decision-making mechanisms. Across benchmark experiments, DR-FREE enables the agents to complete the task even when, in contrast, state-of-the-art models fail. This milestone may inspire both deployments in multi-agent settings and, at a perhaps deeper level, the quest for an explanation of how natural agents – with little or no training – survive in capricious environments.

Introduction
A popular approach to designing autonomous agents is to feed them with data, using Reinforcement Learning (RL) and simulators to train a policy (Fig. 1a). Deep RL agents designed on this paradigm have demonstrated remarkable abilities, including outracing human champions in Gran Turismo 1, playing Atari games 2, controlling plasmas 3, and achieving champion-level performance in drone races 4. However, despite their groundbreaking performance, state-of-the-art agents cannot yet compete with natural intelligence in terms of policy robustness: natural agents have, perhaps through evolution, acquired decision-making abilities so that they can function in challenging environments despite little or no training 5–7. In contrast, for artificial agents, even when they have access to a high fidelity simulator, the learned policies can be brittle to mismatches, or ambiguities, between the model available during learning and the real environment (Fig. 1b). For example, drone-champions and Atari-playing agents assume consistent environmental conditions from training, and if this assumption fails, because, e.g., the environment illumination or color of the objects changes, or the drone has a malfunctioning—so that its dynamics becomes different from the one available during training—learned policies can fail. More generally, model ambiguities—even if minor—can lead to non-robust behaviors and failures in open-world environments 8. Achieving robustness towards these training/environment ambiguities is a long-standing challenge 9–11 for the design of intelligent machines 5,12,13 that can operate in the real world.

Here we present DR-FREE, a free energy 14,15 computational model that addresses this challenge: DR-FREE instills this core property of intelligence directly into the agent decision-making mechanisms. This is achieved by grounding DR-FREE in the minimization of the free energy, a unifying account across information theory, machine learning 16–22, neuroscience, computational and cognitive sciences 23–29. The principle postulates that adaptive behaviors in natural and artificial agents arise from the minimization of the variational free energy (Fig. 1c). DR-FREE consists of two components. The first component is an extension of the free energy principle: the distributionally robust (DR) free energy (FREE) principle, which fundamentally reformulates how free energy minimizing agents handle ambiguity. While models based on the classic free energy principle (Fig. 1c) obtain a policy by minimizing the free energy based on a model of the environment available to the agent, under our robust principle, free energy is instead minimized across all possible environments within an ambiguity set around a trained model. The set is defined in terms of statistical complexity around the trained model. This means that the actions of the agent are sampled from a policy minimizing the maximum free energy across ambiguities. The robust principle yields the problem statement for policy computation. This is a distributionally robust problem having a free energy functional as objective and ambiguity constraints formalized in terms of statistical complexity. The problem has not only an onlinear cost functional with nonlinear constraints but also probability densities over decision variables that equip agents with explicit estimates of uncertainty and confidence. The product of this framework is a policy that minimizes free energy and is robust across model ambiguities. The second key component of DR-FREE—its resolution engine—is the method to compute this policy. In contrast to conventional approaches for policy computation based on the free energy principle, our method shows that the policy can be conveniently found by first maximizing the free energy across model ambiguities—furnishing a cost under ambiguity—and then minimizing the free energy in the policy space (Fig. 1d). Put simply, policies are selected under the best worst-case scenario, where the worst cases accommodate ambiguity. Our robust free energy principle yields—when there is no ambiguity—a problem statement for policy computation naturally arising across learning 21,30,31—in the context of maximum diffusion (MaxDiff) and maximum entropy (MaxEnt)—and control 32. This means that DR-FREE can yield policies that not only inherit all the desirable properties of these approaches but ensures them across an ambiguity set, which is explicitly defined in the formulation. In MaxEnt—and MaxDiff—robustness depends on the entropy of the optimal policy, with explicit bounds on the ambiguity set over which the policy is robust available in discrete settings 31. To compute a policy that robustly maximizes a reward, MaxEnt is used with a different, pessimistic, reward 31; this is not required in DR-FREE. These desirable features of our free energy computational model are enabled by its resolution engine. This is the method tackling the full distributionally robust, nonlinear and infinite-dimensional policy computation problem arising from our robust principle; see “Results” and Sec. S2 in Supplementary Information for details. In Supplementary Information we also highlight a connection with the Markov Decision Processes (MDPs) formalism. DR-FREE yields a policy with a well-defined structure: this is a soft-max with its exponent depending on ambiguity. This structure elucidates the crucial role of ambiguity on optimal decisions, i.e., how it modulates the probability of selecting a given action.

DR-FREE not only returns the policy arising from our free energy principle, but also establishes its performance limits. In doing so, DR-FREE harbors two implications. First, DR-FREE policy is interpretable and supports (Bayesian) belief updating. The second implication is that it is impossible for an agent faced with ambiguity to outperform an ambiguity-free agent. As ambiguity vanishes, DR-FREE recovers the policy of an agent that has perfect knowledge of its environment, and no agent can obtain better performance. Vice-versa, as ambiguity increases, DR-FREE shows that the policy down-weights the model available to the agent over ambiguity.

Results
DR-FREE comprises a distributionally robust free energy principle and the accompanying resolution engine—the principle (Fig. 2a) is the problem statement for policy computation; the resolution engine is the method for policy computation. The principle establishes a sequential policy optimization framework, where randomized policies arise from the minimization of the maximum free energy over ambiguity. The resolution engine finds the solution in the space of policies. This is done by computing—via the maximum free energy over all possible environments in the ambiguity set—a cost associated to ambiguity. Then, the ensuing maximum free energy is minimized in policy space (Fig. 1d).

In Fig. 1a, random variables X k−1 and U k are state at time k − 1 and action at k (see “Methods” and Section S3 of Supplementary Information). The agent infers the state of a stochastic environment p k(x k|x k−1, u k) and the action is sampled from π k(u k|x k−1).

In the decision horizon (e.g., from 1 to N), the agent-environment interactions are captured by p 0:N, defined as p 0(x 0) Π N k=1 p k(x k|x k−1, u k)π k(u k|x k−1), where p 0(x 0) is an initial prior. The trained model available to the agent, p k(x k|x k−1, u k), does not (necessarily) match its environment, and the goal of DR-FREE is to compute a policy that, while optimal, is robust against the training/environment ambiguities. For example, if—as in our experiments—the robot model of Fig. 1a is learned from corrupted/biased data, then it differs from the real robot, and this gives rise to ambiguities. These ambiguities are captured via the ambiguity set of Fig. 1b. For a given state/action pair, this is the set of all possible environments with statistical complexity within η k(x k−1, u k) from the trained model. For this reason, the ambiguity set B η(p k(x k|x k−1, u k)) is captured via the Kullback-Leibler (KL) divergence: the ambiguity set is the set of all possible models, say p k(x k|x k−1, u k), such that D KL(p k(x k|x k−1, u k)||p k(x k|x k−1, u k)) ≤ η k(x k−1, u k). The radius of ambiguity available to DR-FREE, η k(x k−1, u k), is positive and bounded. For a state/action pair, a small radius indicates low ambiguity, meaning that the agent is confident in its trained model. Vice-versa, high values indicate larger ambiguity and hence low confidence in the model. See “Methods” for the detailed definitions.

DR-FREE computes the optimal policy via a distributionally robust generalization of the free energy principle that accounts for ambiguities. This first component of DR-FREE, providing the problem statement for policy computation, generalizes the conventional free energy principle to ensure policies remain robust even when the environment deviates from the trained model. Our principle is formulated as follows: over the decision horizon, the optimal policy sequence {π* k(u k|x k−1)} 1:N is obtained by minimizing over the policies the maximum free energy across all possible environments in the ambiguity set expected under the policy in question. The expected free energy combines two terms: (i) the statistical complexity, D KL(p 0:N ||q 0:N), and (ii) the expected loss E p0:N [Σ N k=1 c k(x k) + c k(u k)]. The principle is formalized in Fig. 2a (see Methods for details) as a distributionally robust policy optimization problem in the probability space. DR-FREE computes the policy via bi-level optimization, without requiring the environment p k(x k|x k−1, u k) and without performing stochastic sampling for the ambiguity set. The ambiguity set defines the problem constraints—these are nonlinear in the decision variable—and the min-max objective is the free energy—also nonlinear in the decision variables (see Section S2 in the Supplementary Information for connections with other frameworks). As also shown in Fig. 2a, the complexity term in the min-max objective regularizes the optimal policy to prevent environment-agent interactions that are overly complex with respect to q 0:N. This is specified as q 0:N = q 0(x 0) Π N k=1 q k(x k|x k−1, u k)q k(u k|x k−1), with q 0(x 0) being a prior. Hence, the first term in the min-max objective biases the optimal DR-FREE solution of our robust principle—DR-FREE policy—towards q 0:N. The second term in the objective minimizes the worst case expected loss across ambiguity. We refer to q 0:N as generative model, although—for the application of our principle in Fig. 2a—this is not necessarily required to be a time-series model. For example, in some of our navigation experiments, q 0:N only encodes the agent goal destination—in this case the complexity term in Fig. 2a encodes an error from the goal. In contrast, in a second set of experiments—where we relate DR-FREE to a state-of-the-art approach from the literature 21,30—q 0:N encodes a time-series model. Our formulation also allows for the generative model q k(x k|x k−1, u k)—provided to DR-FREE—to be different from the true environment p k(x k|x k−1, u k)—not available to the agent—and the trained model p k(x k|x k−1, u k), see also Fig. 1a, b. This feature can be useful in applications where, by construction, the model available to the agent and the generative model differ 24. Both reactive and planned behaviors can be seen through the lenses of our formulation: a width of the decision horizon N greater than 1 means that the policy at time-step k is computed as part of a plan. If N = 1, the agent action is reactive (or greedy). These actions frequently emerge as reflexive responses in biologically inspired motor control 38,39. In brief, this generalization of active inference can be regarded as robust Bayesian model averaging to accommodate epistemic uncertainty about the environment, where the prior over models is supplied by the KL divergence between each model and the trained model. When there is no ambiguity, our robust principle in Fig. 2a connects with free energy minimization in active inference based upon expected free energy 40—which itself generalizes schemes such as KL control and control as inference 37,41–43. See “Methods” and Section S2 of Supplementary Information.

The policy optimization problem in Fig. 2a is infinite-dimensional as both minimization and maximization are in the space of probability densities. This enables a Bayes-optimal handling of uncertainty and ambiguity that characterizes control and planning as (active) inference. DR-FREE resolution engine—the method to compute the policy—finds the policy and returns a solution with a well-defined and explicit functional form. The analytical results behind the resolution engine are in Sections S3 and S6 of the Supplementary Information. In summary, the analytical results show that, at each k, the policy can be found via a bi-level optimization approach, first maximizing free energy across the ambiguity constraint and then minimizing over the policies. While the maximization problem is still infinite dimensional, its optimal value—yielding a cost of ambiguity—can be obtained by solving a scalar optimization problem. This scalar optimization problem is convex and has a global minimum. Therefore, once the cost of ambiguity is obtained, the resulting free energy can be minimized in the policy space, and the optimal policy is unique. These theoretical findings are summarized in Fig. 2b. Specifically, the policy at time-step k is a soft-max (Fig. 2b) obtained by equipping the generative model q k(u k|x k−1) with an exponential kernel. The kernel contains two costs: the action cost c k(u k) and the cost of ambiguity, η k(x k−1, u k) + e c k(x k−1, u k). Intuitively, given the cost-to-go c k(x k), the latter cost is the maximum free energy across all possible environments in the ambiguity set. The infinite dimensional free energy maximization step, which can be reduced to scalar convex optimization, yield a cost of ambiguity that is always bounded and non-negative (Fig. 2c and “Methods”). This implies that an agent always incurs a positive cost for ambiguity, and the higher this cost is for a given state/action pair, the lower the probability of sampling that action is. The result is that DR-FREE policy balances between the cost of ambiguity and the agent beliefs encoded in the generative model (Fig. 2c).

Methods
The agent has access to: (i) the generative model, q 0:N; (ii) the loss, specified via state/action costs c k(x k): X → R, c k(u k): U → R, with X and U being the state and action spaces (see Sec. S1 and Sec. S3 of the Supplementary Information for notation and details); (iii) the trained model p k(x k|x k−1, u k).

Distributionally robust free energy principle
Model ambiguities are specified via the ambiguity set around p k(x k|x k−1, u k), i.e., B η(p k(x k|x k−1, u k)). This is the set of all models with statistical complexity of at most η k(x k−1, u k) from p k(x k|x k−1, u k). Formally, B η(p k(x k|x k−1, u k)) is defined as the set {p k(x k|x k−1, u k) ∈ D: D KL(p k(x k|x k−1, u k)||p k(x k|x k−1, u k)) ≤ η k(x k−1, u k), supp p k(x k|x k−1, u k) ⊆ supp q k(x k|x k−1, u k)}.

The symbol D stands for the space of densities and supp for the support. The ambiguity set captures all the models that have statistical complexity of at most η k(x k−1, u k) from the trained model and that have a support included in the generative model. This second property, explicitly built in the ambiguity set makes the optimization meaningful as violation of the property would make the optimal free energy infinite. The radius η k(x k−1, u k) is positive and bounded. As summarized in Fig. 2a, the principle in the main text yields the following sequential policy optimization framework, where we highlight that the environment can change over time:

π* k(u k|x k−1) ∈ argmin {π k(u k|x k−1)} 1:N max {p k(x k|x k−1, u k)} 1:N [D KL(p 0:N ||q 0:N) + E p0:N [Σ N k=1 c k(X k) + c k(U k)]]

s.t.: p k(x k|x k−1, u k) ∈ B η(p k(x k|x k−1, u k)), k = 1, ..., N:

This is an extension of the free energy principle 27 accounting for policy robustness against model ambiguities. When the ambiguity constraint is removed our formulation links to the expected free energy minimization in active inference. In this special case, the expected complexity (i.e., ambiguity cost) becomes risk; namely, the KL divergence between inferred and preferred (i.e., trained) outcomes. The expected free energy can be expressed as risk plus ambiguity; however, the ambiguity in the expected free energy pertains to the ambiguity of likelihood mappings in the generative model (i.e., conditional entropy), not ambiguity about the generative model considered in our free energy model. In both robust and conventional active inference, the complexity term establishes a close relationship between optimal control and Jaynes’ maximum caliber (a.k.a., path entropy) or minimum entropy production principle 57,58. It is useful to note that, offering a generalization to free energy minimization in active inference, our robust formulation yields as special cases other popular computational models such as KL control 59, control as inference 41, and the Linear Quadratic Gaussian Regulator. Additionally, in special cases the negative of the variational free energy in the cost functional is the evidence lower bound 60, a key concept in machine learning and inverse reinforcement learning 46. With its resolution engine, DR-FREE shows that in this very broad set-up the optimal policy can still be computed.

Resolution engine
Both the variational free energy and the ambiguity constraint are nonlinear in the infinite-dimensional decision variables, and this poses a number of challenges that are addressed with our resolution engine. The resolution engine allows to tackle the sequential policy optimization framework arising from our robust free energy principle. We detail here the resolution engine and refer to Supplementary Information for the formal treatment. Our starting point is the robust free energy principle. This can be solved via a backward recursion where, starting from k = N, at each k the following optimization problem needs to be solved:

min π k(u k|x k−1) D KL(π k(u k|x k−1)||q k(u k|x k−1)) + E π k(u k|x k−1) [c k(u k) + max p k(x k|x k−1, u k) D KL(p k(x k|x k−1, u k)||q k(x k|x k−1, u k)) + E p k(x k|x k−1, u k) [c k(x k) + c k+1(x k)]]

s.t.: p k(x k|x k−1, u k) ∈ B η(p k(x k|x k−1, u k)), k = 1, ..., N:

In the above expression, for compactness we used the shorthand notations π k(u k|x k−1), q k(u k|x k−1), p k(x k|x k−1, u k) and q k(x k|x k−1, u k) for π k(u k|x k−1), q k(u k|x k−1), p k(x k|x k−1, u k) and q k(x k|x k−1, u k), respectively. The term c k(x k) is the cost-to-go. This is given by c k(x k) = c k(x k) + c k+1(x k), where c k+1(x k) is the smallest free energy that can be achieved by the agent at k + 1. That is, c k+1(x k) is the optimal solution of the above optimization problem evaluated at k + 1. When k = N, c k+1(x k) is initialized at 0. This means that, for reactive actions, e.g., N = 1, c k(x k) = c k(x k). The above reformulation is convenient because it reveals that, at each k, π* k(u k|x k−1) can be computed via a bi-level optimization approach, consisting in first maximizing over p k(x k|x k−1, u k), obtaining the maximum expected variational free energy across all possible environments in the ambiguity set, to finally minimize over the policies. Crucially, this does not mean the agent needs to know the actual maximization, but rather it only needs to know what the actual maximum free energy is. In turn, this can be found by first tackling the problem in green in Fig. 2b, i.e., finding the cost of ambiguity, and then taking the minimization in the policy space. While the problem in green in Fig. 2b is infinite dimensional, its optimal value can be found by solving a convex and scalar optimization problem. This is possible because the optimal value of the problem in green in Fig. 2b equals η k(x k−1, u k) + min α≥0 V α(x k−1, u k). In this expression, α is a scalar decision variable and V α(x k−1, u k), detailed in the Supplementary Information, is a function of α. In summary, the free energy maximization step can be conveniently solved with off-the-shelf software tools. In DR-FREE, the cost of ambiguity promotes robustness and contributes to the expected loss for the subsequent minimization problem in Fig. 2b. The optimal solution of this class of problems has an explicit expression (π* k(u k|x k−1) in Fig. 2b) and the optimal value is c k(x k) used at the next step in the recursion.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "The authors have provided a thorough and thoughtful revision of the manuscript, which is appreciated. This reviewer believes that changes made in response to comments have significantly improved the work.\nInitial concerns regarding the positioning of the work within the existing literature and the clarity of the core contributions have been sufficiently addressed. The newly added context in the introduction, along with the new Methods section and Supplementary Section S2, are effective. The clarification of the problem statement is a welcome addition, and the detailed discussion on the relationship between DR-FREE, MaxEnt, and MaxDiff now successfully situates the work and highlights its unique contributions. In turn, the manuscript's conceptual clarity has also improved. The \"resolution engine\" is now clearly defined as a novel method for solving the min-max policy optimization. Similarly, the improved explanation of the \"generative model,\" including the justification for why it is not limited to a time-series model, makes the framework's application potential more apparent and understandable. Lastly, the addition of experiments and comparisons have strengthened the paper's claims. The new comparisons against MaxDiff and the additional validation of the ambiguity-unaware agent with longer planning horizons support the work's central assertions.\nIn short, all major issues raised in the initial review have been successfully addressed, and the work is now recommended for publication.", "reviewer_2": "I thank the authors for their thorough response. I also agree with the reviewers regarding several points that should be better addressed in the paper: literature context, relation to RL (especially Distributionally Robust RL), experimental setup, and simplicity of the studied problem. I believe these concerns should be addressed. However, I will focus on my initial weaknesses. Concerns not explicitly mentioned here should be considered successfully resolved.\n**Weaknesses**\nW1: I thank the authors; the added lines 62–73 have definitely improved the paper. However, I encourage them to further study this question, i.e. by drawing a stronger connection between the studied problem and decision making (RL, MDP, etc.) in the form of a clear problem statement.\nW3: Thanks for the clarification regarding computing the ambiguity radius only via the given $\\bar p_k$ and $q_k$. That was an important point for me. In addition, I would like to highlight that I think it is not good practice to decouple the training of $\\bar p_k$ and $q_k$ from the whole method.\nW5: Thanks for the extra analytical connection. My remaining concerns lie in the empirical field. MaxDiff is compared with well-known RL approaches on several tasks and can be applied to problems out of the box, which does not seem to be the case for DR-FREE. I believe the paper would be strengthened with more experiments in the current domain (e.g. with different goals, and obstacle setups) and different ambiguity levels. Moreover, the approach should be tested in stochastic scenarious, but without ambiguity to asses the influence on the mechanism on the whole setup.\nI would also like to draw the authors’ attention to NN-MPPI [1], which is used for comparison with MaxDiff. I believe it would be a suitable baseline for comparison with DR-FREE.\nW6: I thank the authors for the modifications in the text. However, I still believe that using setups (environments) commonly found in the literature is necessary to better position the approach. In its current form, the empirical evaluation appears to be very simplistic task, tightly tied to the approach, and the applicability of the method to a wide range of tasks is questionable. Could the authors conduct experiments, for example, on the MuJoCo environments used in the MaxDiff paper (Swimmer, Ant, Half-Cheetah)?\n[1] Williams G, Wagener N, Goldfain B, Drews P, Rehg JM, Boots B, Theodorou EA. Information theoretic MPC for model-based reinforcement learning. In2017 IEEE international conference on robotics and automation (ICRA) 2017 May 29 (pp. 1714-1721). IEEE.", "reviewer_3": "Thank you for the effortful revision of the paper by reflecting my review comments adequately. I have no more comments, and recommend the paper to be published in the current journal.", "reviewer_4": "The revision is satisfactory and the paper is ready for publication."}, "author_response": "Authors’ Responses to the Reviewers’ Comments\nNatureCommunicationsManuscriptNCOMMS-25-23197A\nRobustDecision-MakingViaFreeEnergyMinimization\nAllahkaramShafiei,HozefaJesawada,KarlFristonandGiovanniRusso\nDearReviewers,\nPlease,findenclosedtherevisionofourmanuscriptNCOMMS-25-23197AentitledRobustDecision-MakingVia\nFreeEnergyMinimizationsubmittedinNatureCommunications.\nWe would like to thank you for the thoughtful and constructive feedback. As detailed in our point-by-point\nrepliesbelow,intherevisionweaccommodatedtheoutstandinginputsfromReviewer#2. Wehavestructured\nourrepliesinseparateblocks,correspondingtothesuggestions,commentsandobservationsmadebyeachof\nthereviewers. Foryourconvenience, changesthathavebeenmadetothemanuscriptarealsohighlightedin\nblueintherevisedpaper.\nWe hope that the revised version of the paper properly accommodates the inputs that were kindly provided\nandwehopethatthepaperisnowsuitableforpublication.\nWelookforwardtohearingfromyou. Sincerelyyours,\nTheauthors\nNCOMMS-25-23197A:RobustDecision-MakingViaFreeEnergyMinimization 1\nCommentsbyReviewer#1\n“THE AUTHORS HAVE PROVIDED A THOROUGH AND THOUGHTFUL REVISION OF THE MANUSCRIPT, WHICH IS AP-\nPRECIATED. THIS REVIEWER BELIEVES THAT CHANGES MADE IN RESPONSE TO COMMENTS HAVE SIGNIFICANTLY\nIMPROVEDTHEWORK.”\nWe would like to thank the reviewer for the constructive feedback that has kindly provided throughout the\nreviewprocess.\n[R1:1] “INITIAL CONCERNS REGARDING THE POSITIONING OF THE WORK WITHIN THE EXISTING LITERATURE AND THE\nCLARITY OF THE CORE CONTRIBUTIONS HAVE BEEN SUFFICIENTLY ADDRESSED. THE NEWLY ADDED CONTEXT IN\nTHE INTRODUCTION, ALONG WITH THE NEW METHODS SECTION AND SUPPLEMENTARY SECTION S2, ARE EFFEC-\nTIVE. THE CLARIFICATION OF THE PROBLEM STATEMENT IS A WELCOME ADDITION, AND THE DETAILED DISCUS-\nSION ON THE RELATIONSHIP BETWEEN DR-FREE, MAXENT, AND MAXDIFF NOW SUCCESSFULLY SITUATES THE\nWORK AND HIGHLIGHTS ITS UNIQUE CONTRIBUTIONS. IN TURN, THE MANUSCRIPT’S CONCEPTUAL CLARITY HAS\nALSO IMPROVED. THE ”RESOLUTION ENGINE” IS NOW CLEARLY DEFINED AS A NOVEL METHOD FOR SOLVING THE\nMIN-MAX POLICY OPTIMIZATION. SIMILARLY, THE IMPROVED EXPLANATION OF THE ”GENERATIVE MODEL,” IN-\nCLUDING THE JUSTIFICATION FOR WHY IT IS NOT LIMITED TO A TIME-SERIES MODEL, MAKES THE FRAMEWORK’S\nAPPLICATIONPOTENTIALMOREAPPARENTANDUNDERSTANDABLE. LASTLY,THEADDITIONOFEXPERIMENTSAND\nCOMPARISONSHAVESTRENGTHENEDTHEPAPER’SCLAIMS. THENEWCOMPARISONSAGAINSTMAXDIFFANDTHE\nADDITIONALVALIDATIONOFTHEAMBIGUITY-UNAWAREAGENTWITHLONGERPLANNINGHORIZONSSUPPORTTHE\nWORK’SCENTRALASSERTIONS.”\nWewouldliketothanktherevieweronceagainforhis/herpositivefeedbackonourpaper.\nNCOMMS-25-23197A:RobustDecision-MakingViaFreeEnergyMinimization 2\nCommentsbyReviewer#2\n“ITHANKTHEAUTHORSFORTHEIRTHOROUGHRESPONSE. IALSOAGREEWITHTHEREVIEWERSREGARDINGSEV-\nERAL POINTS THAT SHOULD BE BETTER ADDRESSED IN THE PAPER: LITERATURE CONTEXT, RELATION TO RL (ES-\nPECIALLY DISTRIBUTIONALLY ROBUST RL), EXPERIMENTAL SETUP, AND SIMPLICITY OF THE STUDIED PROBLEM. I\nBELIEVETHESECONCERNSSHOULDBEADDRESSED. HOWEVER,IWILLFOCUSONMYINITIALWEAKNESSES. CON-\nCERNSNOTEXPLICITLYMENTIONEDHERESHOULDBECONSIDEREDSUCCESSFULLYRESOLVED.”\nWethankthereviewerfortheconstructiveandthoughtfulcomments. Weappreciatethatmanyoftheearlier\nconcernshavebeensuccessfullyresolved. Followingthisfeedback,wehaverevisedthemanuscripttofurther\naddresstheremainingpointskindlypointedoutbythereviewer.Onceaddressed,theinputsfromthereviewer\nhelpedustostrengthenthepaper’spositioning,improvetheexperimentalevaluations,andclarifythecontri-\nbution. For your convenience, the edits referenced in our replies are also highlighted in blue in the revised\nmanuscript.\n**WEAKNESSES**\n[R2:1] “W1: I THANK THE AUTHORS; THE ADDED LINES 62–73 HAVE DEFINITELY IMPROVED THE PAPER. HOWEVER, I\nENCOURAGETHEMTOFURTHERSTUDYTHISQUESTION,I.E. BYDRAWINGASTRONGERCONNECTIONBETWEENTHE\nSTUDIEDPROBLEMANDDECISIONMAKING(RL,MDP,ETC.) INTHEFORMOFACLEARPROBLEMSTATEMENT.”\nIn response to this suggestion, we have introduced the new Section S2.1 in the Supplementary Infor-\nmation (page SI-3 and SI-4) where we explicitly draw the requested connection between our studied\nproblem and the MDPs formalism. The new section is also referenced in the main text (lines 73 – 75).\nThenewlyintroducedSectionS2.1complementsthecomprehensiveliteraturereviewinSectionS2,intro-\nducedduringthepreviousreviewcycle,whereweframeourproblemformulationinthecontextofthe\nbroadliteratureindecisionmakingacrosslearning,control,andoptimization.\n[R2:2] “W3: THANKSFORTHECLARIFICATIONREGARDINGCOMPUTINGTHEAMBIGUITYRADIUSONLYVIATHEGIVENp¯\nk\nANDq\nk\n. THATWASANIMPORTANTPOINTFORME. INADDITION,IWOULDLIKETOHIGHLIGHTTHATITHINKITIS\nNOTGOODPRACTICETODECOUPLETHETRAININGOFp¯\nk\nANDq\nk\nFROMTHEWHOLEMETHOD. ”\nWe thank the reviewer for this insightful follow-up comment. The reviewer is right that, in the experi-\nments,thetrainingisperformedoffline,separatefromactiongeneration. Thisapproachisalignedwith\ncommonpracticeinmodel-basedlearningandcontrolworkswhereofflinedataareleveragedtotraina\nmodel(e.g.,aneuralnetwork). Forexample,intheNN-MPPIpaper(kindlyhighlightedbythereviewer\nin comment [R2:4] below) the first phase to train the neural network model is from system identification\ndataandthisapproachisusedconsistentlyintheexperiments.Nevertheless,inviewofthiscomment,we\nelaborated upon our Discussion (lines 383 – 388) to explicitly highlight that integrating DR-FREE with\nlearning-based techniques – coupling training with policy computation – represents an important and\npromising direction for future research. Such integration would motivate further analytical studies to\nestimatethebenefits(e.g.,intermsofsamplecomplexity)ofthisintegratedpipelineoverofflinelearning.\n[R2:3] “W5: THANKS FOR THE EXTRA ANALYTICAL CONNECTION. MY REMAINING CONCERNS LIE IN THE EMPIRICAL\nFIELD. MAXDIFF IS COMPARED WITH WELL-KNOWN RL APPROACHES ON SEVERAL TASKS AND CAN BE APPLIED\nTO PROBLEMS OUT OF THE BOX, WHICH DOES NOT SEEM TO BE THE CASE FOR DR-FREE. I BELIEVE THE PAPER\nWOULD BE STRENGTHENED WITH MORE EXPERIMENTS IN THE CURRENT DOMAIN (E.G. WITH DIFFERENT GOALS,\nAND OBSTACLE SETUPS) AND DIFFERENT AMBIGUITY LEVELS. MOREOVER, THE APPROACH SHOULD BE TESTED\nIN STOCHASTIC SCENARIOUS, BUT WITHOUT AMBIGUITY TO ASSES THE INFLUENCE ON THE MECHANISM ON THE\nWHOLESETUP. ”\nNCOMMS-25-23197A:RobustDecision-MakingViaFreeEnergyMinimization 3\nWethankthereviewerforthisconstructiveandencouragingcomment,whichhelpedusnotonlytocom-\nplementtheextraanalyticalconnection–providingguaranteesthatDR-FREEprotectsagainstambiguity\nasdefinedintheambiguityset–butalsotooverallstrengthenourexperiments. Inresponse,wesubstan-\ntiallyextendedourempiricalevaluationsasfollows:\n• differentgoalsandobstaclesetups – we added new experiments confirming that, even under am-\nbiguity, DR-FREE consistently enables the robot to complete the task across all considered obsta-\ncles/goals setups. These results are presented in the main text (lines 216 – 219) and in the newly\nintroducedExtendedData4;\n• differentambiguitylevels–wenowassessDR-FREEundervaryingambiguitylevels(lines255–257\nandExtendedData5).Remarkably,whentheambiguitylevelissettozero,theambiguityconstraint\nbecomesrelaxedandDR-FREEcoincideswiththepolicyoftherelaxedproblem. Thisaspectisnow\nexplicitlyelaboratedinthemaintext(lines249–251),SupplementaryInformation(lines182–186\nandFig. SI-2)andreferencedinthecaptionofExtendedData5;\n• stochasticscenarioswithoutambiguity–following-uponthepreviousexperiments,weaddednew\nexperimentsforstochasticenvironmentswithoutambiguity. Theexperimentsresultsarediscussed\ninthemaintext(lines257–262)andreportedinExtendedData6.\nRemarkably, all experiments results are consistent with our analysis. Moreover, the experiments also\nhighlight that, to apply DR-FREE to these different scenarios, no changes to its code were necessary.\nThis modularity is preserved across diverse tasks, including those from MuJoCo kindly pointed out by\nthe reviewer in his/her comment [R2:5]. In fact, as in the MaxDiff experiments (see lines 189 – 190 in\nSupplementaryInformationintroducedduringthepreviousreviewcycle)adaptingDR-FREEtodifferent\nproblemsrequirescreatingawrapperaroundtheenvironmentandnotchangingthemethoditself.\n[R2:4] “I WOULD ALSO LIKE TO DRAW THE AUTHORS’ ATTENTION TO NN-MPPI [1], WHICH IS USED FOR COMPARISON\nWITHMAXDIFF. IBELIEVEITWOULDBEASUITABLEBASELINEFORCOMPARISONWITHDR-FREE.\n[1] WILLIAMS G, WAGENER N, GOLDFAIN B, DREWS P, REHG JM, BOOTS B, THEODOROU EA. INFORMATION\nTHEORETIC MPC FOR MODEL-BASED REINFORCEMENT LEARNING. IN2017 IEEE INTERNATIONAL CONFERENCE\nONROBOTICSANDAUTOMATION(ICRA)2017MAY29(PP. 1714-1721). IEEE.”\nWethankthereviewerfordrawingourattentiontotheNN-MPPIpaper[1],whichisnowproperlycited\nin the main text. Following this helpful suggestion, we have also included NN-MPPI as an additional\nbaselineinthenewMuJoCoexperimentskindlyhighlightedbythereviewerinhis/hercomment[R2:5].\n[R2:5] “W6: I THANK THE AUTHORS FOR THE MODIFICATIONS IN THE TEXT. HOWEVER, I STILL BELIEVE THAT USING\nSETUPS (ENVIRONMENTS) COMMONLY FOUND IN THE LITERATURE IS NECESSARY TO BETTER POSITION THE AP-\nPROACH. IN ITS CURRENT FORM, THE EMPIRICAL EVALUATION APPEARS TO BE VERY SIMPLISTIC TASK, TIGHTLY\nTIEDTOTHEAPPROACH,ANDTHEAPPLICABILITYOFTHEMETHODTOAWIDERANGEOFTASKSISQUESTIONABLE.\nCOULD THE AUTHORS CONDUCT EXPERIMENTS, FOR EXAMPLE, ON THE MUJOCO ENVIRONMENTS USED IN THE\nMAXDIFFPAPER(SWIMMER,ANT,HALF-CHEETAH)?”\nWe thank the reviewer for this thoughtful suggestion, which helped us to improve our empirical eval-\nuation and demonstrate that DR-FREE performs effectively on standard benchmark environments. In\nresponse, intherevisedpaperwehaveaddednewexperimentsontheAntenvironmentfromMuJoCo,\nwhich features higher-dimensional states/actions than Swimmer and Half-Cheetah. In these experi-\nments,wecompareDR-FREEwithbothMaxDiffandNN-MPPIusingthestandardenvironmentsetup.\nTheresultsshowthatDR-FREEconsistentlyachieveshigherreturnsthanallotherconsideredmethods.\nNCOMMS-25-23197A:RobustDecision-MakingViaFreeEnergyMinimization 4\nMoreover, as detailed in the text, the other methods would terminate prematurely (fail) due to the Ant\nbecomingunhealthy–asdefinedintheenvironmentdocumentation. Incontrast,acrossallDR-FREEex-\nperiments,theAntneverbecomesunhealthy. Resultsareintroducedinthemaintextoftherevisedpaper\n(lines340–354)andinthenewlyintroducedFig. 5. FurtherdetailsareprovidedinMethods(lines688\n–696)andinSupplementaryInformation(lines198–208andthenewlyintroducedFig. SI-3andTab.\nS-4,referencedtherein). Moreover,wealsofine-tunedtheAbstract(lines15–18)Introduction(lines91\n–92)andDiscussion(lines368–371)toproperlyreferencethisnewsetofexperiments.\nNCOMMS-25-23197A:RobustDecision-MakingViaFreeEnergyMinimization 5\nCommentsbyReviewer#3\n“THANKYOUFORTHEEFFORTFULREVISIONOFTHEPAPERBYREFLECTINGMYREVIEWCOMMENTSADEQUATELY. I\nHAVENOMORECOMMENTS,ANDRECOMMENDTHEPAPERTOBEPUBLISHEDINTHECURRENTJOURNAL.”\nWewouldliketothankthereviewerforthehelpfulandconstructivecommentsthathavekindlyprovided.\nNCOMMS-25-23197A:RobustDecision-MakingViaFreeEnergyMinimization 6\nCommentsbyReviewer#4\n“THEREVISIONISSATISFACTORYANDTHEPAPERISREADYFORPUBLICATION.”\nWewouldliketothankthereviewerforhis/herconstructivefeedback.\nAuthors’ Responses to the Reviewers’ Comments\nNatureCommunicationsManuscriptNCOMMS-25-23197A\nRobustDecision-MakingViaFreeEnergyMinimization\nAllahkaramShafiei,HozefaJesawada,KarlFristonandGiovanniRusso\nDearReviewers,\nWe would like to thank you once again for the thoughtful and constructive feedback. The comments and\nsuggestionsledtoimprovementsovertheoriginallysubmittedmanuscript.\nSincerelyyours,\nTheauthors\nNCOMMS-25-23197B 1\nCommentsbyReviewer#1\n“THE AUTHORS HAVE ADDRESSED ALL MY PREVIOUS COMMENTS AND CONCERNS, AND THE REVISION HAS IM-\nPROVED THE PAPER. I AM SATISFIED WITH THE CHANGES AND BELIEVE THE MANUSCRIPT IS NOW SUITABLE FOR\nPUBLICATION, AND I WOULD LIKE TO THANK THE AUTHORS FOR THEIR CAREFUL WORK AND INVOLVEMENT IN\nIMPROVINGIT.”\nWe would like to thank the reviewer for the constructive feedback that has kindly provided throughout the\nreviewprocess."}, {"round": 2, "reviewer_comments": {"reviewer_2": "Report on “Robust Decision-Making Via Free Energy\nMinimization”\nAllahkaram Shafiei, Hozefa Jesawada, Karl Friston, Giovanni Russo\nThis paper presents a highly innovative approach by introducing the DR-FREE mechanism, which embeds robustness directly into an agent’s decision-making process by leveraging free-energy minimization principles, rather than relying on traditional training methods. The theoretical framework is well-structured, and the real-world robotic navigation experiments convincingly demonstrate the method’s effectiveness, particularly in scenarios where conventional free-energyminimizationfails. Thepotentialimpactofthisworkonreliableautonomousagents is significant and deserves recognition.\nSince the method is based on distributionally robust optimization formulated with an ambiguity set, it is designed to ensure robustness against worst-case distributions within a specified uncertainty set. As a result, the obtained solutions prioritize worst-case performance, which may lead to a degradation in average-case performance compared to methods that optimize for expected outcomes. This trade-off is a fundamental characteristic of distributionally robust optimization.\nI have the following questions:\n1. This work appears to be an extension of the research presented in Reference [39]. The problems addressed in Reference [39] are as follows:\nN\np*(u |x ) ∈ arg min { D (p ||q )+E [ (cid:88) c (X ) ] } . (1)\nk k k-1 KL 0:N 0:N p k (x k |x k-1 ,u k ) k k\np (u |x )\nk k k-1\nk=1\nIn contrast, the problem studied in the present work is formulated as\nN\np*(u |x ) ∈ arg min max { D (p ||q )+E [ (cid:88) c (X )+c (U ) ] }\nk k k-1 KL 0:N 0:N k k k k\np (u |x ) p (x |x ,u ) k k k-1 k k k-1 k\nk=1\n(2)\n( ) ( )\ns.t. p (x |x ,u ) ∈ B ( p¯ (x |x ,u ) ), (3)\nk k k-1 k n k k k-1 k\nwhere an ambiguity set B (·) is introduced to enforce distributional robustness. The key\nn\ndistinction is the addition of this ambiguity set, yet the Introduction does not explicitly\nstate how this feature differentiates the current study from Reference [39].\n2. BasedonComment1,Reference[39]doesnotdiscussanyfreeenergyformulation,whereas\nthe current manuscript frames its entire approach around free-energy minimization. It\nis not clear whether the introduction of the ambiguity uncertainty set is what drives the\nshift from the original problem in Reference [39] to a free-energy¨Cbased problem.\n3. Although the variational free energy cost function is supported by well established deriva-\ntions and explanatory formulae, the manuscript introduces an expected free energy objec-\ntivewithoutexplicitattributionorderivation. Itremainsunclearwhetherthisformulation\nis drawn from prior literature or specifically devised for the problem at hand.\n4. Free energy offers a unified framework for perception, action, and learning, but it is not\nclearhowthismanuscriptrealizesthatintegrationundertheexpectedfreeenergyformula-\ntion. In particular, the control policy shapes perceptual predictions via the KL divergence\nterm while simultaneously governing task performance through the expected loss term.\nVariational free energy research often examines the resulting balance¡ªor bias¡ªbetween\nperception and action. Does the expected free energy objective introduce a systematic\nbias toward perceptual accuracy or toward control effectiveness? If so, please discuss this\ntrade off in the text and, ideally, include experimental results that illustrate its impact\non agent behavior.\n5. The experimental results indicate that the expected cumulative loss inherently enforces\nconstraints such as obstacle avoidance. It is unclear whether this capability stems from\nthe inclusion of explicit obstacle avoidance constraints within the optimization problem\nor from embedding these requirements directly into the cost function. Please clarify the\nimplementation details.\n6. Onpage8, theterm¡°ambiguity-unawarefree-energyminimizingagent¡±appearswithout\nan explicit citation. Since Reference [39] does not address an ambiguity-unaware free-\nenergy formulation, please clarify the origin of this baseline¡ªwhether it refers to another\nsource or is newly introduced here¡ªand provide the appropriate reference.\n7. The computational complexity of solving the DR-FREE optimization problem may be\nsubstantially higher than that of the ambiguity-unaware free-energy approach. It would\nbe valuable to include a comparative table of average runtimes for both methods in the\nexperimental section, allowing readers to assess the practical overhead introduced by\nDR-FREE.\nTypographical corrections and suggestions for minor improvements.\n1. Page 1. The Abstract introduces the abbreviation ¡°DR-FREE¡± and includes litera-\nture citations without first defining the full term or providing context, which may im-\npede reader comprehension. It is recommended to spell out the complete phrase (e.g.,\n¡°Distributionally Robust Free Energy (DR-FREE)¡±) upon first use and to defer detailed\ncitations to the main text.\n2. In Figure 2(a), the optimal solution is denoted with the membership symbol ¡°∈¡± rather\nthan an equality sign. Please clarify whether this indicates selection from the arg-min set\n(i.e., p*(u |x ) ∈ argmin{...} by set membership) or if equality was intended. If the\nk k k-1\nlatter, replace ¡°∈¡± with ¡°=¡±.\n3. Page 6. The expression\nN\nE [ (cid:88) c (X )+c (U ) ]\nk k k k\nk=1\nlacks clear grouping of the summand. It should be formatted as\nN\nE [ (cid:88) ( c (X )+c (U ) ) ]\nk k k k\nk=1\n4. ThevariablesXandUareusedthroughoutthemanuscriptbutareneverformallydefined.\nPlease introduce clear definitions (e.g., their domains, dimensions, and interpretation) at\nthe point of first use.\n5. InReference[7],[29],[33],[34],[40], [42], themonthofpublicationisomitted. Pleaseinclude\nthe publication month to ensure the citation is complete and accurate.\n6. In Reference [10], [11], the authors¡¯ names are presented in abbreviated form, whereas\nfull names are used in other citations. Please ensure consistency by providing full author\nnames."}, "author_response": "Authors’ Responses to the Reviewers’ Comments\nNatureCommunicationsManuscriptNCOMMS-25-23197A\nRobustDecision-MakingViaFreeEnergyMinimization\nAllahkaramShafiei,HozefaJesawada,KarlFristonandGiovanniRusso\nDearReviewers,\nWe would like to thank you once again for the thoughtful and constructive feedback. The comments and\nsuggestionsledtoimprovementsovertheoriginallysubmittedmanuscript.\nSincerelyyours,\nTheauthors\nNCOMMS-25-23197B 1\nCommentsbyReviewer#1\n“THE AUTHORS HAVE ADDRESSED ALL MY PREVIOUS COMMENTS CONCERNS, AND THE REVISION HAS IM-\nPROVED THE PAPER. I AM SATISFIED WITH THE CHANGES AND BELIEVE THE MANUSCRIPT IS NOW SUITABLE FOR\nPUBLICATION, AND I WOULD LIKE TO THANK THE AUTHORS FOR THEIR CAREFUL WORK AND INVOLVEMENT IN\nIMPROVINGIT.”\nWe would like to thank the reviewer for the constructive feedback that has kindly provided throughout the\nreviewprocess."}]

Decision letter