FB-MOAC: A Reinforcement Learning Algorithm for Forward-Backward Markov Decision Processes
Authors: Mohsen Amidzadeh
Mario Di Francesco
Venue: Accepted by TMLR ()

Abstract: Reinforcement learning (RL) algorithms are effective in solving problems that can be modeled as Markov decision processes (MDPs). These algorithms primarily target forward MDPs whose dynamics evolve over time from an initial state. However, several important problems in different scenarios including stochastic control and network systems exhibit both a forward and a backward dynamics. As a consequence, they cannot be expressed as a standard MDP, thereby calling for a novel theory for RL in this context. Accordingly, this work introduces the concept of Forward-Backward Markov Decision Processes (FB-MDPs) for multi-objective problems, develops a novel theoretical framework to characterize their optimal solutions, and proposes a general forward-backward step-wise template that allows to adapt RL algorithms for FB-MDP problems. A Forward Backward Multi Objective Actor Critic (FB-MOAC) algorithm is introduced accordingly to obtain optimal policies with guaranteed convergence and a competitive rate with respect to standard approaches in RL. FB-MOAC is evaluated on diverse use cases in the context of mathematical finance and mobile resource management. The obtained results show that FB-MOAC outperforms the state of the art across different metrics, highlighting its ability to learn and maximize rewards.

Published in Transactions on Machine Learning Research (07/2025)
FB-MOAC: A Reinforcement Learning Algorithm for Forward-Backward Markov Decision Processes
Mohsen Amidzadeh
molsen.omidzadeBBolkk.
Deportment of Computer Science
Aulto Universily, Finlond
Mario Di Francesco
mario. ftt..d....ucccecooaaaat.oooooo
Department of Computer Science Aolto University, Finland
Reviewed on OpenReview: openreviee..........mmmu net/ httt/frmm? id= L55DyC6rfS
Abstract
Reinforcement learning (RT) algorithms are effective in solving problems that can be modeled as Markov decision processes (MDPs). These algorithmms primarily target forward MDPs whose dynamics evolve over time from an initial state. However, several important problems in different scenarios including stochastic control and network systems exhibit both a forward and backward dynamics. As a consoquence, they cannot be expressed as a standard MDP, thereby calling for a novel theory for RL in this context. Accordingly, this work introdeees the concept of problems, develops novel theoretical framework to characterize their optimal solutions, and proposes a general forward-backwarr step-wise template that allows to adapt RL algorithms for FB-MDP problems. A Forward Backward Multi Objective Actor Critic (FB-MOAC) ahgorithm is introduced accordingly to obtain optimal policies with guaranteed convergence and competitive rate with respect to standard approaches RL. FB-MOAC is evaluated on diverse use cases in the context of mathematical finance and mobile resource management. The obtained results show that FB-MOAC outperforms the state of the art across different metrics, highlighting its ability to learn and maximize rewards.
Markov Decision Processes (FB-MDPs) for multi-objective
1 Introduction
Reinforcement Learning (RI.) a very important field of artificial intelligence, as enables agents learn from experience and adapt complex, dynamic environments (Mnih et al., 20133] Lillicrap al. 2016 Schulman et al. 2017b). Moreover, recent breakthroughs in deep learning have led to solutions that surpass human performance in wide variety of challenges. As a result, deep reinforcement learning has lately emerged as a combination of these two fields, with successful applications in different use cases (Mnih et al.
2015
Jaderberg et al. 2018 Rigoli et al. 2021
Existing RT. algorithms mainly address sequential decision-making problems modeled as a forward Markov dccision process (MDP) or controlled forward dynamics (Zare ct al.) 2023). However, there are several sequential tasks whose environment cannot be exclusively captured by this type of dynamics, encompass states evolving backwards in time (Lai et al. 2020 Wang et al. 2021). Such backward dynamics describe a trajectory in a reverse chronological order, wherein the future affects the past. Even further, there are environments exhibiting both controlled forward and backward dynamics at the same time (Ji et al.)
they also
2022a Zhang 2022) namely, as the forward-backwarrd MDP (FB-MDP) illustrated in Figure la
FB-MDPs have wide applications (Section 2.1 For instance, they can be employed to discretize forward- backward stochastic differential equations (SDEs)
(see Appendix F for a detailed account), thereby allowing
to solve stochastic optimal control problems 7hang 2017
Ji et. al. 2020 Moreover, they enable accurate

---

Published in Transactions on Machine Learning Research (07 (07/2025)
a ~ T((.
Single Polisy
a
Actor Network
Spure
Forward Dynamics
Forward Evalusaion
Backward Evaluation
Forward Criic Networks
Bidrectional Critic Networks
St
S Plls, a)
Ytur Pol. ar.)
Forward MDP
Backwaard MDP
8
a
Y+
Learning
a.
Eptsodic
V1
Y=1
y,
Backward Dynamics
VTo
(s.. ax)
Forward Backward
Alyy Spa ay.)
Multi Oll---ljeiivee
Tt(-Is.) (s.s)
Oplinization
(a)
(b)
Corward-Backwart Multi-objective Optimization
A(ss, a.)
Ty S.o a.))
x, (s.)
(a 1 s,,)
Too5
rew
fLat
Compute Forward Gradients
Compute Backward Gradients
-0.06
V V
Cumulative Reward
Compute the common descent direction
g(.)
-3.15
Updite policy using mutti-obeective SCD
FB-MOAC (ours)
020
MOAC
q(-
PPO
(c)
(d)
Figure 1: Overview of our approach. (a) A forward-backward MDP in which forward states (sthe and backward states (yr)e(L))) apply the
actions (ar)ee(L,)) but with different ordering in time. b The FB-MOAC
algorithm comprises three steps: forward evaluation, bachward coaluation and bidirectional learning. During the first two steps, the forward and backward dynamics are evaluated and the resulting experiences buffered. The policy distribution is optimized in the bidirectional learning step based on the experiences of both forward and backward dynamics. For this purpose, a forward- backward malti-objectivve optimization employed by following appropriate chronological order. The episodic MCS-averuge add-on boosts the convergence to Pareto-optinal solutions. (c] The multi-objective optimization module of the FB-MOAC algorithm computes: the vectorvvalued gradients of forward and backward objectives; the descent direction q(-) to ensure that all rewards increase simultaneously; and the parameters
actor network based on q--). d The cumulative reward of our approach (FB-MOAC) compared the widely-used proximal policy optimization (PPO) and to MOAC (our multi-objective extension of the advantage actor critic) in the edge caching use case for different metrics (see Section 5.2 for a detailed account). FB-MOAC performs better than the other algorithms in terms of the overall reward.
modeling of delay latency in the context of network communicatinns and of the execution time for task offloading in cloud edge computing systems (Liu et al. 2019 Wei et al. 2019 Chen et al. 2019b). However, forward-backward dynamics have been marginally addressed in the context of RL and MDPs (Section 2.2 In fact, existing research only formulated a deep learning characterization in terms of borward-backward SDEs (Ji et al., 2020) or considered artificial backward trajectories in forward MDPs to increase the sample efficiency of RL algorithms (Goyal al. 2019 Wang al.) 2021). ) In contrast, and fill this gap, we introduce the concept of FB-MDPs for multi-objective problems entailing both forward and backward rewards that conflict with each other throughout the action space. We then extract the properties of Pareto-optimal
2

---

Published in Transactions on Machine Learning Research (07/2025)
solutions and extend the optimality Bellman equation for this class of MDPs. We provide general solution mechanism and then develop an RT. algorithm specifically suited to FB-MDPs accordingly.
In detail, this work establishes the following contributions.
We introduce the notion of mulli-objective FB-MDPs to express sequential multi-task decision-makee problems with both forward and backward controlled dynamics, whose rewards are coupled within the action space. We notably show that these MDPs cannot be expressed as a standard MDP, and we consequently develop novel theoretical framework to characterize their optimal solutions (Section 3
We propose a step-wise mechanism as a general and versatile template to adapt RL algorithms for FB- MDPs. Accordingly, we devise a multi-objective RL algorithm, called Forward Backward Multi-Objectivn Actor-Critic (FB-MOAC), the first of its kind, to obtaín Pareto-optimal solutions. We further provide a rigorous analysis of FB-MOAC, showing that it reaches convergence with rate of OLI/VI), where I is the number of policy updates (Section
We conduct comprehensive evaluation by considering diverse use cases expressed as FB-MDPs in the context of mathematical finance and mobile resource management. The results establish the necessity of modeling these problems as FB-MDPs, since approximating them with standard MDPs leads to a sub-optimal solution. The evaluation also demonstrates the effectiveness of FB-MOAC compared to RL algorithms in the state of the art (Section 5
we use lower-case a for scalars, hold-face lower-case for vectors, and bold-face uppercase A for matrices. Moreover, A is the transpose of A, ||A]| is the induced matrix norm of A, I is the identity matrix, 1 vector with all elements equal to one, 0 a vector with all elements equal to zero, and em a vector with all elements equal to zero except the m-th element which is equal to one. Finally, a = [ai,..., an) are the components of a n-dimensional column vector a, |S| the cardinality of the set S, and [.] indicates the components of row vectors.
Notation:
2 Background
This section first introduces a few motivating examples of FB-MDPs and then reviews the most relevant works in the existing literature.
2.1 Motivating Examples
Problems Described by Stochastic Differential Equations
SDEs exhibiting anti-causal dynamics have several applications in the context of differential games (Hamadene & Lepeltier 1995), diffusion models (Yang et. al. 2023), and mathematical finance (Ji et al., 2022a).. In particular, problems involving FB-SDE: represent significant portion of the ongoing research in the field of stochastic control theory (Yong 2023 Among them, one example is given by an scenario in mathematical finance. Consider a financial market with single risky asset whose price follows a stochastic process. A trader can invest in this risky asset or engage in risk-free borrowing lending. The trader's total wealth Y (t) evolves based on their investment in the asset and the risk-free rate. Now, consider payoff at future time T, which depends on the asset's price. The goal is determine the minimal initial Yo. required to replicate this payoff. The investment, strategy guaranteeing that the final wealth matches the option's 's payoff is characterized by backward dynamics (Ji ct al. 2022a). In such a context, Section 5.1 presents use case related lo mathematical finance, based on general method to transform an FB-SDE into an FB-MDP (see Appendix F for more details).
Network Content Delivery
Let us consider scenario in which a content provider (e.g., in a video streaming context) serves users by transmitting N content items with different, popularity iN over lossy network. Transmissions take
3

---

Published in Transactions on Machine Learning Research (07/2025)
place in time slots indexed by t, and the delivery of content n fails with an error probability e, (t). Upon failure, user requests are re-transmitted until successful delivery. The request probability of content n clearly depends on the success rate of previous requests and the error probability ex (t). Therefore, it establishes a controlled forward dynamics as the content provider affects (t). Now, the average latency (t()) experienced by a typical user to successfully receive file n is obtained by: (1) = d(1) (1 e (1) + (T(1) + (1 + +)) (1), where: d(t) is the transmission delay between the content provider and the user; and =t) is the duration of time slot t. This equation is obtained by the law of total expectation and exhibits a controlled backward dynamics with (t) as a backward state. As a consequence, minimizing the overall latency Pn= (t))()())) in this context makes the problem an FB-MDP. Existing RL algorithms cannot be applied here since a backward MDP cannot be converted a standard forward MDP (see Theorem 3.6). Note that this FB-MDP cannot be completely replaced by standard MDP problem either. In fact, one rewarding policy could be to only track the number of successful transmissions. A higher success rate over [0. [] correlates with lower latency, however, these two metrics are not interchangeable; one can maintain the same success percentage yet obtain a different overall latency Σm=1 Pro (t). For instance, both the number of successful transmissions and the average latency (t) become determined by fixing the crror probability (t) over [o, T). If we then permute en (t) in time,, the total count of successful transmissions remains unchanged, but the overall latency differs due to presence of (t). Therefore, the success (or failure) of content transmissions does not completely map to the overall latency. Content delivery is only one instance of network problems that can be modeled as FB-MDP; Scction 5.2 presents use case edge caching in wireless networks, whereas Appendix D addresses computation ofllodingg.
Pr.
2.2
Related Work
Korward-Backwark MDPs. Our work shares similarity with prior research on RL algorithms (Edwards) et al.
2018 Goyal et al. 2019 Wang et al., 2021 Lai et al. 2020 Archibald et al. 2023 These studies hypothesize that (creating) virtual backward trajectory, in relation to a forward dynamics, enhances the sample efficiency of RL algorithms. Specifically, they employ the generated backward trajectories to augment the training dataset for learning forward MDP (SDE) problems. Edwards et al.] (2018) train a learnable backwards dynamics that generates fictitious reversal steps from known goal states. These backward paths are then used to augment the replay buffer and contribute to the learning procedure of the considered RL algorithm. Similarly, Goyal et al. (2019) learn an artificial backward model (called backtracking model) trained agent experiences to predict precoding state-action pairs leading to high-value states. The backtracking model then enriches the training dataset by alternative trajectories, leading to better performance. [ai et al. (2020) construct a backward dynamics model and leverages it alongside the forward model to generate short branched rollouts for policy optimization. This approach theoretically derives a tighter bound of return discrepancy and demonstrates improved performance compared to a forward-only method. WWan et al. (2021 introduce a learnable backward dynamics trained on offline datasets to generate rollouts largeting goal states. They augment the training dataset to cope with the distribution shift between the learning policy and the given offline dataset. Our reference model is characterized based on real backward and forward dynamics, in contrast with the works described above, wherein backward dynamics are artificially constructed based on a forward MDP (dynamics) and it is not independently controlled. Our model also considers rewards jointly competing in both directions of time. Consequently, our investigation is centered around a class of FB-MDPs for multi-task problems and allows for the development of RL algorithms that inherently consider bidirectional dynamics, rather than relying on fictitious backward trajectories. Therefore, it provides a more integrated and principled mothod for incorporating backward rcasoning into RL.
Multi-objective RL Algorithms. The majority of Multi-Objective Reinforcement Learning (MO-RL) algorithms has primarily been designed for discrete environments. Mossalam et al. (2016) introduce MO-RL algorithm that combines deep Q-learning and optimistic lincar support lcarning. Their approach take into account a scalarized vector and potential optima to formulate a convex combination of all objectives. However, they require searching over all potential scalarizing vectors since an a priori knowledge on the importance of distinct objectives not available. Yang et al.) (2019) employ multi-objective Q-learning together with a single- agent framework to acquire a preferenco-rciated adjustment that can be generalized across different objectives. Such an approach is computationally efficient, however, it often suffers from sample inefficiency and results
4

---

Published in Transactions on Machine Learning Research (07 (07/2025)
in sub-optimal policy. MO-RL algorithms have been specifically developed for continuous environments as well. Zhhan & Cao (219)) establish reward-specific state-value functions based on correlation matrix to obtain the relative importance of objectives with respect to each other. However, their approach requires to adjust the weight of such a matrix to determine an appropriate inter-objective relationship. Abdolmaloki et al. (2020) devise a MO-RI. approach according to the marimam a posteriori policy oplimizalion algorithm. They learn objective-specific policy distributions to identify Pareto-optimal solutions in seale-independent manner. However, objective-specific coefficients must be adjusted to control the impact on the policy update. In contrast, we propose MO-RL algorithm for continuous- valued FB-MDPs (namely, FB-MOAC), without considering any initial preferences for the different objectives. Different from previous works (Abdolmaleki et al.) 2020 Zhan & Cao 2019 Chen et al, 2019a), we devise a single-policy approach to simplify the algorithu and avoid the need for an initial assumption on the reward preference. Moreover, a comprehensive analysis has been conducted to ensure the convergence of FB-MOAC to Pareto-fron solutions at a certain rate. A remarkable result of such analysis is the ability of FB-MOAC to monotonically incrcase all expectod objectives for any reward preference, thereby making the algorithm scale-insensitive.
Convergence Analysis of RL algorithms. A few recent works (Qiu et al. 2021 Xu et al. 2020 Fu
ct al. 2021 Yang et al., 2018 Khodadadian et al. 2022) have characterized RL algorithms with stochastic
policy, such as Actor- (AC) and Policy Gradient (Sutton & Barto 2018). Qiu et al. (022)) conduct a rigorous convergence analysis of the AC algorithm. Notably, their analysis is limited to a linear representation of the state-value function. Xu et al. (2020) provide a comprehensive characterization of the convergence rate and sample complexity of the Natural Actor-Critic (NAC) algorithm (Poters & Schaal 2008). Their analysis requires that the considered MDP is ergodic. Fu et al. (2021) analyze the convergence of the AC algorithm under the assumption that the considered family of neural networks closed under the Bellman operator. Lastly, [Khodadadian et al. (2022) perform an accurate convergence analysis of the Natural Policy Gradient algorithm (Kakade) 2002). However, their investigation assumes that the initialization value of the state-value function is sufficiently close to the optimal value function. All the aforementioned works address the convergence of stochastic policies in single-objective RL algorithuns for forward MDP problems. In contrast, this work targets multi-lask problems involving a FB-MDP. We carry out rigorous convergence analysis as a solid foundation to characterize multi-objective and
RL algorithms for FB-MDPs.
Applications of RL to Network Systems. RL algorithms have also been applied network systems. particularly, to design dynamic caching and offloading policies (Zhang et al. 2021 Chen et al. 2021
Amidzadeh et al. 2021 Jiang et al. 2022 Chen et al. 2022 Zhou et al.) 2023). Chen et al. (2021) devise a multi-agent reinforcement learning for ultra-dense networks, whereas Zhang et al. (2021) employ a deep RL algorithm to jointly optimize resource allocation and caching for Internet-of-
scenarios. Amiizadeh ct al. p leverage a deep RL-bbaod approach to develop an optimal cache policy for multicast-onabled cellular networks. Moreover, Jiang et al. (2022) develop an actor-critic RL algorithm for proactive caching in mobile edge networks. Finally, Chen et al. 2022) and Zhou et al. (2023) employ deep RL for joint caching and offloading problems in edge computing networks. All the works mentioned above only consider forward dynamics, whereas this work entails a more complex characterization that allows to obtain better solution (see Section 5.2 for a detailed account).
3 Multi-Objective FB-MDPs
This section briefly describes multi-objective optimization and its associated Pareto-optimality as a basis to formally define FB-MDPs. The section concludes by characterizing the optimal solution of a multi-objective FB-MDP problem.
3.1 Pareto Optimality
Consider the following multi-objective optimization problem:
Q
min
S
where fy RR - R, X' is the feasible set and the number of objectives.
5

---

Published in Transactions on Machine Learning Research (07/2025)
Definition 3.1. We say that y X Pareto-dominates E X, if f(()) & f(()) for all i e (1....,,r} and there exists E (1,....r) r) such that (y( < fz()..
Definition 3.2. x" E is called a Pareto-optimal solution of Q1, if there no other solution y X that dom- inates x". Accordingly, [fi(=").... fr(x")) is called Parcto-optimal vector, and mingex [fi (z) ....................... (x))) indicates to the set of Pareto-optimal solutions.
The following lemma (Schäffler et al. 2002 Ma et al. 2020 is instrumental to jointly minimize all objectives
of Q
Lemma 3.3. Consider vector-valued mallivariate function f = [fiy-.-, frh, J3 R R for E fI,...,,)) r).
Let q(-) - a; V.)),, then -q(-) is descent direction for all functions {fi()),, where {azli are the solutions of the following optimization problem:
Q2
min {a:]
Σ as aN
aj =1 j=1
Σ
oj 20, jE rl, =
Remark 3.4. The lemma above can be leveraged to develop a multi-objective gradient descent algorithm. To jointly decrease different objectives, it sulfices to oplimize [[ay]]=]] by using the quadratic program Q2 and obtain q(-), which is nonlinear as itself depends on {vf()))
Accordingly, the optimal solution of problem Q2 can be obtained as follows.
Corollary 3.5. I/ Vf(() is invertible and all Oj 2 0, the solution of Q2 is given by:
a = (I. (VF() ".) VFO)) 1..
(1)
where f-()) is an matrix with Vf(() V V)r For the case aj <0 for E So rh....,r.. we set ff( = f+)l e(I....))\S.
3.2 Forward-Backward Markov Decision Processes
We introduce a class of multi-objective FB-MDPs, expressod by a tuple (S.),A, P(), ()),)))) where: S and Y the forward and backward stale-spaces, respectively; is the action space; py: S A [o, 1] is the forward transition probability, which describes the forward dynamics; Ps: ) A backward transition probability, which expressos the backward dynamics; finally, r S x A- Sy and r ) x A + [S are the forward and backward reward functions (respectively), where S₁ and S are the sets of indices of the forward and backward rewards (respectively). The forward transition probability determines the next forward state of the system St+1 ~ (- az) starting from S; E and performing the action ag E A. Moreover, in an anti-causal way, the previous backward state of the system follows Y:-1 ~ Po at) from Y, y by performing action a, A. The initial forward state $ and the final backward state yr assumed to be known. Figure la page illustrates a FB-MDP.
(0, is the
Sy
Assumption. This work constrains the definition of FB-MDPs to the case where the forward (backward) dynamics does not depend on the backward (forward) state.
Remark 3.6 (FB-MDPs cannot be expressed as standard MDPs). The backward dynamics cannot be represented based on standard forward system in presence of forward dynamics. We can consider the transformations ZT-1 Yt and t' T - t to convert the backward MDP with transition probability Y:-1 P,(-ly,,,,) a.) into forward one. Consequently, we get forward MDP over Ze with transition probability z ~ ). However, this is a non-standard MDP as the state 2; becomes a function of actions that are scheduled for future time steps ar- Specifically, the state relics on future actions that are not. available when progressing forward in time. This violation of the conventional causal structure prevents the use of standard RL algorithms.

---

Published in Transactions on Machine Learning Research (07 (07/2025)
The aim of FB-MDP problem is thus to optimize the following discounted multi-objective cumulative reward from the Pareto-optimality perspective:
T Σ x-1
(s), r'r ++1 ar -
max
(2)
(P.E.)
In Equation 2
T N is the finite horizon of the optimization, Y E [0. 1] the discount factor, and the expectation relers to the different realizations of the forward-backward trajectory.
Remark 3.6 highlights that solving FB-MDP problem of the type in Equation (2) requires developing novel theoretical foundations. To do so, we build on the following observation and the resulting optimal solutions.
Remark 3.7. Both the forward and backward dynamics of a FB-MDP problem can be accurately learned through a 0-parametric stochastic policy a, ~ (-(ss))), whereas employing the policy a, (- is unfeasible due to the anti-causal nature of the backward dynamics. Therefore, need to optimize the policy (-(- based the trajectories of both forward and backward dynamics.
The following section delves into this process.
3.3 Characterizing an Optimal Solution
We now analyze Remark 3.7 and provide a theoretical framework to characterize the optimal solution of a multi-objective FR-MDP problem. Accordingly, the probability of a forward-backwarr trajectory t is determined by:
Po(T) PPS), al,.., ST, aT, Y1....Y1)
T-1
T-1
T-1 II
P(ss))
II
Pf (s44 St ar) To (a((a))))
II
(3)
1=1
The problem in Equation
2 is then reformulated the following policy distribution optimization:
Σ Σx (st, r" r' (VT- aT at ( )
O2: :
max BrwPa(T)
s.t. (411
Vt-1 Po(-yc.. a.),
a ~ mo(
(4)
The multivariate objective of O2 can thus be expressed as:
T Σ k 1
T Σ
J(0)= = Em-Pa(T)
(Sk. ax),
ET-Po(T)
g - (y -+1 ar -+1)
k=1
k=1
((
(
To Pareto optimize J(O), we need to first compute its component-wist gradient with respect to 0, ie.,
()(0) PP(r)) For the forward cumulative rewards J (0), have (Grondman al. 2012): voJ(O)= 3Po(T)
VoJ'(0) = E Vo log (s) (sk, ax) 1
55)
where: : - RiSH A ax) = rJ (St. (st + 7V (sx+1)) V5())) the forward advantage
multivariate function; and V S 1g with
T
(sx) = EP((T)
Σ k=
-ky 5 2s)(s2
(6)

---

Published in Transactions on Machine Learning Research (07 (07/2025)
is the forward state-value multivariate function. We finally obtain the following lemma to characterize the optimal backward trajectories and the Pareto-optimal solutions of FR-MDP problem O2-
Lemma 3.8. For the backward cumvulative reward J (0), it is:
(o) =PPo(T) Vo log ( or 8T A 8T or
(7)
where A y S A 4 S is the bidirectional advantage multivariate function:
A (3 1/--++ kr or -) - (yT- k. ar-k) + yV yr 4-1: 8y-k-1)
8T-
and vo: : x S Rl is the bidirectional state-value multivariate function:
T , (yke, ax Dr 87 }
T-1 Σ
V (1T ST 2) = Σ
p (Br- ar w) 88 ST
==
k'=
(8)
which adheres to the backward Bellman's equation:
(yT ST
r (yT (YT- ab) - 11 ST
(9)
- - 1--
Polelur k. oT
1)
For the stalionary forward and backward transition probabililies, Bellman Parelo-oplimality equalion is given by:
E a) +vV0 (y ( ]. (10)
[V'( (s), bb (y, max
{ fr xt ay
(st)]
E
r Pgt-lx P((1))
for (s, a) S A, where [VF 4b (y. s)] Pareto-optimal vector, st E 8 is the forward state following 8, and y E is the backward state preceding y.
Remark 3.9. The formulation Lemma 3.8 differs from its counterpart for forward MDPs. Specifically, defined in Equation (9) so as have a backward Bellman's 's equation. Note that Equation (9) exhibits forward dynamics with a dependency on the policy distribution that itself relies the forward state rather than the backward state. Moreover, the Bellman's Pareto- optimality equation li.e., Equation (10b) characterizes
the bidirectional stat-value (y ST-x) 2)
optimal solution for FB-MDPs, which notably exhibits a bidirectional optimality dynamics, due to presence of s and s on the right-ham side. This requires that both dynamics should be jointly and simultaneously considered to obtain an optimal policy. We leverage these findings in devising our algorithm next.
4 Barkwrd- Kulti-Objectivv RL
We now build upon the results in the previous section to develop an RL algorithm for multi-objective FB-MDP problems. Specifically,
devise Forward- Backurdd Step- Wise (FB-SW) mechanism according to Remark 3.7 and Lemma 3.8 The mechanism comprises three steps: (i) forward evaluation, in which the forward dynamics is evaluated by generating actions using the policy a (=(+));); (ii) backward cvaluation, in which the backward dynamics is evaluated in a time-reversed way by leveraging the actions generated in the previous step; and (iii) bidirectional learning, employing a multi-objective optimization mechanism with a suitable chronological order optimize the policy (-(s)s;) based on the experiences obtained from both the forward and backward dynamics. In the next sections, we utilize this general mechanism as an adaptable framework to devise RT algorithm suited to FR-MDP problems.
8

---

Published in Transactions on Machine Learning Research (07 2025)
Figures
1b and Ic
(on page 2
outline such an algorithm.
4.1 The -arkwad Algorithm
According to Equations 5 and 77. the gradient of J(0) depends on the policy distribution ne(-I-) in addition to the stale-value functions (.()) and (:,-). For the policy distribution ma(-))), we consider an actor agent represented by O o-parametric neural network (NN). For the forward state-value function (f),,, we set a forward-critic network representod by a Φ-parametric NN, denoted by (y( Morcover, we use a backward-crilic nelwork with vt-paramelric NN for the bidirectional state-value function, indicated by (.-). We must now align the evaluation and update procedures for the actor and critic networks with the FB-SW mechanism. In this regard, (e(-)-) and (-) are evaluated during the forward-evaluate FB-SW mechanism, (t.))) is evaluated during the backward-evvalattee step,, then their values are employed to compute (o) and update (-(-))) during the forward-backward optimization step.
step of the
The update mechanism of actor policy (-(-))) depends on the forward and bidirectional state-value functions, i.e, (.) and (.(-)). As consequence, need to set some losses to also update these state-value functions. In line with Bellman's equation V (sx) = Es, 1s. fr (Sk, ax) + (sx++)) and Temporal Difference
(TD) llearning (Grondman et al. 2012), the following forward critic losses are considered to update Φ:
Al to, (Sk, for i i Sy,
(11)
where Ab., (Sk. a&) = ( ax) - (5x+1) are parametric representations of the so-called forward advantage ions. Conversely, we set the following buckward-crilfe losses to update the parameter sb based on the derived backward Bellman's equation [i.e., Equation [
Σ k=0
aT for i Sp. ST
(12)
where (T = Vb (y ST ST -r (VT aT - (1 are the para-
metric bidirectional advantage functions.
Equations 5 and ( indicate multiple directions for optimizing the actor, while Equations and 12
show multiple losses for optimizing the forward / bidirectional critic networks. A straightforward approach to carry out multi-objective optimization involves using the scalarization technique, namely, obtaining a single- objective loss through preference function (or scales) for different rewards. However, Pareto solutions cannot. be necessarily obtained via this method (Kirlik & Sayın| 2014). As consequence, tuning the scalarization setlings might require trial-and-error approach, which is sensitive to the selected selup. Instead, we use a icale-insensitive multi-objective optimization method (Schäffler et al., 2002 Amidzadeh, (2023) to devise a forward-backward RL algorithm. Accordingly, we employ Lemma 3.3 to formulate forward bidirectional critic networks and a multi-objective actor agent shared between the forward and bidirectional critics.
4.1.1 Forward Bidirectional Critic Networks
Equation |Equation
provides multiple losses for the forward (backward) critic nerw By recalling Lemma 3.3 we formulate the multi-objectivve loss (x)) [K*( by using the coefficients By (9b), so that a common descent direction is formulated for all forward (backward) critic losses. Accordingly, we have:
12
K'(Ø) = Σ
E (s)
k² = Σ
8
S 8T aT
(13)
JES;
k=
JES
k=0
where B3 and 07 are tuned by the following problems (see Q2 of Lemma 3.3
9

---

Published in Transactions on Machine Learning Research (07 (025)
T Σ
T-1
87= argmin " 20
Σ JESy
B3 V
A
Sk.
ax
)2
)
g argmin " 20 Σ, E,e,, =y=
Σ
33 V
Σ
1 the
(Y) 8T ar
Eyes, Pa=1"
k=1
JESb
k=0
(14)
These critic networks are then updated via TD-lcarning with the following Stochastic Gradient Descent (SGD) rules (Grondman et al. 2012):
Φ us (K))
サ pm V
(15)
where By and Mb are the learning rales of the forward and bidirectional critic networks, respectively,
4.1.2 Actor Agent
We follow the same strategy as in the previous section devise single-policy multi-objective actor agent shared between the forward and backward processes. The following forward and backward gradicnts follow from Equations and
voJ/
V (a an).
Vo (ar (5
(16)
for E Sy and E S.. We then employ Lemma 3.3 to provide simultancous ascent direction for all forward / backward rewards. Note that, in contrast to the critic network, an ascent direction is desired as the actor maximizes the rewards. Hence, the multi-objective actor agent is updated by the following multi-objective SGD:
0 0-
Σ Prct, E voJ/(0.0) JES,
Bnet, Bact,) Vo v)), JES.
(17)
where μ is the learning rate of the actor agent, and
Bact argmin
Σ
Bg VoJ
Σ
8, B/VoJ
(3:);
JES,
JES,
s.t.
Bg 2 0,
Σ jESyuS,
By =1
(18)
with
T
(0) = E.E. (o)
Vo log (a(ass)))1 (sk, a
0, ] = -VV
(0 E
Ve log
-)) bor -T 0 (o.0))(0.))00 (19)
for j e S, and i Sy. Note that, as opposed to the critic losses, we theoretically leverage the expected gradients (o and ()) to optimize Bact in Equation (18) [compare with Equation (14)). This approach interestingly ensures that all forward and backward cunulative rewards namely, {Jí (0) and (J)) monotonically increase each iteration and, more importantly, facilitates the convergence of the FB-MOAC algorithm (please refer to Theorem 4.3 in the appendix for more details).
estimate the expected gradients (0) and (o)) we employ Monte Carlo Sampling (MCS) together with eaponential moving average, applied to Vo (0, Φ) and Jo (0, 1b). Specifically, we first implement.
10

---

end for Published in Transactions on Machine Learning Research (07/2025)
Algorithm 1 Pseudo-code of the sawwarddBcckward Multi-Objective Nctor-Crithe (FB-MOAC) algorithm.
1: for episode to Emax do
forward-backward 2: Input: Initrial forward- backward state (s1,
Actor, forward-critic and backward-critic parameters: 0, and
4: Step 1: Forward Evaluation:
5:
for 1 do
6:
Select =o(--s)), interact with environment
7:
Observe forward state 8141 and forward rewards {r/ (st, ad}es
8:
Compute (A4,, at),asss by forward state-value (Vo, (st) (st)ives,, Equation 11 Compute log (ar(s)))
9:
10:
end for
11:
Step 2: Backward Evaluation:
12:
for 1= to T do
13:
Observe backward state Yr-c and backward rewards ar--)lsss depending on the action drawn
in step
14:
Compute {Aa., (yr- ST ar -hes, by bidirectional state-value (y Sr t)sss, Equation 12
15:
end for
16:
Step 3: Bidirectional Learning:
17:
Forward / bidirectional Critic Update:
18:
Obtain 87 and S, by Equation (14)
19:
Compute multi-objective forward-critic loss ('(Ø) and backward-critic loss (*
20:
Apply the rules:
- -
+ w - μb V v (bb
21:
Neward-Backward Actor Update:
22:
Obtain 8 through Equation (T) and the outcomes episodic MCS-average average
Compute stochastic forward and backward gradients VoJ. (0, Φ) and Vo v/)) through Equation 16
24:
Apply the SGD rule:
- 0 - 11 Σ
Bucr (0, + Σ Bosts VoJ
JES/
25:
N distinct backward and forward critic networks with learnable paramcters (1))) respectively, and use the approximations
Nues E (.)00(0..))))
Nacs and {
MMCS
NMss
Vo))
E
VoJ.
NMCS
NMCS
addition, we consider different episodes to take an exponential average with a smoothing factor Ymov to estimate E { 01)0) and E (. vb)10). We name this approach episodic MCS-average.
Figure Ib
overviews the proposed forward-Backward Multi-Objective Actor-Crite (FB-MOAC) algorithm, whereas A Agorithm provides its pseudo-code.
4.2 Convergence Analysis
In this section, we analytically characterize the convergence of the FB-MOAC algorithm. Our investigation starts by establishing some foundational assumptions, followed by the presentation of important theorems and corollaries. We then carry out the convergence analysis for the scenario where the expected rewards are Lipschitz-smooth (please refer to Appendix for more details).
We emphasize that stochastic nature of FB-MDP affects the values of Φ, ab and 0, based on the SGD rules [Equations 115 and [17]. they treated as random variables. We make the following assumptions.
11

---

Published in Transactions on Machine Learning Research (07/2025)
Assumption 1: The estimations of state-value functions are unbiased up to the residual terms, ie.,
((V).(()) s, s, = (s((s) + o B[V ( (y + +6J.
i EESp, SES Sb. (y.s)
where {6/ hes, and {6)) JeS, are the forward and backward residuals arising from the approximation of the true value functions V. (s) y) by the neural network parameterization and the stochastic gradient updates 15
Assumption 2: The forward and backward expected rewards (J)(0), (o)) are Lipschitz-smooth functions with constants b and La: respectively, w.r.t.
(oJ](0)-VoJ((0)) VoJs L 011, jES, |[o))(0) -Vo,(10) Lo|l0' 011, J E Sb.
Assumption 2 can be mapped to set of assumptions related to the architecture of neural networks.
Proposition 4.1. Let the actor represented by a P-parametric neural network, where all activation functions are Lipschitz- continuous, Lipschitz-smooke and bounded both above and below. Moreover, assume that either the action space is compact or actions sampled from the policy distribution ae(-I-) are clipped. Assumplion 2 holds for any family of distributions that bounded whenever their porameters and input are bounded.
Proof. Please refer Proposition B.2 in Appendix B
Assumption 3: The stochastic forward backward gradient:
(B [o. (0, ()) ,es, (0)) ]
has a conditional covariance bounded by a positive semi-definite matrix B:
E i
(0. )b) vJ c. vb) 0) ()) VJ (
where (0) [ (()) y ( Jes.
Note that the assumptions outlined in this context align with the those adopted in the literature on convergence
analysis (Tian et al. 2023 Xiong et al. 2022 Zhou et al. 2022 Qiu et al. 2021
We now nced to present definition for the convergence locally Parcto-optimal solutions.
Definition 4.2. The parameter sequence (0.) said to converge locally Pareto-optimal solutions
Zhou et al. 2022 if
- VoJf ||] +0, 0,
lim 10000
E
min B3 20
Σ
jE(SyUSal
where (O) the j-th element of ( with v.J [Vo jesy, (V) jes.
We ready to present the main theorem and the resulting consequences related to convergence.
12

---

Published in Transactions on Machine Learning Research (07/2025)
Theorem 4.3. Assume the Jorward backward stale-value estimalions, iee,, (V ( Hes, and {V (i Obess following Assumplion 1. Moreover, consider the sorward / backworr expected rewards, i.e., {J OJess and (ess,,, and the forward bbckwarrd stochastic rewards, i.e., { - and (3)( Des. complying with Assumptions and 3, and Pact being the solution of Equation (18). Finally, consider the SGD as Equations 144 and 17 characterized by iteration number and actor learning rate {u: with
U4 S min { [na Lo)' max
(V) 11) }
and 0 < ... 14 ... S Mi, which generate the sequences get:
( {sb')/_ and 10' Then, we
B J ( (0') 12c S max L,g, Lbb
B|
F4i 2 μ4 max L1 Lo}
i=1
2
Σ
[JJ - Jf
(20)
I I [S U U.
jES,US,
Proof (sketch). It can be shown that the episodic MCS-average approach ensures that the forward and backward expected rewards {J} (0) and (O)ees))) constantly increase each iteration based on Corollary 3.5 (scc Theorem B.3 We leverage this fact and the characteristic features of Lipschitz-smooth rewards to guarantee the convergence to a locally Pareto-optimal solution. Appendix B.2 provides a complete proof.
Remark 4.4 (Sublinear convergence). Theorem 4.3 implies the convergence to a locally Pareto- optimal solution (Zhou et al., 2022) with convergence rate of O(1/VI) under the learning-rate scheduling 14 OL/VV),, where I is the number of algorithm iterations equivalently, the number of policy updates in FOAM This rate is notably consistent with that of singlo-objective actor-critic methods for forward-MDPP which exhibit. a convergence rate of (11/V)) (Fu et al. 2021
Remark 4.5 (Complexity). The overall architectural complexity of the FB-MOAC algorithm is comparable that of standard actor-critic approaches. However, the episodic MCS-average add-on makes FB-MOAC computationally different from standard algorithms. The computational burden introduced by this add-on depends on the number of critic agents. We have empirically determined that as few as three agents is sufficient to achieve desirable performance (see Section 5 for more details). Given that the convergence rate of FB-MOAC algorithm is comparable to the standard RL algorithms, the computational complexity required to achieve convergence is of the same order as that of standard forward-only methods.
4.3 Deriving the Pareto-Front
Our FB-MOAC algorithm is designed as a multi-objective framework where single-policy agent interacts with multiple reward-specific critic networks. Crucially, these critics are updated through non-linear mochanism with respect to the reward functions, as described by Remark 3.4 in addition to Equations (13 and [1p.. Hence, we consider the critics for developing preference policy with respect to different rewards. To systematically explore the Pareto front,, introduce forward and backward preference parameters, ef e 11)),,, e" 111s,,, which are used to se-scale the corresponding advantage functions:
(St, as) = (SA, ax) + >V/ (SM+1) - v (Sw), for i i Sz - -k) = =c (y ar VV (Y 8/ (aa ST for j Sp. (21)
We then apply the FB-MOAC algorithm with the updated advantage functions. Note that this re-scaling does not lead to linear preference due to Remark 3.4 We thus employ different preference parameters steer the learning process toward different regions of the Pareto front. It is also important to highlight that.
13

---

Published in Transactions on Machine Learning Research (07 (025))
the scalarization technique camnot be applied on the forward and backward rewards to formulate single reward, since the resulting reward would depend on both the forward and backward states. As result, a state-coupled FB-MDP would occur and Lemma 3.8 would no longer hold. This further motivates using multi-objective optimization to find the Parcto-optimal solutions.
The non-linear re-scaling mechanism described above allows to characterize the Pareto-front of a problem. Note that Theorem 4.3 guarantees convergence to (locally) Pareto-optimal solutions. On the other hand, Theorem B.3 ensures that the expected rewards monotonically increase for any preference policy. Consequently, the convergence to (locally) Pareto-optimal solution is preserved regardless of the different preferences. These considerations explain that the mechanism in Equation (21) allows to derive locally Parcto-optimal solutions for the case of Lipschilz-smooth rewards. An evaluation of the proposed mechanism is provided in Appendix
5 Evaluation
FB-MDPs find application stochastic optimal control problems driven by forward-backwarrd stochastic differential equations (FR-SDEs) and networked systems (Zabihi ct al. 2023). Accordingly, we evaluate FB- MOAC against the state of the art through diverse representative problems in these domains: mathematical finance, example of how FB-SDE-driven stochastic control problem can be solved by FB-MOAC; and cacho-assisted content delivery in wireless networks. Appendix D provides additional use case in the context of computation offloading through an edge (cloud) server. The code of FB-MOAC is available at:
https://githinn com/ am dimohsen//MMMOAC
5.1 Use Case: Mathematical Finance
We consider an investmenn-coonuumiioon problem (Ma & Yong 1999: El Karoui 1997.. In particular, we
address a stochastic optimal control problem driven by a Norward-backwant stochastic differential equation (FB-SDE), which then discretized according to the method in Appendix to find an optimal solution with the FB-MOAC algorithm.
5.1.1 System Model
A financial market consists of n risky assets whose prices follow the following F-SDEs:
dt.(()) (t) (o dB())) Pr (0) > 0,
for n E NJ), where (t) E N is the Wiener process with an identity diffusson matrix, rtP is the instantancous appreciation rate, and yoo E RN is the asset volatility. A trader invests in risky assets by fractional investments (0 &i (4) S 1) or borrow / lend money with an interest rate (I). Hence, the wealth w(t) of the trader with consumption plan c(t) can be obtained by F-SDE:
dw(()= = (ox())))))
(6,(0))dd + Σ w(()) (0)00((00xxx)))38))))
(22)
where pdrf is the respective drift function obtained as:
(w(()) (() = , ,n (t)w(t) Σ w(()) (,apP(t) rit c(t))
with w0) = wo and wo the initial wealth. Note that (t)(t) is called the investment portfolio, with ΣN (t) 1.
An utility process u(t) of the investor then taken into account. This process at time t depends on the consumption plan e(t) and the future utility, and is described by the following backward SDE (B-SDE):
du(() = =-e
(c(t), u(t), =-fsen(c(t) dt)) + (a(t),d(())
(f)==f"n((((()
(23)
14

---

Published in Transactions on Machine Learning Research (07/202) 2025)
T+10
1-0.78
Oplimum Value of ofu(0)
0.10
0.05
0.00
2000
4000
8000
8:00
10000
12000
14000
Time Steps
(a)
Risky assets NV
FB-MOAC
Approach in Ji et al.
(202bb
2020
T=0.50
7=0.75
T=1.00
T=0.50
T=00.55
110.0
10
0.121
0.182
0.241
0.122
0.182
0.242
20
0.122
0.181
0.240
0.122
0.178
0.242
50
0.120
0.180
0.235
0.181
0.237
(b)
Figure 2: Evaluation in stochastic optimal control problem: optimal investor utility u(0) for N 50 risky assets over different horizons and (b) comparison against the state of the art.
where (.) is the generator function, z E R is the control process of the backward dynamics, T is the finite horizon, and (-) is the final utility function. The objective of this problem is to optimize the initial backward state u(0) by designing an optimal portfolio and consumption plan. This is formulated based on the following stochastic optimal control problem:
E (w) + /-' fren
max (o)(t)) a o(t)
s.L.
FB-SDE 22 and 23
(24)
To apply FB-MOAC, discretize this stochastic optimal control problem with the Fuler-Maruyama scheme (Kloeden & Platen 1992) (see Appendix F for more details). The forward state is set to the wealth w(t) (i.c., s(t) = w(t),, while the backward state to the utility process u(t) [i.e., y(t) = u(t)]; the action is on the investment, portsolio (1) the control process of the backward dynamics, and the consumption plan c(t) 2 0, i.e., at)) = (t) i=1 c(t), a(t)]; finally, the cumulative backward reward the initial backward state u(O) expressed by {
(f (w(T)) + fod facm (c(t), a(t), 2(t))dt). Note that the discretization of the B-SDE in Equation (24) leads to a Bockward-MDD due to the backward flow of action information (see Remark 3.6). This motivates the use the of FB-MOAC algorithm. We thus partition the time interval [0, into
T N sub-intervals [4x-1, 4x) for & E (1....,NN N), each sub-interval with length A/- = ,æक, where 4a = 0 and tyas T. By applying this discretization, the backward cumulative reward leads to:
Addia
(a 3) F +
Σ pgen ((Gi))) At)
(25)
where a [a( and (v(ta) yolls forward MDP:
k= Further, the F-SDE in Equation 22 maps to the following
w(tx+1) w(tx) = (d"(((((tt) (o.
Σ
(ttx))n vol (B2)
(26)
15

---

Published in Transactions on Machine Learning Research (07/2025)
where (f((s)) is a normal random variable with zero mean and variance At, and the B-SDE in Equation maps to the following backward MDP:
23
u(fx) u(tx+1)+ f (c(te+t))))) u(te+1),, ((4+1)) )(t ((a))))))))
(27)
The respective sequential docision-making problem is then expressed as follows:
max Ra y).
s.b. {a(l)) ~ mo(-)((())), FB-MDP of Equations (26)
(27
5.1.2 Experiment Setup and Hyper-parameter
We use the same scttings as those in (Ji al. 2022a). The number of assets N {10, 20, 50). the generator function (gen(c(()) u (()) -0.05m(1) c(() -c(1), the final utility function f exp(-xx) the interest rate pint (t) = 0.03, the appreciation rate (t) 0.05, the volatility a = 0.1I, for (1, ...,,.. N), the finite horizon T E {0.5, 0.75, 1.0), and the initial wealth x6 100.
As this problem only entails a backward reward, we only establish the backward- -ritic network; moreover, we set NMCS = 1, the number of neurons in the hidden layer for the actor and crities to 8, the actor and bidirectional critic learning rates 2 x 10-2, and the smotthing factor "movv = 1. We use the Dirichlet. distribution for {t)( to jointly motivate the exploration and satisfy Ex @n 1. Finally, the rectified linear unit (ReLU) activation function used for the neuron connections, the number of neurons in the hidden layer for the actor and critics is 100, the actor and forward / bidirectional critic learning rates are 3 x 10-1, and the smoothing factor is "/mov 0.95.
5.1.3 Performance Evaluation
Figure 2a shows the performance of FB-MOAC as function of time steps for different values of finite horizon T E {0.5, 0.75, 1.0}. For comparison purposes, consider the approaches in (Ji et al., 2022b 2020 which develop deep learning methods by focusing stochastic control theory and incorporate the system dynamics priori for the optimization. In contrast, FB-MOAC learns multivariate rewards for FB-MDPs without knowing the transition probability of the underlying dynamics. Table 26 compares FB-MOAC against state of the art in terms of the optimal initial investor utility v()). The solution by FB-MOAC is close to the values obtained by (Ji et al., 2022b 2020) for different values of T despite treating the system dynamics as a black box during the learning process. This demonstrates the ability of the proposod algorithm find an optimal solution for environments characterized FB-MDPs, thereby broadening its application to a variety of stochastic optimal control problems described by FB-SDES
5.2
Case Study: Edge Caching
We now consider a real-world forward-backward multi-task problem in the context of edge caching Nomikos For conciseness, the rest of the section omits details that can be found in Appendix
et al.
2022
5.2.1 System Model
The environment of this experiment. a wireless network with cache-equipped Baso-Stations (BSs) The environment also includes a library containing N different content items as well fixed mobile users requesting these items from the cellular network. The network operates over time slots with a discrete index E (I, where T' is the total duration of the operation. The network thus addresses that user requests at the beginning of cach time slot. Content items have different popularity {t)" where pPoP(t) is the probability that content n is requested by a randomily selected user al time L. The goal is to satisly as many users as possible during the network operation. At the beginning of each time-slot, the BSs cache the most popular content items with probability (1gach(t) (t)) and simultancously multicast them toward users by consuming content-speeific radio resources {w, (t)) The transmission at time-slot completes within d(t) seconds. We thus denote the system action parameters by the vector a(t), which depends on
16

---

Published in Transactions on Machine Learning Research (07/2025)
the content-specific bandwidth allocation and cache placement at BSs, i..., a(I) [(v (4) (() A multicast. outage may occur with probability (On(a(t), ()) As result, certain users fail to receive the requested content in a given time slot and their request is deferred to the subsequent one. Hence, each time-slot sees a distribution of users accounting for the repeated requests and a distribution describing the new preferences loward content items. This leads to a time-varying model for the request probability pfaa()) of content n:
N Σ 0m m=1
Pr (1) = (1 (1 1)00,, 11,a(-1),,00 1),4 + PRO
(0)
- (a(())))-)) 1).
(28)
repeated request
request based the popularity
Note that (t) indicates the request probability of content n averaged over all users. Then, can be simply verified that E (t) = 1, considcring Σ (t) = 1. Equation (28) therefore rep- resents a forward dynamics, with the forward state vector s(t) = (t) and the action vector
a(l) = (2 (
A request for a content item is repeated across several time-slots until successfully fulfilled, resulting in an expected lateney (t( for successful delivery of content n. For this quantity, a time-varying dynamics can be derived by the law of total expectation as follows:
I (a()),) = (a(t)) + I, (a(t + +), 11).++1))0,0000)0) d(1) 2 (1 (1-000((()))) (n(a()))), La.a(T), T) = 0, (29)
where d(t) is the duration of time-slot t in seconds, and we have L, (a()),, =0 since system operations finish at time t = ' and the users do not need to wait any longer. Equation (29 represents backward dynamics, with the backward state vector y()) = ()) and the action vector a()) Note that this model fully captures the trade-offs involved in the delay dynamics and differs from the conventional formulation that does not provide a comprehensive model when accounting for successive slots; for the delivery without d(t)
outage, the expected latency simply becomes (n) = 2 as its realizations follow a uniform distribution with values between 0 and d(t).
Equation 29 may suggest that it is possible to convert it to a standard forward dynamics. For this purpose, we can consider a variable transformation (T- == Te, (a((), L) as well time transformation 1' == T t. We can then obtain the following forward dynamics on K,,(t))
d(T d('-t) (1-
(() = (d(T - (1) + Kn - (1) (')
for l' 2 1,
2
with K2((0)= 0. However, this shows non-standard MDP, the state (t'( depends on the far future of that cannot be revealed by moving forward in lime. This argument aligns with Remark 3.6
action
Equation 29 also shows that for a full-error transmission scheme (i.e., with the outage equal to one) Ln(t) = d(t))= + (t. + 1) holds, which mcans that the expected latency maximally accumulates as one goes backwards in time. This expected, as no successful receptions take place. Moreover, it worth ressing that minimizing the expected latency in Equation (2 enables optimally keep track of the precise time slot at which requests are finally fulfilled. Alternatively, one could track the service time of requests to prioritize those that have waited longer, or track for the failed/ suucceeedd content transmissions. However, these policies do not completely map to the tracking of overall latency, and oversimplify the problem. Consequently, they fail to account for the complex interactions within the system, leading to a sub-optimal solution. The evaluation in Section 5.2.3 empirically proves this. Therefore, the problem is instead modeled as a FB-MDP, coupling forward and backward dynamics through system actions, where the action space is [0, 1]" x [0, oo)" with n as the number of content items.
Three widely-used network performance metrics (Li et al., 2018b) are considered as reward functions to design an optimal policy: the quality of service r-os("); the total bandwidth consumption TBw(-); and the overall expected latency (-).()). The QoS determines the overall probability of unsatisfied UEs and is given by (!) = - (1)0,))) (()O),(((1)))) (a()),)), 0 (I) < 1, namely, the likelihood of UE request.
17

---

Published in Transactions on Machine Learning Research (07/202) 2025)
0.00
0.30
Thein Cunulative Rewards
-0.00
-0.5
0.10
-0.15
Pref
Pref: [0.3, 1.0)
rous
roos
FLat
FLat
2000
10030
15000
20000
25000
32000
35002
12020
19300
20000
23000
20000
20000
(b)
0.00
0.00
Train Curulaaive Rewarde
-3.05
-0.25
-0.10
-0.15
Pref
Pref: [1.0, 0.3,
ross
foss
FLat
flat
5000
1003
15000
zrere
25000
30000
30000
13800
19391
2000
2000
20000
2000
Steps
Time Steps
(d)
Figure 3: Pareto-optimal solutions in the edge caching use case for different settings of the preference parameters e') (a) [0.3, 1.0, 1.0]. (b) [0.3, 0.3, 1.0] (c) [0.3, 1.0, 0.3] and (d) [1.0, 1.0,0.3) Note that these solutions are Pareto-optimal as none of them dominates the others.
remaining unfulfilled during the multicast transmission at time-slot t. The total bandwidth consumption is (t) = - W (a(()),)) = = - (t), where W-) represents the total bandwidth consumption for the network. Finally, the overall expected latency (L) = - En' p (a))) with Ln,t) obtained from Equation (29 Note that these rewards compete with each other. For instance, reducing rew requires increasing W/ which, in turn, decreases the outage On- Furthermore, decrease in O, makes rQos grow but. reduces the latency I,, which, in turn, increases Plat-
Clearly, (t)s(t) and (tw(t) relate to the forward state, and constitute the forward bivariate reward function r (t) = (f), [t]( Instead, (t) relates to the backward state, and constitutes backward univariate reward function (t))) = (t). The respective sequential decision-making problem is then expressed as follows:
max E Σ 6-1 ( (t), P01
s.t. {ar~ ro(sss))) FB-MDP 28
29
5.2.2 Experiment Setup and paramttprs
We select the following parameters for the considered environment. The number of content items is set to N = 200, the spatial intensity of the BSs to Abs = 100 points / /km?, and the transmission rate to 1 Mbps. The total number of time slots is = 256. For the content popularity, we use time-varying Zipf distributions LLi
et al.
201 Sa).
As for FB-MOAC, three separate sets of NNs representing the multi-objective actor in addition the forward-critie and the backward-critic networks. We use NMCs many NNs for the forward critic as well for the backward critic. The forward critic outputs two values representing the reward-specific state-value
18

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
Train Cumulative Rewards
-2000
-4000
5000
Toas. FB-MOAC
Fous, F-PPO
rgss. F-MOAC
Faw. FB-MOAC
Taw. F-PPO
rew. F-MOAC
FB-MOAC
Puat. F-PPO
FLac. F-MOAC
-8000
2500
5000
7800
10000
12500
15000
17500
20000
Time
(a)
Fgos
Faw
Flat
0.0
-0.05
Foos
raw
-0.1
-0.10
Cumulativ Ravard
0.20
-0.25
F-MOAC
LFU
-0.85
250
500
750
1000
1250
1800
1750
2000
Time Steps
(b)
(c)
Figure Performance FB-MOAC for edge caching use case: learning with F-PPO, F-MOAC, and LFU; [b] comparison between the test solution of FB-MOAC with those of F-PPO, F-MOAC, and LFU;
performance comparison against forward
(c) learning in absence backward optimization.
functions Volg()) related to r-)s(-) and rBw(.). On the other hand, the backward critic outputs one value representing the state-value functions V3((), related to (.ae(.). We set the actor and critic learning rates to 10-4 and the discount factor to 0.92.
To leverage the convergence result of Theorem problem so as to give Lipschitz-smooth expected rewards. For this purpose, we resort to Proposition 4.1 The actions {prach((), w,, (1) can be parameterized by a neural network with Sigmoid activation functions and can be sampled from LogNormal (for wa) and Dirichlet (for paac) distributions. They are also bounded, as Pa < 1 and Wn is clipped (though it is not initially bounded) to avoid a large latency and bandwidth consumption. Therefore, Proposition 4.1 can be considered for the neural network of this problem, and log (a)s) inherits Lipschitz-smoothness: in 0.
assess whether neural network can be used for this
5.2.3 Performance Evaluation
Figure 3 iilustraaes the learning results of FB-MOAC in deriving most of the Pareto-optimal solutions, i.e., the resulting solution each plot does not dominate the others. Recall that Section 4.3 describes a mechanism for obtaining Parcto-front; Appendix E further characterizes such front for the use case considered here. For clarity, the performance metrics are normalized based on the value of rqus, so that they can be clearly shown in single plot; more importantly, the value of rqos shows the average percentage of failed requests.
19

---

Published in Transactions on Machine Learning Research (07/2025)
As the results of forward and backward rewards eventually evolve into stable solution, the actor and the forward/ bidirectional crities are effectively learned.
We consider three baselines for comparison purposes: a widely-used rule-based approach for caching, the Least Frequently Used (LFU) strategy (Ahmed et al. 2013): and two learning- algorithms by replacing the backward reward with a related one (for fairness) so that the backward MDP can be safely removed. Specifically, we manage the time slot during which requests are served by optimizing d(t). We further leverage the fact that maximizing rqus reduces Fhat based on Equation (29 Hence, we consider roos and TBw forward rewards, replace the backward reward with optimizing d(t), then use the baseline proximal policy optimization (PPO) algorithm (Schulman et al. (017aa as well as multi-objective extension of the baseline advantage actor critic (A2C) algorithm (Grondman et al. 2012). We term the solutions obtained with these strategies as F-PPO and F-MOAC (respectively), since they are developed for forward MDPs. Figure 4a compares the training performance of FB-MOAC against the F-PPO and F-MOAC baselines terms of normalizod rewards, while Figure 4b shows the solutions of these algorithms during test. We select a solution for FB-MOAC among different ones by prioritizing rQus- Instead, we learn forward rewards and additionally optimize d(t) for the two baselines (F-PPO and F-MOAC) to achieve a FLat comparable to that of FB-MOAC. The results show that FB-MOAC remarkably outperforms both F-MOAC and F-PPO in all rewards. This means that the FB-MOAC strategy can fulfill the content requests considerably better than forward-only strategies. Specifically, more than 15% of the content items are lost due to the quality of service in both F-PPO and F-MOAC, whereas the failure rate of FB-MOAC is only 2%. Moreover, FB-MOAC gives a comparable or better policy than those obtained by F-PPO and F-MOAC. Finally, the solution of FB-MOAC Pareto-dominates those of F-PPO and F-MOAC. The LFU policy does not benefit from any preference scttings with respect to the rewards, even though it is better than FB-MOAC from the bandwidth consumption raw perspective; unfortunately, it is very unreliable because 45% of the requests fail.
These findings show that minimizing the failure of transmissions (i.e., minimizing the outage) leads to a sub-optimal solution for the overall latency, also highlighting the importance of explicitly incorporating the backward MDP instead of trying to remove it through adjustments to the backward rewards. The results also demonstrate the effectiveness of the proposed FB-MOAC algorithm in solving the considered FB-MDP problem.
5.2.4 Ablation Study
We now conduct an ablation study to assess the benefit of the backward evaluation optimization FB-MOAC. For this purpose, we first disable the backward evaluation of the algorithm and only consider the forward actor and critie updates. Figure 4c shows the resulting rewards as function of time steps, highlighting that Tlat does not improve over time. As consequence, the results demonstrate the necessity of the backward evaluation / optimization in FB-MOAC.
We further carry out another ablation study to evaluate the impact of the multi-objective procedure in Equations 17 to [19) on the performance. For this purpose, we replace the proposed multi-objective optimization with single-objective one through a linear scalarization technique. Specifically, we update the actor parameter with the following rule:
0- 0 p( Σ // x)+ Σ [3 JES;
()
jES.
Jes, where sf Hs// E [0.1, 1| and sf [st)ess. [0.1, 1]., are the scalarization settings. Figure shows the train performance of this approach for different scalarization settings [ |0.3, 0.3, 1.0]. [1.0, 1.0, 0.3] and [0.3, 0.3, Clearly, the single-objective mechanism fails to provide stable solutions, in contrast with the proposed multi-objective approach (sce also Figure 3
6 Conclusion and Limitations
We introduced the notion of howaad--ackwarrd Markov decision processes (FB-MDPs), a class of MDPs that cannot be expressed standard MDPs. We then obtained optimality condition for the solution
20

---

Published in Transactions on Machine Learning Research (07/202) 2025)
0.00
0.00
Train Cunulative Rewards
0.10
-2.10
-2.15
0.29
(1.0,
1000
0.29
(0.3, 1.0)
Faw
-0.99
-1.05
1000
2002
3000
5002
CoDO
7030
2000
4000
Time Steps
Time Steps
(b)
0.00
0.00
Train Cumulative Rewards
0.10
0.10
-2.15
-0.20
-2.20
-.28
(D.3, 0.0,
+8.25
(0.3,
fow
row
-0.55
-0.35
1000
2000
2000
530
1992 1730 Time Stops
1250
2000
Time Sleps
(d)
Figure 5: Train performance when single-objective optimization is used with different values of the scalarization settings is s : (a) [0.3, 0.3, (b) [1.0, 1.0, 0.3), (c) [0.3, 1.0, 0.3] and (d) [0.3, 1.0,
of FB-MDPs and devised a multi-objective RI. algorithm called FB-MOAC accordingly. We analytically characterized the optimality and convergence of FB-MOAC, then conducted an extensive evaluation in diverse use cases to demonstrate its effectiveness.
As limitation, our mechanism targeted FB-MDP problems wherein forward and backward dynamics are purely coupled within the action space. Addressing fully-coupled FB-MDPs is interesting direction for future work.
Acknowledgments
This work was partially funded by the Research Council of Finland under grants number 354322 and 357533.
References
Abbas Abdolmaleki, Sandy H. Huang, Leonard Hasenclever, Michael Neunert, H. Francis Song, Martina Zambelli, Murilo F. Martins, Nicolas Heess, Raia Hadsell, and Martin Riedmiller. A distributional view on multi-objective policy optimization. In International Conference on Machine Learning, 2020.
Mohamed Ahmed, Stefano Traverso, Paolo Giaccone, Emilio Leonardi, and Saverio Niccolini. Analyzing the performance of LRU caches under non-stationary traffic patterns. oprint arXiv 1301.4909, 2013.
Mohsen Amidzadeh. A scale-indepeddnt multi-objoctive reinforcement learning with convergence analysis. IEEE Conference on Decision and Control (CDC), pp. 1326 1333, 2023.
Mohsen Amidzadch, Hanan Al-Tous, Olav Tirkkonen, and Junshan Zhang. Joint cache placement and delivery design using reinforcement learning for cellular networks. In Vehicular Technology Conference, pp. 1-6, 2021.
21

---

Published in Transactions on Machine Learning Research (07/2025)
Mohsen Amidzadeh, Hanan Al-Tous, Giuseppe Caire, and Olav Tirkkonen. Caching in cellular networks based on multipoint multicast transmissions. IEEE Transactions on Wireless Communications, 22(4): 2393 2408, 2023.
Richard Archibald, Feng Bao, and Jiongmin Yong. A stochastic gradient descent approach for stochastic optimal control. East Asian Journal on Applied Mathematics, 10(4):635 658, 2020. ISSN 2079-7370.
Richard Archibald, Fong Bao, and Jiongmin Yong. A stochastic maximum principle approach for reinforcement learning with parameterized environment. Journal on Compulational Physics, 488:112238 2023.
Shuangwu Chen, Zhen Yao, Xinofong Jiang, Jian Yang, and Lajos Hanzo. Multi-agent deep reinforcement learning-based cooperative edge caching for ultra-dense next-generation networks. IEEE Transactions on Communications, 69 (0(4))2441224505) 2021. doi:
Xi Chen, Ali Ghadirzadeh, Mårten Björkman, and Patric Jensfelt. Meta-learning for multi-objective reinforcement learning. In REEE/RSJ International Conference on Intelligent Robots and Systems, pp. 977-983, 2019a.
Xianfu Chen, Honggang Zhang, Celimuge Wu, Shiwen Mao, Yusheng Ji, and Mehdi Bemnis. Optimized computation offloading performance in virtual edge computing systems via deep reinforcement learning. IEEE Interned of Things Journal, 6(3):4005 4018, 2019b.
Zhixiong Chen, Wenqiang Yi, Atm S. Alam, and Arumugam Nallanathan. Dynamic task software caching- assisted computation offlodiing for multi-access edge computing. IEEE Transactions on Communications, 70(10)66950 6965, 2022. doi: RLL.L09TTCOMM.222..222222
Ashley D Edwards, Laura Downs and James C Davidson. Forward-backward reinforcement learning. arXiv preprint arXie:1803. 2018
N. El Karoui. Backward stochastic differential equations in finance. Mathematical Finance, 7(1):1-31, 1997.
Zuyue Fu, Zhuoran Yang, and Zhaoran Wang. Single- timescale actor-crite iitorccritic provably finds globally optimal policy. In International Conference on Learning Representations, ICLR, 2021.
Anirudh Goyal, Philemon Brakel, William Fedus, Soumyye Singhal, Timothy P. Lillicrap, Sergey Levine, Hugo Larochelle, and Yoshua Bengio. Recall traces: Backtracking models for efficient reinforcement learning. International Conference on Learning Representations, ICLR. OpenReview.ned 2019.
Ivo Grondman, Lucian Busoniu, Gabriel A. D. Lopes, and Robert Babuska. A survey of actor-critic reinforcement learning: Standard and natural policy gradients. IEEE Trans. on Systems, Man, and Cybernetics, 42(6):129111300) 2012.
S. Hamadene and J. P. Lepeltier. Zero-sum stochastic differential games and backward equations. Syst. Control Lett., 24(4):259 263, March 1995. ISSN 0167-6911. doi: 00.016/0117--01111))000...
Max Jaderberg, Wojeiech M Czarnecki, Tain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. Hunan-level performance in first-person multiplayer games with population-based deep reinforcement learning. arXiv preprint (rXXv:1807.01250 2018.
S. Ji, S. Peng, Y. Peng, and X. Zhang. Three algorithms for solving dign-dimensional fully coupled fbsdes through deep learning. IEEE Intelligent Systems, 35(03):71 84, may 2020.
Shaolin Ji, Shige Peng, Ying Peng, and Xichuan Zhang. A deep learning method for solving stochastic optimal control problems driven by fully-coupled fbsdes. arXiv:2204.0571 05796 [math. OC), 2022a. URL
http:///axivv.org/bbbaa
Shaolin Ji, Shige Peng. Ying Peng, and Xichuan Zhang. Solving stochastic optimal control problem via stochastic maximum principle with deep learning method. Journal of Scientific Computing, (3)1):30, 2022b. doi: 10:1007//100155-02--197777
22

---

Published in Transactions on Machine Learning Research (07/2025)
Wei Jiang, Daquan Feng, Yao Sun, Gang Feng, Zhenzhong Wang, and Xiang-Gos Gen Xia. Proactive content caching based on actor-critic reinforcement. learning for mobile edge networks. TEEE Transaclions Cognitive Communications and Networking, 8(2):1239 1252, 2022, doi: 10.1109/TCCCN22222 20T.CC....221111999000
Sham M. Kakade. A natural policy gradient. In Advances in Newral Information Processing Systems, 2002.
Sajad Khodadadian, Prakirt Raj Jhunjhunwala, Sushil Mahavir Varma, and Siva Theja Maguluri. On linear and super-liness linear convergence of natural policy gradient algorithm. Systems & Control Letters, 164:105214, 2022.
Gokhan Kirlik and Serpil Saym. A new algorithm for generating all nondominated solutions of multiobjective discrete optimization problems. European Journal of Operalional Research, 232(3):479- 488, 2014.
Peter E Kloeden and Eckhard Platen. Numerical Solution of Stochastic Differential Equations. Springer, 1992.
Hang Lai, Jian Shen, Weinan Zhang, and Yong Yu. Bidirectional model-based policy optimization. In International Conference on Machine Learning, ICML, volume 119 of Proceedings of Machine Learning Research, pp. 5618-5627. PMLR, 2020.
K. Li, C. Yang, Z. Chen, and M. Tao. Optimization and analysis of probabilistic caching in n-tier hetorogencous networks. IEEE Trans. Wireless Commun.. 17(2))1283 1297, 2018a.
Liying Li, Guodong Zhao, and Rick S. Blum. A survey of caching techniques in cellular networks: Rescarch issues and challenges in content placement and delivery strategies. IEEE Communications Surveys and Tutorials, 2013):17 2018b.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. Tn International Conference on Learning Representations, ICLR, 2016.
Yi Liu, Huimin Yu, Shengli Xie, and Yan Zhang. Deep reinforcement learning for offloading and resource allocation in vehicle edge computing and networks. IEEE Transactions on Vehicular Technology, 68(11): 11158-11 111611 2019.
Jin Ma and Jiongmin Yong- Forward- Backward Stochastäc Differential Equations and their Applications, volume 1702 of Lecture Notes in Mathematics. Springer, 1999.
Pingchuan Ma, Tao Du, and Wojciech Matusik. Efficient continuous pareto exploration in multi-task learning. In International Conference on Machine Learning, ICML, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and 2013.
Martin Riedmiller. Playing atari with deep reinforcement learning. ar preprint
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540))5529 5333-,33, 2015.
Hossam Mossalam, Yannis M. Assael, Diederik M. Roijers, and Shimon Whiteson. Multi-objective deep reinforcement learning. preprint ArXiv 1610.02707, 2016.
Nikolaos Nomikos, Spyros Zoupanos, Themistoklis Charalambous, and Ioannis Krikidis. A survey on reinforcement learning-aided caching in heterogeneous mobile edge networks. IEEE Access, 10:4380-4413, 2022. doi: AAA109////SS..... /ttp:/CCCEES..2..1..1.11.
2022.3140719.
Jan Peters and Stefan Schaal. Natural actor-critic. Newrocomputing: An International Journal, 71(7-9): 1180-1190, 2008.
Shuang Qiu, Zhuoran Yang, Jieping Ye, and Zhaoran Wang. On finite-time convergence of actor-critic algorithm. IEEE Journal Selected Areas in Information Theory, 2(2):652 664, 2021.
23

---

Published in Transactions on Machine Learning Research (07/2025)
Lillian M Rigoli, Gaurav Patil, Hamish F Stening, Rachel W Kallen, and Michael J Richardson. Navigational behavior of humans and deep reinforcement learning agents. Frontiers in Psychology, 12:725932, 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347 2017a. URL htttpp///aaaxi..orggaa//////////
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017b.
Stefan Schäffler, Simon R. Schultz, and Klaus Weinzierl. Stochastic method for the solution of unconstrained vector optimization problems. Journal of Optimization Theory and Applications, 2002.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, 2018.
Haoxing Tian, Alex Olshevsky, and Yannis Paschalidis. Convergence of actor-critic with multi-layer neural networks. Advances in Neural Information Processing Systems, 2023.
Jianhao Wang, Wenzhe Li, Haozhe Jiang, Guangxiang Zhu, Siyuan Li, and Chongjie Zhang. Offline reinforcement learning with reverse model-based imagination. In Annual Conference on Neural Information Processing Systems, NewrIPS, pp. 19420-294333 2021.
Yifei Wei, F. Richard Yu, Mei Song, and Zhu Han. Joint optimization of caching, computing, and radio resources for fog-ennn engaena iot using natural actor critic deep reinforcement learning. IEEE Internet of Things Journal, 6(2):2061-2073, 2019. doi:
2018.
Huaqing. Xiong, Tengyu Xu, Lin Zhao, Yingbin Liang, and Wei Zhang. Deterministie policy gradicnt: Convergence analysis. In James Cussens and Kun Zhang (eds.), Proceedings of the Thirby-Eightn Conference on Uncertainty in Artificial Intelligence, volume 180 of Proceedings of Machine Learning Research, pp. 2159-2169. PMLR, 01-05 Aug 2022.
Tengyu Xu, Zhe Wang, and Yingbin Liang. Improving sample complexity bounds for (natural) actor-critic algorithms. Advances in Newrol Information Processing Systems, 33:4358 4369, 2020.
Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hswe Hsuan Yang. Diffusion models: A comprchensive survey of methods and applications. ACM Comput. Sure., 56(4), November 2023. ISSN 0360-0300. doi:
Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi-objective rein- forcement learning and policy adaptation. In International Conference on Neural Information Processing Systems, 2019.
Zhuoran Yang, Kaiqing Zhang, Mingyi Hong, and Tamer Başar. A finite sample analysis of the actor-critic algorithm. In IEEE conference on decision and control, CDC, pp. 2759-2764, 2018.
Jiongmin Yong. borward-backwaard stochastic differential oquations: Initiation, development and beyond. Numerical Algebra, Control and Optimization, 13(3):367 391, 2023. ISSN 2155-3289.
Zeinab Zabihi, Amir Masoud Eftckhari Moghadam, and Mohammad Hossein Rezvani. Reinforcement lcarning methods for computation offloading: A systematic review, ACM Comput. Sure., 56(1), aug 2023. ISSN 0360-0300. doi: 10.1145/3600 3603703. URL ftttpp:://do..or//11 10t..//0//...1///0000033 1144//66033303
Maryam Zare, Parham M Kebria, Abbas Khosravi, and Saeid Nahavandi. A survey of imitation learning: Algorithms, recent developments, and challenges. arXiv preprint arXi::2009.02. 02477, 2023.
Huixin Zhan and Yongcan Cao. Relationship explainable multi-objective optimization via vector value function based reinforcement learning, preprint ArXiv 1910.01919, 2019.
Fan Zhang, Guangjic Han, Li Liu, Miguel Martinez-Garcia, and Yan Peng. Joint optimization of cooperative edge caching and radio resource allocation in 5G-enabled massive IoT networks. IEEE Internet of Things Journal, 8(18):14156 1151111100,, 2021. doi: (L.100/JIOT: 22..222233066 20684706
24

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Jianfeng Zhang. Backward Stochastic Differential Equations: From Linear to Fully Nonlinear Theory, volume 86 of Probabilily Theory and Stochastic Modelling. Springer, 2017.
Liangquan Zhang. A BSDE approach to stochastic differential games involving impulse controls and HJBI equation. Journal of Systems Science and Complexily, 35(3):766 801, 2022.
Huan Zhou, Zhenyu Zhang, Yuan Wu, Mianxiong Dong, and Victor C. M. Leung, Energy efficient joint computation offloading and service caching for mobile edge computing: A deep reinforcement learning approach. IEEE Transactions on Green Communications and Networking, 7(2):950 961, 2023. doi: 20wp://GCNN.222..111111
Shiji Zhou, Wenpeng Zhang, Jiyan Jiang, Wenliang Zhong, Jinjie GU, and Wenwu Zhu. On the convergence of stochastic multi-objective gradient manipulation and beyond. Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neurul Information Processing Systems, 2022.
A Proof of Lemma 3.8
Proof. We simply denote the conditional expectation E{-00} by EE{}) for convenience. Accordingly, we have:
(0) = Vo BPO(T)
(y
-k aT
(30)
(P(
a ar
(Vr
V log (ogroaa--lss-))
T
b EPa(T)
V log To (ar
(yr (yT aT
k-=k
E
Vo To (15 E
V/ (y ar /4 ar-k
vT-A
0=
a1
Q'
E
Vo logog lologo(ar prrp::ro_no(.................... (VT
k=
ap-k
T
Ve log Te ( (1
- aT + -" ((T-
n Pal- 27:1--1))
p((
Vo log (l ar - + -
E
==
-Ep(T))
Ve log To( /
(31)
K=U
where (²(.)) : y' x x R|,,, Q((,))) y' x S A RS and A : y x A R are
the bidirectional state-value, backward action-value, and bidirectional advantage multivariate functions (respectively). We also have:
V (T = M (1)
(32)
For (a), we used VoPo(T) Po(7) PologPo(1) and Vo logT) EtD Ve log To (a based on Equation (3) For (b); considered anti-causality, namely, that the current action does affect the future
25

---

Published in Transactions on Machine Learning Research (07 (07/2025)
of backward rewards. For (c), we applied the definition of backward action-value functions is applied. For (d), (KT==k: (T does not affect the result since ( { {Vo log -a(ar-=|sT-A) = 0. For (e),
we derive the Bellman's equation for the backward action-val value function as follows:
Q" ST k. aT-a)))))) k)
R ay x) yT- aT R. - ST ky 87 h
P u)x
$ - +B
( - --
Po
kr=
rh Ry ko ar + (yr 1))
(33)
-T-k-1 Pat o-- p(
where for (a) we considered 1b k- as the only relevant information to compute the expectation E + ) -k (yT w)) The same strategy can be applied to obtain the Bellman's equation for the bidirectional state-value function as follows:
V (YT-
C (y ar + 17 10]
Note that no distinct forward and backward Bellman optimality equations exist for FB-MDPs. However, Bellman Parcto-optimality equation can be found instead. For this purpose, we consider the forward and backward value functions to become stationary when the forward and backward transition probabilities are stationary. By recalling the notion Pareto-optinn carttt-optimality and Pareto front, we then define the Pareto-optimal forward and backward value functions as follows:
[Q" (s, a), Q² ay)
max [Q' (s, (' (v x(1))
[V' (s), 1 (x)) max (V' (s), (y *(1))
for all (s, y, a) E S x A, where [Q" a). Q" a)) and [v) vo s)] are the Pareto-optimal vector for the multi-objective optimization above. Now, we consider the following policy:
"'' <a(ss) = { 1, argmax (Q" as, Q" (y,s.a)
0,
otherwise
Note that here argmax [Q" a)) Q²' (y, s, a)]
returns set of vectors. We then have:
[s) v.'
vrr (s),
(y,s)
E max [Q" (s,a), Q (y, vys,a))
On the other hand, based on Equation 33 and the Bellman's equation for the forward value function, can get:
o (s) = st s.))
(8)(a))) + +75 (s')
Q² (y,s,a)=
E
(r)(y, 1 7/ (y-,-))
Pylly. p()b))
where y is the backward state preceding y and st is the forward state following s. We therefore obtain the following:
[Vf (s), vda (y, E max
E at Py(-an))
(s, + vVV5" (s*)}
E
(r, a) + 7VV=" {
P((s))
26

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Such an equation, referred to the Bellman Pareto-optimalis equation, provides a base to formulate dynamic programming algorithms for multi-objective FB-MOAC problems and motivates employing multi-objective optimization framework.
B Convergence of the FB-MOAC Algorithm
This section presents a comprehensive study the convergence of the FB-MOAC algorithm. Our investiga- tion starts by establishing of some foundational assumptions, followed by introducing some preliminaries. Subsequently, we analyze convergence for the scenario where the expected rewards are Lipschitz-smooth.
For the analysis, we need to emphasize that the stochastic nature of an FB-MDP affects the values of Φ, 2b and 0, based on the SGD rules [Equations 15 and 17 so they are treated as random variables.
We now make the following assumptions.
Assumption 1: The estimations of the state-value functions unbiased up to residual terms, i.e.,
B(V2,(()) s, 0) V (s) + 6/ (). s. 0) = (v, + 6 j E Sp, (y,s) y
i Sf, SES
where / Nes, and [o} are the forward and backward residuals. These terms arise from approximating the true value functions V. (s) y). by the neural network parameterization and the stochastic gradient. updates in Equation 15)
The following corollary is consequence of Assumption 1.
Corollary B.1. Under Assumption 1, the expected forward / backward gradients in Equation wilh the corresponding veward gradients given in Equalions and 7 respeclively.
10 coincide
voJ. (0) (0)= (o)) j SeS,,
VoJ,(0) ()) je S.
Proof. Based on Equations
16 and
19 we get:
T
(o E [Vo 0
-
E Vo log log (Sk. ax) oo]
Egg 81 E Vo log To (ax ) A (Sk, ax) Sk, aus Sk+1
Egg as 1
Vo log 1s |E A4, (sks ax) 0, S, as s6+1 S6+11 }
Voloomo(auls)) (1/ (A5 ax) - 1)0!)
Bk. at 8
T
(A/ Ve log No (ax (A/ (Sk, ax) + + - ())
0 (b) VoJ/ (0)
(34)
where (a) follows from Assumption 1 and the definition of advantage function A az))
ax) = r(sk. a&) +
yV (V Vols (8x), while (b) stems from E. El Vo log (o(a (sa66 00) = 0. Likewise, can be
shown that:
(o E
V log To (a (s)) SR.
()
(35)
27

---

Published in Transactions on Machine Learning Research (07/2025)
Assumption 2: The forward and backward expected rewards Lipschitz-smooth functions with constants Taf and Is, respectively, w.r.t. 0:
(0) - - L, (O) VoJ < Lol10' 0||, e S..
0ll, j j Sj.
Assumption can be related to the assumptions on the architecture of the considered neural networks.
Proposition B.2. Let the actor be represented by a neural network parameterized by 0 E θ where all activation functions are regular, i.e., Lipschitz-smooth with constant La mnaschitz-continuuu with constant T' and bounded both above and below. Moreover, assume that either the action space A is compact or actions sampled from the policy distribution =o(-I-) are clipped. Then Assumption 2 holds for any family of distribulions that are bounded whenever ils poramelers and inpul are bounded.
Proof. With respect to Equations 34 and 35 it suffices to evaluate Lipschitz-smoootness smoothness of the log- policy (a)s)( since the sum of Lipschitz-mmooth functions is itself Lipschitz-smooth. Let denote such policy by a bi-variate function f(a, ((a.g(0.s)) s)) x g - IR wherein a is the sampled actions being detached (so they do not depend on 0), θ x S + G is the output of the d-parametric neural network, and G is the space of neural network output. Based on the chain rule, we have:
VVVa(a, - vol(a. (( 8)) -|]| Jog(O2. 8) +
(a
I (a g(0 8)) (x(( (aa:)l0 0211 + Aw (L // V
v (a
where Jog the Jacobian of g w.r.t. 0, Vgf is the gradient of f w.r.t. g, and fNr and hy are two functions depending on the architecture of the neural network. For (a), we leverage the fact that the composition all the
of bounded and Lipschitz-mootth (continuous) functions remains Lipschitz-mmooth (continuous) activation functions regular. Therefore, the output of neural network is Lipschitz-smooth with constant fxr( (acc) and Lipschitz-conntnmuoni with constant (L Now, as and are bounded (if is inherently unbounded, it can be clipped whenever the problem allows doing such), the norms of the first and second derivatives the log- pelpooicy f(a, g(0, s)) with respect to the network output g(0,s) are bounded for typical policy distributions such as Normal, LogNormal, Dirichlet and Beta. Therefore, there exists a parameter A₁ such that (a, g(0, s))l S A1. The boundedness of the norm of the second derivative is equivalent to Spsschizz-smoothneee thus, there also exists A2 such that (a, s)) - fa, g(02, s))l| < Agllg(0,, (02, s) g(0,. s)]l Therefore, we get:
Vos(a. - Vol(a, ((2,) AISA((iact I 110 0211 + hN
act A2lg((0), - g(0,, s)ll
I Aifx La 0211 + hv AA2l01 02ll
= (AISN(Lact L'a)) + h ( (²)))))))))))))) A2)l10 01 - -02ll
Hence, the log-policy is Lipschitz-smooth with parameter AIfN( L + (L A2 and the statement follows.
Assumption 3: Consider the following stochastic forward / backward gradient:
v v o. Φ) = [ Φ)] jes,, (V. ]²] jes.
then, its conditional covariance is bounded by posilive semi-definite matrix B:
2) v (
bb) 1 ()) ()) = B.
28

---

Published in Transactions on Machine Learning Research (07 (07/2023) 2025)
where
JJ (0)
[VoJ] (0) jes,' [VoJ] jes.
Note that the assumptions outlined in this context align with the conventions in the literature related to convergence analysis (Tian el al. 2023 Qiu et. al. 2021 Zhou et al. 2022 Xiong et al. 2022)
B.1 Preliminaries
The following theorem proves that forward / backward expected rewards monotonically increase with each update iteration.
Theorem B.3. Consider the forward / backward expected rewards, i.e,, {J/ ()Es, and (J)(-))ss., and the forward backward stochastic rewards, i.e., {J] )ies, and (3) complying with Assumptions 2
and 3, and Bact as the solution of Equation (18). Moreover, consider the SGD in Equations 14 and 17 characterized by iteration number and actor learning rate {uileez with
1 B 1 (VJA pi min max (T.g, [o) max (T,g, [.b)
which generate the sequences (Φ hez, Tez and Hezz. We then get:
EE (J/())tt)) z B {J)(00)) jES,
and
E (})( 2 ((
h e
Proof. Based on Assumption 2, we oblain:
1/10 (0') v.J/ (0++1_ - 0¹)
Lg 10 - 0¹112.
(36)
2
On the other hand, the update rule in Equation
17 gives:
o 0¹ + M4 (0', Φ'), V ( B 0² + 14 V (0' 0'' (')b'))
j
(37)
Plugging Equation 37 into Equation
36 yields:
P.L,
1/(0)t1) 1/ (
(0 ("b oi 2b' Dact
Peet V j Φi b') (0' 8i ab 1Bat.
2
Taking the expectation on both sides of the equation above results in:
E (o 1/(02) 2 m
V.
(0',
Φi (5))))
L
B
(B (0)) T f gd
2
nct.
(b)
m
es
2 Biai
(O")
Th Bhen
2 Bren.
"B
P
(e,
1il.
Baet)
(
4hh
11 Iof
|BI|]
(38)
2
act
2
where: for (a), we used Assumption 3; for (b), we leveraged V.f. = e, V J.h and the fact that
B Bf act (V') T Th Φ. 2b)) ) 0²) = Bi act
()'(0') E th 00' Φ. x)) 10)) B
3 act
" V J )0))
29

---

Published in Transactions on Machine Learning Research (07/2025)
according to Corollary [B1 ; for (c), we employed Pact BBaact ||| |/ can derive from Equation (18) that for all Plat.s 0:
|| s |B]].
On the other hand,
Pact = (1
1 111(1 (O) (1)
(0) ) ( (VJ
(v)) -1
(39)
By substituting this into Equation 38
we finally get:
B Ff - 10(00) 2/1 (1 1iLs
1u LI
B
2
I (oL (J( (F)))))())))
2
44 E
2 2
B
0
2
IT ()LD((())) v.JD((O)) (T
where we used μ4 max [a] < for (a). The statement follows by considering that the denominator in the right-hand side of the equation above is positive due to the positive-definttnnsss of (V"" (e))
(VJ -1
The same analysis can be applied to infer B( (O - 10)) 2 0
Remark B.4. Theorem B.3 guarantees all the forward and backward expected rewards {E (0)) jes, and (E (0)) jes, constantly increase with the number of algorithm iterations. Consequently, this enables to jointly improve all of the cumulative rewards, either forward or backward, on average with cach itcration.
JEs,
Corollary B.5. Consider the framework of Lemma B. we then get:
E {Bact T (0') V Bact S _B (T)
Σ uii JESJUS,
Baoss (ott)) - -F }
+ ui ma Lo [B]
Proof.
The statement follows based on Equation
38
and μ. max La S 1.
J
B.2 Analysis for Lipschitz-smooth Rewards
We perform a convergence analysis by focusing on the Lipschite-smoollues condition detailed in Assumption
2. However, we first need to present definition for the convergence to locally Pareto-optimal solutions.
Definition B.6. The parameter sequence {0.)== is said to converge to locally Pareto-optimal solutions (Zhou et al. 2022 if
lim 10000
E
min B3 20
Σ
Vo
+
Py=
jE(S)uSa
where (0) is the j-th clement of (lb with vJ = | (0) (0) jes, [Vo]} ses.)
We thus have the following theorem.
Theorem B.7. Consider the forward/ / backward state-value estimations, i.e., following Assumption 1. Moreover, assume that the forward/ backward expected rewards, i.e., (J) Obes, and (J)sss,,,, and the forward/ /backaard stochastic rewards, i.e.,, {J/ Dies, and (J)())) comply with Assumptions 2 and 3, and that Bact is the solution of Equation (18). Finally, consider the SGD in Equations [1h and 17] characterized by iteration number i and actor learning rate {uitiez with
and (V (.esss
1 mmxx(Ls Lo} max(Ly, Lo) |B| (1T (VJ) (0) }
#4 min
30

---

Published in Transactions on Machine Learning Research (07 (07/2025)
... S Ml, which generate the sequences (Ø' f and (0)) Then, and 0 < pz S... ... S Ms ...
gel:
B V J (0') Bacll 1 max max(Ly: Lb}
B
14 p4 max ( 5. Lb}
1
2
-0)
(40)
1 pr U.S. jESJUS,
Proof.
Based on Equation 38 an its counterpart for (J} we have:
E Σ 1/(0 J/ (0))
2 mB (1
Buot 1S, U (ol)
(0') (0))
u max
B [S U U.B.
2
2
JS,
Mi Wilimay 12((=-be1-)))
Piece T (o') T V. (00) (0')
Mi max
[B]]
2
where T,max == max Lg, La), and we used the identity 1 VJJ J. Bact = |S, u U.oBBacc
J. J.Jhb Bact
based on Equation (39 for (a). Since HiL max S 1, it is:
E
(IVJ" (0)))"
12]
+ -
(
-1((0')) }
L.I
B
Sy Solu
4 muxx
muxx
JESJUS.
(a)
2
E
Σ
(J)) (0'))
14 max
B
(41)
Ur|S, S.
2 L.Lmaxx
JES,
where (a) follows from maL < 1. We then take the summation on both side of Equation
41 and apply
telescopic cancellation. The statement follows by considering that Pact is the solution of Equation (18) for
0 = 0¹.
]
Remark B.8. Under the learning-rate scheduling Mr O(LVVi, both terms 2 and $ Ever M4((2
Tpr p max Lo}) in Equation (40) decay at the rate of O(IvVI), where I is the number of algorithm iterations equivalently, the number of policy updates in FB-MOAC. Consequently, Theorem B.7 guarantees convergence to a locally Pareto-optimal solution (Zhou et al. 2022) with a convergence rate of O11/VI). This rate is notably consistent. with that of single-objective actor-critic methods for forward-MDPs, which exhibit. a convergence rate of (L/VVI) (Fu et al., 2021).
Remark B.9. In contrast, if the learning rate is chosen as #4 = O(/i), the term Tur in Equation 40) does not vanish as iteration becomes large. This prevents a general convergence guarantee in this setting.
C Additional Details on the Edge Caching Use Case
The environment of this experiment is a cellular network with cache-equipped Base-Stations (BSs) similar that in (Amidzadch et al.) 2023 The BSs are spatially distributed across the network with intensitics Aue- The envirounent also includes a library of N different content items as well as fixed mobile users requesting them from the cellular network. The network operates over time slots with index t e {I, where T is the total duration of the operation. The network handles user requests at the beginning of each time slot. Content items have different popularity is requested by a randomly selected user at time t. The goal is to satisfy as many users as possible during the network operation. At the beginning of each time-slot, the BSs cache the most popular content items with probability ( and simultancously multicast them toward users by consuming content-s sppeeit- radio resources (t)) We denote the system action parameters by the vector a(t), which depends the content-specific bandwidth allocation and cache placement of BSs, i.c., a(t) = [[r (1)) (w,, (t)) A multicast outage may occur with probability ta))
where (t) is the probability that content n
x (PAssppach(1)) n/)
(a(t))), erfe
)
(T(()) 23/w,, (1) - -1,
(4
31

---

Published in Transactions on Machine Learning Research (07 (07/2025)
which is obtained by averaging users. As result, certain users fail to receive the requested content in the current timeslot and their request is deferred to the subsequent one. Hence, each time-slot sees a distribution of users accounting for the repeated requests and distribution describing the new preferences toward content items. This leads to a linevarrying model for the request probability of content n, (t):
N Σ (1 (1- 0m (a(t -a(t- 1), 1)) (t 1).
(t) P (t- 1) (a(t - 1),t-1) + P
P (t)
(43)
m=
repeated request
new request based the popularity
Note that (t) indicates the request probability of content n averaged over all users. Then, it can be simply verified that I (t) = 1, considering Σn= (t) 1. Equation (43) therefore represents forward dynamics, with the forward state vector s(() (t) and the action vector a(t).
A request for a content item is repeated across several time-slots until successfully fulfilled, resulting in an expectod latency (t) for the successful delivery of content n. Its timo-varying dynamics can be derived by the law of total expectation as follows:
L., (a(t),) t) E (latenney outage (outaag) + Elaatency| no outage]P(no outage]
d(t)
= (d(t) + Ln 11( + 2 (1 (a(t),t))
(44)
where d(t) is the duration of time-slot in seconds, and L, (aT)) since system operations finish and the users do not need to wait any longer. Equation 44 represents backward dynamics, with the backward state vector y(t) = L(t) and the action vector at)). Note that this model fully captures the trade-offs involved in the delay dynamics and differs from the conventional formalism that does not provide a comprehensive model when accounting for successive slots; for the delivery without outage, the expected d(t) ils realizations follow uniform distribution with values between 0
latency simply becomes (()
2" -
and d(t). Notice that the backward dynamics in Equation (44) based on Theorem 3.6 cannot be expressed as a standard MDP. Thus, Equations (3 and [44] together model a FB-MDP and are coupled through the
action a(t). Hence, they should be jointly considered to obtain an optimal cache policy.
D Case Study: Computation Offloading
We now present an additional use case in the context of computation offloading (Zabihi al. 2023
D.1 System Model
There are Naev mobile devices and N computational intensive tasks with diverse sizes {s A typical mobile device prefers task n with probability pp (t), and it offloads the preferred tasks with probability off prf
Pn (t) to edge server. The server operates in time slots with duration indexed by t and leverages task-specific parallelism mechanism to process the offloaded tasks. Specifically, it employs N buffers and N computational resources with (t) signifying the buffer capacity and Ced 2 Cedg denoting the computational resources allocated to file Cedg is the minimum extent of allocated resource, while E (L) = Bedg and Σ (rd = Cedg represent the total buffer limit Bedg and computational capacity Ceds, respectively. The control parameters for this problem thus ((pfflpr(() (-), B.dR(-), (C))() A typical device offloading task encounters failure if the corresponding buffer overflows, resulting in the need to re-offload the task. The queue length of the n-th buffer Sn is thus described by the following expression:
L. + 1) = max (1) Sapt (1 + 1)am(1+1)-
n (e 1) (1+1)
.0).
(45)
new data buffered
computed data de-buffered
32

---

Published in Transactions on Machine Learning Research (07 (07/2025)
for (I....,Ne N), where Sn = Sn Nawv, (t) is the offloading probability for file n, Supa" (t) is the total amount of data offloaded for the n-th buffer, and
Bt (t 1) I,n (t) Supa f (1 +1 .13.
at + 1) min {
denotes the fraction of data that can be buffered due to the buffer capacity B Therefore, it can be easily verified that Ln (4 + 1) S (t + 1). The overllow probability On for n-th bufler is thus obtained as Ox(() = - an (t). This equation exhibits a controlled forward dynamics, based on which we constitute the forward state [p (t), L(t) for this dynamics.
We now compute the average compulation time needed for typical device preferring task n, i.e., (Pr(()). If the task is preferred and locally computed, (t) depends on the computation capacity of the device itself; if it is offloaded, sprt (t) depends on the computation offloading time raft (t), i.c., the average computation time needed for device to offload task n. Applying the law of total expectation yields:
to (t) =E(computation time task preferred] = Pn Prfflpt(( (f) en (t) + (1 - Po
(1-
8n off]prt (t)) Cdev
(t))
(46)
As a consequence, we need to compute eoth (t). If the task being offloaded faces an overflow, will be re-offoaded in the next time-slot. However, the computation time depends on the queue length and the computation resource allocated to the task if no overflow takes place. Therefore, efff (t) is derived by the total expectation law as follows:
ffff (t) =E| (computation time task offloaded } = On + +)(+++++++++ t (t+ (t+ + n2
(1- +(1 On(t + 1)) tns
(47)
for E (1..... NN,, where t, stands for the needed time to compute task n with size if it is successfully buffered. We thus have:
bn = (t) + jan(tt + 1)paft (t + 1)S, + sn
C (L
obtained considering that n-th buffer has already a queue with length (,) and an additional buffer S, with probability fl, + 1)pa (L+ 1). Equations 46 and (77) logether provide continuous model for the average time required to successfully compute task within different slots. Additionally, they represent controlled backward dynamics with the backward state t (t) = [tp .....
....,.eff]()) (t). We now consider two action-coupled conflicting rewards. The forward reward is related to the overall overflow probability as:
rop(t) = - pp. (t),
while the backward reward is related to the expected computation time:
NV (Cr(t)= = Σ
p (t)epr (t),
This problem thus represents FB-MDP with an action [[Crde],,. In (Bols) [o, 11 x [o, 1 x 1].
In {po off In] and an action space
D.2
Experiment Setup and Hyper-parameters
We set the number of devices Naev = 100, the number of tasks N 20, the file size s, 10 + Kbits, the computational capacity of the devices ddevr 10 Kbits / slot, the minimum exlent of allocated resource Gesd 10-5, the edge computational capacity C 100 Kbits/slot, the edge buffer capacity B 100 Kbits and the slot duration 60 seconds. The paramterrsmesers of FB-MOAC are the same as in the previous experiment excluding the learning rates of actors and crities, which are set to 3 10-3
33

---

Published in Transactions on Machine Learning Research (07 2025)
FB-MOAC
0.8
F-PPO
OОР
ГОР
FC7
Empirical PDF
FCT
0.5
0.4
0.4
0.3
0.3
0.2
02
0.1
0.1
0.0
-0.35
-0.10
Cumulative Rewards
Cumulattive Rewards
(a)
(b)
0.5
F-MOAC
'OP
CCT
Emoiricall PDF
0.2
0.1
0.50
0.45
0.40
0.35
0.30
0.25
0.20
0.15
0.10
Cumulatve Rewards
(c)
Figure 6: Histograms depicting the empirical distribution function of the computation time rer and the overflow probability for for: [a) FB-MOAC, (b) F-PPO, and
[c]
F-MOAC.
D.3 Performance Evaluation
We consider a learning-based strategy to evaluate FB-MOAC this experiment. Accordingly, we leverage the fact that optimizing the overflow probability decreases the expected latency according Equation (17 Consequently, apply the bascline RL algorithms PPO (Schulman et al. 2017a) and A2C (Grondman et al. 2012 to obtain offloading policy. Similar to Section 5.2.3 call the resulting algorithms F-PPO and F-MOAC, they manage the backward reward using forward mechanism.
Figure 6 reports the histograms (i.e., the empirical probability density function) for the performance of FB-MOAC, F-PPO and F-MOAC algorithms (respectively) in terms of ret and Top. Clearly, both the forward and backward rewards of FB-MOAG are higher than those of F-PPO and F-MOAC. Consequently, FB-MOAC outperforms F-PPO and F-MOAC from the perspectives of both expected computation time TCT and overflow probability Top. In other words, the resulting policy of FB-MOAC Pareto-dominates the strategies of F F-MOAC and F-PPO average. As expected, FB-MOAC shows more significant improvement in the computation time compared to the other two algorithms. Nonethcless, FB-MOAC even gives better overflow probability performance than that of F-MOAC and F-PPO as it obtains a more favorable learning mechanism even for the forward dynamics. These results indicate the importance of explicitly incorporating the backward MDP rather than eliminating through adjustments to the backward rewards. They also demonstrate the effectiveness of the proposed FB-MOAC algorithm in addressing the corresponding FB-MDP problem.
34

---

Published in Transactions on Machine Learning Research (07/202) 2025)
Q:5: 00212
Q:5: 059
-0-11 Lat: D5
Let: 40,003
.0,376
5W.-0.119
LaTT FOS
Qo5
024
0.016
100
Qus: D 0:155
Lat 0202
-0.03
OsS: -0,35
Q:S: 10,53
ADURIET pRipaOk
-0.01
Lat: BUB0
BWV O 100
5IM 10 23332
-0.05
-0.06
-0.27
-0.28
-0 0.26
BW
-0000
0..55
-0.08
142
0024
-0.23
-0.22
-0.21
0.01
Bandwidth Consumption
-0.19
-0.02
-0.18
-0.12
1.16
-0.15
00 1414
Quality of--y--f--eee Serviee
-0,05
13
-0.07
0.12
-0.08
11
-0.99
00.1
-011
0.09
-0.11
-0.00 06
Figure 7: The collection of Parelo-optimal solutions for the of edge caching, obtained by applying discrete policy over different preference settings ef E [0.1, and e E [0.1, The radius of the points denotes the probability of occurrence. Given that many of these solutions cannot be dominated by others, the FB-MOAC algorithm efficiently provide the Pareto-front solutions.
E
Deriving Pareto-optimal Solutions
Here, we employ the mechanism explained Scction 4.3 to obtain the Pareto-optimal solutions the use case considered in Section 5.2 namely, edge caching in wireless networks. Figure illustrates the collection of Pareto-optimal solutions with diverse preference parameters e/ E [0.1, 1] and e [0.1, 1]. Note that most of the solutions cannot be dominated by others, therefore, FB-MOAC can provide most of the Parcto-optimal solutions with the proposed preference policy.
F
Transforming a FB-SDE into a FB-MDP
This section shows the wider applicability of FB-MDPs and FB-MOAC to problems described by FB-SDEs. Specifically, it explains general method transform FB-SDE into FB-MDP, which can then be solved with FB-MOAC (as described in Section 5.1 In particular, the Fuler-Maruyama method [Klodden & Platen
35

---

Published in Transactions on Machine Learning Research (07 (07/2025)
1992 can be used to discretize the space of control problems. More specifically, the forward and backward SDEs can be transformed to forward and backward MDPs, respectively.
Consider the following controlled FB-SDE:
(x(()))(tt,t)) du (t), dxx(t) = f(x(t), u(t), dt + E (x(()) u(t),
dy(() - (() (z)) (z'(0,) di + Ei= 21) dw (t),
(48)
x(0) zo. y(T) = yr,
where w(t) (t) is an I-dimensional Wiener process, x(t) R" is the forward process, f(..))) R" x U x [0, R" is the drift function which describes the dynamics of the forward SDE and is governed by an optimization control process u(t) E U C I,, and o(()))) R" x U x [0, + R is the diffusion coefficient determining the extent of noise added to the forward dynamics. Conversely, y()) E I is the
backward process, g(:.....) 1TTh x Rm x [0, T] + In is the generator function and z (t) E R" the dynamics of the control process.
Note that the solution of the backward SDE is determined by the pair (y()), ())() where (2(()) should be found to guarantee that the backward process (t) is properly adapted. The solution of the FB-SDE in Equation 48 is thus denoted by the tuple (x(+), fz (-))).
In this section, we resort to Fuler-type numerical approaches that discretize the evolution of dynamics to numerically solve the considered FB-SDE. In this regard, we partition the time interval [o, T] into
sub-intervals (. tx) for & E ..)....N, each sub-interval with length At F. where to and ty T. By applying the Euler Maruyama method (Kloeden & Platen 1992). we then obtain:
x(ta + At) xo x(tx) + fx((tx), u(tx), tt) + Σ o' (x((x), u(ta), tx) tw) (tx), for h e NO...., 1} (49)
where Aw" (t"(t)) = w (tx + At) (tx) is a Gaussian random variable with variance At. Equation be interpreted as forward MDP with action u(.) and forward state x(-).
49 can
For the backward SDE, we employ a seni-stochastic approach (Archibald et al. 2020 as it is considerably more efficient from the complexity perspective, even though it might require more itcrations for the algorithm to converge to an accurate solution. This can be compared with the stochastic gradient descent that alleviates extreme complexity by estimating the expectation with single sample. Accordingly, get:
ylu) ss y(1+ + At) - g (yy(1 + Al), u(4 + Al), Z(4 + AI), 42 + A1) Al.
(50)
where Z(-)= (2 and (t) is obtained as:
1 (1x)) At y(u + At) (w) for E (1....,1).
(51)
Equation 550 together with Equation 51 represents backward MDP with action (u(-), Z(-)) and backward state y(.).
36


Reviews:
[{"review_id": "poI5UNG214", "reviewer": "Reviewer_mypH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper proposed an algorithm for multi-objective optimization in forward-backward MDP that addresses real-world applications. The proposed approach is supported by mathematical proofs and empirical evidence. \n\nThe paper could be better structured, and some of the discussions could be extended for readability. See the requested changes for a detailed list.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies forward-backward multi-objective reinforcement learning for multi-task control problems and proposes an actor-critic-based algorithm. The proposed method is demonstrated to be effective through convergence analysis and empirical results. The method is also demonstrated to be effective in several real world scenarios, such as mathematical finance and wireless caching.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Can you elaborate more on \"The convergence results of Theorem 4.1 are aligned with those of single-optimization algorithms for forward-MDPs\"? What do we want to align with single optimization algorithms for forward MDP, is that the best results available for forward MDP?\n2. Please add a formal definition of locally Pareto-optimal. When there are multiple Pareto optimal points, which one does it converge to? \n3. Remark 4.3 is unclear. Why is the computational complexity primarily described by the convergence rate? \n4. As the theoretical guarantee is only based on Lipschitz and smooth loss, can you comment on whether the losses of the examples used in the experiments are Lipschitz and smooth? If they are not, I suggest adding an additional experiment on Lipschitz and smooth loss to better understand the performance of the algorithm on Lipschitz/smooth vs non-Lipschitz/smooth, especially when there are limited baseline algorithms. \n5. Although the trained algorithm seems to be converged, I would still suggest specifying the number of random seeds used and report the standard deviation/confidence intervals of the results.", "broader_impact_concerns": "I do not have concerns of the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KHwBCg1Uq5", "reviewer": "Reviewer_tjxc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of the paper lies in its technical rigor and the soundness of its mathematical analysis and proofs. Although the authors employ established proof techniques rather than novel strategies, their derivations are clear, logically coherent, and meticulously presented. \n\nHowever, the case studies are unclear and fail to demonstrate why FB‑MDPs are necessary compared to standard MDPs. In particular, in the motivating example (Section 2.1) it remains ambiguous why the process cannot be modeled as a regular MDP with stochastic dynamics, where agents move to different states and receive different rewards depending on if content transmission was completed or failed. Therefore, the paper does not convincingly illustrate the practical advantages of FB‑MDPs over classical models. For these reasons, I recommend rejecting the manuscript in its current form, although I would be willing to reconsider my evaluation if the authors substantially clarify their empirical evaluation to show how FB‑MDPs yield benefits that standard MDPs cannot capture in the proposed scenarios. I give detailed suggestions in the next section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce Forward-Backward Markov Decision Process (FB‑MDPs) model, an extension of traditional MDPs that incorporates dual dynamics and rewards. In an FB‑MDP, each action is associated with two transition functions—one over the forward state space and one over the backward state space—and two corresponding reward signals. The learning objective is reformulated as finding Pareto‑optimal value functions that balance forward and backward rewards.\n\nTo tackle this multi‑objective optimization, the paper presents the Forward‑Backward Multi‑Objective Actor‑Critic (FB‑MOAC) algorithm. FB‑MOAC leverages actor‑critic architecture to simultaneously learn policies that navigate both dynamics and optimize the trade‑off between rewards.\n\nEmpirical evaluations on three benchmark tasks showcase the effectiveness of FB‑MOAC: in each case, the proposed algorithm outperforms prior methods in converging to higher Pareto‑optimal frontiers, thereby demonstrating its practical advantages for problems with coupled forward and backward dynamics.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Since the authors define an FB‑MDP as a tuple $(S, Y, A, P_f, P_b, r_f, r_b)$, I recommend that each case study be presented by explicitly specifying: the forward state set $S$, the backward state set $Y$, the action set $A$, the forward transition function $P_f$ and backward transition function $P_b$ induced by an action, the forward reward $r_f$ and backward reward $r_b$, and the formulation of the resulting multi‑objective optimization. For each example, I would also recommend to include a brief discussion explaining why the problem cannot be equivalently modeled and solved as a classical MDP.\n\nAdditionally, I have a few minor points:\n\n-Since Problems 5.1 and 5.2 are treated via discretization, please describe them as discrete‐state problems rather than as the original ODE formulations.\n\n-Include the full name “Forward‑Backward Multi‑Objective Actor‑Critic (FB‑MOAC)” in the abstract.\n\n-At the start of Section 4, the text refers to “mechanism according to Theorem 3.7 and Theorem 3.8,” but Theorem 3.7 is actually a remark, and Theorem 3.8 is a lemma—please correct these references.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gPYiXKIbVi", "reviewer": "Reviewer_A32U", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\nThis paper introduces a novel approach to solving Forward-Backward MDPs using multi-objective reinforcement learning. The authors support their method with both theoretical analysis and empirical evaluations across multiple benchmark tasks.\n\n### Weaknesses:\n- The positioning of this paper within the field of Forward-Backward MDPs is not clearly articulated. Prior work, including key statements and theoretical results, should be explicitly cited and clearly distinguished. Additionally, the paper's novel contributions—both in terms of theoretical insights and algorithmic design—should be more prominently emphasized.\n- The theoretical results rely on strong assumptions, such as strong convexity and Lipschitz smoothness. However, most assumptions have been put into the appendix and it's not clear how the assumptions would hold when using NN-based functions.\n- The learning-based baselines used for the Edge Caching benchmark are more like ablations of the proposed algorithm, while the rule-based one is relatively too simple. The athors can consider using more advanced baselines (not a requirement).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work introduces forward-backward Markov decision processes (FB-MDPs), a new class of problems that extend beyond the expressiveness of standard MDPs. Building on a derived optimality condition specific to FB-MDPs, the authors developed a multi-objective reinforcement learning algorithm, FB-MOAC. They provided theoretical guarantees on the algorithm’s optimality and convergence, and validated its effectiveness through comprehensive experiments in two distinct application domains.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Could you elaborate on how restrctive this assumption can be: This work constrains the definition of FB-MDPs to the case where the forward (backward) dynamics does not depend on the backward (forward) state.\n\n- Where is Assumption 4 mentioned in Theorem B.8? Remarks B.9 and B.10 are very inform.\n\n- Please provide a formal and complete version of Theorem 4.1 with all necessary assumptions. Do you assume the optimization problem at each iteration is well solved?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)