Factor Learning Portfolio Optimization Informed by Continuous-Time Finance Models
Authors: Sinong Geng
Houssam Nassif
Zhaobin Kuang
Anders Max Reppen
K. Ronnie Sircar
Venue: Accepted by TMLR ()

Abstract: We study financial portfolio optimization in the presence of unknown and uncontrolled system variables referred to as stochastic factors. Existing work falls into two distinct categories: (i) reinforcement learning employs end-to-end policy learning with flexible factor representation, but does not precisely model the dynamics of asset prices or factors; (ii) continuous-time finance methods, in contrast, take advantage of explicitly modeled dynamics but pre-specify, rather than learn, factor representation. We propose FaLPO (factor learning portfolio optimization), a framework that interpolates between these two approaches. Specifically, FaLPO hinges on deep policy gradient to learn a performant investment policy that takes advantage of flexible representation for stochastic factors. Meanwhile, FaLPO also incorporates continuous-time finance models when modeling the dynamics. It uses the optimal policy functional form derived from such models and optimizes an objective that combines policy learning and model calibration. We prove the convergence of FaLPO and provide performance guarantees via a finite-sample bound. On both synthetic and real-world portfolio optimization tasks, we observe that FaLPO outperforms five leading methods. Finally, we show that FaLPO can be extended to other decision-making problems with stochastic factors.

Published in Transactions on Machine Learning Research (08/2025)
Portfolio Optimization Informed by Continuous-Timg Time Finance Models
Factor Learning
Sinong Geng Princeton University
Houssam Nassif Meta
Zhaobin Kuang
Stanford University
Anders Max Reppen Boston University
Ronnie Sircar Princeton University
Reviewed on OpenReview: https: //tp://ppppe openreveew.vww. net/ forum? id= KLOJUCusVE
Abstract
This work considers financial portfolio optimization in the presence of unknown and un- controlled system variables referred to as stochastic factors. Existing work falls into two contrasting distinct categories: (i) Reinforcement learning employs end-to-end policy learning with flexible factor representation, but does not precisely model the dynamics of asset prices or factors; (ii) Contimouu--imme finance methods take advantage of explicitly modeled dynamics but pre-specily, rather than learn, factor representation. We propose FaLPO (Factor Learning Portfolio Optimization), a framework that interpolates between these two approaches. Specifically, FaLPO hinges on deep policy gradient learn a performant invest- ment policy that takes advantage of flexible representation for stochastic factors. FaLPO also incorporates continuous-time finance models when modeling the dynamics. Tt uses the optimal policy functional form derived from such models and optimizes objective that combines policy learning and model calibration. We prove the convergence of FaLPO and provide performance guarantees via a finite-sample bound. On both synthetic and real-world portfolio optimization tasks, we observe that FaLPO outperforms five leading methods. Finally, we show that FaLPO can be extended to other decision-making problems with stochastic factors.
1 Introduction
Portfolio optimization studies how to allocate investments across multiple risky financial assets (such stocks) and safe assets (such as US government bonds). The investment target is often formulated maximizing the expected utility of the investment portfolio's value at fixed time horizon, which conceptually maximizes profit while constraining risk (von Neumann & Morgenstern 1947). With continuous-time stochastic models of stock prices, great advances in the expected utility maximization framework were made in Merton (1969)
using stochastic optimal control (dynamic programming) methods. More realistic models incorporate factors like economic indices and proprictary trading signals (Merton et al., 1973 Fama French 2015 1992), which (i) affect the dynamics of stock prices; (ii) stochastically evolve over time; (iii) are not affected by individual investment decisions. With greater data availability, it is natural to design and apply data-driven

---

Published in Transactions on Machine Learning Research (08/2025)
Challenges
Our Solution
Existing Works
Huge Noise
Neural Stochastic
Reinforeement
Models
Complicated Factor Effects
Learsing
Fa
Continuos-TIme
Discrete Time
Medet-Regnparized
Finance
Continuous Time
Policy Learning
Figure 1: Demonstration of FaLPO
machine learning methods [Bengio optimization. This work proposes a novel method (Factor Learning Portfolio Optimization) which combines tools from both machine learning and continuous-time finance.
1997 Dixon et al. 2020: De Prado 2018 to handle factors for portfolio
Portfolio optimization with stochastic factors is challenging for three reasons. First, financial data is notoriously noisy and idiosyncratic (Goyal & Santa-Clara 2003), causing complex purely data-driven methods to be unstable and prone to overfitting. Second, the relationship between the factors and their impact on stock prices can be extremely complicated and difficult to model ex ante. Third, many successful finance models are in contimuous time and require interacting with the environment infinitely frequently. As a result, such models cannot be easily combined with machine learning methods, many of which are in discrete time,
Current approaches to portfolio optimization broadly fall into two categories: reinforcement learning (RL) and continuous-time finance moth Many RT solutions to port.ooli optimization are built on deep deterministic policy gradient methods (Silver et al. 2014 Hambly et al. 2023 Section 4.3). Such methods parameterize the policy function as neural network that takes factors as input and learns the neural network by optimizing the corresponding fortoolio performance. However, these approaches (as well as other model-free methods like Haarnoja et al. 2018 Geng et al. al. 2024) have high sample complexity and tend to overfit due to the high noise in the data. Other RL methods explicitly learn representation (Watter et al.] 2015 Lee et al.
2020:Laskin
2020: Laskin al.
2021 and leverage discrete-time models (Deisenroth & Rasmusssn [2011 Gu al.) 2016 Mhammedi et al. 2020: Janner et al.) 2019 Nagabandi et al.) 2018). Nonetheless, these methods are not
informed by contintous-time finance models and,
experiments suggest in Section cannot benefit from
structures inherent in the financial market. As result, RL provides flexible representation for stochasticc
factors but suffers from the high noise in the data.
Stochastic factor models can be used to mathematically derive optimal (or approximately optimal) investment policies (Kim Omberg 1996 Chacko Viceira 2005 Fouque et al. 2017 Avanesyan [002] To this end, one needs domain knowledge to pick and model the factors. Then, model calibration (a.k.a. model fitting, parameter estimation) is conducted by maximizing calibration objective. With the calibrated model, the optimal investment policy can be derived analytically numerically (Merton 1992 Fleming & Soner 2006). This procedure of calibration and optimization effectively constrains the 'learning" in the optimization step, and thus helps reduce overfitting to noisy data. However, this approach cannot capture the complicated factor effects the data, because the factors may be complex and unlikely to be identified manually. Therefore, these methods may end up with oversimplified models and suffer from model bias with suboptimal performance.
To tackle these limitations, we propose factor learning portfolio optimization (FaLPO), interpolates between the two aforementioned solutions (Figure D FaLPO includes (i) a neural stochastic factor model to handle huge noise and complicated factor effects, and (ii) model-regularieed policy lcarning method to combine continuous-time models with discrele-time policy learning methods. First, to reduce the sample complexity and avoid overfitting, FaLPO assumes factors and asset prices follow a parametric continuous-time finance model. To capture the complicated factor effects, FaLPO models the factors by a representation function parameterized by a neural network with minimal parametric constraints. Second, for policy learning, FaL.OO incorporates two types of regularization derived from contimuous-time stochastic
method that
2

---

Published in Transactions on Machine Learning Research (08/2025)
factor models: policy functional form and model calibration. Specifically, we derive policy functional forms from the neural stochastic factor model using stochastic optimal control tools, and apply it to parameterize the candidate policy in FaLPO. The use of this form in the learning algorithm effectively acts as a regularizer. Then, model calibration and policy learning are conducted jointly, such that the learned policy is informed by continuous-lime models.
Theoretically, we prove that the added continuous-time regularization leads to the optimal portfolio per- formance the trading frequency increases. Empirically, we demonstrate the improved performance of the proposed method by both synthetic and real-world experiments. We review the related literature in Appendix A We also discuss how FaLPO is extendable beyond portfolio optimization, and can be applied to other decision-making problems with stochastic factors in Appendix H
2 Background
In this section, we first formulate the portfolio optimization problem. We then review two major solutions to this problem: deep deterministic policy gradient in reinforcement learning (RL), and stochastic factor models in contimuus-tiime finance.
2.1 Portfolio Optimization
Problem Formulation Portfolio optimization seeks to derive a policy of asset allocation that yields high return while maintaining low risk for the invest- ment. Formally, consider dg risky assets with prices S. = [St S8 ... S and risk-free money mar- ket account with, for simplicity, zero interest rate of return (like cash). We observe dy stochastic factors (e.g. economic indices, market benchmarks) denoted as Y, Such stochastic factors (i) affect the dynamics of asset prices; (ii) evolve over time stochastically; (iii) are not affected by investment decisions. Given an initial investment capital (or wealth) 20 and the initial values for Y, and St as Yo and Su, use ds 1 vector T. lo denote the fractions of wealth invested in the dg assets at time point t. Note that negative values are allowed in indicating short positions. At the terminal time T > 0, the target is to maximize the expectation of a given utility function EUU (27)),
10-
7/s
Utility
0.0
Power Utility
1
0.0
1.5
Exponentis
2
10r
1.5
3.5
20
2.0
7.5
2.5
5.0
Wealth
Wealth
Figure 2: Power and exponential utilities.
Intuitively, a utility function reflects the risk preference of an investor. It is an increasing function of wealth that is also concave: it changes significantly when the wealth low but less so when the wealth high (Figure 2). This work focuses on the power utility 7(z;)) r-==-= with Z R 0, and 7 / 1; and the exponential utility U(2:7) := - exp(->s) with Z = RR and 7 > 0. Here, y is the investor's risk aversion coefficient and is hand-picked (instead of tuned) by the user. A larger y corresponds to more risk aversion, while a smaller y corresponds to more risk tolerance. Note that potentially, one can learn suitable y from data, which is related inverse reinforcement learning (Geng al. 2020 2023). Our method also applicable to other utility functions and other objective functions for port optimization (see Appendix B
Discrete- and Continuous-Time Policies Discrete- and contimuous-time policies are two major types of investment policies, differing on how frequently the portfolio is rebalancod. A discrete-time policy rebalances the fortoolio finitely frequently, leading to a discrete-time dynamics for the wealth. Such policies are often considered in RL methods (Section 2.2 Continuous-tinn time policies rebalance the investment infinitely frequently, leading to a continuous-time dynamics for the wealth. These policies are often found explicitly in continuoustiims finance models (Section 2.3
It impossible reba lance portfolio infinitely frequently in practice, continuous-time policies are mostly analytical tools.
3

---

Published in Transactions on Machine Learning Research (08/2025)
2.2 Deep Deterministic Policy Gradient
We review deep deterministic policy gradient (DDPG, Silver et al. 2014 a quintessential example of RL methods. DDPG does not explicitly model the dynamics, but instead directly learns tiscrete-time policy for portfolio optimization. DDPG parameterizes the policy function as a deep neural network and conducts gradient- based policy learning. Denote by (t, St, Ze, Yx: the deep policy function with parameter dD. Without explicitly modeling the dynamics of St or Xt, DDPG directly maximizes the following performance objective to learn a policy:
max V(OD) with ((OD) == [2T (1)))))
where the expectation is over the terminal wealth z7! (=(=p)) following the policy <<;0p)- A key step is to compute the gradient of (0D)) to update 0D. Following the procedure in Appendix C this can be achieved by sampling the trajectories of S, and Y, to approximate the expectation and thus the gradient of ((O
Typically, DDPG learns a discrete-time policy that rebalances the portfolio finitely frequently. To see how the policy rebalances the portfolio, we study its corresponding wealth process 2z that characterizes the changes in wealth over time. Let At > 0 be the time interval (e.g. daily, weekly) to rebalance the portfolio and, for integer M > O, let T == M At be the fixed investment horizon (e.g. one or two months). At time mat with m E {0, 1,2, ... .M- - 1}, define ThAs := x (m,tt Smat: Zmai, Ymari Bp) as the fraction of current wealth invested in the ith risky asset. Then, the wealth change at time måt is: Zx(10p)
(=(00)) Ede Ma mAt
Smtiie Smar] ) where (Z(110) S(mtiia -S is the wealth change duc to the Smot
2mat Tma Saa
investmment in the 7th risky asset. Note that the number of shares invested in an asset (Zm(sl)) Tmat does not change during (mAt, (m + 1)At): the portfolio rebalances every At time.
Smat
RL methods like DDPG provide flexible representation for factors: the hidden layers of the neural network are considered as the representation learned for Y,, providing strong representation power. Nonetheless, RL methods do not provide an explicit parametric model for the learned representation and asset prices. Consequently, such methods require lots of data and tend to overfit (Aboussalah et. al. 2022).
2.3 Stochastic Factor Models
We review stochastic factor models in continuous- finance. These models can explicitly formulate the dy- namics and can also be used to mathematically derive the functional form of the optimal continuous-tin invest- ment policy. Stochastic factor models are described by stochastic differential equations (SDEs) (see Oksendal 2013 and Appendix D to formulate the dynamics of asset prices S, Specifically, let W. = [W] W?. W." be a dw 1 Brownian motion that characterizes random fluctuations. Let E {1, 21, ... .ds) Then, S, and Yú are assumed to follow
dS; S
dw
+ Σ oed (Y:
(2)
dy. = fy (Y::0' 0g dt + gy (Ya:3) dW.
In Equation 2 fs Rdy + Rds fy Rd + Rdy gs Rd + ded and gy Rdy - Rdw are parametric functions pre-specified by domain knowledge. Further, Is and fr are often referred to as the drift, and gs and gy as the volatility of S, and Y, respectively. Intuitively, SDEs formulate the change of a variable in an infinitesimal time stop as the sum of a deterministic part (dt) and a stochastic part (dW),, and we use 03 to parameterize the SDE. The factor Y, appears both the drift and volatility of the asset prices, thus affecting the price transition. With the parametric functional forms in Equation (2) we can use tools in stochastic optimal control to derive the functional form of the optimal continuous- time investment policy.
Contimuous time policies change the investment in each asset every time point. For continuous time dZ: de #jas; with 2, = -o: So =
investment policy #e. the dynamics of wealth 2 is defined as 2
=
-==1
4

---

Published in Transactions on Machine Learning Research (08/2025)
So and Yo yu- Crucially, this is different from the discrete-time wealth process 2 in Section 2.2 as the number of shares 2, in asset now changes continuously over time, as opposed to being rebalanced at finite intervals. This discrepancy creates obstacles to directly apply the results derived from stochastic factor
models to discrete-time policy learning.
Stochastic factor models reduce sample complexity for portfolio optimization, since the assumed functional forms in Equation 2 significantly constrain the solution space. However, crucial step to apply stochastic factor models is to pick or even construct Y, that perfectly follows a pre-specified model. This step often relies on domain knowledge and thus may end up with oversimplified models suffering from model bias and eventually leading to suboptimal performance.
3 Factor Learning Portfolio Optimization
We now propase factor learning portfolio optimization (FaLPO), new decision-making framework that interpolates between DDPG and stochastic factor models. FaLPO has two components: (i) neural stochastic factor model to handle huge noise and complicated factor effects and (ii) model-rogularized policy learning to combine continuous-line models with discrete-time policy learning methods.
3.1 Neural Stochastic Factor Models
We describe neural stochastic factor models (NSFM) and discuss their benefits. Specifically, neural stochastic factor model is defined as
ds; = f3(( 0z
dw
Σ
g3 Xx: 02) dW??
S
dX, fx, 0z) + gx (x): dW.
X, (Y);
On the one hand, neural stochastic factor model assumes the existence of representation function ø for factors such that X; o(Y;; 65) along with asset prices S, follows a stochastic factor model (e.g. Equations (2) and (6), which reduces the sample complexity and avoids overfitting. On the other hand, á is formulated neural network with parameter 03. As result, FaLPO avoids hand-picking factors following SDEs as is the case in stochastic factor models (Section 2.3). The neural network representation has only few parametric constraints and thus able to capture complicated factor effects in the data. Note that the notation of X, is only provided for the case exposition: it does not represent any external data other than the factors Y. demonstration of NSFM is provided in Figure 3
3.2 Model-Regularic Policy Learning
Under the proposed neural stochastic factor model, we aim to learn the discrete-time optimal policy function of and the representation function ((:;0)) However, while the policy learning is for discrete-time policies, our proposed model is in continuous time. To bridge this gap, we incorporate two types of contimuuus-time model regularization into discrete-time policy learning: (i) the policy functional form of Equation and (ii) the model calibration objective of Equation (4)
Policy Functional Form From our model, we apply the functional form of a continuous time optimal policy into our discrcto-time policy learning. Using tools in stochastic optimal control, we can derive the functional form of an optimal continuous-time poliey: #े = [I((,, S, Z₁, X;) where the functional form of IT can be obtained in many existing stochastic factor models {Kim & Omberg 1996, Chacko & Viceiral 2005 Avanesyan 2021 Zariphopoulou 2001 Wachter 2002 Kraft 2005), and 0g is an optimal parameter for II.
FaLPO uses the functional form of II in policy learning and parameterize the candidate policy as
(1, St, Ze, Yo 0g 0x) IT,,, Z, <(Y;; (Vv::o));0),
(3)
5

---

Published in Transactions on Machine Learning Research (08/2025)
Seerr
Stochastic Factor
Models
X
(()
Factor Representation
Not Specified
Y
Ya
Time
Figure 3: Demonstration for neural stochastic factor models.
Algorithm FaLPO Algorithm
1: Input: number of iterations N. Initialize 0g and 0,
for [MV] do Parameterize the policy function according to Equation
3
5: Estimate the policy gradient for H in Equation (5)
(Appendix
6: Update Os. On, and Os.
7: end for
8: Return 0. 0x)
where ó the representation function for the factors in Section 3.1 space and acts regularization. Importantly, although IT is derived for contimuous-time policics, it can still provide guidance for discrete-time policy learning when At is small. We rigorously prove the soundness of using Equation
As result, II constrains the policy
33] in Section
Model Calibration FaLPO also hinges on model calibration to regularize policy learning. Given the specific functional forms
2 FaLPO conducts model calibration to estimate the parameters of the SDE. The calibration procedure can be summarized as maximizing model calibration objective:
max L(fs,
(4)
0g
practice, with discrete data, one may use likelihood (Phillips 1972 Beskos Roberts 2005 or other
likelihood- based objective functions (Bishwal 2007 Ait-Sahalia & Kimmel 2010 for L (sce Appendix E for concrete examples).
To harness the information provided by model calibration in policy learning, FaLPO combines the model calibration ovjective L in Equation 4 with the performance objective V in Equation [ and facilitates a joint optimization over the two. Note that naively combining L(0s, 0s) and ((Op) is not effective since the two in general do not share common parameters: the paramcter of the policy network θD has no overlap with the SDE parameter Os or factor representation parameter 0g- However, by constraining the policy space Equation 3 in FaLPO, we can show that 0g is also part of the policy parameterization. Thus, V can be derived V(es, 0x) (27(((00)))) )). In other words, 0g is shared in both V and L, and hence FaLPO can carry out joint optimization over the two:
max 0g.000s)e4.
H(0g, 0z 0s), with
(5)
(00,, 0z. 0s) = (0. 0x) + AL(As,
where the candidate policy follows the functional form of the optimal continuous-time policy Equation 3 (see Algorithm D. and A denotes the considered parameter set. The model calibration objective also acts model regularization, where 10 is hyperparameter determining its effect. In practice, we

---

Published in Transactions on Machine Learning Research (08/2025)
optimize Equation [5] by gradient- baste methods, facilitating easy and end-to-end learning procedure (see Appendix for gradient estimation details).
3.3 FaLPO example
In portfolio optimization, one can use different types of stochastic factor models. FaLPO can be applicd to many such lypes (see Appendix F.. In this section, we use the Kim- Oumbrg model (Kim & Omberg 1996 as an example to illustrate FaLPO's modeling and policy learning. Kim-Omberg is a standard model for portfolio optimization with stocahstic factors, which has been extensively studied empirically (Welch & Goyal 2008) Muhle-Katee Karbe al. 2017). For modeling, FaLPO with Sim-Omberg model formulates the dynamics of asset prices and factors as
ds; Xi dt
dw
Σ 0 aw? j=1
S
(6)
dX,- = pاw X1) dt + ddW and X, = ((Y(;
where SDE parameters 4; a, and μ are constant matrices or vectors.
For policy learning, we detail the policy functional form and model calibration. Under the Kim-Omberg model, we can derive the optimal policy functional form II in Equation (3. Specifically, for power utility IT(t, Sr, Zz (Y: 0g); 0-) = hi (t: 0z (Y{;; 0g) + (2; 0z), for exponential utility [T,,, St Ze, (YY;; (Yr; ki (t; ((r)))((;)))))))))))
(ttt where ki (-: 0x) [0, + Rdxxdx and A2(:;0x) [0, 27 + (Rdsxds are two time dependent functions (Appendix F.2. We can derive the functional forms of K and l/ since the two are solutions to systems of ODEs related to algebraic Riccati equations (Appendix Gb. We also dircctly use function approximators like neural networks or kernel methods for the two. For model calibration, we use a negative mean square loss with the derivation deferred to Appendix
(s),00s)
-E Eds =1 [log(S)+<r) - log(S;) of (Y: 0/] where in this case 0g is a dg X vector. Therefore, to implement FaLPO, we can parameterize the candidate policy function using and optimize Equation 5
Note that the methodology of FaLPO is also generally applicable to other decision-making problems besides portfolio optimization. In Appendix H we use linear quadratic control with stochastic factors example to demonstrate the generality of FaLPO.
4 Theory
We theorctically analyze both the asymptotic and non-asymptotic characteristies of FaLPO.
4.1 Asymptotic Analysis
FaLPO applies the policy functional form and model calibration derived from contimous-time models to discrete-time policy learning. We show that FaLPO can achieve the optimal performance asymptotically (i.e. with infinite data and perfect optimization). In the following, we describe the assumptions and results and provide the formal theorem in the end.
We provide an intuitive description on the assumptions, with the formal statements provided in Appendix I.I First, we assume that the portfolio optimization problem satisfies some standard regularity conditions (Higham et al. 2002): the drift and volatility are locally Lipschitz continuous; meanwhile, the asset prices, the stochastic factors, and the wealth process under the optimal policy have bounded moments. Second, we assume that the utility function (=) has linear growth on z E Z. Note that some widely used cases like power utility with wealth satisfy this assumption. Third, we consider only and we assume that covers the optimal contimuous-time policy
< 1 and exponential utility with admissible policies with parameters parameter. A policy is admissible if it predictable and the wealth process Z E Z for any t e [0, almost surely. It is a common practice to only consider such admissible policies in portfolio optimization.
7

---

Published in Transactions on Machine Learning Research (08/2025)
The last two assumptions are artifacts of the current theoretical analysis; in practice FaLPO can achieve reasonable performance without enforcing them (see Section
With the foregoing assumptions, we show that the performance of FaLPO can asymptotically converge to that of the best policy in discrete time. Define VA := V(=*) where x an optimal discrete-time admissible policy with time interval At. That is Vā is the optimal performance obtained without constraining to the functional form of Equation 3 or leveraging model calibration like Equation (5). Define 0% = (0 Ae 0 (,,))) e arg (0,00 A (f x. 0g) with the policy functional form of Equation
3 such that V(6% is the
performance that FaLPO achieve with infinite data and perfect optimization. Then, Theorem shows that the gap between Vå and (.) converges to zero as At goes to zero.
Theorem 4.1. With assumptions above, limar-o (Vă+ v()))) = 0.
Theorem
4.1 justifies the methodology of FaLPO under a small time interval. Proof in Appendices 1.2 and L33
4.2
Asympttti Analysis
We now study the finite-sample performance of FaLPO. In each iteration, we collect B independent trajectories estimate the gradicnt of H. Let 6,, be the estimate after the th itcration, and N the total number of
iterations. We analyze the average estimate 0 0= Ingg" On instead of ON, which is common technique for stochastic optimization analysis. Specifically, we aim to bound the expected difference between V&r and V()). Note that it is extremely challenging to theoretically analyze a non-convex stochastic optimization equation 5 without further specifications in problem setup and assumptions (Polyak 1963 Bhandari & Russo 2024 Jin
ct al. 2021 Ma 2021 Wang et al. 2020). Therefore, we consider a projection-basee variant of FaLPO, under which the oplimization process is conducted in a bounded parameter set B A. Furthermore, we assume that the objective function H is strongly concave in B with a local maximal point 0%, := (0) Pr.... (.AA.) Similar local curvature assumptions are commonly used to analyze non-convex problems (Bach et al. 2017 Loh 2017). With the above setup and assumptions (also detailed in Appendix J.2 the expected gap between Vir and (00 satisfies the following sampl bound in Theorem 4.2
Theorem 4.2. With the aforementioned projection-based FaLPO algoritlm and assumplions, there exist positive constants C₁, O2, O3, and C4 such that
BIVĀ V(0)
Ext
(1(as) H(O))
C Nog(NN
1
+
N(1 - -)
(7)
Ci log(N) BN(1- A)
(1-A))²C + X²Os + 22(1 - A)Ca]
where A [0, and ear is ervor term not related to N or B but dependent on At with lim eat 0.
Theorem 4.2 provides a non-asymptotic upperbound on the gap between the optimal performance V and the one achieved by FaLPO (ō). We briefly comment on each term in the upperbound. 1 bounds the asymptotic performance gap caused by leveraging the contimuuu--time policy functional form constraint and model calibration, and we explain its connection to Theorem 4.1 in Appendix J.4
H(0%.)-H()..) controls the
1-X
performance gap between the local optimal point ot and the global optimal point 0z.- The remaining terms characterize the performance gap between 0 and 6)r-
Theorem 4.2 has two implications. First, the bound in Equation 7 is a rational function of A. Accordingly, there exist situations where a À e (0, 1) results a smaller upper bound than x = 0, indicating the possibility that tuning can provide better performance (sce experiments in Appendix J]. Second, when 1, the
bound diverges to infinity. This makes sense since when À = 1, H (0,, 0z 0g) ((0g.0s) does not contain f,:
the algorithm does not learn the policy. We prove the theorem in Appendices J.1 J.3 and J.4
8

---

Published in Transactions on Machine Learning Research (08/2025)
5 Experiments
By incorporating outinuous-time finance models into policy learning, FaLPO can deal with high data noise and complex factor effects. In this section, we demonstrate the improved performance of FaLPO against existing portfolio optimization methods, over synthetic and real-world experiments.
5.1 Synthetic Experiments
Metrics We compare different methods using the average terminal utility since it is the ultimate goal in our portfolio opti- mization problem formulation and is com- monly used in continuous-time finance models. There exist other statistics mea- suring the performance of portfolios (see Section Bj. These statistics are not equiv- alent or consistent with the utility, and thus we do not emphasize them.
Tahle 1: Competing methods and their characteristics.
Methods
Factor Representation
Parumetric Modeling
Joint Optimization
MMMC
x
x
DDPG
SLAC
RichlD
V/
CT MIB-LBL
*
V
X
FaLPO
V
Methods FaLPO combines 1) flexible factor representation learning, 2) parametric modeling, and 3) joint optimization of policy learning and model calibration. To demonstrate the performance improvement of the combination of these three features, we compare FaLPO with five competing methods in prior work (Table [ (i) Merton Model with Model Calibration (MMMC): model calibration of a classic continuous- time finance bascline, which does not consider stochastic factors [Merton] 1969). (ii) Deep Deterministic Policy Gradient (DDPG): a tfato-of-the-art model-free RL method with deterministic policy (Silver et al.] 2014). which is at the core of many empirical portfolio optimization methods (Appendix (A). DDPG directly treats factors input of neural networks for policy without parametric modeling and joint optimization. (iii) Stochastic latent actor critic (SLAC): a state-of-the-arr representation lcarning RL method that jointly learns the policy and the representation of variables (a.k.a factors in our case) (Lee el al. 2020). But SLAC does not use any parametric models in representation learning or policy learning. (iv) Model-based RL with rich observations (RichID): a state-of-the-art model-based RL method with representation learning (Mhammedi et al., 2020). RichID learns the representation of factors using parametric models, but it does not conduct joint optimization on performance and model calibration like FaLPO. (v) Continuous-time model-based R.L (CT-MB-RL): modcl-based policy gradient where factors fed into the policy functional dcrived from continuous-tine models (Herzog et al. (0044) without representation learning. Nole that all the considered methods except MMMC observe the same data with stock prices St and factors Y..
For policy gradient methods, we pick a deterministic policy approach like DDPG as, when compared to non-deterministic policy gradient alternatives, they are more suitable to portsolio optimization due to the continuous action space, high exploration cost, and high noise in financial data (Appendix A.2). For portfolio optimization, different variance reduction methods for policy gradient (Schulman et al. 2015 2017 Xu et al. 2020) only provide minor performance improvements (Aboussalah et al. 2022). We hence do not report such results. For areas of RT, with representation learning and model-based RI. we focus on those (SLACC and RichID) explicitly learning a representation of latent variables, since such methods are more closely related to FaLPO. There are other techniques like data augmentation, feature construction, adversarial training, and regularization that can improve the empirical performance of portsolio optimization (See Hambly et al. 2023 and Appendix Ap. In this work, we focus on the central methodological task of policy learning, and most such techniques can be directly combined with our proposed method.
Protocol We simulate environments with the Sim-Omberg model and implement the considered methods to compare their performance. Note that key data generating parameter in portfolio optimization is the signal-to-noise ratio, which can be roughly characterized by the ratio between the scale of the drift and the scale of the volatility (see Appendix K.1 for detailed explanation). We test our method under different signal-to-noise ratios. To this end, we randomly generate stock drifts around 10% (to mimic the real-world average return of stocks in SP500), vary the scale of volatility in (10% 20%, 30%), and simulate data following the procedure in Appendix K.2 Then, we apply the considered methods to maximize the terminal power and
9

---

Published in Transactions on Machine Learning Research (08/202))
Table 2: Average terminal utility after tuning with standard deviation for synthetic data
Annual Volatility
0.1
0.2
0.3
FaLPO
0.465 + 0.446
1.35 + 0.155
-2.737 + 0.219
DDPG
-1.650 + 0.456
-3.30
1.294
-5.495 1.269
SLAC
-0.750 + 0.210
-5.50 + 0.011
-6.160 0.012
RichID
-3.350 0.111
-5.65 + 0.102
-6.325 + 0.048
CT-MB-RL
-
-5.35
0.020
-6.160 0.026
MMMC
-4.723 7.619
-5.602 + 4.299
-6.124 + 3.217
Table 3: Average terminal utility for real-world data. Mix denotes mix of stocks in the previous three sectors.
Methods
Energy
Material
Industrials
Mix
FaLPO
2.4 E 1,9
-3.2 1 1.0
6.3 2.3
3.5 + 1.5
DDPG
-6.6 12
-7.3
-7.3 2.1
-2.5 x 104 3.3 x 10
SLAC
-68
-7.0 +
-342.4 886.8
-3.0 x 10 +.3 1012
RichID
-..110.11
-6.9 1.4
6699
1.9
CT-MB-RL
6.2
4.
-11655
32947.5
-.7
MMMC
-8.547.6
1.7
-11.0 5.4
44.
exponential utility with different y's. For cach mothod, we tune the learning rate and other method-specific hyperparameters with early stopping (Appendix K.3 With each geelood-ipeeppraaneee--vviiiiooo combination, we repeat training, validation, and testing five times.
Results For exponential utility maximization, Table summarizes the average test utility after hyperparameter selection with 10 stocks, 10 factors, and = 5. FaLPO outperforms all the competing methods in terms of average terminal utility. This performance gain may be explained by the factor representation learning informed by the continuous- time finance model, as other methods are incapable of doing so. Meanwhile, MMMC and CT-MB-RL underperform, which suggests the disadvantage of using oversimplified models. Compared with the more sophisticated RI. methods like SLAC and RichID, the simple DDPG is fairly competitive. This is consistent with the existing observation that always be suitable for portfolio optimization due to the large noise and idiosynerasy in the data. In the appendix, we provide additional experimental results with different problem dimensions (Appendix K.4.1). other y values (Appendix K.4.2), and different utility functions (Appendix K.4.3). The results are consistent. Appendix K.4.4 studies a simplified case, where the optimal performance can be mathematically derived. FaLPO achieves a similar performance as the theoretically optimal one.
complicated RL solutions may not
5.2 Real-World Stock Trading
In this section, we present an application of FaLPO real-world stock trading problems. Following the synthetic portfolio optimization setup, we study the six considered methods for 21-days (one month) stock trading in four different stock sectors using the daily stock price data from Yahoo finance between January 4, 2006 and April 1, 2022. For factors, we follow existing works (Aboussalah et al. 2022 De Prado 2018 Dixon ct al. 2020) and consider economic indexes, technical analysis indexes, and soctor-specific factors such as oil prices, gold prices, and related ETF prices, leading to around 30 factors for each sector. In each sector we select 10 stocks according to the availability and trading volume the considered time range (Appendix L.]) The training, validation, and testing data are constructed using rolling windows (Appendix L.3). Table 3 reports the achieved average utility of each method under the selected hyperparamters. FaLPO achieves the highest average utility in all four sectors.
10

---

Published in Transactions on Machine Learning Research (08/2025)
0.8
Average Return
0.6
0.4
0.2
0.0
Q1 2018
Q4 2018
Q1 2020
Q1 202:
Date
Figure 4: FaLPO average return portfolio terminal dates.
Next, we conduct the training-tuning-ten tuning-tusttiinn teetting procedure above with E {5, 7.9), and report the returns of FaLPO in each quarter Figure Recall that smaller y corresponds taking more risk. This consistent. with the observation in Figure that the smaller the y the bigger the return but the larger the fluctuations. Also, the return fluctuates and drops around late 2018 and early 2020. The former corresponds to the abrupt bear market at the end of 2018, and the latter is consistent with the time period that COVID-19 bursts. Under these two time periods, the financial market was especially noisy and unprodictable. In Appendices L.6 to L.S. we conduct additional experiments to further demonstrate the characteristics of FaLPO under various scenarios including discussions on transaction costs, sensitivity analysis, and different initial wealth First, we provide experimental results with short-selling transaction costs in Appendix L.6 This is in contrast to many existing works of utility maximization (Merton) 1969 Kim Omberg 1996. Avanesyan 2021 Guan Liu 2021), which do not consider transaction costs even with shorting. We also believe that further studying other market frictions is interesting direction for future work. Second, we implement sensitivity analysis À in Appendix L.4 and observe that non-zero small À works well in practice. Then, compare different methods using Sharpe ratio and largest drawdon in Appendix L.7 Finally, we experiment with different initial wealth values with results provided in Appendix L.8
6 Conclusion and Future Work
This work proposes FaLPO, new decision-making framework for portfolio optimization with stochastic factors. By using continuous-time finance models to regularize policy learning, FaLPO is able to handle high noise and complex effects in financial data.
FaLPO has two potential limitations. First, while we show the extension FaLPO to problems beyond portfolio optimization, FaLPO is not applicable when there no suitable parametric model. Second, the performance of FaLPO still relies on good factors (Yi). In the presence of unpredictable market events (like COVID-19), when the factors do not contain any useful signals (like the Merton case in Appendix K. K.4.4 additional caution needs to be taken.
Future work includes applying FaLPO to other areas like paycheck optimization (Alaluf et al.
2023 circuit
design (Mina et al. 2023), discase treatment (Geng et al. 2019a),, and diagnosis (Geng et al.
2018
2019b
Kuusisto al. 2013). And leveraging other learners for the policy function (Li el al.] 2024 Hazimel el. al.
2020 Gorishniy al. 2021
References
Amine Mohamed A bboussalah, Ziyun Xu, and Chi-Guhn Lee. What is the value of the cross-sectional approach to deep reinforcement learning? Quantitative Finance, 2296):1011 2022.
Alekh Agarwal, Sham M Kakade, Jason Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine. Learning Research, 22
11

---

Published in Transactions on Machine Learning Research (08/2025)
(98):1-76, 2021.
Yacine Ait-Sahalia. Closed-form likelihood expansions for multivariate diffusions. The Annals of Slatistics, 36(2):906-99) 937, 2008.
Yacine Ait-Sahalia and Robert L. Kimmel. Estimating affine multifactor term structure models using closed-form likelihood expansions. Journal of Pinancial Economics, 98(1):113 144, 2010.
Melda Alaluf, Giulia Crippa, Sinong Geng, Zijian Jing, Nikhil Krishnan, Sanjeev Kulkarni, Wyatt Navarro, Ronnie Sircar, and Jonathan Tang. Reinforcement learning paycheck optimization for multivariate financial goals. Risk and Decision Analysis, 9(1):11-18, 2023.
Levon Avanesyan. Optimal Investment in Incomplete Markets with Multiple Brownian Enternalities. PhD thesis, Princeton University, 2021.
Levon Avanesyan, Mykhaylo Shkolnikov, and Ronnie Sircar. Construction of forward performance processes in stochastic factor models and an extension of Widder's theorem. Finance and Stochastics, 1184)1181 2020.
Stephen H Bach, Bryan He, Alexander Ratner, and Christopher Ré. Learning the structure of generative models without labeled data. In International Conference on Machine Learning, pp. 273 282, 2017.
Krzysztof Bartoszek, Sylvain Glémin, Ingemar Kaj, and Martin Lascoux. Using the Ornstein Uhlenbeck process to model the evolution of interacting populations. Journal of Theorelical Biology, 429:35 45, 2017.
Maximilian Behr, Peter Benner, and Jan Heiland. Invariant Galerkin Ansatz spaces and Davison-Makt methods for the numerical solution of differential Riccati equations. Applied Mathematics and Computation, 410:126401, 2021.
Yoshua Bengio. Using financial training criterion rather than prediction criterion. International Journal of Neural Systems, 8(4):433 443, 1997.
Alexandros Beskos and Gareth O. Roberts. Exact simulation of diffusions. The Annals of Applied Probabilily, 15(4):2422-244) 2005.
Jalaj Bhandari and Daniel Russo. Global optimality guarantees for policy gradient methods. Operations Research, 1996) 2024.
Homanga Bharadhwaj, Mohammad Babacizadch, Dumitru Erhan, and Sergey Levine. INFOrmation Priori- lization through EmpoWERmeet in Visual Model-Based RL. In International Conference on Learning Representations, 2022.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https: //www/ wandb. com/ available from wandb.com.
Software
Jaya P. N. Bishwal. Parameter Estimation in Stochastic Differential Equations. Springer, 2007.
Tomas Björk. Arbitrage Theory in Continuous Time Oxford University Press, 2009.
Simone Blomberg, Suren I Rathnayake, and Cheyenne M Morcau. Beyond brownian motion and the Ornstein Uhlenbeck process: Stochastic diffusion models for the evolution of quantitative characters. The 2020.
American Naturalist, 195(2):145
George Chacko and Luis M Viceira. Dynamic consumption and portfolio choice with stochastic volatility in incomplete markets. The Review of Financial Studies, 18(4):1369 1402, 2005.
Yevgen Chebotar, Karol Hausman, Marvin Zhang, Gaurav Sukhatme, Stefan Schaal, and Sergey Levine. Com- bining model-based and model-free updates for trajectory-centti reinforcement learning. In International Conference Machine Learning, pp. 703-177770 -11, 2017a.
12

---

Published in Transactions on Machine Learning Research (08/2025)
Yevgen Chebotar, Mrinal Kalakrishnan, Ali Yahya, Adrian Li, Stefan Schaal, and Sergey Levine. Path integral guided policy search. In International Conference on Robolics and Aulomation, pp. 3381-3388 2017b.
Marco Corazza and Francesco Bertoluzzo. Q-learning- based financial trading systems with applications. Technical report, University Ca Foscari of Venice, Dept. of Economics, 2014.
Marcos Lopez De Prado. Advances in Financial Machine Learning. John Wiley & Sons, 2018.
Mare Deisenroth and Carl E Rasmussen. PILCO: model-based and data-efficient approach policy search. In International Conference on Machine Learning, pp. 465 472, 2011.
Matthew Dixon, Igor Halperin, and Paul Bilokon. Machine Learning in Finance. Springer, 2020.
Kenji Doya. Reinforcement learning in contimuous time and space. Newral Compulation, 12(1):219 245, 2000.
Benjamin Eysenbach, Alexander Khazatsky, Sergey Levine, and Russ R Salakhutdinov. Mismatched no more: Joint model-policy optimization for model-based RL. In Advances in Neural Information Processing Systems, volume 35, pp. 18230-232400 2022.
Eugene F Fama and Kenneth R French. The cross-section of expected stock returns. The Journal of Finance, 47(2):427-46) 427,466,, 1992.
Eugene F Fama and Kenneth R. French. A five-factor asset pricing model. Journal of Financial Economics, 116(1):1-22, 2015.
Vicky Fasen. Statistical estimation of multivariate Ornstein Uhlenbeck processes and applications to integration. Journal of Econometrics, 172(2):325 337, 2013.
Damir Filippoié and Eberhard Mayerhofer. Affine Diffusion Processes: Theory and Applications, pp. 125-164. De Gruyter, 2009.
Wendell Fleming and Raymond Rishel. Determinislic and Stochastic Optåmal Control. Springer, 1975.
Wendell H Fleming and Sanjoy K Mitter. Optimal control and nonlinear filtering for nondegenerate diffuusion processes. Stochastics: An International Journal of Probability and Stochastic Processes, 8(1):63 77, 1982.
Wendell H Fleming and Halil Mete Soner. Controlled Markov Processes and Viscosity Solutions. Springer, 2006.
Jean- Pierre Fouque, Ronnie Sircar, and Thaleia Zariphopoulou. Portfolio optimization and stochastic volat ility asymptotics. Mathematical Finance, 27(3):704- 745, 2017.
S Geng, Z Kuang, P. L. Peissig, D Page, L Maursetter, and K. E. Hansen. Parathyroid hormone independently predicts fracture, vascular events, and death in patients with stage 3 and 4 chronic kidney discase. Osteoporosis International, 30:2019 2025, 2019a.
Sinong Geng, Zhaobin Kuang, Peggy Peissig, and David Page. Temporal poisson square root graphical models. In International Conference on Machine Learning, pp. 1714-1723, 2018.
Sinong Geng, Minhao Yan, Mladen Kolar, and Sanmi Koyejo. Partially linear additive gaussian graphical models. In International Conference Machine Learning, pp. 2180 2190, 2019b.
Sinong Geng, Houssam Nassif, Carlos Manzanares, Max Reppen, and Ronnie Sircar. Deep PQR: Solving inverse reinforcement learning using anchor actions. In International Conference Machine Learning, pp. 3431-3441, 2020.
Sinong Geng, Houssam Nassif, and Carlos A. Manzanares. A data-driven state aggregation approach for dynamic discrete choice models. In Uncertainty in Artificial Intelligence (UAI), pp. 647-657, 2023.
Sinong Geng, Aldo Pacchiano, Andrey Kolobov, and Ching-An Cheng. Improving offiine RL by blending heuristics. In International Conference on Learning Representations, 2024.
13

---

Published in Transactions on Machine Learning Research (08/2025)
Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, Aviv Tamar, et al. Bayesian reinforcement learning: A survey. Foundations and Trendsh in Machine Learning, ((5-6):399 (3359-44,, 2015.
Xavier Glorot and Yoshua Bengio. Understanding the difficully of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics, pp. 249-256, 2010.
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34:18932 18943, 2021.
Amit Goyal and Pedro Santa-Clas Idiosyncratic risk matters! The Journal of Finance, 58(3):975- 1007,, 2003.
Shixiang Gu, Timothy Lillicrap, Πya Sutskever, and Sergey Levine. Contimuous deep Q-learning with model-based acceleration. In International Conference on Machine Learning, pp. 2829 2838, 2016.
Mao Guan and Xiao- Liu. Explainable deep reinforcement learning for portfolio management: An empirical approach. In International Conference on Al in Finance, pp. 1 9, 2021.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pp. 1861 1870, 2018.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference Machine Leurning, pp- 2019.
Ben Hambly, Renyuan Xu, and Huining Yang. Recent advances in reinforcement learning in finance. Mathematical Finance, 33:37-503, 2023.
Elad Hazan and Satyen Kale. Beyond the regret minimization barrier: An optimal algorithm for stochastic strongly-convex optimization. In Conference Learning Theory, pp. 421-436, 2011.
Hussein Hazimeh, Natalia Ponomareva, Petros Mol, Zhenyu Tan, and Rahul Mazumder. The tree ensemble layer: Differentiability meets conditional computation. In International Conference on Machine Learning, pp. 4138-4148, 2020.
Florian Herzog, Gabriel Dondi, Hans P Geering, and Lorenz Schumann. Continuous-time multivariate strategic asset allocation. In Annual Meeting of the German Finonce Association, pp. 1-34, 2004.
Desmond J Higham, Xucrong Mao, and Andrew M Stuart. Strong convergence of culer-type methods for nonlinear stochastic differential equations. STA M Journal on Numerical Analysis, 40(3):1041110000 1063, 2002.
Vladimír Holy and Petra Tomanová. Estimation of Ornstein Uhlenbeck process using altra-high-frequeenn data with application to intraday pairs trading strategy. arXie preprint 181iv:1811.0000 1009312 2018.
Guosheng Hu, Yuxin Hu, Kai Yang, Zehao Yu, Flood Sung, Zhihong Zhang, Fei Xie, Jianguo Liu, Neil Robertson, Timpathy Hospedales, and Qiangwei Miemie. Deep stock representation learning: From candlestick charts to investment decisions. In International Conference on Acoustics, Speech and Signal Processing, pp. 2706 2710, 2018.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in Neural Information Processing Systems, 32, 2019.
Zhengyao Jiang, Dixing Xu, and Jinjun Liang, A deep reinforcement learning framework for the financial portfolio management problem. arXiv preprint urXio:1706 (ooe:1006..009, 2017.
Chi Jin, Pranecth Netrapalli, Rong Ge, Sham M Kakade, and Michael Jordan. On nonconvex optimization for machine learning: Gradients, stochasticity, and saddle points. Journal of the ACM, 68(2):1-22) 2021.
Hilbert J Kappen. Path integrals and symmetry breaking for optimal control theory. Journal of Statistical Mechanics: Theory and Experiment, 2005(11):P11011, 2005.
14

---

Published in Transactions on Machine Learning Research (08/2025)
Hilbert Johan Kappen and Hans Christian Ruiz. Adaptive importance sampling for control and inference.
Journal of Statistical Physics, 162(5):1244-1266 2016.
Toannis Karatzas and Steven Shreve. Browmian Motion and Stochastic Calculus. Springer, 1987.
Belhal Karimi, Blazej Miasojedow, Fric Moulines, and Hoi-To Wai. Non-asymptotic analysis of biased stochastic approximation scheme. In Conference on Learning Theory, pp. 1944 1974, 2019.
Taesup Kim, Sungjin Ahn, and Yoshua Bengio. Variational temporal abstraction. Advances in Newral Information Processing Systems, 32, 2019.
Tong Suk Kim and Edward Omberg. Dynamic nonmyopic portfolio behavior. The Review of Financial Studies, 9(1):141-161, 1996.
Holger Kraft. Optimal portfolios and Heston's stochastic volatility model: An explicit solution for power utility. Quantitative Finance, 5(3):303 313, 2005.
Finn Kuusisto, Inês Dutra, Houssam Nassif, Yirong Wu, Molly E Klein, Heather B Neuman, Jude Shavlik, and Elizabeth Burnside. Using machine learning to identify benign cases with non-definitive biopsy. International Conference on E-Heallhh Networking, Application Services, pp. 283 283-285, 2013.
Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. URLB: Unsupervised reinforcement learning benchmark. In Conference Newral Information Processing Systems, 2021.
Alex X Lee, Anusha Nagabandi, Picter Abbecl, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. In Advances in Newral Information Processing Systems, volume 33, pp. 741-752, 2020.
Boyao Li, Alexander J Thomson, Houssam Nassif, Matthew M Engelhard, and David Page. On neural networks infinite tree-structured probabilistic graphical models. Advances in Neural Information Processing Systems, 37:4598-4666 4628, 2024.
Zhaoqi Li, Houssam Nassif, and Alex Luedtke. Estimation of subsidiary performance metrics under optimal policies. Statistion Sinica, 37(3,, 2027.
Xiao Xiao-Yang Liu, Hongyang Yang, Jiechao Gao, and Christina Dan Wang. FinRL: Deep reinforcement learning framework to automate trading in quantitative finance. In International Conference on Al in Finance, pp. 2021.
Po-Ling Loh. Statistical consistency and asymptotic normality for high- high-dimensional robust m-estimators. The Annals of Statistics, (52))6866 896, 2017.
Tengyu Ma. Why Do Local Methods Solue Noncoweer Problems?, pp. 465-485. Cambridge University Press, 2021.
Sidra Mehtab and Jaydip Sen. A robust predictive model for stock price prediction using deep learning and natural language processing. In International Conference on Business Analylics and Intelligence, 2019.
Robert C Merton. Lifetime portfolio selection under uncertainty: The contimuous-time case. The Review of Economics and Statistics, pp. 247 257, 1969.
Robert C Merton. Continuous- Time Finance. Blackwell, 1992.
Robert C Merton et al. An intertemporal capital asset pricing model. Econometrica, 41(5):867 887, 1973.
Zakaria Mhammedi, Dylan J Foster, Max Simchowitz, Dipendra Misra, Wen Sun, Akshay Krishnamurthy, Alexander Rakhlin, and John Langford. Learning the linear quadratic regulator from nonlinear observations. A Advances in Newral Information Processing Systems, 45344-154443 2020.
15

---

Published in Transactions on Machine Learning Research (08/2025)
Rayan Mina, George E Sakr, and Houssam Nassif. Enhancing transistor sizing in analog IC design using circuit-focused semi-supervised learning. In International Mullidisciplinary Conference on Engineering Technology, pp. 223 228, 2023.
Johannes Murle-Karbe, Max Reppen, and H. Mete Soner. A primer on portfolio choice with small transaction costs. Annual Review of Financial Economics, 9:301-331, 2017.
Rémi Munos. Policy gradient in continuous time. Journal of Machine Learning Research, 7:771-791, 2006.
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In International Conference on Robotics and Automation, pp. 7559 7566, 2018.
Abhishek Nan, Anandh Perumal, and Osmar R Zaiane. Sentiment and knowledge based algorithmic trading with deep reinforcement learning, In International Conference Databose and Expert Systems Applications, pp. 167-180, 2022.
Arkadi Nemirovski, Anatoli Juditsky, Guanghui Lan, and Alexander Shapiro. stochastiic approximation approach to stochastic programming. SIAM Journal on Optimization, 19(4):1574-109, 2009.
Bernt Oksendal. Stochastic Differential Equations: An Introduction with Applications. Springer, 2013.
Matteo Papini, Damiano Binaghi, Giuseppe Canonaco, Matteo Pirotta, and Marcello Restelli. Stochastic variance-reduced policy gradient. In International Conference Machine Learning, pp. 4026 4035, 2018.
P. C. B. Phillips. The structural estimation of a stochastic differential equation system. Econometrica: Journal of the Econometric Society, pp. 1021-1041, 1972.
Peter C. B. Phillips and Jun Yu. Maximum likelihood and gaussian estimation of contimuous-tine models in finance. In Handbook of Financial Time Series, pp. 497-530. Springer, 2009.
Boris T Polyak. Gradient methods for the minimisation of functionals. USSR Computational Mathematics and Mathematical Physics, 3(4):864 878, 1963.
Konrad Rawlik, Marc Toussaint, and Sethu Vijayakunar. On stochastic optimal control and reinforcement learning by approximate inference. In International Joint Conference on Artificial Intelligence, pp. 3052 3056, 2013.
A. Max Reppen and H. Mete Soner. Bias-variance trade-off and overlcarning in dynamic decision problems. Xixiv preprint arxiv:2011, orXi::2011.03449, 2020.
Rori V Rohlfs, Patrick Harrigan, and Rasmuu Nielsen. Modeling gene expression evolution with an exlended Ornstein Uhlenbeck process accounting for within-species variation. Molecular Biology and Evolution, 31 (1):201-211, 2014.
John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. Xiv prepriat 1ow::1777.06334. 2017.
Shai Shalev-Shwartz, Yoram Singer, Nathan Srebro, and Andrew Cotter. Pegasos: Primal estimated sub-gradient solver for SVM. Mathematical Programmingg 127(1):3 30, 2011.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International Conference on Machine Learning, pp. 387-395, 2014.
Freek Stulp and Olivier Sigand. Path integral policy improvement with covariance matrix adaptation. In International Conference on Machine Learning, pp- 1554, 2012.
16

---

Published in Transactions on Machine Learning Research (08/2025)
Jingrui Sun and Jiongmin Yong. Stochastic Linear- Quaaratiicc Optimal Control Theory: Open-Loop and Closed-Loop Solulions. Springer Nature, 2020.
Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.
Peter Tankov. Pinancial Modelling with Jump Processes. CRC Press, 2003.
Evangelos Theodorou, Jonas Buchli, and Stefan Schaal. A generalized path integral control approach to reinforcement learning. The Journal of Machine Learning Research, 11:3137-3181, 2010.
Brandon Trabucco, Xinyang Geng, Aviral Kumar, and Sergey Levine. Desigu-beuch: Benchınrkk for data-driven offline model-based optimization. In International Conference on Machine Learning, pp- 21677 21691, 2022.
Vladimir Vapnik. Principles of risk minimization for learning theory. In Advances in Neural Information Processing Systems, pp. 831 838, 1992.
Oldrich Vasicek. An equilibrium characterization of the term structure. Journal of Financial Economics, 5 (2):177-188, 1977.
J. von Neumann and O. Morgenstern. Theory of Gomes and Economic Behavior. Princeton University Press, 1947.
Jessica Wachter. Portfolio and consumption decisions under mean-reverting returns: An exact solution for complete markets. Journal of Pinancial and Quanditative Analysis, 37(1):63 91, 2002.
Haoran Wang. Thaleia Zariphopoulon, and Xunyu Zhou. Exploration versus exploitation in reinforcement learning: A stochastic control approach. arXiv preprint arXiv:1812.01552. 2018.
Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural policy gradient methods: Global optimality and rates of convergence. In International Conference on Learning Representations, 2020.
Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally lincar latent dynamics model for control from raw images. Advances in Neural Information Processing Systems, 28:2746 2754, 2015.
Ivo Welch and Amit Goyal. A comprehensive look at the empirical performance of equity premium prediction. The Review of Financial Studies, 21 (4):14455-155) 2008.
Zhuoran Xiong, Xiao-Yar Yang Liu, Shan Zhong, Hongyang Yang, and Anwar Walid. Practical deep reinforcement learning approach for stock trading. In Neural Information Processing Systems Workshops, 2018.
Pan Xu, Felicia Gao, and Quanquan Gu. An improved convergence analysis of stochastic variance-reduced policy gradient. In Uncertainty in Artificial Intelligence, pp. 541-551. PMLR, 2020.
Jiongmin Yong and Xun Yu Zhou. Stochastic Controls: Hamiltonian Systems and HJB Equations. Springer, 1999.
Pengqian Yu, Joon Sern Lee, 1lya Kulyatin, Zekun Shi, and Sakyasingha Dasgupta. Model-based deep reinforcement learning for dynamic portfolio optimization. In International Conference on Machine Learning Workshops, 2019.
Thaleia Zariphopoulou. A solution approach to valuation with unhedgeable risks. Finance and Slochastics, 5 (1):61-82, 82, 2001.
17

---

Published in Transactions on Machine Learning Research (08/2025)
Appendix
A Related Literature
this section, we discuss related literature.
A.1 Continuous-Ting Finance Models
Existing strategies for solving portfolio optimization using contimuous-time finance models can be loosely summariied as performing three sleps:
1. Choosing the model for the dynamics, i.e. the type of stochastic differential equation (SDE).
Estimating the parameter of the selected model (which is also referred to as model fitting, model identification, or calibration).
3. Solving for the optimal policy under the estimated model. This step leverages stochastic optimal
control tools (Fleming & Rishel 1975 Fleming & Mitter 1982 Fleming & Soner 2006 Yong & Zhou 1999
Finding and estimating an appropriate model for stochastic optimal control requires significant domain knowledge. For instance, in finance, the modeler must specify both which factors are relevant and how they affect stock prices (Fama & French 1992). Unless every relevant factor is correctly specified, optimal control can hardly lead to good performances. As result, stock trading, control methods require hand picking three five economic indices the factors, and assuming they follow simple (often linear) SDE. Trading can benefit from much richer datasets including related option prices, technical indicators, and interest
rates (Aboussalah et al. 2022
De Prado 2018
Dixon et al. 2020 Mchtab & Sen
2019
Further, even with a correctly specified model and factors, likelihood-based estimation for SDE control models can be very challenging (Phillips Yu 2009). As a result, methods like Ait-Sahalia 2008 ; Ait-Sahalia &
Kimmel 2010) sock to replace the exact likelihood with other likelihood-based objective functions, while maintaining theoretical guarantees. However, the proposed objective function needs be derived for each specific problem, and the derivation can be challenging. Other methods like Fasen] (2013) Holy & Tomanová (2018) rely more specific parametric low-dimensional setups. To alleviate these issues, our framework extends the existing continuons-tine finance models by allowing for a flexible and generalized definition of stochastic factor dynamics. Furthermore, we simultaneously conduct policy learning and model calibration in an RL manner, with a squarc-loss objective that avoids the calculation of an exact likelihood.
A.2 Reinforcement Learning
RL aims to conduct the aforementioned three steps by (i) relying more on data (ii) in an end-to-end fashion. Methods like model-free RL assume no parametric forms on the dynamics, and directly learn the optimal policy while explicitly learning the model (step 1) and estimating the parameters (step 2).
Discrete-Time Model-Free RL There exist many discrete- soerettetimme model-free RL methods (Sutton Barto 2018). In this category, deep deterministic policy gradient (DDPG) is the most relevant one, and is empirically most widely used for portfolio optimization (Hambly et al. 2023 The reason is twofold. First, DDPG is policy-gradient based method, and thus can naturally handle continuous states and actions in portfolio optimization with simple procedures. Second, DDPG learns deterministic policy instead of a stochastic one like Haarnoja et al. (2018). This characteristic is especially important in portfolio optimization where the policy learning goal deterministic policy since the cost of a stochastic policy is extremoly expensive.
Contimuou--Time Model-Free RL Contimuous-time model-free RL Wang et al. 2018 Doya 2000 Munos 2006) aims to solve for a continuous policy. However, such methods do not use or assume any SDE structure, and thus struggle with the common open questions in model-free RL like poor stability
18

---

Published in Transactions on Machine Learning Research (08/2025)
and sample complexity. As an example, path integral methods stem from the theoretical result that the value function of a type of contimuous-time decision-making problems can be expressed in closed form a Feyman-Kac path integral (Fleming & Rishel 1975 Kappen 2005). A series of control methods follow the rationale of optimizing the policy to maximize such an integral. Specifically, Thcodorou et al. (2010) propose an open-loop control strategy; Kappen & Ruiz (2016) build RL. with importance sampling: and Chebotar et al. (2017a by: Stulp & Sigaud (2012) combine path integral with other model-based or model-free RL methods. However, the core derivation only holds for decision- making satisfying [Kappen & Ruiz 20166 Equation (1)), which is equivalent to assuming that the action does not affect the randomness in decision-making. Such an assumption is limiting, and does not hold for portfolio optimization, where how to allocate the wealth in order to minimize the risk is key to a successful policy.
Model-based RL and RL with Representation Learning Model-based RL and RL with representation learning are two active research arcas but without clear general statc-of-tho-art (Bharadhwaj ct al. 2022
Eysenbach al.
2022
Trabucco et al.)
2022
Janner al.
2019
Deisenroth Rasmussen 2011 Nagabanda
et al.
2018 Laskin
2021 ; Lee et al.
2020 Watter et
2015
Chebotar et al. 2017a Halner et al.) 2019:
Kim et al.
2019).
The closest to FaLPO
those that learn an explicit representation of latent variable
like Lee Lee et al. (2000); Mhammedi et al. (2020).
But such methods are unable to leverage continuous-time
finance models for portfolio optimization.
Bayesian RL Our proposed framework also related to Bayesian models (Ghavanzadeh et al. 2015 Raw
et al. 2013), we treat the learned representation of factors as the hidden variable. Strictly formulating an NSFM a Bayesian model requires assumptions specifying the conditional distributions, and thus requires more domain knowledge. The optimization of Bayesian methods is also more challenging.
RL for Stock Trading
Various authors applied RL to stock trading Corazza & Bertoluzzo 2014 Hambly
et al.
2023 Nan et al. 2022; Xiong et al.) 2018 Guan & Liu, 2021 Liu et al. 2021 Hu et al. 2018 Yu
et al. 2019 However, these methods focus more on feature selection or empirical verforaance-mprovinng
techniques. Methodologically, they do not take advantage of continuous-time finance models.
A.3 Empirical Risk Minimization
Another related area is Empirical Risk Minimization (ERM) (Vapnik 1992 ERM studies the minimization of an objective function using the averages over training data to construct an empirical loss function. Recent work connected ERM with simulation-base based and data-based offline decision decision-making methods (Reppen & Soner 2020). More specifically, when the random input is observable and unaffected by actions, and training set is available, the decision-making problem be formulated the portfolio optimization may be reformulated as an ERM extension.
ERM problem. As result,
B Other Objective Functions
The goal of portsolio optimization is to maximize the return while minimizing or constraining the risk. In practice, one can use different objective functions for such a goal, like mean-variance (Hambly et al. 2023). Sharpe ratio, and so on. In this work, we consider utility maximization with power utility and exponential utility. The proposed method also works with other objective functions, as long as we can derive (part of) the optimal policy structure. The selection among these objective functions is user-preference question. We note that, in principle, one could also combine multiple objective functions into composite loss (Li et al. however, we do not explore this direction in the present work.
2027;
C Gradient Estimates
In this section, we discuss the gradient estimation for both V and L. Assume that we collect B independent trajectories for S, and Yz, denoted as
D ((S0,,, bo,k), (SACk, yAr,k) U2Ar,)), (SMALK (.MMA)......M.. (MAA.))
19

---

Published in Transactions on Machine Learning Research (08/2025)
Then, the gradient estimate for V( 0x) is defined as
B
Ox) = Σ (V.(0.)0m) with (Vi(Os, 0z) = Vo U (zT, (
s(:8+),At Let TT
denote the terminal wealth in trajectory & under policy r(:30)):
M
dg
s (1+ - s mds
T
50
x (mit, Mm 0s 0z)
8) 8mat
m=
Next, we consider the gradient of L:
VL 0s)
(0,
B
Am
Specifically for likelihood and negalive mean square loss, we have
1 M-1
LLiketthood
Σ
Ves.os log(P(s(( (8 0z)
M
m==
&mat,e O(Umst,ti@a);:
M
ds
L (0,,
1
Vos.0s
(log(0) - log(smmse.))
M
1=1
(m+1)At
TF
(x(x;;)
1 ds
Smack dlUns 08))
gg' (X,; 2s
2
As a result, in each iteration, we collect B trajectories to estimate the gradient H(f,, Oz).
Primer on Stochastic Differential Equations (SDEs)
We provide general formulation of SDEs with two examples.
D.1 Formulation of SDEs
SDEs are a generalization of ordinary differential equations dynamic systems influenced by random fluctuations. The structure of the randomness can in principle be quite general, such as with jump processes where the state evolution is no longer continuous (Tankov] 2003). Although our method can be generalized to all SDEs, restrict ourselves to practical settings where the source of randomness is Brownian motion.
Let W. be a multi-dimensional independent standard Brownian motion. For random process S,, an SDE is typically expressed using a differential form as
dS, f(S.) dt + dW,, S, So / [(S.) dt [' ((S)
where f(-) and g(-) are functions of S., The stochastic integral f (S.) is the accumulation of influence to the state due to the noise. We refer the reader to Karatzas & Shreve (1987) for details on the construction of stochastic integrals and SDE theory. Important here is that Equation (8) defines the transition of S, in an infinitesimal time step. The drift coefficient (S.) characterizes the deterministic part of the change of S,, and the diffusion coefficient. g(S)) models the randomness in the transition of S..
20

---

Published in Transactions on Machine Learning Research (08/2025)
D.2 Examples
As concrete examples, discuss two families of SDEs widely used in finance, economics, and biology: Geometric Brownian motion (GBM) and Ornstein Uhlenbeck (OU) processes (Merton et al., 1973 Vasicek 1977 Bartoszek et al. 2017: Blomberg et al.) 2020 Rohlfs ct al. 2014). The OU structure appears in both applications below, and the financial application uses GBM as base, but extends it with OU drift coefficients.
The two types of SDEs are given by
dSc S, d + a dWW.
GBM:
OU: dS; = μSe dt + a dW,.
d8 = ASa where S;
Su denotes the component-wisd wise division of S,, and the matrices j and a define the drift and diffusion coefficients. We refer the interested reader Fleming Soner (2006) for more information on these topics.
D.3 Itõ's Formula
Itô's Formula a fundamental analytical tool for SDEs, and crucial for their analysis. We only provide simple version here, which is sufficient for our analysis. A more general and rigorous statement with assumptions and proof of Itô's formula and integral can be found in Karatzas & Shreve (1987 Theorem 3.3).
Lemma D.1 (Itō's Formula). Consider vwice differentiable function G, and S, following
dS: f(S)) dt + ((S))
Then, we have
(GG Ot
(GG as,)
1 g(Si)) G. ase 9(S,)
}ot (aG [GG T as,
dGtt, +
f(S.) +
((S)ddW,
Lemma D.2. For suitable bounded process St, the Itō integral Jo SidW, satisfies:
I S.dW. 0, E [[. Sadw.)
EE U Siat
E
The latter is also referred to as Itt's isometry
E Model Calibration
We discuss two model calibration loss functions, log-likelihood and negative mean square loss.
E.1
Log- Likellhoddo
We can use log-likelihood as L for model calibration. The log- likellidd of SDEs is derived in a sequential manner. Specifically, for EQuation 2 the log-likelihood derived as
Log- Likelihood 0g) = E[log(Pes (S44A ((v+ssc)0s) S, (YY:o)))),
(9)
where Pas denotes the conditional likelihood according to Equation (2) but with parameter Os instead of 0%. In specific models, one can derive Pes (S++ (()++a)) S, o(Y)): 0) (Phillips 1972 Holý & Tomanová 2018
Beskos & Roberts 2005) or the approximation of it (Ait-Sahalia & Kimmel 2010
E.2 Negative Mean Square Loss
For the SDE system of Equation 2 one can also use a negative mean square loss (NMSL) as the calibration objective. To derive this loss function, first derive the dynamics of log price by applying Itt's Formula
21

---

Published in Transactions on Machine Learning Research (08/2025)
Lemma D.1 to Equation 2
dw (X (s,: dt + dw
d (log((;)) = (s):
Σ
(X
(10)
2
Then, combined with Lemma D.2 under proper assumptions of Lemma D.2 both sides of the above equation, and derive
we pose an expectation over
dw E[dlog(S:) fs(Xx;
Σ (x::
In words, the expectation of the log price change within an infinitesimal time (3(xx:3s)d 1 E (a' (X,; dt.. Therefore, one can estimate the parameter 0s by minimizing the mean square loss between the log price change and the expected log price change:
++
LNMSL(0S) -E
log( log(Si+A1) -lggs)))))()) IE
fs(X,:0s)
(11)
dw
}.
Lo's (x,;
St. X
2
Tt can be easily proved that the true data generating SDE parameter satisfies
83 arg (0s))
Further, if we take X, = 0();; and parameterize the objective as LNMSL(00,0s) 0s) =
dg i-l llgg(S)+ log(S)) (Y::: At 0//1² we can prove
0% 03 arg arg LNMSI 0s).
Note that in practice it can be very hard to calculate the expectation E[J, (xx;00))) ES (o (X,; (s)) X.). Therefore, when At small, replace the conditional expectation via 0( Ee Co (Xs; 0s)) Sr, X.]. Accordingly the calibration objective is defined as
ds
dw Σ Lo (X:05))))
LNMSL(OS) se -E log(S(+ar) log(() f((x.:0s))) 2 ==1
(12)
E.3 Other Model Calibration Objective
Another potential model calibration objective following the same rationale as Equation
11
dx
(0ss)) -F
Σ
((C+seci (V);;
++tt
0z) (s) OCY:: 00)
E
Fx( 0s)
which is derived using the conditional expectation of X1+ given X,. However, the true data-generating parameter (8), (3)) is not maximal point of LNMSL-X- To more clearly this, this loss function encourages the representation function & take constant output so that X, constant over time with f3 (X, (x) = 0, and LNMsL-x(BA) (9s) O. We also try this loss in experiments, and leads to poor validation and test performances.
22

---

Published in Transactions on Machine Learning Research (08/2025)
F Applications of FaLPO to Different Stochastic Factor Models in Continuous-Timg Finance
FaLPO can be used with many stochastic factor models continuous-time finance models. In this section, discuss the Merton model (Appendix F.I), Kim Omberg (Appendix F.2 and EVE (Appendix F.3 These models differ in how factors influence asset price dynamics. In the Merton model, factors have no effect; in the Kim Omberg model, factors affect only the drift; and in the EVE model, factors influence both drift and volatility. Choosing the appropriate model depends on the role of factors in driving price behavior. For instance, if asset prices are uncorrelated with the factors, the Merton model may be sufficient. If there is evidence that volatility also depends on the factors, then the EVE model may be more appropriate.
F.1
Merton Model
Merton model (Merton 1969) is classic setup for portfolio optimization. It studies the allocation of capital across a set of financial assets in order to maximize profits and minimize risks.
F.1.1 Modeling
Consider p risky assels with prices St = (S and an additional risk-free money market account with, for simplicity, zero interest rate of return (like cash). The Merton model does not include factors. The dynamics for asset prices is formulated as
dsp S
= pdl + odW,
(13)
The parameters μe, p matrices, with denoting the volatility of assets. Further, use 2 to denote the wealth at lime point / under the continuuustinme policy म. Under the famous and widely used self-finam financing assumption (Börkk 2009), we have
dž
Ruptt + W.d.
2
An investor's goal is maximize the expected utility of capital U(ZT) at some future time point T::
max ( Zu 2, Su =
(14)
Negative values in the policy output are allowed, meaning the agent can short any asset.
F.1.2 Policy Functional Form
For the Merton model, policy 11 in Equation can be explicitly derived.
Lemma F.1 (Policy Functional Form for Merton Model). For Merton model defined in Equation under common assumptions, the optimal policy for portfolio optimization [4] with power utility is
13
* pl T)-
The optimal policy for portfolio optimization
with exponential udility is
(((aT)-
2%
Proof. Lemma F.1 is a classic result in continuous- time finance, proposed in Merton (1969
In words, in Merton model, the optimal policy independent of time, stock prices, and factors. The optimal investment strategy to keep constant fraction amount of wealth in each asset all along, depending
23

---

Published in Transactions on Machine Learning Research (08/2025)
the choice of utility function. Let 8, be ds 1 parameter vector. According to Lemma F.1 in FaLPO, we parameterize the candidate policy function for power utility
St, Ze Y1; 0g 02) [I(,, S, 2, (V;; 0x
and for exponential utility as
0x
(1,, Ze 2, (T,, Sr 2(000)))::
Ze
F.1.3 Model Calibration
According the Merton model formulation, there exist no factors affecting the evolution of stock prices. Therefore, we do not add the model calibration objective FaLPO for Merton problems.
F.2
Kim-Omberg Model
Kim Omberg model (Kim & Omberg 1996) is a standard model for portfolio optimization with predictable
asset returns, extensively discussed literature (Welch & Goyal 2008 Mluhle-Karba et al. 2017
F.2.1
Modeling
In Sim-Omberg model, the stock dynamics are formulated
dS; X dt + Σ v dw?
dw
S
j=1
dX, سlw X,) dt dWW,
The portfolio optimization goal is formulated as
max (##) with ((t,) == E,, (U( = 2, Zo = 2, Su 8).
(15)
F.2.2 Policy Functional Form
An optimal policy function is derived in Lemma
Lemma F.2. Under common assumplions in Kim & Ombery (1966) Herzog et al. 2004). an optimal policy
functional form for Equation 15 with power utility is derived as
(00 () + ov
where kz(t) and ktt() satisfy
(t)
+ Tr (o (1)( + ka(t))
+ (ww))
h2(())
2(7 - (1a(()) vo (z((t))) = 0,
dt
2
(tz(
+ kg (t) T k2(t) - 4 (t())) + ks)
7 - 7-1 -- (lokk()) + kz( kx((t) = O,
dt
dks(t)
+ ks ka(t) Ks (t) - kz
7 - ((00 1 (( (oo To oo kt(())))))))) los (o) (ooa
(oa (t()vuuTkss))
with ky (T) 00 (T) 0, and (T) Note that (t) scalar, k2()) is a ds 1 vector, and ky is dx dx. The ODE of kt(()) is the famous matrix Riccati equation (Appendic G
24

---

Published in Transactions on Machine Learning Research (08/2025)
Similarly, optimal policy functional form for Equation 15
with exponentiol utility is derived as
मे = (ao )-- -72 x(t) ov on (ks(t)), + Ra(0)] +
1
where (1) and (1)(1) satisfy
dki(t) (t) dt
+ Tr {u (12(())22))) + ka(t))o} + (mw) (kw)) 1
2 (Aa(tt) "o ha(()) = 0,
dk2(t)
+ kz(t)ov' h2(() ph2(t) + ks ((00 ((00))- ouk2(t) ka (t(v) (2(()) (t) 0,
dt dka(() + kss() (t(()) kz(t) p kt((t)
dt -((aT) (ao ---'ou (k((() + kg (a)uuo (oaT)-1 + ka(1)oo kz(()), ks)().
with ln (T) = 0, (T) = 0, and (T) = 0.
F.2.3 Model Calibration
Following the derivation in Appendix E.2
the negative mean square loss for Kim fim--Ombeg model can be log Mog(S)+40 log(S;) of (Y): 0/s2] where this case 0g is a derived as L(B,, As) = D
ds x 1 vector.
F.3 EVE Model with Stochastic Markovian Factors
We take EVE model
|Avanesyan et al. 2020 with stochastic Markovian factors as another example.
F.3.1
Modeling
We first detail the modeling of EVE, which formulates the dynamics of asset prices by
dS;
dw
= (:(,;; 0s)
Σ 1=1
(x(((;0s))W)) i= 1,2, ... dss.
S
(16)
dX, = (M X, + dt + (((e; As) dBi
B. = P W, + A W.
We use W. and W.L. to denote two sets of independent Brownian motions. denotes a correlation matrix with components p² E [-1, 1]. Therefore, B indeed another Brownian motion. We let μ, a, and be parametric functions, with 0g denoting all the parameters. The SDE parameters include all the parameter matrices and B's. Again, with Z, as the wealth, we aim to maximize the power utility at the terminal time T > 0 the performance objective:
(2
((+.) = Ex
1-7
EVE model poses further assumptions on dynamics
16
Assumption F.3. M has non-uegative off-diagonal entries and E [0, oo Further, we assume that there exist X(x), A and L, and N such that pl-), of-), x(-) and p satisfy
x(x) u())) (o()))((())( (c(x)) m(x) = A ATx
x(x) T x(x) diag(L) L2 LuoK), with L1, L2, L 2 TR(x) T TT M(x))=)>>)))))) = TT
(17)
25

---

Published in Transactions on Machine Learning Research (08/2025)
The conditions in Assumption F.3 necessary for the process of X, to be [0, 00) and affine. Under these conditions, SDE (16) has a unique weak solution which is affine and takes values in [0, 00)* (Filipovié & Mayerhofer 2009). Further, the EVE model requires the following two assumptions:
Assumption F.4 (Assumption 2.2 in Avanesyan et al. (0200)). The functions μe IR + RR a: R Rd xds are continuous. The columns of p belong to the range left-multillication by oxx) for all Rdx
Assumption F.5 (EVE Condition in Avanesyan al. (2020)). ). For some p [0,1],
p=p=p' p-pp, pl,
where 7 is the identity matrix. Note that when p 1, is vector and thus we define == p p.
F.3.2 Concrete Example
We consider Specifically, use Du: Dr, and Dx to denote diagonal matrices. Let D(x) denote the diagonal matrix whose diagonal is z. We use z for any & E IR to denote the component-wise power operation (Hadamard power).
concrete example of EVE satisfying the formulation and assumptions in Appendix F.3.1
We define
m(x) = D Duo 0
a(x) = D2 (xx)
x(x) p Dr (z)
Then, we have
M(x) = =D D D
Further, we pose
10 p = pp I.
Then,
X(x) (o(x) u(x) D=1 and A(x) (z))>(z>) X(x) Do²r,
n(x) [(())()==))))))(())) D2 (x. (( 1) ')_ DD D(x),,),
TR(x)" pT ((x) TD(x> (p 1) D
-
Such a setup satisfies Equation 16
F.3.3 Policy Functional Form
The policy functional form of the EVE model with Markovian stochastic factors can be derived as: Lemma F.6. Under the assumptions in Appendices F.3.1 and F.3.2. the optimal policy function follows:
n. = -0x(
+ gox 9xx (xx((())2))
with
(t)
dx
r
L.2k(tt)))
(M (M+ N) A'= 0. i= 1,2, .dx
29
dk (t)
+ T (() =.
dt
For the specific example in Appendix F33.2 in FaLPO, we parameterize the candidate policy as
(1, Y,, Z5 0 0,)
TT(,, Yr, Zz 0+) =(()))a((t.))
26

---

Published in Transactions on Machine Learning Research (08/2025)
F.3.4 Model Calibration
Following the derivations Appendix E.2 we can derive the calibration objective as:
L(Os) ;- min (g=(Ci,Ca)
E
A log (S;)
G Solutions of Riccati Differential Equations
According to the analysis in Section F the optimal policy function is closely related to the solutions of Riccati differential equations, which also have closed-form solutions. Specifically, with abuse of notation, let A, B and D be p p matrices, and (t) [0. I] Rexp as function of solving the following Riccati differential
equation:
(1) = x()(1) + X())BB' x()) + DTD.
dt
(18)
X(0) = Xo-
Following the analysis and assumptions in solution of (t) follows:
Behr et al. 2021). the unique symmetric positive stabilizing
X(t) Xx0 - clA" (xx Xx) (X2 dxxzet 18 -Xo]] -od
where
AX + XLA + +B = 0,
with A:= A-BB B and
0= A + X + D.
Note that with Equation 18 we can further derive the policy functional forms without using neural networks to parameterize time-dependent functions.
Extension to Linear Quadratic Control (LQC)
FaLPO can also be applied to decision-making problems other than portfolio optimization. To implement FaLPO, needs to first construct neural stochastic factor model combining factor representation with continuous time model. Then, the policy learning is conducted while leveraging policy functional form and model calibration. As example, we implement FaLPO to linear quadratic control (T.QC). and detail modeling (Section H.1 policy functional form (Section H.2 and model calibration (Section H.3
H.1 Modeling
We consider the problem of LQC (Sun & Yong 2020) but with stochastic factor X, following an OU process. With slight abuse of notation, we use S. to denote the sate variable this section:
dw
ds. = (BS, +UA X,) D DjA, dw?
(19)
1=1
X; μX, dt + dW, where B, U, M, v, and Dj are redefined matrices with appropriate dimensions. With A. x(-) following the policy n, we aim to solve
max (TT) with V(T) := Ex [ [(QS.) (RA,) A.] dt + (GSz) S1
(20)
with Q, R, and G as known matrices with appropriate dimensions, and T terminal time. Further, we apply the modeling stratogy in Section 3.1 and aim to learn the representation of stochastic factors:
X, (5):
27

---

Published in Transactions on Machine Learning Research (08/2025)
H.2 Policy Functional Form
By laking E, (the combination of St and Xx) as the stale variables, reformulate the problem classic LQC problem:
dw
d = B
Σ A, +
with all the coefficients redefined. Then, under common assumptions in Sun & Yong 2020 Yong & Zhou (1999), can be derived that the optimal policy satisfies:
A
(T))),,
where
dw
- = R+ > (D] K (t)D,))
Also, (t) solves the differential Riccati equation:
(1( (
(T-T)
with
K² = 0.
Therefore, we can formulate the candidate policy
(t, St. Yú; 0s 0x) = [Tt,, St, Y;; 0g 0z) = kin (t: 0,) + (VV;
(21)
H.3 Model Calibration
According to Equation
19 and following the derivation stratogy in Appendix E.2
derive the negative
mean square loss for LQC as
L(O,, 0s) = : - C24.
(22)
with 0s ((1,, As summary, to apply FaLPO to LQC with stochastic factors, parameterize candidate policies following Equation (21 and maximize
- AV 0,)) + AL(0. 0s).
with V in Equation 200 and I Equation 22
Extended Results for Theorem
Assumptions and Definitions
To start with, we consider stochastic factor models such that the optimal feedback admissible policy admits functional form as
मं. IT(,, X 02).
Also, we consider the L function such that the true data generating parameters 03. og satisfy
0%.0% 0 arg max L(As, 0s).
(23)
0..08
while other options for L also be empirically used in method.
28

---

Published in Transactions on Machine Learning Research (08/2025)
Definition I.1. For contimuous-time policy #, [It St. Zz, (Y);;))) 0z), define its value function
( 0x) = (Z))]
Accordingly, we define the contimuous-time version objective function as
(0, 0z. 0s) == (1 (0s 0n) + (L 0s).
Definition I.2. For E [måt, (m + 1) define [t] := mAt and [ == [[+]
Definition I.3. For the continuous-time optimal policy म", we use [ to denote the piece-wise constant.
version #*. We use Z denote the wealth process when implementing the optimal continuous-time
policy ((; 0g) in the piece-wise constant manner. Specifically,
( yds;
dž,
d2, =
W
dy
S 7 2 d1 [c]
6
dw
+ (L]
S (x,;0%
2 |
dw:
8
Further, remember z5- is used to denote the continuous- wealth process under the policy øt. By the dynamics of contimuous time wealth process 2 in Section 2.3 the dynamics of Z is derived as
dzr 2
[T,,, 02 fs(X, 03) [[t, x,; 02) gs (3) d.W.
Assumption 1.4. For cach R > 0, if | < R and t S T, we assume that there exists a Cn > 0 such that [I fs x 03)l V (æ 03) IIfx (2; 03)1l (1xx(x;; 03)1l < CR.
and II(t, z; 02) is locally Lipschitz with Lipschitz constant CR- For some p > 2, there cxists a constant A such that
B[ sup OSIST
12 P V [ sup
1Z!** [
VB[ sup |X|||]] VE sup [loo((S))IP] s A. OSIST
USIST
USIST
Note that Assumption I.4 requires the stochastic processes to have bounded high-order moments. For a specific model like Kim Omberg, this is not guaranteed hold for every initial value and SDE coefficient, but one can derive model-specific sufficient conditions for Assumption I.4 In practice when implementing the method, we caleulate the empirical moments of wealth, factors and asset prices to approximately check whether Assumption 1.4 holds.
With Assumption
1.4
we define a stopping time:
Definition I.5. For R > 0, define a stopping time
|1 1 R or |X,1| 2 or [log((S)) 2 R or 12 2 R
Tn inf 0S1ST
Assumption I.6. The utility function U(2) has linear bound on 2:
[U(=)] Co([=++1),
Note that Assumption L6 holds as long as the utility function bounded at the smallest value in 2. Specifically for power utility, Assumption L6 is equivalent to setting y (0, Assumption I.7. There exists At' > such that for any At Al' [a is also admissible policy.
29

---

Published in Transactions on Machine Learning Research (08/2025)
1.2 Lemmas
Lemma I.8. Consider non-negolive conslunts CI, C2, Cn. The following inequality is true:
Proof. The proof follows the Cauchy-Schwartz inoquality. Lemma 1.9. (0), 0z 0z) mazimizes both (00g, 0z)) and I(Og, 0s)
Preof. Since ag- is defined to be the optimal parameter for the contimuous time policy and 03 is defined to be the true data generating parameter, then (0g, 07) 07) maximize V. By Equation Lemma I.10. For any 8 > 0,
23 (0 fz) maximize L. O
E sup (Z)- 1 2 y S E 0S1ST
(2 Z )
2P+18A 8A +
(p 2)8A
sup 0SiST
p
p82/(p 2) P2
Proof. Proof by applying Young's inequality. See derivation in Higham et al. 20002 Equation (2.8)).
Lemma L.1. For any S TR. the difference between the coefficients of the dynamics of 2 and 2 are bounded by:
ds T1,, X1; 07503 Xx: 25 m(4 X (xx
S
S/4
1=1
5dg exp (2 (2R) 1 (xp(22)A [t] + exp(2R) R2 X X 1
+ Rº ||| - Sell + exp((2)) 23-= 2 2
+ exp 22) 21 - 1
and
ds
di
St 2 (9') (x;:83)
Σ (x) |"
dw
Σ
[[[t], [taa (e]:8t) Σ
Σ
II(t, X1; 07)
|
<5dg exp(22)
ex |t]|² + exp(2R) R2 |x - X(el| ?
+ R² |s - sill + exp(2 ". - 2:5²1² + exp((2)) 2 _/ 27
Proof. By triangle inequality
de Σ
de
2 S
Σ
(xx:
[4] Si
ds
(Xe: 033
(#/ S # 23 1i11
#1 2100 |ماا + |
S/g
- -
2 s
For any 15 TR, we can further bound the right hand side by Assumption
L.4
Σ #x,5(Xx;
Σ
#1 (3) 03) || [
S
[c]
Sic
<ds exp( C2 ( [exp( R) R)R(
[t]l+ || - -X
+ exp(R)) 2,5" 2/
2 2// '1)
R | []
30

---

Published in Transactions on Machine Learning Research (08/2025)
Then, by Lemma
L8
ds
Σ 1 (x; 1 Si
Σ (3)
[c] Sie)
55d exp(2 exp(2 12 (1t-110 1 + X, - -x11113
+ exp((2) /5 2| 11
| + + | 1 + R²² || - sall²1
5d3 (xp((2))C exp(2R) R²1 |&]12 exp(2R) R² Xx -x(u 1
+ | S 2| + exp(2R) ". - - eexp(2R) [,,00 - [x*0 |
Similarly
ds
[ Σ
dw
00 (x
St
2
Σ
ds
dw
(0 (x):
1
1/4
1=1
S 5 exp (cp(2R)CH C+ exp(2 122 |t-
|t]| exp(2R) R² | - 111]
1 exp(2 ,,"" 7/.0 |2
2
+ exp 2R) 2/55 7 1 "
R2 ||
Lemma I.12.
With TR defined in Definition I.5
E XEATR X [UATTl
< ( + At),
E SIATR -
- 2exx + At),
2
E
[EAFR]
- 2R expl CX + At).
Proof.
By the dynamics of X, and Lemma L8 we can derive
thTR
I gx 9x( dW,
LATR
E XIATn X(enTK] 2| 2AtE
(fx((x,0331' ds + 2E
MeNTR
where MEATR satisfies miArg Al S (LATR) (MAATR 1)A1. Further, we apply Ito's isometry with stopping time (Lemma) D.2,, and derive
MATR
LATR
E XLATR - XluATal 112 < 2A1E
[xx((x;00311 ds ds 2B
(xxx(xx033)1 ds.
m
At
By Assumption 1.4
we derive
E XINTH X[eATR] 2 S2Ch(AA + +t))
Similarly
E SEATR - -Seernll 1|2 $2exp(2R)C((AF + +AI),
and
E
2 [tATR]
22R exp -
31

---

Published in Transactions on Machine Learning Research (08/2025)
Lemma I.13.
B sup (A ) 0giST
+ (A +
+ +2C% + At) + 2eep( + Ac)
+ +10(T + 44 C exp(4
/
E sup 0SP5s
ZTATR
- ds.
Proof. By Cauchy Schwarz inequality and the dynamics of 2 and Zi for any T TST T
E[ sup 051ST
EALK 12
LAte
MATR
ds
de
S
2
2TT
sup LOS1ST
F 03)25" -
Σ x Laj
ds
i=1
2
LATR
2E
sup DSEST
# (xx;03)
dW.
Then, by Doob's martingale inequality
E sup 051ST
LAtE
LATA de
ds
S
<2TE
sup USSSS
# (X, 03)
Σ
* [S) 03)
ds
dw
MATR
dg
4 g 0z) - (X (x:] 03)5 dWp )
+ STE
(X,;
Next, we apply Lemma
E sup
(Z
0StST
TARR
<10(T + 4)d exp(2R)E
(exp(2R) [s]] + exp((2)) | - X|,1
+ R² [|| - -.1l + cxp(2
2.rh 2 + +xxp(2)) 21 2|5] [3as]]
Therefore, combined with Lemma
I.12
E sup (2 UStST
2mne
4 + 20 + +
+ 220 Al) 2 + AC]
+ +0TT + 4)d C exp(4R)
l
E sup 05:50
2
"1**1 ds.
Lemma I.14. With the definitions and assumptions in Appendiz I.1
lim B[
ž 1 0.
Ar-0
32

---

Published in Transactions on Machine Learning Research (08/2025)
Proof.
By Lemma
L.13
apply the Gronwall inequality and obtain
E
sup 0S1ST
(Zii
)²]
$ 10(T + C (s + (At
At)
+ (A + At) + 2cxxp(4R)CH + At)] exp(10(T + 4)d exp(4R))
Then, combined with Lemma 1.10 for any 8 > 0,
E sup ( 0S1ST
2/, 1)
< 10(T + 4) d3 C# exp(4 At + +2 + -1)
(A At)
2exp(4 At + At) exp(10(T + 4)d C 4 exp(4Rt)
SA 20-2
p
162/(e-21RP
Therefore, Elsuposier (2 - 71551 [2] converges to 0 as Al goes to 0.
1.3 Proof
For ease of presentation, we define
0 (0s 0x 0z), 0" = (0%, 02, 03) and 0%. (03.01: Or. (5..0)
Note that every discrete-tine admissible policy is continuous-time admissible policy. Thus, the continuous- time admissible policy set includes the discrete-time admissible policy set. Therefore,
V(0)) Z Va
In other words, is enough to bound V((') V(0%.) for the proof. By Lemma 1.9 0" maximizes V and L simultaneously, leading to
v(00) v(02.) < H(00) H())))
1
By Assumption I.7 for At S At', 0t also an admissible paramcter 0" E A, leading to H(0") H( s Further, for any 8 > by adding and subtracting equal terms,
( - v((A))
1
<
[A(0)) - -H" + H(0') (02.))
(24)
1
(A((0) H()))|-
Next, we focus on ()) 5)), which by definition has
8)) - H(00; 6) = (1- BIU :0) U
where AL(O") in both ( 8) and 0) omit each other. By Lemma 1.14 we have lim 2 Since 02; is a continuous function, we implement the continuous mapping theorem and derive
Pg
27
lim U(2 1:8) P, U(F :6). Ar
(25)
with different finite At is uniformly integrable. Then, following As- sumption T66 U UZZ 1:8) is also uniformly integrable since U(2; 6) has a linear bound. Combined with
By assumption
I.4 2
AKAr
Equation 25 we derive
lim
"* 1;, + [(U(Z
Ar
which finishes the proof.
33

---

Published in Transactions on Machine Learning Research (08/2025)
J Extended Results of Theorem 4.2
In this section, we study the non-asymptotic guarantees on the performance of FaLPO.
J.1 Another Version of Theorem 4.2
Definition J.1. For two random vectors v and w, we define the trace of the covariance matrix as
Var(v) [ || Cov(u, w) E=Elu wo Ei,] Ekw]
Further, Var (()) and Covo(u, w) to denote the conditional version of the two given 0:
Varo(u) [ ||E (0) Cova(u, w) :=Elu" T w/00 Elv 0J B[w
Note that it is challenging to theoretically analyze non-convex stochastic optimization 5 while there are various ad-hoc procedures providing good empirical performances. To provide theoretical analysis, in this section, we study projection-based version of FaLPO (Algorithm 2 Specifically, the learning/ learning/optimiztt /otimmzaiinnnnnn process is conducted in a bounded parameter space B, under which we assume that the objective function is strongly concave regarding the parameters.
Algorithm Projected FaLPO
Input: Hyperparameter X, learning rate v, number of iterations N, the strongly concave region B, and batch size B.
2: Output: 0g, 0x, and 0s
3: Initialize neural networks with initial parameters (0g, 0z 0s) B.
Parameterize the policy function by (3)
for n E [N] do Collect B trajectories.
7: Estimate the gradients of H following the procedure in Appendix C with the parameter X.
8: Update 0g and On with learning rate by gradients.
9: Project the achieved update to B.
10: end for
11: Return 0s. 0T. and Os.
For ease of presentation, we define
8* == (6), 02 03), 0h. = (0%.
O o7 03,0)) and = (0t ( Ø! o. de Ø! Al.
Let 0, be the estimation after the th iteration, and 0 = e. technique to consider the average estimation 0 instead of the final estimations dy for such analysis. Then, we provide version of Theorem 4.2
the average estimation. It is a common
Definition J.2. With the gradient estimations discussed in Appendix define
(H) = (1 -)-V-(()) + (Q).
34

---

Published in Transactions on Machine Learning Research (08/2025)
Theorem J.3.
With assumptions in Section J.2 À [0, and ny Co as the learning rate,
B V(0)
H(0)) H())))
H(OX)) H(t))
CB log(N)
1 +
+
1- X
2N - -
N-1
1
E
[11 - Var e.. (VV)(0n)) + Vare,, (VL)(0,))
2NB(1 - X)
72
1+11
R=0
21(1 - A)Cove, (vV) (Lo(ON))]
Also, there exits situations where a X E (0, provides smaller value for (1 - Var (VHt (0w)) + Var (B.) 2X(1 ()Cor(VH.(...) (O,)) than À O. In other words, there exist cases where tuning À may provide better performances.
J.2 Assumptions
A Assumption J.4. There exits a constant CB > 0 such that the parameter region B is convex set and satisfies the following conditions
n A, ((ss. an. is locally m-strongly concave with a local maximal point (0.A0 P 0g, E B.
2. For any 0 B, ||||| < CB
For any 0 E B, the expectation of the gradient estimation bounded by
EE
(0)
(6)
E
< CB.
B
These assumptions are widely used in existing analysis (Papini et al. 2018 Karimi et al. 2019
Agarwal
et al. 2021 Bhandari & Russo 2024 Wang et al. 2020 Xu et al., 2020).
However, even with Algorithm 2, it remains difficult to guarantee that training is confined to region B, which is typically unknown a priori. In practice, a more effective approach is to use widely adopted weight initialization schemes such as those proposed in GGlorot & Bengio 2010).
Assumption J.5. At the th iteration,
the learning rate Mh+1
mm
Note that in practice we will tune the learning rate 17 as hyperparameter, since we may not know m. However, it is a common practice to set the learning rate in Assumption J.5 (Hazan & Kale 2011 Nemirovski et al.
2009
Shalev-Shware et al. 2011
J.3
Technical Lemmas for Theorem 4.2
Lemma J.6.
With Assumption J.A and J.5
we have
H(Ot) (0) <
1
Σ
N-1
ΣB
2k=1
((,)))
2N
1
B
n=0
N-1
ΣBe (H )
+
Σ
VH(O.))
(0) -0)-
N
B
n=0
35

---

Published in Transactions on Machine Learning Research (08/2025)
Proof. By the strong concavity of H in Assumption J.4
VH((O)) - -o,) 2 H(00) - H(O,) + m
00, - 001
(26)
2
_km (H)(),))
Further, since Pnt1 is the projection of 0n Mn+1
to B, the projection satisfies
B
E ((n))))
On 1
gt
| 0
B
which suggests
0 - 8/|
0 - -111
Σ (bx))))
2 |10, 171
On + 1a+1
8t
Σ k=1
(Hn))))
Σk=1
B H (fn)
(27)
11++1
(20, + mh+11
20²))
B
B
7n+1
E
B
(H+(0.)
2
- -2mm+ 10, -8¹)
B
B
We reorder Equation 27
and derive
B T2k
-k=
(On)
- (0n,
|1 - 1
21
EB AH (0,) 2
2 + Mn+1
-==
| - -81112
B
Taking the result back to Equation 26
EB kml
HH (0n()) (0m)
H(01) H(Om) < (H.)))
(0" - -
B
Σ ((n))
||0, -01
- 1 + 2 n
+
2
B
771
0
0111
(28)
2
EB
VHH (0n) T
(H.))
(8² - -.,) +
1
2 - m) 1|o, - -t11
B
2
EB V (0,.) 1
1
|10m-1 - 01 2 7
27m+1
B
36

---

Published in Transactions on Machine Learning Research (08/2025)
By averaging over n with Assumption J.5
get
-(01)-11() - (0) S
(H - H(O.))/N
2=0
1
N-1
Σ k= (O,)
2N
n+1
B
n=0
N-1
Σk (Ha(On)) )
+
Σ
VH(O,))
(0 - On)
N
B
n=0
where the first incquality due to condition of Assumption J.4
J.4 Proof of Theorem
Recall that every discrete-time admissible policy is continuous- time admissible policy. Thus, the continuous- time admissible policy set includes the discrete-time admissible policy set. Therefore,
v(0") 2 V2--
Therefore, it is enough to bound V(00) (0) for the proof. By Lemma 1.9 0" maximizes and L simultaneously. Therefore,
- < H(00)) H (0) H()))) H(O%)) + H()))) H(Ot) + H() H())
(29)
1-
1-
Then, we use the convergence result for Algorithm 2 detailed by Lemma J.6
N-1
D (Om))
H(0t) - H()) -
Σ
2N
n n+1
B
n=0
(30)
N-1
ΣB k=
V ( )
VH(Om))
(0t (0)))m) 0m)
Then, we take expectation on both sides
1
Σ (On)
E(H(0 H00] -
E
2N
Ti n+ 1
B
7=0
(31)
N-1
Σ
(H(O.))
DB k
(Ha(
(0t (m)-
+
B
0=0
For the last component of Equation
31
since (0x))) is an unbiased gradient estimator:
En (On) T (0' - -nn)
B
E
(H()
B
ΣB
((n)
= E
E
(H()..)
(0) Bm.) 0n 0.
37

---

Published in Transactions on Machine Learning Research (08/2025)
Then, for the first component in Equation 31
(On)
71
+
B
12
Σ k=
(H(((nn))
E
E
18k
1
B
Ek=1 (00m)
B
(0n))
V H (On)
Varg.
)
On
B
B
Ek=1
B
HH
Varg,.
CB
B
where the last inequality is due to condition 3 of Assumption J.4
Equation
can be further derived as
N-1
(On)
1
B[H(O!)- - H(O)
E
Varg.
R=
CB
2N
B
===0
As result,
()H( H(0)
(02) (00) -
CB (N)
<0((0))- (1O0)
2N
N-1
7/(1 Var o., (VVi(On)) + Xar o. (VL)()
BN
122 - A)Cova, (VV)(On), (LN))))
E
BN
Taking the results back to Equation
29 we finish the proof.
K Extended Results for Synthetic Experiments
For syuthetic portfolio optimization, we provide details for drift and volatility (Appendix K.1 data generation (Appendix K.2), hyperparameter tuning (Appendix K.3), and extended experimental results (Appendix K.4). We consider 21-day trading, and generate 1000 trajectorics with 21 observations for training, 1000 for validation, and 1000 for testing. We use average terminal utility a metric to compare different methods.
K.1 Drift and Volatility
Drift and volatility are two important concepts characterising the strength of signal and noise in financial markets. To demonstrate this, for an asset price St and time interval At, define return as:
returnt St St S
St
The returnf can be daily, monthly or yearly, depending on the length of At. For a specific asset, drift (fr(X,;;3) in Equation (2) approximately the expectation of the return, while volatility (g' (x); is approximately the return's standard deviation. Given multiple assets, drift (f(X;;03) volatility (g(Xe: 03)) is a matrix. When generating synthetic data (Appendix K.2), we fix the scale of drift and vary the scale of volatility, which is defined the average value of each component.
a vector and
38

---

Published in Transactions on Machine Learning Research (08/2025)
K.2 Data Generation
We simulate data for St and X, following SDE 66. To this end, drift and volatility are randomly picked while mimicking the historical stock price data, with an average annual return of 0.1 and average annual volatility in {0.1, 0.2, 0.3), leading to daily return around 252 and a daily volatility around {0.1/252, /.2/2,..2/25. 0.2/252, 0.3 222/) The true representation function is selected as component-wise exponential operation. Then, we discritize the SDE following the explicit Euler method, and generate data accordingly (Beskos & Roberts 2005).
The specific configurations for data generation is:
Define two scalars: Ca, and C. determining the scale of drift and volatility:
Ca = 0.1/252, and C2 E
is selected as random matrix, whose components follow uniform distribution in [0.5C., 1.5C)).
is selected random matrix, whose components follow uniform distribution in
-,55C, 1.5C.
j is selected diagonal matrix whose diagonal components follow uniform distribution in [0.9, 1].
The initial values of X are randomly generated from uniform distribution on [-2Cd, 2Ca].
The initial prices of assets are randomly generated from a uniform distribution in [20, 30).
Note that the design makes sure that the simulated price has approximately a yearly return of 0.1 and yearly volatility in (0.1,0.2, 0.3). Table reports the experimental setup parameters.
Experiment Configurations
Values
Time Interval A/
1 (Day)
Terminal Time T Scale of Annual Drift
21
0.1
Scale of Anmual Volatility
{0.1, 0.2, 0.3)
Number of Simulated Trajectories
1000
Utility Function
(Power, Exponential)
Risk Aversion
3,,1,3,5,10
Number of Replications under Each Hyperparameter
Compute Resources
AWS ec2 m5ad. 24xlarge
Table Setup for synthetic experiments
K.3 Early Stopping and Hyperparameter Tuning
For better performance, we conduct carly stopping for all methods using the average validation utility, with the patience as 5 steps (if the performance does not improve for 5 epochs, the training stops.). The considered hyperparameters include the learning rate, X, and batch size. For each configuration, we conduct training for times, and average the results. We pick the configuration that provided the best average validation utility, and test it on the test data and calculate the average test utility per trajectory. The tuning process is conducted using the software wandb (Biewald 2020). Table reports the hyperparameter values.
39

---

Published in Transactions on Machine Learning Research (08/2025)
Hyperparameters
Values
patiBatch Size
{100,50)
{0, 05, 0.1, 0.9)
Learning Rate
{0.0005, 0.001, 0.01, 0.1}
Table 5: Hyperparameters for synthetic experiments
K.4 Synthetic Experiment Results
To gain a more holistic understanding the performance of FaLPO in a variety of scttings, we conduct experiments with different number of stocks to be traded (Appendix K.4.1p, different risk preferences (Ap.endix[K.4.2) and alternative utility functions (Appendix K.4.3). Finally, we also compare the performance of various methods under the Merton model sanity check (Appendix K.4.4
K.4.1 Synthetic Experiment Results with Different Dimensions
Tables G and 7 report the synthetic experiment results with the number of simulated stocks (ds) varying in {10, 15}. FaLPO outperforms the competing methods in all scenarios. Note that the performance is not strictly negatively correlated with the number dimensions of the problem or the anmual volatility simulation. The reason that the noise the problem is indeed determined by the whole volatility matrix o, which is randomly generated the synthetic experiment (Appendix K.2) In other words, the dimension and average scale cannot fully characterize the extent of the noise in synthetic task.
Annual Volatility Simulation
0.1
0.2
0.3
FaLPO
0.465 + 0.446
-1.35 + 0.155
-.7737 + 0.219
DDPG
-1.650 + 0.456
-3.30 + 1.294
-5.495 +'2 1.269
SLAC
-0.750 0.210
-5.30 0.011
-6160 + 0.012
Methods
RichID
-3.350 = 0.111
-5.65 = 0.102
-6.25 + 0.048
CT-MB-RL
-2.80 + 0.014
-5.35 + 0.020
6.160 + 0.026
MMMC
-4.723 7.619
-5.602 4.299
6.124 + 3.217
Table Average terminal utility after tuning with standard deviation for synthetic data with ds and dw 10.
Annual Volatility in Simulation
0.2
0.3
FaLPO
2.463 3.744
-1.021 + 0.278
-2.243 + 0.547
DDPG
-3.976 + 1.428
-1.443 + 0.751
-5.205 + 1.858
SLAC
-4.749 + 0.139
-6.129 + 0.016
-6.526 0.012
Methods
RichID
-4.973 + 0.448
-6.321 + 0.038
-6.641 = 0.022
CT-MB-RL
33.074 + 0.014
-5.714 + 0.023
6.363 0.021
MMMC
55.388 + 5.688
-6.465 + 4.978
-7.155 = 5.965
Table 7: Average terminal utility after tuning with standard deviation for synthetic data with ds 15 and dw 15.
K.4.2 Synthetic Experiment Results with Different values of 7
Tables and report experimental results with ds 10, dw 10, and 7 E {3, 10) for an exponential utility. FaL.OO outperforms the competing methods in most scenarios.
40

---

Published in Transactions on Machine Learning Research (08/2025)
Annual Volatility in Simulation
0.1
0.2
0.3
FaLPO
0.003 0.0021
0.0055
0.0008
0.0132
0.0028
DDPG
-0.003 + 0.001
-0.0105 + 0.006
-0.0205 + 0.0034
SLAC
0.003 1 0.0007
0.0153 0.0013
0.0192 1 0.0011
Methods
RichTD
0.012 0.0005
0.0188 0.0002
-0.0211 0.0
CT-MB-RL
-0.01 + 0.0
-0.0179 + 0.0
-0.0206 + 0.0
MMMC
0..0162 + 0.0212
-0.019 + 0.0135
-0.0210 + 0.0102
Table 8: Average terminal utility after tuning with standard deviation for syuthetic data with 10.
Annual Volatility in Simulation
0.1
0.2
0.3
FaLPO
-4.575 +: 3.325
117.9358
2.3349
-56.0405 +: 13.0502
DDPG
-23.113 + 4.3472
-51.6559 = 11.7981
-50.0399 + 13.8451
SLAC
-23.514 11.7077
-886616
0.0254
-76.1371 + 0.0355
Methods
RichID
-44.629 + 1.8797
-69.881 = 0.9413
-77.232 + 0.1091
CT-MB-RL
-34.842 0.4686
65.41 0.1331
-75.4364 0.122
MMMC
59.2333 + 77.4511
70086667 + 49.2500
-76.8619 + 37.448
Table Average terminal utility after tuning with standard deviation synthetic data with y
K.4.3 Synthetic Experiment Results with Power Utility
We also conduct synthetic experiments maximizing the expected power utility for portfolio optimization. The results are summarized in Figure 5 FaLPO outperforms the competing methods in all scenarios.
Exille
Avcesaee ENIRY
Mathods
Mathods
Methods
(a)
(b) 7 0.2
(c) 7 0.3
Figure 5: Average Terminal Power Utility
K.4.4 Synthetic Experiment for the Merton Case
As a sanity check, we study Merton problem where the optimal performance can be mathematically derived, in order to compare the performance of FaLPO to the optimal one. We simulate data following Merton model in Appendix F.1 with ds 10, dw 10. With exponential utility function with 7 5, according to Lemma F.1 the optimal policy can be derived
p(00
(32)
Zr
Further, by taking Eqqation 132 back into Equation
14
we derive the optimal expected terminal utility
max E [U (27) =
bH (ao Tao T>-A
7
which is the theoretically optimal performance. Then, to implement FaLPO, we generate fake factors which are independent from the asset prices: the optimal policy not dependent on these factors. Idcally, FaLPO should be able automatically ignore the fake factors, and deliver performance similar the theoretically optimal derivation. The results of FaLPO, MMMC, and the theoretically optimal derivation reported in
41

---

Published in Transactions on Machine Learning Research (08/2025)
Figure 6 Note that FaLPO achieves slightly worse performance compared to the other two. The reason for the slight suboptimality of FaI.PO in the Merton case is twofold: i) the expected terminal utility is derived for continuous-tine policy while FaLPO learns a discrete-time policy with time interval At; ii) FaLPO uses an over-complicated model with stochastic factors, while the true data generating process follows a Merton model without stochastic factors.
Negative Averuge, Expecteo Terminal Utllity
PaLPO
XMMMC
Optand
Methods
Figure 6: Negative average terminal utility of FaLPO, MMMC, and negative expected optimal terminal utility. The smaller the better.
L
Extended Results of Real-World Stock Trading
L.1 Protocol
We consider 21-day stock trading in four different stock sectors using the daily stock price data from Yahoo finance between January 4, 2006 and April 1, 2022. More specifically, we use the adjusted close price as the daily trading price. For factors, we consider economic indexes, technical analysis indexes (generated by python package TA), and sector-specific factors such as oil prices, gold prices, and related ETF prices, leading around factors for each sector. In each sector we select stocks according to the availability and trading volume in the considered time range. The considered sectors, stocks, and the factors are provided in Table TO We consider the same competing methods in Section 5.1 and compare the performance using the average achieved terminal utility over different trajectories the metric. The larger the utility the better.
Sectors
Stocks
Factors
SP500 returns, MACD of stock prices, RSI of stock prices, oil prices, gasoline prices US Dollar Index Cash (DX-Y.NYB)
APA, COP, CVX, HAL, HES, MRO, OKE, OXY, VLO, WMB
Energy
SP500 returns, MACD of stock prices, RSI of stock prices, ETF prices including DIA, EXT, TYJ and VIS SP500 returns, MACD of stock prices, RSI of stock prices, gold prices, silver prices, ETF prices including TYM and VAW
BA. CAT, DE, EMR, ETN, GE. HON, LMT, LUV, PNR,
Industrial
APD, AVY, BLL, DD, ECL, FMC, IFF, IP, NEM, VMC
Materials
Table Selected stocks factors
L.2 Extra Penalties
For real-world experiments, consider two extra penalty terms for better stability. The first penalty is the model calibration loss discussed in Appendix E.3 Given trajectory with time interval At, == St, Ft, / [m]}, it is defined as
AI min C.D
Σ (l Co(X..) b1l2-
where C' matrix and vector of proper dimensions. As discussed in Appendix E.3 this penalty encourages simple representation function Φ. The second penalty is the negative sample variance of the
42

---

Published in Transactions on Machine Learning Research (08/2025)
terminal wealth, with the parameter X2 determining its strength. The intuition of this penalty is to further penalize the instability of the algorithm performance. The second penalty is implemented for all the competing methods except MMMC for a fair comparison.
L.3 Train-Validation- Test Split by Sliding Window
We detail the sliding window method for train-validation-test split. for real-world portfolio optimization experiments (Figure [ In financial markets, the dynamics under asset prices and factors vary over time, leading us to construct a sliding window on the dataset for training, validation and testing. Spocifically, given dataset of asset prices and observed factors, we construct. several windows of observations of equal length. We divide each window into three contiguous periods, the first used for training, the second for validation, and the third for testing. We refer to the length (in days) of the training period the training size (same for validation and test periods).
After constructing one window, we move the start time point by a fixed number of days (the window gap), and construct the second window. A given method trained on the training set each window separately, and then validated and tested on the corresponding validation and test sets. The final validation and test performances calculated by averaging each window. The experimental setup is summarized in We report the considered hyperparameter values in Table 12
Table II
Window Gap
Time
Window
Train
Validation
Test
Window
Train
Validation
Test
Window
Train
Validation
Test
Figure 7: Demonstration of sliding window.
Experiment Configurations
Values
Time Interval At
1 (Day)
Terminal Time T
21 Eapooential
Utility Function
Risk Aversion
Number of Replications under Each Hyperparameter
10 AWS ec2 m5ad 24xlaage
Compute Resourees
Table Sctup for real-world experiments.
43

---

Published in Transactions on Machine Learning Research (08/2025)
Hyperparameters
Values
Batch Size
{100, 200, 200,400) 400)
{0.1, 0.9}
Learning Rate
{0.0005, 0.001, 0.01, 0.1}
Window Gap
(63, 126]
Train Size
{1260)
Validation Size
{63]
Test Size
{63)
Table 12: Hyperparameters for real-world experiments
L.4 Sensitivity Analysis on À
According to Equation ( the value of À determines the weight of the FaLPO model calibration. In this section, we conduct sensitivity analysis of FaLPO on X. Under the same protocol as the experiments Section 5.2 we also report FaLPO with different values of when applied to different sectors. The results are reported in Figure g Compared
the case without model calibration (A 0), a small non-zero provides higher terminal utilities and lower variance. This observation justifies our method of incorporating model calibration into policy learning. Then, when À gets bigger and close to one, the performance of FaLPO decays while the variance also gets smaller.
Avergg thiby
Awveage EhiRy
Aveeage Unilyy
(a) Oil
(b) Material
(c) Industrials
(d) Mix
Figure 8: Sensitivity analysis for
L.5
More Information on Competing Methods
Here we provide more information for the implemented competing methods. We focus on policy learning methods, without studying other performance improving techniques like data augmentation or feature engineering which be casily applied to FaLPO. Further, for thorough comparison, we select four diverse methods that summarize the existing policy-learning methods for portfolio optimization. For fair comparison, all methods take asset price data and factors input.
DDPG is implemented with the gradient estimation detailed in Appendix C and also discussed Nan et al. (2022); Xiong et al. (2018); Jiang et al. (2017). This design makes sure that DDPG can leverage offline data without exploration.
SLAC Lee et al. 2020 learns representation of factors jointly with policy learning. But in this process, no parametric models are used.
RichID (Mhammedi al. 2020 falls into the category of model-based policy learning like Yu el. al. (2019). It first learns the representation of factors and then conducts policy learning, In this process, both steps take advantage of parametric model. For better performance in portfolio optimization, as the used model, instead of LQR as RichID originally proposed.
we pick
CT-MB-RL a policy gradient method optimizing the performance objective using the policy functional form derived from contimuous-time models, but without factor representation learning.
44

---

Published in Transactions on Machine Learning Research (08/2025)
We also implement MMMC representative of continuous-timg finance methods. More complicated and advanced continuous-time finance methods are hard to implement. for two reasons. First, to implement such methods, we need to estimate all the parameters of multivariate SDE (like 0, μ and w in Section 3.3). It is challenging since the derivation of likelihood requires solving multivariate stochastic intograls (Ait-Sahalia & Kimmell 2010), Second, deriving explicit optimal policy functions also difficult, which involves solving high-dimennional PDEs (like (2(() and Ra(t) in Lemma F.2 Further, such pure continuous-time models are expected to underperform, since they assume that the data exactly follows parametric SDE and tend to underfit. This can also be seen by our comparison with CT-MB-RL (Section 5 which is model-based RL method by relying Kin-Omberg model. As a result, we focus our empirical comparison to more competitive RL methods.
Note that FaLPO circumvents the two aforementioned challenges. First, our model calibration does not aim to fit all the parameters in an SDE, but only those related to learning 9g and 0z. That is why our model calibration loss in Section 3.3 has such an casy-to-calculate form with the paramcter Os as a simple vector. Second, FaLPO does not. need fully derived closed-form solution for the optimal policy. Like in Section 3.3, use neural networks to parameterize K(t) and o(), instead of fully deriving them like contimouus-time finance methods. Being able to bridge this gap between contimnous-tinn time finance models and high multidimensional stock trading problems is one of our contributions.
L.6 Experiments with Transaction Costs
In this section, we consider the case with transaction costs. Usually, the cost of borrowing stock to short can vary but typically ranges from 0.3% to 3% per year. Therefore, we take 1% annual transaction cost for short selling an asset. The fees are applied daily. Under this setting, we replicate our real-world experiments for the oil sector, using the same protocol. After the tuning procedure in Appendix L.3] the achieved results are reported in Table 13 The results are consistent with those without transaction costs.
Methods
Average Utility
FaLPO
22.25 + 1.649
DDPG
-6.795 + 0.8247
SLAC
-7.115 = 0.8872
RichID
-6.365 = 0.5989
CT-MB-RL
5.557 + 5.036
Table 13: Average terminal utility in oil sector with transaction costs
L.7 Experiment Results in Sharpe Ratio and Largest Drawdown
To provide more thorough review for the real-world experiment, provide the annual Sharpe ratio and maximum drawdowns for the real-world experiments conducted in the mixed sector.
Methods
Sharpe Ratio
Largest Drawdown
FaLPO
4.705
-1105.046
DDPG
2.828
-4778.342
SLAC
0.6434
-5262.523
RichID
0.418
-643.022
CT-MB-RL 4.467
-993.661
Table 14: Annual Sharpe ratio and maximum drawdowns for the real-world experiments in the mixed sector
2 drawdown the decline a partfollo from running historical maximum subsequent minimum.
45

---

Published in Transactions on Machine Learning Research (08/2025)
FaLPO achieves superior Sharpe ratios compared to competing methods. However, emphasize that utility is a more appropriate metric than returns or Sharpe ratios for portfolio optimization. Return alone is not a reliable metric for comparing portfolios, as it fails to account for risk. The utility function addresses this limitation by incorporating risk into the evaluation (see Figure 2 This is a critical feature in portfolio optimization. Utility widely recognized as a Ioundational concept in portfolio optimization, underpinned by numerous seminal works (Merton) 1969). Many finance models explicitly rely on utility. To effectively leverage such models, utility becomes an indispensable metric. Portfolio optimization is defined as maximizing expected utility. Therefore, comparing methods based on other metrics (e.g., average returns Sharpe ratio) may not be meaningful, as these metrics are not directly equivalent to utility. A policy excelling in utility may not necessarily yield high average returns or a high Sharpe ratio, as discussed in Appendix B For this reason, our paper focuses on utility when comparing different methods.
L.8 Experiments with Different Initial Wealth
We vary the initial wealth (3000, 5000, 8000, 1000} for portfolio optimization using stocks the oil sector following the same protocol the experiments in Section 5.2 The results are summarized in Table [5] Specifically, FaLPO achieves superior performance to the competing methods with different initial wealth. Note that all the methods achieve higher terminal utility given more initial wealth.
Initial Wealth
3000
5000
8000
10000
FaLPO
-21.08 + 16.775
-2.4 1.9
-0.243 = 0.1209
-0.03595 0.008896
DDPG
-909.5 = 3443.846
-6.6 1.2
-0.34665 = 0.01809
-0.046755
0.00444
SLAC
-11865 = 47260.664
0.2
0.35465 = 0.07427
-0.04558 = 0.00142
RichID
-45.51 + 8.368
-6.5
-0.33125 + 0.009791
-0.045363 = 0.0001446
CT-MB-RL
28.715 + 18.303
-4.2 6.2
-0.30995 + 0.1441
-0.043055 0.002061
Table 15: Average terminal utility in oil sector with different initial wealth
L.9 Experiments with Different Continuous- Time Models
As demonstrated in Appendix F. FaLPO is compatible with various continuous-time financial models. In this section, evaluate its performance real-world oil sector dataset, comparing results across three SDE families: the Merton model, the Kim Omberg model, and the EVE model.
Sectors
Average Utility
Kim Hinn-Omberg
-2.4
Merton
-855
EVE
-2.5
L.10 Computation Time and Memory Usage
In this section, we report the memory usage and computation time of FaLPO. All experiments are conducted AWS EC2 instances of type 24xlarge, using CPUs only. The experiments require less than GB of CPU memory. The computation times summarized in Figure
46

---

Published in Transactions on Machine Learning Research (08/2025)
0.010
Frequency
Frequency
00.002
0.010
0.001
0.000
0.000
0.00
Rum Time
Run Time
(a) Syuthetic
(b) Real-World
Figure 9: Histograms for computation time of synthetic and real-world experiments under all the considered hyperparameters.
L.11 Performance of FaLPO over Different Market Regimes
As the market regime arguably changes before and after 2020, delve deeper into the performance of FaLPO across these two periods. Specifically, report the return and standard deviation of FaLPO on the oil sector before and after 2020 in Table 16 While performance decline is observed after 2020, the decrease
is relatively mild.
Period
Return (%)
Std (%)
Before 2020
7.8
2.8
After 2020
7.0
3.6
Table 16: Performance of FaLPO Before and After 2020
47


Reviews:
[{"review_id": "8cTkhe63d4", "reviewer": "Reviewer_oYDW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n*  Learns a neural mapping from high‐dimensional market signals to low‐dimensional latent factors and embeds those factors and asset prices into a structured SDE, thereby combining the flexibility of deep networks with the interpretability and regularization of parametric continuous‐time models.\n\n* Proves that the restricted policy class converges to the true continuous‐time optimum as rebalancing frequency increases and provides a finite‐sample performance bound that quantifies trade‐offs between utility maximization, model calibration, and policy‐gradient noise.\n\n* Demonstrates significant outperformance over five strong baselines on both synthetic Kim–Omberg data and real‐world stock portfolios.\n\nWeakness:\n\n- The framework requires a known analytical solution for the continuous‐time optimal control. This severely limits FaLPO’s applicability to toy SDE families and means the approach cannot generalize to most real‐world settings where continuous‐time optima must be approximated numerically or do not exist analytically.\n- The asymptotic and finite‐sample guarantees rely on strong conditions such as Lipschitz continuity, bounded gradients, and local strong concavity of the joint loss in the parameter space. However, deep neural networks used for factor extraction rarely satisfy these smoothness or convexity properties, making the theoretical results practically irrelevant.\n- The evaluation lacks out‐of‐sample or cross‐market tests. As a result, there is no evidence FaLPO would remain stable under genuine market stress or in broader investment universes, undermining confidence in its real‐world utility.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces FaLPO, a novel framework that unifies deep representation learning with continuous‐time portfolio theory by using a neural network to extract low‐dimensional latent factors from a high‐dimensional set of market signals, embedding those learned factors and asset prices in a parametric stochastic differential equation, and constraining the trading policy to follow the functional form of the continuous‐time optimum. By jointly optimizing factor extraction, SDE calibration, and policy performance through a single objective that balances expected utility against model‐fit, FaLPO dramatically reduces overfitting and sample complexity compared to unconstrained, model‐free reinforcement learning. The authors prove that as the rebalancing frequency increases to continuous time, FaLPO’s discrete‐time policy converges to the true continuous‐time optimum and provide a finite‐sample performance bound that quantifies how algorithmic choices affect suboptimality in practice. Extensive experiments on both synthetic Kim–Omberg data and real‐world stock portfolios from 2006 to 2022 show that FaLPO consistently outperforms five strong baselines, including model‐free methods like DDPG and SLAC as well as model‐based approaches like RichID and CT‐MB‐RL, achieving higher terminal utility, greater robustness under transaction costs, and improved risk‐adjusted metrics.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Discuss how to select or validate the SDE class before applying FaLPO\n- Provide a small experiment comparing at least two different SDE families and report both calibration quality and policy performance\n- Explicitly list all technical assumptions required for the convergence theorems\n- Discuss whether typical neural network architectures satisfy these assumptions in practice, or propose modifications\n- Describe the ranges of values tried for the balance parameter λ, learning rates, and network sizes\n- Provide wall‐clock training times and GPU memory usage for FaLPO on the synthetic problem and one real‐world sector\n- Split backtests into distinct market regimes (e.g., pre‐2008, post‐2009, 2020-2022) and report performance separately for each period", "broader_impact_concerns": "- Advanced algorithmic trading strategies that leverage FaLPO could be deployed by well‐capitalized firms, potentially crowding out smaller investors who lack access to similarly sophisticated tools.\n- The neural factor extractor learns from historical data, which may embed biases. If these biases go unchecked, policies could steer capital away from underrepresented or emerging markets.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IMRA94I33s", "reviewer": "Reviewer_CJ1x", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength** \n\n1. The interpolation between continuous-time finance models and discrete-time reinforcement learning is novel and well-motivated.\n\n2. The paper includes both asymptotic convergence guarantees (Theorem 4.1) and non-asymptotic finite-sample bounds (Theorem 4.2).\nThis combination strengthens the method’s credibility and positions it well against both RL and traditional finance methods.\n\n3. Experiments span synthetic data (with controlled volatility and factor complexity) and real-world portfolios.\n\n**Weakness**\n\n1. While the NSFM setup allows for flexible representations ${\\phi}$, the paper does not inspect or interpret what the learned factors capture.\n\n2. The asymptotic convergence (Theorem 4.1) and non-asymptotic bound (Theorem 4.2) rely on: strong concavity of H, lipschitz continuity and bounded moments, and admissibility constraints.These assumptions may not hold in high-noise financial markets or with real-world data, especially regarding the curvature of H.\n\n3. The paper compares FaLPO to five baselines, but does not ablate its own components (e.g., with/without model calibration, with/without policy regularization).\n\n4. The paper uses utility (power, exponential) as the primary performance metric, but does not report risk-adjusted metrics like: Sharpe Ratio or Conditional Value-at-Risk.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes FaLPO, a model-based reinforcement learning framework designed for continuous-time financial markets. It integrates policy gradient methods with learned low-dimensional latent factors that drive market dynamics. The method builds on Neural Stochastic Differential Equations, utilizes MLE and policy gradients, and is evaluated both theoretically (finite-sample and convergence guarantees) and empirically (simulated and real-world markets).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Provide insight into the learned representation\n\n2. Verify the assumptions empirically for theoretical analysis if possible.\n\n3. Conduct some ablation studies such as with/without model calibration, with/without policy regularization", "broader_impact_concerns": "The paper lacks a formal Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "a27wsIXBgr", "reviewer": "Reviewer_S9uZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I found this paper difficult to evaluate. While it is true that the machine‐learning literature contains few studies that combine continuous‐time finance models with reinforcement learning, such an approach is actually classical and fairly standard in finance. In finance, intertemporal asset allocation problems are formulated through the Bellman equation, and it is common to assume a particular stochastic process. When researchers “solve’’ the Bellman equation, they typically rely on dynamic programming or approximate dynamic programming techniques rather than reinforcement learning, even though the underlying mathematical formulation is identical. Intertemporal portfolio‐choice models of this type have been studied since at least the 1980s, including within macroeconomics.\n\nBecause the topic is highly domain specific, the machine‐learning community has not been accepted such studies. On the other hand, the finance community has traditionally emphasized analytical solutions to the Bellman equation; algorithmic approaches akin to reinforcement learning are less common, which creates a gap between the two fields. Nevertheless, I do not think that there is no previous related work.\n\nIf my research had been limited to machine learning only, I might not have opposed acceptance. However, I am active in both economics and machine learning, and this dual perspective leaves me uncertain. Regrettably, I suspect that the paper might be better suited to a finance or macroeconomics journal than to TMLR. For instance, Mathematical Finance—where, for example, Professor Elad Hazan, a prominent machine‐learning scholar, has also published—and the Journal of Economic Dynamics and Control, which regularly features similar work, seem more appropriate venues.\n\nThere are specialists who focus precisely on this area and may be aware of closely related research. Because I am not an expert in the subfield that rigorously specifies stochastic processes and solves intertemporal optimization problems, it is somewhat difficult for me to judge the quality of this study. I will not object if the other reviewers lean toward acceptance, but from an academic standpoint I believe that submitting the paper to a finance journal, where it can be properly assessed, would be preferable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript develops a reinforcement learning (RL) framework for continuous-time portfolio optimisation, called FaLPO (Factor Learning Portfolio Optimization). The authors model asset price dynamics using a neural stochastic factor model, which embeds a flexible, data-driven factor representation into a continuous-time stochastic differential equation framework. Building on this, they propose a model-regularized policy learning algorithm that incorporates both the functional form of optimal continuous-time policies and model calibration objectives into a joint learning scheme.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See Strength and Weakness.", "broader_impact_concerns": "See Strength and Weakness.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UmwjB60Itd", "reviewer": "Reviewer_BASm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think the idea is very interesting and potentially applied to other situations. For example, we can choose an LQR-like dynamics as a stochastic factor model, compute an optimal policy, and then, use it to guide the learning of a more flexible neural network policy.\n\nA potential weakness is that the paper is probably not easy for most of machine learning researchers not familiar with financial methods. From this perspective, maybe this submission is not well-suited for TMLR since __\"TMLR's objective is to publish original papers that contribute to the understanding of the computational and mathematical principles that enable intelligence through learning, be it in brains or in machines\"__. (A quote taken from TMLR's Submission Guidelines and Editorial Policies) The current form of this submission does not seem to well align with this objective, given that it focuses on financial portfolio optimization.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Disclaimer: I am not an expert on financial methods. (I am not sure why this paper is assigned to me) My review is an educated guess from a perspective of an RL researcher.\n\nThis paper proposes a novel way to combine continuous-time finance methods and reinforcement learning algorithms. Concretely, the proposed algorithm learns a feature mapping that maps stochastic factors, such as economic indices, and market benchmarks, to a latent space in which an existing stochastic factor model well predicts the evolution of asset prices. Since for many stochastic factor models, an optimal solution is known, this optimal solution is computed and used to guide the learning of reinforcement learning algorithm.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I have no requested change.", "broader_impact_concerns": "There is no concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)