Factor Learning Portfolio Optimization Informed by Continuous-Time Finance Models
Authors: Sinong Geng
Houssam Nassif
Zhaobin Kuang
Anders Max Reppen
K. Ronnie Sircar
Venue: Submitted to ICLR 2023 (2023)

Abstract: We study financial portfolio optimization in the presence of unknown and uncontrolled system variables referred to as stochastic factors. Existing work falls into two distinct categories: (i) reinforcement learning employs end-to-end policy learning with flexible factor representation, but does not precisely model the dynamics of asset prices or factors; (ii) continuous-time finance methods, in contrast, take advantage of explicitly modeled dynamics but pre-specify, rather than learn, factor representation. We propose FaLPO (factor learning portfolio optimization), a framework that interpolates between these two approaches. Specifically, FaLPO hinges on deep policy gradient to learn a performant investment policy that takes advantage of flexible representation for stochastic factors. Meanwhile, FaLPO also incorporates continuous-time finance models when modeling the dynamics. It uses the optimal policy functional form derived from such models and optimizes an objective that combines policy learning and model calibration. We prove the convergence of FaLPO, and provide performance guarantees via a finite-sample bound. On both synthetic and real-world portfolio optimization tasks, we observe that FaLPO outperforms five leading methods. Finally, we show that FaLPO can be extended to other decision-making problems with stochastic factors.

Under review as a conference paper at ICLR 2023
FACTOR LEARNING PORTFOLIO OPTIMIZATION INFORMED BY CONTINUOUS -TIME FINANCE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
We study financial portfolio optimization in the presence of unknown and un- controlled system variables referred to as stochastic factors. Existing work falls into two distinct categories: (i) reinforcement learning employs end-to-end policy learning with flexible factor representation, but does precisely model the dy- namics of asset prices or factors; (ii) continuous-ting finance methods, in contrast, take advantage of explicitly modeled dynamics but pre-specify, rather than learn, factor representation. We propose FaLPO (factor learning portfolio optimization), framework that interpolates between these approaches. Specifically, FaLPO hinges on deep policy gradient to learn a performant investment policy that takes advantage of flexible representation for stochastic factors. Meanwhile, FaLPO also incorporates continuous- time finance models when modeling the dynamics. It uses the optimal policy functional form derived from such models and optimizes objective that combines policy learning and model calibration. We prove the convergence of FaLPO and provide performance guarantees via a samite-sample bound. On both synthetic and real- world portfolio optimization tasks, we observe that FaLPO outperforms five leading methods. Finally, we show that FaLPO can be extended to other pecision-making problems with stochastic factors.
INTRODUCTION
Portfolio optimization studies how to allocate investments across multiple risky financial assets such as stocks and safe assets such as US government bonds. The investment target is often formulated as maximizing the expected utility of the investment portfolio's value fixed time horizon, which conceptually maximizes profit while constraining risk (von Neumann & Morgenstern 1947). With continuous- stochastic models of stock prices, great advances in the expected utility maximization framework were made in Merton (1699) using stochastic optimal control (dynamic programming) methods. More realistic models incorporate factors like economic indices and proprietary trading signals (Merton et al. [9733 Fama & French 2015 1992). which (i) affect the dynamics of stock prices; (ii) stochastically evolve over time; (iii) are not affected by individual investment decisions. With greater data availability, it is natural to design and apply data-driven
machine learning methods (Bengio 1997 Dixon et al., 2020 De Prado (018)) to handle factors for portfolio optimization. This work proposes novel method Factor Learning Portfolio Optimization WhiPch combines tools from both machine learning and continuous- finance.
Portfolio optimization with stochastic factors is challenging for three reasons. First, financial data is notoriously noisy and idiosyncratic (Goyal & Santa-Clare 2003), causing complex purely data-driven methods be unstable and prone to overfitting. Second, the relationship between the factors and their impact on stock prices can be extremely complicated and difficult to model ante. Third, many successful finance models are in continuous time and require interacting with the environment infinitely frequently. As result, such models cannot be casily combined with machine learning methods, many of which are in discrete time.
Current approaches to portfolio optimization broadly fall into two categories: reinforcement learning (RL) and continuous- time finance methods. Many RL solutions to portfolio optimization are built on deep deterministic policy gradient (Silver al. 2014 Hammly et al. 2021, Section 4.3). Such methods parameterize the policy function as a neural network with strong representation power and learn the neural network by optimizing the corresponding portfolio performance. However, these approaches

---

Under review as a conference paper at ICLR 2023
(as well as other model-free methods like Haarnoja al. (0118) have high sample complexity and tend to overfit due to the high noise in the data. Other RL methods explicitly learn representation (Watter
ct al. 20133 Lee al. 2020; Laskin et al. [202T) and leverage pascrete-time models (Deisenroth & Rasmussen 2011: Gu et al. 2016; Mhammedi et al. 2020) Janner et al., 2019, Nagabandi et al.
2018). Nonetheless, these methods are not informed by continuous-time finance models and, as
experiments suggest in Section cannot benefit from structures inherent in the financial market.
Stochastic factor models can be used mathematically derive optimal (or approximately optimal) investment policies (Kim & Omberg 1996] Chacko & Viceira, 2005 Fouque et al. 2017 Avanesyan 2021). To this end, one needs domain knowledge to pick and model the factors. Then, model calibration (a.k.a. model fitting. parameter estimation) is conducted by maximizing a calibration objective. With the calibrated model, the optimal investment policy can be derived analytically or numerically (Merton 1992 Fleming & Soner 2006). This procedure of calibration and optimization effectively constrains the 'leaniigg' the optimization step, and thus helps reduce overfitting to noisy data. However, this approach cannot capture the complicated factor effects in the data, because the factors may be complex and unlikely to be identified manually. Therefore, these methods may end up with oversimplified models and suffer from model bias with suboptimal performance.
To tackle these limitations, we propose factor learn- ing portfolio optimization (FaLPO). new method that interpolates between the two aforementioned solutions (Figure D FaLPO includes (i) neural stochastic factor model to handle huge noise and com- plicated factor effects and (ii) a model-regularized policy learning method to combine continuous-timg models with discrete- sescrete-time policy learning methods. First, to reduce the sample complexity and avoid over- fitting, Fa assumes factors and asset prices follow parametric continuous-time finance model. To cap- ture the complicated factor effects, FaLPO models the factors by representation function parameterized by neural network with minimal paramet- ric constraints. Second, for policy learning, FaL.PO incorporates two regularizations derived from continuous- time stochastic factor models: a policy functional form and model calibration. Specif- ically, we derive policy functional forms from the neural stochastic factor model using stochastic optimal control tools, and apply it to parameterize the candidate policy in FaLPO. The use of this form in the learning algorithm effectively acts as regularizer. Then, model calibration and policy learning are conducted jointly, such that the learned policy informed by continuous-tima models.
Challenpss
Our Sehuiinn
Existing Works
Hegg Narkr
Youna Maur booke
Lanin
Fscar Eliics
FaLPO
ve. Evetnann Thay
Mueit Repakaiont Iuaayy Forning
Plaane
Figure 1: Demonstration of FaLPO
Theoretically, we prove that the added continuous-ting regularization leads to the optimal portfolio performance as the trading frequency increases. Empirically, we demonstrate the improved perfor- mance of the proposed method by both synthetic and real-world experiments. We review the related literature in Appendix[A] A We also discuss how FaLPO extendable beyond portfolio optimization, and can be applied to other decision-making problems with stochastic factors in Appendix H
2 BACKGROUND
In this section, we first formulate the portfolio optimization problem. We then review two major solutions this problem: deep deterministic policy gradient in reinforcement learning (RL) and stochastic factor models in continuous-time finance.
2.1 PORTFOLIO OPTIMIZATION
Problem Formulation Portfolio optimization seeks to derive a policy of asset allocation that yields high return while maintaining low risk for the investment. Formally, consider ds risky assets with prices S. : S .... sds]T and risk-free money market account with, for simplicity, zero interest rate of return (like cash). We observe dy features (c.g. economic indices, market benchmarks) denoted as Y. From Y,, we can derive dx factors denoted as X, which (i) affect the dynamics of asset prices; (ii) evolve over time stochastically; (iii) are not affected by investment decisions. Given an initial investment capital (or wealth) za and the initial values for Y, and S, yo and sp, we use
2

---

Under review as a conference paper at ICLR 2023
ds x 1 vector to denote the fractions of wealth invested in the dg assets at time point t. Note that negative values are allowed in Te indicating short positions. At the terminal time T > 0. the target is to maximize the expectation of a given utility function E(U where U Z R with Z C R is the utility function and Z denotes the terminal wealth under π.
Intuitively, utility function reflects the risk preference of an investor. It is an increasing function of wealth that is also concave: it changes significantly when the wealth is low but less so when the wealth is high (Figure 2). This work focuses on the power utility 72 := 1-
7/8
Reponine Utility
Power Utility
with 2 = R+, > 0, and + 1; and the exponential utility (2; == exp(-22) with Z R and 7 > 0. 7.
Wealth
Here, is the investor's risk aversion coefficient and is hand-picked (instead of tuned) by the user. A larger yy corresponds to more risk aversion, while smaller y cor- responds to more risk tolerance. Beyond these two utilities, our method is also applicable to other utility functions and other objective functions for portfolio optimization (see Appendix B
Figure 2: Power & exponential utilities.
Discrete- and Continuous-Tine Policies Discrete- and continuous-time policies are two major types of investment policies, differing on how frequently the portfolio is rebalanced. A discrete-time policy rebalances the portfolio fintely frequently. leading to a discrete-time dynamics for the wealth. Such policies are often considered in RL methods (Section 2.2p. Continuous-time policies rebalance the investment infinitely frequently, leading to a continuous time dynamics for the wealth. These policies are often found explicitly in continuous- time finance models (Section 2.3
2.2 DEEP DETERMINISTIC POLICY GRADIENT
We review deep deterministic policy gradient (DDPG, Silver al. 2014) a quintessential example of RL methods. DDPG does not explicitly model the dynamics, but instead directly learns discrete- time policy for portfolio optimization. DDPG parameterizes the policy function as deep neural network and conducts gradient-based policy learning. Denote by (t., St, Zt, Y;; 0p) the deep policy function with parameter 0p. Without explicitly modeling the dynamics of S, or X,, DDPG directly maximizes the following performance objective to learn a policy:
max ((D) with (0D) = EEU ((0))) 8p
(I)
where the expectation is over the terminal wealth Z (-:9p) following the policy T(; 0p). A key step of DDPG is to compute the gradient of V ((0)) to update 0p. Following the procedure in Appendix this can be achieved by sampling the trajectories of S, and Y, approximate the expectation and thus the gradient of ((D)).
Typically, DDPG learns discrete-time policy that rebalances the portfolio finitely frequently. To see how the policy rebalances the portfolio, we study its corresponding wealth process 2, that characterizes the changes in wealth over time. Let Al > be the time interval (e.g. daily, weekly) to rebalance the portfolio and, for integer M > 0, let = MAt be the fixed in- vestment horizon (c.g. one or two months). At time mat with m E 1,, M 1}. define Tmat := ('mAt, Smat: Zmar, Ymar; 0D) as the fraction of current wealth invested in the 7th risky asset. Then, the wealth change at time mál is: z:(:(0)
(1(p)
(m++))a:
(1180)) Ets T Stna mmat
Am- - is the wealth change Si Stnae where ZmAt "mAr S
due the investment in the 7th risky asset. Note that the number of shares invested in asset (Z
( kmaa) does not change during (mAt, (m + 1)) the portfolio rebalances every At time.
Snat
RL methods like DDPG provide flexible representation for factors: the hidden layers of the neural network are considered as the representation learned for Y,, providing strong representation power.
Note that it is impossible to rehalance portfolio infinitely frequently in practice. Thus, continuous-time policies are more useful as analytical tools.
3

---

Under review as a conference paper at ICLR 2023
Nonetheless, there is not explicit parametric model for the learned representation and asset prices. Consequently, such methods require lots of data and tend to overfit (Aboussalah 2020).
2.3 STOCHASTIC FACTOR MODELS
We review stochastic factor models in continuous-time finance. These models explicitly formulate the dynamics and can also be used to mathematically derive the functional form of the optimal continuous-time investment policy. Stochastic factor models are described by stochastic differential equations (SDEs) (see Oksendal 2013 and Appendix D to formulate the dynamics of asset prices Sr. Specifically, with dx x 1 factor variable Xt, let W, == [W: W.? ... w:dw be dw x 1 Brownian motion that characterizes random fluctuations. Then, St and X₃ are assumed follow
ds; = fs(X1; dt
dw
dt Σ (X:
i f1,2, .ds).
S
(2)
dx, = (x Ug dt + gx (X);05) dW,.
In (2). fs R + R fx Rdx + Rdx, Is Ráx + Rdaxxyy and gx R + Rdx xdw parametric functions pre-specified by domain knowledge. Further, fs and fx are often referred to as the drift, and Os and 9x as the volatility of S. and X2 respectively. Intuitively, SDEs formulate the change of a variable in infinitesimal time step the sum of deterministic part (dt) and stochastic part (d and we use f to parameterize the SDE. The factor X, appears in both the drift and volatility of the asset prices, thus affecting the price transition. With the parametric functional forms in 2 we can use tools in stochastic optimal control to derive the functional form of the optimal continuous-timg time investment policy.
Continuous-time policies change the investment in each asset at every time point. For a continuous- dē- time investment policy क. the dynamics of wealth 2, defined as zr := ds #;ds;
with 2
St
20, So Sn and X, -z. Crucially, this is different from the discrete-ting wealth process Z in
Section 2.2 as the number of shares 27 in asset now changes continuously over time, opposed
S7
being rebalanced at finite intervals. This discrepancy creates obstacles to directly apply the results derived from stochastic factor models to discrete-time policy learning.
Stochastic factor models can reduce sample complexity for portfolio optimization, since the assumed functional forms in (2) significantly constrain the solution space. However, crucial step to apply stochastic factor models to pick or even construct X, from the observed Y; that perfectly follows pre-specified model. This step often relies domain knowledge and thus may end up with oversimplified models suffering from model bias and eventually leading to suboptimal performance.
FACTOR LEARNING PORTFOLIO OPTIMIZATION
We propose factor learning portfolio optimization (FaLPO), a new decision-making framework that interpolates between DDPG and stochastic factor models. FaLPO has two components: (i) neural stochastic factor model to handle huge noise and complicated factor effects and (ii) model-regularized policy learning to combine continuous- time models with discrete-time policy learning methods.
3.1 NEURAL STOCHASTIC FACTOR MODELS
We describe neural stochastic factor models (NSFM) and discuss their benefits. On the one hand, neural stochastic factor model assumes the existence of representation function á such that the factors of the problem can be directly learned from its features: X, = o(Y;: 62). Here, o is formulated as a neural network with parameter 6% (Figure 3h. As a result, FaLPPO avoids hand-picking
factors from features as the case in stochastic factor models (Section 2.3 The neural network representation has only few parametric constraints and thus is able capture complicated factor effects in the data. Furthermore, factors X, and asset prices S, are assumed to follow a stochastic factor model (e.g. and (6)), which reduces the sample complexity and avoids overfitting.
4

---

Under review as a conference paper at ICLR 2023
3.2
POLICY LEARNING
Under the proposed neural stochastic factor model, we aim to learn the discrete-time optimal policy function z and the representation function ((;;0))) However, while the policy learning for discrete-time policies, our proposed model is in continuous time. To bridge this gap, we incorporate Iwo types of continuous-time model regularization into discrete-time policy learning: (i) the policy functional form (3) and (ii) the model calibration objective
Policy Functional Form From our model, we apply the functional form of continuous-time optimal policy into our discrete-time policy learning. Using tools stochastic optimal control, we can derive the functional form of an op- timal continuous- time policy: क. = [[t S, Ze. Xy; f where the functional form of II can be obtained in many existing stochastic factor models (Kim & Omberg, 1996; [hacko & Viceiral 2005 Avanesyan 202 1 Za riphopoulou] [001] Wachter 2002 Kraft 2005). and 02 an optimal parameter for I1. FOLPO uses the functional form of II in policy learning and parameterize the candi- date policy as
Stochastic Factor Models
X
Factor
Representatio
%
Not Speoiickke
Tire
Figure 3: Demonstration for neural stochastic factor models.
(t,, Ze. 0s, 0x) = 11(t, St,, Z,, (Yri
(3)
Algorithm 1 FaLPO
where o is the representation function for the factors Section 3.1 As result, II constrains the policy space and acts as regularization. Importantly, although II is derived for continuous-time policies, it can still provide guidance for cerrete-time policy learning when At is small. We rigorously prove the soundness of using in Section 4
1: Input: number of iterations N,
2: Initialize 0g and 0z,
3: for [MV] do
4: Parameterize the policy function according 3
5: Estimate the policy gradient for in (5) (Appendix C).
Model Calibration FaL.PO also hinges on model cali-
6: Update 0g. Ou. and 0s.
bration to regularize policy learning. Given the specific functional forms ( FaI.PO conducts model calibra- tion to estimate the parameters of the SDE. The calibration procedure can be summarized as maximizing a model cal- ibration objective:
7: end for
8: Return <.;;0a. 0x))
max I(O0,, 0s)- 0.
In practice, with discrete data, one may use likelihood (Phillips) 1972 Beskos Roberts 2005) or other likelihood- based objective functions (Bishwal 2007 Satalia & Kimmel| 2010) for I (see Appendix E for concrete examples).
To harness the information provided by model calibration in policy learning, FaLPO combines the model calibration objective L. in (4) with the performance objective V in (T) and facilitates a joint optimization over the two. Note that naively combining T.(Og, 0g) and V(Op) is not effective since the two in general do not share common parameters: the parameter the policy network 0p has no overlap with the SDE parameter 0g or factor representation parameter 0s. However, by constraining the policy space to (3) in FaL.O,, we can show that 0g is also part of the policy parameterization.
Thus, V can be derived as V(0g, 0,)) == E[U (Zzz (=9+.9x))) }]. In other words, 0g shared in both V and L, and hence FaLPO can carry out a joint optimization over the two:
H(0 Oz. 0s), with ((0,, Oz 0s) = (1 (0,, 0.) + AL 0s), 00.00..ss)a.
max
(5)
where the candidate policy follows the functional form of the optimal continuous-time policy 3 (see Algorithm|| [] and A denotes the considered parameter set. The model calibration objective also acts as model regularization, where À E (0, 1) is hyperparameter determining its effect. In practice, we can optimize (5) by gradient-base based methods, facilitating a easy and end-to-end learning procedure (see Appendix for gradient estimation details).
5

---

Under review as a conference paper at ICLR 2023
3.3 EXAMPLE OF FALPO
In portfolio optimization, one can use different types of stochastic factor models. FaLPO can be applied to many such types (Appendix F.. In this section, we use the Kim-Omberg model (Kim & Omberg 1996) as an example to illustrate FaLPO''s modeling and policy learning. Kim-Omberg standard model for portfolio optimization with stocahstic factors, which has been extensively studied empirically (Welch & Goyal 2008; Muhle- Karbe et al. 2017). For modeling, FaLPO with Kim- tin-Omberg model formulates the dynamics of asset prices and factors as
dsp S
Xj dt
ta dw?, dX, = p(w X2) dt + odW, and X, = ((Y;;)))
(6)
where SDE parameters w, a, and μ constant matrices or vectors.
For policy learning, we detail the policy functional form and model calibration. Under the Kim- Omberg model, we can derive the optimal policy functional form II in (3). Specifically, for power utility (I,,, S. Ze, (Ye; fa); 0x)) = k (1; (YY; + k2 (x); for exponential utility St,, Zı, (Y;; 0s);;;;); 0x) = kn 0- (Y: + (t; 0z)/2z, where (=; [ Rdsxdx and (x))
[0. - Rdaxdx are two time dependent functions (Appendix F.2 We can derive the functional forms of kn and l22 since the two are solutions to systems of ODEs related to algebraic Riccati equations (Appendix GG. We can also directly use function approximators like neural networks or kernel methods for the two. For model calibration, we use negative mean square loss with the derivation deferred to Appendix E L(Og, 0s)
[lgg(S++a)) log(S)) / (Yt;; 0z) 0/] where in this case Og is a ds x 1
-E vector. Therefore, to implement FoLPO, we can parameterize the candidate policy function using II and optimize (5).
Note that the methodology of FaLPO is also generally applicable to other decision-making problems besides portfolio optimization. In Appendix [ use linear quadratic control with stochastic factors as an example to demonstrate the generality of FaLPO.
4 THEORY
We theoretically analyze both the asymptotic and non-asymptotic characteristics of FaLPO.
4.1 ASYMPTOTIC ANALYSIS
FaLPO applies the policy functional form and model calibration derived from continuous-time models to discrete-time policy learning. We show that FaLPO can achieve the optimal performance asymptotically (i.e. with infinite data and perfect optimization). In the following, we describe the assumptions and results and provide the formal theorem in the end.
We provide intuitive description on the assumptions, with the formal statements provided in Appendix [.T] First, we assume that the portfolio optimization problem satisfies some standard regularity conditions (Higham al., 2002): the drift and volatility are locally Lipschitz continuous; meanwhile, the asset prices, the stochastic factors, and the wealth process under the optimal policy have bounded moments. Second, we assume that the utility function Uz) has linear growth on
E Z. Note that some widely used cases like power utility with y < 1 and exponential utility with lower-bounded wealth satisfy this assumption. Third, we consider only admissible policies with parameters in A and we assume that A covers the optimal continuous-time policy parameter. A policy admissible it is predictable and if the wealth process Z E 2 for any E (0, T] almost surely. It is common practice to only consider such admissible policies in portfolio optimization. The last two assumptions are artifacts the current theoretical analysis; in practice FaLPO can achieve reasonable performance without enforcing them (see Section
With the foregoing assumptions, we show that the performance of FaLPO can asymptotically converge to that of the best policy in discrete time. In detail, we define V = V(T*) where x² is optimal discrete-time admissible policy with time interval At, i... Vi is the optimal performance obtained without constraining to the functional form 3 or leveraging model calibration like (5). Next, define

---

Under review as a conference paper at ICLR 2023
0. = (03. 03,.0 E arg (e.. (8s, 0x. 0s) with the policy functional form (3) such that (,0 is the performance that FaL.PO can achieve with infinite data and perfect optimization. Then, Theorem 4.1 shows that the gap between Vä. and V ((2)) converges to zero as At goes to zero.
THEOREM 4.1 With assumptions in Appendix [.1
limar-+ (V3. - V((%)) 0.
Theorem 4.1 justifies the methodology of FaL.PO under small time interval. The proof is provided in Appendix 1.2 and Appendix 1.3
4.2 NON-ASYMPTOTIC ANALYSIS
We study the finite-sample performance of FaLPO. We describe the problem setup, major assumptions, and then provide the theorem. In each iteration, we collect independent trajectories to estimate the gradient of H. Let O,, be the estimate after the nhth iteration, and N the total number of iterations.
we analyze the average estimate 0 := Σm= On instead of dx. which is common technique for stochastic optimization analysis. Specifically, we aim bound the expected difference between Vi. and (0). Note that is extremely challenging to theoretically analyze non-convex stochastic optimization (5) without further specifications in problem setup and assumptions (Polyak) [963
Bhandari Russo 2019] Jin al. [02T;; Mal 2020; Wang al. 2019). Therefore, we consider vrojection-based variant of FaLPO, under which the optimization process is conducted in a bounded parameter set B S A. Furthermore, we assume that the objective function H is strongly concave in
with a local maximal point 0/, == (f).::0% pp (.,,))) Similar local curvature assumptions are commonly used to analyze non-convex problems (Bach et al. 2017 Loh 2017). With the above setup and assumptions, the expected gap between V/,, and V(0) satisfies the following finite-sample bound Theorem
THEOREM 4.2 With the aforementioned projection-based FaLPO algorithm and assumptions, both detailed in Appendix J.2
there exist positive constants C1, C2. C3. and C₁ such that
H( H(tt)
C1 log(N)
B V(0)
+ N
1-1
C1 (N)
BN(1 (1-1)90
X) + C + 2X(1 A)CA],
where E [0,1]. Also, est is an error term not related to N or B but dependent on At with lim Cat 0.
Theorem 4.2 provides non-asymptotic upperbound the gap between the optimal performance VX and the achieved by FaL.PO (0).. We briefly comment on each term in the upperbound. eat bounds the asymptotic performance gap caused by leveraging the continuous-time policy functional form constraint and model calibration, and we explain its connection to Theorem 4.1 in Appendix J.4
H(0%.)-((11)) controls the performance gap between the local optimal point 0h, and the global 1-X optimal point 8%. The remaining terms characterize the performance gap between and 0
Theorem 4.2 has two implications. First, the bound in (7) is rational function of X. Accordingly. there exist situations where À E (0, 1) provides smaller upper bound than À indicating the possibility that tuning À can provide better performance (see experiments in Appendix [). Second, when 1, the bound diverges to infinity. This makes sense since when 1, (0 0z: 0s) = L(Os, 0s) does not contain 0,: the algorithm does not learn the policy. We prove the theorem in
Appendices J.I J.3 and
5 EXPERIMENTS
By incorporating continuous-timg time finance models into policy learning. FaLPO deal with high data noise and complex factor effects. this section, we demonstrate the improved performance of Fal.PO against existing portfolio optimization methods, over synthetic and real-world experiments.
7

---

Under review as a conference paper at ICLR 2023
Cantinan-Tiime
Daneet-TTmee Madel
Annual Volacility
0.1
0.2
0.3
Repecentstion
FaLPO
-0.465 + 0.446
-1.35 4: 0.165
-2.737 = 0,219
MMMC DDPG
DDPG
-1,600 0.500:0.00 0.456
-3.30 1.291
-5.195
1.299
SLAC
SLAC
-0.730
-5.50 0.011
-0.160 0.012
RichiD
V
x
RichID
0.102
0.048
CT-MB-RL
2,850 0.014
5.35 + 0.020
6.160 0.026
FalPo
MMMC
-4.723 7.619
5. 3.602 4.290
6.124 3.217
Table 1: Competing methods and their characteristics.
Table 2: Average terminal utility after tuning with standard deviation for synthetic data
5.1 SYNTHETIC EXPERIMENTS
Metries We compare different methods using the average terminal utility since is the ultimate goal in our portfolio optimization problem formulation and commonly used in continuous-time finance models. There exist other statistics measuring the performance of portfolios (see Section B). These statistics are not equivalent consistent with the utility, and thus we do not emphasize them.
Methods We compare FaLPO with five competing methods representing various approaches in prior work (Table I] (i) Merton Model with Model Calibration (MMMC): model calibration of classic continuous time finance baseline. which does not consider stochastic factors (Merton 1969), combined with closed-from policy function. (ii) Deep Deterministic Policy Gradient (DDPG): state-of-the-art model-free RL method with deterministic policy (Silver et 20.44) which many empirical portfolio optimization methods build on. (iii) Stochastic latent actor critic (SLAC): a ofte--of-the-art representation learning RL method that explicitly learns representation of latent variables (Lee et al.) 2020). (iv) Model-based RL with rich observations (RichID): a state-of-the art model- based RL method with representation learning (Mhammedi et al., 2020). (v) Continuous time model-basee RL (CT-MB-RL)) policy gradient optimizing the performance objective using the policy functional form derived from continuous-time models, but directly treating the features the factors.
For policy gradient methods, we pick a deterministic policy approach like DDPG as, when compared to nonddeteriinistic policy gradient alternatives, they are more suitable to portfolio optimization due to the continuous action space, high exploration cost, and high noise in financial data (Appendix (A.2). For portfolio optimization, different variance reduction methods for policy gradient (Schulman et al. 2015: 2017; Xu et al.) 2020) only provide minor performance improvements (Aboussalah 2020 We hence do not report such results. For areas of RL with representation learning and model-basee RL, we focus on those (SLAC and RichID) explicitly learning a representation of latent variables, since such methods are more closely related to FaI.PO. There are other techniques like data augmentation, feature construction, adversarial training, and regularization that can improve the empirical performance of portfolio optimization (see survey in Hammll et al. 2021 Section 4.3). In this work, we focus on the central methodological task of policy learning, and most such techniques can be directly combined with our proposed method.
Protocol We simulate environments with the Kim fim-Omberg model and implement the considered methods to compare their performance. Note that a key data generating parameter in portfolio optimization is the signal-to-noise ratio, which can be roughly characterized by the ratio between the scale of the drift and the scale of the volatility (see Appendix [K. for detailed explanation). We test our method under different signal-to-noise ratios. To this end, we randomly generate stock drifts to around 10% (to mimic the real-world average return of stocks in SP500), vary the scale of volatility in {10%, 20%, 30%), ), and simulate data following the procedure in Appendix K.2 Then, we apply the considered methods to maximize the terminal power and exponential utility with different 7 's. For each method, we tune the learning rate and other -eetood-specific hyperparameters with early stopping (Appendix K.3 With each training, validation, and testing five times.
combination, we repeat
Results For exponential utility maximization, Table 2 summarizes the average test utility after hyperparameter selection with 10 stocks, 10 features, and 7 competing methods in terms of average terminal utility. This performance gain may be explained by the factor representation learning informed by the continuous time finance model, as other methods are incapable of doing so. Meanwhile, MMMC and CT-MB- RL underperform, which suggests the disadvantage using oversimplified models. Compared with the more sophisticated RI. methods like SLAC and RichII,, the simple DDPG is fairly competitive. This is consistent
FaLPO outperforms all the
8

---

Under review as a conference paper at ICLR 2023
Methods
Energy
Musenal
Induarials
Mix
Pa
-24
-3.2
-6.3
-5.5 +
Average Resurn
SLAC
6.5222
-7.0
5424 586.8
-1.0 x. 10 4.3 1022
RichDD
632.1
8.1
CTMB.BL
-64#44
-11653 22947.5
-37
MMMC
-85 7.
-0.5+117
-11.0+-544
-7.5++44 +-4
Table 3: Average terminal utility for real-worke data. Mix denotes mix of stocks the previous three sec- tors.
Dute
Figure 4: FaLPO average return over portfo- lio terminal dates.
with the existing observation that more complicated RL solutions may not always be suitable for portfolio optimization due to the large noise and idiosyncrasy in the data. In the appendix, we provide additional experimental results with different problem dimensions (Appendix K.4.T). other values (Appendix (.nniiixxXK444))) K.2.2), and different utility functions (Appendix (oppendix(K.4.3).. The results are consistent. Appendix K.4.4 studies simplified casc, where the optimal performance can be mathematically derived. Fa achieves a similar performance as the theoretically optimal one.
5.2 REAL-WWOLLD STOCK TRADING
In this section, we present an application of FaLPO for real-world stock trading problems. Following the synthetic portfolio optimization setup, we study the six considered methods for 21-day (one month) stock trading four different stock sectors using the daily stock price data from Yahoo finance between January 4, 2006 and April 1, 2022. For factors, we follow existing works (Aboussalah) 2020) De Prado 2018 Dixon et al. 2020) and consider economic indexes, technical analysis indexes, and sector-specific features such as oil prices, gold prices, and related ETF prices, leading to around 30 factors for each sector. each sector we select 10 stocks according to the availability and trading volume in the considered time range (Appendix L. 1b. The training, validation, and testing data
SolleBrrpoos reports the achieved average utility of each method under the selected hyperparamters. FaLPO achieves the highest average utility in all four sectors.
constructed using rolling windows (Appendix/ L.3 Table
Next, we conduct the training-tuning-testing procedure above with 7 7,9,, and report the returns of FaLPO in each quarter in Figure Recall that a smaller corresponds to taking more risk. This is consistent with the observation in Figure that the smaller the 7 the bigger the return but the larger the fluctuations. Also, the return fluctuates and drops around late 2018 and early 2020. The former corresponds the abrupt bear market the end of 2018, and the latter consistent with the time period that COVID-19 bursts. Under these two time periods, the financial market was especially noisy and unpredictable. We also implement sensitivity analysis on À in Appendix L.4 and observe that non-zero small À works well in practice.
6 EPILOGUE
Conclusion This work proposes FaLPO, new decision-making framework for portfolio optimiza- tion with stochastic factors. By using continuous-time finance models to regularize policy learning, FaLPO is able to handle high noise and complex effects in financial data. We demonstrate FaLPO's benefits both thcoretically and empirically. We focus on policy learning and defer more advanced feature engineering methods to future work.
Limitations FaLPO has two potential limitations. First, while we show the extension of FaLPO to problems beyond portfolio optimization, FaLPO is not applicable when there is no suitable parametric model to derive the optimal policy functional form. In such cases, the model negell-rgulariized policy learning of FaLPO cannot implemented. Second, the performance of FaLPO still relies on good features (Y₂) to generate factors (X,). ). In the presence of unpredictable marker events (like COVID- 19), or when the features not contain any useful signals (like the Merton case Appendix K.4.4). additional caution needs to be taken when applying FaLPO.
The assumptions and proof details are provided in Appendices I and The experiment implementation details are reported in Appendices and
Reproducibility

---

Under review as a conference paper at ICLR 2023
REFERENCES
Amine Mohamed Aboussalah. What is the value of the cross-sectional approach to deep reinforcement learning? Available at SSRN, 2020.
Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1-76, 2021.
Yacine Ait-Sahalia. Closed-form likelihood expansions for multivariate diffusions. The Annals of Statistics, 36(2):906-933) 937, 2008.
Yacine Ait-Sahalia and Robert L Kimmel. Estimating affine multifactor term structure models using closed-form likelihood expansions. Journal of Financial Economics, 98(1):113-144 2010.
Levon Avanesyan. Optimal investment in incomplete markets with multiple Brownian externalities. PhD thesis, Princeton University, 2021.
Levon Avanesyan, Mykhaylo Shkolnikov, and Ronnie Sircar. Construction of forward performance processes in stochastic factor models and an extension of widder's ' theorem. arXiv preprint arXiv: 04535, 2018.
Stephen Bach, Bryan He, Alexander Ratner, and Christopher Ré. Learning the structure of generative models without labeled data. In International Conference Machine Learning, pp. 273-282. PMLR, 2017.
Krzysztof Bartoszek, Sylvain Glémin, Ingemar Kaj, and Martin Lascoux. Using the Ornstein- Uhlenbeck process model the evolution of interacting populations. Journal of theoretical biology, 429:35-45, 2017.
Maximilian Behr, Peter Benner, and Jan Heiland. Invariant Galerkin Ansatz spaces and Davison- Maki methods for the numerical solution of differential Riccati equations. arXiv preprint arXiv: 13362, 2019.
Yoshua Bengio. Using financial training criterion rather than a prediction criterion. International Journal of Neural Systems, 8(04):333 443,,4434111 1997.
Alexandros Beskos and Gareth O Roberts. Exact simulation of diffusions. The Annals of Applied Probability, (5(4))2422-2444) 2005.
Jalaj Bhandari and Daniel Russo. Global optimality guarantees for policy gradient methods. arXiv preprint arXiv: 1906. 01786, 2019.
Homanga Bharadhwaj. Mohammad Babaeizadeh, Dumitru Erhan, and Sergey Levine. Information
prioritization through empowerment in visual model-based rl. arXiv preprint 2rX0::2204.08588, 2022.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https: http::://wwwwwaddd com/ Software available from wandb.com.
Jaya PN Bishwal. Parameter estimation in stochastic differential equations. Springer, 2007.
Tomas Björk. Arbitrage theory in continuous time. Oxford university press, 2009.
Simone P Blomberg, Suren Rathnayake, and Cheyenne M Morcau. Beyond brownian motion and the onsteein-uhlenbeck process: Stochastic diffusion models for the evolution of quantitative characters. The American Naturalist, 195(2):145-165. 2020.
George Chacko and Luis M Viceira. Dynamic consumption and portfolio choice with stochastic volatility in incomplete markets. The Review of Financial Studies, 18(4):1369-144 2005.
Yevgen Chebotar, Karol Hausman, Marvin Zhang, Gaurav Sukhatme, Stefan Schaal, and Sergey Levine. Combining mrodel-based and model-free free updates for trajectory-centric reinforcement learning. In International conference machine learning, pp. 703-711. PMLR, 2017a.
10

---

Under review as a conference paper at ICLR 2023
Yevgen Chebotar, Mrinal Kalakrishnan, Ali Yahya, Adrian Li, Stefan Schaal, and Sergey Levine. Path integral guided policy search. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 3381-3388. IEEE, 2017b.
Marco Corazza and Francesco Bertoluzzo. Q-leaming- financial trading systems with ap- plications. University Ca' Foscari of Venice, Dept. of Economics Working Paper Series No, 15, 2014.
Marcos Lopez De Prado. Advances in financial machine learning. John Wiley & Sons, 2018.
Mare Deisenroth and Carl E Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML-I1), pp. 465 472. Citeseer, 2011.
Matthew F Dixon, Igor Halperin, and Paul Bilokon. Machine Learning in Finance. Springer, 2020.
Kenji Doya. Reinforcement learning in continuous time and space. Neural computation, 12(1): 219-245, 2000.
Benjamin Eysenbach, Alexander Khazatsky, Sergey Levine, and Ruslan Salakhutdinov. Mismatched no more: Joint model-policy optimization for model-based rl. arXiv preprint 10.02221 2021.
Eugene F Fama and Kenneth R French. The cross-section of expected stock. The Journal of Finance, 47(2):427-466 1992.
Eugene F Fama and Kenneth R French. A five-factor asset pricing model. Journal of financial economics, 116(1):1-22, 2015.
Vicky Fasen. Statistical estimation of multivariate omstein- uhlenbeck processes and applications to co-integration. Journal of Econometrics, 172(2):325-337, 2013.
Damir Filipović and Eberhard Mayerhofer. Affine diffusion processes: theory and applications. In Advanced financial modelling, pp. 125-164. De Gruyter, 2009.
Wendell Fleming and Raymond Rishel. Deterministic and stochastic optimal control. Springer, 1975.
Wendell H Fleming and Sanjoy K Mitter. Optimal control and nonlinear filtering for nondegenerate diffusion processes. Stochastics: An International Journal of Probability and Stochastic Processes, 8(1):63-77, 1982.
Wendell Fleming and Halil Mete Soner. Controlled Markov processes and viscosity solutions, volume 25. Springer Science & Business Media, 2006.
Jean-Pierre Fouque, Ronnie Sircar, and Thaleia Zariphopoulou. Portfolio optimization and stochastic volatility asymptotics. Mathematical Finance, 27(3):704-745, 2017.
Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, Aviv Tamar, al. Bayesian reinforcement learning: A survey. Foundations and Trends@ in Machine Learning, 8(5-6):359 483, 2015.
Amit Goyal and Pedro Santa-Clara. Idiosyneratic risk matters! The journal of finance, 58(3): 975-1007, 2003.
Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration. In International conference machine learning, pp. 2829-2838.
PMLR, 2016.
Mao Guan and Xiao Yang Liu. Explainable deep reinforcement learning for portfolio management: an empirical approach. In Proceedings of the Second ACM International Conference on Al in Finance, pp. 1-9, 2021.
Tuomas Haamojja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning, pp. 1861-1870. PMLR, 2018.

---

Under review as a conference paper at ICLR 2023
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on PMLR, 2019.
Machine Learning, pp.
Ben Hambly, Renyuan Xu, and Huining Yang. Recent advances in reinforcement learning in finance. arXiv preprint arXiv:21 12.2212...45553 2021.
Elad Hazan and Satyen Kale. Beyond the regret minimization barrier: an optimal algorithm for stochastic strongly-conee optimization. In Proceedings of the 24th Annual Conference on Learning Theory, pp. 421-436. JMLR Workshop and Conference Proceedings. 2011.
Florian Herzog, Gabriel Dondi, Hans P Geering, and Lorenz Schumann. Continuous time multivariate strategic asset allocation. In Proceedings of the 11th Annual Meeting of the German Finance Association, Session 2B, pp. 1-34. Citeseer, 2004.
Desmond J Higham, Xuerong Mao, and Andrew M Stuart. Strong convergence of euler-type methods for nonlinear stochastic differential equations. SIAM Journal on Numerical Analysis, 40(3): 1041-1063, 2002.
Vladimír Holy and Petra Tomanová. Estimation of constein-uhlenbed process using ultra-high- frequency data with application to intraday pairs trading strategy. arXiv preprint arXiv: 1811.9931, 2018.
Guosheng Hu, Yuxin Hu, Kai Yang, Zehao Yu, Flood Sung, Zhihong Zhang, Fei Xie, Jianguo Liu, Neil Robertson, Timpathy Hospedales, al. Deep stock representation learning: From candlestick charts to investment decisions. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP). pp.
IEEE, 2018.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in Neural Information Processing Systems, 32, 2019.
Zhengyao Jiang, Dixing Xu, and Jinjun Liang. deep reinforcement learning framework for the financial portfolio management problem. arXiv preprint arXiv: 10059, 2017.
Chi Jin, Praneeth Netrapalli, Rong Ge, Sham M Kakade, and Michael Jordan. On nonconvex optimization for machine learning: Gradients, stochasticity, and saddle points. Journal of the ACM (JACM), 68(2):1-29, 2021.
Hilbert Kappen. Path integrals and symmetry breaking for optimal control theory. Journal of statistical mechanics: theory and experiment, 2005 (10((1))P11111, 2005.
Hilbert Johan Kappen and Hans Christian Ruiz. Adaptive importance sampling for control and inference. Journal of Statistical Physics, 162(5):1244-13 1266, 2016.
Ioannis Karatzas and Steven Shreve. Brownian motion and stochastic calculus, volume 1 113. springer, 1987.
Belhal Karimi, Blazej Miasojedow, Eric Moulines, and Hoi-To Wai. Non-asymptotic analysis of biased stochastic approximation scheme. In Conference on Learning Theory, pp. 1944-1974. PMLR, 2019.
Taesup Kim, Sungjin Ahn, and Yoshua Bengio. Variational temporal abstraction. Advances in Neural Information Processing Systems, 32, 2019.
Tong Suk Kim and Edward Omberg. Dynamic nonmyopic portfolio behavior. The Review of Financial Studies, 111:111,,,,, 1996.
Holger Kraft. Optimal portfolios and heston's 's stochastic volatility model: an explicit solution for power utility. Quantitative Finance, 5(3):303-3133 2005.
Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. Urlb: Unsupervised reinforcement learning benchmark. arXiv preprint arXiv:2/ 10. 15.11 2021.
12

---

Under review as a conference paper at ICLR 2023
Alex X Lee, Anusha Nagabandi, Picter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. Advances in Neural Information Processing Systems, 33:741-752. 2020.
Xiao- Yang Liu, Hongyang Yang, Jiechao Gao, and Christina Dan Wang. Finrl: Deep reinforcement learning framework automate trading in quantitative finance. In Proceedings of the Second ACM International Conference on in Finance, pp. 1-9, 2021.
Po-Ling Loh. Statistical consistency and asymptotic normality for high-dimensional robust m- estimators. The Annals of Statistics, 45(2):866- 2017.
Tengyu Ma. Why do local methods solve nonconvex problems? Beyond the Worst-Case Analysis of Algorithms, pp. 465, 2020.
Sidra Mehtab and Jaydip Sen. A robust predictive model for stock price prediction using deep learning and natural language processing. Available SSRN 3502624, 2019.
Robert C Merton. Lifetime portfolio selection under uncertainty: The continuous-time case. The review of Economics and Statistics, pp. 247-257, 1969.
Robert C Merton, Continuous-tiin time finance. Blackwell Cambridge, MA, 1992.
Robert C Merton et al. An intertemporal capital asset pricing model. Econometrica, 41(5):867-887, 1973.
Zakaria Mhammedi, Dylan Foster, Max Simchowitz, Dipendra Misra, Wen Sun, Akshay Krish- namurthy, Alexander Rakhlin, and John Langford. Learning the linear quadratic regulator from nonlinear observations. Advances in Neural Information Processing Systems, 33:14532- 2020.
Johannes Muhle-Karbe, Max Reppen, and Mete Soner. primer on portfolio choice with small transaction costs. Annual Review Financial Economics, 9:301-331, 2017. 2017
Rémi Munos. Policy gradient in continuous time. Journal of Machine Learning Research, 7:771-791, 2006.
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynam- ies for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 7559-7566. IEEE, 2018.
Abhishek Nan, Anandh Perumal, and Osmar R Zaiane. Sentiment and knowledge based algorithmic trading with deep reinforcement learning. International Conference on Database and Expert Systems Applications, pp. 167-180. Springer, 2022.
Arkadi Nemirovski, Anatoli Juditsky, Guanghui Lan, and Alexander Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on optimization, 19(4) 1609, 2009.
Bernt Oksendal. Stochastic differential equations: an introduction with applications. Springer Science & Business Media, 2013.
Matteo Papini, Damiano Binaghi, Giuseppe Canonaco, Matteo Pirotta, and Marcello Restelli. Stochas- tic variance-reduced policy gradient. In International Conference Machine Learning, pp. 4026-4035. PMLR, 2018
PCB Phillips. The structural estimation of stochastic differential equation system. Econometrica: Journal of the Econometric Society, pp. 1021-1041, 1972.
Peter CB Phillips and Jun Yu. Maximum likelihood and gaussian estimation of continuous time models in finance. Handbook of financial time series, pp. 497-530. Springer, 2009.
Boris T Polyak. Gradient methods for the minimisation functionals. USSR Computational
Mathematics and Mathematical Physics, 3(4):864 8788 1963.
13

---

Under review as a conference paper at ICLR 2023
Konrad Rawlik, Mare Toussaint, and Sethu Vijayakumar. On stochastic optimal control and rein- forcement learning by approximate inference. Proceedings of Robotics: Science and Systems VIII, 2012.
Max Reppen and Mete Soner. Bias-variance trade-off and overlearning in dynamic decision problems. arXiv preprint 1.944990 2020.
Rori V Rohlfs, Patrick Harrigan, and Rasmus Nielsen. Modeling gene expression evolution with an extended Omnstein-Uhlenbeck process accounting for within-species variation. Molecular biology and evolution, 31(1):001-211 2014.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on muchine learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Shai Shalev-Shwartz, Yoram Singer, Nathan Srebro, and Andrew Cotter. Pegasos: Primal estimated sub- sub-gradient solver for svm Mathematical programming. 127(1))3-30, 2011.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pp. 387-395. PMLR, 2014.
Freek Stulp and Olivier Sigaud. Path integral policy improvement with covariance matrix adaptation. In Proceedings of the 29th International Coference on International Conference on Machine Learning, pp. 1547-1554, 2012.
Jingrui Sun and Jiongmin Yong. Stochastic Linear- Shnear-Quadratir Optimal Control Theory: Open-Loop and Closed- Loop Solutions. Springer Nature, 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Peter Tankov. Financial modelling with jump processes. CRC press, 2003.
Evangelos Theodorou, Jonas Buchli, and Stefan Schaal. A generalized path integral control approach to reinforcement learning. The Journal of Machine Learning Research, 11:3137-3181, 2010.
Brandon Trabucco, Xinyang Geng, Aviral Kumar, and Sergey Levine. Design-bench: Benchmarks for data-driven offline model-based optimization. arXiv preprint arXiv:2202.08450, 2022.
Vladimir Vapnik. Principles of risk minimization for learning theory. In Advances in neural information processing systems, pp. 831-838, 1992.
Oldrich Vasicek. An equilibrium characterization of the term structure. Journal of financial economics, 5(2):177-188, 1977.
von Neumann and Morgenstern. Theory of games and economic behavior. Princeton University Press, 1947.
Jessica A Wachter. Portfolio and consumption decisions under mean-reverting returns: An exact solution for complete markets. Journal of financial and quantitative analysis, 37(1):63-91, 2002.
Haoran Wang, Thaleia Zariphopoulou, and Xunyu Zhou. Exploration versus exploitation in reinforce- ment learning: a stochastic control approach. arXiv preprint arXiv:1812.07553 2018.
Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural policy gradient methods: Global optimality and rates of convergence. arXiv preprint arXiv: arXiv:1909.00 /909.01 150, 2019.
Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. Advances in neural information processing systems, 28, 2015.
14

---

Under review as a conference paper at ICLR 2023
Ivo Welch and Amit Goyal. A comprehensive look at the empirical performance of equity premium prediction. The Review of Financial Studies, 21((4:11455-335 1508, 2008
Zhuoran Xiong, Xiao-Yana Yang Liu, Shan Zhong, Hongyang Yang, and Anwar Walid, Practical deep reinforcement learning approach for stock trading. arXiv preprint arXiv:1811.07522 2018.
Pan Xu, Felicia Gao, and Quanquan Gu. An improved convergence analysis of stochastic variance- reduced policy gradient. In Uncertainty in Artificial Intelligence, pp. 541-551. PMLR, 2020.
Jiongmin Yong and Xun Yu Zhou. Stochastic controls: Hamiltonian systems and HJB equations,
volume 43. Springer Science & Business Media, 1999.
Pengqian Yu, Joon Sern Lee, Ilya Kulyatin, Zekun Shi, and Sakyasingha Dasgupta. Model-based deep reinforcement learning for dynamic portfolio optimization. arXiv preprint (eXix:990100770 2019.
Thaleia Zariphopoulou. A solution approach to valuation with unhedgeable risks. Finance and stochastics, 5(1):61-82, 2001.
15

---

Under review as a conference paper at ICLR 2023
APPENDIX
A RELATED LITERATURE
In this section, we discuss related literature.
A.1 CONTINUOUS-TIME FINANCE MODELS
Existing strategies for solving portfolio optimization using continuous-time time finance models can be loosely summarized as performing three steps:
1. Choosing the model for the dynamics, i.e. the type of stochastic differential equation (SDE).
2. Estimating the parameter of the selected model (which is also referred to as model fitting, model identification, or calibration).
3. Solving for the optimal policy under the estimated model.
The third step leverages stochastic optimal control tools (Fleming & Rishel 1975 Fleming & Mitter 1982: Fleming & Soner 2006 Yong & Zhou 1999).
Finding and estimating an appropriate model for stochastic optimal control requires significant domain knowledge. For instance, in finance, the modeler must specify both which features are relevant and how they affect stock prices (Fama & French| 1992). If not every relevant factor is correctly specified, optimal control can hardly lead to good performances. As a result, in stock trading, control methods would hand pick three to five economic indices as the factors and assume they follow a simple (often linear) SDE. But indeed trading can benefit from much richer datasets including related option prices, technical indicators, and interest rates (Aboussalah) 2020 2018 Dixon et al. 2020 Mehtab & Sen 2019).
De Prado
Further, even with correctly specified model and factors, likelihood-based estimation for SDE control models can be very challenging (Phillips Yu 2009). As a result, methods like Sat--ahaliia (2008); Ait-Sabalia & Kimmel (2010) seek replace the exact likelihood with other likelihood- objective functions, while maintaiiing thcoretical guarantees. However, the proposed objective function needs to be derived for each specific problem, and the derivation
be challenging. Other methods like Fasen| (2013); HHoly & Tomanová (2088) rely on more specific parametric or low- dimensional setups. To alleviate these issues, our framework extends the existing continuous-time finance models by allowing for a flexible and generalized definition of stochastic factor dynamics. Further, we simultaneously conduct policy learning and model calibration in an RL manner, with a square-loss objective that avoids the calculation of an exact likelihood.
A.2 REINFORCEMENT LEARNING
RL aims to conduct the aforementioned three steps by (i) relying more on data (ii) an end-to-end fashion. Methods like model-free RL assume no parametric forms on the dynamics, and directly learn the optimal policy while explicitly learning the model (step 1) and estimating the parameters (step 2).
Discrete-Time Model-Free RL There exist many discrete-time model-free RL methods (Sutton & Barto 2018). In this category, deep deterministic policy gradient (DDPG) is the most relevant one, and is empirically most widely used for portfolio optimization (Hambly et al. [022]. The reason is twofold. First of all, DDPG is a policy-gradies based method, and thus can naturally handle continuous states and actions in portfolio optimization with simple procedures. Second, DDPG learns a deterministic policy instead of a stochastic one like Haarnjja et al.) (2018). This characteristic is especially important in portfolio optimization where the policy learning goal is a deterministic policy since the cost of a stochastic policy is extremely expensive.
Continuous- Time Model-Free RL Continuous-timm model-free RL (Wang et al. 2018 Doya 2000 Munos 2006) aims to solve for continuous-time policy. However, such methods do not or assume any SDE structure, and thus struggle with the common open questions in model-free RL like poor stability and sample complexity. As one example, path integral methods stem from the
16

---

Under review as a conference paper at ICLR 2023
theoretical result that the value function of type of continuous-time decision-making problems can be expressed closed form as a Feyman- Kac path integral (Fleming Rishel 1975 Kappen 2005). A series of control/RL methods follow the rationale of optimizing the policy to maximize such an integral. Specifically, (Thoooroo et al. (010)) propose an open-loop control strategy: Kappen & Ruiz 2016) builds RL with importance sampling; [hebotar al. (2017a b): (Sulp & Sigaud (2012) combine path integral with other model-based or model-free RL methods. However, the core derivation only holds for decision-makinn equivalent to assuming that the action does not affect the randomness in decision decision-making. Such an assumption limiting, and does not hold for portfolio optimization, where how allocate the wealth in order to minimize the risk is key to successful policy.
satisfying Kappen Ruiz (2016) Equation (I)), which is
Model-based RL and RL with Representation Learning Model-based RL and RL with represen- tation learning two active research areas but without clear general state-of- of-tth--rttttttt (Bharadhwaj Eysenbach et al.) 2021] Trabucco et al.) 2022] Janner et al. 2019 Deisenroth & Ras
et al.
2022
mussen [01
Nagabandi et al., [0018] Laskin al., [021 Lee al., 2020 Watter et al. 2015
Chebotar et al.
7a Hafner et al. 2019; Kim et al. 2019). The closest to FaLPO are those that
learn explicit representation of latent variable like Lee al. (2020); Mhhmmedii al. (2020).
But such methods are unable to leverage continuous- time finance models for portfolio optimization.
Our proposed framework also related to Bayesian models (Ghavamzadeh et al. 2015 Rawlik et al.) 2012), if we treat the learned representation of factors as the hidden variable. Strictly formulating an NSFM as a Bayesian model requires assumptions specifying the conditional distributions, and thus requires more domain knowledge. The optimization of Bayesian methods is also more challenging.
Bayesian RL
RL for Stock Trading Various efforts have been made applying RL to stock trading (Corazza & Bertoluzzo 2014 Hambly et al. 2021 Nan et al. [2222] Xiong et al. 2018 Guan & Liu 202 Liu
al. 2021 Hu et al. [2188] Yu et al. (2019). However, these methods focus more on feature selection techniques. Methodologically, they do not take advantage of
or empirical aerformance-impronn
continuous-tims finance models.
A.3 EMPIRICAL RISK MINIMIZATION
Another related area is Empirical Risk Minimization (ERM) (Vapnik 1992). ERM studies the minimization of an objective function using the averages over training data to construct an empirical loss function. Recent work connected ERM with simulation-based and data-based offline decision- making methods (Reppen & Soner 2020). More specifically, when the random input is observable and unaffected by actions, and a training set is available, the decision-making problem can be formulated as an ERM problem. As a result, the portfolio optimization may be reformulated as an ERM extension.
B OTHER OBJECTIVE FUNCTIONS
Note that the goal of portfolio optimization is to maximize the return while minimize or constrain the risk. In practice, one can use different objective functions for such goal, like mean-variance objective (Hambly al., 202T), Sharpe ratio, and so on. In this work, we consider utility maximization with power utility and exponential utility. The proposed method also works with other objective functions, as long as we can derive (part of) the optimal policy structure. Note that the selection among these objective functions is more user-preeerence question.
17

---

Under review as a conference paper at ICLR 2023
C GRADIENT ESTIMATES
In this section, we discuss the gradient estimation for both V and Assume that we collect B independent trajectories for St and Y,, denoted as
DD ((50, k0, (8 k: Vsc,k); 92ALK), (MMA))
Then, the gradient estimate for ((g 0x) is defined as
B
1
(V 0z) = B
(V,(00,0 0z) with Oz), = Vog.0 U
(
=(-0-),Ar
The terminal wealth in the trajectory under the policy where (:; 0z) is denoted as TTK
with st mas Smat
M
8 (m+ 1)2t
r' (m,tt, Vmaii 0g 0z)
TT
20
Y12
Next, we consider the gradient of L:
1 VL 0s)
(z 0s).
B
Specifically for likelihood and negative mean square loss, we have
M-1
LL (0g,
Σ
Ve..s
(1 0z)
M
m==
SmAt.: ((umscii0a):03)
M-1
(LNMSL.((00.s)) =
Vos.0s (og(s((+4))se.) log(smmeek)
M
m=0 1=1
(m+1)At
E
fx(xx;3))
mdl
1 de
Ig (X,; 0s) Smark AlUmAr,: (2)) 0a))
2
As result, in each iteration, we collect trajectories to estimate the gradient of H (0.. Oz).
D A PRIMER ON STOCHASTIC DIFFERENTIAL EQUATIONS (SDES)
We provide general formulation of SDEs with two examples.
D.1 FORMULATION OF SDES
SDEs generalization of ordinary differential equations to dynamic systems influenced by random fluctuations. The structure of the randomness can in principle be quite general, such as with jump processes where the state evolution is longer continuous (Tanovv] 2003). Although our method can be generalized to all SDEs, we restrict ourselves to practical settings where the source of randomness a Brownian motion.
Let W, be a multi-dimensiona.. independent standard Brownian motion. For random process S,, an SDE is typically expressed using differential form as
dS, f(S) dt ((S:) dWe or S. So ( f(S)) ( g(S.) aw,,
18

---

Under review as a conference paper at ICLR 2023
where f()) and g(-) are functions of St. The stochastic integral Sn g dWW is the accumulation of influence to the state due to the noise. We refer the reader to Karatzas & Shreve (1987) for
details on the construction of stochastic integrals and SDE theory. Important here is that Equation (8)
defines the transition of St in an infinitesimal time step. The drift coefficient f(St) characterizes the deterministic part of the change of St, and the diffusion coefficient g(St) models the randomness in the transition of Sg.
D.2 EXAMPLES
As concrete examples, we discuss two families of SDEs widely used in finance, economics, and biology: Geometric Brownian motion (GBM) and
Uhlenbeck (OU) processes (Merton al. 1973: Vasicek 1977, Barrtoszkk al. 2017 Blomberg al., 2020, Rohlfs et al.. 2014). The OU structure appears in both applications below, and the financial application uses GBM as base, but extends it with OU drift coefficients. The two types of SDEs are given by
ds. GBM:
dtt + dW,,
OU: dS, - pS, dt dW,,
5.
where S,
denotes the component- wise division St, and the matrices js and define the drift and diffusion coefficients.
as.. Sit
We refer the interested reader to Fleming & Soner (2006) for more information on these topics. We now briefly formulate two classic stochastic optimal control models for decision-making with stochastic factors. The stochastic factors appear as the drift coefficients of other state variables and themselves modeled SDEs.
D.3 ITÔ'S FORMULA
Itô's Formula a fundamental analytical tool for SDEs, and crucial for their analysis. We only provide simple version here, which is sufficient for our analysis. A more general and rigorous statement with assumptions and proof of Itō's formula and integral can be found in Karatzas & Shreve (1987 Theorem 3.3).
LEMMA D.I (Itô's Formula) Consider a twice differentiable function G, and S, following
dS: = f(S)) dt + g(Se) dW,-
Then, we have
- {aG Ot
(DG) f(S.) + Tr g(S)) PG as? 9(5,) }at- (aG)
dG
ass ((S))dW.
LEMMA D.2 For suitable bounded process St, the Itô integral So S,dW, satisfies:
27 = V S7 [ S.dW, = O, E [V. s.aw:)
E
The latter is also referred to as Itô's isometry
E MODEL CALIBRATION
We discuss two model calibration loss functions, log- likelihoodd and negative square loss.
E.
LOG-LIKELIHOOD
We can use log-likelihood as for model calibration. The log-likelihood of SDEs is derived in sequential manner. Specifically, for 2 the likelihkehooo is derived
Log (.,, 0s) Ellog(Pes (St+ o(Y - 06) S, (YY; (s)))
where Pe. denotes the conditional likelihood according to 2 but with parameter 0g instead of 0
Then, in specific models, one can derive Pa. (St+ae o(Y++ Ar] S, o(Y)); (Phillips) 1972 Holy &
Tomanová 2018 Beskos & Roberts 2005) the approximation of it (Ait-Sahalia & Kimmel 2010)
19

---

Under review as a conference paper at ICLR 2023
E.2 NEGATIVE MEAN SQUARE LOSS
For the SDE system (2). one can also use negative mean square loss (NMSL) as the calibration objective. To derive this loss function, first drive the dynamies of log price by applying Itö's Formula Lemma D.I to (
dw 2 Σ
d (log(S)) f5((x05)dl
(X (X.: +
Σ (Xx;0s)))
(10)
Then, combined with Lemma D.2 under proper assumptions Lemma D.2 over both sides of the above equation, and derive
we pose expectation
dw
E(dlog(S;) ff((,, 0g)
(X.:
In words, the expectation of the log price change in an infinitesimal time is (X 0g)dt <>=m(0 (n5 (X,; R Therefore, one can estimate the parameter 0g by minimizing the mean square loss between the log price change and the expected log price change:
++
(NMSL(0s):= -E
log( log(S log(S;) IE
(Xsi
(11)
1 dw
Lo' (X): Sc, Xz 1)
2
It can be easily proved that the true data generating SDE parameter satisfies
0' E arg max LNMSL (0s)-
Further,
we take X, = 0s) and parameterize the objective as INMSL(O0 0s) =
E
log(S)+ (S:) (Yr: - 0/. we can prove
0% 0g E arg max LNMSL
Note that in practice it can be very hard to calculate the expectation E [[ fz(X,;0s) "E (0 (X,; (s)) X.]. Therefore, when At small, we replace the conditional ex- pectation via [3 0s) 1 (9' (Xs: 0s) S,, Xr}. Accordingly the calibration objective is defined as
dw
LNMSI(0s) ~ -E
-
Σ
(K::
2
(12)
E.3
OTHER MODEL CALIBRATION OBJECTIVE
Another potential model calibration objective following the same rationale as 11 11p is
dx
LNMSI- 0s) = EE
Σ
& ((Yersrs0s)
H+D (' (s): Os o(y:: ).
which derived using the conditional expectation of X,+a given X, However, the true data- generating parameter (8%, (3)) is not maximal point of LNMSL-X- x- To clearly this, this loss function encourages the representation function ó to take constant output that X, is constant over time with f (X,; fs) = O, and LNMSt.-x(0A) = We also try this loss experiments, and it leads to poor validation and test performances.
20

---

Under review as a conference paper at ICLR 2023
F APPLICATIONS OF FALPO TO DIFFERENT STOCHASTIC FACTOR MODELS IN CONTINUOUS TIME FINANCE
FaLPO can be used with many stochastic factor models in continuous-time time finance models. In this section, we discuss the Merton model (Appendix F.I.. Kim-Omberg (Appendix) F.2) and EVE (Appendix
F.1
MERTON MODEL
Merton model (eerton 1969) is a classic setup for portfolio optimization. It studies the allocation of capital across set of financial assets in order to maximize profits and minimize risks.
F.1.1 MODELING
Consider p risky assets with prices S, {S:) and additional risk-free money market account with, for simplicity, zero interest rate of return (like cash). The Merton model does not include factors. The dynamies for asset prices formulated
dsp S
jdt + odW.
(13)
The parameters μ4, p p matrices, with denoting the volatility of assets. Further, use 2, to denote the wealth at time point t under the continuous-time policy i.. Under the famous and widely used self-financing assumption (Björk 2009), we have
dZp 2/
Reputt + #.ddW.
An investor's goal maximize the expected utility of capital (ZT) some future time point T:
max Ea. U ((Z 2, So
(14)
Negative values in the policy output allowed, meaning the agent short any asset.
F.1.2 POLICY FUNCTIONAL FORM
For the Merton model, II in 33 can be explicitly derived.
LEMMA (Policy Functional Form for Merton Model) For a Merton model defined in (13), under common assumptions, the optimal policy for portfolio optimization (14). with power utility follows
p(aa )-
The optimal policy for portfolio optimization
14p with exponential utility follows
p( )
2
Proof. Lemma F.I is classic result in continuous-time finance, proposed in Merton 1969
In words, in Merton model, the optimal policy is independent of time, stock prices, features, and factors. The optimal investment strategy is to keep a constant fraction or amount of wealth in each asset all along, depending on the choice of utility function. Let 6x be a ds parameter vector. According to Lemma in FaLPO, we parameterize the candidate policy function as
(t, Ze, Yr: 0s 0x) II(t, S, Ze, (Y(;; Ox
for power utility, and
0
(t, Z, Y1; 07) II(,, 24
Z.
for exponential utility.
21

---

Under review as a conference paper at ICLR 2023
F.1.3 MODEL CALIBRATION
According the Merton model formulation, there exist no factors affecting the evolution of stock prices. Therefore, we do not add the model calibration objective in FaLPO for Merton problem.
F.2 KIM-OMBERG
Kim-Omberg model (Kim & Omberg 1996) is a standard model for portfolio optimization with predictable asset returns, which has been discussed extensively in the empirical literature (Welch & Goyal 2008 Muhle- Muhle-Karbe al. 2017).
F.2.1
MODELING
In Kim Omnerg model, the stock dynamics formulated
dsp
dw
x di +
J dw?
S
j=1
d.X, = سlم X.) dt + udW,
The portfolio optimization goal is formulated as
max Vite with (+t,) == Es, (U x, Zo S. s].
(15)
Rr
F,2.2 POLICY FUNCTIONAL FORM
An optimal policy function is derived in Lemma
LEMMA F.2 Under common assumptions in Kim & Omberg (199)]: Herzog at. 2004), optimal policy functional form for [5) with power utility is derived as
(oo - + ov
where (t) and (t) satisfy
dki(t) (t)
Tr fuT (t)) + ka(())b + (mw)) k2((t)
2(7 - 1) (k2(t) T vo k()))) = 0,
di
2
dk₂ (t) dt
+ (tt TonyouTk2l0 kt(() p kk2(t) kt) 7
7-1 ( + ks()) ke())) o,
dkg
(t)
ks(t)vv kt(() - ks(() nT kt()t)
dt
7 7-1 (100 1)- + (0o T Tks(t))) les (t)(a (go
(aaT) T) 11 + ks(()vv kt((t) (())t) 0,
with (T) 0, (T) and lss (T) 0. Note that ky (t) is sealar, (a(t) is ds 1 vector, and kt(t) is a dx x dx. And the ODE of kt(() is the famous matrix Ricatti equation.
Similarly, optimal policy functional form for 15) with exponential utility is derived as
मे
(aa T)-1
1
x(1) ov To (ka(1) ((((()))+++(()))
27
where (t( and kt(t) satisfy
dk,(t) (t)
dt + Tr {u (k2)())22()) + ka(t)) + (mw) (k2()) 1 (b2(t)) ou k2(()))) = 0,
dk2(t)
+ ka(t)ou k2(()) (h2(t) + kw) - ((ooT) (vkk(() (oouk2())) ka(t)ve k2(()) = 0, (ka(1) (t) (t(t) kz(t) - 4 kt(()
dt
dt
- -(0o 1)-1 (0o )
''o ov k(( + lis()) (aa (aa =a(())
with (T) 0, (T) 0. and kT) 0.
22

---

Under review as a conference paper at ICLR 2023
F.2.3 MODEL CALIBRATION
Following the derivation in Appendix E.2 the negative mean square loss for Kim-Omberg model can be derived as L(0g, 0s) := E this case 0g a dg x 1 vector.
log(S)+40) log(S;) of (Y::0s) 003]² where in
F.3 EVE MODEL WITH STOCHASTIC MARKOVIAN FACTORS
We take EVE model (Avanesyan et al. 2018) with stochastic Markovian factors another example.
F.3.1 MODELING
We first detail the modeling of EVE, which formulates the dynamics of asset prices by
dsp
dw
St (u(Xi;0s))
Σ 3=1
(xx((x i= 1,2, ... ,ds, ..,....ds.
(16)
d.,, = (MT X, w) + K ((x;0s) dBr,
B₁ = p W, TW, +
A W,!
We use W, and W,! to denote two sets of independent Brownian motions, denotes correlation matrix with components pt E [-1, 1]. Therefore, B indeed another Brownian motion. We let μ. a, and be parametric functions, with 0s denoting all the parameters. The SDE parameters include all the parameter matrices and ('s. Again, with Ze as the wealth, we aim to maximize the power utility at the terminal time T as the performance objective:
1-
((+)) E
1-7
EVE model poses further assumptions on
16
ASSUMPTION F.3 M has non- non-negative off-diagonal entries and w E [0, oc) Further, we assume that there exist A(x), A and I., and N such that (()), x<()) and p satisfy
A(x) X(x) = u(x) o(x)) oz)x (o(x) u(x)= A
<(x:) m(())= = diag(TTT)) (1 L2 LERR), with L1, T2, La 2 0 T<(x)] p ('(xx)= NTx.
(17)
The conditions in Assumption R3 necessary for the process of X, to be [0, < and affine.
Under these conditions. the SDE in 16 has unique weak solution which affine and takes values in [0, ox (Filipović & Mayerhofer 2009). Further, the EVE model requires the following two assumptions:
ASSUMPTION F.4 (Assumption 2.2 in Avanesyan et al. (2018) The functions μ Rdx Rds, Rd + R dw xds are continuous. More over, the columns of belong to the range of left- multiplication by (x)) for all x Rdx
ASSUMPTION E.5 (EVE Condition in Avanesyan et al. 2018)) For some E [0, 1].
pT p=p=p) =pp
where / the identity matrix. Note that when p p is vector and thus define p = pT p.
F.3.2 CONCRETE EXAMPLE
We consider concrete example of EVE satisfying the formulation and assumptions in Ap- pendix F.3.1 Specifically, we use D,,. D.. and D, to denote diagonal matrices. Further, let D(x) denote the diagonal matrix whose diagonal is æ. Also, we use sk for any lo E IR to denote the component-wise power operation (Hadamard power).
23

---

Under review as a conference paper at ICLR 2023
Then, define
1(x:) = D pu
a(x) == DT (xx)
k(x) p- D'DDD(I
Then, we have
M(x) D; D +
Further, we pose
p p pp 1.
Then.
((x) = (o(x)=1) u(x) = x and A(z) \(x) = D=²Ø,
n(x)]<(x)) D2 (xx 11 Dr D(x D2
Tx(x)" p A(x)))==-)0)))))><))) ID(x Dx p D
-ID( DT'.
Such setup is shown satisfies (6
F.3.3 POLICY FUNCTIONAL FORM
The policy functional form of the EVE model with Markovian stochastic factors ca be derived as:
LEMMA F6 Under the assumptions in Appendix F.3.1 and Appendix F.3.2 the optimal policy function follows:
n. = -xxx)) (A(X,) + 9px((x))B((tT))
with
(t:)(t) dl
(.0()20)))+++ + Σ (M N))) _ 0, i= 1,2, 1,,,
dx
24
j=1
dk- w kk)) =0
dt
For the specific example in F.3.2 in FaLPO, we parameterize the candidate policy as
T1, Y,, Ze; 0g 02) = II(,, Y,, Z. Og. 0z) (YY)) k(t;0x).
F.4 MODEL CALIBRATION
Following the derivations in Section E.2 we can derive the calibration objective as:
L(0s) == min
E
A (og) Cio(Y;; 0x) A O2;
0s (C-(Cr.Ca)
G SOLUTIONS OF RICCATI DIFFERENTIAL EQUATIONS
According to the analysis in Section F. the optimal policy function is closely related to the solutions of Riccati differential equations, which also have closed-fon solutions.
Specifically, with abuse of notation, let A, B and D be p x matrices, and X [0, + R as a function solving the following Riccati differential equation:
dd A X(t) dt
X(t) D D.
(18)
X(0)
Xo-
24

---

Under review as a conference paper at ICLR 2023
Following the analysis and assumptions in (Behr et al. 2019), the unique symmetric positive stabiliz- ing solution of X (t) follows:
X(t) Xxo CAA" (Xxo - Xx) [I (X2 x T 1Xxo -Xo) cdd _cd
where
AX2 + XLA + BB = 0,
with A A BBTBBBX
Xoo, and
0= xT X D.D.
Note that with (88 we can further derive the policy functional forms without using neural networks parameterize tmmeddependent fucntions.
H EXTENSION TO LINEAR QUADRATIC CONTROL (LQC)
The methodology of FaL.PO also be applied decision-making problems other than portfolio optimization. To implement Fal PO, one needs to first construct a neural stochastic factor model combining factor representation with continuous-time model. Then, the policy learning conducted while leveraging policy functional form and model calibration. As an example, we implement FaLPO linear quadratic control (LQC), and detail modeling (Section H.T. policy functional form (Section H.2) and model calibration (Section H.3)
H.1 MODELLNG
We consider the problem of LQC (Sun & Yong 2020) but with stochastic factor X, following an OU process. With slight abuse of notation, S, denote the sate variable in this section:
dw
dS; = (BS, + + X.) dt + D, dw?,
(19)
X, = ,X, dl dW,,
where B, U. μe, and Dj are redefined as matrices with appropriate dimensions. With A, = T(-) following the policy æ, we aim to solve
max ((x) with ((T) == Ey J." [(QS.) (RA.)
+(RA).)) Ar] di + (GSr) ST]
(20)
with Q, R. and G as known matrices with appropriate dimensions, and T is terminal time. Further, we apply the modeling strategy in Section 3.1 1. and aim to learn the representation of stochastic factors from the available features Y;:
X, = (Yr;
H.2 POLICY FUNCTIONAL FORM
By taking E, (the combination of S₁ and X,) the state variables, can reformulate the problem as a classic LQC problem:
dw
d= BB
Σ (DJ JWW? j=1
with all the coefficients redefined. Then, under common assumptions in Sun & Yong 2020 Yong Zhou 1999), it can be derived that the optimal policy satisfies:
#" #* =
(U=) " k=))))
where
dw
AF = R+ Σ (D) (F()D))
25

---

Under review as a conference paper at ICLR 2023
Also, K=t(t solves the differential Riccati equation:
k k=(()) (t)
(T-t) (( G2
( (F (
with
(T) =0. 0.
Therefore, formulate the candidate policy
(t, St, Yú; 0g an) 11(t, St, Y;: 02 0m) k1 (t;;0))s Aa(t
(21)
H.3 MODEL. CALIBRATION
According to 19 and following the derivation strategy in Appendix E.2 we can derive the negative mean square loss for LQC
L(0,, 0s) == -
- -1 C2A.1]))
(22)
with 0s [1,,
As summary. apply FaLPO to LQC with stochastic factors, we parameterize candidate policies following 21 and maximize
(1 - (V + AL( 0s).
with V in 20 and L in
22
EXTENDED RESULTS FOR THEOREM
1.1 ASSUMPTIONS AND DEFINITIONS
To start with, we consider stochastic factor models such that the optimal feedback admissible policy admits functional form
#ं [T,,, Xx; 02).
Also, consider the L function such that the true data generating parameters 0g. 8 satisfy
0% 0 E arg max L(Os, 0s),
(23)
0.
while other options for L can also be empirically used in method.
DEFINITION L.1 For a continuous tietiuuuus-imeeeeeee policy मे := Illt, S,, Z, ((Y:: 0x), we define its value function
(0s, 0z) = {Z))-
Accordingly, we define the continuous-time version objective function as
(9,,, 0z. 0s)) == - (es, 0x) + AL(Hs, DEFINITION 1.2 For [mAi, (m 1)) define [4] mA and [
DEFINITION I.3 For the continuous-time optimal policy म", we use [ to denote the piece-wise constant version #*. We use _[*1] I to denote the wealth process when implementing the optimal continuous-time policy (:; Φ. 07) in the piece-wise constant manner. Specifically,
ds yds
d2.
=2 |
[t]
[E
dp
St 2
([E) X i 0 (X,;
dt
St
dw
S 7 |
Σ
93 (X,;
03)
dw?
5 [
j=1
26

---

Under review as a conference paper at ICLR 2023
Further, remember Z is used to denote the continuous-time wealth process under the policy #* By the dynamics of continuous-time wealth process Z in Section 2.3 the dynamics of 7 is derived as
d2t (T, X1; (;) [s(Xx;3))) [T, x 0 gs (x(;;;33) dWW,
Z
ASSUMPTION 1.4 For each R > 0, if ||]||| & and f $ T, we assume that there exists a Cn >>
such that
[[[(t, x; 02) [I 03)1l v 1las(2; 03)1 [1xx (2) (xx((;0))))) < Cr.
and [t, xx; (7) is locally Lipschitz with Lipschitz constant CR- For some p > 2, there exists constant A such that
B[ sup OSiST
2 P
VEL sup OSEST
12, 1 V OS1ST
sup
|
V sup [[og((.))IP] A. OSIST
Note that Assumption 1.4 requires the stochastic processes have bounded high-order moments. For a specific model like Kim-Omberg, this is not guaranteed to hold for every initial value and SDE coefficient, but one can derive model-specific sufficient conditions for Assumption [.4 In practice when implementing the method, we calculate the empirical moments of wealth, factors and asset prices to approximately check whether Assumption I. I.4.holss
With Assumption
.4 we define stopping time:
DEFINITION 1,5 For 0, define stopping time
TR
inf OSIST
or |Xx|ll 2 R or |log((:))] 2 R or 12, #* 11 R
ASSUMPTION 1.6 The utility function (z) has a linear bound on 2:
[U(z)] S Cr([z++11 1I)
Note that Assumption [.6 holds as long as the utility function is bounded at the smallest value in 2. Specifically for power utility, Assumption I.6 is equivalent to setting 7 E (0, 1).
ASSUMPTION 1.7 There exists At' 0 such that for any At < At' [ is also admissible policy.
1.2 LEMMAS
LEMMA 1.8 Consider non-negative constants C12 C2 "* Cr. The following inequality is true:
n
Proof. The proof follows the Cauchy-Schwartz inequality.
LEMMA 1.9 (0), 8% 0%) maximizes both V(0 (z) and L(Os,, Os)s,0s).
Proof. First of all, since 0N- is defined to be the optimal parameter for the continuous-time policy and 8% is defined to be the true data generating parameter, (0%, 07)) maximizes V. Then, by (23).
(0) oo 03) maximize L.
LEMMA L10 For any 8 > 0,
20-18A
(p 2)8A
sup
(2!T"
**
S
sup 0S1ST
[=' )
0StST
P
p52/ (p-2) Re
Proof. Proof by applying Young's inequality. See derivation in Higham et al. 2002 Equation (2.8)).
27

---

Under review as a conference paper at ICLR 2023
LEMMA L11 For any 8 S Tu. the difference between the coefficients of the dynamics of 27' and |
are bounded by:
dg Σ ((
dg
[11 S
Σ (1)
S/4.
5dg expl (2R) C1 exx((2)) [t]| + exp(2R) R² [ X(u] 1
+ R² ||ااا - Sall + exxp(22)
-.-h 2 {
2
+ exp (2R) 21 - 2 1 13,
and
[[(Et) [A]]: (7) dw
S - 1
Σ (x:: |]
dw
Σ
Σ 603 (X:; 03))
Σ I1(t, Xx; 0
Iod3 exp cx exp(2R) R²It It - |t| + exx(22) R² X - -X(e]l 1
+ R2 |/g - Sill + exp(2 / - -1551( 12
+ exxp(2 2 -2,15/- 17]
Proof.
By triangle inequality
de Σ (xr:
- Σ (Xx:003)) [1 S
de
Si
de
15's (X:: 03) ( | S(10 - :1e 25 si15
1=1 (t))
+
$1 + M - 20
Stu
2|-" ss
For any t S TR, we can further bound the right hand side by Assumption I.4
ds Σ (x)5((x;03)22
S
Σ (Xx: 2/) [4] 5i11
Lds (xp(R)) R(p(R) - |t]|+ -x(yll
+ eexp(R) 2/" - -2,*'1 + 2 - 11 + R || - S
Then, by Lemma 1.8
2
Σ (X,; (3) 1('' St
Σ (x);)))))
[0] S
35d' exp(2 (xp(2 (2pp2)22)) (1= (1- + |x, - -110 13)
+ +exp(2 255 - 2.  | + +| - | /3/ + n22 |||1]] - S -s.l
=5d3 exp( (xp(2) exp(2R) R² |t||2 + exp(22) R² Xx 10--pp XXwl 1
+ R² 11 Ss 2 + exp(2R) -..¹ - 2! exp(2R) 77,-'J - 2 [ 19.
[(]
28

---

Under review as a conference paper at ICLR 2023
Similarly
d,
dw
Xx; f3) St 1
ds
dw
(g' (X:
1/11
1=1
j=1
j=1
S 5d exp(2R) (2 R - |t]| +
(2R)
X [111| 1
+ R² ||اا]] Sz 1 + exp (2
" - -1 2
+ exp(2 R)
7.
11 1 19.
LEMMA 1.12 With TR defined in Definition
1.5
1 XENTR X[eATR]
2 2.0 (At + +
E SEATH - -
1 2cxxp(2R) C (At At),
E
Z LeATR]
S 2R exp( ( (AP + At).
Proof. By the dynamics of Xx and Lemma L8 we can derive
LATR
LATR
E El|TeR X 1| S 2
(Xx: ds+2B
ddW, gx (3) dW,
Imnen At
ming At
where MIATA satisfies MENTA At < (t (n) < (MEATA + 1)At. Further, we apply Itô's isometry with stopping time (Lemma) D.2. and derive
LATR
E XEATR - XjenTRll 11
S 2
19xx(Xx:03)1 ds. [xxx(xx)03)) ds 2B
At
mmmm Al
By Assumption
we derive
XEATH [ 112 S22 + At).
Similarly
E SEATA S(eATH] 112 <2exp(2R)CH(AF+ At), At))
and
E 2
22 (F Z
LEMMA I.13
E sup
(2 )
+10(T+4)Tdg ( + 20 + +1t)
OStST
+ +2C7 At) 2 2exp(4 + At)
2
C exp(4R) + +10(T + C
E sup 05r5s
2
-TTTT
ds.
Proof.
By inequality and the dynamics of 2, and 2|**1 for any T S
E[ sup
(2
ZEAtH
051ST
ds MATRR
dg
(" St
2
<2TE
sup LostSr_
(X,s 03)25' - Σ
03)
ds
Si
1=1
1=1
2
dw
dy
2E
sup DSAST
# 03:
(x
dwi
29

---

Under review as a conference paper at ICLR 2023
Then, by Doob's martingale inequality
BAAE )2]
E sup 0StST
(ZEAEN
tATR de
ds
2(F11 S 2
<2T
USSIS JO
sup
$ # (Xg;
-E
क (X,:
[e]
S/:
ds
dw
EATR
Σ 3 (Xa: 02)2. - (X ( dWp }
dg
+ STE
Next, we apply Lemma 1.11
E sup 0StST
(
-Z(
TATR
<10(T 4)d exp(2R)
(exp(2R)
[s]1 exx((2)) Xx X(1)
+ + - Ssll + (2xp(21) 2 - -2 (2Xp(2 | )dsl
Therefore, combined with I.12
E sup (Z - )T S10(T + C 0SeST
+ 20 + +t)
+ (AF + +1) + 2xxp(4R) /A22 + Ar)
+ 10(T + d) exp(4R) [
2
sup 05r5e
7 -TATRR
ds.
LEMMA I.14 With the definitions and assumptions in Section
lim B[(2]=' A6+0
-[ 1
Proof.
By Lemma
I.13
we apply the Gronwall inequality and obtain
E sup (Zir.
Z )²]
0S1ST
< 10(T + 4)Td& exp(4R) Ch (AP + + + At)
+ (A At) + 2exp (RR)) (A + At) exp(10(T + 4)xxChex)(4AT).
Then, combined with Lemma 1.10 for any 8 8>0 0.
( - "!**11
E sup USIST
< 10(T + 4)T C [ At +
(At + At)
+ (At +A) 2xxp((RR)) At2 + At)] exp(10(T + C exp(4R))
2p+ 8A
2)8A
1821(2-2)12'
Therefore, E(suposisT 17. 2," # 111
converges to Al goes
30

---

Under review as a conference paper at ICLR 2023
1.3 PROOF
For ease of presentation, we define
θ == (0g. 1x. 0s), 0" == 07. 03) and 0/.. (0% 07 AsS
Note that every discrete-time admissible policy is continuousttime admissible policy. Thus, the continuous-time admissible policy includes the discrete-time admissible policy set. Therefore,
V(0)) V
In other words, it enough to bound V(0*) V()),) .) for the proof. By Lemma I.9 8" maximizes V and simultancously, leading to
- V 5 H())) H
1
By Assumption 1.7 for At < At', 0* is also admissible parameter 0 E A, leading H ( H (0)) I O. Further, for any å > 0, by adding and subtracting equal terms,
v(0') ) - V(
1
$ - [100)) ((")) + H())) (02,))
(24)
1 H(0")- H(
1
Next, we focus on ( 8) : 6)/. which by definition has
()) :0:0;)0) 0) = =(1 -0) U(( :00]
where AL(O") in both (0:; 8) and 8)) omit each other. By Lemma I. .4 we have lim 2 1a*") E, 7 Since (2; is continuous function, implement the continuous
mapping theorem and derive
lim U(Z 1:8) P, U(F :8). Ar-0
(25)
By assumption I.4 2
with different finite At is uniformly integrable. Then, following 5) is also uniformly integrable since (2; 8) has linear bound. Combined
A<ca
Assumption 166 U ( :
#1 #1
with (25). we derive
lim BIU 5)] + E 2 5)]
4150
which finishes the proof.
J EXTENDED RESULTS OF THEOREM 4.2
In this section, we study the non-asymptotic guarantees on the performance of FaLPO.
J.1 ANOTHER VERSION OF THEOREM 4.2
DEFINITION J.1 For two random vectors and w, we define the trace of the covariance matrix as
Var(e) [| ||[(c]|6]
Cov(r, w) Eu T w) [0] R[.].
Further, we use Var (v) and Covo(u, w) denote the conditional version of the two given 0:
Varę(v) ==B[]| - ||L[ 10
Cova(e, w) Bu T w| [[]|o0 [|]]
31

---

Under review as a conference paper at ICLR 2023
Note that it is challenging theoretically analyze non-convex stochastic optimization ( while there are various ad-hoc procedures providing good empirical performances. To provide theoretical analysis, in this section, we study projection based version of FaLPO (Algorithm ( Specifically, the learning/optimization process is conducted in a bounded parameter space B, under which we assume that the objective function is strongly concave regarding the parameters.
Algorithm 2 Projected FaLPO
1: Input: Hyperparameter X, learning rate y, number of iterations N, the strongly concave region B, and batch size B.
2: Output: 0g. On. and 0g
3: Initialize neural networks with initial parameters (0s. 0z 0g) E B.
4: Parameterize the policy function by (3).
5: for E [M] do
6: Collect B trajectories.
7: Estimate the gradients of H following the procedure in Appendix C with the parameter À.
8: Update A and 0g with learning rate 1) by gradients.
9: Project the achieved update to B.
end for
11: Return lg. 0T,, and Os-
For ease of presentation, we define
f* (6), f/ 0g), 0%. == (0 Ar (r and 8 := (0) o! 7. 0 (,A)))
Let On be the estimation after the th iteration, and == >N o., the average estimation. It is common technique to consider the average estimation instead of the final estimations Oy for such analysis. Then, we provide new version of Theorem 4.2
N
DEFINITION J.2 With the gradient estimations discussed in Appendix we define
(+) = (1 - \)VV& (0) + (O).
THEOREM J.3 With assumptions in Section J.2
X E [0, 1), and <
1 C₁ as the learning rate,
E[VL. [())
A(0)) H(8)))
H(O)))
H(0))
C8 Coggg(N)
+
1
2N A) 1
N-1
+ 1
[11 2) Vare. (FVz(On)) + Var (FL)(0.))
E
2NB X)
n=0
+ 2X(1 - A)Cove, (V.). (.
Also, there exits situations where a À E (0, 1) provides smaller value for (1 2ar ($H)(B.)) -
Var L (0.w) + 2X(1 - A)Cov(HH (4,)), (&,)) than = 0. In other words, there exist
cases where tuning À may provide better performances.
J.2 ASSUMPTIONS
ASSUMPTION J,4 There exits constant CB > 0 such that the parameter region B is convex set and satisfies the following conditions
1. B A, H(0g At. 0r, 05.Ar) is locally m-strongly concave with local maximal point (0). A. 0g s B.
2. For any E B, |0|| < Cs.
32

---

Under review as a conference paper at ICLR 2023
3. For any E B, the expectation of the gradient estimation is bounded by
E
k=
CB.
B
These assumptions are widely used in existing analysis (Papini et al., 2018 Karimi et al. 2019
Agarwal al. 2021 Bhandari & Russo 2019 Wang et al.) 2019 Xu et al. 2020)
ASSUMPTION J.5 At the nhhh iteration, we use the learning rate as 7h++1
Note that in practice we will tune the learning rate 7) as hyperparameter, since we may not know m..
However, it is common practice to set the learning rate as in Assumption J.5 (Hazan & Kale 2011 Nemirovski al. 2009 Shalev-Shwartz al. 2011
J.3
TECHNICAL LEMMAS FOR THEOREM 4.2
LEMMA J.6 With Assumption 1.4 and
J.5
we have
N
Exs1 (ll)
27
H(0)) - H())
-
Σ
2N
+
B
n=0
N-1
Σ k
V HH
(0x))
(On)
VH(0,))
(0T 0m)
B
Proof. By the strong concavity of H in Assumption
1.4
(H(O)) - O.)) 2 H()0) - H(O))) + 2 10,, - 0' 10112.
(26)
2 He((ln)
5
Further, since ax+1 is the projection of 0, + 1h+11
to B, the projection satisfies
B
B Ek ((m))))
On + 1n+
01
|10001 - 0'112
B
which suggests
00, - 00112 - -110n-1
0tl1
ΣB (0.)
|10, 0-1
On + 7at+1
gl
B
ΣB
A (O,)
(20,. + me+11
2
B
Hz((O.)
20
(27)
1
B
B
Σ
B
H4
2
2mp+ (0, -00)
-K-1 (0,)
B
m
We reorder 27 and derive
B
& (O.)
-0,, - 81) T
|00, - -0111
2mh+1
+ 2
Σ bi 4HH (0x)
00+11~ - -0(113)
1n+1
B
33

---

Under review as a conference paper at ICLR 2023
Taking the result back to
26
Ek k= (On) (0' 0n)
H() (1(0)) ~ (0,)
B
Σ
VH* (On)
|10, - 0 11 - -0i nh+1
+ 2
B
m 11 01112
2
ER=n (0n))
1 (8 - 00,) + ((7771
(0,))
- m) 10,, - -t111
B
2
Ek=1 HHH(On)
110m-1 - 0'112+ 2 nh-
21J0+1
B
(28)
By averaging over n with Assumption J.5
we get
H(Ot) (0) S
(H H(Oo)))NN
2k=1 (1m))
2N
n n+1
B
N-1
Ee (..))))
)
VH(O,))
(0) (0) - 0n)
B
where the first inequality is due to condition of Assumption I.4
J.4 PROOF OF THEOREM 4.2
Note that every discrete-ting time admissible policy a continuous time admissible policy. Thus, the continuous-time admissible policy set includes the discrete-time admissible policy Therefore,
V(0") 2 Vá.-
Therefore, it is enough to bound (((0)) - (0) for the proof. By Lemma L.9 0* maximizes V and L simultaneously. Therefore,
1100)) H (0) (00)) - (100.)) + ((22)) (())) + 1(00 - (1)0) ((00)-((' 1
(29)
Then, we use the convergence result for Algorithm
2
detailed by Lemma L.
H(Ot) (0) S
1
Σ
N-1
ΣB k=
(f,))
2N
n+1
B
n=0
(30)
N-1
Σ
VH(Om)
Eke
A HH (Om)
(0) (0'- (0'
N
B
n=0
Then, take expectation both sides
N
Ek=1 (0,)
E(H((t)) H())) S
Σ
E
2N
n 1+1
B
(31)
N-1
ΣB (H,) )
B
(HH(0))
(0t (0t-
B
m=
34

---

Under review as a conference paper at ICLR 2023
For the last component of (1p.. since (B,) is unbiased gradient estimator: (0) -On)
B
(0n)
(O,)
k=
E (Hn)) (H(),)
(0) - mn) On 11 =0.
E
E
B
Then, for the first component in (31)
V H (0,))
71
B
Σ k=
A H (O.))
B8
II
10x
1
B
Σ Sk=1
B
H (0,)
B
(On)
Varg.
)
0n
B
Σ -==1
B H4 (0n))
Varg,
+
C3
11
where the last inequality is due to condition 3 of Assumption J.4 Then,
31
can be further derived as
N-1
V (O.)
1
B[H(0t) H(0)
E
Varo..
+
CB
2N
n
n +
As a result,
(0H() H())
(0(H(0%.) H(O0) + (O00) H())
<H())) - H(O0) + CB log(N))
2N
N-1 7|(1 - 2a on (VV2(0%)) + Var (vL)(O)) )
+
BN(++1
+ 122(1 A)Cova, (VVi(O.), Lo.)))))
N-1
BN +1)
Taking the results back to
29 we finish the proof.
K EXTENDED RESULTS FOR SYNTHETIC EXPERIMENTS
For synthetic portfolio optimization, we provide details for drift and volatility (Appendix K.J). data generation (Appendix K.2). hyperparameter tuning (Appendix K. and extended experimental results (Appendix K.4) We consider 21 21-day trading, and generate 1000 trajectories with observations for training, 1000 for validation, and 1000 for testing. To compare different methods, calculate the average terminal utility as the metric.
K.1 DRIFT AND VOLATILITY
Drift and volatility are two important concepts characterising the strength of signal and noise in financial markets. To demonstrate this, for an asset price S and time interval At, define return as:
StAA: S
return;
St
35

---

Under review as a conference paper at ICLR 2023
The return) can be daily, monthly yearly, depending the length of At. For specific asset, drift (fxr (%) in (2) is approximately the expectation of the return, while volatility (g' (Xx; 63)) is approximately the return's 's standard deviation. Given multiple assets, drift (f(X;;02) (2) is vector and volatility (g( x ()) a matrix. When generating synthetic data (Appendix K.2). we fix the scale of drift and vary the scale of volatility, which is defined the average value of each component.
K.2 DATA GENERATION
We simulate data for S, and X, following SDE 6 To this end, drift and volatility are randomly picked while mimicking the historical stock price data, with an average annual return around as 0.1 and average annual volatility in {0.1, 0.2, 0.3). leading volatility around {0.1/252, 0.2/252, 0.3/252).. The true representation function is selected component exponential operation. Then, we discritize the SDE following the explicit Euler method, and generate data accordingly (Beskos & Roberts 2005).
daily return around 0.1 and daily
252
The specific configurations for data generation is:
Define two scalars: Ca. and Cp determining the scale of drift and volatility:
Ca 0.1/252, and C, (0.1/252, 0.2/252, 0.3/252),
is selected a random matrix, whose components follow uniform distribution in (05C,,,,55C,
v is selected as a random matrix, whose components follow a uniform distribution in [- -,,5,,C, 1.CCC..).
r is selected as diagonal matrix whose diagonal components follow uniform distribution in [0.9.
The initial values of are randomly generated from a uniform distribution on [-2Ca. 2Ca]
The initial prices of assets are randomly generated from uniform distribution in [20, 30].
Note that the design makes sure that the simulated price has approximately a yearly return of 0.1 and yearly volatility in (0.1, 0.2, 0.3). Table reports the experimental setup parameters.
Experiment Configurations
Values
Time Interval At
I (Day)
Terminal Time T Scale of Annual Drift
21
0.1
Scale of Annual Volatility
{0.1, 0.2, 0.3)
Number of Simulated Trajectories
1000
Utility Function
(Power, Exponential)
Risk Aversion y
3,.1,3,5, 5, 103
Number of Replications under Each Hyperparameter
5
Compute Resources
AWS ec2 m5ad. 22xllarge
Table 4: Setup for synthetic experiments
K.3 EARLY STOPPING AND HYPERPARAMETER TUNING
For better performance, we conduct early stopping for all methods using the average validation utility with the patience as 5 steps. The considered hyperparameters include the learning rate, X, and batch size. For each configuration, we conduct training for times, and average the results. Then, we pick the configuration providing the best average validation utility, and test it the test data and calculate the average test utility per trajectory. The tuning process is conducted using the software wandb (Biewald 2020). Table rieports the hyperparameter values.
36

---

Under review as a conference paper at ICLR 2023
Hyperparameters
Values
Batch Size
{100, 50}
05, 05, 0.1, 0.9}
Learning Rate
{0.0005, 0.001, 0.01, 0.13
Table 5: Hyperparameters for synthetic experiments
K.4 SYNTHETIC EXPERIMENT RESULTS
To gain a more holistic understanding of the performance of FaLPO a variety of settings, we conduct experiments under different number of stocks to be traded (Appendix K.4.1) different risk preferences (Appendix K.4.2), and alternative utility functions (Appendix K.4.)). Finally, we also compare the performance of various methods under the Merton model as sanity check (Appendix K.4.4).
K.4.1 SYNTHETIC EXPERIMENT RESULTS WITH DIFFERENT DIMENSIONS
Tables 6 and 7 report the synthetic experiment results with the number of simulated stocks (ds) varying in {10, 15}. The performance is not strictly negatively correlated with the number of dimensions the problem or the annual volatility in simulation. The reason is that the noise in the problem is indeed determined by the whole volatility matrix which is randomly generated in the synthetic experiment (Appendix K.2). other words, the dimension and average scale cannot fully characterize the extent of the noise in a synthetic task.
Annual Volatility in Simulation
0.1
0,2
0.3
FaLPO
-0.465 + 0.446
-1.35 + 0.155
-2.737 + 0.219
DDPG
-1.65 + 0.456
-3330 + 1.294
-5.495 + 1.269
SLAC
00.750 0.210
-5.50 0.011
-6.160 0.012
Methods
RichID
3.350 + 0.111
-5.65 0.102
6.325 1 0.048
CT-MB-RL
2.850 + 0.014
-5.35 + 0.020
-.160 + 0.026
MMMC
4.723 + 7.619
-5.602 + 4.299
-6.124 + 3.217
Table 6: Average terminal utility after tuning with standard deviation for synthetic data with ds 10 and dw 10.
Annual Volatility Simulation
0.3
FaLPO
-2,463 3.744
-1.021
0.278
-..443 4 0.547
DDPG
-3.976 + 1.428
-1.443 0.751
-5.205 + 1.858
SLAC
-4.749 + 0.139
-6.129 +.001
-6.526 + 0.012
Methods
RichID
-4.973 : 0.448
-6.321 + 0.038
-6.641 = 0.022
CT-MB-RL
3.077 +: 0.014
-5.714 0.023
6.363 + 0.021
MMMC
5..88 1: 5.688
-6.465 4.978
-7.155 1 5.965
Table 7: Average terminal utility after tuning with standard deviation for synthetic data with ds = 15 and dw = 15.
K.4.2 SYNTHETIC EXPERIMENT RESULTS WITH DIFFERENT VALUES OF
Tables and report experimental results with ds = 10, dw = 10. and y E {3, 10) for an exponential utility. FaI.PO outperforms the competing methods in most scenarios.
37

---

Under review as a conference paper at ICLR 2023
Annual Volatility in Simulation
0.1
0.2
0.3
FaLPO
-0.003 0.0021
-0.0055
0.0008
-0.0132
0.0028
DDPG
-0.003 = 0.001
-0.0105 + 0.006
-0.0205 + 0.0034
SLAC
-0.003 0.0007
0..0153 0.0013
0.0192 0.0011
Methods
RichID
-0.012 + 0.0005
-0.0188 + 0.0002
-0.0211 + 0.0
CT-MB-RL
-0.01 = 0.0
-0.0179- 0.0
-0.0206
MMMC
0.0162 + 0.0212
0..019 + 0.0135
-0.0210 + 0.0102
Table 8: Average terminal utility after tuning with standard deviation for synthetic data with y 10.
Annual Volatility in Simulation
0.1
0.2
0.3
FaLPO
4.575 3.325
17.9358 2.3349
56.0405
13.0502
DDPG
-33.113 4.3472
-51.6559 + 11.7981
-50.0399
13.8451
SLAC
23.514 = 11.7077
688 68.6816 = 0.0254
76.1371 +: 0,0355
Methods
RichID
44.629 1.8797
69.481 0.9413
77.233 0.1091
CT-MB-RI.
-34.842 0.4686
-65.41 0.1331
-75.4364 + 0.122
MMMC
- 59. 2338 77.4511
77088667 + 49.2500
-76.8619 + 37.448
Table 9: Average terminal utility after tuning with standard deviation for synthetic data with
3.
K.4.3 SYNTHETIC EXPERIMENT RESULTS WITH POWER UTILITY
We also conduct synthetic experiments maximizing the expected power utility for portfolio optimiza- tion. The results summarized in Figure
Aneran Uiillyy
Enilley
Methods
Mothods
10.00
Muthods
y 0.1
(b) = 0.2
(c) 0.3
Figure 5: Average Terminal Power Utility
K.4.4 SYNTHETIC EXPERIMENT FOR THE MERTON CASE
As a sanity check, we study a Merton problem where the optimal performance can be mathematically derived, in order to compare the performance of FaLPO to the optimal one. We simulate data
following a Merton model in Appendix F.T with ds 10, dw = 10. With an exponential utility function with 7 5, according Lemma F. the optimal policy can be derived
M(a0T) T)-
(32)
Z₂
Further, by taking 32 back into
14 we can derive the optimal expected terminal utility as
720 tu
max E. U
#1
which is the theoretically optimal performance. Then, to implement FaLPO, we generate fake features which are independent from the asset prices: the optimal policy is not dependent on these features. Ideally, FaLPO should be able to automatically ignore the fake features, and deliver performance similar to the theoretically optimal derivation. The results of FaLPO. MMMC, and the theoretically optimal derivation are reported in Figure 6 Note that FaLPO achieves slightly worse performance compared to the other two. The reason for the slight suboptimality of FaLPO the Merton case is twofold: i. the expected terminal utility is derived for continuous-time policy while Fal.OO learns
38

---

Under review as a conference paper at ICLR 2023
discrete-time policy with time interval At; ii. FaLPO uses over-complicated model with stochastic factors, while the true data generating process follows a Merton model without stochastic factors.
Average, /Ex Expeecedd
Terminal Utility
Negative
PILPO
Oytana
Methods
Figure 6: Negative average terminal utility of FaLPO and MMMC, and negative expected optimal
terminal utility. The smaller the better.
L EXTENDED RESULTS OF
STOCK TRADING
L.1 PROTOCOL
We consider 21-day stock trading in four different stock sectors using the daily stock price data from Yahoo finance between January 4, 2006 and April 1, 2022. More specifically, close price as the daily trading price. For factors, we consider economic indexes, technical analysis indexes (generated by python package TA), and sector-specific features such as oil prices, gold prices, and related ETF prices, leading to around 30 factors for each sector. In each sector we select 10 stocks according to the availability and trading volume in the considered time range. The considered sectors, stocks, and the features are provided in Table IO We consider the same competing methods Section 5. and compare the performance using the average achieved terminal utility over different trajectories as the metric. The larger the utility the better.
the adjusted
Sectors
Stocks
Features for Factors
SP500 returns, MACD of stock prices, RSI of stock prices, oil prices, gasoline prices US Dollar/USDX Index Cash (DX- (.XYYYYB)
APA, COP, CVX. HAL, HES, MRO, OKE, OXY, VLO, WMB
Energy
SP500 returns, MACD of stock prices, RSI of stock prices. ETF prices including DIA, EXI, IYJ and VIS
BA. CAT, DE, EMR, ETN. GE, HON, LMT, LUV, PNR,
Industrial
SP500 returns, MACD of stock prices, RSI of stock prices, gold prices, silver prices, ETF prices including IYM and VAW
APD, AVY, BLL, DD, ECL, FMC, IFF, IP, NEM, VMC
Materials
Table 10: Selected stocks and features
L.2 EXTRA PENALTIES
For real-world experiments, we consider two extra penalty terms for better stability. The first penalty is the model calibration loss discussed in Appendix E.3 Given trajectory with time interval At,
T == (t,, St, Tt, [m]], it defined as
A1 min Σ (1(()))) Co(X..) bbll C
where C' is matrix and b vector of proper dimensions. As discussed in Appendix E.3 this penalty encourages a simple representation function d. The second penalty is the negative sample variance of the terminal wealth, with the parameter A determining its strength. The intuition of this penalty is to further penalize the instability of the algorithm performance. The second penalty is implemented for all the competing methods except MMMC for fair comparison.
39

---

Under review as a conference paper at ICLR 2023
L.3 TRAIN- VALIDATION- SPLIT BY SLIDING WINDOW
We detail the sliding window method for train ceinn-vlldatiinntess split for real-world portfolio optimiza- tion experiments (Figure 7 In financial markets, the dynamics under asset prices and factors vary over time, leading us to construct sliding window on the dataset for training, validation and testing. Specifically, given dataset of asset prices and observed features, construct several windows of observations of equal length. We divide each window into three contiguous periods, the first used for training, the second for validation, and the third for testing. We refer the length (in days) of the training period as the training size (same for validation and test periods).
After constructing one window, we move the start time point by fixed number of days (the window gap), and construct the second window. A given method is trained on the training set of each window separately, and then validated and tested the corresponding validation and test sets. The final validation and test performances are calculated by averaging over each window. The experimental setup is summarized in Table We report the considered hyperparameter values in Table
Window Gap
Time
Window
Train
Validation
Test
Window
Train
Validation
Test
Window
Train
Validation
Test
Figure 7: Demonstration of sliding window.
Experiment Configurations
Values
Time Interval At
1 (Day)
Terminal Time T
21 Exponential
Utility Function
Risk Aversion ?
Number of Replications under Each Hyperparameter
10
Compute Resources
AWS ec2 m5ad 22xlarge
Table 11: Setup for real-world experiments.
40

---

Under review as a conference paper at ICLR 2023
Hyperparameters
Values
Batch Size
2000,200,440
{0.1, 0.5, 0.9)
Learning Rate
{0.0005, 0.001, 0.01,0.1)
Window Gap
{63, 126}
Train Size
{1260}
Validation Size
{63]
Test Size
{63)
Table 12: Hyperparameters for real-world experiments
L.4 SENSITIVITY ANALYSIS ON X
According to (5), the value of determines the weight of the FaLPO model calibration. In this section, we conduct sensitivity analysis of FaLPO X. Under the same protocol as the experiments in Section 5.2 we also report Fal PO with different values of À when applied to different sectors. The results are reported in Figure Compared the case without model calibration (A = 0), small non-zero À provides higher terminal utilities and lower variance. This observation justifies our method of incorporating model calibration into policy learning. Then, when gets bigger and close to one, the performance of FaLPO decays while the variance also gets smaller.
Aatree Utiles
Unilas
Aserage
Aueraan
(a) Oil
(b) Material
(c) Industrials
(d) Mix
Figure 8: Sensitivity analysis for À
M MORE INFORMATION FOR COMPETING METHODS
Here we provide more information for the implemented competing methods. First of all, we focus on policy learning methods, without studying other performance improving techniques like data augmentation or feature engineering. (See Appendix A.2 for review of such methods.) Such techniques can be easily applied FaLPO.. Further, for a thorough comparison, we summarize the existing policy-learning methods for portfolio optimization with the following four representatives. Note that, all the following methods take asset price data and features as the input for a fair comparison.
DDPG is implemented with the gradient estimation detailed in Appendix and also dis- cussed Nan et al. (2022); Xiong et al. (2018); Jiang al., (2017). This design makes sure that DDPG can leverage offline data without exploration.
SLAC (Lee et al. 2020) learns representation of factors jointly with policy learning. But in this process, no parametric models are used.
RichID (Mhammedi et al. 2020) falls into the category of model-based policy learning like Yu et al. (2019). first learns the representation of factors and then conduct policy learning. In this process, both steps take advantage of parametric model. For better performance in portfolio optimization, we pick Kim im--mberg model as the used model, instead of the LQR model original proposed with this method.
CT-MB-RI L is policy gradient method optimizing the performance objective using the pol- icy functional form derived from continuous time models, but without factor representation learning.
41

---

Under review as a conference paper at ICLR 2023
We also implement MMMC as representative of continuous-time finance methods. More compli- cated and advanced continuous-ting finance methods are hard to implement for two reasons. First, to implement such methods, we need to estimate all the parameters of multivariate SDE (like 0, e, and w in Section 3.3). It is challenging since the derivation of likelihood requires solving multivariate stochastic integrals (Ait-Sahalia & Kimmel (0100), Second, deriving explicit optimal policy functions is also difficult, which involves solving nigh-dimensional PDEs (like k2 (t) and ka (t) in Lemma F2). Further, such pure continuous- time models are expected to underperform, since they assume that the data exactly follow parametric SDE and tend to underfit. This can also be seen by our comparison with CT-MB-RL (Section()) [5). which is model-based RL method by relying on a Kim CannOmeerg model. As result, we focus our empirical comparison to more competitive RL methods.
Note that Fa circumvents the two aforementioned challenges. First, our model calibration does not aim to fit all the parameters in an SDE, but only those related to learning 0g and 0.. That is why our model calibration loss in Section 3.3 has such an easy-to-calculate form with the parameter 0g as a simple vector. Second, FaLPO does not need a fully derived closed-form solution for the optimal policy, Like in Section 3.3, we use neural networks to parameterize (t) and &(), instead of fully deriving them like continuous-timg finance methods. Being able to bridge this gap between continuous time finance models and high multidimensional stock trading problems one of our contributions.
N EXPERIMENTS WITH TRANSACTION COSTS
In this section, we consider the case with transaction costs. Usually, the cost of borrowing a stock short can vary but typically ranges from 0.3% 3% per year. Therefore, we take 1% annual transaction cost for short selling an asset. (The fees are applied daily.) Under this setting, we replicate our real-world experiments for the oil sector, using the same protocol. After the tuning procedure in Appendix L.3 the achieved results are reported in Table 13 consistent with those without transaction costs.
It should be noticed that the results are
Methods
Average Utility
FaLPO
-2.25 + 1.649
DDPG
-6.795 + 0.8247
SLAC -7.115 0.8872
RichID
-6.365 + 0.5989
CT-MB-RL
-5.57 + 5.036
Table 13: Average terminal utility in oil sector with transaction costs
O EXPERIMENTS WITH DIFFERENT INITIAL WEALTH
We vary the initial wealth in (3000, 5000, 8000, 1000} for portfolio optimization using stocks in the oil sector following the same protocol as the experiments in Section 5.2 The results are summarized in Table 14 Specifically, Fal.PO achieves superior performance the competing methods with different initial wealth. Also, it should be noticed that all the methods achieve higher terminal utility given more initial wealth.
Initial Wealth
3000
5000
8000
10000
FaLPO
-21.08 + 16.775
-2.4 1.9
-0.243 0.1209
-0.03595 + 0.008896
DDPG
-909.5 + 3443..33444..8 3443.846
-6.6 1.2
-0.34665 = 0.01809
-0.046755 + 0.00444
SLAC
-11865 + 47260.664
-6.8
0.35465 + 0.07427
-0.04558 1: 0.00142
RichlI
-45.51 8.368
0.1
-0.33125 + 0.009791
-0.045365 + 0.0001 446
CT-MB-RL
28.715 + 18.303
6.2
-0.30995 0.1441
-0.013055 0.002061
Table 14: Average terminal utility in oil sector with different initial wealth
42


Reviews:
[{"review_id": "nGaUApxA3se", "reviewer": "Reviewer_oc9r", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes Factor Learning Portfolio Optimization (FaLPO), which combines tools from both machine learning and continuous-time finance. It has three learnable parts for: (i) learning factors ($\\theta_\\phi$) (ii) learning trading policy ($\\theta_\\pi$) (iii) calibrating SDE ($\\theta_S$). The role of factor learning part (i) has an influence on both the objective of (ii) and (iii), therefore these three parts can be jointly optimised. On both synthetic and real-world portfolio optimization tasks, FaLPO outperforms some existing methods.", "summary_of_the_review": "Overall, I think this paper address an important (and popular) problem in finance, but the current experiment design can not deliver a convincing result.", "strength_and_weaknesses": "* Strength\n\n(i) The work is well written and motivated, compared to a policy network that observes pricing data only, the proposed method involves factors.\n\n(ii) The factors are not manually defined, instead they are learned, within the unified objective that considers both the evolving of factor and asset (through an SDE) and the RL-based loss (through DDPG).\n\n(iii) The comparison against several baselines showed some improvements.\n\n* Weakness\n\n(i) I think the experiment design has a major flaw. The key difference here is that the proposed method can access some external data -- factors, either in raw or derived. Therefore, to make a fair comparison, the authors should keep RL part (DDPG) the same, and try to feed (i) no factors (ii) \"off-the-shelf\" factors, e.g., Fama-French-Five (iii) \"off-line\" learned factors, e.g., by fitting SDE or training NN-based model separately. By doing so, we can identify the improvement is indeed credited to the joint learning of factors.\n\n(ii) Some details on backtesting (e.g., transaction cost, as short is allowed -- the cost of borrowing assets, etc.) are missing, and it's a concern why Table 3 (Mix) got some extreme values like 10^8.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jJ0HObo_i8M", "reviewer": "Reviewer_TKCJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A portfolio optimization method with a stochastic factor model combined with reinforcement learning is proposed. Financial time series have a low signal-to-noise ratio, which is easy to fall into overfitting. The stochastic factor model is robust to such noise as its model noisy terms but it is possible to oversimplify the complex relationships between factors. Meanwhile, the RL approaches try to model every detail and easy to falls into overfitting. FaLPO is proposed to interpolate the RL and stochastic processes to take benefit of the robust modeling power of the stochastic factor model and the flexible modeling power of the RL approach. ", "summary_of_the_review": "Overall, it is technically a novel paper, but the balance of the contents needs to be improved. As many papers in finance literature consume large portion of the robustness checking procedure, experimental results need to be improved. ", "strength_and_weaknesses": "The overall idea is strong and makes sense, as there are many well-written papers in ML fields incorporating the flexible modeling power of ML models and domain knowledge. It is true that the ML models are prone to fall into overfitting and stochastic models underfitting. And it is a technically challenging procedure to bridge the discrete-time RL models and continuous-time stochastic models. Policy functional form is quite novel as it bridges the discrete and continuous model. \n\nOverall, the paper is easy to read, but the background contents are too much and could be simplified and adequately referenced. It is kind to provide details of finance fields (as it is self-contained), but the amount of appendix is too much. \n\nExperiments seem to be improved. As the Kim-Omberg model could not model every detail(properties) of real-world data, the synthetic experiments seem to be not enough to show the superiority of the proposed method. Also, as the RL models improved over time, the stochastic models also improved over time, but only MMMC (written in 1969) are compared. The FaLPO may capture (cross-sectional) stock selection performance or market timing, but those components are not systematically compared as in finance papers.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "FpS6I7c4JJn", "reviewer": "Reviewer_gA89", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the portfolio optimization problem with factor learning, where the authors use deep reinforcement learning models to learn the factors, and combined that with the classic continuous time finance factor model. The deep RL component is based on the deep deterministic policy learning (DDPG) model from the literature; the stochastic factor model is applied with the learnt factors. \n\nThe authors provided theoretical analysis for the proposed framework (asymptotic convergence and finite-sample convergence rate), and conducted comparative experiments with several other baselines using synthetic and real-world stock data.", "summary_of_the_review": "Overall I think this is a well-written paper which provided a natural solution to the portfolio optimization problem. However I had some concern on the experimental comparison side where some stronger baselines might be missing. I was also not able to find implementation details.", "strength_and_weaknesses": "Strength:\n- Overall the paper is well-written and the structure is fairly clear\n- The proposed method appears to be a natural solution that combines the classic continuous-time stochastic factor model with deep reinforcement learning. By design it takes advantage of the continuous-time model which models the dynamics explicitly, and the advantage of the neural networks which learn the complex correlation between the raw features and the useful factors.\n- Both convergence guarantees on the theoretical analysis side, and the comparisons with other baselines are provided.\n\nWeakness:\n- I found the theoretical analysis section written in a bit confusing way where the exact assumptions are all referred to the sections in the appendix. For example, in the theorem statements, the needed assumptions are delayed until several places in the appendix. It is not clear to me the precise assumptions that are needed.\n- In terms of the baselines used, the selected baselines are a bit on the weak side. The authors compared to the baselines that either uses a vanilla stochastic factor model (MMMC), or those use general deep RL only (not designed in particular for stock trading). The authors did not compare with other existing works which also incorporate RL techniques for portfolio optimization or trading, such as the related works mentioned in Appendix A.2. It is unclear if the proposed method is indeed outperforming the state-of-art approaches. \n\nOther questions:\n - I had some confusion when interpreting the numbers in Table 3. It is said that the experiments were conducted with a rolling window, using end-of-day data from 2006 to 2022. Are the numbers in Table 3 averaged values across the entire 2006-2022 period? \n- the results are reported in terms of averaged terminal utility. What about the results in terms of average daily/ monthly return?\n- How does the proposed method’s performance change in terms of the size of the initial portfolio?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject