MultiSTOP: Solving Functional Equations with Reinforcement Learning

Authors: alessandro  trenta
davide  bacciu
andrea  cossu
pietro  ferrero
Venue: ICLR_cc_2024_Workshop_AI4DiffEqtnsInSci
Published: 2024-01-30 12:02:26

Abstract: We develop MultiSTOP, a Reinforcement Learning framework for solving functional equations in physics. This new methodology is also able to find actual numerical solutions instead of bounds. We extend the original BootSTOP algorithm by adding multiple constraints derived from domain-specific knowledge, even in integral form, to improve the accuracy of the solution. We investigate a particular equation in a one-dimensional Conformal Field Theory.

Main paper: 
 introduction: 
 Functional equations appear in many scientific areas of study due to their ability to efficiently model complex phenomena. Unfortunately, when analytical solutions are not available, finding approximate solutions using numerical methods such as semidefinite programming (Simmons-Duffin, 2015) can often be computationally expensive. Although numerical methods can be quite effective, often they only provide bounds on the parameters of the equation. Differential Equations (DEs) and Partial Differential Equations (PDEs) can also be interpreted as functional equations and have been widely studied with both numerical (Butcher, 2000) and Machine Learning methods (Raissi et al., 2017a; b) . We focus on a particular class of equations in the Conformal Field Theories (CFTs) of physics which take the form h(z, z) + n C 2 n F ∆n (z, z) = 0, with unknowns being the coefficients C 2 n and the functions' parameters ∆ n . Historical approaches (Simmons-Duffin, 2016; Poland et al., 2019) for conformal bootstrap equations involve truncating the equation to all terms with ∆ i ≤ ∆ max and applying the aforementioned semidefinite programming. With this methodology, only bounds on the unknowns are found (Kos et al., 2014) . Recently, Reinforcement Learning is emerging as a new paradigm to solve functional equations in physics. A recent approach, called Bootstrap STochastic OPtimization, briefly BootSTOP (Kàntor et al., 2022; 2023) , has been used to find actual numerical solutions for the above equation. Our main contributions are: (a) MultiSTOP, an extension of BootSTOP which enables the enforcement of additional constraints derived from the physics of the model into the algorithm, in order to increase the accuracy of the solutions. (b) A detailed empirical analysis on the effectiveness of the proposed approach on a 1D defect CFT. (c) An analysis of the degeneracy problem that occurs when two or more functions have similar parameters ∆ n in two limiting cases for the model. 
multistop: 
 We start with the description of the functional equation we are interested in. We then introduce our MultiSTOP (Multiple STochastic OPtimization) approach solving the conformal bootstrap equation by introducing additional constraints on the unknowns derived from the physical domain.We focus on the conformal bootstrap equation of a Conformal Field Theory (CFT), the 1D defect CFT defined by a straight1 2 -BPS Wilson line in 4D N = 4 super Yang-Mills theory (Maldacena, 1998; Drukker & Kawamoto, 2006; Giombi et al., 2017) . In this 1D setting, the equation reads:EQUATIONand must hold for almost all the points x ∈ C. Function h(x) is known, while the F ∆n (x) are analytical with respect to ∆ n and are defined in terms of the so called conformal blocks f ∆n (x) through the formulaF ∆n (x) = x 2 f ∆n (1 -x) + (1 -x) 2 f ∆n (x) (see appendix A.1 for a review).Our objective is to find solutions of equation 1 with unknowns given by the squared OPE coefficients C 2 n ≥ 0 and the scaling dimensions ∆ n ≥ 0, which we indicate together as CFT data. In this model of interest the CFT data depends on a parameter g called coupling constant. Previous studies were able to identify the values of the scaling dimensions ∆ n for the first 10 terms of equation 1 as a function of the coupling constant with high precision for many values of g (see Grabner et al. (2020) and references therein). For our purposes, we give these values as input to the MultiSTOP algorithm, halving the number of unknowns we have to find and significantly reducing the search space.In the specific model under investigation, the CFT data are further subject to the integral constraints (equation 2), where RHS i are functions of g alone, while Int i are integrals of the conformal blocks f ∆ evaluated with respect to a certain measure 1 : Constraint i:EQUATIONThese constraints were derived in Cavaglià et al. (2022) ; Drukker et al. (2022) from the requirement that deformations of the straight line parametrized by the CFT data appearing in equation 1 are compatible with the constraints of supersymmetry, and have been shown to drastically improve the bounds on the numerical determination of the CFT data.See section A.2 for a brief introduction to the main concepts of Reinforcement Learning (RL). We start by making two approximations. (a) Since equation 1 consists of an infinite number of terms, we truncate it up to a certain number of terms or to all functions with ∆ n ≤ ∆ max . In this case, we consider the first 10 elements of the sum as their scaling dimensions ∆ n are known with very high precision (Grabner et al., 2020) . For a discussion on how this truncation may affect the results or how to mitigate it through the analysis of the tails see Niarchos et al. (2023) . (b) Being equation 1 a functional equation, it has to be verified in a continuous space. In order to make it tractable, we choose a set of 180 testing points in the complex plane where the equation is evaluated. While this does not guarantee the CFT data found to hold for every x ∈ C, it has proven to be effective with the specific points taken from Kàntor et al. (2023) .We now describe the BootSTOP algorithm, first introduced in Kàntor et al. (2022; 2023) , which constitutes the basis for our MultiSTOP approach. In order to apply RL to our framework we need to define a suitable environment with states, actions and, most importantly, a reward signal that guides the agent to find a solution to the equation.States: a state is the current agent's guess of the solution to the truncated version of equation 1, that is the vector of CFT data S t = ∆ 1 , . . . , ∆ 10 , C 2 1 , . . . , C 2 10 ∈ [0, ∆ max ] 10 ×[0, C 2 max ] 10 . For the 1D CFT of interest, it is safe to assume that ∆ max = 10 and C 2 max = 1. We indicate states as (∆, C 2 ). Actions: an action by the agent is the change of one couple of values in the current guess S t . In particular, the agent cycles from n = 1 to 10 and, at each step, selects the new values for (∆ n , C 2 n ) obtaining S t+1 . Note that for this environment the state transitions are deterministic. Rewards: the reward has to guide the agent towards a solution of the conformal bootstrap equation. Evaluating the truncated equation on the aforementioned 180 complex points and with our current guess of the CFT data, we obtain a vector of evaluations E(∆, C 2 ). We want this to be as close as possible to the null vector, in order for equation 1 to be satisfied. The closer the norm ∥E(∆, C 2 )∥ 2 is to 0, the closer the state is to the actual solution and the higher has to be the reward for learning. Possible choices are R 1 = -∥E(∆, C 2 )∥ 2 as in Kàntor et al. (2022) and Kàntor et al. (2023) . In this work, we consider the second formulation. The RL algorithm used to optimize the agent's policy is Soft Actor-Critic (SAC) (Haarnoja et al., 2018) . For the complete algorithm, implementation details and speed-up techniques applied, see appendix A.3.R 2 = 1 ∥E(∆,C 2 )∥2 as inWe now introduce our MultiSTOP, which extends the BootSTOP algorithm by including additional physical constraints. This is an important feature as functional equations or, more generally, DEs or PDEs can feature multiple constraints that must be satisfied, such as initial conditions. Since the physical constraints in equation 2 have proven to be very effective in improving the accuracy of the bounds on the first three values of C 2 n of our model as in Cavaglià et al. (2022) , we want to impose these constraints into our RL framework. To do so, we first notice that equation 2 has the same general form and the same unknowns as our objective 1. Hence, we can apply the same reasoning from equation 1: we evaluate the physical constraints on our current guess (∆, C 2 ), with a reward that should be higher as I 1 and I 2 in equation 2 are closer to zero. With this in mind, we want to include two terms similar to 1 |Ii| , i = 1, 2 into the reward formulation. We have two possibilities:R 1 = 1 ∥ ⃗ E t ( ⃗ ∆, ⃗ C 2 )∥ + w 1 1 |I 1 | + w 2 1 |I 2 | , R 2 = 1 ∥ ⃗ E t ( ⃗ ∆, ⃗ C 2 )∥ + w 1 |I 1 | + w 2 |I 2 | (3)where w 1 and w 2 are weights. After the initial experiments with both forms of reward, with the objective of matching the known bounds for C 2 1 , C 2 2 , C 2 3 with g = 1, we found that R 2 produces better and more stable results, with weights w 1 = 10 4 and w 2 = 10 5 . While R 2 forces all three terms of the denominator to be small to improve the overall reward, when using R 1 the agent was sometimes able to optimize one term independently of the others, leading to poor optimization of the total reward. 
results: 
 We assess the effectiveness of MultiSTOP by studying the 1D CFT described in section 2 in two regimes: (a) weak coupling (g ≤ 1), where we investigate the behavior of C 2 2 , C 2 3 as g → 0 and (b) Strong coupling (g ≥ 1), where we focus on the behavior of C 2 n , n ≥ 4 as g → 4. In each case, the values for ∆ n are given as input as they are known with high precision from Grabner et al. (2020) . We remark the effectiveness of MultiSTOP with respect to the baseline BootSTOP algorithm: when trying to match the known values for C 2 1 , C 3 2 , C 2 3 , integrating the constraints in equation 2 leads to a reduction in relative error from 2x to 10x. This effect has already been observed in similar contexts (Cavaglià et al., 2022; Chester et al., 2022; Chester, 2023) . See appendix B.1 and C for further details from a theoretical and experimental point of view.Since the numerical bounds on C 2 1 from Cavaglià et al. ( 2022) are tight, we provided their middle value as input to the algorithm to reduce the search space and help the agent. In figure 1 we plot the results of the best 25 runs based on reward for the tested values of g and compare them with the available bounds in literature. We notice that the results are less coherent when g → 0, but the values for the coefficients are outside of the bounds in a symmetric way. Each value for C 2 2 is below the expected ones and the opposite happens for C 2 3 . If we consider the sum C 2 2 + C 2 3 , most of the values are in the green regions and therefore acceptable. We found that this is strictly related to the fact that as g → 0, the scaling dimensions ∆ 2 and ∆ 3 converge to the same value of ∆ = 2 (Cavaglià et al., 2022; Grabner et al., 2020) . This is known as degeneracy and affects the ability of the agent to find accurate values of both the coefficients at the same time (appendix B.2). Since the functions F ∆n (x) are analytical with respect to ∆ n , up to terms of O(∆ 2 -∆ 3 ) we can write equation 4, showing that only C 2 2 + C 2 3 can be determined with good precision in the limitEQUATION)To solve this issue, more conformal bootstrap equations like 10 of the same CFT could be integrated. Since at strong coupling all bounds are tight enough, the middle points for C 2 1 , C 2 2 , C 2 3 are given as input to reduce the search space. In this case, we expect a higher precision since our experiments show that the best and average rewards increase monotonically with g and the results are stable with multiple runs in parallel (see appendix A.4 for further results). Figures 2a and 2b show the results for C 2 4 , C 2 5 at strong coupling. We can see that we have a higher precision on C 2 4 , which is in line with the fact that the terms with higher ∆ have a greater influence in the equation and are more optimized. However, the standard error for the best 25 runs seems to increase as g → 4. Again, this is due to degeneracy as the scaling dimensions ∆ 4 , ∆ 5 (together with ∆ 6 , ∆ 8 ) converge to the same value of ∆ = 6 at infinity. Investigating on the sum C 2 4 + C 2 5 + C 2 6 + C 2 8 we found that our mean values are very close to a theoretical result from Ferrero & Meneghelli (2021) ; Ferrero & Meneghelli (2023) ; Ferrero & Meneghelli (2023) , which indicates that the approach is working well (see figure 2c ). We also noticed that if the standard deviation on the sum C 2 4 + C 2 5 + C 2 6 + C 2 8 is calculated using error propagation from the errors on the individual coefficients we obtain figure 6 , with error between 1% and 10%. If we instead calculate the sum C 2 4 + C 2 5 + C 2 6 + C 2 8 for each experiment and calculate statistics directly on these values, the results are surprisingly precise with errors always below 0.1% and the bars of one standard deviation being invisible as in figure 2c . This is another confirmation of the effects of degeneracy: when two or more elements of the sum have similar values of ∆ n , the sums of the corresponding coefficients are more precise than individual values. 
conclusion: 
 In this work, we developed MultiSTOP, an extension to the BootSTOP algorithm that is able impose additional physical constraints into the framework. We experimented on a one-dimensional CFT where some values of ∆ n and C 2 n where known with enough precision to be given as input. At the weak coupling regime, the individual coefficients found were outside of the expected bounds due to the degeneracy problem. We found a similar issue at strong coupling, where the precision on results decreases with g. In both cases, the sum is much more aligned with previous results and has a higher precision. To solve this issue, we could include more equations into the same framework to decouple the coefficients or provide additional physical constraints and information. In the future, MultiSTOP can be applied on PDEs by using linear combinations of parametrized basis functions.where we have introduced the symbol (q) n as(q) n = 1, n = 0, q(q + 1) • • • (q + n -1), n > 0 (7)We also introduce the Bessel functions of the first kind I α asI α (x) = ∞ m=0 1 m!Γ(m + α + 1) x 2 2m+α (8)solutions to the differential equationEQUATION)Starting from Cavaglià et al. (2022) , we now describe how the conformal bootstrap equation is expressed and implemented within our framework. To define equation 1 we begin withEQUATION)whereEQUATION)The individual terms of g defined asEQUATIONwhile the constant C 2 BP S is given byF(g) = 3I 1 (4πg)((2π 2 g 2 + 1)I 1 (4πg) -2gπI 0 (4πg)) 2g 2 π 2 I 2 (4πg) 2 C 2 BP S (g) = F(g) -1 (13)Finally, the functions in equation 1 are defined asEQUATIONThe integral constraints are defined as Constraint 1:EQUATION)whereInt 1 [f ∆n ] = - 1 2 0 (x -1 -x 2 ) f ∆n x 2 ∂ x log(x(1 -x))dx Int 2 [f ∆n ] = 1 2 0 f ∆n x 2 (2x -1)dx (16)and the values RHS i , i = 1, 2 are expressed by where, finally,RHS 1 = B(g) -3C(g) 8B(g) 2 + 7 log 2 - 41 8 (F(g) -1) + log 2 RHS 2 = 1 -F(g) 6 + (2 -F(g)) log 2 + 1 - C(g) 4B(g) 2 (17)EQUATIONand the curvature C(g) can be found in Cavaglià et al. (2022) as an expansion in terms of g.Figure 3 , taken from Cavaglià et al. (2022) , shows the first 10 values of ∆ n as a function of the coupling constant g, which were used in our experiments. 
 
 Appendix: Reinforcement Learning is a particular Machine Learning technique involving two interacting components: an agent and the environment. At each time step, the agent has a state S t , representing the stage of interaction, and does an action A t following a policy π(A t |S t ), which expresses the agent's behavior based on the state. Then, the environment outputs a reward signal R t+1 and a new state S t+1 is reached based on unknown dynamics p(S t+1 , R t+1 |S t , A t ). The objective of the agent is to find the optimal policy to maximize the expected discounted rewardE p [G t ] where G t = ∞ i=t γ i-t R i and γ ∈ [0, 1]is the discount factor which expresses the importance of future rewards. RL techniques often involve studying objects such as the value functionv π (s) = E π,p [G t |S t = s] and the action-value function q π (s, a) = E π,p [G t |S t = s, A t = a]which are measures of the performance of the policy based on the current state and action.The complete algorothm description of Bootstop can be found in algorithm 1.In order to improve the search and the precision of the final solution, two additional techniques are adopted. First, if the agent does not improve the maximum reward ever obtained after a certain number of steps, the Neural Networks of SAC are reset to start the search from scratch and try to find other possible values for the CFT data. Second, after some re-initializations of the networks, the search window is reduced around the previous best guess by a vertain factor to improve the precision. These two processes are repeated iteratively until enough precision is reached.In our experiments, to further improve the accuracy and the performance of the agent, we fixed the numerical values for some parameters. In particular, the scaling dimensions ∆ n , 1 ≤ n ≤ 10 are Figure 4 shows the best and averaged rewards on the best 25 runs as a function of the coupling constant g. As we can see, apart near g = 1 where the setting changes a bit from only C 2 1 to both C 2 i , i = 1, 2, 3 given as input, the reward monotonically increases. We also notice that the best reward is almost indistinguishable from the average of the top 25 with an almost null standard deviation. This shows the stability of the model under multiple runs. Figure 5 shows the experimental results on coefficient C 2 6 which have similar precision to the ones for C 2 5 in figure 2b . Figure 6 shows the results on the sum C 2 4 + C 2 5 + C 2 6 + C 2 8 with the standard deviation calculated with error propagation from the individual ones as described at the end of section 3.2.We briefly describe the theoretical and experimental motivations for the addition of the integral constraints in equation 2. The conformal bootstrap, which leads to the main equation 10, is a powerful method that puts non-perturbative constraints on conformal field theory data (∆, C 2 ). Previous approaches produced only allowed regions for the CFT data (∆, C 2 ). The fact that the inclusion of the additional integral constraints in equation 2 significantly shrinks such regions, allowing in some cases to identify the theory under investigation with great precision, has been already observed in several papers in the physics literature, including the case that we study. We refer to Cavaglià et al. (2022); Chester et al. (2022) ; Chester (2023) for further details. It is therefore important to investigate whether the integral constraints are useful as it has been in many other contexts. A clear example of this is the improved precision between figures 6, 9, and 10 in Cavaglià et al. (2022) . . On a theoretical level, individual components of the conformal bootstrap equation 10 represent physical states of the theory. We refer to degeneracy as the situation where two states i and j have conformal dimensions ∆ i and ∆ j (which we recall are functions of g) which become similar, to the point of being indistinguishable, for a certain value of g. This is the case at weak coupling (g = 0), and this degeneracy of states is responsible for the fact that the terms with n = 2, 3 can no longer be distinguished, so there is a significant loss in precision for the determination of their individual OPE coefficients C 2 2 and C 2 3 . On the other hand, given the structure of the bootstrap equation, it is still possible to determine with good precision their sum C 2 2 + C 2 3 . The fact that precision is lost at weak coupling is visible from the plots in figures 6, 9, 10 of Cavaglià et al. (2022) , which focus on the first three states, while the OPE coefficients for other states are not shown. The fact that the degeneracy is particularly high at weak coupling (much higher than at strong coupling) can be seen from table 5 and figure 1 of Ferrero & Meneghelli (2023) . While their terms in the equation are very similar, different states are physically different and can in theory be distinguished. This can be achieved by adding more conformal bootstrap equations of the same CFT into the MultiSTOP framework, which we leave for future works.To show the improvements of MultiSTOP compared to its baseline BootSTOP also on from an experimental point of view, we conducted experiments with g = 1 after the definitive weights were found to be w 1 = 10 4 , w 2 = 10 5 . We gave as input the conformal dimensions ∆ n and tried to match the known values for C 2 1 , C 2 2 , C 2 3 , defined as the middle point of the available tight bounds from Cavaglià et al. (2022) . In figure 7 we compare the most relevant metrics based on the number of integral constraints applied (the best 10 runs in each case are considered): from "no contstraints", the BootSTOP case, to "one constraint" or "two constraints" with MultiSTOP.• The norm of the crossing equation 1 increases by a factor of two (figure 7a. This is most probably due to the harder optimization problem given by the addition of the integral constraints.• Most importantly, the absolute relative error with respect to the ground truth (figures 7b,7c,7d) decreases when the integral constraints are applied, most notably when both of them are included. The reduction in error goes from a factor of at at least 2 for C 2 2 and C 2 3 to a factor of almost 10 in the case of C 2 1 .We can conclude that MultiSTOP is able to find more accurate solutions to the crossing equation when compared to the baseline method. See appendix A.1 for a more detailed description. 


Decision: Thanks for the great points made by the reviewer. After going through the comments and paper I also think overall the paper is interesting. I expect the authors to address all the questions raised by reviewer in the final revision. Under this condition, this work can be accepted as poster.

Reviews: review: title: title: The authors present an extension of the BootStop algorithm, termed MULTIStop, by adding additional constrains derived from domain experts’ knowledge
main_review: main_review: The authors present an extension of the BootStop algorithm, termed MULTIStop, by adding additional constrains derived from domain experts’ knowledge. The authors use an 1D conformal bootstrap equation to show their method.

I think the paper is well written and make clear the main goals. I am not familiar with the example the authors use of the conformal bootstrap equation. I provide my comments below from an optimization perceptive and from a parameter non-identifiability angle of view. 

(a)	The authors seem to be familiar with the BootStop Algorithm and here they propose their method MULTIStop which includes additional constrains to improve the accuracy of the bounds on their coefficients. I would expect to see at least a comparison with the BootStop algorithm to showcase that indeed the additional constrains are useful.

(b)	The authors choose to work in regimes for which there is literature suggesting what might be good OPE coefficients and scaling dimensions. I think this is the rational thing to do but it seems that when they use those “good initial conditions” the obtained coefficients C_n^2 are still off. Can they please comment on why this is the case?

(c)	The authors in many parts of the paper discuss about degeneracy; in my understanding this has to do with non-identifiability of the systems parameters (the coefficients that give a particular response cannot be determined uniquely –> the systems can reach the same state from different parameters) is this is case?

If yes, then in my opinion showing comparisons in terms of the inputs/parameters/coefficients space is misleading. I would expect so see how the norm (the output) ||E(Δ,C^2)||_2 looks like or how close to zero Equation 1 becomes for the obtained coefficients. If the values of their coefficients produce an error comparable to the prior work or smaller, then their method is successful. Based on the results showing in the manuscript is unclear to me the efficacy of their method.

(d)	As a follow up to my previous comment, I think this is why they observe much smaller errors when they use the sum C_4^2 + … + C_8^2. This might be an effective coefficient of the model and thus it has one-to-one correspondence with the output of interest.

In my opinion, the lack of a comparison between their approach and BootStop and the comparison only in the input space and not also in the output/response/behavior space makes the current paper marginally below acceptance threshold. 

Minor Comment:
Abstract: Check the English of the second sentence.