Speeding up Policy Optimization with Vanishing Hypothesis and Variable Mini-Batch Size
Authors: Tamás Tardi
Gergo Bogacsovics
Andras Hajdu
Venue: Submitted to ICLR 2023 (2023)
Keywords: reinforcement learning
policy optimization
noisy evaluation
variable mini-batch size
vanishing hypothesis
optimal placement of sensors

Abstract: Reinforcement learning-based algorithms have been used extensively in recent years due to their flexible nature, good performance, and the increasing number of said algorithms. However, the largest drawback of these techniques remains unsolved, that is, it usually takes a long time for the agents to learn how to solve a given problem. In this work, we outline a novel method that can be used to drastically reduce the training time of current state-of-the-art algorithms like Proximal Policy Optimization (PPO). We evaluate the performance of this approach in a unique environment where we use reinforcement learning to help with a practical astronomical problem: where to place a fixed number of observatory stations in the Solar System to observe space objects (e.g. asteroids) as permanently as possible. That is, the reward in this scenario corresponds to the total coverage of the trajectories of these objects. We apply noisy evaluation for calculating the reward to speed up the training, which technique has already been efficiently applied in stochastic optimization. Namely, we allow the incorporation of some additional noise in the reward function in the form of a hypothesis term and a varying mini-batch size. However, in order to follow the theoretical guidelines, both of them are forced to vanish during training to let the noise converge to zero. Our experimental results show that using this approach we can reduce the training time remarkably, even by 75%.

Under review as a conference paper at ICLR 2023
SPEEDING UP POLICY OPTIMIZATION WITH VANISH- ING HYPOTHESIS AND VARIABLE MINI-BATCH SIZE
Anonymous authors Paper under double-blind review
ABSTRACT
Reinforcement learning-based algorithms have been used extensively in recent years due to their flexible nature, good performance, and the increasing number of said algorithms. However, the largest drawback of these techniques remains unsolved, that is, it usually takes long time for the agents to learn how to solve a given problem. In this work, we outline novel method that can be used to drasti- cally reduce the training time of current state-of-the-art algorithms like Proximal Policy Optimization (PPO). We evaluate the performance of this approach in a unique environment where we use reinforcement learning to help with practical astronomical problem: where to place a fixed number of observatory stations the Solar System to observe space objects (e.g. asteroids) permanently as pos- sible. That is, the reward in this scenario corresponds to the total coverage of the trajectories of these objects. We apply noisy evaluation for calculating the reward to speed up the training, which technique has already been efficiently applied in stochastic optimization. Namely, we allow the incorporation of some additional noise in the reward function in the form of hypothesis term and a varying mini- batch size. However, in order to follow the theoretical guidelines, both of them are forced to vanish during training to let the noise converge to zero. Our experimental results show that using this approach we can reduce the training time remarkably, even by 75%.
INTRODUCTION
Reinforcement learning (RL) was in the focus of numerous studies in the last few years. As such, plenty of new algorithms have emerged and are continuously surfacing that improve the already existing RL algorithms or combine the traditional methods with modern deep learning techniques, like deep Q-learning (Mnih et al.] 2013), Proximal Policy Optimization (PPO) (Schulman et al.) 2017 Asynchronous Advantage Actor-critic (Mnih et al.) (01 or Soft Actor-Critic (Haarnoja) et al. 2018). Researchers have used these algorithms with great success, solving various challeng- ing problems, panngngg from using RL to play table games like Go (Silver et al., 2016 2017), playing Massively Multiplayer Online (MMO) games (Suarez et al., 2019), J, to solving complex problems in train a single RI. agent in a way that it can be used to solve different tasks, indicating the high gener- alization capabilities of these types of algorithms. However, one of the biggest problems these algorithms - especially on-policy methods still remains: the optimization process, and hence the training of the agent, takes really long time (Yarats al., 202T Yu 2018). This is especially true in the case of policy-based algorithms which are notorious for being sample inefficient (Bastani 2020). Although PPO, which is also policy-based method, has better sample complexity (Schul- man et al. 2017) than the original policy gradient algorithm (Mnih et al. (016a), it still suffers from this phenomenon. Despite this drawback, PPO still widely used due to its convergence to least local optimum, in contrast to value- based methods, where convergence is not necessarily guaran- teed, which phenomenon is referred to as the "deadly triad" by /Sutton & Barto (2018). this work, show that convergence and achieving the same cumulative rewards by treating the loss as an energy function and incorporating some noises in the optimization process.
robotics (Plappert al. 2018). Recent research (Reed et al. 2022 has also shown that
remarkably reduce the training time of PPO agent while still preserving its
In real-world applications it is quite common that the energy function cannot be evaluated precisely because of certain noise corruptions. If the noise is too large then meaningful optimization cannot

---

Under review as a conference paper at ICLR 2023
take place. On the other hand, we also inject some CorrUption deliberately to speed up the optimization process via applying noisy evaluation (Gelfand & Mitter 1989). The original idea was to perform only rough evaluation which approach can save lot of computational time if the energy function is complex, and its proper evaluation is laborious. It has already been shown that this approach is valid only in the case when the corruption vanishes as the search progresses to be able to keep up the convergence behavior of the original (uncorrupted) model. Specifically, for simulated annealing (SA) Gelfand Mitter (1989) have shown in what extent the noise should converge to 0 to preserve the search efficiency of the uncorrupted case.
In this work, we focus on policy-bane based approaches and further refine the noisy evaluation idea to integrate it in the RL framework in different ways. The first integration is possible only in such applications - including ours -, where a batch size is used during the gradient descent or ascent update. This is true for policy gradient algorithms which measure the goodness of a policy π as the cumulative reward achieved in a set time frame via r(x) "i ar s')ls st, a ae s' stt1
for each timestep E I in the given time frame and use a gradient estimate to update x. In such cases we can apply the well-known mini-batch approach with including only a part of these timesteps in one update cycle. However, opposite to the traditional mini-batch approach that keeps the batch size constant, we also consider as noise w.r.t. a full-batch validation. Consequently, we also follow the above requirement to vanish the noise, which can be achieved here with continuously increasing the batch size during training till reaching the total number of observations in the episode.
Beyond considering variable mini-batch size, we also include hypothesis term h(s') ( for every sub- sequent state s' in the reward function r(s, a, s') as a direct feedback for our agent. This procedure can also be interpreted addition is to speed up the training process and improve the sample efficiency, as we believe that an appropriately formulated hypothesís can direct the search to the optimal location and hence make the convergence faster. However, we can never be sure that our hypothesis indeed has this behaviour, since we can be mistaken with it. Moreover, to take the above considerations also into account, the noise term should tend to 0 to keep up the convergence behavior of the original RL model, Thus, the hypothesis term will be added in such a way that it will vanish as the search progresses. As we will see, both the variable mini-bated and the vanishing hypothesis term are able to speed up the training process, however their simultaneous integration will improve it further.
adding noise in the light of the above summary. Our intention with this
The rest of the paper is organized as follows. In section 2 we properly exhibit how the hypothesis term and the variable mini-batch can be integrated a reward function considered in policy op- timization. Our special application domain that originally motivated the research is introduced in section 3 Then, section 4 presents the details of the proper implementation. In section S we show our experimental results suggesting a meaningful speed up in finding the solution for our problem. Finally, some conclusions are drawn in section 6
2
METHODOLOGY
Policy gradient methods like PPO use an estimation of the gradient to update their policy i param- eterized by set of weights 0 (mo for short) so that the expected cumulative reward
Exo Σ relre r(s, a. s'), = $t, = at, s' s141
increases. To this end, they use estimation similar to the following expression (Schulman et al. 2017) for each timestep and perform gradient ascent:
g Ra log (,(a,,,)
where Ar is the estimation of the advantage. PPO also uses a clipped surrogate objective to prevent large updates during the optimization process. Although this facilitates training to a certain degree, making PPO converge faster than the vanilla policy gradient method (Mnih et al., 2016a) and achieve higher scores as shown in Schulman al. (0077). the training still takes long time. What is even more important is the fact that in the PPO paper it can also be observed that in several environments the agents achieved really low scores in significant chunk of the total training time before finally
2

---

Under review as a conference paper at ICLR 2023
converging to optimum. This raises the question whether this part of the training could be further improved to reduce the total time required for training the agent.
In our work, we focus on reducing this time by incorporating some additional noise and performing noisy evaluation during the training process, which technique has been shown to work well with regular stochastic optimization problems. Namely, for SA it has been shown in [elfand & Mitter 1989) that a normally distributed noise with mean 0 and variance (o()))² > 0 in the k-th iteration let the search converge the globally optimal solution with probability if
o (k) =0 (T(())
Besides the temperature, an alternative constraint has also been formulated in Gutjahr Pflug (196) for the same purpose for the number of iterations as
(k) O (k-9) with some >>> 1.
(4)
Though the conditions and require the variance of the noise to decrease along with the progress of the exploration, it is obvious that the noise should also vanish to preserve convergence.
very attractive expectation towards any optimization methods is that the applied heuristics that is used to speed up the search should guarantee the original convergence properties; esp. to preserve convergence to the global optimum. Though numerous very efficient strategies have been proposed to improve exploration, unfortunately, the theoretical preservation of the convergence has been rarely proved and even in such cases strict limitations had to be applied. For example, even for the original SA algorithm it has been shown (Geman & Geman 1984) that only the logarithmic cooling schedule can preserve global convergence, while in practice the linear, and the exponential schemes became very popular. Regarding our current study, we can have very similar claims for the PPO algorithm. As relatively new approach, in spite of its very popular and well-applicable nature, the theoretical foundations have already just begun recently (Bhandari & Russo 2019; Agarwal al., 202T). Thus, regarding the above discussion, we will follow the theoretical suggestions for noisy evaluation to incorporate our speed-up considerations as noises in the PPO algorithm. Accordingly, they should vanish by the end of the training to fulfill (3) and (4)
2.1 VARIABLE MINI-BATCH SIZE
The first approach that we have used to incorporate noise in the training process relies on the number of samples used in the rollout during the gradient update. For this to work, the most important requirement is to have a batch of samples to work with. Therefore, we will use the rollout itself and E, from (2) which is the average over batch of samples, but instead of defining a static, fixed batch size K, we will start from a small K and gradually increase it up until the total number of observations N.
The batch size K can be changed with different dynamics for the k-th iteration step. In Töth et al. 2020 it has been proved that the variance of the noise coming from this sampling should fulfill
(k) ( e)*,
0<<<1
(5)
using SA to preserve convergence. Accordingly, it was also derived that
2 man
(6)
(N
1)o (k)
2
should hold for the minimal batch size at the k-th iteration step in SA, where maa is the worst-case maximum value of the whole population standard deviation.
The trend to increase K depends on the cooling profile selected in SA to calculate the temperature (Tk) for the k-th iteration step in The linear cooling schedule
(T()) 7,)) ak, 0<0 a
leads to the requirement
(k) k ~ (T(0) - -ak) (1 ()=, 05a51,0<e<>
+)²,
(8)
3

---

Under review as a conference paper at ICLR 2023
by applying 5 and also to
No mer
k=
(9)
(N -)(T (1
c)2k
2k
0 maz
by
6 Similarly, for the exponential schedule
T = T a with 0 S a 1
(10)
have
K 2 (1)) a² c)* with sasl, and 0 < 1
(11)
and
No 1) o c)) + o o
K
(12)
meet the requirements.
Similarly to various SA applications, we have found the trend 12 corresponding to the exponential cooling schedule the most efficient in our experiments adjust K. To incorporate the variable mini-
batch size, we slightly modify the loss function described in Schulman et al. (2017) and arrive at the following formula for the loss function L(0):
L(0)
Σ min (0) cip
(13)
where (0) the probability ratio calculated between the current (TT) and the old (Te.. policies, and is defined as in the original paper as:
(T(()((:)) R2
(14)
To.as (at (8))
2.2 HYPOTHESIS AS NOISE
The second approach that have realized is using hypothesis noise. To this end, have decided on a fixed hypothesis h S - IR, where the state space that we have incorporated in the reward function r(s, a, s') in the following way. For each timestep within the k-th iteration, when the agent used an action in state s; and arrived at the next state 8141. it received the reward r; = r(tt, at, 8141) through the formula
(k)
(k) h M8841),
r(8,, Ut, 8141)
K
K
(15)
max(h(s)) sES
max(h(s) #ES
where Fr denotes the traditional reinforcement learning reward that the agent would receive without taking any hypothesis into account, h(s') is the reward for arriving in the next state s' 81+1 according to our hypothesis h, and max.es (h(s)) is the theoretically obtainable maximal reward (k) at timestep. The role of the trade-off parameter oj (max)es(hs))) final reward r(s, a, s') between the traditional and hypothesis-based rewards. Moreover, with this selection of the trade-off we incorporate the hypothesis as a vanishing noise which follows the same trend the variable mini-batee bini-batch size. In this way preserve the original convergence behavior of PPO and by using the trade-off parameter to weight both types of rewards we also make sure that the final reward r(s, a, s') is the same scale the traditional reward f to make them comparable.
(15) to balance
The adjustments of the parameters discussed throughout this section are task-dependent and will be given properly in section 4
APPLICATION DOMAIN
Our practical task motivating the research focuses optimal sensor placement. The goal is to place sensors spatial domain in such a way that we achieve the maximum result according to particular reward function. This task has received great interest in recent years RL problem.
4

---

Under review as a conference paper at ICLR 2023
Research has been carried out wide variety of topics, such placing sensors in greenhouses
to measure humidity and temperature (Uyeh et al.) 202T), but there have also been publications
generalizing the idea (Wang al., 2019). Perhaps the greatest scientific acclaim of all, however, came from the Google Brain team, who Mirhoseini et al. (2017); [Goldie & Mirhoseini (2020) presented RL algorithm that solved the chip placement dilemma significantly faster and better than human experts.
By studying the approaches that set out to solve this type of task, we can observe broadly consistent methodology for dealing effectively with the optimal placement problem. The solution starts with the transformation of the problem into a Markov decision process. Typically, the environment is thought of as a discrete mapping of the domain, so the problem can be formulated in general terms finite number of possible locations to provide optimal
as follows: the goal is to place M sensors coverage. In these problems, the agent is the entity that chooses the actions that usually place or move a sensor. The reward function varies from problem to problem.
Figure I: Observing space objects in the Solar System from the Earth.
We investigate the applicability of RL in a time-varying continuous model with geometric coverings. Specifically, it strives to find out how to optimally position observatories on the surface of the Earth, so they provide the maximum possible coverage for monitoring the celestial bodies of the Solar System for a chosen time interval (see Figure D]. Specific subtasks may relate to the fast detection and observation of the trajectories of space objects e.g. asteroids unidentified ones. Generally speaking, the problem at hand is so complex that with theoretical models it is virtually impossible to handle. However, the presented RL approach is able address it and can be extended to even more complex problems. One simple extension can be e.g. to let the observatories to be installed also on other planets/moons of the Solar System.
To achieve optimal placement, the software uses the methodology described above. In our case, this means that each step a location is chosen for an observatory. It is achieved by generating two values from 1], where the first represents latitude mapped to [-90, 90]. and the second means longitude mapped [-180, 180). Although giving partial rewards after each step seemed reasonable, it did not help the learning process. When all the observatories placed, rewards are calculated based on coverage earned, and the episode ends.
The solution was implemented in Unity Haas 2014), using the open-source Pytorch-based Unity ML-Agents Toolkit (Juliani et al. 2018). The main advantage of this framework is that it can deal exceptionally well with our geometry- based problem, while also having case of implementation through the ML-Agents toolkit. On top of that, it provides a well-optimized pre- pre-implemented ver- sion of PPO, the algorithm most often used to solve positioning problems recently.
5

---

Under review as a conference paper at ICLR 2023
4 IMPLEMENTATION
4.1 DATA ACQUISITION
To find the optimal sensor placement, essential to have the location of the celestial objects of the Solar System at different time instants to handle occlusions. For this task, the DE 421 planetary and lunar ephemeris (Folkner et al. 2009) was used, which contains estimates for the orbits of all objects from 1900 to 2050; our examples will fall in this time interval.
In ephemerides, the positions of the celestial bodies are typically described by three values, Alpha, Delta (angles), and distance from given center (origin). In astronomy, different reference frames are used for different problems. The frame we have chosen for this task the International Terres- trial Reference System (ITRS) (Petit & Luzum 2010). ITRS is coordinate system whose origin the Earth's center of mass, and as the Earth rotates, so does the coordinate system itself. It also identifies the location of points with 3D coordinates (X, Y, Z triplets), with the XY plane going through the Earth's ' equator and the Z-axis pointing towards the North Pole, which makes it ideal for working with Unity.
The coordinates of the ephemerides represent the actual positions of the celestial bodies in space. However, the task requires information about where given celestial body appears in the sky when observed from Earth. This conversion was done with the Skyfield Python package Rhodes] (2019). designed to easily obtain the positions of different planets and celestial bodies. The procedure to convert the values obtained from DE421 to ITRS is as follows:
Selecting the Earth's s center of mass as the observation point.
Selecting target observe.
Generating the astrometric position, which considers the effect of light travel time.
Generating the apparent position, which applies additional effects that might affect where celestial objects appear in the sky.
Converting the position to ITRS.
4.2
LEARNING PROCESS
For the implementation to address multiple problems, a flexible framework has been developed that can be parameterized with different data. These parameters include information about the observa- tories and celestial bodies, which were collected into a CSV file with a structure of:
1st row: Angle of View (AOV) of the observatories be placed. For example, '30,20,10* means that three observatories are placed, with AOVs of 30%, 20 and 10", respectivell.
2nd row: Diameter of bodies to be observed. As with the AOV, the number of values displayed is the same as the number of celestial bodies being monitored. Celestial bodies are represented by a regular sphere.
3rd row: The importance value of the objects to be observed (used for reward calculation). For example, if we rewarded closer objects to the Earth more, we could have an importance value based on distance.
4th and the following rows: Observations in ITRS reference frame in X Y Z' format. A row represents an observation at a time instant, with a column describing the position of the celestial body. For example, on January 2000, the position of the Sun after scaling is: 015.3266911992666
At the beginning, the only entity in the environment is the Earth, since it is the only object, whose position will not change during the learning process (no need to move or rotate it thanks to the reference system). Solving the task starts with processing the parameters. We read the problem to be solved and save the data related to the observatories and celestial bodies. Then, we create the corresponding spheres representing these objects in the environment at the appropriate size. The learning process is as described above, with a new observatory being placed at each step. After all the observatories are placed, the reward fr in (15) is calculated, and the episode ends. The whole process can be described more formally by Algorithm

---

21: end for Under review as a conference paper at ICLR 2023
Algorithm
Optimal placement of observatories
1:
Input
2: genPos
generated positions
3: obs
observatories to be placed
4: planets
planets to observed
5: maxPoints
maximum achievable points based importance values
6: Output
7:
reward earned reward
8: reward 6-
9:
for each gp E gemPas do
10: distinetPlanetsseen f- 0
11:
position planets according to gp
12:
for each E obs do
planets CarreetsInCoee.. for each E planetsInnCone do
all planets that inside o.angle cone
14:
15:
send ray from o to if ray collides first with p and Plantscttlnnettseeen pp=1 WisstinctPlonetsSeer
16:
=0 then
disttnet Up
18: end if
end for
20: reward+
22:
end for
23:
return reward/(len(geng * max
The way done by going through all the generated positions from the CSV file and checking the number of distinct objects seen by that placement. A celestial body is considered seen, if it is within the observatory's AOV, and is not obscured by another object. The reward earned in a given generated position is the sum of the importance values assigned to the objects seen. The sum of the rewards earned by the placement divided by the number of generated positions and the maximum achievable points, it is always less than 1.0 (Unity ML-Agents best practice).
Our main example considers placing Al = 3 observatories, each with an AOV of 30% trying to find the optimal placement for the time interval from 01.01.2023 to 12.12.203. with four daily samples, at am, am, 12 am, and 6 pm (N = 15 992). The main objective is to monitor the planets of our Solar System plus Pluto, the Sun, and the Moon, each with an importance value of This way the maximum achievable point umarrPoing is 10.
When noise is considered the reward is simply given the agent. Meanwhile, when seeking to speed up the training by using a hypothesis, the reward is further refined by (15). During the development, several hypotheses were tested, most of which were based on knowledge of the Solar System. It is known that the Earth's 's orbit around the Sun defines a plane 23.5 from the equator, called the ecliptic. Since the shape of the Solar System is more similar to that of flattened disk, the planets move more or less in this plane. The orbits of these planets are at most 3 to the ecliptic, except for Mercury, which is 70, and Pluto, which classified as dwarf planet, being at an angle of 17 Using this knowledge, a hypothesis can be constructed that rewards placements that are not too far off the ecliptic.
We also know that for observatory to object, it has be within cone defined by its AOV. Therefore, the closer the observatories are to each other, the higher the chance that their cones will intersect. We specify a minimum distance longitude and if the placement meets the condition, we can reward it.
minimum difference in either latitude or
The variable mini-batch approach described in section 2.1 does not change how rewards calcu- lated but uses a subset of generated positions instead as given in (13). Digureeieppin how the batch size K changes during the training when using the parameters from Table In the beginning, the

---

Under review as a conference paper at ICLR 2023
algorithm will only get rewards based the chosen 64 samples, but it slowly increases 15992 according to (12) as discussed in section 2
14000
12000
Mini- Mini-batch bil-batchhhc size
10000
6000
4000
2000
Episode number (k)
Figure 2: Mini-batch sizes K used for the k-th episodes.
Table 1: Parameters used for adjusting the mini-batch size.
PARAMETER
VALUE
0.0000009
Tma
40
T-0)
0.99997
N
992
5 EXPERIMENTAL RESULTS
We have tried various heuristics, but in the end. h($(+1) was calculated in 15) as a combination of two different terms. The first gives linearly decreasing reward for each observatory whose latitude value was below 20 with maximizing at 08, while the second rewards the algorithm if the longitude or latitude of pair of observatories differs by least 50' Intuitively, we have formulated that expect the observatories close to the equator with a remarkably large distance from each other. For our experiments with M = 3 observatories we have fixed the maximum heuristics reward max.es(h(s)) = (0.5 + 0.9) 4.2 in (15) empirically which is similar scale fz.
Using traditional PPO, the maximum reward achieved was 0.4568704, which means that, on aver- age, the observatories can monitor 4.5 celestial objects out of the ten chosen ones. Figure 3 plots how the rewards earned evolved during training. It can be observed that using heuristics the algo- rithm reaches the original PPO maximum slightly faster. The variable mini-batch size, however, shows a significant improvement in training speed, which can be further escalated by combining more precisely indicates, we could speed up the learning pro-
with the hypothesis noise. As Table 2 cess by almost 75% in this way. The entries in the table were calculated by taking the elapsed times when the different approaches reached the maximum reward earned by the original PPO algorithm.
Table 2: Learning speed increments according to incorporating variable mini-batch and hypothesis.
Method
Speed increase (%)
Hypothesis Mini-batch
0.686
70.409
Hypothesis and batch
74.219
8

---

Under review as a conference paper at ICLR 2023
014
01.4
Noware carned (rl
RPO
PPO hypothesz
PPO with hypothese
PPO minitachh
PPO with manitatch
PPO with hypothess and babch
PPO with hypothoss and minibatch
0.3
0.
spent learning (hours)
spent learning (hours)
(a) 2 hours training.
(b) hours training.
Figure 3: Comparison of the rewards earned (10-years period).
Using the hypothesis term solely did not increase the training speed remarkably in our above exam- ple. However, for harder task including 50-years time period with capped maximum amount of steps and decreasing learning rate, it apparently outperformed the simple PPO. As depicted in Fig- ure a) the learning process reached the maximum reward with 47.0499.9999 17.099% faster using the heuristics term. Figure 4bb demonstrates also the optimal locations found for the 50-years example.
0.05
Reward esmmed Ir!
0.500
0.23
0.10
0.05
with typothess
0.000
Time spent on lcarning (hours)
(a) Speeding up learning with adding heuristics.
(b) Optimal locations of 3 observatories.
Figure 4: An example for 50-years observation period.
CONCLUSIONS
In this work, we have introduced two new approaches, a heuristics term and variable mini-batch size, to speed up the PPO algorithm in RL. They have been incorporated as noises that vanish as the learning progresses to keep up the original convergence characteristics. remarkable gain in the training time in experimental analysis focusing finding optimal sensor placement to monitor space objects. Both the approaches (esp. the variable mini- batch) were found to be efficient with additional gain when they were combined. As further step, plan to let the sensors be placed a more flexible way to other types of locations, which extension expected to induce further modifications current solution.
have demonstrated

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alekh Agarwal, Sham M. Kakade, Jason D. Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1-76, 2021.
html
Osbert Bastani.
Sample complexity of estimating the policy gradient for nearly deterministic dynam- ical systems. In International Conference on Artificial Intelligence and Statistics, pp. 3868. PMLR, 2020.
Jalaj Bhandari and Daniel Russo. Global optimality guarantees for policy gradient methods. CoRR, /es//996.017880 2019. |ttp:::///axxx wwt.oorgg//oo.ooooo.ooo..../..st abs abs/1906. 01786
W. Folkner, James Williams, and Dale Boggs. The planetary and lunar ephemeris de 421. Inter- planetary Network Progress Report, 42-178:1-, 08 2009.
B. Gelfand and K. Mitter. Simulated annealing with noisy or imprecise energy measurements. Journal of Optimization Theory and Applications, 62(1):49 62. 1989. ISSN 1573-2878. doi: 10.1007/BF00999929.
Stuart Geman and Donald Geman. Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. IEEE Transactions Pattern Analysis and Machine Intelligence, PAMI-6
(6):721-741, 1984. doi: 0..11099PAAMI.119444777775
Anna Goldie and Azalia Mirhoseini. Placement optimization with deep reinforcement learning- In Proceedings of the 2020 International Symposium Physical Design, ISPD '20, pp. 3-7, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 978 155377011. doi:
10.1145/3372990.338881744
org. 1145 3378174
Walter J. Gutjahr and Georg Ch. Pflug. Simulated annealing for noisy cost functions. Journal of Global Optimization, 8(1):1-13, 1996. doi: 10.1007/6000229298.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. CoRR, hts/1801.00 2901,99 1801. C1290
2018. URL
John Haas. A history of the unity game engine. 2014.
Arthur Juliani, Vincent-Pierre Berges, Ervin Teng, Andrew Cohen, Jonathan Harper, Chris Elion, Chris Goy, Yuan Gao, Hunter Henry, Marwan Mattar, and Danny Lange. Unity: A general platform for intelligent agents, 2018. /ttts:/tt:///arrtt
orw. wwwas//s80. 1809.02627
Azalia Mirhoseini, Hieu Pham, Quoc V. Le, Benoit Steiner, Rasmus Larsen, Yuefeng Zhou, Naveen Kumar, Mohammad Norouzi, Samy Bengio, and Jeff Dean. Device placement optimization with reinforcement learning, 2017. URL https: /ttls://arxiva wtw.aaxvivvv...a///aaa//aaaaaava http:/aaxxvv...aaaaa/
1706. 04972
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Joannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013. URL/attp://axxx wwtp/aaa////a//eee
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937. PMLR, 2016a.
Volodymyr Mnih, Adrià Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. CoRR, abs/1602.01783, 2016b. URL http:://aaatt wwtpoo/gg//////eeeeeeeeeee 00783118388133
Gérard Petit and Brian Luzum. Iers conventions (2010). Tech. Rep. DTIC Document, 36:31-42, 01 2010.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej. Peter Welinder, Vikash Kumar, and Wojeiech Zaremba. Multi-goal reinforcement learning: Challenging robotics environments and request for research. CoRR, abs/1 802,09944, 2018. Utth:t/:://arx wtwp://wwwwwwo/////////2// 8028804...44.
10

---

Under review as a conference paper at ICLR 2023
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXtv:2205.061775 2022.
Brandon Rhodes. Skyfield: High precision research-grade positions for planets and Earth satellites generator. Astrophysics Source Code Library, record ascl: 1907 .024,002,04 July 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1 00070,6477 2017. URL http://arxiv.org/ /bs/
1707. 06347
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 4897)44944 2016.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-355 2017.
Joseph Suarez, Yilun Du, Phillip Isola, and Igor Mordatch. Neural MMO: A massively multiagent game environment for training and evaluating intelligent agents. CoRR, abs/1903.00784, 2019.
URL http:://axii.. org/ abs/ 110./.993... 00784
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
János Tóth, Henrietta Tomán, and András Hajdu. Efficient sampling-ban baspi energy function evalu- ation for ensemble optimization using simulated annealing. Pattern Recognition, 107:107510, 2020. ISSN 0031-3203. doi: ott.o//ccc.c.cceeeaaaaa////////a
URL https:
www. sc acieiieie inciicediiic rect. com
Daniel Dooyum Uyeh, Blessing Itoro Bassey, Rammohan Mallipeddi, Senorpe Asem-Hiablie, Maryleen Amaizu, Seungmin Woo, Yushin Ha, and Tusan Park. A reinforcement learning ap- proach for optimal placement of sensors in protected cultivation systems. IEEE Access, 9:100781- 100800, 2021. doi: 10.1 A09//ACCESS.22213000600
Zhi Wang, Han-Xiong Li, and Chunlin Chen. Reinforcement learning based optimal sensor place- ment for spatiotemporal modeling. IEEE Transactions on Cybernetics, PP, 03 2019. doi:
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improv- ing sample efficiency in model-free reinforcement learning from images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 10674-10681, 2021.
Yang Yu. Towards sample efficient reinforcement learning. In ICCAI, pp. 5739-5743, 2018.


Reviews:
[{"review_id": "teRBpEcwFz", "reviewer": "Reviewer_Cmm4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose two heuristics for speeding up policy gradient approaches and demonstrate it with their application in observatory placement (a coverage problem). The first heuristic is a mini-batch size schedule, where they start with small batches. The second heuristic is injecting noise in the reward function.", "summary_of_the_review": "While the mini-batch scheduling idea could be useful, the proposed theoretical motivation is unclear, and the experiments would have to be greatly expanded. ", "strength_and_weaknesses": "Strengths:\n- Adjusting the mini-batch size is an interesting idea, and it has been proposed for (at least) supervised learning before.\n\nWeaknesses:\n- The overall mathematical rigor could be improved, and the writing is sometimes confusing.\n- To test it only on your own application (and without error bars - did you make multiple runs?) makes it impossible to know if it will generalize to other problems. As a method paper, I would expect it to at least test on some gym environments (c.f. the PPO paper you cite).\n- In more detail: I do not understand the motivation (or naming) of the second heuristic of injecting noise into the reward function that they call \"hypothesis\". The connection of both approaches to the temperature noise in simulated annealing is also not obvious. What I get is that the authors argue that since a decreasing noise sequence does not *impede* converge there, it will not here either. However, that does not mean it will *improve* convergence, and I don't understand how this is supposed  to work. In SA the noise is to find a better global minima, I do not think injecting it into the policy gradient will work as well. Is this what you are trying to do?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QV-3zL3AOiw", "reviewer": "Reviewer_RTtu", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose adding noise to the reward function and a variable batch size to improve PPO.  They test their method on a simulated astronomy problem.\n", "summary_of_the_review": "This paper needs some work before it is ready for a conference like ICLR.  I encourage the authors to bring more ML/RL knowledge to the project, and work on improving the writing, their knowledge of the related work, the experiments, and the overall contribution.", "strength_and_weaknesses": "**A minor weakness:** “incorporating some additional noise and performing noisy evaluation during the training process”.  In RL, this could mean many things, such as adding noise to the action selection or the policy parameters, noise to observations, etc.  From the abstract, we know that the authors mean to add noise to the reward function, but except for the phrasing in the abstract, this fact is not clear from sections 1-2 (there are plenty of phrases like the one above, which talk about adding noise, but exactly what that means is not made clear except for in the abstract).\n\n**Major weaknesses:**\n- T is first used in (3), but is undefined.  It is then used heavily throughout 2.1, but since it has not been defined, I cannot understand 2.1.\n- “The second approach that we have realized is using a hypothesis as noise. To this end, we have decided on a fixed hypothesis h : S → R, where S is the state space that we have incorporated in the reward function r(s, a, s′) in the following way.”\n    - small clarity problem: the hypothesis function and its codomain (the reals) are stated in the sentence above, but why is the codomain the reals?  What do the reals represent in this case?  It turns out the answer is that “h(s′) is the reward for arriving in the next state”, but this second quote is quite far below the quote above.  This makes it confusing and difficult to read.\n    - larger clarity problem: h(s) maps states to rewards, but the details are unclear in Section 2.2.  Is it similar to a learned critic (predicting reward instead of return)?  Is it the true value of the next reward given the state and action?  Is it the expected value of the reward, given the state and policy?  Something else?  It is not clear from 2.2.\n- There is no related work section.  The authors do not seem to have a firm understanding of the relevant related work in RL.  For example, the quote “converge faster than the vanilla policy gradient method (Mnih et al., 2016a)” may hint at a lack of breadth and depth knowledge about policy gradient methods.  There is a long history of policy gradient algorithms going back to Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning (Williams, 1992) that I suspect the authors are unaware of.\n- The experiments are not sufficient to support the authors’ claims.  There are no error bars, there appears to be only one trial per learning curve, and there is only one environment.  Unless if I missed it, there is no mention of hyperparameters or hyperparameter selection.\n\n**Comments:**\n- It seems like it might be trivial to formulate this problem as a supervised learning problem instead of an RL problem.  If that’s the case, treating this as an RL problem (without a *very* good justification) is not a great approach to the problem.\n- One might argue that this is more of an applied paper.  Based on the title and abstract, I would argue that that is not the case/intent.  However, even for an application (astronomy) paper, most of the weaknesses above are still relevant.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Ng01QHBIe7", "reviewer": "Reviewer_qi1M", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper discusses the application of simulated annealing to the automatic tuning of hyperparameters in PPO, specifically applied to batch size and reward shaping (which they call a 'hypothesis term').\n\nEvaluating on a sensor placing problem, which involves determining the optimal placement of observatory stations for the purposes of observing space objects, they see that they can converge to the optimal solution quicker than a standard PPO approach.", "summary_of_the_review": "Overall the submission seems to be low quality, and doesn't merit acceptance. I think significant work needs to be done to improve this submission, and have given some guidance as to how to improve the paper along these lines.", "strength_and_weaknesses": "Strengths:\n* The problem setting is interesting, and is a valid use case for RL.\n\nWeaknesses:\n* The paper has a significant number of shortcomings:\n    * The idea of automatically tuning hyperparameters is not novel in RL. Indeed the entire sub-field of AutoRL [1] places emphasis on this, and even automatically discovering similar batch-size strategies to the ones here [2]. Furthermore, their notion of having a 'hypothesis term' designed to guide the reward is just a type of reward shaping and curriculum (e.g., in their case encouraging elliptical solutions). Again, no reference seems to be made to this fact.\n    * There is no comparison of their method on standard benchmark suites, and no comparison to alternative scheduling methods.\n    * It appears there's only a single seed used in experiments.\n\n[1] Automated Reinforcement Learning (AutoRL): A Survey and Open Problems, Parker-Holder et al., JAIR2022\n\n[2] Bayesian Generational Population-Based Training, Wan et al., AutoML2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7POzBMMXqq", "reviewer": "Reviewer_XSaN", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a novel method to drastically reduce training time for on-policy algorithms such as PPO. The formal method involves an exponential schedule for batch size and adding noise to reward. Their method is evaluated on an astronomical problem.", "summary_of_the_review": "The paper has fundamental flaws and needs significant revision. If the authors can show the generality of their method on different RL agents and environments, I am willing to raise to weak reject.", "strength_and_weaknesses": "Strengths:\n- Organized\n\nWeaknesses:\n- Writing is not formal (i.e. \"takes a really long time\", \"remarkably reduce training time\" in introduction)\n- Evaluation is limited. To show the effectiveness of a method, must be applied to multiple environments (such as DMC control suite).\n- Does your method also apply for different agents (outside of PPO)?\n- Changing batch size/making it variable is not new. It has shown to be effective in supervised learning, so why not RL?\n- Much of the evaluation, such as data acquisition, and learning process, should be moved to appendix\n- Where is related work? Noise + variable batch size is not new.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject