Using semantic distance for diverse and sample efficient genetic programming
Authors: David Saxton
Chrisantha Fernando
Venue: Submitted to ICLR 2023 (2023)
Keywords: genetic programming
meta learning

Abstract: Evolutionary methods, such as genetic programming, search a space of programs to find those with good fitness, often using mutations that manipulate the syntactic structure of programs without being aware of how they affect the semantics. For applications where the semantics are highly sensitive to small syntactic mutations, or where fitness evaluation is expensive, this can make learning programs intractable.

We introduce a mutation operator that yields mutated programs that are semantically far from previously evaluated programs, while still being semantically close to their parent. For function regression, this leads to an algorithm that is one to two orders of magnitude more sample efficient than other gradient-free methods, such as genetic programming, or learning the weights of a neural network using evolutionary strategies.

We show how this method can be applied to learning architecture-specific and general purpose neural network optimizers, and to reinforcement learning loss functions. The learnt components are simple, interpretable, high performance, and contain novel features not seen before such as weight growth.

USING SEMANTIC DISTANCE FOR DIVERSE AND SAM- PLE EFFICIENT GENETIC PROGRAMMING
Anonymous authors Paper under double-blind review
ABSTRACT
Evolutionary methods, such as genetic programming, search space of programs find those with good fitness, often using mutations that manipulate the syntactic structure of programs without being aware of how they affect the semantics. For ap- plications where the semanties are highly sensitive to small syntactic mutations, or where fitness evaluation is expensive, this can make learning programs intractable.
We introduce mutation operator that yields mutated programs that are semantically far from previously evaluated programs, while still being semantically close to their parent. For function regression, this leads an algorithm that is one two orders of magnitude more sample efficient than other gradient-fres methods, such as genetic programming, or learning the weights of neural network using evolutionary strategies.
We show how this method can be applied learning architecture-specifie and general purpose neural network optimizers, and to reinforcement learning loss functions. The learnt components are simple, interpretable, high performance, and contain novel features not seen before such as weight growth.
INTRODUCTION
A program is a discrete structure, such as a symbolic expression, or sequence of instructions. Given a fitness function, genetic programming is an evolutionary algorithm for searching over the space of programs to find one with high fitness. An example of program is a machine learning (ML) component such as an optimizer or loss function, specified using calls to a library such as TensorFlow or JAX. They contain an ordered list of nodes, where each node has an operator (such addition or multiplication), and a list of inputs where each input is either an external input, a constant, or previous output. These programs thus compactly describe mathematical function that can be executed on hardware, or can be further transformed in non-trivial ways, such as being differentiated.
Can learn such programs? An important lesson from the history of ML research is that methods that scale and leverage computation will typically eventually outperform hand-designed priors and heuristics given enough compute (Sutton) 2019). The lesson of replacing hand-designed with learnt features has often been applied to learning neural networks using gradient descent. Parts of the model such as the optimizer or loss function are important for controlling aspects such how fast the model trains, but typically have less useful gradient information available to train (Metz et al. 2019) and are harder to specify using parameterized functions.
In this paper we hypothesize that one issue with learning via evolution is that mutation operators that naively manipulate the syntactic structure of program can be highly sample inefficient, due to proposing mutations that are either too close to a previously evaluated program, or too far from the parent of the mutation (and thus likely to be deleterious). We introduce mutation operator that uses semantic distance information to mutate safely and with diversity.
We demonstrate the effectiveness of this method in a variety of ways. For function regression, we show our method is one to two orders of magnitude more sample efficient than other gradient-free methods. We then leam cebiteeturr-sppeciie and general- within the training distribution, and with simple, interpretable and novel features that can be used to generate research hypotheses. Finally, we learn reinforcement learning losses in goid-world environments, with novel interpretable features that transfer to Atari.
optimizers, which perform well

---

2 RELATED WORK
Semantic genetic programming See Vanneschi al. (2014) for survey. This includes point mutations and crossover mutations. Semantic point mutations replace a node in a program with a new one that semantically close, but not too close (guyen al. 2009). Semantic crossover mutations combine two programs by swapping two nodes which are either semantically non-equivalent, or again close, but not too close (Beadle Johnson 2008 [y et al. 2009a One can crossover whole programs in semantic crossover, such as summing two programs and simplifying (Moraglio et al.) 2012) which can suffer from exponential size blowup; and although not genetic programming per-se, Gangwani & Peng (2017) crossover neural network policies via distillation. Our method is most similar to semantic point mutation, although crucially (I) we only use semantic information at the program level, which allows applications where nor-node semantic information is not defined, and (2) we also use semantic information for diversity in the mutation objective.
Diversity Diversity in evolution usually refers population diversity, where diverse range of programs is kept in the gene pool to avoid premature convergence (Burke et al., 2004). Methods include reducing fitness for crowded areas of the phenotype space (Goldberg al. 1987), or grouping within the phenotype space (Mouret & Clune 2015). Our use of diversity is orthogonal to, and could be combined with population diversity: we ensure mutations are diverse from previously evaluated programs before evaluating them, which reduces evaluations of expensive fitness functions. Ensuring programs are not semantically identical to previous programs has been done before (Alet al., 2020 Real et al. (2020) but we are the first to use semantic distance in the mutation objective, which generalizes to continuous search spaces.
Learning reinforcement learning (RL) algorithms, optimizers, and loss functions Previous work for learning loss functions and optimizers include many ways of parameterizing these components (hyperparameters, neural networks, or symbolic computation graphs), and many ways of learning (genetic programming, evolutionary strategies, meta-gradients, and Bayesian methods). Learning RL algorithms includes using meta graadggaassets (Oh et al. 2018 2020) and evolving programs Ceo-Reyes ct al.
2021
Faust et al., 2019). Genetic programming has been used to learn loss functions (Bengio 1994 Trujillo & Olague 2006; Gonzalez & Mikkulainen 2020). Methods for learning
ct al.
optimizers include gradiegrs (Hochreiter et al. |001 Andrychowicz et al. 2016 Wichrowska al., 2017 Lv et al., 2017; Metz et al. (020)), RI. (Bello et al.) [017]] Li & Malik 2016), and
evolutionary strategies (Houthooft et al. (2018). More generally, evolutionary techniques have most commonly been applied neural architecture search (Stanley & Mükkulainen [0022] Real al.
2019; Liu et al. 2018; Elsken et al.) [019 Pham et al.] et 2019). and AutoML aims to automate the machine learning training process (Hutter et al. 2019; Real et al., 2020). Our method uses genetic programming to learn symbolic computation graphs: this has the advantage of interpretable results and avoiding noisy gradient information (Metz et al. 2019). Co-Reyes et al.) (202T) is most similar to our work for RI. algorithms we use semantic information in the mutation operator, but do not directly compare empirical performance due to the complexity of reproducing the same environments and evolutionary setup.
3 METHOD
Let & be the space of programs. The two key points are to define distance function d between pairs of programs which captures semantic information; and then given G' E G, to define mutation M (G) which respects this distance function. This mutation can then be used in an evolutionary algorithm, for example, hill climbing where the best program far is repeatedly mutated and evaluated.
3.1 THE MUTATION OBJECTIVE
Let H C G be the set of previously evaluated programs, let v(H) be a measure of program complexity (for example the number of nodes), and let μ. > 0 be constants. Our objective to minimize
(G)) ((M G) pl 4tnnh( min d(M H)/p) + (Bo(M(G)). HEH
(I)
This objective contains three terms.
2

---

1. The first term requires that d(M G) y for some mutation rate μ O; i.e. that G and (G) are semantically similar (which means that the mutation is less likely to be deleterious), but not too similar (since evaluating almost identical programs would also lead to wasted computation). This similar to similarity-base -based mutation (Uy et al. 2009a b). although we only ever consider distances between programs rather than nodes within program: this allows us to work with programs where internal nodes do not have meaningful semanties by themselves, such as in optimizers or loss functions.
2.
The second term encourages diversity by penalizing programs that close to previously evaluated program. Note that the function d + μ tanh(d/ (d/ () is approximately the identity when |d] μ4, and is otherwise bounded from above by 14, thus this term gives "repulsive force" between M (G) and H only for H E H. where di (G), H) S p. We use min rather than swm since we want the first two terms to be balanced.
3. The third term penalizes programs that are complex, and is similar to parsimony pressure (Koza 1992, Zhang& & Mühlenbein 1995); this is to counteract the phenomena of bloat where the number of nodes grows without corresponding increase in fitness.
Distance function What distance function d should we use? We build the distance function by mapping each program to vector, and then using the L1 distance (i.e., mean absolute difference) between vectors. More precisely, we build s : g + R (which we refer to as the semantics of a program), a scaling p(G) RT where G is the graph mutate, and define the distance between J, E G to be
1 d(J,k) = m Σ
(ss(()) s(K)a)
For all our applications, the semantics relies on program input samples (xz) where each x; is some input previously evaluated program (see $3.3.. Note that although the distance function is a function of G and the and so varies as evolution proceeds, it in practice tends to vary slowly.
The best scaling and semantics mapping depend on the application. For example, in function regression in where the program H is used as scalar-valued function H RR* - R. we use the identity semantics %aa (H) (H))).... (xr))) E R. and the scaling p = 1. If H used as a loss function, the gradient semantics Sgnad (H) = (V (Z .... (z,))) is more appropriate, since this does not distinguish between programs whose outputs differ by a constant, but does take into account stop-gradient nodes.
3.2 IMPLEMENTATION DETAILS: MUTATING COMPUTATION GRAPHS
In our applications we restrict to programs in the form of computation graphs, i.e., ordered lists of nodes, where each node has inputs from previous outputs or external inputs, and performs some operation such as addition. We minimize CMA-ES fine-tune the constants.
( by generating several candidate mutations, and then use
Generating raw mutations See Appendix A for details. In the first step, generate several candidate programs GI...., G (we take l: 16) by repeatedly taking the original program G, then applying up to two changes to it:
With probability 0.5 adjust a uniformly chosen node, Adjusting a node a involves replacing it with one of the following expressions sampled uniformly: {sc, z + c, + cf, z(1 2f), + clff)), where: c E f0, 1} is a "gating" constant that allows continuous linear interpolation between the old and new node and is initially set value such that the new node has the same value as the old node; and f' is either an existing node, or a newly constructed node which is a randomly chosen function of existing nodes. Thus after this step, the program still has the same semantics as the previous program.
2. With probability 0.5, replace randomly chosen node with one of its inputs. This allows program simplification.
We simplify the resulting program, for example by removing unused nodes, and merging adjacent add and multiply nodes. We repeat these steps until & distinct programs (ignoring constants) have
3

---

been generated. We experimented with different raw mutation operators. mutation probabilities, and the number of mutations applied, but it did not make a noticeable difference to performance.
Optimizing The second step uses the black-box optimization algorithm CMA-ES (Hansen & Ostermeier 2001 Lange 2022) to optimize the constants appearing in each of the programs G; to minimize (T] the program with the smallest value of (I) then the mutation output. We use CMA-ES population size of 128, initial standard deviation of 0.01, and run for 200 generations. (Thus we evaluate (T) a total of 16 x 128 200 10 times, which is fast on modern ML hardware; this is one source of how we are able to achieve orders of magnitude speed-up; see
To make optimization more stable, CMA-ES does not directly tune the constants (which can have dramatically different magnitudes). Instead, for candidate program, let E R be the initial constants appearing in the computation graph, let v E R be the values that CMA-ES is tuning: where
we set the new program constants to be
R is custom scaling, defined as T; = max(10-5 min [ [4 - 1/, [u + 1}}}. Observe that when the initial value u, of a constant is large, then CMA-ES "step size" will be much bigger than if it is close to 0, multiplicative constant close to such as 0.999.
3.3 IMPLEMENTATION DETAILS: DISTRIBUTED SETUP
We evaluate programs in a distributed setup consisting of single central controller, and several evaluation workers (ranging from 8 to 64 in our experiments) each consisting of machine with TPUv2-8 Tensor Processing Units.
Sampling and/or mutation The central controller keeps track of the fitness of evaluated programs. When a evaluation worker is free, it requests program to evaluate. A program is selected from those with highest estimated fitness (or the original none have yet been evaluated), mutated on the evaluation worker, and evaluated. (Additionally for applications with noisy fitness measurements, such as RL in Section 7 with probability 0.2 we re-evaluate the program without mutating it; this reduces fitness uncertainty.)
We found it beneficial to bias the selection of the program towards simpler programs, by first sampling the max number of nodes uniformly from the node count range already seen, and then sampling the best program conditioned on having at most this many nodes. This can be seen as maintaining diversity in complexity, and as such is similar to Mouret & Clune (2015).
Mutation rates and parsimony pressure We resample ze and / for (T) in each mutation. The optimal value of μ depends on the fitness landscape around G; for example, if we are close to optimum, then smaller will likely work better. 3 trades off between program complexity and the distance terms, and there likely no single value that works better either. Rather than picking a single fixed value, we sample ~ (O, 0.44 and B ~ exp(U -1)) where U (a, b) is uniform random variable the range (a, b). These values were chosen using hyperparameter sweeps on the function regression and optimizer (with a reduced set of tasks) applications.
Collecting sample inputs for the semantics Recall that the semantics function s is defined in terms of sample program inputs (a:))_ These are collected by the evaluation workers, and stored in the central worker, which uniformly samples the v, from across all evaluated programs.
4 ANALYSIS: LEARNING SIMPLE FUNCTIONS
For initial analysis we learn a program G that approximates some unknown function f. where the fitness one minus the mean square error over 1000 uniformly sampled points yi, i.e.
G) = - (G(v)) - 2/11)
We evaluate on f(y) = sin for in the range [- πT] and 1-2 2n and fly 2) = atan2 2) for y. 2 in the range 1-1, 1)2 since these cover range of function complexities, with both scalar and vector inputs. For the distance function (2) we use the identity semantics and p = 1. We restrict graph operators add, sub, mul, div, abs, and max.
4

---

4.1 PERFORMANCE AND ABLATIONS
Figure shows the performance according to which terms of the mutation objective ① are included. Diversity is very important for fitness, and simplicity mildly important. One hypothesis for why simplicity also helps fitness is that since graphs with fewer nodes give a lower dimensional set of functions, there are simply fewer distinct graphs to explore over. Figure 2a shows the average number of nodes in the graph with the highest fitness over time; interestingly both the diversity and simplicity terms are required for obtaining small graphs. Figure shows the effect of the diversity term: the new mutation is pushed to be distinct from nearby previously evaluated functions.
elar2
5m
si_blg
1.0
1.00
1.00
diveriiyy
simpicity
diversity,
ftteess
0.9
0.99
smpliitty
0.90
diversity. no simplicity diversity. smplkity
0.85
0.0
0.0
0.80
st0p
step
step
Figure I: Performance with and without the diversity and/or simplicity terms of ① Also shown is "plain mutation" which is the raw mutation operator $3.2 without optimizing for any objective.
$n_big
80
group
X
elocestDD clases_1 I elesest_2 2
40
20
original
8.0
0.2
0.4
0.6
0.8
1.0
1c4
X
(a) Node count learnt graphs for (b) The diversity term in action. This shows sin(x) over time, according to which the original function, new function, and the terms are included.
three closest previously evaluated functions.
(c) Example learnt graph.
Figure 2: Insights into the algorithm.
4.2 SAMPLE EFFICIENCY COMPARISON
We compare our method with other approaches for learning a function given scalar fitness measure- ments, including plain point mutation with and without crossover, other semantic based methods, and also using CMA-ES to learn parameterized function. See Appendix B for details. We also tried gplearn (Stephens] 2016), which learns tree-based programs, but it had much worse performance. As Figure
shows, we are one two orders of magnitudes more efficient than other methods.
$n_big
1.00
10000
CMA ES
0.95
99755
0.68
Point mutation, no crossover SSM, cressaver
0.90
9950
0.94
Point
0.85
9925
3.92
Point mulationn
0000
0.00
<tep
164
Figure 3: Comparison in fitness sample efficiency. CMA-ES NN learns the weights of neural network through evolutionary strategies. Point mutation the raw mutation algorithm described in 3.2 with parsimony pressure. SSM is Semantic Similarity Mutation (Uy et al. (009a).. Random crossover randomly swapping nodes. SSC is Semantic Similarity Crossover (Nguyen al. 2009
5

---

5 EXTENDING TO SEMANTIC CROSSOVER
We briefly remark we can generalize this approach to create a crossover operator C, and we show how. Given given two programs G, H 9, write C C(G, H), and let
Serossvver (C) max d),
This objective is minimized when C is semantically halfway between G and H. This be minimized following the same approach as in 3 This further improves performance; see Figure
which-atar2
which-sin
which-sin
1.0000
1.000
0.9995
0.995
farea
0.90
SSC
0,9900
0.990
Random
0.9985
0.985
0.96
0.2
0.4
0.5
0.8
8.9800
0.2
0.4
0.6
0.8
0
0.980
0.2
0.4
0.6
0.6
10
stop
164
step
1ed
step
1e4
Figure 4: This shows sample efficiency when the mutation operator described by T is used in conjunction with crossover. Average is the semantic crossover operator described by (4) SSC is swapping semantically similar nodes (Uy et al. 2009a). Random swaps nodes at random.
APPLICATION: LEARNING AN OPTIMIZER
We demonstrate the method on non-toy tasks. Can we learn a neural network optimizer? An elementwise optimizer is a function f : R x R x R + R x R that maps f (g. 0, + (0' 8'), where g E R is a gradient, 8, 8' E R are the previous and new parameter values, and s, s' R are the previous and new state values for some k 2 0. For example, gradient descent with momentum has 1 and f (g. m) (0- rym', m²) where m' (1 x)m + kg for some
1. By
transforming by additive constant, it is always possible to take the initial state to be 0 E R and so this formalism captures all optimizers that act elementwise, such as Adam (Kingma Ba 2014).
Recall that we need to define optimizer semantics to define distance between programs. The identity semantics applied to is inappropriate, since we care about the (long term) effect on parameters rather than the immediate change in optimizer state. So we instead build the semantics as follows:
The samples (==) (see (3.3) used to construct the semantics are pairs x; = =0., (g.;);=1). where 0g E R the value of some parameter at some point during training and (g:) 1 is sequence of gradients for that parameter subsequent timesteps. We collect 500 trajectories each of length s 2000.
2. For each Tú, let O(G, .x) R be the parameter rollout starting 0, and applying optimizer updates to it using the gradients 9ıj (we set the initial optimizer state to 0).
3. Define the semantics to the set of single step parameter changes at 1000 randomly sampled points. Lee., sample S ii...., 81000 I n and 1 < ji,..., J1000 S 8 and set
Sugi (G) = (O(G, xi, )i,. O vi, bi 1) E R 1000
55)
The semantics in (5) a sequence of parameter value changes. These may be dramatically different in magnitude, so for the scaling in (2) we normalize these by
p(x) = 2/( + median((::l))
where |oc| is the elementwise absolute value of æ. For mutation, in addition to the operator described in (3.2, we also allow changing the state size k although in the experiments below our best performing optimizers did not make use of this.
6.1 LEARNING A GENERAL PURPOSE OPTIMIZER
Can we learn a hyperparameter-fiie free optimizer that achieves good eval performance across a wide range of network architectures and datasets? We measure fitness by training and evaluating the

---

set of fixed tasks T in the learnt optimization task suite with the hope that an optimizer trained on sufficienlly many tasks will generalize. Each task T E T defines a parameter initialization function, train and eval datasets, and loss function. The fitness of optimizer G' is then given by Z where eval iral_loss(G, G, T) is the evaluation loss after training for 10 steps, lr < Lo are constants hand-picked by examining the typical loss range, and train frac_ finite(G, T) [0, 1] is the fraction of train steps that the train loss is finite, which gives a signal on tasks which diverge during train.
max (0. LT eval loss(G.T)) + train frac f (i (7)
Fopt(G)
Σ TET
IT
LT LT
We set the initial optimizer equal to momentum with a learning rate of 10-3. We use 64 parallel evaluation workers, with each fitness evaluation taking approximately 30 minutes.
Results The initial optimizer has a fitness of 0.331. After approximately 3500 optimizer evaluations, the following optimizer with fitness of 0.551 was discovered, with single state variable m (initialized at zero as described above) and parameter & update formula
(8)
1.0002 + 10.881 g
0.99986(# 0.01636m') + 2.67 10-6
This optimizer performed well across all tasks, with average performance approximately equal to that of Adam (which gets fitness of 0.554), but with only single state variable and far fewer nodes. The 0.99986 term is L.2 parameter decay. However the "momentum" update (8) has very interesting properties, and to the best of our knowledge has not been seen before. Rather than scaling momentum uniformly at every step, it only scales down by large gradients; in particular parameters that receive small gradients accumulate "momentum" over more steps. Additionally the change in 0 is always bounded, since mm| S 1/10.881 no matter how large [gl is.
Interestingly, this optimizer failed to generalize outside the task suite, failing to train large trans- former. This was unexpected, since the optimizer has fewer constants than the number of tasks [T| R 50, so we did not expect it to overfit, but it is possible that the tasks in 'T lack diversity, for example the length of training, or gradient magnitude. Nonetheless this is a promising initial result: (I) we are able to learn a simple symbolic optimizer that performs well within training distribution, so we are reduced to the problem of finding a representative set of tasks T, (2) the optimizer contains novel features not seen before, which can generate interesting research insights in themselves, and (3) there many potential avenues to be explored, such allowing the learnt optimizer to make use of extra inputs such as gradients from other parts of the network.
6.2 LEARNING NETWORK- SPEEIIIC OPTIMIZER
We also learn an optimizer to fine-tune single model: CIFAR- 10 classification using VGG (Simonyan & Zisserman 2014). The fitness is the evaluation accuracy after 10 training steps. To allow for discovery of things like learning rate decay, provide scalar input t E [0, 1]. which is the fraction
through training. We set the initial optimizer equal to momentum, and the best performing optimizer is discovered after around 1000 evaluations:
0' (0.9992 + 0.001t) 0.0012g)
(10)
This is very simple optimizer (momentum has been unlearnt). But it is high performing: matches the performance of a highly tuned momentum plus learning rate-schedule with weight decay. The term (0.9992 + 0.001t) corresponds to an L2 parameter regularization term, i.c., weight decay. However, it varies with time. At t O it corresponds to a loss term of 8 x 10-482 /2; and more interestingly at t = it corresponds to a loss term of -2 x 10 10-402 #22/2 /2, i.e., the final 20% of training the network actually undergoes weight expansion. This is not a superfluous feature, as shown in Figure weight expansion helps final performance, and is an example of the sorts of research insights that can be generated using evolved computation graphs.
http:////t thub. http://ggthuub..coooooooogeeo/e/ httt:///...hht....oooogg//e/eeee
7

---

0.950
0.945
top 1 accuracy
0.945
top accuracy
0.940
0.935
0.935
0.930
0.930
0.925
0.925
0.00
0.25
0.50
0.75
1.00
1.25
Initial L2 loss coeff
1e-3
final L2 loss coeff
1e-4
Figure 5: Final eval accuracy according to initial and final L2 parameter coefficients. Having positive L2 loss coefficient required through most training; however, slight negative L2 loss coefficient (weight expansion) at the end of training helps performance; this is not something we are aware has been investigated before. (Error bars show standard deviation, over seeds.)
7 APPLICATION: LEARNING REINFORCEMENT LEARNING LOSS
A typical deep reinforcement learning agent contains neural network that maps observation to policy and a value. The parameters of the network are trained using a loss function, which takes in these predictions together with other inputs such as the environment rewards. The loss function affects the agent performance, with innovations such as reward scaling, handling off-policy data Espeholt et al. (2018). trust region policy updates (Schulman al., (015), etc leading to better performing agents. Can our method can learn loss functions that are effective across a range of environments?
Loss graph and semantics We use the following notation. At time step f, the agent network outputs policy and value V, given an observation 8t. An action is sampled from Te, and the agent then receives a reward r,, with the mask m, E {0, 1} indicating that the environment has not yet terminated. We restrict to loss functions that be expressed using backward trajectory with n-step bootstrapped returns R₁. The graph G defines per-step loss the n-step trajectory, where 20. We additionally input the policy entropy H, Ea-T, log Te, since reduction nodes in the computation graph are not yet supported.
G: : (V,, Vi+1, Tt+1, my m4+1, log Ti (aols)), He, R4+1) H (loss, Rs)
(11)
For the semanties we collect 500 input trajectories (i.e., the left hand side (TI). Given a graph G, for each trajectory calculate the gradient of the loss defined by G with respect to the three differentiable inputs; this yields three vectors TV 0H Um
aL
E R 500x20 We then randomly sample 1000 points from
each these vectors, yielding SRL( (G) E R 3x 1000
We use the scaling defined by (6).
Evaluation We evaluate the learnt loss function on the 15 enviromments introduced by Oh al. (2020), which consist of 5 tabular grid worlds, 5 random grid worlds, and 5 delayed chain MDPs. These environments are small and implemented in JAX (Bradbury al. 2018) which allows quickly training agents with the environment in the loop. We use approximately the same network architecture, environment settings, and number steps per lifetime as in (Oh al. 2020), except we cap at maximum of 107 steps per task for speed of training. The fitness is defined as the average scaled episode return across the different environments after training.
As baseline, we use advantage actor critic (A2C) (Mnih et al. 2016)) with discount factor y and loss coefficient Centropy This is defined by the following equations:
R2 = Ft- + (mg+
(12)
A, RG (V.)
(13)
loss = A, log ग (a) + (RR V1)² Centropy Hi,
(14)
where an advantage, and sg denotes a stop- gradient operator. Our graph was initialized to the baseline A2C and evolved from it.
8

---

CIAPUFICT
1.00
08
A2C
631
Leamt
00
144
185
Lot
160
1er_am Lioeree
14/5_107 J11_
15
0.79
0.50
913
141
125
1d1
155
Litr Jegg
LEAd- wwge Jare
-1.1
180
3029
300
Figure 6: Performance of the best performing learnt loss function and a tuned A2C baseline on 15 training environments from (Oh al. 2020).
Results Figure 6 shows loss function outperformed the baseline A2C across 15 environments
The learnt loss function made various changes to the original A2C loss function, including discovering TD(A) returns (Sutton & Barto 2018). It also made the following simple change to the policy gradient term, namely replacing the advantage coefficient in Eq (14) with
A'= A + 3.15 max(0, A).
This has the effect of more heavily weighting policy updates when the advantage is positive rather than negative (i.e., actions with higher-than-expeee expected returns). In fact, this term was proposed by Oh al. (2018) as self-imitation learning combined with A2C. We confirm the learnt loss function transfers to Atari: a modified version of it (replacing the 3.51 constant with 1) on Impala (Espeholt jt al., 2018) trained for 200M frames without any further hyperparameter tuning performed better than A2C on 26 out of 57 games and worse on 22 out of 57 games. On 5 games it significantly improved the performance of A2C (see Appendix C). These results shows that our method is capable of both performing well within-training distribution, as well as generating research insights due to the interpretable nature of the learnt programs.
CONCLUSION
We have shown how use whole-program semantic information in a mutation objective achieve safe, diverse mutations. This method significantly improves sample efficiency, allowing us to discover algorithmic features for optimization and reinforcement learning. These methods may generalize to other program spaces too, for example neural architecture search.
There are some limitations of the method. The use of distance function requires engincering infrastructure to collect program inputs. Although not specific to our approach, evolution in general still data intensive, and the solutions discovered are limited to the search space, but we hope that as the amount of compute and how the programs are used increases, that these become less of an issue, in the same way that neural networks are sufficiently flexible to capture many transformations we care about.
The experiments we have ran so far have been relatively limited, with small amounts of data (the largest experiments trained for less than two days), small tasks such as neural networks with only a few layers and grid-world RL environments. The methods can be easily scaled up with more compute, across more tasks, and for longer, well increasing the expressivity of the program (such as reduction operators) and how the programs are used, and we hope that this will lead to increasingly powerful programs being discovered.
Althoug we approximately the same set of environments Oh al. 2020), the settings were slightly different. Thus we compare against our own tuned A2C haseline which our program was initialized.

---

REFERENCES
Ferran Alet, Martin Schneider, Tomas Lozano-Perez, and Leslic Pack Kaelbling. Mcta-leaming curiosity algorithms. arXiv preprint arXiv:2003.05325, 2020.
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew Hoffman, David Pfau, Tom Schaul, Brendan Shillingfod. and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29, 2016.
Lawrence Beadle and Colin G Johnson. Semantically driven crossover in genetic programming. In 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence), pp. 111-116. IEEE, 2008.
Irwan Bello, Barret Zoph, Vijay Vasudevan, and Quoe V Le. Neural optimizer search with reinforcement learning. In International Conference Machine Learning, pp. 459-468. PMLR, 2017.
Samy Bengio, Yoshua Bengio, and Jocelyn Cloutier. Use of genetic programming for the search of a new leaminng rule neural networks. In Proceedings of the First IEEE Conference on Evolutionary Computation. IEEE World Congress on Computational Intelligence, pp. 324-327. IEEE, 1994.
James Bradbury, Roy Frostig. Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin. George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. htthttp::
http:httpp/ttte http:::/ggithb..../ com/ google/ jax
Edmund K Burke, Steven Gustafson, and Graham Kendall. Diversity in genetic programming: An analysis of measures and correlation with fitness. IEEE Transactions Evolutionary Computation, 8(1):47-62, 2004.
John D Co-Reyes, Yingjie Miao, Daiyi Peng, Esteban Real, Sergey Levine, Quoc V Le. Honglak Lee, and Aleksandra Faust. Evolving reinforcement learning algorithms. arXiv preprint arXiv:2101.000 001553958. 2021.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. The Journal of Machine Learning Research, 20(1 1997:1997-2017 2019.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-RL RL with importance weighted actor-learner architectures. International Conference on Machine Learning, pp. 1407-1410 1416. PMLR, 2018.
Aleksandra Faust, Anthony Francis, and Dar Mehta. Evolving rewards automate reinforcement learning. arXiv preprint arXiv: 1iX::1905000628. 2019.
Tanmay Gangwani and Jian Peng. Policy optimization by genetic distillation, arXiv preprint 1.00022 2017.
David E Goldberg, Jon Richardson, et al. Genetic algorithms with sharing multimodal function optimization. In Genetic algorithms and their applications: Proceedings the Second International Conference on Genetic Algoritims, volume 4149, Hillsdale, NJ: Lawrence Erlbaum, 1987.
Santiago Gonzalez and Risto Miikkulainen. Improved training speed, accuracy, and data utilization through loss function optimization. In 2020 IEEE Congress on Evolationary Computation (CEC), pp. 1 1-8. IEEE, 2020.
Nikolaus Hansen and Andreas Ostermeier. Completely derandomized self-adaptation in evolution strategies. Evolutionary computation, 9()):59-1155 2001.
Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In International Conference on Artificial Neural Networks, pp. 87-94. Springer, 2001.
Rein Houthooft, Yuhua Chen, Phillip Isola, Bradly Stadie, Filip Wolski, OpenAl Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. Advances in Neural Information Processing Systems, 31, 2018.
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren. Automated machine learning: methods, systems, chal- lenges. Springer Nature, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1r1 2014.
JRGP Koza. On the programming of computers by means of natural selection. Genetic programming, 1992.
Robert Tjarko Lange. evosax: Jax-based evolution strategies, 2022. URL. http: /ttp://gion wwg.ttthb... com/ (kbbrrt/llange/evoss
10

---

Ke Li and Jitendra Malik. Learning optimize. arXiv preprint arXiv:1606.01885, 2016.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceedings of the European conference on computer vision (ECCV), pp. 19-34,7 2018.
Kaifeng Lv, Shunhua Jiang, and Jian Li. Learning gradient descent: Better generalization and longer horizons. In International Conference on Machine Learning, pp. 2247-2255. PMLR, 2017.
Luke Metz, Niru Maheswaranathan, Jeremy Nixon, Daniel Freeman, and Jascha Sohl-Dickstein. Understanding and correcting pathologics in the training learned optimizers. International Conference on Machine Learning, pp. 4556-4565. PMLR, 2019.
Luke Metz, Niru Maheswaranathan, C Daniel Freeman, Ben Poole, and Jascha Sohl-Dickstein, Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves. arXiv preprint arXiv:2009. /243, 2020.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement Icarning. In International conference machine learning, pp. 1928-1937. PMLR, 2016.
Alberto Moraglio, Krzysztof Krawice, and Colin G Johnson. Geometric semantic genetic programming. In International Conference on Parallel Problem Solving from Nature, pp. 21-31. Springer, 2012.
Jean- nan--Batistte Mouret and Jeff Clune. Illuminating search spaces by mapping clites.
arXiv preprint
arXiv:1504.0m 04909, 2015.
Quang Uy Nguyen, Xuan Hoai Nguyen, and Michael O'Neill. Semantic aware crossover for genetic program- ming: the case for real-valued function regression. Eurupean Conference on Genetic Programming, pp. 292-302, 302. Springer, 2009.
Junhyuk Oh, Vijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. International Conference on Machine Learning, pp.
PMLR, 2018.
Junhyuk Oh, Matteo Hessel, Wojciech Czamecki, Zhongwen Xu, Hado P van Hasselt, Satinder Singh, and David Silver. Discovering reinforcement learning algorithms. Advances in Neural Information Processing Systems, 33:1060-1070, 2020.
Hicu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International conference muchine learning, pp. 4095-4104. PMLR, 2018
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pp. 4780-4789, 2019.
Esteban Real, Chen Liang, David So, and Quoe Lc. Automl-zero: Evolving machine learning algorithms from scratch. In International Conference on Machine Learning, pp. 8007-8019. PMLR, 2020.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. International conference on machine learning, pp. 1889-1897, PMLR, 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.15 2014.
David So, Quoe Lc, and Chen Liang. The evolved transformer. In International Conference on Machine Learning, pp. 5877-5886. PMLR, 2019.
Kenneth O Stanley and Risto Miikkulainen. Evolving neural networks through augmenting topologies. Evolu- tionary computation, 10(2):99-127, 2002.
Trevor Stephens. Genetie programming in Python with a Scikit-lcar inspired API: gplearn, 2016.
Rich Sutton. The bitter lesson. http::/www..o. http://wwww..eeaaaaaaaaaaaaaaaaa htmi 2019. 2019-03-30.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Leonardo Trujillo and Gustavo Olague. Synthesis of interest point detectors through genetic programming. In Proceedings of the 8th annual conference on Genetic and evolutionary computation, pp. 887-894, 2006.
11

---

Nguyen Quang Uy. Nguyen Xuan Hoai, and Michael O'Neill. Semantics based mutation in genetic programming: The case for real-valued symbolic regression. In 15th international conference on soft computing, Mendel, volume 9, pp. 73-91, 2009a.
Nguyen Quang Uy. Michael O' Neill, Nguyen Xuan Hoai. Mckay, Edgar Galván-López. Semantic similarity based crossover in gp: The case for real-valued function regression. International Conference on Artificial Evolution (Evolution Artificielle), pp. 170-181. Springer, 2009b.
Leonardo Vanneschi, Mauro Castelli, and Sara Silva. A survey of semantic methods in genetic programming. Genetic Programming and Evolvable Machines, 15(2):155 2014.
Olga Wichrowska, Niru Maheswaranathan, Matthew Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando Freitas, and Jascha Sohl-Dickstein. Learned optimizers that scale and generalize. In International Conference on Machine Learning, pp. 3751-3760 PMLR, 2017.
Byoung- Troung-Tak Tak Zhang and Heinz Mühlenbein. Balancing accuracy and parsimony in genetic programming. Evolutionary Computation, 377),17-38, 1995.
12

---

A GENERATING RAW MUTATIONS
We found that the precise form of the raw mutation operator which generates candidate programs to fine-tune did not matter, i.e., that the performance of the final algorithm was largely insensitive to precise implementation details and probabilities. But for completeness we detail the raw mutation operator used in our experiments.
Given an the original program G, we generate a mutation by the following procedure.
1. With probability 0.5, do the following. Select a node & uniformly at random. Select an expression from {ac, + C, cf, x1 cf), 2/(1 cffl) uniformly at random. Here, is a constant, that is chosen from {0, } to be such that the resulting expression is equal to (c is later fine-tuned optimize the mutation objective.) Further, selected as follows. With probability 0.5, f is an existing node in the program selected uniformly at random. And otherwise, f is generated by first uniformly selecting a function in (abs, add, div, exp. log, max, min, mul, neg, pow, sqrt, square, sub, cos, cosh, sin, sinh, tan, tanh), and then selecting the appropriate number of inputs for the arity of the function uniformly from the existing nodes in the program.
2. With probability 0.5, replace randomly chosen node with one of its inputs. This allows program simplification.
We simplify the resulting program, for example by removing unused nodes, and merging adjacent add and multiply nodes.
B DETAILS OF FUNCTION REGRESSION COMPARISONS
We describe here the methods compared in Figure
B. SEMANTIC SIMILARITY CROSSOVER (SSC)
This method follows Nguyen et al. (2009. For two programs represented as computation graphs, we calculate the semantic distance between each pair of nodes (one in each program) by using the collected sample program inputs (x and taking the L2 distance between the intermediate outputs. We then uniformly sample two nodes to crossover among nodes whose distance is within the range 10.1, 1.0] (chosen using a hyperparameter sweep), or uniformly sample between all pairs there are nodes within this range.
B.2 SEMANTIC SIMILARITY MUTATION (SSM)
This method follows Uy et al. (2009)). Given program represented as a computation graph, we first select node uniformly random, and then generate candidate mutations using the same method as described in 3.2 to find one which is at an 1.2 distance of from the original node, and then replacing the original node with the new (sub) computation graph. Here, μ (0, 0.4) as Section 3.3
B.3 LEARNING THE WEIGHTS OF NETWORK USING CMA-ES
We use CMA-ES (Hansen & Ostermeier 2001 Lange 2022 to tune the weights of a fully connected neural network. The tollowing hyperparameters, obtained using a sweep, used:
Number of intermediate hidden layers 2
Total parameter count 128
CMA-ES Population size 16
CMA-ES Initial standard deviation = 0.01
C GRAPH VISUALIZATION: A2C, AND LEARNT GRAPHS
D POSITIVE-ADVANTAGE-BE ON ATARI
13

---

update
0.8842
mask
returss
mul
reward
value_tm1 tml_mm__g
value_tmi tml
add
sub
sub
returms
logps
neg
integer pow y=2
0.258694
entropy
mul
div
0.00433399
neg
add
mul
add
loss'
Figure 7: The A2C graph. This shows with constants tuned for maximizing the RL fitness (but nodes added or removed). The "_sg" suffix indicates stop gradient has been applied. Performance shown in Figure as A2C.
14

---

update
loges
-3
0.0078889
1.2025
muskc tmll
[_$
mask_
rahansa
min
div
0.15577
ndd
-7 7.30611e005
rewar.]
eulsopy
40 40.221282
add
mill
-1.12676
add
Valte mmll
Value tul
retanas
add
sub
subh
inleger pow y-2
nat
1.00114
-0. 03.00422422
add
Figure 8: A graph learnt from search initialized with A2C (see Figure 7.. The " "_s"" suffix indicates a stop gradient has been applied.
15

---

2000
400000
180
50
180
Seviing
5rdig
2300
150
409
2000
0.5
0.5
15
3.9
05
15
30
02
05
11
35
13
35
16
15
80
10
15
Hinee
484
100000
400
300
1807
0000
4000
790
2003
1000
300
18K
0.52
360
-14
-1.2
3000
750
5000
180
1mns
2x Herteyy
3300
250
180
0.5
03
1.5
F7M
13000
20.00
500
1900
-00
2000
1985
580
-064
100
500
13000
13000
2000
100
085
0.5
35
13
05
14
0.
10
81
05
38
35
13
35
1.0
15
83
35
10
15
30
03 BPM
430
300
200
2500
180
2000
1500
0.2
100000
20000
750401
2000
baseline
positive advantage
2000
Figure 9: Performance with the advantage term used in the policy gradient loss in Impala (Espeholt) et al. 2018 replaced by advantage + max(0, advantage). Trained over 200M frames, 3 seeds each.
16


Reviews:
[{"review_id": "qrUA2NIemfa", "reviewer": "Reviewer_zufj", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a semantic mutation operator for genetic programming (GP), which aims at creating new diverse candidate solutions while keeping a good connection with the parent solutions. Semantic GP is a common approach that aims at exploiting the behavior of programs or subprograms (e.g., the output generated for given test cases used for evaluation) instead of their structure. This signal has been shown as relevant to be included in the optimization methods of GP, optimization guided purely on the structure of GP solutions being non-smooth or continuous, with simple changes in the structure leading to huge differences in the behavior. The proposed semantic mutation operator is presented and evaluated for three specific problems, that is real-valued function approximation, gradient-descent-based optimization function, and a loss function for reinforcement learning.", "summary_of_the_review": "The proposal is simple, but the explanations are not very clear, many elements are not justified, there is a lack of comparison with other related methods and the problems tackled are non standard (not common benchmark in the field) nor of practical interest by themselves.", "strength_and_weaknesses": "Strength\n- Straightforward proposal for a novel semantic GP mutation operator.\n- Apparently good empirical performances in three applicative contexts.\n\nWeaknesses\n- The proposal is relatively simple and incremental considering the work presented so far on semantic GP. There is no detailed overview of semantic GP, nor a good analysis of the similarity and differences of the current approach with the literature on the topic is missing.\n- The experiments are done on specific problems and lack detailed comparison with other common semantic GP approaches.\n- The paper writing quality is below average, and details on the method proposed are missing or unclear at times.\n- The overall proposal appears relatively niched.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "R1jRNDyUt0m", "reviewer": "Reviewer_kHGQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new sampling method for genetic programming based on the semantics. The idea is that for each step, the algorithm would sample a new variant with different semantics (based on some distance function) that is also different from previous historical samples. Based on this idea, the authors demonstrate that this method improves sample complexity on a bunch of tasks, including function regression, learning an optimizer, and learning RL losses.", "summary_of_the_review": "This paper mainly provide experiments on small scale datasets and lack comparison with existing methods. I'm not an expert on genetic programing, so I'm not sure how important these tasks are for the community --- from my perspective the tasks (e.g., learning optimizer and learning RL loss) are not super relevant these days especially when the experiments are only carried out on small datasets. Thus, I'll vote for rejection based on my current understanding, but would love to follow other reviewer's suggestion if they are more confident on this topic.", "strength_and_weaknesses": "Strength:\n- The paper provides several instantiations of the method, all seem quite interesting.\n\nWeakness:\n- The method seems pretty simple and not very surprising, so not sure how much conceptual novelty there is. I'm not an expert on genetic programming, so would leave it to other reviewers who have more domain expertise on this topic.\n- For the learned optimizer experiment, there's no comparison with other methods. In fact there is a rich literature of previous works on this problem, i.e., how to select the hyperparameters in an algorithmic way. More comparison with them would be great.\n- The experiments are definitely too small scale. E.g., for the learned optimizer, it's unclear whether it works on larger NN models, so the findings in this paper may be not relevant to modern deep learning. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-_X3FmGTQXS", "reviewer": "Reviewer_6oRb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a mutation operation to generate semantically close and diverse individuals in genetic programming. The proposed method generates a candidate program minimizing the mutation objective that takes into account the distance between parent, child, and previously evaluated programs. Then, the proposed method enables to the generation of a candidate program that is close but not too close to the parent and is not close to previously evaluated programs. To validate the effectiveness of the proposed concept, the authors apply the proposed method to simple regression, learning an optimizer, and learning reinforcement learning loss tasks. In the simple regression task, the proposed method can accelerate the search efficiency compared to existing mutation operations. In addition, the proposed method could find a better solution than existing algorithms in learning an optimizer and learning reinforcement learning loss tasks.", "summary_of_the_review": "The topic treated in this paper is interesting and important. However, the reviewer feels that the technical novelty and empirical evidence of the advantage of the proposed method are unclear. Such weakness should be addressed to accept the paper.", "strength_and_weaknesses": "[Strength]\n* A novel mutation operation taking into account the semantic similarity is introduced.\n* The effectiveness of the proposed method is demonstrated for not only simple problems but also meta-learning tasks.\n\n[Weaknesses]\n* The reviewer is not an expert on this topic and is not confident whether the proposed semantic distance between programs defined in (2) is novel or not. The technical novelty of the distance function should be clarified. \n* In the proposed method, it seems that minimizing the objective of (1) and distance calculation requires generated programs' execution, and such cost should not be ignored. The reviewer is not sure that the experimental comparison with existing methods was fair. It would be better to compare the performance under the same number of program executions or computational budgets.\n* In the experiments on learning an optimizer and learning a reinforcement learning loss, although the use case of the proposed method is demonstrated, the advantage of the proposed method on such tasks compared to other genetic programming methods is unclear.\n* The concept of the proposed method can be understandable. However, it seems difficult to re-implement it because the authors did not provide the code and detailed information on the implementation.\n\n[Minor Comments]\nWhat does `sin_big` mean in Figure 1-4?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject