Proppo: a Message Passing Framework for Customizable and Composable Learning Algorithms
Authors: Paavo Parmas
Takuma Seno
Venue: NeurIPS 2022 Accept (2022)
Keywords: machine learning frameworks
Monte Carlo gradient estimation
Automatic Differentiation

Abstract: While existing automatic differentiation (AD) frameworks allow flexibly composing model architectures, they do not provide the same flexibility for composing learning algorithms---everything has to be implemented in terms of back propagation. To address this gap, we invent Automatic Propagation (AP) software, which generalizes AD, and allows custom and composable construction of complex learning algorithms. The framework allows packaging custom learning algorithms into propagators that automatically implement the necessary computations, and can be reused across different computation graphs. We implement Proppo, a prototype AP software package built on top of the Pytorch AD framework. To demonstrate the utility of Proppo, we use it to implement Monte Carlo gradient estimation techniques, such as reparameterization and likelihood ratio gradients, as well as the total propagation algorithm and Gaussian shaping gradients, which were previously used in model-based reinforcement learning, but do not have any publicly available implementation. Finally, in minimalistic experiments, we show that these methods allow increasing the gradient accuracy by orders of magnitude, particularly when the machine learning system is at the edge of chaos.

O Proppo: a Message Passing Framework for Customizable and Composable Learning Algorithms
Paavo Parmas'
Takuma Seno Keio University
Kyoto University
paavodsys. i wyo.o-y.oc.......
seno@ailab. www.iis......e... keio.ac.......... jc.
Abstract
While existing automatic differentiation (AD) frameworks allow flexibly com- posing model architectures, they do not provide the same flexibility for compos- ing learning Sggorithms-everyee
has to be implemented in terms of back- propagation. To address this gap, we invent Automatic Propagation (AP) software, which generalizes AD, and allows custom and composable construction of complex learning algorithms. The framework allows packaging custom learning algorithms into propagators that automatically implement the necessary computations, and be reused across different computation graphs. We implement Proppo, a prototype AP software package built on top the Pytorch AD framework. To demonstrate the utility of Proppo, we use it to implement Monte Carlo gradient estimation techniques, such as reparameterization and likelihood ratio gradients, as well as the total propagation algorithm and Gaussian shaping gradients, which were previ- ously used in model-based reinforcement learning, but do not have any publicly available implementation. Finally, in minimalistic experiments, we show that these methods allow increasing the gradient estimation accuracy by orders of magnitude, particularly when the machine learning system is at the edge of chaos.?
Introduction
Application
Application
Application
Software library
Software library
Computation software
Computation software
Computation software
Automatic propagation
(a) Direct approach
(b) Library approach
(c) Automatic propagation software approach
Figure 1: A simplified taxonomy of machine learning (ML) software development methodologies. The computation seftware refers to existing ML tools such as Py (Paszke et al., 2019) or NumPy (Harris et al., 2020); the software library reusable code for new ML method.
When a machine learning (ML) researcher comes up with new algorithm, they are faced with a dilemma: do they directly implement an application of their algorithm using existing ML tools (Fig. la); do they implement& a library for their algorithm, then the library implement the desired application (Fig. 1b). The first approach is faster, because the researcher only has to worry about the details of the specific application; the second approach may take longer to implement, but will enable reuse of the code and speed up implementing future applications. Clearly, the second
Affiliitted with the Okinawa Institute of Science and Technology (OIST) during much project. I Hired by OIST during his contribution to the project. Currently working at Sony Al. 'The code is available at https:/ / //gtttb.b..////// www./ooo////eeeeee prtppppppp//pppp.....
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

approach is preferable in the long-term. Our work aims to overcome this dilemma by simplifying the intermediate stage of implementing reusable code for the algorithm. The approach is illustrated in Fig. Ic, where, our invention, automatic propagation software, ameliorates the shortcomings of existing tools that we utilize concurrently to implement reusable code for new algorithm. This interusability was key design choice, as it allows benefitting from the rich ML software ecosystem.
In ML vast amount of effort has already been placed in developing computational software related to back propagation, ranging from established automatic differentiation (AD) libraries (Paszke et al., 2019: Tokui et al., 2015; Abadi et al., 2016; Bradbury al., 2018); newer AD libraries(Oktay et al., 2020; Paszke et al.. 2021); deep learning libraries (Chollet et al., 2015); extensions to AD libraries (Dangel et al., 2020: Rogozhnikov, 2021) to frameworks that use these AD libraries as tools to implement algorithms (Krieken et al.. 2021: Tran et al., 2016; Gardner et al., 2018). Moreover, some libraries do not directly build on these frameworks, but can be combined together, e.g., Ray (Moritz et al., 2018) for distributed computing. Creating each one of these libraries takes significant effort, and it makes sense to build on the work of others to simplify implementing new algorithms. Due to this, new implementations tend to follow the old design paradigms of existing software.
Barham and Isard (2019) described this situation with the sentence "Machine learning systems are stuck in a rut". They argued that current ML systems are optimized for computations encountered by popular algorithms, yet, other calculations may be slow, despite in principle requiring a similar amount of operations. Such an imbalance would discourage researchers from trying new ideas that require novel forms of computation. Our own observation is that existing AD frameworks encourage implementing algorithms by designing a differentiable "surrogate loss" (Schulman et al., 2015; Foerster et al., 2018) often employed for implementations of Monte Carlo (MC) gradient estimators (Krieken et al., 2021). However, this may not be the most natural way to express the algorithms, or indeed, it may not even be possible. Minka (2019) argued that a general purproe ML language should "simplify implementation of all ML algorithms", and promoted a message passing approach.
Message passing (MP) is a promising candidate for a general ML framework because of the many already existing efficient algorithms that are based on it. Some of the prominent MP algorithms are belief propagation (Pearl, 1982) and expectation propagation (Minka, 2001) in probabilistic modeling. and turbo codes in information theory (McEliece et al., 1998). In modern deep learning. MP is used in graph neural networks (Wu et al., 2020). Also, back propagation can be seen as an MP algorithm that passes the gradient messages backwards. A closely related framework to MP is the Actor Model of computation (Hewitt et al., 1973), where programs operate by actors passing messages between each other. The Actor Model has been suggested as promising framework for parallel Al systems (Hewitt and Lieberman, 1983) and has been noted to enable modularity (Agha, 1986). We aim to incorporate these strengths into a practical ceneral-purposs message passing framework that enables conveniently implementing a variety of MP algorithms, and propose automatic propagation (AP).
We list our desiderata for a general-purpose framework, and the approach we take to meet these desired properties in Proppo, our prototype AP software package.
Ease of use. Proppo is implemented entirely in Python, a language favored by ML. researchers.
Flexibility. Proppo allows implementing arbitrary message passing algorithms.
Computational speed. Proppo interfaces with existing computational frameworks such PyTorch to achieve efficient computation. Proppo adds functionality to such frameworks; the computational speed will depend on how the user implements their algorithm.
Reusability of code. Proppo allows packaging algorithms into propagators- the key building blocks of AP coftware--thr encapsulate an algorithm and can be reused across different computation graphs.
Composability. Proppo includes special sequence propagators (See. 3.3) that allow combining multiple propagators together to form new propagators. These new propagators may then be further composed, allowing to deal with arbitrary complexity.
The main functionality of Proppo is to provide base classes for the propagators, and to implement automatic message passing. When user wants to implement a new algorithm, they need create the propagators to represent their algorithm as a message passing program. When a user wants to use an existing algorithm, they can simply download the existing propagators, and apply them in their code.
2

---

To demonstrate the utility of Proppo, we use it to implement the total propagation (TP) and Gaussian shaping (GS) gradient algorithmss (Parmas et al., 2018; Parmas, 2018). Both of these are techniques for MC gradient estimation without previously existing publicly available implementations, and they were one of our key motivations for this research. particular, the probabilistic computation graph framework (Parmas, 201 8, 2020) influenced our thinking on MC gradient estimation, as it makes clear that MC gradient estimators can be implemented as MP programs passing information backwards. Our experimental results in Sec. 4.2 and App. C.3 demonstrate orders of magnitude improvement in gradient accuracy for these methods. The improvement was particularly large when the system was at the edge of chaos. This implies that AP software can achieve fundamentally better computational results compared to the current AD software paradigm. To demonstrate the scalability and practicality of Proppo, in concurrent work (Anonymous, 2023) we use Proppo to replicate the model-buse -based RL results of (Parmas et al., 2018) and we apply TP to the high dimensional visual MBRL algorithm Dreamer (Hafner et al., 2020). All our results imply that Proppo is scalable, leads to more modular code, simplifies implementation of some algorithms and can lead to greatly improved performance.
2 Preliminaries
2.1 Automatic differentiation software
A recent review of AD in MI. was written by Baydin et al. (2018). To explain the functionality of AD software, suppose that user has written program to compute the output of function y f(x) given the input æ. In such a situation, AD software can automatically obtain the gradient V (x). Such functionality is achieved by differentiating each intermediate computation and applying the chain rule. A series of computations can be represented as a directed acyclic graph with nodes and their associated variables H; and with an edge from node i to j when the variable 2y is computed with function fi ..) taking the variable at node i as an input. In this case, the total derivative from to y can be obtained by summing the product of derivatives across all paths between the two nodes (Bauer, 1974),
drk Ox
V Fe fa)
Σ
II
PachEPaths(--13))
Edge[[ [akc][=+k)ees Eath
naive way to implement AD would to compute the intermediate Jacobians Oes by differentiating del
the operations fo ..)....... However, note that this will require storing full Jacobian matrices having size K T., where K and I. are the respective dimensionalities of the variables. Moreover, if the gradient were computed by multiplying the Jacobians in the forward direction from x to y. this would require matrix-matric multiplications each step, and may incur a large computational cost. Reverse mode automatic differentiation (also known as back propagation). overcomes this issue by performing the computations backwards from the y node, As y just I dimensional scalar, its local Jacobian is a vector. Therefore the computation at the last step is a vector-matrix product, yielding another vector. This vector is propagated backwards, yielding a vector at each backward step. This allows computing the total gradient using only vector- matrix products, leading to large computational saving compared to computing the gradients the forward direction using -aatrix products.
Another subtle point is that it is not necessary to compute the intermediate Jacobians
dzg, one Ban
merely needs a means to compute oJ Jey where v is the back propagated gradient vector. So in practice, AD software often only stores the variables necessary to evaluate, v 0æs and never 8es, explicitly computes 8iz This leads to further computational and memory savings, as the full Jacobian matrices with size K L need not be stored. In PyTorch, this is implemented in the torch. torch.autorn autograd backward (tensors, grad_tensors) function, where tensors corresponds to Hg and gradg tensors corresponds to v. All in all, AD software implements fordwa forward functions to cache the necessary data at each node, and backward functions to evaluate the rector-Jacobiaa product at each node using the cache, resulting in efficient and automatic computation of the gradient.
Daj
2.2 Basics of Monte Carlo gradient estimation
In ML., the computations may be stochastic, and the objective is an expectation J= =R [f())). In this case, it is not clear how to directly differentiate the stochastic operations, and we use
3

---

Monte Carlo (MC) methods to obtain an estimator, 9. for the gradient d.f methods have been reviewed in ML. by Mohamed et al. (2020). The two major MC gradient estimators are the reparameterization (RP) (Rezende et al., 2014; Kingma and Welling, 2013) and likelihood ratio (LR) (Williams, 1992; Glynn, 1990) gradient estimators. The LR method uses f()) to construct an estimator, whereas RP uses V=f(2). Total propagation (TP) (Parmas et al., 2018) combines LR and RP using inverse variance weighting. Moreover, TP does so a step-wise level at each sampling node. The required step-wist wee- computations interfere with the natural operation of back propagation, necessitating AP software to implement TP. We give more background on these methods in App. A.
d3 s.t. E E
dj Such dă
3 Proppo explanation
Propagation Graph
Propagation Manager
Propagation Node
Computation Graph
Computation Node
Figure 2: Automatic propagation framework (Sec. 3). A subset of the nodes O are designated as propagation nodes O forming the propagation graph. The propagation manager ? traverses the propagation graph and activates the propagators to ensure the correct functioning of the implemented algorithm. Here, the propagation graph is illustrated as a chain, but may also have a graph structure. The gray dashed lines show the corresponding same nodes in the propagation and computation graphs.
schematic illustration of automatic propagation software is provided in Fig. 2. Automatic propaga- tion is not alternative to existing frameworks, such as AD packages, rather, it adds functionality top of them, and can be used conjunction. We first explain the Proppo framework, its simplest explain the different components of
form, which is specific implementation of AP software; the framework (Sec. 3. then the processes by which they operate (Sec. 3.2). Finally, we explain AP from more general perspective (App. B.1), loosening some of the specific choices in Proppo.
3.1 Components of Proppo
Computation nodes. Automatic propagation software can be combined together with any other computation package program code. These external packages can be used regularly as one would usually write code. However, at certain locations in the code, the user can specify special computations that will determine what algorithm is imple- mented using the AP software. These special computations will create propagation nodes O.
Propagation nodes. The propagation nodes determine the structure of the operations per- formed in the AP software. In Proppo, each propagation node includes the three components: (1) pointer to propagator -O- that determines the forward and backward computations be- longing to the node, (2) a cache/memory that stores necessary information for the computations occurring at the node, (3)= a message box for incoming messages X from other propagation nodes.
Propagators. The propagators are the key components of AP software. The propaga- tors determine the computations performed at each propagation node. In Proppo, each propagator implements a forward and backward mode of computation. The forward function is used when the node first created; may take an input from the program code, perform com- putations, store data in the memory of the node, and produce an output. The backward function used in later processing; it performs computations using the stored data in the memory and incoming messages to the node, and sends messages X to other nodes in the propagation graph.

---

Messages. Globally, the algorithms are implemented by passing messages between the prop- agation nodes. Each message includes the two components: (1) the contents of the message. (2) an address determining where the message should be sent. A typical address may be pointer to the parent node of the current propagation node, so that the messages are passed backwards.
Propagation manager. In Proppo, all of the messages are first passed to the propagation manager, then the propagation manager decides where to send the message based on the address in the message. The propagation manager also has other important roles: constructs the propagation graph, keeps track of the nodes, activates the propagators and passes messages.
3.2 Typical program flow and usage of automatic propagation software
Usage AP software can be broadly categorized into two: (1) implementing a new algorithm into an AP package, (2) using algorithm already implemented in AP software. We start by explaining how one would typically use an already existing algorithm packaged into AP software. We begin by explaining minimalistic code snippet in Fig. 3 of MC gradient estimation with Proppo.
1 # Configuration code
2 prop RPProp (config) Choose RPProp or LRProp
(PProp(cenfig)
3 manager PropagationMausger
defaull..propgaatorrpp
ttrch. corch (batch_ size. nun_dinensions t range (borizon_lengtt))
Progran cade
randn 6 state torch.randa (batch__sse nun_dimansions)
range (borizon,length)):
state rnn (state)
state ran (atate)
state manager (atxerd
10
11 loss loss_func (stato)
loss fun__func(state manager.. backuard(los))
12 loss backward
(a) PyTorch code
(b) Proppo code combined with PyTorch code
Figure 3: Code for estimating the gradient w.r.t. the parameters recurrent neural network (rnn).
Minimalistic code example. In the example Fig. 3a, batch of initial states (state) is sampled from Gaussian distribution (lines 5-6), this batch is pushed through the recurrent neural network (rnn) for horizon horizoh_length times, then a loss is computed using the function _oss_func, finally the gradients are backpropagted from the loss node by calling loss. backward (). As explained in the preliminaries (Sec. 2.1). the gradients will backpropagate until the parameter node in the computation graph. The regular Py PyTorch code in Fig. 3a computes the gradient using back propagation; the Proppo code in Fig. 3b modifies the computation by adding noise into the forward computation, and estimating the gradient using either RP gradients LR gradients. Conveniently, only the definition of the propagator on line has to be changed to switch between using LR or RP gradients. Moreover, settings such as whether the LR gradient should without having to modify the program code. Finally. note that instead of changing the behavior use MC gradient estimators by using RPProp or LRProp, one could also simply use standard back propagation by setting an appropriate different propagator in the configuration. This was a minimalistic example to give taste of how Proppo might be used. In practice both the configuration and program code may be more complicated with multiple types of propagators, We will explain how existing AP algorithms would typically be used with reference to this minimalistic example.
baseline not can be specified in the config ig
Using an AP algorithm. The code for using AP software can be broadly separated into two: the configuration code that sets up the AP software and selects the algorithm be used, and the program code that determines how and where the algorithm is applied. It may be beneficial to keep these sections of the code distinct improve the readability and modularity.
In the minimalistic example Fig. 3b the configuration code is on lines 2-4. When configuring the code, the user instantiates the propagator and propagation manager objects from their existing class code, while giving them the appropriate configuration. propagator may be created using
propagator Propagator (contfi)
There are different propagator classes, and the user would use different command for the different types of propagators. Some propagators may allow for configuration input to make small tweaks to
5

---

the behavior. The propagation manager may be created using
manager PropagationWanager (config).
Here the configuration input may define default propagators to be used for specified computations, default message passing styles, etc. The configuration in the RNN example was simple, but in general the configuration may be more complicated and include setting up many different types of propagators and equipping them to the manager by assigning them names by which the manager identifies them.
After selecting the desired algorithm by properly configuring the AP software, the user must modify the program code to make it compatible with the AP software and properly implement the algorithm. In the minimalistic example in Fig. 3b, the program code is on lines 6-12, and the modifications to use Proppo are on lines and 12. We emphasize that the same modifications may be compatible with multiple algorithms, so that only the configuration code has to be modified to switch between the algorithms. The main purpose of the modifications to the program code are to activate the propagators and construct the propagation graph. A propagator is activated using the command
output manager . forward (input, arguments).
This command will add a node into the propagation graph of the manager. In the current case, the manager has been equipped with the default propagator prop. When manager calls the forward function, it will create new node, and assign the prop propagator to it. Then, the forward method of prop is called with the input. The forward method applies some modification to the input, stores data in the memory of the node, and produces an output. Note that the data stored in the memory of the node will be used later during the backward pass of the algorithm.
The arguments input to the forward function serves multiple purposes. One of the main purposes is to modify the behavior of the propagator in some way. For example, in the minimalistic code in Fig. 3b the propagator injects noise into state produce the output, so one part of the arguments may be the type of noise and the parameters of the noise distribution. Note that often these settings can be defined in the configuration without having to modify the arguments, c.g. the type of distribution may just be fixed in the configuration code. Another role of the arguments is to modify the targets where the messages should be sent if the default setting is unsuitable. Furthermore, multiple types of propagators are necessary to implement the algorithm, the arguments may also be used to select between different propagators equipped to the manager.
Finally, the backward pass of the algorithm is invoked by calling
manager.. backward
When the manager. backward function called, the manager will typically call the backward function of the propagator of the last node, pass the message to its target node, call the backward function of the previous node, and iterate this process from the end of the graph until the beginning. If the propagators are correctly implemented, and the propagation graph is correctly constructed, this framework allows automatically using an algorithm available in the AP software.
A keen reader may have noticed that in the minimalistic example in Fig. 3b an additional loss input was given to the backward call. This is a convenience implemented in Proppo, where the manager will automatically add propagator corresponding to the loss as the final node before starting to iterate backwards on the propagation graph. Alternatively, the same operations could have been implemented by the commands manager append_loss (loss) followed by manager. backward (). Internally, append_ loss (.) simply calls the forward method of a special propagator designed for adding a node corresponding to a loss. In Proppo, yet another way how the lines 11-12 for adding the loss could have been implemented is the following: manager.. apwend__pppp____p__ sopped_llsssee moad__osssstttte (state, 1ossfunc=lose func) fol- lowed by manager. backward (). In this casc, the loss function loss_ ffnc is passed to the prop- agator directly via the lossfunc keyword, and the propagator will compute the loss by inputting state into loss_ func. The last method of implementation is beneficial for compatibility between different types of loss propagators, for example, it is useful for easily switching between the loss operations necessary for Gaussian shaping gradients and regular loss computation. We that there are many different ways to implement the same algorithm using propagators.
Creating an AP algorithm. In the previous section, we saw that the functionality of an AP algorithm is determined by the implementation of the propagators. Consequently, creating a new algorithm requires implementing the appropriate propagators corresponding to the desired algorithm.

---

In Proppo, several base classes for new propagators are already provided, and the user must only implement the forward and backward methods of the propagators.
In principle, the forward and backward methods can contain arbitrary code, and there is no restriction what they can do. The programmer must figure out how implement their algorithm through a process of storing data in the forward pass, and further computations and message passing in the backward pass. The programmer must also decide how many different types of propagators are necessary for their algorithm, and also how they are intended to be used in the program code. Like with any software, good design of the code and clear documentation are crucial for usability.
Throughout our discussion of AP software, we have promised composability and customizability of the algorithms. We note that merely using AP software to implement algorithm does not guarantee that the algorithm will be composable. It is up to the creator of the algorithm to design the propagators so to facilitate such desirable properties. However, good framework by which to construct an algorithm in a composable way. In particular, in Sec. 3.3 we highlight common propagator types that can lead to composable and flexible algorithms.
find that the paradigm of propagators is
3.3 Example elementary propagators
In AP software, composability is achieved by deriving new propagators from already existing propagators. The simplest form of this may be connect propagators into a chain, and activate their methods in sequence. In Proppo, we provide two propagators with such functionality, ComboProp and ChainProp. While both of these methods activate the forward and backward methods of the constituent propagators sequentially, they differ in how the messages and nodes are handled.
ComboProp. This propagator is a simple way to connect propagators into chain. The output of the forward method of one propagator is fed into the input of the next propagator, and each propagator operates on the same node. Likewise, the backward methods use the data the same node, and the messages are simply combined together and passed to the backward method of the previous propagator. ComboProp is suitable for simple combinations of propagators: however, can run into problems when trying to implement more complex behaviors. Namely, the different propagators may overwrite data stored by other propagators leading to unexpected consequences.
ChainProp. This propagator provides a more systematic way to combine propagators, so as to overcome the sometimes unpredictable behavior of ComboProp. To rigorously define the behavior, ChainProp internally creates its own propagation graph, connecting nodes with propagators into chain, and assigning propagation manager to handle the message passing. The constituent propagators may themselves also be of the ChainProp type. This way, propagation graphs can be embedded into other propagation graphs, creating hierarchy of connected propagators all handled by their own managers. This system allows safely composing propagators to define complex behaviors.
BackPropagator. This example propagator is often composed with a base propagator using ComboProp. In Pytorch, the torch. autograd. backward (tensors, grad_tensone allows inputting lists of tensors and gradients, so that the backpropagation can be commenced at multiple tensors at the same time using a single call. In the composition, the base propagator produces tensor and the gradients, and BackPropagator commences the back propagation on the AD computation graph. Importantly, usually the config of the propagator allows determining whether BackPropagator should be included in the combo sequence or not. This is useful, if one wants to omit immediately backpropagating gradients in favor of grouping multiple tensors together and initiating each backprop simultaneously with a single call.
method
3.4 Example Monte Carlo gradient estimator propagators
One of our main motivations for developing AP software to enable easy implementation of advanced MC gradient estimation methods, such as total propagation or Gaussian shaping gradients. We provide detailed pseudo code for these propagator implementations in App. B.2. Here, we briefly outline some of the more prominent propagator implementations.
Reparameterization. In PyTorch, RP gradients are already implemented, and it is possible to obtain reparameterized samples from a distribution, so that the gradients automatically flow through. Al- ternatively, the output can be detached, and the gradients passed backwards using BackPropagator.
7

---

Likelihood ratio. Typically, LR gradients are implemented in AD software by constructing surrogate loss (Schulman et al., 2015) of the form p(;; fx)), and then differentiating this w.r.t. 0 to obtain the LR gradient. In Proppo, we do not need to construct such surrogate loss. We instead store logp(x; 0) in the node as a tensor, then during the backward pass, we set f(x:) as the input gradient, and back propagate it into log p(:; f), using the BackPropagator in Sec. 3.3.
Total propagation. TP requires storing both data for RP and LR in its node. In addition. requires setting a target tensor in the AD computation graph where the variances of LR and RP will be computed for the inverse variance weighting. We implement this by storing a pointer to the target tensor, and use tor corch. autograd grad to compute the LR and RP gradients. After computing the gradient variances, the mixture weight can be computed, and BackPropagator from Sec. 3.3 is used for one last back propagation with the combined gradient.
3.5 Implementation and computation speed considerations
Primarily, Proppo does not perform any major computations on its own; it merely manages the operations performed by other computational tools (Sec. 3). This overhead is negligible for moderate size computations as confirmed by benchmarking experiments (App. B.3) and real applications (Anonymous, 2023). Experiments also confirmed that Proppo can scale to millions of propagation nodes (App. B.4). The scalability could be further increased by running multiple propagation managers on the propagation graph in parallel, and we plan to implement this in future work.
4 Chaotic net minimalistic experiments for comparing gradient estimators
In this section show minimalistic easily reproducible experiments using the Proppo framework. The main aim is to demonstrate that there exist situations where TP and GS gradients can improve the gradient accuracy by multiple orders of magnitude compared to conventional gradient estimation methods. The main result for TP is in Fig. 4, while the experiments for GS are in App. C.3.
4.1 Setup: gradient variance of a recurrent neural network with chaotic behavior
In our experiments, we estimate the gradient, of an expected loss Ep(nn)) [xx), w.r.t. a parameter 3, using various different MC gradient estimation methods. We repeat this estimation many times, and compute the empirical variance of the gradient estimators, V [g]. from the set of estimates (M) Jk=1- In terms of the particular situation we consider, we were motivated by PIPPS (Parmas et al., 2018), where they showed a situation where chaos-like like properties of the dynamics caused the RP gradient variance to explode, while LR gradients were robust. Parmas et al. (2018) considered a cart-pole swing-up up and balancing task; while this task appears simple, there still redundant complexity as the control policy had nigh-dimensional parameter. Moreover, while the fractal input-output patterns that they visualized are strongly suggestive that the system was chaotic, it falls short of a rigorous proof. Therefore, we devised a similar simplified system based on recurrent neural network (RNN).
We consider a particular 2-dimensional sigmoid recurrent neural network that was mathematically proven to exhibit chaotic behavior by Wang (1991). Formally, the system dynamics are given by
ct+1 Sigmoid(BWar xx) +
5 5
where Sigmoid(y) == TFexp(-9] W-
3 is the inverse temperature, a variable parame-
25 25
ter of the network, s ~ (0, Io²) is Gaussian noise added to the state at each time step, a = 0.001 is the standard deviation of the added activation noise, and the initial state is zy |0.35; 0.55). We use a batch size, B = 1000, and simulate this system for time steps. We compute a quadratic loss at the final time step L(x#)) 1/2 (I 1)((HH 1) and estimate the influence through all time steps to obtain the gradient w.r.t. B. Wang (1991) explained that / controls the dynamics of the system. When is small, the system is well-behaved, but as 3 is increased, the system becomes chaotic through period-doubling mechanism. We discuss this system in greater depth in App. C.I.
"Note that Wang (1991) did not add any noise to the system, but the system still shows similar behavior even when such noise is added. We added this noise be able use LR gradients.
8

---

Horisoo
Hotizon
Horzonn 50
Horiwon 100
10a
10P
107
Crudant variaase
10-"4
10"
10'
10-1
10
10-1
10-11
10
10-5
10-3
10-1
10
2.0
3.0
1.0
2.0
3.0
1.0
2.0
3.0
2.0
3.0
Iniereee lemperaturr, 8
Inveree temperaturr, 8
Inverse temperaatinn,
/
Inverse tempuratiirs, 3
Total propagation
Likelihood ratio
Reparameserization
(a) Variance of RP, LR and TP gradient estimators against the inverse temperature 3 for various horizons.
Horizon
Horizon 25
Horzznn 50
Horizon 100
10'
10'
10P
10F
10F
10'
101
107
Cradleat
10
2.0
Invense tempersture, 8
Inveree temperatuue, ,
Enverse temperatture, /
Inverse temperstture, 3
Total propagation, ratio of improvement
Optimal naively combined gradient
(b) Improvement of TP over the best between LR and RP for each B.
Figure 4: Comparison of the RP. LR and TP gradient estimators the chaotic RNN task. See Sec. 4.1 and Fig. 9 for the description of the task, and Sec. 4.2 for the discussion of the experiment. The main result that TP provides a large improvement at the edge of chaos regime around 2.5.
4.2 Single path experiment: showing the advantage of total propagation
Experimental setup. We consider an experiment on the simple RNN system (Sec. 4.1). We compute the gradient variances of RP, LR and TP gradients, and plot these against B. We considered the horizons H E (10, 25, 50, 100). The batch size was B = 1000 for gradient estimate, g (averaged over the batch), and we performed this estimated K 800 times. We compute the empirical gradient variance of the K estimates, (g(*)) Jk- We use the percentile bootstrap method to estimate 95% confidence intervals. The results are in Fig. 4.
Results. The gradient variances are plotted in Fig. 4a. We see that RP is much more accurate than LR at low 8, but the gradient variance of RP explodes as the system transitions chaos around B = 2.5. This effect is stronger as the horizon is increased. Fig. 4b shows the ratio of improvement of TP over the best between RP and LR at each B. We see that around the edge of chaos at / = 2.5 TP improves by up to around 100 times compared to the best choice between RP and LR.
Discussion.
Previously (Parmas et al., 2018) already showed an example where the RP gradient variance explodes. They also showed that LR was robust to this issue, and demonstrated that TP improves the performance. However, in their results, while the improvement of both TP and LR was huge compared to RP in the chaotic regime, the improvement of TP over LR was comparatively small, up around a times improvement. Here, on the other hand, we showed that TP can simultaneously improve over both RP and LR by multiple orders of magnitude. We also elucidated that the improvement may be particularly large at the edge of chaos. The simplicity of our RNN example allowed examining the advantage of TP in greater depth with better reproducibility.
In Fig. 4b, we have additionally plotted the performance for an optimal "naive" gradient combination based on applying inverse variance weighting to LR and RP computed separately, 5 without taking advantage of the graph structure of the computations. Such a naïve method was first mentioned by Parmas et al. (2018) who immediately skipped it for TP, because of its conceptual issues; however,
"Strictly speaking, we did not actually combine the gradient estimates, but only computed the variance of an idealized version of this estimator by using the precise variances V [gæz] and V [g.x]. In practice, one also has to estimate these variances and the mixing ratio from each batch of samples.

---

later Metz et al. (2019) "followed the insight" from Parmas et al. (2018) and used the naïve method in the context of metalearning. In Fig. 4b, we see that the naive method performs poorly, leading to hardly any improvement in gradient accuracy compared to the best between RP and L.R.
Indeed, when combining statistical estimators, the theoretical maximum accuracy is the sum of the accuracies of the individual estimators. If two estimators are equally good, this would maximally lead to doubling of the accuracy. However, in Fig. 4b we see improvements far exceeding this limit. Such a result was possible because TP takes advantage of the graph structure of the computations. It not only improves the accuracy of the final result, also improves the accuracy of the intermediate gradient computations that are propagated backwards. The improvements in the accuracy of the intermediate calculations have a compounding effect, leading to orders of magnitude improvement of the final result. This advantage of TP highlights that automatic propagation software such as Proppo can produce results unattainable by standard back propagation procedures.
5 Related work
From the point of view of MC gradient estimation, the concurrent work, Storchastic (Krieken et al., 2021) is the most closely related to ours. Storchastic is library for MC gradient estimation implemented in PyTorch based on the surrogate loss formalism (Schulman et al., 2015; Foerster et al., 2018). Their user interface of using a backward function is similar to ours, as both were inspired by the PyTorch interface that is the same. However, Storchastic not an AP software, and for instance, it does not enable implementing Total Propagation.
From conceptual point of view, the Actor Model (Hewitt et al., 1973) related. In the Actor Model, everything in the program are actors and they communicate with each other via message passing. However, practically, Proppo is not implemented as an actor system. Unlike in an actor system, the propagators acting at the nodes can interact with outside objects directly without explicitly implemented messaging mechanisms. From theoretical point of view, one could consider these interactions as "passing messages" but practically, these interactions are not implemented based on the actor formalism. Instead, AP software aims to be pragmatic and flexible to enable interusability with existing ML tools. A further difference is that AP automates the default messaging targets. Prominent Actor Model implementations, such as Akka (Bonér al., 2010) are typically used for server and web apps, and they are not primarily meant for implementing machine learning algorithms. Nevertheless, studying existing actor systems may inspire improvements to AP implementations.
6 Discussion & Conclusions
Recently, many impressive results were achieved by leveraging large scale data and computation together with basic algorithms (Brown et al., 2020; Ramesh al., 2022; Chowdhery al., 2022). We ask whether simple algorithms are used, simply because researchers and engineers have difficulty with properly implementing more complicated ones? There is guarantee that merely scaling up existing algorithms will be sufficient for satisfactory performance in the long term of ML research; we must also explore algorithmic improvements. We hope that automatic propagation will enable the proliferation of complex algorithms by providing a standard for sharing and building ML tools.
Author Contributions
Paavo Parmas invented automatic propagation software, designed and implemented most of the code including the key parts, the experiments, and wrote the paper. Takuma Seno worked part-time to help write the code and give advice on the software engineering aspects in the early stages of the project. He also gave comments the paper.
Acknowledgements
The project was started with funding provided by the Proof of Concept Program at the Okinawa Institute of Science and Technology (OIST). Most of the key contributions were made during this stage. PP finished the project while hired under the Cyborg-Al project supported by NEDO, Japan. PP would like to thank the Neural Computation Unit OIST for administrative support while the project was being conducted at OIST.
10

---

References
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, Devin, M., Ghemawat, S., Irving. G., Isard, M., al. (2016). (TensorFlow): A system for TLarge-Scaler machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16), pages 265-283.
1
Anonymous (2022). Anonymous. Anonymous, page 0000. 1, 3.5, B.3
Barham, P. and Isard, M. (2019). Machine learning systems are stuck in a rut. In Proceedings of the Workshop Hot Topics in Operating Systems, pages 177-183.
Bauer, L. (1974). Computational graphs and rounding error. SIAM Journal on Numerical Analysis, 11(1):87-96. 2.1
Baydin, A. G., Pearlmutter, B. A., Radul, A. A., and Siskind, M. (2018), Automatic differentiation in machine learning: a survey. Journal of Marchine Learning Research, 18:1-43. 2.1
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke,
A., VanderPlas, J., Wanderman-Milnn S., and Zhang, Q. (2018). JAX: composable transformations of Python+NumPy programs.
Chollet, et al. (2015). Keras, https://keras.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022). Palm: Scaling language modeling with pathways. arXiv preprint arX1v:2204.02311. 6
Dangel, F., Kunstner, F,, and Hennig, P. (2020). BackPACK: Packing more into backprop. In International Conference on Learning Representations.
Fairbank, M. (2008). Reinforcement learning by value gradients. arXiv preprint arXiv:0803.3534 A.2
Foerster, J., Farquhar, G., Al-Shedivat, M., Rocktäschel, T., Xing, E., and Whiteson, (2018). Dice: The infinitely differentiable Monte Carlo estimator. In International Conference on Machine Learning, pages 1529-1538. 1,5
Glynn, P. W. (1990). Likelihood ratio gradient estimation for stochastic systems. Communications of the ACM, 33(10):75-844 2.2
Greensmith, E., Bartlett, P. L., and Baxter, J. (2004). Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research, 5(Nov):1471-1530 A.1
Hafner, D., Lillicrap, T., Ba, J., and Norouzi, M. (2020). Dream to control: Learning behaviors by latent imagination. In International Conference Learning Representations. 1
Jankowiak, M. and Obermeyer, (2018). Pathwise derivatives beyond the reparameterization trick. In International Conference on Machine Learning, pages 2240-2249, A.1
Kingma, D. P. and Welling, M. (2013). Auto-encoding variational Bayes. arXiv preprint 131iv6 2.2
Krieken, E., Tomezak, J., and Ten Teije, A. (2021). Storchastie: A framework for general stochastic automatic differentiation. Advances in Neural Information Processing Systems, 34. 1,
Metz, I.., Maheswaranathan, N., Nixon, J., Freeman, C. D., and Sohl-Dickstein, J. (2019). Un- derstanding and correcting pathologies in the training of learned optimizers. In International Conference on Machine Learning. 4.2, C.1
Minka, T. (2019). From automatic differentiation to message passing. Slides for Presentation at Advances and challenges machine learning languages workshop. [Online; http:: htttps://tmink.... gi github.r it/pei/rs///////p//ooooeeeeeeset /amierssaammm1199011 wwwmaam11/mmmaaaammaaaaa hwimmminnaa//a aan.ka-9mm12219---11 slides...essseeeee slidsss.ddddd.... .df].

---

Mohamed, S., Rosca, M., Figurnov, M., and Mnih, A. (2020). Monte Carlo gradient estimation in machine learning. Journal of Machine Learning Research, 21:1-62. 2.2
Moritz, P., Nishihara, R., Wang. S., Tumanov, A., Liaw, R.,, Liang, E... Elibol, M., Yang, Z., Paul, W.,
Jordan, M. I., et al. (2018). Ray: A distributed framework for emerging {AT) applications. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pages 561-577.
Oktay, D., McGreivy, N., Aduol, J., Beatson, A., and Adams, R. (2020). Randomized automatic differentiation. In International Conference on Learning Representations. 1
Owen, A. B. (2013). Monte Carlo theory, methods and examples. A.1
Parmas, (2018). Total stochastic gradient algorithms and applications in reinforcement learning. In
Advances in Neural Information Processing Systems, pages 10204-10214. 1, A.1, A.2, A.2, A.2,
A.2, A.2, C.3
Parmas, P. (2020). Total stochastic gradient algorithmss and applications model-based reinforcement learning. PhD thesis, Okinawa Institute of Science and Technology Graduate University. A.1
Parmas, P., Rasmussen, C. E., Peters, J., and Doya, K. (2018). PIPPS: Flexible model-based policy search robust to the curse of chaos. In International Conference on Machine Learning, pages 4062-4071. 1, 2.2, 1.1, 4.2, 1. A.2, A.2, C.1,8 8
Parmas, P. and Sugiyama, M. (2021). A unified view of likelihood ratio and reparameterizztion gradients. In International Conference on Artificial Intelligence and Statisties, pages 4078-4086. PMLR. A.1
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L.,, et al. (2019). Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
Paszke, A., Johnson, D. D., Duvenaud, D., Vytiniotis, D., Radul, A., Johnson, M. J., Bagan-Kelley J., and Maclaurin, D. (2021). Getting to the point: index sets and parallelismmprreervving autodiff for pointful array programming. Proceedings of the ACM on Programming Languages, 5(ICFP):1-29.
Pearl, J. (1982). Reverend bayes on inference engines: a distributed hierarchical approach. In Proceedings of the Second AAAI Conference on Artificial Intelligence, pages 133-136. B.1
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. (2022). Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125. 6
Rezende, J., Mohamed, S., and Wierstra, (2014). Stochastic backpropagation and approximate inference in deep generative models. In International Conference Machine Learning, pages 1278-1286. 2.2, A.I
Rogozhnikov, A. (2021). Einops: Clear and reliable tensor manipulations with einstein-like notation. In International Conference on Learning Representations.
Schulman, J., Heess, N., Weber, T., and Abbeel, P. (2015). Gradient estimation using stochastic computation graphs. In Advances in Neural Information Processing Systems, pages 3528-3536. 1, 3.4, A.I
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. (2014). Deterministic policy gradient algorithms. In International conference on machine learning, pages 387-395. PMLR. A.2
Sutton, R. S., McAllester, D. A., Singh, S. P.,, and Mansour, Y. (2000). Policy gradient methods for reinforcement learning with function approximation. In Advances neural information processing systems, pages 1057-1063, A.2
12

---

Tokui, S., Oono, K., Hido, S., and Clayton, J. (2015). Chainer: a next- generation open source frame- work for deep learning. In Proceedings of workshop on machine learning systems (Leariingyss) in the twenty-ninth annual conference on neural information processing systems (NIPS), volume 5, pages 1-6.
Wang, X. (1991). Period-doublings to chaos in a simple neural network: An analytical proof. Complex Systems, 5(4) 441. 4.1, 4.1, 4, C.I,8 8
Weaver, and Tao, (2001). The optimal reward baseline for gradient- -based reinforcement learning. In Proceedings of the Seventeenth conference Uncertainty in artificial intelligence, pages 538-545. Morgan Kaufmann Publishers Inc. A.1
Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8(3-4):229-2) 256. 2.2
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A] Our paper about a framework, not direct application.
(d) Have you read the ethics review guidelines and ensured that your paper conforms
them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] No novel theoretical results.
(b) Did you include complete proof's of all theoretical results? [N/A] No novel theoretical results.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [No] The experiments are simple, well documented and easy to reproduce. The code is available at at https://github. com/proppo.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No] The experiments in this paper were small scale run on CPU that anyone can replicate.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/]] The used assets are well-known, e.g., PyToreh, so this is redundant.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] We included code in Appendix. D. The code is also available at https: /::::://ttttut com/
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] We did not use or curate data.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] We did not use curate data.
5. If you used crowdsourcing conducted research with human subjects...
13

---

(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "KSfkoBpbjF7", "reviewer": "Reviewer_KJqN", "summary": "This paper introduces a class of software called Automatic Propagation (AP), an extension of autodifferentiation software as well as a specific instance of it called Proppo. AP allows users to implement custom learning algorithms into the base unit of a \"propagator\" that can be reused. The authors note that many ML software packages adhere to the same paradigm of a surrogate loss to be used with backpropagation, while other forms of calculations are not optimized for or considered as they are not the standard. As such, they design AP to be a general message passing framework for ML.  The basic unit of Proppo is a propagator that implements both a forward mode (which performs computation, and stores any necessary data for the backward mode ) and backward mode of computation, which performs backward computation sending and recieving messages to other nodes. The whole process is managed by a propagation manager. \n\nThe utility of Proppo and the AP general system is demonstrated by the implementation of total propagation and gaussian shaping gradient algorithms for MC gradient estimation, which are then shown to perform (in terms of gradient accuracy) orders of magnitude better than the standard surrogate loss + autodiff method for certain classes of problems.", "questions": "In the minimalistic code example section starting on line 163, it could be useful for some readers to understand how Proppo configuration code could be modified to do the default pytorch behavior (normal backpropagation) (I acknowledge that you introduce the BackPropagator later but it might provide some clarity to new readers if mentioned in the first minimal code section that you could simply also use backprop in addition to changing the behavior of the code completely.\n\nI suggest rewording the sentence on 167-169, as it is confusing what the intention is due to the grammatical errors. \n\nIf you could open source some code for Proppo, it would make an even more impactful contribution.", "limitations": "The authors have addressed some limitations of the work (i.e. that it is ultimately up to the user to use the framework well to achieve the desired aims of composability and customisability. Giving further guidelines and documentation for users on the best ways to use the framework would be worthwhile, especially if open sourced.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper is well written and motivated. The contribution is important and original. The experimental results are thoughtful and validate the claim that there are situations where total propagation algorithm can be more accurate gradient estimator than those that can be achieved by standard AD software. \n\nWeaknesses: The paper does not seem to link to any open sourced code, which would be a nice addition to the paper, to see some examples of how the more sophisticated propagators could be implemented. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "G6q1iI9LBsj", "reviewer": "Reviewer_JEb5", "summary": "The authors present a software framework (Proppo) for constructing propagation algorithms on generic computational graphs. These can be used to implement a variety of learning approaches. The authors make the case that existing ML frameworks, because of their limited capabilities of operating on generic computation graphs, discourage researchers from pursuing particular directions. The authors present example approaches to Monte Carlo gradient estimator and the Total Propagation algorithm implementations using Proppo, then show an experiment demonstrating Total Propagation can improve stability of an RNN-based system with nearly-chaotic dynamics.\n", "questions": "> implemented entirely in Python\n\nGiven that the entire framework is implemented in Python, how does performance hold up for large graphs? Large graphs emerge in many learning paradigms (e.g. transduction, graph learning). Performance benchmarks are missing from the paper.\n\n> [Messages are implemented via] a pointer to a parent node\n\nMore implementation details regarding statements like this would be very helpful.\n\n> propagation managers working in parallel\n\nHow is this accomplished? Threading? Efficient graph traversal?", "limitations": "Proppo as a framework allows users to implement RP and LR algorithms as discussed, but these algorithms are not new, and the authors do not detail why an implementation enabled by Proppo is any better than existing implementations of these algorithms, beyond positing that there may be correctness issues with existing implementations that Proppo helps avoid. These ideas are very underspecified.\n", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "##### Strengths\n\nThe idea of developing a generic framework to perform propagation on computational graphs is important and indeed opens the doors for many new learning algorithms that may not be based on gradient-based learning or may involve gradient computation with higher order or other methods. There is a need for such a generic framework to implement such algorithms.\n\n##### Weaknesses\n\nThe paper is very light on the details of the framework itself — the APIs, semantics for implementing other learning algorithms, or implementation details are not explained at all in neither the paper nor the appendix. This makes verifying and validating the quality and correctness of the authors’ implementations difficult.\n\nThe work is missing a set of performance benchmarks, which are especially relevant for Monte Carlo-based estimation methods. Total Propagation or similar methods are not new, and while an implementation of them using a generic framework has utility, it is not clear if such an implementation is competitive computationally with other implementations.\n\nThe authors present no new learning algorithms imagined with Proppo, instead benchmarking existing ones. The paper would be made stronger if the authors were able to present case studies demonstrating the utility of their approach in more detail. The Monte Carlo gradient estimation example in the paper, while an interesting setting in which to study stability, is not convincingly broadly useful. The authors should make such a case and should expand the set of learning settings in which they explore these algorithms, taking advantage of their interoperability with PyTorch.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "E3pjLJE4ZPo", "reviewer": "Reviewer_GmXe", "summary": "Existing automatic differentiation (AD) frameworks do not provide flexibility for composing learning algorithms. This paper invents Automatic Propagation (AP) software, which generalizes AD, and allows the construction of complex learning algorithms.\n\n", "questions": "See \"Strengths And Weaknesses\"", "limitations": "See \"Strengths And Weaknesses\"", "rating": 2, "confidence": 1, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This work serves as, in my opinion, a kind of software illustration that incorporates modern learning algorithms. As a result, I didn't find this paper to make a major academic contribution.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept