Title: Learning to superoptimize programs
Authors: Rudy Bunel
Alban Desmaison
M. Pawan Kumar
Philip H.S. Torr
Pushmeet Kohli
Venue: ICLR 2017

Abstract:   Code super-optimization is the task of transforming any given program to a more efficient version while preserving its input-output behaviour. In some sense, it is similar to the paraphrase problem from natural language processing where the intention is to change the syntax of an utterance without changing its semantics. Code-optimization has been the subject of years of research that has resulted in the development of rule-based transformation strategies that are used by compilers. More recently, however, a class of stochastic search based methods have been shown to outperform these strategies. This approach involves repeated sampling of modifications to the program from a proposal distribution, which are accepted or rejected based on whether they preserve correctness, and the improvement they achieve. These methods, however, neither learn from past behaviour nor do they try to leverage the semantics of the program under consideration. Motivated by this observation, we present a novel learning based approach for code super-optimization. Intuitively, our method works by learning the proposal distribution using unbiased estimators of the gradient of the expected improvement. Experiments on benchmarks comprising of automatically generated as well as existing (``Hacker's Delight'') programs show that the proposed method is able to significantly outperform state of the art approaches for code super-optimization.



Review discussion:
The paper mentions that an MLP was used with three hidden layers, but does not specify any other hyperparameters. How many hidden units? Was the MLP tuned on a separate validation set, or on the final test results  shown in the paper?
Were any other models tried? Seems like logistic regression or random forests could do very well on this task, also.
Thank you for your questions.
The model that was used is the following (in our case the number of opcodes is 3874) :
Bag of Words over the instructions -> Linear Layer (nb of opcodes -> 100) + ReLU -> Linear Layer (100 -> 300) + ReLU -> Linear Layer (300 -> 300) -> Embedding
Embedding -> Linear Layer (300 -> nb of type of move) + Softmax -> Probability over the types of move
Embedding -> Linear Layer (300 -> nb of opcodes) + Softmax -> Probability over the opcodes
I will add an appendix with those details to the paper.
We didn't perform any exhaustive fine tuning on the model's architecture. The only parameter that needed to be selected was the learning rate to ensure that
the training loss was going down.
We haven’t explored the use of other models but any other differentiable architecture can be used in place of the MLP. A vanilla logistic regression baseline would have a lot of parameters ( nb of opcodes * (nb of opcodes + nb of type of move) ~= 15e6.
In contrast, our current model is at 1.7e6 parameters). We will conduct experiments with a linear model and will add the results to the paper. This will show whether or not the depth of the model is actually helping in this context.
Typically, in MCMC-based synthesis, it's important to be able to make a large number of proposals/evaluations. If you're using an MLP to propose changes to a program, does this not impact the number of evaluations per second metric? I would be interesting in knowing how does the proposed method compare with the baselines in terms of the wall clock time to completion. This dimension has consequences wrt the scalability of the approach to bigger synthesis/superoptimization tasks.
Thank you for your questions.
The impact on runtime is negligible. The proposal distribution is
sampled once at the beginning of the optimisation and remains the same throughout
one run. The only difference comes from the fact that Vanilla Stoke samples from
a uniform distribution while we sample from a categorical distribution.
Running a quick experiment on my desktop, using uniform sampling, we can reach a throughput of 200 000 proposal per second. Sampling from the learned distribution, we can reach 50 000 proposals per seconds.
However, the proposal time is negligible in comparison to the evaluation of the cost function so the throughput is similar.
Two things I really liked about this paper:
1. The whole idea of having a data-dependent proposal distribution for MCMC. I wasn't familiar with this, although it apparently was previously published. I went back: the (Zhu, 2000) paper was unreadable. The (Jampani, 2014) paper on informed sampling was good.
So, perhaps this isn't a good reason for accepting to ICLR.
2. The results are quite impressive. The rough rule-of-thumb is that optimization can help you speed up code by 10%. The standard MCMC results presented on the paper on randomly-generated programs roughly matches this (15%). The fact that the proposed algorithm get ~33% speedup is quite surprising, and worth publishing.
The argument against accepting this paper is that it doesn't match the goals of ICLR. I don't go to ICLR to hear about generic machine learning papers (we have NIPS and ICML for that).
Instead, I go to learn about how to automatically represent data and models. Now, maybe this paper talks about how to represent (generated) programs, so it tangentially lives under the umbrella of ICLR.
But it will compete against more relevant papers in the conference -- it may just be a poster. Sending this to a programming language conference may have more eventual impact.
Nonetheless, I give this paper an "accept", because I learned something valuable and the results are very good.
We believe this paper has important implications for representation learning.
Apart from the reviewer’s remark on representing (generated) programs, we believe super-optimization provides a natural benchmark for evaluating representations of programs. This is particularly true in light of recent resurgence of interest  in machine learning on Program Induction/Synthesis and its connections to Hierarchical RL.
Answers to the questions posted by reviewers would help for a more high-quality review. Thanks.
This is an interesting and pleasant paper on superoptimization, that extends the  problem approached by the stochastic search STOKE to a learned stochastic search, where the STOKE proposals are the output of a neural network which takes some program embedding as an input. The authors then use REINFORCE to learn an MCMC scheme with the objective of minimizing the final program cost.
The writing is clear and results highlight the efficacy of the method.
comments / questions:
- Am I correct in understanding that of the entire stochastic computation graph, only the features->proposal part is learned. The rest is still effectively the stoke MCMC scheme? Does that imply that the 'uniform' model is effectively Stoke and is your baseline (this should probably be made explicit )
- Did the authors consider learning the features instead of using out of the box features (could be difficult given the relatively small amount of data - the feature extractor might not generalize).
- In a different context, 'Markov Chain Monte Carlo and Variational Inference:Bridging the Gap' by Salimans et al. suggests considering a MCMC scheme as a stochastic computation graph and optimizing using a variational (i.e. RL) criterion. The problem is different, it uses HMC instead of MCMC, but it might be worth citing as a similar approach to 'meta-optimized' MCMC algorithms.
Thank you for your comments. We provide our responses below.
Comment: Only the features->proposal part is learned?
Response: Indeed, only the mapping from the features to the proposal distribution is learned. The ‘Uniform’ model is indeed Stoke as it was described in the work of Schkufza et al. We will make this more clear in the paper.
Comment: Learning the features?
Response: Our multi-layered network learns a representation on top of the Bag Of Words (BOW) descriptor. As another reviewer suggests, we did consider directly using the program tree as input to a tree structured neural network.
However, this option requires the availability of a large amount of data to learn a generic feature representation, which is unfortunately not available in our current setting.
While we used the BOW descriptor as our base representation, we intend to investigate more generic tree and sequence architectures in the future.
Comment: Related work by Salimans et al.
Response: Thank you for the reference. It is indeed relevant and presents another interpretation of the “traces” of MCMC algorithms. The key difference of our work lies in the objective: approximating the posterior vs. finding the mode more efficiently, for which we optimize directly by specifying it as our reward function. We will add a discussion in the related work section.
"However, this option requires the availability of a large amount of data"
Your data is being generated on-the-fly, right? You technically have infinite data?
While we would technically have infinite data using synthesized programs, these don't correspond to "proper" training data as they lack the internal coherence that actual programs exhibit.
Indeed, as they are obtained from a completely random walk, they exhibit almost no structure and would therefore be of poor quality to learn a structured representation.
This experiment on synthetic data was intended to show that the proposed method was not entirely limited to a restricted class of bitwise manipulations of a few registers. We will make this goal more clear in the paper's revision.
This work builds on top of STOKE (Schkufza et al., 2013), which is a superoptimization engine for program binaries. It works by starting with an existing program, and proposing modifications to it according to a proposal distribution. Proposals are accepted according to the Metropolis-Hastings criteria. The acceptance criteria takes into account the correctness of the program, and performance of the new program.
Thus, the MCMC process is likely to converge to correct programs with high performance. Typically, the proposal distribution is fixed. The contribution of this work is to learn the proposal distribution as a function of the features of the program (bag of words of all the opcodes in the program). The experiments compare with the baselines of uniform proposal distribution, and a baseline where one just learns the weights of the proposal distribution but without conditioning on the features of the program. The evaluation shows that the proposed method has slightly better performance than the compared baselines.
The significance of this work at ICLR seems to be quite low., both because this is not a progress in learning representations, but a straightforward application of neural networks and REINFORCE to yet another task which has non-differentiable components. The task itself (superoptimization) is not of significant interest to ICLR readers/attendees. A conference like AAAI/UAI seem a better fit for this work.
The proposed method is seemingly novel. Typical MCMC-based synthesis methods are lacking due to their being no learning components in them.
However, to make this work compelling, the authors should consider demonstrating the proposed method in other synthesis tasks, or even more generally, other tasks where MH-MCMC is used, and a learnt proposal distribution can be beneficial. Superoptimization alone (esp with small improvements over baselines) is not compelling enough.
It is also not clear if there is any significant representation learning is going on. Since a BoW feature is used to represent the programs, the neural network cannot possibly learn anything more than just correlations between presence of opcodes and good moves. Such a model cannot possibly understand the program semantics in any way. It would have been a more interesting contribution if the authors had used a model (such as Tree-LSTM) which attempts to learn the semantics the program. The quite naive method of learning makes this paper not a favorable candidate for acceptance.
We thank the reviewer for the comments. The main critique is the relevance of the work in the context of representation learning, which we clarify below.
While superoptimization may not directly be one of the main interests of the ICLR audience, it does provide a natural benchmark for evaluating representations of programs. Representations need to be evaluated on tasks and Superoptimization as a task requires the representation to decouple  the semantics of the program (its specification / desired behaviour) from its superfluous properties (the exact implementation).
We hope that our work will spur interest in the ML community on the problem of learning programs representation.
While we used the BOW descriptor as our base representation, we intend to investigate more generic tree and sequence architectures in the future.
We thank the reviewers for their helpful feedback. We uploaded a new version of
the paper. The elements that have changed are:
Clarity of experiments:
- The complexity of each model in term of number of parameters is given in Table 1 and the hyperparameters / architecture were added as Appendix A.
- Addition of a better representation of the consequences of a learned bias (Figure 3). We compare Stoke (uniform distribution)  running for (200,400) iterations with the learned optimizer running for (100, 200) iterations (at training time, the horizon is set at 200 iterations). Even with four times less iterations, a better distribution of the score is achieved.
- The numbers for the impact on throughput of the learned proposal distribution have been added (Table 3). This measures not just the time for the proposal (which we gave as answer to the question of AnonReviewer2) but the whole time of an MCMC iterations which represent more accurately the throughput of the algorithm.
Explanations:
- We added the citation that AnonReviewer3 suggested (Section 2 - Related Works).
- We added our argument that superoptimization provides a good benchmark to estimate representation of programs to the introduction (Section 1 - Introduction, 1st paragraph).
- Clarified throughout the paper that the "Uniform" model is Stoke (Section 3.2.2-Parametrization and Section 4.1-Models, Table 1).
- Fixed some typos

Meta-review:
role: PC
data: comment: This paper applies REINFORCE to learn MCMC proposals within the existing STOKE scheme for super-optimization. It's a neat paper, with interesting results.
 
 It's not clear whether interesting representations are learned, and the algorithms are not really new. However, it's a neat piece or work, that some ICLR reviewers found interesting, and could inspire more representation learning work in this area.

Decision: Accept (Poster)