The Cakewalk Method

1 INTRODUCTION . Combinatorial optimization is one of the foundational problems of computer science . Though in general such problems are NP-hard Papadimitriou ( 2003 ) , it is often the case that locally optimal solutions can be useful in practice . In clustering for example , a common objective is to divide a given set of examples into a fixed number of groups in a manner that would minimize the distances between group members . As enumerating all the possible groupings is usually intractable , local search methods such as k-means MacQueen et al . ( 1967 ) are frequently used to approach such problems . We find the persistent use of k-means in a wide variety of applications as convincing evidence that from a practical perspective , locally optimal solutions can be useful . In the combinatorial setting however , solution neighborhoods are not always available , and even when they are , in many interesting cases they only connect small parts of the search space . For example , when the search space involves computer programs , it is not clear how replacing one operation with another ( for example , an if clause with an addition operation ) impacts the program behavior even if the program validity is preserved . Though one can define a limited but sensible set of neighboring solutions ( e.g. , replace an addition with a multiplication ) , neighborhoods that build on those usually connect only a tiny fraction of the search space . Another interesting case involves natural language sentences where replacing one word with another ( say , ’ very ’ to ’ not ’ ) , or changing clauses order can completely change the meaning of a sentence . A third popular scenario involves sequential decision making as is the case in reinforcement learning problems with discrete action spaces , where it is not always clear that two action sequences can be related if the initial actions are different . In such combinatorial problems , methods that transforms one solution to another ( either directly or through smoothing ) might be confined to a small sub-space , and therefore in such problems , searching the solution space directly is unfavorable . One type of algorithms which are suitable to such combinatorial problems , and have drawn considerable interest in the last few years are policy gradient methods Sutton & Barto ( 2017 ) . The general strategy these methods adopt is to construct a parametric sampling distribution over the search space , and to optimize the expected value of some given objective function by applying gradient updates in the parameters ’ space . In spite of their apparent generality , these gradient updates require special attention . In particular , the sampled objective values affect both the sign and the magnitude of the gradient step size . On the one hand , such dependence on the objective values is what allows these algorithms to give higher likelihood to examples which achieve better objective values . On the other hand , such direct dependence makes it hard to tune the step sizes by means of predetermined hyper-parameters . As our goal is to extend such constructions to any objective in a generic fashion , we seek to transform the construction so that it will only be sensitive to the order relation the objective induces . In this construction however , the objective is essentially a random variable whose distribution changes from one problem to another , and not only that , it keeps on changing throughout the optimization . As a result , it seems that finding a generic rule for tuning various hyper-parameters in a manner that fits all scenarios seems impractical . Following this understanding , we purpose to utilize a generic surrogate objective function that has the following two properties . First , the surrogate should preserve the set of locally optimal solutions if solution neighborhoods can be defined . Second , the surrogate should have a fixed and predetermined distribution for every possible objective , and this distribution should remain fixed throughout the optimization . Once in this form , generic rules for setting various hyper-parameters can be found , and that can provide us with a generic stochastic optimizer . Though it might seem that such general purpose surrogate objectives could be hard to find , we show that by utilizing the empirical cumulative distribution function ( CDF henceforth ) of the original objective , these can be easily constructed . We discuss few possible surrogate objectives , and purpose one such version which makes the basis our method . Since the crux of our method is based on capitalizing on the CDF of the original objective , we refer to our method as CAkEWaLK which stands for CumulAtivEly Weighted LiKelihood . We start by considering policy gradient methods as stochastic optimization algorithms for combinatorial problems in section 2 , and proceed to present the Cakewalk method in section 3 . In section 4 we discuss how Cakewalk is related to the cross-entropy ( CE henceforth ) method , to policy-gradient methods in reinforcement learning , and to multi-arm bandit algorithms . Since we are interested in methods that can recover locally optimal solutions when these can be defined , we use the problem of finding inclusion maximal cliques in undirected graphs as a controlled experiment for testing this property in a non-trivial setting . For that matter , in section 6 we investigate how to apply such methods to the clique problem , and in section 7 we report experimental results on a dataset of graphs on which results are regularly published . Lastly , as an additional experimental task , we show in appendix section B how Cakewalk can be used to produce an algorithm that outperforms the most commonly used algorithms for k-medoids clustering , the combinatorial counterpart of k-means . Notably , we use this task to demonstrate that Cakewalk can also be used to optimize the starting point of greedy algorithms that search the input space directly , thus providing empirical evidence that supports Cakewalk ’ s effectivity in a greater variety of combinatorial problems . 2 BACKGROUND . We set out on constructing a stochastic optimization algorithm for combinatorial optimization problems , and start by stating the problem . Let f be an objective function which we need to maximize , and let x ∈ [ M ] N be a string that describes N items such that each xj is one of a discrete set of M items . In this text we denote discrete sets { 1 , . . . , K } using [ K ] . Our goal is to search the space X = [ M ] N for some x ? that achieves an optimal f ( x ? ) = y ? . Since X is discrete , in general this problem is NP-Hard ( maximum clique can be reduced to this description ) , hence we focus only on finding locally optimal solutions rather than the global optimum x ? . For the purpose of defining locally optimal solutions , we ’ ll rely on a neighborhood function N that maps each x to its neighboring set , though the methods we consider treat f as a black-box , and don ’ t require suchN for their operation . Our goal is to find some locally optimal solution x∗ ∈ X ∗f where the set of locally optimal solutions is defined as X ∗f = { x ∈ X |∀x′ ∈ N ( x ) .f ( x ) ≥ f ( x′ ) } . Preferably , we would like to find some x∗ whose objective value y∗ = f ( x∗ ) is as large as possible , though in general , this can not be guaranteed . We describe a stochastic optimization algorithm for problems of this form . Let X be a random variable that is defined over X , and which is distributed according to a parametric distribution Pθ that the algorithm maintains . In addition , let Y be a random variable that is defined over the values of the objective function f , i.e . Y = f ( X ) . We emphasize that in this text we refer to random variables using capital English letters in bold such as X or Y , and we use x and y to refer to elements in their appropriate sample spaces ( deterministic quantities ) . During the optimization , the algorithm iteratively samples solutions x according to Pθ , and updates the parameters θ ∈ Rd which govern Pθ in a manner that reflects how good is the objective value y = f ( x ) with which x is associated . Initially Pθ ( X = x ) is multi-variate uniform ( fully specified in section 5 ) , but as the optimization continues , the algorithm decreases the entropy in the distribution until eventually only few solutions become likely , and sampling some x from Pθ , with high probability , returns some locally optimal solution . Since we discuss an iterative algorithm that at each iteration t updates the parameters θt , we refer to the random variables that are associated with Pθt byXt and Y t. Lastly , as a short hand notation , we refer to Pθ ( X = x ) simply by Pθ ( x ) . Since we learn a distribution function , we say that our learning objective J ( θ ) is to maximize the expectation over x ∼ Pθ of the original objective which we denote as Eθ [ Y ] . To find the parameters θ which maximize Eθ [ Y ] , we derive a gradient ascent algorithm which relies on estimates of∇θEθ [ Y ] . To calculate the gradient , we use the log-derivative trick , ∇θEθ [ Y ] = Eθ [ Y ∇θ logPθ ( X ) ] . Next , we can use Monte Carlo estimation Wasserman ( 2013 ) to estimate Eθ [ Y ∇θ logPθ ( X ) ] . Traditionally , at each iteration t , a large sample St = { xkt , y k t } K k=1 of some fixed size K is sampled using Pθt . Denoting this estimate by ∆t , then the update at iteration t takes the following form , θt = θt−1 + ηt∆t ( 1 ) ∆t = 1 K K∑ k=1 [ ykt∇θ logPθ ( xkt ) ] ( 2 ) where ηt is a learning rate parameter that is predetermined . We describe the update step using a vanilla gradient update mostly for illustratory purposes , though in practice any gradient based update such as AdaGrad Duchi et al . ( 2011 ) or Adam Kingma & Ba ( 2014 ) can be used instead . Turns out that for positive learning rates this stochastic optimization scheme converges to a local maximum of J , and when using the optimal parameters θ∗ , sampling from Pθ∗ returns locally optimal solutions x∗ ∈ X ∗f with high probability Williams ( 1992 ) . Nonetheless , such gradient estimates are known to be highly variable Paisley et al . ( 2012 ) , which requires drawing large samples at each iteration which is costly . Though there are techniques for reducing the variance of such estimates Ross ( 2013 ) , these are mostly useful when tied to the specifics of a given objective . We approach this problem differently , and consider instead how can we adapt the optimization objective in a manner that allows us to rely on noisy gradient estimates that only involve a single example ( i.e. , setting K = 1 ) , while ensuring we converge to a distribution that still allows us to sample some x∗ ∈ X ∗f . Since we focus on online updates , for the reminder of the text we drop the superscript k when referring to xkt and y k t .

Summary: The authors argue that not knowing the distribution of rewards observed in the policy gradient algorithm hinders learning (and the tuning process). They propose to replace the reward term in the policy gradient algorithm with its centered empirical cumulative distribution, which has a fixed and known U[-1, 1] distribution. They test their methods on a toy task that consists in finding inclusion maximal cliques (which tests for local optimality) against REINFORCE (including their variants: centering the rewards with a mean baseline or normalizing them), the cross-entropy method and Exp3.