Package: pepg-es
Parameter-exploring Policy GradientsPython Implementation of Parameter-exploring Policy Gradients[3]Evolution StrategyRequirementsPython >= 3.6NumpyOptionalgymmpi4pyInstallFrom PyPIpip3installpepg-esFrom Sourcegitclonehttps://github.com/goktug97/PEPG-EScdPEPG-ES
python3setup.pyinstall--userAbout ImplementationI implemented several things differently from the original paper;Applied rank transformation[1]to the fitness scores.Used Adam[2]optimizer to update the mean.Weight decay is applied to the mean, similar to[4].UsageRefer toPEPG-ES/examplesfolder for more complete examples.XOR ExampleFind Neural Network parameters for XOR Gate.Black-box optimization algorithms like PEPG are competitive in the
area of reinforcement learning because they don't require
backpropagation to calculate the gradients.  In supervised learning
using backpropagation is faster and more reliable. Thus, using backpropagation
to solve the XOR problem would be faster. I demonstrated library by solving XOR
because it was easy and understandable.frompepgimportPEPG,NeuralNetwork,Adam,sigmoidimportnumpyasnpnetwork=NeuralNetwork(input_size=2,output_size=1,hidden_sizes=[2],hidden_activation=sigmoid,output_activation=sigmoid)# Adam Optimizer is the default optimizer, it is written for the exampleoptimizer_kwargs={'beta_1':0.9,'beta_2':0.999,'epsilon':1e-08}# Adam Parameterses=PEPG(population_size=100,theta_size=network.number_of_parameters,mu_init=0,sigma_init=2.0,mu_lr=0.3,sigma_lr=0.2,optimizer=Adam,optimizer_kwargs=optimizer_kwargs)truth_table=[[0,1],[1,0]]solution_found=FalsewhileTrue:print(f'Step:{es.step}')solutions=es.get_parameters()rewards=[]forsolutioninsolutions:network.weights=solutionerror=0forinput_1inrange(len(truth_table)):forinput_2inrange(len(truth_table[0])):output=int(round(network([input_1,input_2])[0]))error+=abs(truth_table[input_1][input_2]-output)reward=(4-error)**2rewards.append(reward)es.update(rewards)ifes.best_fitness==16:print('Solution Found')print(f'Parameters:{es.best_theta}')breakOutput:Step:233Step:234Step:235Step:236Step:237SolutionFound
Parameters:[1.25863047-0.73151503-2.533777231.018023553.027235071.23112726-2.00288859-3.667892424.56593794]DocumentationPEPG Classes=PEPG(self,population_size,theta_size,mu_init,sigma_init,mu_lr,sigma_lr,l2_coeff=0.005,optimizer=Adam,optimizer_kwargs={})Parameters:population_size:int: Population size of the evolution strategy.theta_sizeint: Number of parameters that will be optimized.mu_initfloat: Initial mean.sigma_initfloat: Initial sigma.mu_lrfloat: Learning rate for the mean.sigma_lrfloat: Learning rate for the sigma.l2_coefffloat: Weight decay coefficient.optimizerOptimizer: Optimizer to useoptimizer_kwargsDict[str, Any]: Parameters for optimizer except learning rate.solutions=self.get_parameters(self)Creates symmetric samples around the mean and returns a numpy array with the size of[population_size, theta_size]self.update(self,rewards)Parameters:rewards:List[float]: Rewards for the given solutions.Update the mean and the sigma.self.save_checkpoint(self)Creates a checkpoint and save it into created time.time().checkpoint file.es=PEPG.load_checkpoint(cls,filename)Creates a new PEPG class and loads the checkpoint.self.save_best(self,filename)Saves the best theta and the mu and the sigma that used to create the best theta.theta,mu,sigma=PEPG.load_best(cls,filename)Load the theta, the mu, and the sigma arrays from the given file.NeuralNetwork ClassNeuralNetwork(self,input_size,output_size,hidden_sizes=[],hidden_activation=lambdax:x,output_activation=lambdax:x,bias=True):Parameters:input_size:int: Input size of network.output_size:int: Output size of the network.hidden_sizes:List[int]: Sizes for the hidden layers.hidden_activation:Callable[[float], float]: Activation function used in hidden layers.output_activation:Callable[[float], float]: Activation function used at the output.bias:bool: Add bias node.self.save_network(self,filename)Save the network to a file.network=NeuralNetwork.load_network(cls,filename)Creates a new NeuralNetwork class and loads the given network file.Custom Optimizer ExamplefrompepgimportPEPG,Optimizer,NeuralNetworkclassCustomOptimizer(Optimizer):def__init__(self,alpha,parameter,another_parameter):self.alpha=alphaself.parameter=parameterself.another_parameter=another_parameterdef__call__(self,gradients):gradients=(gradients+self.parameter)*self.another_parameterreturn-self.alpha*gradientsnetwork=NeuralNetwork(input_size=2,output_size=1)optimizer_kwargs={'parameter':0.3,'another_parameter':0.2}es=PEPG(population_size=100,theta_size=network.number_of_parameters,mu_init=0.0,sigma_init=2.0,mu_lr=0.3,sigma_lr=0.2,optimizer=CustomOptimizer,optimizer_kwargs=optimizer_kwargs)ReferencesDaan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters and Jurgen Schmidhuber. Natural Evolution Strategies. 2014Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. 2014F. Sehnke, C. Osendorfer, T. Ruckstiess, A. Graves, J. Peters and J. Schmidhuber. Parameter-exploring policy gradients. 2010Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor and Ilya Sutskever. Evolution Strategies as a Scalable Alternative to Reinforcement Learning. 2017