Package: agnes
AGNES - Flexible Reinforcement Learning Framework with PyTorchStatus:This framework is under active development and bugs may occur.ResultsMuJoCo(Current results)MuJoCo "Ant-v2" training with 1M steps.Singlerunner withPPOalgorithm,MLPNN and 32 number of envs. The curve is an average of 3 runs.You can get the Tensorboard log file by clicking the image above(You will be redirected to the destination GitHub folder). The default config for the MuJoCo environment was used. Plotted byexamples/plot.pyAtari(Old results)Atari "BreakoutNoFrameskip-v4" with frame stack training with 10M steps.DistributedMPIrunner withPPOalgorithm,LSTMCNNand 16 number of envs.You can get the Tensorboard log file by clicking the image above(You will be redirected to the destination GitHub folder). The default config for the Atari environment was used.Grad-cam technique was used for sampled action chosen by trained LSTMCNN(previous point).RunnersSingleOne worker and trainer.agnes.make_vec_envcan also be used here.importagnesimporttimeif__name__=='__main__':env=agnes.make_env("InvertedDoublePendulum-v2")runner=agnes.Single(env,agnes.PPO,agnes.MLP)runner.log(agnes.log,agnes.TensorboardLogger(".logs/"),agnes.CsvLogger(".logs/"))runner.run()agnes.log- object ofStandardLoggerclass that outputs parameters to console.agnes.TensorboardLogger- class for writing logs in Tensorboard file.agnes.CsvLogger- class for writing logs in csv file. (required for plotting)DistributedMPIUnlike inSinglerunner, inDistributedMPIrunner due to async executing, weights are delayed by one rollout but this has no effect on learning because weights are delayed only by one update as it is inSinglerunner. So all parameters like probabilities ratio stay the same.Runs withmpiexec-n3python-mmpi4pyscript_name.py
or
mpirun-n3python-mmpi4pyscript_name.pyThis command will run 2 workers and 1 trainer.# script_name.pyimportagnesif__name__=='__main__':env=agnes.make_vec_env("BreakoutNoFrameskip-v4")runner=agnes.DistributedMPI(env,agnes.PPO,agnes.CNN)runner.run()AlgorithmsA2CSync version of Advantage Actor Critic is implemented in this framework and can be used simply:importagnesif__name__=='__main__':runner=agnes.Single(env,agnes.A2C,agnes.MLP)runner.run()PPOProximal Policy Optimization is implemented in this framework and can be used simply:importagnesif__name__=='__main__':runner=agnes.Single(env,agnes.PPO,agnes.MLP)runner.run()Neural Network ArchitecturesMulti Layer PerceptronCan be used with both continuous and discrete action spaces....runner=agnes.Single(env,agnes.PPO,agnes.MLP)...Convolutional Neural NetworkCan be used only with discrete action spaces....runner=agnes.Single(env,agnes.PPO,agnes.CNN)...Recurrent Neural NetworkCan be used with both continuous and discrete action spaces....runner=agnes.Single(env,agnes.PPO,agnes.RNN)...Convolutional Recurrent Neural NetworkCan be used only with discrete action spaces....runner=agnes.Single(env,agnes.PPO,agnes.RNNCNN)...Convolutional Neural Network with last LSTM layerCan be used only with discrete action spaces....runner=agnes.Single(env,agnes.PPO,agnes.LSTMCNN)...Make environmentmake_vec_env(env, envs_num=ncpu,config=None)**Parameters:env(str or function) is id of gym environment or function, that returns initialized environmentenvs_num(int) is a number of environments to initialize, by default is a number of logical cores on the CPUconfig(dict) is a dictionary with parameters forMonitorand for initializing environment, by default is None(uses default config)Returns:dict of"env"(VecEnvobject)"env_type"(str)"env_num"(int) is a number of envs inVecEnvobject"env_name"(str) is the name of envs inVecEnvobject(Id in gym or class name)The whole tuple should be put in arunner.make_env(env, config=None)is an alias ofmake_vec_envwithoutenvs_numargument that will be setted to 1.Notice:Some plot functions and environment wrappers were taken fromOpenAI Baselines(2017).