Package: nanoPPO
nanoPPOnanoPPO is a Python package that provides a simple and efficient implementation of the Proximal Policy Optimization (PPO) algorithm for reinforcement learning. It is designed to support both continuous and discrete action spaces, making it suitable for a wide range of applications.InstallationYou can install nanoPPO directly from PyPI using pip:pipinstallnanoPPOAlternatively, you can clone the repository and install from source:gitclonehttps://github.com/jamesliu/nanoPPO.gitcdnanoPPO
pipinstall.UsageHere are examples of how to use nanoPPO to train an agent.On the MountaionCarContinuous-v0 environment:fromnanoppo.train_ppo_agentimporttrain_agentimportpickleppo,model_file,metrics_file=train_agent(env_name="MountainCarContinuous-v0",max_episodes=50,policy_lr=0.0005,value_lr=0.0005,vl_coef=0.5,checkpoint_dir="checkpoints",checkpoint_interval=10,log_interval=10,wandb_log=False,)ppo.load(model_file)print("Loaded best weights from",model_file)metrics=pickle.load(open(metrics_file,"rb"))print("Loaded metrics from",metrics_file)best_reward=metrics["best_reward"]episode=metrics["episode"]print("best_reward",best_reward,"episode",episode)Use Custom LR Scheduler and Custom PolicySet Cosine Annealing Learning Rate SchedulerSet CausalAttention Policy instead of Linear Policyfromnanoppo.train_ppo_agentimporttrain_agentfromnanoppo.cosine_lr_schedulerimportCosineLRSchedulerfromnanoppo.policy.actor_critic_causal_attentionimportActorCriticCausalAttentionlr_scheduler=CosineLRScheduler(learning_rate=config['cosine_lr'],warmup_iters=config['cosine_warmup_iters'],lr_decay_iters=config['cosine_decay_steps'],min_lr=config['cosine_min_lr'])policy_class=ActorCriticCausalAttentionppo,model_file,metrics_file=train_agent(env_name=env_name,env_config=env_config,max_episodes=config['max_episode'],stop_reward=config['stop_reward'],policy_class=policy_class,lr_scheduler=lr_scheduler,policy_lr=config['policy_lr'],value_lr=config['value_lr'],vl_coef=config['vl_coef'],betas=config['betas'],n_latent_var=config['n_latent_var'],gamma=config['gamma'],K_epochs=config['K_epochs'],eps_clip=config['eps_clip'],el_coef=config['el_coef'],checkpoint_dir=checkpoint_dir,checkpoint_interval=10,log_interval=10,wandb_log=wandb_log,debug=True)On the CartPole-v1 environment:fromnanoppo.discrete_action_ppoimportPPOimportgymenv=gym.make('CartPole-v1')ppo=PPO(env.observation_space.shape[0],env.action_space.n)# Training code here...ExamplesSee theexamplesdirectory for more comprehensive usage examples.examples/train_mountaincar.shpython nanoppo/train_ppo_agent.py --env_name=MountainCarContinuous-v0 --policy_lr=0.0005 --value_lr=0.0005 --max_episodes=50 --vl_coef=0.5 --wandb_logexamples/train_pointmass1d.shpython nanoppo/train_ppo_agent.py --env_name=PointMass1D-v0 --policy_lr=0.0005 --value_lr=0.0005 --max_episodes=50 --vl_coef=0.5 --wandb_logexamples/train_pointmass2d.shpython nanoppo/train_ppo_agent.py --env_name=PointMass2D-v0 --policy_lr=0.0005 --value_lr=0.0005 --max_episodes=50 --vl_coef=0.5 --wandb_logDocumentationFull documentation is availablehere.ContributingWe welcome contributions to nanoPPO! If you're interested in contributing, please see ourcontribution guidelines.LicensenanoPPO is licensed under the Apache License 2.0. See theLICENSEfile for more details.SupportFor support, questions, or feature requests, please open an issue on ourGitHub repositoryor contact the maintainers.ChangelogSee thereleasespage for a detailed changelog of each version.