Package: afrl
AFRL - All Forms of Reinforcement LearningThe main goal of this project is to provide a framework for reinforcement learning research. The framework is designed to be modular and easy to extend. It is written in Python and built on top ofPyTorch. The framework is still in its early stages and is under active development.UsageThe framework is designed to be modular and easy to extend. The main components of the framework are:Environments: The environments are the tasks that the agent is trying to solve.Agents: The agents are the algorithms that are trying to solve the environment.Trainers: The trainers are the methods the agents are training with.Here's the list of the trainers along with their current implementation status:DQNDDQNDueling DQNDouble Dueling DQNDDPGSDDPGPrioritized Experience ReplayTRPOPPOTD-LambdaSARSAREINFORCEActor-CriticA2CA3CSACTo make it easier to choose the right trainer for the right environment, here's a table that shows the different trainers and the environments that they support:MethodDiscrete Action SpaceContinuous Action SpaceSingle-AgentMulti-AgentLow-Dimensional ObsHigh-Dimensional ObsDQN✔️❌✔️❌✔️❌DDQN✔️❌✔️❌✔️❌Dueling DQN✔️❌✔️❌✔️❌Double Dueling DQN✔️❌✔️❌✔️❌DDPG❌✔️✔️❌✔️✔️SDDPG❌✔️✔️❌✔️✔️Prioritized Experience Replay✔️✔️✔️❌✔️❌TRPO✔️✔️✔️❌✔️✔️PPO✔️✔️✔️✔️✔️✔️TD-Lambda✔️✔️✔️❌✔️❌SARSA✔️❌✔️❌✔️❌REINFORCE✔️✔️✔️❌✔️✔️Actor-Critic✔️✔️✔️✔️✔️✔️A2C✔️✔️✔️✔️✔️✔️A3C✔️✔️✔️✔️✔️✔️SAC❌✔️✔️❌✔️✔️Characteristics of Different Environments for RL AlgorithmsHere is a list of characteristics of different environments, along with some environment examples that have those characteristics and some generic rules of thumb in order to choose the right method for your environment:1. Discrete vs Continuous Action SpaceExamples:Discrete: Tic-Tac-Toe, Grid WorldsContinuous: Robotic arm control, Portfolio managementRules of Thumb:For discrete action spaces, DQN variants and PPO are commonly used.For continuous action spaces, DDPG and SAC offer better performance.2. Single-Agent vs Multi-AgentExamples:Single-Agent: Mountain Car, CartpoleMulti-Agent: Poker, Multi-robot coordinationRules of Thumb:Single-agent tasks often use DQN, PPO, or DDPG.Multi-agent tasks typically benefit from specialized algorithms like MADDPG, or generic methods like PPO that are adapted for multi-agent scenarios.3. Low-Dimensional vs High-Dimensional Observation SpaceExamples:Low-Dimensional: Frozen Lake, Taxi-v3High-Dimensional: Atari games, Visual navigationRules of Thumb:Low-dimensional problems can be tackled with simpler algorithms like SARSA or TD-Lambda.High-dimensional problems often require methods capable of handling complex function approximation, such as Convolutional Neural Networks (CNNs) in DQN for image-based tasks.4. Partially Observable vs Fully ObservableExamples:Partially Observable: Poker, Hide and SeekFully Observable: Chess, GoRules of Thumb:Fully observable environments can make use of simpler methods such as DQN or PPO.Partially observable environments may require methods with memory capabilities like LSTM or GRU incorporated into the algorithm (e.g., DRQN).5. Sparse vs Dense RewardExamples:Sparse Reward: Maze navigation, robotic graspingDense Reward: Cartpole, Lunar LanderRules of Thumb:Dense reward problems can use most algorithms effectively.Sparse reward problems often benefit from algorithms designed for exploration, such as those utilizing curiosity-driven mechanisms or hierarchical methods.By considering these environment characteristics and rules of thumb, one can make a more informed decision when selecting an appropriate reinforcement learning algorithm for a specific task.