Package: hindsight-replay
Hindsight Experience Replay (HER)=================================Hindsight Experience Replay (HER) is a reinforcement learning technique that makes use of failed experiences to learn how to achieve goals. It does this by storing additional transitions in the replay buffer where the goal is replaced with the achieved state. This allows the agent to learn from a hindsight perspective, as if it had intended to reach the achieved state from the beginning.ImplementationThis repository contains a Python implementation of HER using PyTorch. The main class isHindsightExperienceReplay, which represents a replay buffer that stores transitions and allows for sampling mini-batches of transitions.TheHindsightExperienceReplayclass takes the following arguments:state_dim: The dimension of the state space.action_dim: The dimension of the action space.buffer_size: The maximum size of the replay buffer.batch_size: The size of the mini-batches to sample.goal_sampling_strategy: A function that takes a tensor of goals and returns a tensor of goals. This function is used to dynamically sample goals for replay.TheHindsightExperienceReplayclass has the following methods:store_transition(state, action, reward, next_state, done, goal): Stores a transition and an additional transition where the goal is replaced with the achieved state in the replay buffer.sample(): Samples a mini-batch of transitions from the replay buffer and applies the goal sampling strategy to the goals.__len__(): Returns the current size of the replay buffer.UsageHere is an example of how to use theHindsightExperienceReplayclass:# Define a goal sampling strategydefgoal_sampling_strategy(goals):noise=torch.randn_like(goals)*0.1returngoals+noise# Define the dimensions of the state and action spaces, the buffer size, and the batch sizestate_dim=10action_dim=2buffer_size=10000batch_size=64# Create an instance of the HindsightExperienceReplay classher=HindsightExperienceReplay(state_dim,action_dim,buffer_size,batch_size,goal_sampling_strategy)# Store a transitionstate=np.random.rand(state_dim)action=np.random.rand(action_dim)reward=np.random.rand()next_state=np.random.rand(state_dim)done=Falsegoal=np.random.rand(state_dim)her.store_transition(state,action,reward,next_state,done,goal)# Sample a mini-batch of transitionssampled_transitions=her.sample()ifsampled_transitionsisnotNone:states,actions,rewards,next_states,dones,goals=sampled_transitionsIn this example, we first define a goal sampling strategy function and the dimensions of the state and action spaces, the buffer size, and the batch size. We then create an instance of theHindsightExperienceReplayclass, store a transition, and sample a mini-batch of transitions. The states, actions, rewards, next states, done flags, and goals are returned as separate tensors.Customizing the Goal Sampling StrategyTheHindsightExperienceReplayclass allows you to define your own goal sampling strategy by passing a function to the constructor. This function should take a tensor of goals and return a tensor of goals.Here is an example of a goal sampling strategy function that adds random noise to the goals:def goal_sampling_strategy(goals):
    noise = torch.randn_like(goals) * 0.1
    return goals + noiseIn this example, the function adds Gaussian noise with a standard deviation of 0.1 to the goals. You can customize this function to implement any goal sampling strategy that suits your needs.ContributingContributions to this project are welcome. If you find a bug or think of a feature that would be nice to have, please open an issue. If you want to contribute code, please fork the repository and submit a pull request.LicenseThis project is licensed under the MIT License. See theLICENSEfile for details.