Package: ballbeam-gym
Ball & Beam GymBall & beam simulation as OpenAI gym environments.InstallationRun command:pip install ballbeam-gymor clone the repository and run the following inside the folder:pip install -e .System DynamicsSimulated as a frictionless first order system that takes the beam angle as input. The equation that describe the system is as follows:dx/dt = v(t)
dv/dt = -m*g*sin(theta(t))/((I + 1)*m)EnvironmentsBallBeamBalanceEnv- Objective is to not drop the ball from the beam using key state variables as observation space.VisualBallBeamBalanceEnv- Same as above but only using simulation plot as observation space.BallBeamSetpointEnv- Objective is to keep the ball as close to a set position on the beam as possible using key state variables as observation space.VisualBallBeamSetpointEnv- Same as above but only using simulation plot as observation space.BallBeamThrowEnv- Objective is to throw the ball as far as possible to the right possible using key state variables as observation space.VisualBallBeamThrowEnv- Same as above but only using simulation plot as observation space.AliasBallBeamBalance-v0VisualBallBeamBalance-v0BallBeamSetpoint-v0VisualBallBeamSetpoint-v0BallBeamThrow-v0VisualBallBeamThrow-v0APIThe environments use the same API and inherits from OpenAI gyms.step(action)- Simulate one timestep.reset()- Reset environment to start conditions.render(mode='human')- Visualize one timestep.seed(seed)- Make environment deterministic.BallBeamBalanceEnvBall is given a random or set initial velocity and it is the agent's job to stabilize the ball on the beam using a set of key state variables.Parameterstimestep- Length of a timestep.beam_length- Length of beam.max_angle- Max abs(angle) of beam.init_velocity- Initial speed of ball (Nonefor random).max_timesteps- Max timesteps in an episode (Nonefor infinate).action_mode- Continuous or discrete action space.Observation SpaceBeam angle.Ball position on beam.Ball velocity.Action SpaceContinuous:Set angle.Discrete:Increase angle.Keep angle.Descrease angle.RewardsA reward of 1 is given for each timestep ball stays on beam.ResetResets when ball falls of beam or max timesteps are reached.VisualBallBeamBalanceEnvBall is given a random or set initial velocity and it is the agent's job to stabilize the ball on the beam using a image data from the simulation plot.Parameterstimestep- Length of a timestep.beam_length- Length of beam.max_angle- Max abs(angle) of beam.init_velocity- Initial speed of ball (Nonefor random).max_timesteps- Max timesteps in an episode (Nonefor infinate).action_mode- Continuous or discrete action space.Observation SpaceRGB image [350x260x3].Action SpaceContinuous:Set angle.Discrete:Increase angle.Keep angle.Descrease angle.RewardsA reward of 1 is given for each timestep ball stays on beam.ResetResets when ball falls of beam or max timesteps are reached.BallBeamSetpointEnvThe agent's job is to keep the ball's position as close as possible to a setpoint using a set of key state variables.Parameterstimestep- Length of a timestep.beam_length- Length of beam.max_angle- Max abs(angle) of beam.init_velocity- Initial speed of ball (Nonefor random).max_timesteps- Max timesteps in an episode (Nonefor infinate).action_mode- Continuous or discrete action space.setpoint- Target position of ball (Nonefor random).Observation SpaceBeam angle.Ball position.Ball velocity.Setpoint position.Action SpaceContinuous:Set angle.Discrete:Increase angle.Keep angle.Descrease angle.RewardsAt each timestep the agent is rewarded with the squared proximity between the ball and the setpoint:reward = (1 - (setpoint - ball_position)/beam_length)^2.ResetResets when ball falls of beam or max timesteps are reached.VisualBallBeamSetpointEnvThe agent's job is to keep the ball's position as close as possible to a setpoint using a image data from the simulation plot.Parameterstimestep- Length of a timestep.beam_length- Length of beam.max_angle- Max abs(angle) of beam.init_velocity- Initial speed of ball (Nonefor random).max_timesteps- Max timesteps in an episode (Nonefor infinate).action_mode- Continuous or discrete action space.setpoint- Target position of ball (Nonefor random).Observation SpaceRGB image [350x260x3].Action SpaceContinuous:Set angle.Discrete:Increase angle.Keep angle.Descrease angle.RewardsAt each timestep the agent is rewarded with the squared proximity between the ball and the setpoint:reward = (1 - (setpoint - ball_position)/beam_length)^2.ResetResets when ball falls of beam or max timesteps are reached.BallBeamThrowEnvThe agent's job is to throw the ball as far as possible to the right using a set of key state variables.Parameterstimestep- Length of a timestep.beam_length- Length of beam.max_angle- Max abs(angle) of beam.init_velocity- Initial speed of ball (Nonefor random).max_timesteps- Max timesteps in an episode (Nonefor infinate).action_mode- Continuous or discrete action space.Observation SpaceBeam angle.Ball position on beam.Ball velocity.Action SpaceContinuous:Set angle.Discrete:Increase angle.Keep angle.Descrease angle.RewardsIs rewarded the calculated distance the ball would travel in x direction when it leaves the beam. Reward for a negative distance is set to 0.ResetResets when ball leaves the beam or max timesteps are reached.VisualBallBeamThrowEnvThe agent's job is to throw the ball as far as possible to the right using a image data from the simulation plot.Parameterstimestep- Length of a timestep.beam_length- Length of beam.max_angle- Max abs(angle) of beam.init_velocity- Initial speed of ball (Nonefor random).max_timesteps- Max timesteps in an episode (Nonefor infinate).action_mode- Continuous or discrete action space.Observation SpaceRGB image [350x260x3].Action SpaceContinuous:Set angle.Discrete:Increase angle.Keep angle.Descrease angle.RewardsIs rewarded the calculated distance the ball would travel in x direction when it leaves the beam. Reward for a negative distance is set to 0.ResetResets when ball leaves the beam or max timesteps are reached.Example: PID Controllerimportgymimportballbeam_gym# pass env arguments as kwargskwargs={'timestep':0.05,'setpoint':0.4,'beam_length':1.0,'max_angle':0.2,'init_velocity':0.0,'action_mode':'continuous'}# create envenv=gym.make('BallBeamSetpoint-v0',**kwargs)# constants for PID calculationKp=2.0Kd=1.0# simulate 1000 stepsforiinrange(1000):# control theta with a PID controllerenv.render()theta=Kp*(env.bb.x-env.setpoint)+Kd*(env.bb.v)obs,reward,done,info=env.step(theta)ifdone:env.reset()Example: Reinforcement Learningimportgymimportballbeam_gymfromstable_baselines.common.policiesimportMlpPolicyfromstable_baselines.common.vec_envimportDummyVecEnvfromstable_baselinesimportPPO2# pass env arguments as kwargskwargs={'timestep':0.05,'setpoint':0.4,'beam_length':1.0,'max_angle':0.2,'init_velocity':0.0,'action_mode':'discrete'}# create envenv=gym.make('BallBeamSetpoint-v0',**kwargs)# train a mlp policy agentenv=DummyVecEnv([lambda:env])model=PPO2(MlpPolicy,env,verbose=1)model.learn(total_timesteps=20000)obs=env.reset()env.render()# test agent on 1000 stepsforiinrange(1000):action,_=model.predict(obs)obs,reward,done,info=env.step(action)env.render()ifdone:env.reset()