Package: pokerenv
PokerenvPokerenv is an openAI gym (https://gym.openai.com/docs/) compliant reinforcement learning environment for No Limit Texas Hold'em. It supports 2-6 player tables.The environment can be configured to output hand history files, which can be viewed with any pokerstars compatible tracking software (holdem manager, pokertracker, etc.), allowing you to easily track the learning process.Installation and dependenciespipinstallnumpy
pipinstalltreys
pipinstallpokerenvUsage informationThe rewards are output as a numpy array, where the nth element corresponds to reward given to the agent who was playing when the value of the acting player flag contained in the observation was n.The acting player flag contained in the observation does not mean the agents position in the table.
Each player inside the table gets a unique id when the table instance is created, and this id is passed as the acting player flag in the observation.
This way agents can keep reacting to the same acting player flag value, while still playing from all possible table positions.Invalid actionsThe environment deals with invalid actions by ignoring them, and either checking or folding automatically.
If configured to do so, the environment also applies an invalid action penalty to the corresponding reward. The observation contains entries which can be used to implement invalid action masking.All of the required (from the learning loop perspective) observation entries have human readable index definitions in the obs_indices.py module.Toy exampleDefine an agentimportnumpyasnpimportpokerenv.obs_indicesasindicesfrompokerenv.tableimportTablefrompokerenv.commonimportPlayerAction,Action,action_listclassExampleRandomAgent:def__init__(self):self.actions=[]self.observations=[]self.rewards=[]defget_action(self,observation):self.observations.append(observation)valid_actions=np.argwhere(observation[indices.VALID_ACTIONS]==1).flatten()valid_bet_low=observation[indices.VALID_BET_LOW]valid_bet_high=observation[indices.VALID_BET_HIGH]chosen_action=PlayerAction(np.random.choice(valid_actions))bet_size=0ifchosen_actionisPlayerAction.BET:bet_size=np.random.uniform(valid_bet_low,valid_bet_high)table_action=Action(chosen_action,bet_size)self.actions.append(table_action)returntable_actiondefreset(self):self.actions=[]self.observations=[]self.rewards=[]Create an environmentactive_players=6agents=[ExampleRandomAgent()for_inrange(6)]player_names={0:'TrackedAgent1',1:'Agent2'}# Rest are defaulted to player3, player4...# Should we only log the 0th players (here TrackedAgent1) private cards to hand history filestrack_single_player=True# Bounds for randomizing player stack sizes in reset()low_stack_bbs=50high_stack_bbs=200hand_history_location='hands/'invalid_action_penalty=0table=Table(active_players,player_names=player_names,track_single_player=track_single_player,stack_low=low_stack_bbs,stack_high=high_stack_bbs,hand_history_location=hand_history_location,invalid_action_penalty=invalid_action_penalty)table.seed(1)Implement learning loopiteration=1whileTrue:ifiteration%50==0:table.hand_history_enabled=Trueactive_players=np.random.randint(2,7)table.n_players=active_playersobs=table.reset()foragentinagents:agent.reset()acting_player=int(obs[indices.ACTING_PLAYER])whileTrue:action=agents[acting_player].get_action(obs)obs,reward,done,_=table.step(action)ifdone:# Distribute final rewardsforiinrange(active_players):agents[i].rewards.append(reward[i])breakelse:# This step can be skipped unless invalid action penalty is enabled,# since we only get a reward when the pot is distributed, and the done flag is setagents[acting_player].rewards.append(reward[acting_player])acting_player=int(obs[indices.ACTING_PLAYER])iteration+=1table.hand_history_enabled=False