# Decoupled Monte Carlo Tree Search for Cooperative Multi-Agent Planning
## 6 Results
### Experimental Setup
#### 6.1.2 MMDP Problems  
We extended our results to Multi-agent Markov Decision Processes (MMDP) to test action selection policies on problems from literature: _meeting in the grid_[39] and _fire fighting_[40]. Although these problems were decentralized partially observable problems, we changed them to be centralized and fully observable to model as MMDP. Experimental Setup  
We redefined _meeting in the grid_ and _fire fighting_ problems as MMDP problems. The _meeting in the grid_ problem had two agents that were, initially, positioned at the top left corner and bottom right corner of the grid, as seen in Figure 8. The grid was defined as a square and its size determined the complexity of the problem. The state of the problem was defined as the position of the agents. They could move _north, south, east, west_, or _stay_ in the current cell. However, actions failed with a probability of 0.4 and when they failed, a random movement was executed. The agents received a reward of 1 when they met in any cell, and 0 otherwise. The horizon of the problem was defined as two times the grid size; for a 3 by 3 grid, the horizon was 6. All agents received the full state of the problem so that every agent knew where the other agent was. The _fire fighting_ problem had 2 agents and \(m\) houses positioned side to side, as seen in Figure 8. Every house had a fire level that denoted whether the house was on fire and the severity of the fire. In our experiments, we used 3 fire levels: 0 being the no fire level and 2 being the severe fire. The state of the problem was defined as the position of the agents and the fire levels of the houses. Agents could move from one house to another in a single time step, even if the houses were far away from each other. If two agents were in a house, the fire was completely extinguished. If there was one agent, the fire level of the house decreased with 0.6 probability. A house's fire level increased if there was no agent in that house, with different probabilities determined by its current fire level and the fire levels of its neighbors. The reward of the problem was negative of the sum of fire levels so that the agents were penalized for houses that were on fire. The agents could see the fire levels of houses and their positions. In the beginning, the agents started at an out of house location. The horizon was set as two times the number of houses. We ran different experiments for the different difficulties of problems. For _meeting in the grid_ problem, we tested for different sizes of the grid. For _fire fighting_ problem, we tested for a different number of houses. We kept the number of agents as two for both problems. We ran 2000 simulations at each decision step and results were provided over 100 experiments. We normalized rewards and kept them between 0 and 1 at each step for the _EXP3_ action selection policy. We used different action selection policy parameters for different problems. For _meeting in the grid_ problem, we found \(\epsilon=0.61\) and \(\gamma=0.05\) were the best parameters when we ran experiments between 0 and 1 with an increment of 0.01 for a grid size of 3. We show the change of the performance over the parameter space for _meeting in the grid_ problem in Figure 9. For the _fire fighting_ problem, we found \(\epsilon=0.33\) and \(\gamma=0.15\) were the best parameters when we ran experiments between 0 and 1 with an increment of 0.01 for a problem with 2 agents and 6 houses. We show the change of the performance over the parameters for the _fire fighting_ problem in Figure 9. Although we chose the best-performing parameters for the experiments, we observed that the selection policies were not sensitive to the parameters as there was no significant performance difference between the parameter values in the range of 0.2 and 0.6. We  
Figure 8: MMDP Problems. **(left)** The fire fighting problem. **(right)** The meeting in grid problem. also ran experiments with _UCT_ algorithm to compare the decoupled-MCTS method to centralized planning.