# Training a Minesweeper Agent Using a Convolutional Neural Network
## 3 Model Design
### Reward Function  
The success of reinforcement learning methods, including DQN, heavily relies on a well-designed reward function. However, designing such a function is often challenging and requires careful analysis based on the specific context. In the case of Minesweeper, each action leads to a state change in the minefield. Even for a 6 \(\times\) 6 grid with four mines, the number of possible states in the minefield can be extremely large. However, by analyzing the state changes before and after the action, the actions can be classified into five types,  
Figure 5: Convolutional neural network model of the Minesweeper game agent.  
each assigned a reasonably intuitive reward value. Through testing and comparison, a satisfactory reward function can be obtained.  
1. Action leading to a Win state: The goal of Minesweeper is to uncover all non-mine cells. Successfully achieving this means correctly determining the status of each cell (whether it is a mine or not). The reward for winning the game is set to the number of cells, \(m\times n\), where m is the number of rows and n is the number of columns in the minefield;
2. Action leading to a Lose state: In contrast to winning, if an uncovered cell is a mine, the game is lost. The reward for losing the game is set to \(-m\times n\). This design aligns with the characteristic of Minesweeper that \"one wrong move leads to complete failure\";
3. Progress state: Uncovering a non-mine cell brings the agent closer to winning the game and is rewarded with +1. This reward value is based on the idea that successfully uncovering a cell has a value of +1, consistent with the reward settings for the Win and Lose states;
4. Guess state: If all eight neighboring cells of an uncovered cell are still hidden, even if the uncovered cell is not a mine, no reward should be given. Instead, the agent should incur a penalty (negative reward). There are two reasons for this: first, such an action is essentially a random guess and does not contribute to improving the agent's Minesweeper skills; second, unless the mine density is very low, repeatedly taking such actions is likely to hit a mine, making it almost impossible to win the game. Based on preliminary testing, the reward for random guessing was set to \(-0.5\);
5. YOLO state: YOLO is an acronym for \"You Only Live Once\", which encourages people to embrace risk and live boldly. This state refers to attempting to uncover an already uncovered cell, which does not change the minefield and results in no progress. \"You can't beat something with nothing\"--avoiding risk may prevent failure, but it also forfeits the opportunity for success. A penalty for such actions is necessary to prevent the agent from choosing a \"no-action\" strategy (lie flat or tang ping), where it repeatedly selects already uncovered cells to avoid the penalty for losing while giving up the reward for winning. A reasonable penalty for this \"no-action\" behavior helps the agent escape local optima and encourages it to continue striving for the global optimum.  
The five types of actions and their corresponding rewards are shown in Figure 6. In this paper, the reward data structure is designed as a dictionary. For a 6 \(\times\) 6 grid with four mines, the reward structure is defined as follows: rewards = [\"Win\": 36, \"Loss\": \(-\)36, \"Progress\": 1, \"Guess\": \(-\)0.5, \"YOLO\": \(-\)0.5].