# Training a Minesweeper Agent Using a Convolutional Neural Network
## 2 Algorithmic Principles
### Training a Minesweeper Agent with DQN
```
0: Coded minesweeper grid state
0: Q-network parameters
1: Initialize replay buffer \(D\)
2: Randomly initialize Q-network and target Q-network, \(\theta^{-}=\theta\)
3: Observe the initial state \(s\)
4: Repeat
5: Select an action \(a\)
6: With probability \(\varepsilon\), choose a random action
7: Otherwise, select an action based on \(\pi(s)=\underset{a}{\arg\max}Q(s,a;\theta)\)
8: Execute the action \(a\)
9: Observe the reward \(r\) and new state \(s^{\prime}\)
10: Store the experience \(\langle s,a,r,s^{\prime}\rangle\) in \(D\)
11: If \(D\) is full, remove the oldest experience and add the new data
12: Randomly sample a batch of size batch_size from \(D\)
13: Compute the target \(x_{i}\) for each sample in the batch
14: If the state is terminal, \(x_{i}=r_{i}\)
15: Otherwise, \(x_{i}=r_{i}+\gamma\max_{a^{\prime}}Q^{\prime}(s^{\prime}_{i},a^{\prime}; \theta^{-})\)
16: Compute the MSE loss between the Q-function and the target Q-function
17:\(L(\theta)=\frac{1}{batch_{i}size}\sum_{i=1



### Training a Minesweeper Agent with Supervised Learning

The goal of supervised learning is to train the model using the true labels of the Minesweeper grid to predict the probability of each cell containing a mine. Given a grid state \(s\), the model \(f(s,\theta)\) is trained to predict the probability \(p_{i}\) (for \(i=1,2,\cdots,MN\)) that each cell contains a mine. The model is optimized by minimizing the cross-entropy (abbreviated as CE) loss function, defined as follows:

\[L_{CE}(\theta)=-\underset{i}{\sum}[y_{i}\log(p_{i})+(1-y_{i})\log(1-p_{i})] \tag{5}\]

Here, \(p_{i}\) is the predicted mine probability, and \(y_{i}\) is the true label (1 for mine, 0 for non-mine). During gameplay, the agent makes decisions based on the cell with the lowest mine probability, consistently following the \"safest\" strategy for each move. In this way, the agent continuously explores and accumulates more game data, gradually optimizing the model and improving the accuracy of mine predictions. Each cell is either a mine or not, with a label of 1 for mines and 0 for non-mines (viewing this as the probability of a cell being a mine is a useful way to understand the problem). For each game of Minesweeper, the grid labels are actually known, or at least can be obtained. This can be explained in two ways: one case is when the Minesweeper environment is designed by us, and the internal data structure of the game is transparent, allowing the grid labels to be directly used as training data; another case is treating the Minesweeper process as an exploration of a black box. If the game is won, it means all non-mine cells have been revealed, and the remaining unrevealed cells are mines. If the game ends in failure due to stepping on a mine, all mines will be revealed (as standard in Minesweeper), and the non-mine cells will also be disclosed. Therefore, regardless of the outcome, the true labels of the grid can be obtained. By backtracking through a game, a set of training data can be gathered. Similarly, humans discover key mistakes through review and reduce the likelihood of repeating them by accumulating experience. Both machines and humans essentially use similar methods: backtracking and reviewing are simply different expressions of the same concept. Machines use a loss function to measure the severity of errors and train the model by minimizing the loss function, thus reducing the frequency of errors and lowering their severity. Overall, this approach is intuitive and effective. The supervised learning algorithm used in this paper is shown in Algorithm 2.