## 4 The Propose Transmission Optimization Algorithm
### Dynamic Learning Algorithm  
A game theoretical approached is used to solve this problem. Each sensor nodes in this system acts as a game player to interact with other sensors. In a dynamic environment, the reactive action of the game can obtain better performance than a deterministic method. For the problem under investigation, the location of each social sensor node follows a dynamic process. Hence, we use a dynamic learning algorithm to solve this problem. In this work, we first present the game as \(\Gamma=<M,\{\mathcal{A}_{i}\}_{i\in M},\mathbf{X},\{\mu_{i}\}_{i\in M}>\), where \(M\) denotes the number of players, \(\mathbf{X}\) represents the state of dynamic environment, and \(\mu_{i},i=1,\cdots,M\) is the utility function of the player. In this work, the action set of sensor node includes the two transmission power levels \(\mathcal{A}_{i}=\{0,P_{max}\}\), which refer to the sleep and active states of each sensor node, respectively. For the utility function \(\mu_{i}\), we first analyze the optimization objective and the constraints. In particular, the objective is to maximize the SNR at the intended BS, which is proportional to the transmission rate. The action of each sensor node, i.e., \(c_{i}=1\) or \(c_{i}=0\), determines the value of the SNR. When \(c_{i}=1\), the transmission power is \(P_{max}\), otherwise, it is 0. We consider the INR for the unintended BSs as a penalty term of the objective function. Then, the corresponding utility function \(\mu_{i}\) can be derived as follows:  
\[\mu_{i}=\mathcal{S}(\varphi_{0},\mathbf{cP})-\kappa(\mathcal{INR}(\varphi_{l}, \mathbf{cP})-INR_{0}). \tag{23}\]  
Here, since all \(M\) sensor nodes use the collaborative beamforming technique to achieve cooperatively the same optimization objective, the utility function of each node is the same, i.e., \(\mu_{1}=\mu_{2}=\cdots=\mu_{M}\). For the game theoretical approach, reinforcement learning methods are generally used to solve the problem. Although the Markov decision process (MDP) can also be used to solve the problem, the accurate state information is required, which is difficult to obtain in practical applications. In addition, the implementation of MDP is centralized. For the CPSS, the state of each social sensor node is determined by human activity. Thus, the accurate prior knowledge is hardly obtained in dynamic environment. In addition, the centralized optimization is hard to be implemented due to the difficulty of collecting the global state information of all sensor nodes. For the WSN with a distributed architecture in CPSS, a distributed scheme needs to be considered. Therefore, we first present a distributed reinforcement learning algorithm based on regret matching to improve the transmission performance of the WSN with social and physical sensor nodes in CPSS. The regret matching scheme focuses on the regret related to the actions of each player. For any time, each player (node) calculates the regret value by adjusting its strategy. The specific definition is given by  
\[R_{i}^{a_{i}}(t)=\frac{1}{t}\sum_{\tau=1}^{t}[\mu_{i}(a_{i},a_{-i}(\tau))-\mu_ {i}(a(\tau))] \tag{24}\]where \(\mu_{i}(a(\tau))\) denotes the received utility for each player at time \(\tau\), and \(\mu(a_{i},a_{-i}(\tau))\) is the utility when the player switch to action \(a_{i}\) while other players retaining theirs unchanged. Then, each player updates its strategy \(p_{i}(t)\) as follows:  
\[p_{i}^{a_{i}}(t)=\frac{[R_{i}^{a_{i}}(t)]_{+}}{\sum_{a_{i}^{\prime}\in\mathcal{A }_{i}}[R_{i}^{a_{i}^{\prime}}(t)]_{+}} \tag{25}\]  
where \([R_{i}^{a_{i}}(t)]_{+}=\max\{R_{i}^{a_{i}}(t),0\}\). The proposed algorithm is presented in Algorithm 1, where the regret matching method is used to maximize the utility in Equation (23). In the initialization stage, each social or physical sensor node initializes the location of node, state, and probability of action selection. Then, each sensor node selects the specific action (corresponding to the transmission power) according to the initial probability.