## 5 Precoder and Combiner Design Based on Reinforcement Learning
### Joint Precoding and Combining Algorithm Based on MADDPG  
MADDPG is actor-critic algorithm that utilizes the idea of centralized training and decentralized decision-making. The actor makes a decision over time steps, while the critic evaluates the value of the decision. Assume the set \(\mu=\{\mu_{1},\cdots,\mu_{2N_{s}}\}\) is the policies of all agents and set \(\theta=\{\theta_{1},\cdots,\theta_{2N_{s}}\}\) as the parameters of corresponding policies. For the \(i\)-th agent's policy \(\mu_{i}\), the object function \(\mathcal{I}(\theta_{i})\) is the expected reward. The gradient of the object function w.r.t \(\theta_{i}\) can be depicted as:  
\[\
abla_{(\theta_{i})}\mathcal{I}(\mu_{i})=\mathbb{E}_{(\mathbf{S},\mathbf{A},\mathbf{R},\mathbf{S}^{\prime})}\Big{[}\
abla_{(\theta_{i})}\mu_{i}(a_{i}|s_ {i})\
abla_{(a_{i})}Q_{i}^{\mu}(\mathbf{S},a_{1},...,a_{i})\Big{]}|_{(a_{i}= \mu_{i}(s_{i}))}, \tag{33}\]  
where \(\mathbf{S}=(s_{1},\cdots,s_{2N_{s}})\), \(\mathbf{A}=(a_{1},\cdots,a_{2N_{s}})\). \(\mathcal{D}\) is the replay buffer, which stores the transition \(\big{(}\mathbf{S},\mathbf{A},\mathbf{R},\mathbf{S}^{\prime}\big{)}\), where \(\mathbf{S}^{\prime}\) is the next time step state and \(R=(r_{1},...,r_{2N_{s}})\). \(Q_{i}^{\mu}(\mathbf{S},a_{1},...,a_{i})\) is the Q-value function. The Q-function can be updated by minimizing the function below:  
\[\mathcal{L}(\theta_{i})=\mathbb{E}_{\mathbf{S},\mathbf{A},\mathbf{R},\mathbf{ S}^{\prime}}\Big{[}\Big{(}Q_{i}^{\mu}(\mathbf{S},a_{1},...,a_{i})-y\Big{)}^{2} \Big{]}. \tag{34}\]  
In (34), \(y=r_{i}+\gamma Q_{i}^{\mu^{\prime}}\big{(}\mathbf{S}^{\prime},a_{1}^{\prime},...,a_{i}^{\prime}\big{)}|_{a_{i}^{\prime}=\mu_{i}^{\prime}(s_{i})}\), \(\gamma\) is the \(Q\)-value discounting factor, and \(\mu^{\prime}\) represents the policies with the delayed network parameter \(\theta^{\prime}\). Each agent maintains two sets of actor-critic network pairs, known as the behavior pair and the target pair. Relative to the behavior pair, the target pair of networks makes a soft replacement with parameters. We show the algorithm as a pseudo-code type in Algorithm 1.  
The hyperparameters of MADDPG are shown in Table 1.  
```
1 Start;
2 Initialize the environment of downlink MIMO communication system;
3 Randomly initialize the weights of behavior actor-critic pair;
4 forepisode = \(1\to M\)do
5 Initialize a Gaussian random process \(\mathcal{N}\);
6 Reset the photon-counting MIMO downlink system environment and obtain initial global state \(\mathbf{S}\);
7 for\(t=1\to N\)do
8 Each agent selects action \(a_{i}=\mu_{i}(s_{i})+\mathcal{N}_{t}\);
9 Reformulate actions \(\mathbf{A}=(a_{1},\cdots,a_{2N_{k}})\) into precoding matrix \(\mathbf{W}\) and combining matrix \(\mathbf{F}^{T}\) and obtain new state \(\mathbf{S}^{\prime}\);
10 Store \(\left(\mathbf{S},\mathbf{A},\mathbf{R},\mathbf{S}^{\prime}\right)\) in replay buffer \(\mathcal{D}\);
11\(\mathbf{S}\gets\mathbf{S}^{\prime}\);
12 foragent \(i=1\to 2N_{s}\)do
13 Randomly sample a minibatch of transitions \(\left(\mathbf{S},\mathbf{A},\mathbf{R},\mathbf{S}^{\prime}\right)\) from \(\mathcal{D}\);
14 Update the critic network by (34);
15 Update the actor network by (33);
16 Update the target network parameters of each precoding and combining agent \(i\);
17\(\theta_{i}^{\prime}\leftarrow\tau\theta_{i}+(1-\tau)\theta_{i}^{\prime}\);
18 End;
```  
**Algorithm 1**MADDPG for joint precoding and combining optimization