# RIS-Assisted Multi-Antenna AmBC Signal Detection Using Deep Reinforcement Learning
## 4 Proposed Solution
### TD3-Based Framework  
Analyzing the above algorithm, DQN is not suitable to solve the problem **P1**, because DQN can only handle the discrete action spaces. It is worthy that both the active beamforming and the passive RIS phase shift are continuously variable. While DDPG is easily influenced by the overestimation of the \(Q\) value, the optimal performance cannot be achieved, and DDPG is very sensitive to the external conditions. In this paper, an optimization algorithm based on TD3 is proposed to solve the non-convex optimization in problem **P1**, and the block diagram of TD3 is shown in Figure 2. We used six fully connected deep neural networks (DNNs) to build the algorithm, containing one Actor network, one Actor target network, two Critic networks, and two Critic target networks. Each DNN network contains one input layer, two hidden layers and one output layer. 1. Construction of TD3: In the paper, the AmBC system and the RIS controller are regarded as the environment and the agent, respectively. In order to perform the TD3 algorithm, the state, action, and reward are defined as follows: **State space:** The state space represents the current state of the system, which consists of the optimization  
Figure 2: The TD3-based framework. parameters of the previous time. Since the optimization parameter \(\mathbf{f}\) denotes complex number value, and the neural network cannot handle the complex number, \(\mathbf{f}\) is decomposed into two parts: the real part \(\mathbf{f}\) and the imaginary part \(\mathbf{f}^{\prime}\), and they are propagated to the neural network at the same time. \[S_{t}=\left[f^{\prime(t-1)}_{\ 1},\dots,f^{\prime(t-1)}_{\ K},f^{*(t-1)}_{1}, \dots,f^{*(t-1)}_{\ K}\right] \tag{24}\]  
**Action space: There are two key components in the action space: the combining vector \(\mathbf{f}\) and the phase shift \(\mathbf{\phi}\). ** Similarly, each complex variable is decomposed into the real and the imaginary part. \[A_{t}=\left[\phi^{\prime t}_{\ 1},\dots,\phi^{\prime t}_{\ N},\phi^{*t}_{1}, \dots,\phi^{*t}_{N},f^{\prime t}_{\ 1},\dots,f^{\prime t}_{\ K},f^{*t}_{1},\dots,f^{*t}_{K}\right] \tag{25}\]  
**Reward:** In this article, the optimization goal is to achieve the maximum energy ratio \(\gamma\). The reward at step \(t\) is a linear transformation of maximum energy ratio \(\gamma\), i.e.,  
\[r^{t}=100(\gamma-1) \tag{26}\]  
(2) Algorithm Description: The detail of the TD3 based framework is shown as follows (Algorithm 1):  
```
1:Input: The batch size, the soft update coefficient, the replay capacity, the policy
2: network learning rate, the \(Q\) network learning rate, the policy target update interval,
3: the explore steps, the action noise for exploration and evaluation
4:Output: The optimal phase shift matrix \(\mathbf{\phi}_{opt}\), the combining vector \(\mathbf{f}_{opt}\), the optimal
5: energy ratio \(\gamma\), and the BER for backscatter tag detection
6:Initialization: Randomly initialize the six networks
7:for episode \(j=1,\dots,M\)do
8: Obtain the current CSI
9: Randomly chose combining vector to obtain \(s_{0}\)
10:for step \(t=1,\dots,T\)do
11:if\(t>\exp\)\(\mathit{lore\ steps}\), choose action based on the policy \(s_{t}=\mu(s_{t};\theta_{\mu})+\mathcal{N}\)
12:else generate action randomly
13: Obtain the reward \(r_{t}\) and observe new state \(s_{t+1}\) by the environment, store
14: the tuple \(\{s_{t},a_{t},r_{t},s_{t+1}\}\) into the replay memory
15:if the length of replay buffer \(>\) batchsize, sample a minibatch from the
16: experience replay memory
17: Learn and update the Critics network by minimizing the loss
18: if reach the policy target update interval, update the actor policy network
19: Update the target network parameters of Actor and Critics network
20:endfor
21:endfor
22: Obtain the optimal phase shift matrix \(\mathbf{\phi}_{opt}\) and the combining vector \(\mathbf{f}_{opt}\) by final action \(A_{t}\).