# RIS-Assisted Multi-Antenna AmBC Signal Detection Using Deep Reinforcement Learning
## 4 Proposed Solution
### Overview of Deep Reinforcement Learning  
Deep reinforcement learning is the combination between reinforcement learning and deep neural network [19]. A conventional DRL system consists of agents, environment, and the mutual transfer of information between them constituting a Markov process. The current state \(s_{t}\) of the environment is obtained through the agent, then the action \(a_{t}\) is taken based on the policy \(\Pi\) and it is transmitted to the environment; \(a_{t}\) is evaluated by the environment to send reward \(r_{t}\), and the new state is updated to \(s_{t+1}\). With the goal of maximizing long-term returns, deep neural networks and the accumulated empirical data are combined in the agent to continuously optimize the policies. So far, three value-based algorithms, including DQN, DDPG, and TD3, have emerged successively. 1. DQN: Deep \(Q\) network (DQN) aims to solve the optimization problem of the discrete action space [20]. A \(Q\) function is constructed with a deep neural network to evaluate the action-state pair, which is expressed as \(Q(s,a;\theta)\) and is updated by the maximum value of the \(Q\) function as Equation (16). However, Equation (16) cannot be used to solve the continuous action space problem well. \[Q(S,A)\gets Q(S,A)+\alpha\Big{[}R+\gamma\underset{a}{\text{max}}Q(S^{\prime },a)-Q(S,A)\Big{]}\] (16)
2. DDPG: To solve the untractable problem that DQN cannot deal with the continuous action space, deep deterministic policy gradient (DDPG) is proposed [21]. There are two important components in DDPG: an Actor network and a Critic network. In the Actor network, a deep neural network is used to generate a deterministic policy \(\mu(s;\theta_{t})\) and then select the optimal action from the continuous action space, which can obtain the maximum value of \(Q\). In the Critic network, \(Q(s,a;\theta_{q})\) is used to evaluate the action's effect as the same as DQN. To solve the problem that the updated target continues to change when the network is updated, which makes the updating difficult, fixed network technology is used to generate the copies of the Actor network and the Critic network as the target network, and to generate the target value. The target network is softly updated by the following formula: \[\theta_{t^{\prime}}=\tau\theta_{i}+(1-\tau)\theta_{t^{\prime}},\ i=\mu\text{ or }q\] (17) Since a certain policy is generated by the Actor network, fully exploring the environment, the noise \(N_{a}\sim\mathcal{N}(a_{t},1)\) is added to \(\mu\) to overlap a random sample of normal distribution with \(a_{t}\) as the mean. Here, the new policy \(\widetilde{\mu}\) can be expressed as \[\widetilde{\mu}(s_{t})=\mu\big{(}s_{t};\theta_{u}\big{)}+\mathcal{N}\] (18)
3. TD3: The Twin Delayed Deep Deterministic (TD3) policy gradient algorithm is an optimized version of DDPG [22]. DDPG is mainly improved by TD3 through three aspects: double Q network, delayed actor update, and the target policy smoothing regularization. **Target policy smoothing regularization:** When the target policy network outputs actions, noise following standard normal distribution is added, so that the estimation of Q is more accurate and the network is more robust. The output action is denoted as \(\widetilde{a}\), which is expressed by \[\widetilde{a}=\Pi_{\varphi^{\prime}}(s^{\prime})+\varsigma\] (19) where \(\varsigma\sim\text{clip}(\mathcal{N}(0,\sigma),-c,c)\) represents the normal noise with its upper and lower limits \(\pm c\). **Double Q network:** Two sets of networks are designed in TD3 to estimateQ, and the smaller one is regarded as the update target. The update target Q network is denoted as \(y_{t}\) and expressed as follows:  
\[y_{t}\gets r_{t}+\gamma\text{min}_{i=1,2}Q_{\theta_{i}}(s^{\prime}_{t+1}, \bar{a}) \tag{20}\]  
where \(s^{\prime}_{t+1}\) denotes the next state in minibatch, \(\bar{a}_{t}\) is the action after smoothing regularization, \(\gamma\) is the discount factor, \(r_{t}\) is the reward what we designate at the current state.