## 3 Anti-Interference Power Control Based on Reinforcement Learning  
RL enables agents to optimize actions through feedback from environmental interactions. In this study, we developed a transmitter anti-interference communication scheme, HPTR, which leverages RL-based optimization to select relay nodes and transmission power under dynamic interference and uncertain channel conditions. The proposed scheme employs a test reference least squares (TRL) algorithm based on mixed strategies. By incorporating prior knowledge from similar scenarios, the algorithm accelerates the learning process, improving efficiency and effectiveness in complex and dynamic underwater environments. State space: The transmitter's performance is directly influenced by the state space. At slot \(k\), the transmitter evaluates the previous communication performance, including the signal-to-noise ratio (SINR) of the direct link \(SINR_{5,T}^{(k-1)}\) and the bit error rate (BER) over the entire communication range \(b^{(k-1)}\). Thus, the state space is represented as \[s^{(k)}=\Big{[}SINR_{S,T}^{(k-1)},b^{(k-1)}\Big{]} \tag{1}\]  
Action space: Based on the policy distribution \(\pi\Big{(}s^{(k)},\cdot\Big{)}\) and Q-value \(Q\Big{(}s^{(k)},\cdot\Big{)}\) of the current state, the transmitter selects an anti-jamming strategy \(a^{(k)}=\Big{[}x^{(k)},p_{s}^{(k)}\Big{]}\in A\). Here, the relay triggers factor \(x^{(k)}\in\{0,1\}\) and the transmitting power \(P_{S}^{(k)}\in\{j\cdot P_{S}^{\max}/M|0\leq j\leq M\}\), where \(M\) represents the number of discrete power levels. The probability of selecting strategy \(a^{(k)}\) is as follows:  
\[\Pr\Big{(}a=a^{(k)}\Big{)}=\pi\Big{(}s^{(k)},a^{*}\Big{)},a^{*}\in A \tag{2}\]  
Reward function: The reward function is a key component of the learning process, as it directly impacts the decision-making strategy of the transmitter. Based on the selected anti-jamming strategy of the transmitter, when \(x^{(k)}=0\), the transmitter transmits the signal to the receiver through a direct link. Otherwise, the transmitter triggers the relay node for information transmission. After receiving feedback from the receiver, the signal-to-noise ratio (\(SINR\)) and the relay trigger factor at the destination are calculated. The reward function is expressed as  
\[ \begin{split} R^{(k)}&=x^{(k)}\max\Big{\{}SINR_{S, T}^{(k)},\min\Big{\{}SINR_{S,R}^{(k)},SINR_{K,T}^{(k)}\Big{\}}\Big{\}}\\ &+\Big{(}1-x^{(k)}\Big{)}SINR_{S,T}^{(k)}-C_{S}P_{S}^{(k)}\end{split}  \tag{3}\]  
where the first term represents the effectiveness of the selected transmission strategy (either direct or relay-assisted) based on the \(SINR\). The second term penalizes energy consumption by weighting the transmitter's power cost. The Q-function (\(s\)), representing the expected utility of taking action \(a\) in state \(s\), is updated iteratively using the Bellman equation as follows:  
\[Q(s,a)\leftarrow(1-\alpha)Q(s,a)+\alpha\big{(}U_{R}(s,a)+\delta V\big{(}s^{ \prime}\big{)}\big{)} \tag{4}\]  
where \(s^{\prime}\) is the next state resulting from action \(a\), and \(V(s^{\prime})\) is the value function, defined as the maximum \(Q(s,a)\) across all possible actions:  
\[V(s^{\prime})\leftarrow\max_{a\in A}Q(s,a) \tag{5}\]  
In addition, the learning rate \(\alpha\in(0,1]\) represents the weight of current experience, while the discount factor \(\delta\in[0,1]\) represents the degree of uncertainty about future utility. Based on the mixed strategy table represented by \(\pi(s,a)\), the probability of other actions is reduced by increasing the probability corresponding to the actions with the largest Q-value. \[\pi(s,a)\leftarrow\pi(s,a)+\left\{ \begin{array}{l}\beta,a=\underset{a\in A }{\operatorname{argmax}}Q\Big{(}s^{(k)},a\Big{)}\\ \frac{-\beta}{|A|-1},o.w.\end{array} \right. \tag{6}\]  
The proposed TRL algorithm leverages experience from large-scale UACNs in anti-jamming tasks to initialize Q-values and policy distributions for the current scenario.