## 3 Intelligent Tuning Based On DDPG
### Reward Shaping Inspired by Human Knowledge
Therefore, we can use the sum of each cosine value of the angle \(\alpha_{j}\) between \(\mathbf{v}_{j}^{\text{(current)}}\) and \(\mathbf{v}_{j}^{\text{(target)}}\) to calculate the directional difference of two S-parameters curves, i.e.,  
\[\cos\alpha_{j}=\frac{\mathbf{v}_{j}^{\text{(current)}}\cdot\mathbf{v}_{j}^{ \text{(target)}}}{\left|\mathbf{v}_{j}^{\text{(current)}}\right|\left|\mathbf{v }_{j}^{\text{(target)}}\right|}, \tag{8}\]  
where  
\[\alpha_{j}=\Big{\langle}\mathbf{v}_{j}^{\text{(current)}},\mathbf{v}_{j}^{ \text{(target)}}\Big{\rangle}. \tag{9}\]  
The intuition to use cosine values is pretty simple. In order to adjust screws to maximally drive the current S-parameters curve to reach the target, we want each point on the curve to move as far as possible toward the goal. The cosine value of two vectors within the range \([-1,1]\) describes the similarity between the current direction and the target direction. The closer the current vector is to the target, the higher the value. More specifically, the cosine value of angle \(\alpha_{j}\) in Figure 4 is larger than 0 because \(\alpha_{j}\) is less than 90\({}^{\circ}\), representing that \(\mathbf{v}_{j}^{\text{(current)}}\) and \(\mathbf{v}_{j}^{\text{(target)}}\) almost share the same direction. Conversely, the value of angle \(\alpha_{j}\) in Figure 4 will be less than 0 since \(\alpha_{j}\) is larger than 90\({}^{\circ}\), which indicates that the two vectors have opposite directions. Therefore, the third shaping reward is formulated as  
\[r_{t}^{\text{direction}}(s_{t},a_{t},s_{t+1})=\sum_{j=1}^{n}\cos\alpha_{t,j}, \tag{10}\]  
where  
\[\cos\alpha_{t,j}=\frac{\mathbf{v}_{j}^{\text{(t)}}\cdot\mathbf{v}_{j}^{\text {(target)}}}{\left|\mathbf{v}_{j}^{\text{(target)}}\right|\left|\mathbf{v}_{j }^{\text{(target)}}\right|}, \tag{11}\]  
in which \(\mathbf{v}_{j}^{\text{(t)}}\) denotes the vector connecting the point \(j\) on the \(S_{11}\) curve before and after tuning at time step \(t\). The cosine value only describes the direction of movement of each point on the curve. In addition, the length of each vector represents the extent of the movement. So, we multiply  
Figure 4: Formulate the change of S–parameter curves with vector fields: (**a**) The angle \(\alpha_{j}\) is obtuse denoting that the current change direction is moving away from the target. (**b**) The angle \(\alpha_{j}\) is acute denoting that the current change direction is approaching the target. the cosine value by the moving amount of that point in the target direction. Therefore, the human-experience-shaped reward can be improved by  
\[r_{t}^{\text{direction}}(s_{t},a_{t},s_{t+1})=\frac{1}{n}\sum_{j=1}^{n}\frac{ \left|\mathbf{v}_{j}^{(t)}\right|}{\left|\mathbf{v}_{j}^{(\text{target})} \right|}\cdot\cos\alpha_{t,j}. \tag{12}\]  
Therefore, the generated shaping reward inspired by human experience naturally shows the overall degree to which the curve changes in the direction of the target. The overall reward function is the sum of these three parts:  
\[r_{t+1}=\lambda_{1}\cdot r_{t+1}^{\text{goal}}+\lambda_{2}\cdot r_{t+1}^{ \text{limit}}+\lambda_{3}\cdot r_{t+1}^{\text{direction}}, \tag{13}\]  
where \(\lambda_{1}\), \(\lambda_{2}\), and \(\lambda_{3}\) are weights for a trade-off.