## 3 Intelligent Tuning Based On DDPG
### Reward Shaping Inspired by Human Knowledge  
After taking the action \(a_{t}\), a reward \(r_{t}\) will be obtained. The value of the reward is an evaluation of the action taken, and the goal of RL is to learn a policy \(\pi(s)\to a\) so that the expected future reward is maximized. In some cases, the reward value can be derived directly from interaction with the environment, such as playing Atari games. In other cases, the reward function should be defined manually, such as the self-driving problem. Other tasks have reward functions that are difficult to define, such as robot manipulations. In these cases, the reward is always learned by demonstration. In this work, we use the experience of human tuning to manually design the reward function to make RL have better performance. The reward \(r_{t}\) of each step consists of three parts. \(r_{t}^{\text{goal}}\) directly describes the cost of each step taken by an action. Any step other than reaching the goal is rewarded with \(-0.01\). If the action results in an eligible S-parameters curve, i.e., reaching the tuning target, the reward will be added by \(+10\). \[r_{t}^{\text{goal}}(s_{t},a_{t},s_{t+1})=\left\{ \begin{array}{ll}r^{\text{ target}}&\text{if }s_{t+1}=s_{\text{goal}}\\ r^{\text{penalty}}&\text{otherwise}\end{array} \right.. \tag{4}\]  
To impose a penalty for one or more screws exceeding the length limit, a total reward of \(-0.1\) is added. More specifically, if the action causes the screw to exceed its limit, the screw will remain unchanged and the agent will be rewarded with \(-0.1/m\), where \(m\) is the number of valid screws. The reward \(r_{t}^{\text{limit}}\) is formulated as  
\[ \begin{array}{l}r_{t}^{\text{limit}}(s_{t},a_{t},s_{t+1})=\\ -\frac{01}{m}\cdot\sum\limits_{i=1}^{m}\mathbf{1}\left[z_{t+1}^{(i)}\in\left( z_{\text{max}}^{(i)},+\infty\right)\cup\left(-\infty,z_{\text{min}}^{(i)} \right)\right],\end{array}  \tag{5}\]  
where \(\mathbf{1}[\cdot]\) is the indicator function with  
\[\mathbf{1}[x]=\left\{ \begin{array}{ll}1&\text{if }x\text{ is True}\\ 0&\text{otherwise}\end{array} \right., \tag{6}\]  
and \(z_{t}^{(i)}\) denotes the insertion position of the \(i\)-th tuning screw at the time step \(t\). \(z_{\text{max}}^{(i)}\) and \(z_{\text{min}}^{(i)}\) denote, respectively, the maximum and minimum insertion position of the \(i\)-th screw. The third part is the most important reward \(r_{t}^{\text{direction}}\), which is formed in the evaluation of transitions from the current state to the next state, inspired by human tuning experience. Take into account the experienced tuning process. Technicians are able to optimize the S-parameters curve because they can evaluate changes to the curve. In other words, they do their best to make the S-parameters curve always move in the right direction. Therefore, the change direction of the S-parameters curve to the new curve can be expressed as a set of vectors. Each \(S_{11}\) curve is truncated first, leaving only the central core area. The two curves are then discretized into the same number of sampling points. Thus, the vectors connecting two corresponding points of the curves can form a vector field. See Figure 4 for more details. In Figure 4a, the vector \(\mathbf{v}_{j}^{(\text{current})}\) represents the moving direction of the point \(A_{j}\) on the blue curve (current state) to the point \(B_{j}\) on the green dashed curve (a new state). \[\mathbf{v}_{j}^{(\text{current})}=\big{[} \begin{array}{cc}x_{bj}-x_{aj}&y_{bj }-y_{aj}\end{array} \big{]}. \tag{7}\]  
Similarly, to represent the changing direction of a curve to a target line, a set of vectors connecting each two corresponding points on the curve and the target line can be applied, as shown in Figure 4, denoted by \(\mathbf{v}_{j}^{(\text{target})}\).