# Safety-Aware Optimal Attitude Pointing for Low-Thrust Satellites
## 2 Methods
### Reinforcement Learning  
In this subsection, we specify the observation and reward functions that we used in conjunction with MATLAB's Reinforcement Learning Toolbox software. In this software, the policy that selects the best action to take is represented in the form of a neural network. In this case, the action is an amendment of the values \(c_{1}^{0},c_{2}^{0},c_{3}^{0}\); the action output is added to the \(c_{i}^{0}\) values selected in Step 1 in order to choose a new path such that the obstacle is avoided.  
The observation sent to the RL block consists of detecting the boundary of the sets \(\mathcal{S}_{1}\) and \(\mathcal{S}_{2}\), and the reward function \(r(t)\) is set up in order to ensure that the condition in (8) is maintained. In particular, when the antenna intersects the boundary of the pointing set \(\mathcal{S}_{2}\), the condition (denoting \(\mathbf{b}=[x_{b},y_{b},z_{b}]\)) is expressed by a logical value:  
\[obs_{2}(t)=\begin{cases}1&\text{if }x_{b}^{2}+y_{b}^{2}>=r_{2}^{2}\;\;\text{ and }\;z_{b}>=0.791\\ 0&\end{cases} \tag{18}\]  
where \(r_{2}=\sin(2\phi)\) and \(\phi=0.175\). Since the antenna axis is taken as \([0,0,1]\), the boundary of the region is clearly a plane parallel to the x-y-axis, and \(z_{b}=0.791\) where it intersects the z-axis arises by choosing \(\phi=0.174\); therefore, the coordinates of the boundary projected  
Figure 5: Correcting for antenna pointing. The initial trajectories constructed between the points (**a**) and the same trajectories after choosing the \(c_{i}\) values using reinforcement learning (**b**). The initial points are indicated by \(+\) and the final points by \(*\) (initial and final points from Equation (23)).  
on the plane) are given by \(X_{b}=\sin(2\phi)\cos(\theta)\), \(Y_{b}=\sin(2\phi)\sin(\theta)\) (for \(\theta\in[0,2\pi]\)), and \(z_{b}\) can be calculated from the inverse stereographic projection through the south pole,  
\[z_{b}=\frac{1-X_{b}^{2}-Y_{b}^{2}}{1+X_{b}^{2}+Y_{b}^{2}}.\]  
To determine the boundary of the set \(\mathcal{S}_{1}\), we used the projection from sphere to plane,  
\[[X,Y]=\left(\frac{x}{1-z},\frac{y}{1-z}\right)\]  
to map the vector \(\mathbf{a}=[x_{a},y_{a},z_{a}]\) to the plane where it has the form \([X_{a},Y_{a}]\) and the Sun vector \(\mathbf{s}=[x_{s},y_{s},z_{s}]\) to the plane where it has the form \([X_{s},Y_{s}]\), and we calculated the value:  
\[A=(X_{a}-X_{s})^{2}+(Y_{a}-Y_{s})^{2}.\]  
The intersection of \(\mathbf{a}\) with the boundary of the set \(\mathcal{S}_{1}\) is then described by:  
\[obs_{1}(t)=\begin{cases}1&\text{if }A\leq r_{1}^{2}\\ 0&\end{cases} \tag{19}\]  
where \(r_{1}=0.25\). The reward is correspondingly calculated as:  
\[r(t)=\begin{cases}-10&\text{if }obs_{1}(t)=1\ \ \vee\ \ obs_{2}(t)=1\\ 10&\text{if }\ obs_{1}(t)=0\ \ \wedge\ \ obs_{2}(t)=0.\end{cases} \tag{20}\]  
The RL-block then uses a neural network to calculate the three action outputs based on the values of the reward and observation functions. We chose the simple reward function (20) to avoid the issues of \"reward shaping\", which may lead the agent to make unnecessarily elaborate maneuvers to increase the reward. A drawback of the Reinforcement Learning Toolbox is that the neural network defining the policy function component is a \"black box\", and so, we had no real insight into the scaling involved. In this case, it was noticed that the policy was outputting actions to the order of \(10^{-3}\), so it was required to introduce a scaling of the actions to produce reasonable results. After trial and error, we determined that a scaling of 100 was efficient. Some further insight into the neural networks used in the policy function could have assisted in choosing a scaling term less heuristically.