# Say What You Are Looking At: An Attention-Based Interactive System for Autistic Children
## 2 Methods
### Visual Attention Prediction  
Before carrying out visual attention prediction, we need to process the results of gaze tracking. Since gaze estimation is greatly affected by the noise, we adopt head posture estimation that adjusts weights and generates more stable and accurate results according to the confidence level. We can see how to calculate the sightline from Equations (1) and (2). \[p=\frac{p_{l}+p_{r}}{2} \tag{1}\]  
\[d=\alpha_{e}\frac{d_{l}+d_{r}}{2}+\alpha_{h}d_{h} \tag{2}\]  
Parameters \(p_{l}\) and \(p_{r}\) represent the left and right eye positions, respectively. The gaze direction of the left and right eyes are \(d_{l}\) and \(d_{r}\) respectively. The parameter \(d_{h}\) represents the head direction, and \(\alpha_{e}\), \(\alpha_{h}\) are standardizing confidence levels. Equation (1) determines the equation of the sightline in 3D space, where \(p\) is the starting point of the sightline and \(d\) is the gaze direction. We then transform the linear equation from the camera coordinate to the world coordinate and represent the sight line with Equation (3). Notably, the sightline has a direction, and therefore the value of parameter \(t\) is greater than 0. \[p^{\prime}=p+td \tag{3}\]  
We then obtain the visible point cloud within the map, which is marked as \(S\), as shown in Figure 3. First, we construct a mathematical cone whose main axis has an inclination angle of \(\epsilon\). We specify that the points within this round table to be the visible point cloud \(S\) that reflects potential regions focused by people. For a point in the map, we first obtain the perpendicular foot to the sightline. After that, we calculate the distance from this point to the perpendicular foot and the distance from the perpendicular foot to the eye. By Equation (4), we determine whether this point is in the visible point cloud \(S\). \[p_{i}\in S,\ if\ tan(\epsilon)>\frac{max\Big{(}dist\Big{(}p_{i},p_{f}\Big{)}- r,0\Big{)}}{dist\Big{(}p,p_{f}\Big{)}} \tag{4}\]  
where \(p_{i}\) is a point in the map, \(p_{f}\) is the perpendicular foot of \(p_{i}\) to the sightline, \(\epsilon\) is the inclination angle, \(r\) is the radius of the circle above the frustum, and \(dist(*,*)\) is the distance between two points. The visible point cloud \(S\) determines the area where a person focuses on. However, some of the points in \(S\) are occluded. These points are not visible. Imprecise assignment of weights may distract the attention, and therefore in this paper, we propose a method to address this problem. For each visible point, we construct a cone. The direction  
Figure 2: The system takes a video as input. It builds a map and estimates the gaze simultaneously. Then, an attention heat map is predicted. Finally, the system selects the attention region to generate a sentence from the image. of the central axis of the cone is identical to the direction of the sightline. We specify that the point cloud within the cone is the occluded point cloud \(R\), as shown in Equation (5). \[p_{j}\in R,\ if\ tan(\theta)>\frac{dist\Big{(}p_{i},p_{f}\Big{)}}{dist\Big{(}p _{f},p_{j}\Big{)}} \tag{5}\]  
where \(p_{i}\) is a point in visible point cloud \(S\), \(p_{f}\) is the perpendicular foot of \(p_{i}\), \(p_{j}\) is a point that needs to be checked, and \(\theta\) is the inclination angle. We eliminated all the points in \(R\) so that the remaining point cloud reflects the actual attention region of the person. We assigned weights to the visible point cloud by constructing a Gaussian model, which is guided by the distance from points to sightline. The weight of each point cloud is inversely correlated with the distance, which implies that closer points allocated bigger weights. For convenience, we specify that the weight is in an interval from 0 to 1. On the basis of the 3D map with weight, we use a camera model to project this map onto an image plane. Subsequently, the discrete points with weight can be transformed into a continuous probability map using image processing techniques, such as dilation and filtering.