# A Study of Eye-Tracking Gaze Point Classification and Application Based on Conditional Random Field
## 2 Proposed Method
### The Fusion of Features  
The relationship between the target of gaze image and gaze point features is established, and the relationship between gaze points is described.  
The graph model relationship based on CRF is shown in Figure 4, where the yellow and red points, respectively, represent the classification marker layer and the gaze point after classification, and the blue points represent the gaze point sequence described by different features. In the classification marker layer, each node is connected to its neighbors. The solid yellow line indicates that the nodes are connected to their neighboring nodes, and the red nodes are jointly affected by the class markers of the surrounding yellow nodes. In the observation layer, each blue node indicates the corresponding features, the image saliency features and gaze point features, and they are connected to the nodes in the corresponding classification marker layer, which indicates that these features affect the classification marker results.  
Each node interacts with its neighbors in the classification and observation layer, and these nodes together determine the final classification label. A fusion module about image saliency and gaze point features is constructed by applying different weights to different feature channels in order to express the attention region, which can represent the output position of the gaze point. The LSTM model [18] is introduced to predict the channel weight vector in order to fuse the image visual features with the gaze point features for training the prediction of the attention region at the next gaze region. The module takes \(G_{t-1}^{s}\) and \(G_{t-1}^{a}\) as input and outputs the predicted channel weight vector \(\omega_{t}\). \(G_{t-1}^{s}=m_{p}\) is the \(t-1\) moment salience feature of the target in the image, which is also related to \(G_{t-1}^{a}\).  
The fusion module framework is presented in Figure 5.  
Figure 4: Graph model relations based on CRF.  
In the Figure 5, the channel weight extractor \(C\) takes the previous moment saliency features \(G_{t-1}^{s}\) and the gaze point features \(G_{t-1}^{a}\) as input. From each channel, the features of the gaze point location \(G_{t-1}^{a}\) are projected to the region \(G_{t-1}^{s}\) in order to obtain the channel weights \(\omega_{t-1}\):  
\[\omega_{t-1}=C\big{(}G_{t-1}^{s},G_{t-1}^{a}\big{)}, \tag{3}\]  
where \(C\) is the nonlinear function denoting the cropping and averaging operation, and \(\omega_{t-1}\) denotes the feature representation of the region of attention around the gaze point at moment \(t-1\).  
The transfer probability \(P\) score of the gaze state is \(f_{t-1}^{P}=P\big{(}G_{t-1}^{a}\big{)}\in[0,1]\), which indicates how likely the first gaze point occurs in the target. A channel weight vector is extracted for each gaze point to learn the transitions between gaze points, and learn the relationship between gaze points. That is, a series of channel weight vectors extracted from images with gaze points are used to train the LSTM and output the probability score of gaze transfer. In the testing process, given a channel weight vector, the training outputs a channel weight vector which represents the region of gaze point at the next gaze. The predicted gaze probability channel weight vector is expressed as:  
\[\omega_{t}=f_{t-1}^{P}\omega_{t-1}+\Big{(}1-f_{t-1}^{P}\Big{)}\cdot L(\omega_{ t-1}), \tag{4}\]  
where \(L(\omega_{t-1})\) is the channel weight vector output by training.  
The mapping of gaze points predicted by gaze transfer probability is then computed as:  
\[G_{t}^{a}=\sum_{c=1}^{n}\omega_{t}[c]\cdot G_{t}^{s}[c], \tag{5}\]  
where \(c\) is the channels dimension, denotes the \(c\)-th dimension/channels of \(\omega_{t}/G_{t}^{s}\).