## 4 Design
### Model
#### 4.3.1 Factor Machine  
Based on the characteristics of the datasets, it is essential to first focus on capturing low-order interactions between the features. To achieve this, we compare several regression models capable of learning these interactions, including linear regression, generalized linear models (GLM), FM, and polynomial regression. These models are evaluated for their effectiveness in modeling low-order feature relationships. Among them, FM stands out due to its ability to handle large-scale sparse data, automatically learn feature interaction information, and offer strong generalization abilities. In contrast, other models may perform less effectively on small sample datasets and large-scale sparse data. Consequently, FM is selected as the component for the learning of low-order interaction information.  
FM [32] extends the linear model to capture low-order interactions in sparse datasets [33]. Structurally, FM consists of two components: order-1 and order-2. The order-1 component is the LR linear combination, which is used to learn the linear weight \(w_{i}\) of each feature \(x_{i}\). The order-2 component is the cross-term to address feature combination by using the hidden vectors \(v_{i}\) to measure the interaction effects between features. To avoid dimension explosion when dealing with feature interactions, FM expresses these interactions as the inner products of implicit vectors, rather than resorting to a comprehensive parameter matrix, reducing the model's parameter count. Moreover, FM capitalizes on data sparsity, computing the inner products of implicit vectors solely when the feature values are non-zero. This approach not only captures the interactions between feature pairs but also enhances the model's ability to represent complex relationships, thus improving the prediction accuracy. The function of FM is depicted in Equation (2), which illustrates how these components work together to achieve accurate predictions.  
\[f(x)=\omega_{0}+\sum_{i=1}^{n}w_{i}x_{i}+\sum_{i=1}^{n-1}\sum_{j=i+1}^{n}(v_{ i}^{T}v_{j})x_{i}x_{j} \tag{2}\]  
where \(\omega_{0}\) is the bias term and \(w_{i}\) is the weight of a single feature. \((v_{i}^{T}v_{j})\) represents the inner product of the feature hidden vectors as a way to define the feature cross-weight values of features \(x_{i}\) and \(x_{j}\). \(\omega_{0}+\sum_{i=1}^{n}w_{i}x_{i}\) represents the expression in order-1, and \(\sum_{i=1}^{n-1}\sum_{j=i+1}^{n}(v_{i}^{T}v_{j})x_{i}x_{j}\) represents the cross-term of features in order-2.  
In Figure 7, the calculation process of the order-2 cross-terms for a total of n features is illustrated. The inputs are features from the dataset; they are then processed by the embedding layer, which transforms them into hidden vectors. The order-2 cross-terms are derived from the hidden vector and the features. Then, these order-2 cross-terms are used  
Figure 6: Architecture of MovePerf’s prediction model.  
as inputs to the DFNN. The DFNN learns high-order feature interactions by processing these cross-terms and further refining the predictions.  
This process allows the model to integrate both low- and high-order feature interactions, enhancing its predictive performance.