## 3 Evaluation Settings
### Evaluation Procedure and Performance Metrics  
With respect to the model implementation, similar to [18; 34], we re-implement _[PERSON]'s_, _[PERSON]'s_, _[PERSON]'s_, _[PERSON]'s_, _[PERSON]'s_, and _[PERSON]'s_ models based on the corresponding publications since their implementations are not publicly available [6; 7; 8; 9; 11; 12]. For the _LSTM_, _ATLAS_, _CQM_, _biQPS_, _SQI_, and _KSQI_ models, we employ the implementations publicized by the respective authors. Regarding the _P.1203_ model, an implementation of the standard that is free to use for research purposes is used [30; 35]. Note that, although the _P.1203_ model has four input modes, of which model#3 provides the highest performance, it is not applicable to some databases (namely _W-SQe-I_, _W-SQe-II_, _W-SQe-III_, and _W-SQe-IV (full)_) due to the lack of input data. Therefore, for these databases, the _P.1203_ model is tested with mode#0 while mode#3 is employed for the others. The impact of initial delay could be separately modeled by a function of its duration which is then simply integrated into QoE models as an additive component [36; 37]. Hence, in the same way, the models of _[PERSON]'s_, _[PERSON]'s_, and _[PERSON]'s_ that do not originally consider the impact of initial delay are tested in two cases of (1) original (denoted _ori_) and (2) modified (denoted _mod_) by adding the impact of initial delay. It is expected that this addition will help obtain more understanding of the performance of the models and the impact of factors as well. Particularly, since most existing models use logarithm functions to model the impact of initial delay [4; 36], such a function that is proposed in [4] with a very high prediction performance is added into _[PERSON]'s_, _[PERSON]'s_, and _[PERSON]'s_ models as follows:  
\[QoE_{pred}=Q+I_{ID}, \tag{1}\]  
and  
\[I_{ID}=-0.862 log(d+6.718), \tag{2}\]  
where \(Q\) and \(QoE_{pred}\) are respectively the predicted QoE values before and after modifying, and \(I_{ID}\) denotes the impact of the initial delay with the duration of \(d\) seconds. Because each model was developed using one or several specific training databases, these databases will be excluded from the performance evaluation of that model. In other words, the performances are calculated on only test databases. In particular, Table 3 describes the training and test databases for each model. In addition, it should be noted that, because of the lack of input data (e.g., objective quality values of segments), _[PERSON]'s_, _ATLAS_, _SQI_, and _KSQI_ models are not evaluated over some databases such as _TR04f_, _TR04p_, and _VL04_ databases as indicated by NA (i.e., not applicable) in Table 3. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Original Database** & **Sub-Database** & **Viewing Device** & **Codec** \\ \hline \multirow{8}{*}{_W-SQe-IV (full)_} & _W-SQe-IV\_pH264_ & Smartphone & H.264 \\ \cline{2-4}  & _W-SQe-IV\_pHEVC_ & Smartphone & HEVC \\ \cline{2-4}  & _W-SQe-IV\_pH264_ & FHD & H.264 \\ \cline{2-4}  & _W-SQe-IV\_pH264_ & FHD & HEVC \\ \cline{2-4}  & _W-SQe-IV\_pH264_ & UHD & H.264 \\ \cline{2-4}  & _W-SQe-IV\_uHEVC_ & UHD & HEVC \\ \hline \multirow{2}{*}{_TR04 (full)_} & _TR04p_ & Smartphone & H.264 \\ \cline{2-4}  & _TR04f_ & FHD & H.264 \\ \hline \multirow{2}{*}{_TR06 (full)_} & _TR06p_ & Smartphone & H.264 \\ \cline{2-4}  & _TR04f_ & FHD & H.264 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Settings of sub-databases. In addition, given a combination of a model and a database, a first order linear regression is applied for subjective and predicted quality values following Recommendation ITU-T P.1401 [38]. The aim is to compensate for possible variances of different databases related to subjective experiments. Regarding the rating scale, besides the 5-point scale from 1 to 5, some models (i.e., _Liu's_, _ATLAS_, _SQL_, and _KSQI_) and databases (i.e., _W-SQoE-I_, _W-SQoE-II_, _W-SQoE-III_, and _W-SQoE-IV (full)_) use the 100-point scale from 0 to 100.