# Crowdsourcing Framework for QoE-Aware SD-WAN +
## 6 Performance Evaluation  
### HPC Experiment Analysis  
For the HPC experiment, there were 2430 paired comparison questions with 259 participants. A total of 243 participants were successful, and 16 failed the reliability test. Ranks were computed using the Crowd Bradley-Terry model [24]. Figure 7 shows participant ratings for Video 1 for each protocol and scenario of events. We collected similar results for other videos in Table 1. Using MSU-VQM, we computed VQM for processed videos from Experiment 1 against the original video. Table 3 shows VQM values for each resulting video.  
Table 4 shows the highest ranked protocol in HPC against the best VQM value for each video across different scenarios. Among all the events and scenarios that we studied in the HPC experiment, RTP had the highest rating on average. The same results were concluded from VQM values, i.e., RTP had the lowest VQM values, followed by TCP. UDP only stood out with better values and ranking in Video 3. For Video 1, it was noticed that TCP had the highest VQM values across all scenarios except in scenario 5, with a very low difference compared to RTP; however, in HPC, it was ranked either second or first with a very close rating difference to RTP. UDP on the other hand had the lowest rating among all scenarios in Video 1. It was noted during Video 2 analysis that there was a high ranking gap between RTP and the other two rated protocols in HPC experiment; however, this gap was not noticeable in VQM values. UDP was ranked the lowest among three scenarios out of five in Video 3 at HPC ranks, whereas, in VQM it had the highest values in only two scenarios. For Video 4, TCP ratings in HPC were consistent with VQM calculated values.  
By comparing subjective and objective results, we found that HPC and VQM results were consistent, where the highly ranked protocols in HPC mostly had the best (lowest)  
Figure 6: SD-WAN testbed network topology.  
VQM value across the three protocols per video. This result demonstrates that analysis and human-perceived quality do provide consistent results.  
We note that the sequence of events in the scenarios in Table 2 was designed to allow us to monitor how protocols differ in recovering from packet loss and delay (research question 3), how the QoE will be affected, and if participating users will detect changes with one parameter fixed and the others changed. It was found in our experiments that, for the packet loss events, the quality degradation was most noticeable; however, changing  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Scenario** & **1** & **2** & **3** & **4** & **5** \\ \hline \multicolumn{6}{c}{**Video 1**} \\ \hline
**UDP** & 3.45 & 3.15 & 5.29 & 3.18 & 3.86 \\
**RTF** & 3.51 & 3.15 & 3.45 & 3.15 & 3.32 \\
**TCP** & 7.28 & 8.52 & 5.35 & 3.68 & 4.28 \\ \hline \multicolumn{6}{c}{**Video 2**} \\ \hline
**UDP** & 4.31 & 4.15 & 4.39 & 4.10 & 3.94 \\
**RTF** & 4.33 & 4.08 & 4.36 & 4.18 & 4.37 \\
**TCP** & 4.10 & 4.10 & 4.11 & 4.09 & 4.30 \\ \hline \multicolumn{6}{c}{**Video 3**} \\ \hline
**UDP** & 2.15 & 1.82 & 2.46 & 2.39 & 2.37 \\
**RTF** & 2.71 & 1.63 & 3.05 & 1.84 & 2.74 \\
**TCP** & 2.49 & 2.33 & 2.20 & 2.26 & 2.47 \\ \hline \multicolumn{6}{c}{**Video 4**} \\ \hline
**UDP** & 2.53 & 2.42 & 2.45 & 2.49 & 2.64 \\
**RTF** & 2.55 & 2.09 & 2.90 & 2.96 & 2.57 \\
**TCP** & 2.01 & 2.60 & 2.09 & 2.18 & 2.61 \\ \hline \hline \end{tabular}
\end{table}
Table 3: HPC experiment scenarios.  
Figure 7: HPC QoE Rating results for Video 1 using Crowd Bradley–Terry model.  
the delay parameters in some cases was not noticeable by participants, likely due to the buffering ability of the video player on the client side.