# Crowdsourcing Framework for QoE-Aware SD-WAN +
## 5 Test Bed Implementation
### Use Case Scenarios  
Two experiments were conducted using human participants, one based on paired video comparison using the QoE crowdsourcing technique and the other based on real-time feedback during video streaming in a controlled lab environment. The purpose of these experiments was to attempt to provide answers to the research questions that were posed in Section 1, i.e., to have a better understanding of how quickly and reliably the participants can identify quality degradation, how their results compare to the objective analysis of videos, and how different transport protocols perform with regard to QoE of multimedia content. The first experiment, human-paired comparison (HPC), was tested using a QoE crowdsourcing campaign of a paired video comparison on a set of processed videos using the subjecty.us web service [29]. Comparable video sets were created on the basis of four selected HD videos. Each video was 40 s in time, which allowed sufficient time for introducing quality degradation and user reactions without being too long. The videos also varied in bitrate and frame rate, as well as in content type, so that the analysis of results for research questions 1-3 could be reasonably separated from video quality and content. Details of these videos are presented in Table 1.  
The scenario files were created by changing delay and loss events every 10 or 5 s, as shown in Table 2, while RTP, Legacy UDP, and RTP over TCP were applied as streaming protocols in different scenarios. These processed videos were used as the golden question for verifying the reliability of participants in comparing the original HD video to the processed (deteriorated) video. If participants chose the processed copy, then their feedback results were considered untrustworthy. The choice of different transport protocols and network conditions allowed us to evaluate results for research questions 4-5. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & **Content** & **Frame Width** & **Frame Height** & **Bit Rate** & **Frame Rate** \\ \hline Video 1 & Beach waves & 1920 & 1080 & 17,388 kbps & 50 fps \\ Video 2 & Ski views & 1920 & 1080 & 5141 kbps & 30 fps \\ Video 3 & Animation & 1920 & 1080 & 1249 kbps & 60 fps \\ Video 4 & Wild animals & 640 & 360 & 1554 kbps & 30 fps \\ \hline \hline \end{tabular}
\end{table} 
Table 1: HPC video specifications. For the second experiment, random scenario files of 40 s in length were created with timed events changing every 0.5 s during the video streaming session. These scenarios were applied to three HD-quality videos, while RTP, Legacy UDP, SCTP, RTP over UDP, and RTP over TCP were applied as streaming protocols with a result of 150 processed videos. Using the MSU-VQM objective analysis tool [30], comparative results were computed against the original videos in terms of PSNR, VQM, and SSIM. For these experiments, a network topology was created consisting of one controller, one switch, and two hosts. By using the minievents module script [31], we applied dynamic changes to adjust delay and loss over a period of time during video real-time streaming, in order to measure the impact of changes in network conditions on QoE. The parameters of our event scenarios were selected to check the following three network condition scenarios:  
1. Fixed link delay and changing packet loss over time. 2. Switching from high delay to a substantial lower delay with a fixed packet loss. 3. Gradually decreasing delay over time and then gradually increasing it while applying packet loss. The selection of these particular scenarios was to evaluate how different transport protocols and user perception respond to abrupt or gradual packet loss or delay. In our experiments, link delay was set in the range of 0-80 ms, and packet loss was set in the range of 0-1%. The flow bandwidth was fixed at 50 Mbps. The values of link delay, loss, and bandwidth were chosen through trial and error to generate sufficiently large quality degradation at the user end, having no other significance in our study.