# Crowdsourcing Framework for QoE-Aware SD-WAN +
## 2 Related Works  
A comprehensive discussion of network and service performance indicators for multimedia applications was presented in [1]. The most important performance indicators include the following:  
* One-way end-to-end delay (including network, propagation, and equipment) for video or audio should be within 100 to 150 ms.
* Mean opinion score (MOS) levels for audio should be within 4.0 and 5.0. MOS levels for video should be between 3.5 and 5.0. * End-to-end delay jitter must be short, normally less than 250 us. * Synchronization of intermedia and intramedia should be maintained using suitable algorithms. To maintain intermedia synchronization, the differential delay between audio and video transmission should be within \(-\)20 ms to +40 ms. The following parameters should also be taken into consideration while designing a QoE framework for multimedia services [4]:  
* Video quality at the source. * How the content is delivered over the network and QoS quality parameters. * User perception, expectations, and ambiance. Objective QoE measurements rely on quantitative measures from the network and end-devices to predict users' perception of video quality. As mentioned earlier, PNSR is one such metric that has been used extensively [3]. One drawback of PSNR is that it does not take into consideration how human perception works. The structural similarity index (SSIM) [8] is another measurement approach for estimation of perceived visual distortion based on the structural distortion of the video. SSIM addresses the shortcoming of PSNR by combining other factors such as contrast, luminance, and structure similarity, and it compares the correlation between the perceived video images and the original video images; hence, it can be considered a full-reference model, where a higher ratio denotes a higher structural similarity. Another objective approach is the Video Quality Metric (VQM) [9], which is a metric to measure the perception of video quality as closely as possible to human perception. The VQM metric is designed to be a general- purpose quality model for a range of video systems with different resolutions, frame rates, coding techniques, and bit rates. VQM measurement takes noise, blurring, and block and color distortions into consideration. VQM gives an output value of zero if no impairment is perceived, and it increases with a rising level of impairment. Research on quantitative QoE estimates has more recently focused on the adaptation of such measures in cloud and content streaming networks. For instance, in [10], the authors used Server- and Network-Assisted DASH (SAND) to provide communication between the end device and the network node, such as a WiFi access point, and to reallocate network resources to avoid empty buffers on the client side that cause QoE degradation. In [11], the authors proposed a framework for dynamic adaptation of bitrate to the end nodes to provide a fair allocation of resources to all users. In both cases, network edge nodes are involved in quality estimation and resource allocation. Subjective QoE measurements focus on adopting and improving MOS scores for various multimedia services. The focus is mainly on definition of the right score, as well as on methods to collect them efficiently from the users. More recently, crowdsourcing techniques have been considered for collecting user QoE feedback. In [5], the authors designed a crowdsourcing framework that overcomes some of the disadvantages of the MOS technique, namely, (1) difficulty and inconsistency for participants to map their ratings to five-point MOS scaling, (2) rating scale heterogeneity, and (3) the lack of a cheat detection mechanism. This new approach provides comparable consistency to the MOS methodology by introducing the ability to have QoE measured in a real-life environment using crowdsourcing rather than a controlled environment in a laboratory.