# Crowdsourcing Framework for QoE-Aware SD-WAN +
## 2 Related Works
### Crowdsourcing QoE Models  
A general crowdsourcing framework for QoE capture was discussed in [5]. The objective in that work was to overcome some disadvantages of the MOS technique by utilizing the paired comparison technique, as well as the ability to have QoE measured in a real-life environment using crowdsourcing rather than a controlled environment in a laboratory. Four case studies were conducted using audio and video content to evaluate the effectiveness of the proposed framework. The key features of using this framework for QoE evaluation are as follows:  
1. It can be generalized for different types of multimedia content with no need for adjustments. 2. Use of the pair-comparison rating technique provides a simpler user feedback comparing to MOS technique
3. Results from compared judgements can be evaluated by probability models
4. Reward and punishment schemes are used, where users are given appropriate incentives to give honest feedback in order to obtain trustable quality measures. This framework is a promising evaluation technique to measure QoE; however, as the authors of the study pointed out, this is not a QoE evaluation; rather, it is the quality of perception (QoP). As they pointed out, \"QoP reflects a user's detectability of a change in quality or the acceptability of a quality level\" [5]. The OneClick Framework [12] captures user perception in a simple one-click procedure. Whenever a user is not satisfied with the quality of the viewed content, they can click on a button that informs the system of their dissatisfaction. In contrast to the MOS technique, a user does not have to decide between different grading scales and what best suits their perception. OneClick is a real-time framework which means that the clickable button is available along the whole viewing experience. The user can record their dissatisfaction several times along the process where each click is time-captured. This framework is based on the PESQ (Perceptual Evaluation of Speech Quality) and VQM (Video Quality Metric), which are both objective measurements. The key advantages of OneClick Framework are as follows:  
1. Initiative: Participants are not required to decide about the perceived quality, whereas they only report their dissatisfaction through one click of a button. 2. Lightweight: The framework does not require any specific deployments and is not expensive to roll out. 3. Time-aware: Participants can record their dissatisfaction several times along the process where each click is time-captured, which indicates how perception can change over time. 4. Independent: OneClick can be used in conjunction with several applications simultaneously. OneClick includes two main steps: (1) experiments are held to gather users' perception feedback during different network conditions; (2) collected data are then processed to identify QoE measurements. Figure 2 shows the full OneClick process assessment technique with the following steps: (1) preparing test materials (optional); (2) asking the subjects to do the experiments; (3) inferring average response delays; (4) modeling the relationship between network factors and click rates; (5) predicting the click rate given each network factor; (6) summarizing an application's QoE over various network conditions by comfort region. Given the simplicity of OneClick's approach for user feedback collection, a similar approach was used in our work for the user side of our framework, i.e., to allow service users to express their displeasure through a click when the quality of the streaming deteriorates. Most past research contributions in this area have focused on the quality of video streaming, video analytics, network QoS, and metrics for conducting QoE. In those works, QoE was usually measured and assessed after completion of streaming session/or paired comparisons, and results were sent back to management systems to make changes for enhancing the user QoE in future streaming.