# DevEmo--Software Developers' Facial Expression Dataset
## 5 DevEmo Dataset
### Threats to Validity  
During the construction of the dataset, a methodological analysis of threats to the validity of the study was carried out in accordance with the approach proposed by [PERSON] et al. [28]. This allowed threats to be identified and categorised into three types: internal validity, external validity and construct validity.  
The first type of threats is internal validity, which refers to factors that can affect the independent variables [28]. Within this group, the identified threat is the manual annotation of emotions, a subjective assessment of facial expressions that depends on the experience of the annotator, which can lead to misinterpretation of the participant's behaviour (e.g., mistaking a frown for confusion). To minimise this threat, all recordings were assessed by three annotators. In addition, a detailed annotation protocol was prepared which all researchers were required to follow.  
The highest number of threats was identified in the external validity group, which refers to factors that limit the ability to generalize results [28]. The first of the potential threats considered is the Hawthorne effect, which means that the participant's awareness of being observed may change the way they behave [29]--either intensifying or weakening the expression of emotions. Participants were aware that the purpose of the study was to identify emotions while solving programming tasks, which may have resulted in some bias. However, the design of the study, which involved participants in their natural working environment, ensured that this threat was not minimized in this study.  
However, such a natural working environment raises another threat to external validation. Lighting conditions and the position of the camera capturing participants' facial expressions can affect the reliability of annotations. This threat cannot be mitigated, as the primary objective was to develop the dataset in an uncontrolled environment.  
Covering the face is another of the risks identified. However, as with the previous threat, due to the desire to record the natural behaviour of the participants, no instruction was given in any way on how they should behave. In order to minimise this risk, it was decided not to include fragments in the dataset where participants covered their face with their hand or other objects.  
Finally, the most important of the risks identified is the sampling of participants. Only 33 Polish students aged between 19 and 21, studying at the same university and department, took part in the study. This means that deep learning models trained on data sets may be less effective at recognising the emotions of people of different ages or races.  
The final threat identified belongs to the group of construct validity, which refers to the ability to generalize the results of an experiment to the concept or theory underlying the experiment [28]. An incomplete or inadequate selected emotion model, which does not cover the entirety of the respondents' expressions, may compromise the validity of the labelled data. To address this threat, we relied on industry standards (the Ekman model). However, the results of the study indicate that when it came to recognising students' emotions while solving programming tasks, this was not the most accurate choice. Of the seven basic emotions, only three of them (happiness, surprise and confusion) were identified more than once in the annotation process.