TITLE: Identify fetal yawns based on temporal dynamics of mouth opening. 
AUTHORS: Damiano Menin, Angela Costabile, Flaviana Tenuta, Harriet Oster, Marco Dondi

[DESCRIPTION]
<div class = "text-blocks"><div class = "text-block">In these two studies, we examined the temporal dynamics of yawning and non-yawn mouth openings in preterm neonates comparable in developmental level to fetuses observed in ultrasound studies (about 31 weeks postmenstrual age, PMA).  In Study 1 we tested the reliability and construct validity of the only quantitative measure for identifying fetal yawns in the literature, by comparing its scores with a more detailed behavioral coding system (The System for Coding Perinatal Behavior, SCPB) adapted from the comprehensive, anatomically based Facial Action Coding System for Infants and Young Children (Baby FACS). In Study 2 we developed and tested a new machine learning system based on support vector machines (SVM) for identifying yawns.</div></div>

[STEPS]
?. [Procedure and video-recording]
Neonates were observed while they were lying supine in a cot. Behavior was video-recorded (24 frames per second) for 10 to 30 minutes (M =18.63, SD = 6.31), at a midpoint in the feeding cycle, when the neonates were not receiving any stimulation through routine nursing or medical care.
?. [Coding methods]
Frame by frame coding of the video-recordings was performed by two independent coders expert in FACS, Baby FACS and micro-analysis of neonatal behavior. The secondary coder did reliability coding for 41% of the video recordings. Videos were coded using ELAN, professional software for the creation and management of complex annotations on video and audio (Max Planck Institute for Psycholinguistics, The Language Archive, Nijmegen, The Netherlands; http://tla.mpi.nl/tools/tla-tools/elan/).
?. [Coding methods]
Coders identified mouth opening in the preterm neonates every time the lips parted (AU 25) and the mouth stretched widely open (AU 27) simultaneously.
?. [Coding methods]
In order to replicate the Reissland et al. (2012) procedure for identifying mouth openings as yawns, coders scored two different time intervals for each mouth opening:total duration was coded from the onset of mouth opening, the first frame where the mouth opening motion was visible, to the offset, the last frame where mouth opening was visible. plateau duration was coded from the first frame (plateau onset) to the last frame (plateau offset) during which maximum mouth opening was maintained.Mouth openings were subsequently categorized as yawns or not yawns based on whether or not they satisfied the Reissland et al. criterion defining yawns as mouth openings in which “the time to maximum opening of the mouth was of longer duration than the time from maximum opening to closing” (Reissland et al., 2012; p. 3).Consistent with the definition provided in the Fetal Observable Movement System, the plateau (the portion of the episode where mouth opening remained at its apex), albeit scored separately from the opening and closing phases, was considered “to be part of the opening rather than the closing phase” (p. 169), and the closing phase was timed from the end of the plateau.
?. [Coding methods]
Two independent expert FACS, and Baby FACS coders identified mouth openings as yawns according to the following definition from The System for Coding Perinatal Behavior (SCPB) based on the Action Units (AUs) described in the comprehensive, anatomically based Facial Action Coding System for Infants and Young Children (Baby FACS, Oster 2017)Yawning (AU 94) is a stereotyped behavior characterized by a slow mouth opening with deep inspiration, followed by a brief apnea and a short expiration and mouth closing. One of the characteristic features of yawning is its timing, with a gradual acceleration followed by an abrupt deceleration of the facial actions involved. Yawning usually emerges from a relaxed face, initially involving mouth stretching widely open (AUs 25 + 27) and upper eyelids drooping (AU 43).Although the specific AUs accompanying yawns vary, at apex they may include tightly closed eyelids (AUs 6+7+43), flattened tongue shape (AU 76b), and swallowing (AU 80). During the plateau, brow knitting (AU 3), brow knotting (AU 4), nose wrinkling (AU 9), lateral lip stretching (AU 20), nostril dilatation (AU 38) and head tilting back (AU 53) may occur. In this phase, the expansion of the pharynx can quadruple its diameter, while the larynx opens up with maximal abduction of the vocal cords [21]. Yawning is often accompanied by limb stretching and other bodily movements.The SCPB is a recently developed coding scheme based on frame-by-frame analysis of video-recorded material. The system focuses on fetuses, preterm and full-term neonates and aims to identify and reliably code the repertoire of complex patterns of behaviors observable from the last trimester of pregnancy to the first month of age. The SCPB is a system based on the Action Units (AUs), Action Descriptors (ADs), Miscellaneous Actions and Supplementary Codes described in Baby FACS for identifying discrete facial muscle actions and more complex, configurationally defined patterns of facial behaviors.
?. [Inter-rater reliability]
Cohen’s Kappa was used to assess inter-rater reliability between the primary and secondary coders. Reliability was separately assessed for identifying mouth opening and for identifying yawns. The assessment of inter-rater reliability for identifying mouth opening, using a time window of one second for both onset and offset, resulted in an acceptable agreement between coders (kappa = .72).Inter-rater reliability in identifying yawns was then assessed for Criterion A and criterion B. Acceptable reliability was obtained using Criterion A (kappa = .64), while perfect inter-rater agreement was found using criterion B (kappa = 1).
?. [Study 1]
Data analysisEach mouth opening episode was categorized dichotomously as a yawn (1) or non-yawn mouth opening (0) according to Criterion A and Criterion B. In order to assess the accuracy of the measure adopted by Reissland et al. (2012), we used a contingency table to assess its sensitivity (i.e. the true positive rate) and specificity (i.e. the true negative rate), and compute Cohen’s Kappa.
?. [Study 2]
In Study 2 we conducted a more detailed analysis of the temporal dynamics of yawning to assess the feasibility of distinguishing yawning from non-yawn mouth opening based solely on temporal cues coded from videos or US scans of mouth and jaw movements.Following a first exploratory phase, we adopted a support vector machine (SVM) approach that was cross-validated on the same sample of preterm neonates used in Study 1. SVMs are supervised learning models with associated learning algorithm that analyze data used for classification and regression analysis, supporting high dimensional data. These methods are widely used in different research fields such as bioinformatics, text mining, face recognition and image processing and are regarded, along with neural networks and fuzzy systems [28], as a state-of-the-art tool for machine learning.
?. [Study 2]
CodingIn addition to coding the total duration of mouth opening and duration of the plateau, as defined in Study 1, we calculated the following variables:Duration of the opening phase: from mouth opening onset to plateau onsetDuration of the closing phase: from plateau offset to mouth opening offsetOpening/closing asymmetry: difference between the durations of the opening and closing phasesOpening/closing ratio: ratio of the duration of the opening phase to the duration of the closing phase
?. [Study 2]
ReliabilityIn order to establish the reliability of these measures, we compared the duration of the three phases (opening, plateau and closing) as scored by the two independent coders for Study 1. The differences between the two coders were deemed acceptable for the duration of each of the three phases: opening (Median = .08 s), plateau (Median = .10 s) and closing (Median = .12 s). Adopting a tolerance window of .5 seconds, the percentage agreement between coders was 93 % (39 out of 42) for both opening and plateau duration and 95 % (40 out of 42) for closing duration.
?. [Study 2]
Hierarchical logistic regressionsPreliminary analyses were carried out, via hierarchical linear regression models, in order to identify specific features of yawns compared to other mouth openings in terms of their temporal dynamics, and to investigate the relations between different parameters for the two classes of episodes.
?. [Study 2]
Machine learning algorithmBased on findings from the exploratory analysis, the use of Support Vector Machine (SVM) classifiers was deemed appropriate to maximize the classification margin and minimize the risk of type 1 as well as type 2 errors in distinguishing yawns and non-yawn mouth openings. We used LibSVM with radial basis functions (RBF) kernel  in R, version 3.5.2 (package "e1071"), to build our models and to generate predictions for our test cases. C and γ were set at 100 and 1, respectively.Classification performance was compared via percent agreement and Cohen’s Kappa calculation for different SVM models by changing the subset of episodes used respectively for training and testing, in order to identify and cross-validate the optimized model. Since yawns are expected to share distinctive temporal dynamics, models including interaction effects only, as well as models also including main effects were fitted. In particular, based on the differences highlighted in the regression analyses, five different SVM models were implemented for classification.Three models only captured three-ways interaction effects, including respectively total duration, plateau duration and duration of the opening phase (Model A), total duration, plateau duration and opening/closing asymmetry (Model B) and total duration, plateau duration and opening/closing ratio (Model C). The remaining two models, with both main effects and interaction terms, included respectively total duration, plateau duration and opening/closing asymmetry (Model D) and all of the available variables, namely total duration, plateau duration, durations of both the opening and closing phases, opening/closing asymmetry and ratio (Model E).We compared the classification performance of different SVM methods via hold-out validation. On each iteration, two thirds (n = 87) of the sample (N = 130) were randomly assigned to training, while the remaining 43 episodes were used for testing. For each model, we computed overall agreement, Cohen’s Kappa, sensitivity (true positive rate) and specificity (true negative rate). The candidate model with the highest Cohen’s Kappa coefficient was deemed the best model for yawn identification.