## 3 Auditory-Perceptual Screening  
Most participants experienced difficulties in producing consistent breathy and pressed voice types, as reported by the on-site monitoring SLP. A mixed voice quality or other quality dimensions such as vocal tremor were sometimes perceived in the formal recording session. The recorded microphone data were perceptually assessed independently by five SLPs in order to validate each token's membership to a voice quality category. The SLPs were blinded to the purpose of the study and the information of voice samples. Since there is no validated perceptual tool to facilitate assessment of voice quality in two-second tokens, a customized website was developed to allow the SLPs to categorize the single vowel samples. Screenshots of the online system are shown in Figure 3. The SLPs were required to select a voice quality category (modal, breathy, pressed or none of the above) and a confidence level for the specific token's categorical membership. Distractor samples, with other voice quality dimensions, such as tremor, were added into the dataset to ensure use of the \"none of the above\" category and reduce any selection bias towards the three target types. The comment box was utilized to describe perceptually any samples categorized as \"none of above\". The SLPs were required to pass a training test (eight of 10 correctly categorized sample tokens) before proceeding to the formal rating.  
Figure 3: Screenshots of the online voice type assessment system.  
Inter-rater reliability analysis was used to evaluate the agreement among raters. Fleiss' Kappa method was used to assess the SLPs' reliabilities on categorical ratings. The parameters for the analysis were five raters, four categories (modal, breathy, pressed and none of the above) and 1595 samples. According to the \(\kappa\) value interpretation proposed by [PERSON] and [PERSON], the resulting (\(\kappa=0.7322\)) value indicated a substantial reliability among SLPs. Intra-rater reliability was evaluated by calculating the ICC (Intra-class Correlation Coefficient) using IBM SPSS Statistics 24 on 200 duplicated samples. The ICC was calculated using the two-way random model, as participants and raters were chosen randomly from a large population. Absolute agreement was targeted. The \(\kappa\) values for all five SLPs revealed \"almost perfect\" reliability (\(0.802,0.863,0.874,0.931,0.952\)) based on their rating results on the 200 duplicated samples, as interpreted in [33].  
The auditory-perceptual screening process was used to obtain ground-truth voice type labels for single vowel phonation and to screen out samples with mixed or inconsistent voice qualities [11]. In this study, \"pure samples\" were defined as any sample whereupon at least four out of five raters agreed on the voice type label with a minimum confidence level of 80%. The screening results showed that 952 out of 1395 samples were rated as \"pure\". The numbers of samples rated as \"pure\" voice types were 285, 395 and 273, for modal, breathy and pressed voice types, respectively. The number of breathy voice samples was approximately 40% greater than that of each other voice type sample, indicating that breathy voice was easier to mimic and perceive than modal or pressed voice. Through auditory-perceptual screening, the labels of the \"pure\" samples provided a ground truth dataset for developing an automatic voice type classification algorithm.