# Collaborative System Usability in Spaceflight Analog Environments through Remote Observations
## 3 Methods
### Processing Voice Recordings  
All crew members wore Philips Audio Recorders (DVT4010) throughout each mission day, providing a substantial bank of naturalistic audio data. These data serve as a valuable resource for characterizing scheduling-related collaboration as a crucial step in confirming Playbook's usability and inferring crew behavior. To post-process the data, we first identified timeframes in which TPM and SS took place and extracted the relevant audio recordings. We then combined Whisper, a speech recognition system developed by OpenAI [23], with manual transcription. We used Whisper to automate the initial transcription and corrected errors through manual review. Transcripts were formatted to represent different speakers and verbalizations on each line. While reviewing the transcripts, we synthesized our research question in a bottom-up fashion as recommended by [PERSON] [5]: what is the focus and frequency of different types of verbal collaboration during TPM and SS? After a comprehensive examination of the topics discussed by the crews, we identified four categories of interest (Table 1): collaboration regarding the timeline, collaboration regarding the task, collaboration regarding Playbook, or off-topic conversation. Each of these categories could provide insight into group behavior and attitudes; for example, collaboration regarding the nature of the self-scheduling task might reflect confusion about the instructions. Similarly, lots of off-topic chatter might represent low task burden or unengaging demands. Next, we appointed two independent raters to categorize the crews' conversations into one of the four categories. These raters were provided with the definitions of the collaboration categories from Table 1 and were instructed to tally the instances of each category for all TPM and SS. Given our primary interest in the topical focus of collaboration, raters were specifically instructed to closely monitor content shifts in blocks of verbalizations. To avoid potential confusion, we provided an example of a same-category content shift: if the discussion shifted from scheduling preferences for hygiene to preferences for when to do surveys, it would be counted as two instances of CRTimeline. To address the possibility of low interrater reliability, we predetermined that a third independent mediator would meet with the raters to resolve transcripts with significant count discrepancies. We calculated an intraclass coefficient (ICC) to assess interrater reliability and found it to be excellent (ICC = 0.91, 95% CI = [0.83, 0.95]) according to guidelines set forth by [PERSON] and [PERSON] [24], removing the need for any follow-up mediation. We then averaged the tallies from our independent raters to produce a single score for each category per transcript. Our primary variable of interest for analysis is the total counts for a discussion category during \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Label** & **Category** & **Description** & **Example** \\ \hline \multirow{2}{*}{CRTimeline} & \multirow{2}{*}{Collaboration Regarding Timeline} & Discussion on timeline & “I personally like the questionnaires \\  & & content and preferences & stacked together \& knock ’em all out.” \\ \hline \multirow{2}{*}{CRTask} & \multirow{2}{*}{Collaboration Regarding Task} & Discussion on the nature of & “We can talk about what our \\  & & the assigned task at hand & preferences are.” \\ \hline \multirow{2}{*}{CRPlaybook} & \multirow{2}{*}{Collaboration Regarding Playbook} & Discussion on how to use the & “But where do you see the tasks?” \\  & & tool or navigate the interface & “They’re right there on add-to-plan.” \\ \hline \multirow{2}{*}{OT} & \multirow{2}{*}{Off-Topic} & Jokes, tangents, or unrelated & “Have you watched the latest season of \\  & & topics & that show?” \\ \hline \hline \end{tabular}
\end{table} 
Table 1: The four observed conversation types and respective examples. a TPM or SS. We also considered the total duration of the TPM or SS as extracted from the length of our trimmed audio files.