# Awareness to Action: Student Knowledge of and Responses to an Early Alert System
## 2 Materials and Methods
### Materials  
Survey items were informed by conversations with faculty and student success staff; these conversations involved asking faculty and staff about their experiences discussing ASRs with students and their observations of student responses to ASRs. We also conducted a review of peer-reviewed literature on early alert systems. The author team then held a series of discussions to ideate on themes that emerged from these conversations and literature review to write the initial survey items. The survey items were reviewed for clarity and appropriateness by an undergraduate research assistant, resulting in minor changes to the wording of items. None of the survey items were taken from established measures of knowledge and experiences with early alert systems. Indeed, it appears that there is a current lack of validated measures regarding this topic, as prior survey studies of student experiences with early alert systems have also used measures developed specifically for those investigations (e.g., [34]).  
The survey included 34 items, excluding demographic information items. These items investigated five key areas regarding student engagement with the ASR system. Pertaining to Research Question 1, What do students know and believe about ASRS? were (1) knowledge of ASRs and (2) beliefs about ASRs. Pertaining to Research Question 2, How do students respond to receiving an ASR?, were (3) experiences receiving an ASR and (4) emotional and behavioral responses to receiving an ASR. Finally, pertaining to Research Question 3, What recommendations do students have to improve the system? was (5) recommendations for system improvement. The items were a mixture of open-text and closed-response (e.g., Likert scale) items. Eight of the 11 open-response items were coded by human raters (two additional items did not have sufficient data for analysis and one additional item which asked participants to list sources did not produce responses of qualitative depth needed for coding). Two raters inductively coded responses by separately  
\begin{table}
\begin{tabular}{l l l l} \hline
**Category** & **Response** & **n** & **\%** \\ \hline Hispanic or & & & \\ Latino/a/x & & & \\  & No & 263 & 73.9 \\  & Yes & 86 & 24.2 \\  & Prefer not to say & 3 & 0.8 \\ Year in Undergraduate & & & \\ Career & First & 256 & 71.9 \\  & Second & 75 & 21.1 \\  & Third & 14 & 3.9 \\  & Fourth & 5 & 1.4 \\  & Fifth & 2 & 0.6 \\ Transfer Student & & No & 325 & 92.3 \\  & Yes & 27 & 7.6 \\ Major & & Business & 179 & 50.3 \\  & Psychology & 32 & 9.0 \\  & Social and Behavioral Sciences & 29 & 8.1 \\  & Computer Science & 12 & 3.4 \\  & Biology & 11 & 3.1 \\  & Arts (e.g., Film) & 11 & 3.1 \\  & Health-Related (e.g., Nursing) & 10 & 2.8 \\ \hline \end{tabular} Four participants declined to provide any demographic information. Items asking participants for their gender, race, and major were open response.  
\end{table}
Table 1: _Cont._reading participant responses, developing a coding schema, and then creating a consensus coding schema through discussion. Then, 80-100% of the items were coded separately and checked for reliability. As responses were allowed to be included in multiple categories, 80% inter-rater agreement was set as the threshold for acceptable reliability, as done in prior research [42]. If the raters met that threshold, disagreements were resolved through discussion and then one rater coded the remaining responses. In certain cases, disagreement resulted in adjusting the coding schema to better reflect the data and raters repeating the process until the threshold was met. The final range of percentage agreement among raters for the items was 80.9-95.1%.