DOI: 10.17603/ds2-3qzm-0464
title: Interviews with National Weather Service (NWS) Forecasters related to coastal fog and new artificial intelligence/machine learning (AI/ML) guidance: Interview materials for the “AI/ML” version
title: in Interviews with National Weather Service (NWS) Forecasters related to coastal fog and new artificial intelligence/machine learning (AI/ML) guidance

description: These PDFs are the interview materials used for the “AI/ML” version of the interviews. The protocol includes all questions asked, as well as notes for interviewers to guide the interview process. The information board is a PDF of the interactive slide deck used for the interviews. Detailed instructions on how the information board was used can be found in the slide deck. The only differences between this version and the “No AI/ML” version is that this protocol mentions AI/ML in the beginning of the interview, has the AI/ML related questions in the beginning of the interview, and the information board mentions AI/ML.
description: This project developed an interview guide and materials for conducting interviews with expert users to better understand how they assess and make use decisions about new AI/ML guidance. Forecasting and providing decision-support for coastal fog are challenging, in part because there are limited guidance options for predicting fog events that can have costly impacts for industries like shipping and aviation. At the same time, artificial intelligence (AI) is gaining momentum from researchers and developers as a tool for developing new guidance. For this study, we leveraged these advances to explore how National Weather Service (NWS) Forecasters perceive the use of new AI guidance for forecasting coastal fog. We also specifically examine which guidance features are important for how forecasters assess the trustworthiness of new AI guidance. To this aim, we conducted virtual, structured interviews with NWS forecasters from across the Eastern, Southern, Western, and Alaskan Regions. The interviews covered the forecasters’ approaches and challenges for forecasting coastal fog, perceptions of AI and its use in forecasting, and reactions to a prototype of new AI coastal fog guidance. During the interview, the forecasters went through a self-guided review of different sets of information about the development (spin-up information, AI technique, developers of the guidance, training, and inputs) and performance (explainable AI analyses, verification, a case study, and a comparison of the new product to existing guidance) of a prototype for new non-operational coastal fog AI guidance (Kamangir et al., 2021). The forecasters then assessed how the information influenced their perception of how trustworthy the guidance was and whether or not they would consider using it for forecasting. 

This project includes the interview protocols and accompanying information boards used for the interviews. As described below, there is one set of interview materials in which AI/ML are mentioned throughout and another set where AI/ML were only mentioned at the end of the interviews. We did this to better understand how the label “AI/ML” did or did not affect how interviewees responded to questions and reviewed the information board. We also leverage think aloud methods with the information board, the instructions for which are included in the interview protocols.