# Adapting Satellite Soundings for Operational Forecasting within the Hazardous Weather Testbed
## 4 Discussion  
Realistically, forecasters have limited time and will typically incorporate only the most reliable and relevant products into their decision making. For developers, the only way to accurately assess if a product is both reliable and relevant is by participating in a realistic operational environment, such as the NOAA testbeds. Below we provide suggestions for product evaluation in an operational environment:  
* **Develop a clear understanding of user needs, express realistic possibilities from the research community and be prepared to provide alternative products, tools or data delivery methods where possible.** As product developers, we often focus our efforts on meeting statistical requirements (e.g., producing data within a target error threshold) to increase product utilization in the research community. Through the HWT, we found that product latency was also of primary importance to forecasters. Improving product delivery into AWIPS and reducing the NUCAPS latency to 30 min drastically increased the utility for severe weather forecasting. This is example also illustrates how developers, who are aware of what changes are feasible, can help forecasters make achievable requests for improvements. Additionally, providing alternative visualization tools, which for NUCAPS include BUFKIT and SHARPpy, gives forecasters flexibility to view low-latency data in whatever way is most effective for their decision making.
* **Developers should document and communicate the limitations of their product(s) for the scenario being tested.** Each testbed focuses on a specific scenario; in the case of the HWT EWP, the focus is on issuing warnings on time scales of approximately 0-2 h. If forecasters repeatedly do not find utility in the product for certain situations, this should be documented in HWT training materials such as quick guides and communicated in pre-testbed training. As part of the operations to research process, developers should incorporate these limitations in general user guides for the product.
* **For mature products, foster more sophisticated analysis using training.** As a product matures, the assessment should focus on _how_ a product is used and with _what_ other datasets. From HWT case studies, we suggest developing a screen capture of a live demonstration of the product. In addition to scenarios where the product works well, product limitations should be clearly described in the training.
* **Surveys should contain a mixture of quantitative and qualitative questions.** Quantitative results permit easier comparison between HWT demonstrations and are faster for forecasters to fill out. We recommend repeating a core set of 2-3 quantitative questions every year. The remainingquestions can be in qualitative, written narrative form. Written responses to questions give forecasters the opportunity to describe their experiences in detail. Qualitative questions should be broad and include the _what_, _when_, _where_, and _how_ to encourage a more detailed response.
* **Products should meet quality requirements and be usable in AWIPS.** No product development should occur in isolation; if possible, have a developer be physically present in the room. Screen captures show developers how the fixed product looks, but watching forecasters interact with the AWIPS system also provides insight. For instance, menu design enables quick data access and comparison. Accessing gridded NUCAPS was a challenge the first year, but menus were improved following feedback. NUCAPS has 100 different pressure levels and not every combination is used by forecasters; we found that a well-curated list of options is better than an exhaustive one. Not all changes can be easily made, however. Updates to AWIPS visualizations often take time to develop or update; for NUCAPS, forecasters did not like how they could not open and compare multiple soundings in the display or see which profile they were observing. These changes require updates to the baseline code and will take longer to push into operations.