Evaluation of Syndromic Surveillance Data Streams in Animal Health

Objective To implement a systematic and uniform approach to evaluating data sources for syndromic surveillance within the United States Department of Agriculture (USDA) Animal and Plant Health Inspection Services (APHIS) Veterinary Services (VS) group. Introduction USDA-APHIS-VS utilizes several continuous data streams to increase our knowledge of animal health and provide situational awareness of emerging animal health issues. In addition, USDA- APHIS-VS often conducts pilot projects to see if regular data access and analysis are feasible, and if so, if the information generated is useful. Syndromic surveillance was developed for three goals: a syndromic monitoring system to identify new diseases, as an emerging disease early warning system, and to provide situational awareness of animal health status. Current efforts focus on monitoring diverse data, such as laboratory accessions or poison center calls, grouped into syndromic or other health indicator categories, and are not intended to identify specific pre-determined diseases or pathogens. It is essential to regularly evaluate and re-evaluate the effectiveness of our surveillance program. However, there are difficulties when using traditional surveillance evaluation methods, since the objectives and outcomes of monitoring novel data streams from pilot projects are not easily measurable. An additional challenge in the evaluation of these data streams is the identification of a method that can adapt to various context and inputs to make objective decisions. Until recently, assessment efforts have looked at the feasibility of regular analysis and reporting, but not at the utility of the information generated, nor the plausibility and sustainability of longer term or expanded efforts. Methods Methods for surveillance evaluation, syndromic surveillance evaluation, and specifically for animal health syndromic surveillance evaluation were researched via a literature review, exploration of methods used in-house on traditional surveillance systems, and through development over time of criteria that were seen as key to the development of functioning, sustainable systems focusing on animal health syndromic surveillance. Several methods were adapted to create an approach that could organize information in a logical manner, clarify objectives, and make qualitative value assessments in situations where the quantitative aspects of costs and benefits were not always straight forward. More than 25 articles were reviewed to determine the best method of evaluation. Results The RISKSUR Evaluation Support Tool (EVA) provided the majority of the methodology for the evaluations of our data sources. The EVA tool allows for an integrated approach for evaluation, and flexible methods to measure effectiveness and benefits of various data streams. The most useful and common factors found to evaluate pilot data sources of interest were how well the information generated by the data streams could provide early detection of animal health events, and how well and how often situational awareness information on animal health was generated. The EVA tool also helps identify and organize criteria that are used to assess the objectives, and assign value. Conclusions The regular evaluation of syndromic surveillance data streams in animal health is necessary to make best use of resources and maximize benefits of data stream use. It is also useful to conduct regular interim assessments on data streams in pilot phase to be certain key information for a final evaluation will be generated during the project. The RISKSUR EVA tool was found to be very flexible and useful for allowing estimates of value to be made, even when evaluating systems that do not have very specific, quantitatively measurable objectives. This tool provides flexibility in the selection of attributes for evaluation, making it particularly useful when examining pilot project data streams. In combination with additional review methodologies from the literature review, a systematic and uniform approach to data stream evaluation was identified for future use.