As in our previous study, the standard for a true positive was to obtain a system alert on a country-disease event on or before the silver standard alert. To allow for compatibility and comparison we kept the period for a qualifying system alert as up to 7 days prior to and including a qualifying ProMED report on the same topic. Other history period lengths might be more or less effective but were not the target of the investigation in this study. True positives were increased by 1 if there was any system alert that fell within the 7 day period. Multiple system alerts did not count twice. False positives were increased by 1 for each system alert that fell outside of the 7 day window. False negatives were counted as the number of qualifying alert periods when there were no system alerts. True negatives were counted as the number of days outside of any qualifying alert period when no system alert was given. In testing we tried to maximize F1 together with timeliness. Figure 1 shows the 16 event streams that we explored. The events chosen for this study were determined based on diversity of geographical and media coverage rather than random selection. The 16 event streams contain 2064 surveillance days with 153 events (7.4% of alerting days) (Note that system data from the study will be made publicly available online for re-use via the GENI database interface on BioCaster). Since we wanted to explore the hypothesis that linguistic coverage in multiple languages could strengthen detection rates and timeliness we compared English news coverage against all languages including English for each of 16 disease outbreaks. English was chosen as the baseline because of its overall geographic representativeness. An alternative and perhaps more realistic approach might have been to use the native language for each outbreak country as the baseline which we will consider in future investigations. Because cross-lingual events on the 13 languages were only available in our system from December 2009, the trial period was from January to May 2010.