# Influenza forecasting for French regions combining EHR, web and climatic data sources with a machine learning ensemble approach

Canelle Poirier, Yulin Hswen, Guillaume Bouzillé, Marc Cuggia, Audrey Lavenu, John S Brownstein, Thomas Brewer, Mauricio Santillana

<figure><img src="image_1.png" /></figure>

## To cite this version:

Canelle Poirier, Yulin Hswen, Guillaume Bouzillé, Marc Cuggia, Audrey Lavenu, et al. Influenza forecasting for French regions combining EHR, web and climatic data sources with a machine learning ensemble approach. PLoS ONE, 2021, 16 (5), pp.e0250890. 10.1371/journal.pone.0250890. hal-03246277

HAL Id: hal-03246277

https://hal.science/hal-03246277

Submitted on 2 Jun 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
RESEARCH ARTICLE

# Influenza forecasting for French regions combining EHR, web and climatic data sources with a machine learning ensemble approach

Canelle Poirier<sup>1,2,9,10*</sup>, Yulin Hswen<sup>3,4</sup>, Guillaume Bouzillé<sup>1,2,5</sup>, Marc Cuggia<sup>1,2,5</sup>, Audrey Lavenu<sup>6,7,8</sup>, John S. Brownstein<sup>4,9</sup>, Thomas Brewer<sup>4</sup>, Mauricio Santillana<sup>9,10*</sup>

1 INSERM, U1099, Rennes, France, 2 Université de Rennes 1, LTSI, Rennes, France, 3 Department of Social and Behavioral Sciences, Harvard T.H. Chan School of Public Health, Boston, MA, United States of America, 4 Innovation Program, Boston Children's Hospital, Boston, MA, United States of America, 5 CHU Rennes, Centre de Données Cliniques, Rennes, France, 6 Université de Rennes 1, Faculté de médecine, Rennes, France, 7 INSERM CIC 1414, Université de Rennes 1, Rennes, France, 8 IRMAR, Institut de Recherche Mathématique de Rennes, Rennes, France, 9 Department of Pediatrics, Harvard Medical School, Boston, MA, United States of America, 10 Computational Health Informatics Program, Boston Children's Hospital, Boston, MA, United States of America

* canelle.poirier@outlook.fr (CP); msantill@g.harvard.edu (MS)

<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

OPEN ACCESS

**Citation:** Poirier C, Hswen Y, Bouzillé G, Cuggia M, Lavenu A, Brownstein JS, et al. (2021) Influenza forecasting for French regions combining EHR, web and climatic data sources with a machine learning ensemble approach. PLoS ONE 16(5): e0250890. https://doi.org/10.1371/journal.pone.0250890

**Editor:** Ka Chun Chong, The Chinese University of Hong Kong, HONG KONG

**Received:** March 10, 2021

**Accepted:** April 16, 2021

**Published:** May 19, 2021

**Copyright:** © 2021 Poirier et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

**Data Availability Statement:** All data cannot be shared publicly, in particular EHR data, due to the protection of patient data. For a data access, contact the local Ethics Committee of Rennes Academic Hospital (Brittany, France) email: comite.ethique@chu-rennes.fr.

**Funding:** CP, MC, GB and AL were partially funded by the French National Research Agency inside the Integrating and Sharing Health Data for Research Project (Grant No.452ANR-15-CE19-0024). MS

## Abstract

Effective and timely disease surveillance systems have the potential to help public health officials design interventions to mitigate the effects of disease outbreaks. Currently, health-care-based disease monitoring systems in France offer influenza activity information that lags real-time by one to three weeks. This temporal data gap introduces uncertainty that prevents public health officials from having a timely perspective on the population-level disease activity. Here, we present a machine-learning modeling approach that produces real-time estimates and short-term forecasts of influenza activity for the twelve continental regions of France by leveraging multiple disparate data sources that include, Google search activity, real-time and local weather information, flu-related Twitter micro-blogs, electronic health records data, and historical disease activity synchronicities across regions. Our results show that all data sources contribute to improving influenza surveillance and that machine-learning ensembles that combine all data sources lead to accurate and timely predictions.

## Introduction

Influenza is a major public health problem causing up to five million severe cases and 500,000 deaths per year worldwide [1–3]. In France alone, the epidemic of 2018–2019 caused 9,500 deaths. During epidemic peaks, large increases of visits to general practitioners and to emergency departments are observed and often lead to disruptions to healthcare delivery and thus increase the risk of undesirable outcomes in patients with influenza infections. To reduce the impact of influenza outbreaks in the population and to better design timely public health
and CP were partially funded by the National Institute of General Medical Sciences of the National Institutes of Health under Award Number R01GM130668. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

**Competing interests:** The authors have declared that no competing interests exist.

interventions, surveillance systems that produce accurate real-time and short-term forecasts of disease activity may prove to be instrumental.

In France, an important influenza monitoring system was implemented by the Sentinelles network in 1984 [4, 5]. This system centralizes information obtained from a group of volunteer (1,314 in 2018) general practitioners and (116 in 2018) pediatricians that each week report the proportion of patients with Influenza-Like-Illness (ILI, any acute respiratory infection with fever above 38°C, cough and onset within the last ten days) seeking medical attention. Data collection, processing, aggregation and distribution processes of this information, at the national and regional levels, introduce up to three weeks delays in the availability of flu activity information. This temporal data gap prevents public health officials from having the most up-to-date epidemiological information, and thus leads to the design of interventions that do not take into consideration recent changes in disease activity [2, 6]. For example, if estimates were available in real-time, information campaigns and vaccination prevention could be deployed earlier and could lead to greater impact. Additionally, healthcare facilities could be better prepared to respond to unexpected increases in the flux of high-risk patient during time periods of increased disease activity.

With the motivation to alleviate this time delay, mathematical modeling and machine learning approaches have been proposed to produce disease estimates in real time and ahead of healthcare-based surveillance systems in multiple nations around the world. Most of these studies have been designed and tested in developed nations, such as the United States and France, where information on disease outbreaks has been collected historically for decades [2]. Numerous research studies have been conducted on the use of traditional statistical methods, like temporal series or compartmental methods, as well as the inclusion of disparate data sources such as meteorological or demographic data to track flu activity, as discussed in Nsoesie et al. 2014 and Yang and Shaman 2014 [7, 8]. And in recent years, multiple more studies have emerged exploring the use of Internet-based data sources that capture aspects of human behavior and environmental factors to track the spread of diseases. With over 3.2 billion web users, data flows from the internet are huge and of all types. Some studies have used data from Google [2, 3, 9–13], Twitter [14–18] or Wikipedia [19–22] to monitor flu specifically.

One of the first and most prominent studies on the use of internet data for monitoring influenza epidemics is Google Flu Trends (GFT) [23, 24]. This web-based platform, created in 2009 and designed and deployed by Google, used the volume of selected Google search terms to estimate ILI activity in real time. GFT led to multiple prediction errors during the 2009 H1N1 Flu Pandemic (due to changes in people's search behaviour as a result of the exceptional nature of the pandemic) and later produced large overestimations during the 2012–2013 US flu season (due to the announcement of a pandemic that finally did not appear). These events show the lack of robustness of their algorithm and led to eventual discontinuation of this disease monitoring platform [25]. Since then, multiple research teams have proposed improved methodologies that are capable of extracting information more efficiently from flu-related Google searches and produce improved flu estimates [2, 3, 9–13]. Among these methods, the work of Shihao Yang et al. [2] explored a penalized regression methodology that combines historical flu activity with Google search activity dynamically, called ARGO, to better predict flu.

Additional data sources have been explored to monitor flu activity such as clinicians' searches, electronic health records (EHR), crowd-sourced flu monitoring apps [26–28]. Among these, electronic health records have been shown to track flu accurately and timely in the US and France [6, 29–31]. Specifically, in United States, Santillana et al. [6] showed that a model leveraging EHR data and a machine learning algorithms was capable to monitor flu activity in multiple spatial resolutions that included the regional level. In France, Poirier et al.
[29] similarly showed multiple statistical models that incorporate EHR and Internet-search data, can yield accurate ILI incidence rates in real time at the national level.

In early 2019, Fred S. Lu et al. [9] extended the ARGO methodology to accurately track flu activity in multiple states of the United States. In their approach, they included Google search data, EHRs and historical flu trends. They developed also a spatial network approach, called Net, to capture the synchronicity observed historically in flu activity between each states. Finally, by dynamically combining estimates from ARGO and Net, they showed that an ensemble approach, named ARGONet, led to improved results.

## Our contribution

In this study, we propose a forecasting platform that combines multiple data sources and statistical models to track flu activity in France at a spatial resolution that, to our knowledge, has not been explored before. Our forecasting platform produces accurate region-specific real-time and short-term flu activity forecasts for the twelve continental French regions. In our approach, we incorporated data sources such as Google data or Twitter microblogs, Electronic Health Record data, and weather that were not considered in the US study [9]. In addition, the EHR Data used here came directly from a clinical data warehouse rather than cloud-based billing and EHR company which required integrating structured and unstructured clinical data. Additionally, historical synchronicities across regions are captured with a Network model. A machine learning ensemble approach is proposed to improve predictions by dynamically combining estimates from these two distinct approaches. Near real-time estimates as well as one- and two-week ahead forecasts are presented.

## Materials and methods

All the data used for this research were fully anonymized. For the EHR data, the IRB ethics committee from the Rennes Academic Hospital approved this research (Approval number 16.69) and the data were fully anonymized before we accessed them. All other data sources are publicly available and appropriately anonymized. The data collected from Google and Twitter complied with the terms and conditions for each website.

### Data sources

**Sentinelles network data.** We obtained weekly ILI incidence rates (per 100,000 inhabitants) for the French regions (twelve) from the French Sentinelles network (websenti.u707.jus-sieu.fr/sentiweb). We retrieved these data in August 2018 from 05 January 2004 to 13 March 2017. We considered these data as the gold standard and as our task for our prediction models.

**Google data.** We obtained the frequency per week of the 100 most correlated internet queries (if correlation ≥ 0.60) by French users from Google Correlate (https://www.google.com/trends/correlate). Because our prediction period spans 05 January 2015 to 20 February 2017, we utilized the ILI signal for each French region, from January 2004 to December 2014 to obtain the most highly correlated search terms using the tool Google Correlate. In this way, we obtained different search terms for each individual region. The signals obtained correspond to queries performed by French users at the national level. We retrieved Google Correlate data in August 2018 for the period going from 05 January 2004 to 13 March 2017.

**Electronic health record data.** We retrieved EHR data from the clinical data warehouse (CDW) of Rennes University Hospital (France). This CDW, called eHOP, integrates structured (laboratory test results, prescriptions, ICD-10 diagnoses) and unstructured (discharge letter, pathology reports, operative reports) patients' data. It includes data from 1.2 million inpatients and outpatients and 45 million documents that correspond to 510 million
structured elements. eHOP consists of a powerful search engine system that can identify
patients with specific criteria by querying unstructured data with keywords, or structured data
with querying codes based on terminologies.

The first approach to obtain eHOP data connected with ILI was to perform different man-
ual queries to retrieve patients who had at least one document in their EHR that matched the
following search criteria: (1) Queries directly connected with flu or ILI with the keywords “flu”
or “ILI”; (2) Queries connected with flu symptoms with the keywords “fever”, “pyrexia”, “body
aches” or “muscular pain”; (3) Queries connected with flu drugs with the keyword “Tamiflu”;
(4) Queries with the ICD-10 terminology; (5) Queries connected with flu tests, positive or neg-
ative results.

In total, we performed 34 manual queries. For each query, the eHOP search engine
returned all documents containing the chosen keywords (often, several documents for one
patient and one stay). For query aggregation, we kept the oldest document for one patient and
one stay and then calculated, for each week, the number of stays with at least one document
mentioning the keyword contained in the query.

From the CDW eHOP, we built a database containing the time series constructed from the
structured data. In all, we have 1,335,347 time series. As Google Correlate, the Pearson correla-
tion between each signal of each region and the time series from the database was calculated.
In this way, for each region, the second approach was to retrieve the 100 most correlated sig-
nals to ILI signal. Because our test period is from 05 January 2015 to 20 February 2017, we cal-
culated the correlation between January 2004 and December 2014.

As a result, for each region, we obtained 134 variables from the CDW eHOP where there
are at least 34 variables common to all regions (manual queries). We retrieved retrospective
data in August 2018 for the period going from 03 January 2005 to 13 March 2017.

**Weather data.** We obtained region-specific weather data from the French climatological website Info Climat (https://www.infoclimat.fr). It has been shown in several studies that humidity is correlated with the spread of influenza [32]. In the absence of humidity data on the Climat website, we retrieved precipitation and temperature data. This choice was made knowing that both variables [33, 34], can be used as a proxy for humidity since they are directly related by the Clausius–Clapeyron relation [35]. We obtained temperatures and precipitations per day for the largest city of each region, and calculated the weekly mean for both temperature and precipitation. We retrieved climatic data in August 2018 for the time period going from 07 January 2008 to 13 March 2017.

**Twitter data.** Geotag tweets were extracted as the national scale for France from Boston Children's Hospital Geotweet dataset with the following keywords pertaining to influenza ("grippe", "grippé", "syndrome grippal", "fièvre", "toux", "congestion", "malade", "faiblesse", "courbatures", "tamiflu", "la crève"). From there, we aggregated tweets to get weekly counts. In this way, we obtained 11 variables from Twitter. We retrieved Twitter data in December 2018 for the period going from 30 December 2013 to 13 March 2017.

**Statistical models**

**The ARGO model.** The ARGO model is a regularized regression dynamically calibrated weekly using the LASSO method [36] to combine multiple external data sources with historical flu information. We performed the LASSO regression with the R package caret and the associated function fit with the method glmnet [37, 38]. We optimized the shrinkage parameter lambda via a ten-fold cross-validation. To test the stationarity and whiteness of residuals, we used Dickey Fuller's and Box-Pierce's tests available from the R packages tseries and stats [39]. The formulation of our model is:
* Real time estimates:

$$y_{it} = \sum_{j=1}^{52} \eta_j y_{it-j} + \sum_{k=1}^{10} \alpha_k x_{kit} + \sum_{l=1}^{10} \beta_l z_{lit} + \sum_{p=1}^{11} \gamma_p v_{pt} + \sum_{m=1}^{2} \delta_m w_{mit} + \epsilon_{it}$$

* One-week ahead forecast:

$$y_{it+1} = \sum_{j=1}^{52} \eta_j y_{it-j} + \sum_{k=1}^{10} \alpha_k x_{kit} + \sum_{l=1}^{10} \beta_l z_{lit} + \sum_{p=1}^{11} \gamma_p v_{pt} + \sum_{m=1}^{2} \delta_m w_{mit} + \epsilon_{it+1}$$

* Two-week ahead forecast:

$$y_{it+2} = \sum_{j=1}^{52} \eta_j y_{it-j} + \sum_{k=1}^{10} \alpha_k x_{kit} + \sum_{l=1}^{10} \beta_l z_{lit} + \sum_{p=1}^{11} \gamma_p v_{pt} + \sum_{m=1}^{2} \delta_m w_{mit} + \epsilon_{it+2}$$

where $y_{it}$ corresponding to the flu incidence rate at time $t$ for the region $i$, $\sum_{j=1}^{52} \eta_j y_{it-j}$ corresponding to the historical flu incidence rates for the region $i$, $\sum_{k=1}^{10} \alpha_k x_{kit}$ corresponding to the 10 most correlated variables from Google data for the region $i$, $\sum_{l=1}^{10} \beta_l z_{lit}$ corresponding to the 10 most correlated variables from hospital data for the region $i$, $\sum_{p=1}^{11} \gamma_p v_{pt}$ corresponding to Twitter data, $\sum_{m=1}^{2} \delta_m w_{mit}$ corresponding to climatic data for the region $i$, $\epsilon_t$ corresponding to residuals. We applied this model for each region. The model was dynamically recalibrated every week by incorporating all data available. In this way, the size of our training dataset increases every week. We obtained estimates from January 2011 to March 2017.

**The Net model.** The Net model is a LASSO model dynamically calibrated weekly and using the relationship between the regions to know how synchronicity could improve forecasts. Indeed, S1 Fig in S1 File (Heatmap of pairwise correlations between all regions) shows that the flu incidence rates of the different areas are correlated. For each region, we used historical data of all regions and estimates obtained with ARGO model for all regions expected the region to be predicted. The formulation of our model is:

* Real time estimates:

$$y_{it} = \sum_{l=1}^{2} \sum_{j=1}^{12} \alpha_j y_{jt-l} + \sum_{j=1}^{12} \beta_j \hat{y}_{jt} + \epsilon_{it}$$

* One-week ahead forecast:

$$y_{it+1} = \sum_{l=1}^{2} \sum_{j=1}^{12} \alpha_j y_{jt-l} + \sum_{j=1}^{12} \beta_j \hat{y}_{jt+1} + \epsilon_{it+1}$$
* Two-week ahead forecast:

$$y_{it+2} = \sum_{l=1}^{2} \sum_{j=1}^{12} \alpha_j y_{jt-l} + \sum_{j=1}^{12} \beta_j \hat{y}_{jt+2} + \epsilon_{it+2}$$

where $y_{it}$ corresponding to the flu incidence rate at time $t$ for the region $i$, $\sum_{l=1}^{2} \sum_{j=1}^{12} \alpha_j y_{jt-l}$ corresponding to two weeks of historical flu incidence rates for all regions, $\sum_{j=1}^{12} \beta_j \hat{y}_{jt}$ corresponding to ARGO predictions for all regions excepted the region $i$ to be predicted and $\epsilon_t$ corresponding to residuals. We applied this model for each region. We used a two years' training dataset. We obtained estimates from January 2013 to March 2017.

**The ARGONet model.** The ARGONet model is an ensemble approach combining the predictive power of ARGO and Net models. To combine the results of both models, we tested three methods:

* First, for a given week, we choose ARGO's estimate if it leads to the lowest mean prediction error in the previous K weeks (compared to the Net model's estimate). If this is not true, we choose Net's estimate. The values of K were inspired by Lu et al. [9] study and verified using cross-validation during the training time period.

* A second method consists of calculating the mean value of the estimates produced by the ARGO and Net models for a given week.

* In the final method, for a given week, ARGONet's estimate is built as a linear combination of estimates produced by ARGO and Net. The coefficients are dynamically calculated each week to best predict new ground truth data available each week.

**The autoregressive model.** To assess the importance of external data sources, we built an autoregressive model of order 52 (AR(52)). We used the LASSO regression with the previous 52 weeks of ILI incidence rates to predict the current week and the two weeks after.

* Real time estimates:

$$y_{it} = \sum_{j=1}^{52} \alpha_j y_{it-j} + \epsilon_{it}$$

* One-week ahead forecast:

$$y_{it+1} = \sum_{j=1}^{52} \alpha_j y_{it-j} + \epsilon_{it+1}$$

* Two-week ahead forecast:

$$y_{it+2} = \sum_{j=1}^{52} \alpha_j y_{it-j} + \epsilon_{it+2}$$
where $y_{it}$ corresponding to the flu incidence rate at time $t$ for the region $i$, $\sum_{j=1}^{52} \alpha_j y_{it-j}$ corresponding to the previous 52 weeks, $\epsilon_t$ corresponding to residuals. We applied this model for each region. We used a six years' training dataset. The model was dynamically recalibrated every week.

**The baseline model.** Finally, we included a baseline model that simply predicts that the number of new flu cases in a week will be exactly the number of cases observed in the past week.

## Evaluation

Our test period consists on 115 weeks starting from January 2015 to March 2017.

**Metrics.** To assess the performance of the models, we compared estimates to the official incidence rates from the Sentinelles network by calculating two metrics: the root mean squared error (RMSE) and the Pearson correlation coefficient (PCC).

$$ 
\begin{aligned}
\text{RMSE} &= \sqrt{\frac{1}{n} \sum_{i=1}^{n} (\hat{y}_i - y_i)^2} \\
\text{PCC} &= \frac{\sum_{i=1}^{n} (y_i - \bar{y})(\hat{y}_i - \bar{y})}{\sqrt{\sum_{i=1}^{n} (y_i - \bar{y})^2 \sum_{i=1}^{n} (\hat{y}_i - \bar{y})^2}}
\end{aligned}
$$

where $\hat{y}_i$ is the predicted value for the week $i$, $\bar{y}_i$ is the mean of predicted values, $y_i$ the real value for the week $i$, $\bar{y}_i$ is the mean of real values.

We also estimated the relative efficiency of ARGONet model compared to the autoregressive model AR(52) with 95% confidence interval (CI) by using a Bootstrap method. A relative efficiency, calculated by $\frac{RMSE_{ARGONET}}{RMSE_{ARGONET}}$ higher than one, suggests increased predictive power of ARGONet compared to the autoregressive model AR(52). The CI and relative efficiency have been computed based on 100 Bootstrap samples of length 52. The 52 weeks were randomly selected from estimates from January 2015 to February 2017.

**Comparisons.** First, we assessed the importance of adding external data sources by comparing:

* RMSE and PCC of the AR(52) model and the ARGO model including historical data plus the ten most correlated variables from hospital data and Google data. The individual contribution of hospital data and Google data has already been shown in a previous study [29]. But, we added in appendices, two comparisons: A comparison with the ten most correlated variables from hospital data and a comparison with the ten most correlated variables from Google data.

* RMSE and PCC of the AR(52) model and the ARGO model including historical data plus climatic data.

* RMSE and PCC of the AR(52) model and the ARGO model including historical data plus Twitter data.

Second, we compared the baseline model, AR(52) model, ARGO model (including all the data sources), Net model and ARGONet model.

## Results

### Evaluation of data sources as predictors

In order to assess the predictive value of each and all external data source, we compared ARGO models that incrementally included external data sources with an autoregressive
<table><caption>Table 1. Real time estimate—RMSE and PCC for ARGO models including only historical data (AR(52)) and the ten most correlated variables from hospital and Google data, for the period starting from January 2015 to March 2017.</caption><thead><tr><th></th><th>Auv.</th><th>Bour.</th><th>Bre.</th><th>Cen.</th><th>Gd Est</th><th>Ht Fra.</th><th>Ile Fra.</th><th>Norm.</th><th>Aqui.</th><th>Occi.</th><th>Loi.</th><th>Pro.</th></tr></thead><tbody><tr><th colspan="13" style="text-align: left;">RMSE</th></tr><tr><td>AR(52)</td><td><strong>59.02</strong></td><td>36.24</td><td>65.41</td><td>77.00</td><td><strong>50.05</strong></td><td><strong>67.52</strong></td><td>61.34</td><td>75.29</td><td>71.55</td><td>58.48</td><td>102.38</td><td>82.49</td></tr><tr><td>ARGO</td><td>65.24</td><td><strong>34.99</strong></td><td><strong>56.83</strong></td><td><strong>64.52</strong></td><td>52.39</td><td>82.30</td><td><strong>56.67</strong></td><td><strong>64.17</strong></td><td><strong>65.87</strong></td><td><strong>50.19</strong></td><td><strong>97.84</strong></td><td><strong>67.71</strong></td></tr><tr><th colspan="13" style="text-align: left;">PCC</th></tr><tr><td>AR(52)</td><td>0.958</td><td>0.898</td><td>0.916</td><td>0.919</td><td>0.952</td><td>0.915</td><td>0.935</td><td>0.879</td><td>0.919</td><td>0.946</td><td>0.815</td><td>0.929</td></tr><tr><td>ARGO</td><td><strong>0.971</strong></td><td><strong>0.928</strong></td><td><strong>0.950</strong></td><td><strong>0.960</strong></td><td><strong>0.966</strong></td><td><strong>0.935</strong></td><td><strong>0.949</strong></td><td><strong>0.912</strong></td><td><strong>0.939</strong></td><td><strong>0.980</strong></td><td><strong>0.846</strong></td><td><strong>0.963</strong></td></tr></tbody></table>

https://doi.org/10.1371/journal.pone.0250890.t001

model, AR(52), model that only uses historical information as input. As shown in the next sections, we found that all external data sources improve flu estimates, specially EHR Data and Google Data.

**EHR data and Google data.** Our first modeling experiment involved comparing ARGO models that use Google search and EHR data simultaneously with the AR(52) in all French regions. A detailed analysis on the individual contribution of Google data and EHR data into predictions, separately, is provided for completeness in the supplementary materials. Our findings suggest that each of these data sources individually improves predictions in all time-horizons. This is consistent with the findings of a previous study conducted at the national-level and the French region of Brittany [29], where both Google and EHR information were found meaningful, but EHR data was shown to possess a stronger predictive power.

The joint contribution of both EHR and Google data on predictions is presented below. In real time (Table 1), in terms of correlation, estimates produced using EHR data and Google data improve the accuracy for all the regions and for 9 regions in terms of error metrics. The combination of both sources lead to correlation improvements of up to 5% for the region BrTAGNE and decreases in error of up to 20% for the region Provence-Alpes-Côte d'Azur.

For one-week ahead estimate (Table 2), estimates obtained with EHR and Google data are more accurate or comparable for eleven of the twelve regions in terms of correlation and nive of the twelve region in terms of error metrics. The combination of both sources lead to correlation improvements of up to 15% for the region Bourgogne and decreases in error of up to 25% for the region Provence-Alpes-Côte d'Azur.

For two-week ahead predictions (Table 3), estimates obtained with EHR and Google data are more accurate for all the regions in terms of correlation and for eleven of the twelve regions in terms of error metrics. The combination of both sources lead to correlation improvements of up to 30% for the region Centre and decreases in error of up to 25% for the region Provence-Alpes-Côte d'Azur.

<table><caption>Table 2. One-week ahead estimate—RMSE and PCC for ARGO models including only historical data (AR(52)) and the ten most correlated variables from hospital and Google data, for the period starting from January 2015 to March 2017.</caption><thead><tr><th></th><th>Auv.</th><th>Bour.</th><th>Bre.</th><th>Cen.</th><th>Gd Est</th><th>Ht Fra.</th><th>Ile Fra.</th><th>Norm.</th><th>Aqui.</th><th>Occi.</th><th>Loi.</th><th>Pro.</th></tr></thead><tbody><tr><th colspan="13" style="text-align: left;">RMSE</th></tr><tr><td>AR(52)</td><td><strong>105.36</strong></td><td>59.41</td><td>99.54</td><td>122.54</td><td><strong>83.21</strong></td><td>103.89</td><td>101.55</td><td>110.63</td><td>117.00</td><td>102.94</td><td><strong>139.49</strong></td><td>134.43</td></tr><tr><td>ARGO</td><td>119.89</td><td><strong>57.54</strong></td><td><strong>84.41</strong></td><td><strong>97.50</strong></td><td>91.32</td><td><strong>98.36</strong></td><td><strong>96.13</strong></td><td><strong>91.69</strong></td><td><strong>107.28</strong></td><td><strong>93.53</strong></td><td>153.10</td><td><strong>101.72</strong></td></tr><tr><th colspan="13" style="text-align: left;">PCC</th></tr><tr><td>AR(52)</td><td>0.868</td><td>0.692</td><td>0.782</td><td>0.779</td><td>0.861</td><td>0.782</td><td>0.813</td><td>0.707</td><td>0.758</td><td>0.860</td><td><strong>0.645</strong></td><td>0.804</td></tr><tr><td>ARGO</td><td><strong>0.896</strong></td><td><strong>0.827</strong></td><td><strong>0.881</strong></td><td><strong>0.903</strong></td><td><strong>0.909</strong></td><td><strong>0.875</strong></td><td><strong>0.847</strong></td><td><strong>0.813</strong></td><td><strong>0.834</strong></td><td><strong>0.923</strong></td><td>0.600</td><td><strong>0.914</strong></td></tr></tbody></table>

https://doi.org/10.1371/journal.pone.0250890.t002
<table><caption>Table 3. Two-week ahead estimate—RMSE and PCC for ARGO models including only historical data (AR(52)) and the ten most correlated variables from hospital and Google data, for the period starting from January 2015 to March 2017.</caption><thead><tr><th></th><th>Auv.</th><th>Bour.</th><th>Bre.</th><th>Cen.</th><th>Gd Est</th><th>Ht Fra.</th><th>Ile Fra.</th><th>Norm.</th><th>Aqui.</th><th>Occi.</th><th>Loi.</th><th>Pro.</th></tr></thead><tbody><tr><th colspan="13">RMSE</th></tr><tr><td>AR(52)</td><td><strong>144.22</strong></td><td>69.89</td><td>116.89</td><td>155.47</td><td>111.85</td><td>124.06</td><td>129.82</td><td>132.51</td><td>145.67</td><td>128.13</td><td>157.68</td><td>164.00</td></tr><tr><td>ARGO</td><td>147.70</td><td><strong>69.30</strong></td><td><strong>114.15</strong></td><td><strong>116.30</strong></td><td><strong>104.74</strong></td><td><strong>123.84</strong></td><td><strong>112.41</strong></td><td><strong>109.75</strong></td><td><strong>135.96</strong></td><td><strong>112.28</strong></td><td><strong>137.85</strong></td><td><strong>121.01</strong></td></tr><tr><th colspan="13">PCC</th></tr><tr><td>AR(52)</td><td>0.731</td><td>0.522</td><td>0.665</td><td>0.609</td><td>0.733</td><td>0.658</td><td>0.688</td><td>0.539</td><td>0.610</td><td>0.731</td><td>0.528</td><td>0.699</td></tr><tr><td>ARGO</td><td><strong>0.820</strong></td><td><strong>0.708</strong></td><td><strong>0.779</strong></td><td><strong>0.832</strong></td><td><strong>0.834</strong></td><td><strong>0.823</strong></td><td><strong>0.775</strong></td><td><strong>0.714</strong></td><td><strong>0.720</strong></td><td><strong>0.849</strong></td><td><strong>0.664</strong></td><td><strong>0.858</strong></td></tr></tbody></table>

https://doi.org/10.1371/journal.pone.0250890.t003

<table><caption>Table 4. Real time estimate—RMSE and PCC for ARGO models including only historical data (AR(52)) and only climatic data, for the period starting from January 2015 to March 2017.</caption><thead><tr><th></th><th>Auv.</th><th>Bour.</th><th>Bre.</th><th>Cen.</th><th>Gd Est</th><th>Ht Fra.</th><th>Ile Fra.</th><th>Norm.</th><th>Aqui.</th><th>Occi.</th><th>Loi.</th><th>Pro.</th></tr></thead><tbody><tr><th colspan="13">RMSE</th></tr><tr><td>AR(52)</td><td>59.02</td><td>36.24</td><td>65.41</td><td>77.00</td><td>50.05</td><td>67.52</td><td>61.34</td><td>75.29</td><td>71.55</td><td>58.48</td><td>102.38</td><td>82.49</td></tr><tr><td>ARGO</td><td><strong>57.89</strong></td><td><strong>35.65</strong></td><td><strong>62.53</strong></td><td><strong>76.56</strong></td><td><strong>48.76</strong></td><td><strong>64.21</strong></td><td><strong>60.13</strong></td><td><strong>73.30</strong></td><td><strong>69.29</strong></td><td><strong>58.41</strong></td><td><strong>102.37</strong></td><td><strong>80.72</strong></td></tr><tr><th colspan="13">PCC</th></tr><tr><td>AR(52)</td><td>0.958</td><td>0.898</td><td>0.916</td><td>0.919</td><td>0.952</td><td>0.915</td><td>0.935</td><td>0.879</td><td>0.919</td><td>0.946</td><td>0.815</td><td>0.929</td></tr><tr><td>ARGO</td><td><strong>0.962</strong></td><td><strong>0.901</strong></td><td><strong>0.922</strong></td><td><strong>0.919</strong></td><td><strong>0.954</strong></td><td><strong>0.921</strong></td><td><strong>0.939</strong></td><td><strong>0.884</strong></td><td><strong>0.925</strong></td><td><strong>0.951</strong></td><td><strong>0.828</strong></td><td><strong>0.934</strong></td></tr></tbody></table>

https://doi.org/10.1371/journal.pone.0250890.t004

**Climatic data.** When combining climatic data with historical activity via ARGO was shown to consistently improve prediction results across all regions (Table 4). However, this improvement is lower than the one observed with EHR and Google data. Indeed, climatic data lead to correlation improvements of 2% for the region Pays de la Loire and decreases in error of 5% for the region Hauts-de-France.

For one-week ahead estimate (Table 5), in term of correlation and error, results obtained with Climatic data are better or comparable for all regions. Climatic data lead to correlation improvements of up to 5% for the region Bourgogne-Franche-Comté and decreases in error of up to 7% for the region Hauts-de-France.

For two-week ahead estimate (Table 6), results obtained with Climatic data are better for all the regions. Climatic data lead to correlation improvements of up to 25% and decreases in error of up to 11% for the region Bourgogne-Franche-Comté.

**Twitter data.** Overall, we found that national-level flu-related Twitter data improves prediction results for all regions.

<table><caption>Table 5. One-week ahead estimate—RMSE and PCC for ARGO models including only historical data (AR(52)) and only climatic data, for the period starting from January 2015 to March 2017.</caption><thead><tr><th></th><th>Auv.</th><th>Bour.</th><th>Bre.</th><th>Cen.</th><th>Gd Est</th><th>Ht Fra.</th><th>Ile Fra.</th><th>Norm.</th><th>Aqui.</th><th>Occi.</th><th>Loi.</th><th>Pro.</th></tr></thead><tbody><tr><th colspan="13">RMSE</th></tr><tr><td>AR(52)</td><td>105.36</td><td>59.41</td><td>99.54</td><td>122.54</td><td>83.21</td><td>103.89</td><td>101.55</td><td>110.63</td><td>117.00</td><td>102.94</td><td>139.49</td><td>134.43</td></tr><tr><td>ARGO</td><td><strong>105.11</strong></td><td><strong>56.37</strong></td><td><strong>94.10</strong></td><td><strong>120.20</strong></td><td><strong>81.48</strong></td><td><strong>96.76</strong></td><td><strong>99.72</strong></td><td><strong>106.13</strong></td><td><strong>112.20</strong></td><td><strong>99.30</strong></td><td><strong>137.04</strong></td><td><strong>129.76</strong></td></tr><tr><th colspan="13">PCC</th></tr><tr><td>AR(52)</td><td><strong>0.868</strong></td><td>0.692</td><td>0.782</td><td>0.779</td><td>0.861</td><td>0.782</td><td>0.813</td><td>0.707</td><td>0.758</td><td><strong>0.860</strong></td><td>0.645</td><td>0.804</td></tr><tr><td>ARGO</td><td>0.867</td><td><strong>0.726</strong></td><td><strong>0.809</strong></td><td><strong>0.788</strong></td><td><strong>0.867</strong></td><td><strong>0.808</strong></td><td><strong>0.825</strong></td><td><strong>0.740</strong></td><td><strong>0.787</strong></td><td>0.852</td><td><strong>0.669</strong></td><td><strong>0.820</strong></td></tr></tbody></table>

https://doi.org/10.1371/journal.pone.0250890.t005
<table><caption>Table 6. Two-week ahead estimates—RMSE and PCC for ARGO models including only historical data (AR(52)) and only climatic data, for the period starting from January 2015 to March 2017.</caption><thead><tr><th></th><th>Auv.</th><th>Bour.</th><th>Bre.</th><th>Cen.</th><th>Gd Est</th><th>Ht Fra.</th><th>Ile Fra.</th><th>Norm.</th><th>Aqui.</th><th>Occi.</th><th>Loi.</th><th>Pro.</th></tr></thead><tbody><tr><td><strong>RMSE</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>AR(52)</td><td>144.22</td><td>69.89</td><td>116.89</td><td>155.47</td><td>111.85</td><td>124.06</td><td>129.82</td><td>132.51</td><td>145.67</td><td>128.13</td><td>157.68</td><td>164.00</td></tr><tr><td>ARGO</td><td><strong>141.10</strong></td><td><strong>62.27</strong></td><td><strong>112.50</strong></td><td><strong>148.99</strong></td><td><strong>106.08</strong></td><td><strong>115.28</strong></td><td><strong>127.36</strong></td><td><strong>125.57</strong></td><td><strong>138.09</strong></td><td><strong>123.71</strong></td><td><strong>153.15</strong></td><td><strong>157.00</strong></td></tr><tr><td><strong>PCC</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>AR(52)</td><td>0.731</td><td>0.522</td><td>0.665</td><td>0.609</td><td>0.733</td><td>0.658</td><td>0.688</td><td>0.539</td><td>0.610</td><td>0.731</td><td>0.528</td><td>0.699</td></tr><tr><td>ARGO</td><td><strong>0.743</strong></td><td><strong>0.651</strong></td><td><strong>0.709</strong></td><td><strong>0.647</strong></td><td><strong>0.759</strong></td><td><strong>0.719</strong></td><td><strong>0.701</strong></td><td><strong>0.610</strong></td><td><strong>0.655</strong></td><td><strong>0.758</strong></td><td><strong>0.590</strong></td><td><strong>0.726</strong></td></tr></tbody></table>

https://doi.org/10.1371/journal.pone.0250890.t006

In real time (Table 7), we see that between Twitter data and AR(52) results are comparable. Twitter data lead to correlation improvements of 2% for the regions Occitanie and Pays de la Loire and decreases in error of 5% for the region Centre.

For one-week ahead estimate (Table 8), estimates obtained with Twitter data are more accurate for all the regions in terms of correlation and for ten regions in terms of error metrics. Twitter data lead to correlation improvements of 10% for the region Pays de la Loire and decreases in error of 7% for the region Bretagne.

For two-week ahead estimate (Table 9), results obtained with Twitter data are more accurate for all the regions in terms of correlation and for nine regions in terms of error metrics. Twitter data lead to correlation improvements of 20% and decreases in error of 6% for the region Bourgogne-Franche-Comté.

## Evaluation of statistical models

Here, we compare the predictive performance of five different modeling approaches the baseline model, AR(52), ARGO, Net, and ARGONet for three time horizons: real-time, one-week

<table><caption>Table 7. Real time estimate—RMSE and PCC for ARGO models including only historical data (AR(52)) and only Twitter data, for the period starting from January 2015 to March 2017.</caption><thead><tr><th></th><th>Auv.</th><th>Bour.</th><th>Bre.</th><th>Cen.</th><th>Gd Est</th><th>Ht Fra.</th><th>Ile Fra.</th><th>Norm.</th><th>Aqui.</th><th>Occi.</th><th>Loi.</th><th>Pro.</th></tr></thead><tbody><tr><td><strong>RMSE</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>AR(52)</td><td>59.02</td><td><strong>36.24</strong></td><td>65.41</td><td>77.00</td><td>50.05</td><td>67.52</td><td>61.34</td><td><strong>75.29</strong></td><td>71.55</td><td>58.48</td><td><strong>102.38</strong></td><td>82.49</td></tr><tr><td>ARGO</td><td><strong>58.86</strong></td><td>36.93</td><td><strong>64.69</strong></td><td><strong>72.79</strong></td><td><strong>49.98</strong></td><td><strong>66.45</strong></td><td><strong>59.86</strong></td><td>76.72</td><td>71.74</td><td><strong>53.66</strong></td><td>102.51</td><td><strong>82.28</strong></td></tr><tr><td><strong>PCC</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>AR(52)</td><td>0.958</td><td><strong>0.898</strong></td><td><strong>0.916</strong></td><td>0.919</td><td>0.952</td><td><strong>0.915</strong></td><td>0.935</td><td><strong>0.879</strong></td><td>0.919</td><td>0.946</td><td>0.815</td><td>0.929</td></tr><tr><td>ARGO</td><td><strong>0.960</strong></td><td>0.893</td><td>0.914</td><td><strong>0.931</strong></td><td><strong>0.954</strong></td><td>0.914</td><td><strong>0.945</strong></td><td>0.871</td><td><strong>0.919</strong></td><td><strong>0.961</strong></td><td><strong>0.830</strong></td><td><strong>0.933</strong></td></tr></tbody></table>

https://doi.org/10.1371/journal.pone.0250890.t007

<table><caption>Table 8. One-week ahead estimate—RMSE and PCC for ARGO models including only historical data (AR(52)) and only Twitter data, for the period starting from January 2015 to March 2017.</caption><thead><tr><th></th><th>Auv.</th><th>Bour.</th><th>Bre.</th><th>Cen.</th><th>Gd Est</th><th>Ht Fra.</th><th>Ile Fra.</th><th>Norm.</th><th>Aqui.</th><th>Occi.</th><th>Loi.</th><th>Pro.</th></tr></thead><tbody><tr><td><strong>RMSE</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>AR(52)</td><td>105.36</td><td>59.41</td><td>99.54</td><td>122.54</td><td><strong>83.21</strong></td><td>103.89</td><td><strong>101.55</strong></td><td>110.63</td><td>117.00</td><td>102.94</td><td>139.49</td><td>134.43</td></tr><tr><td>ARGO</td><td><strong>102.71</strong></td><td><strong>57.98</strong></td><td><strong>92.92</strong></td><td><strong>114.90</strong></td><td>87.64</td><td><strong>101.23</strong></td><td>103.45</td><td><strong>107.90</strong></td><td><strong>112.16</strong></td><td><strong>96.11</strong></td><td><strong>131.67</strong></td><td><strong>128.60</strong></td></tr><tr><td><strong>PCC</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>AR(52)</td><td>0.868</td><td>0.692</td><td>0.782</td><td>0.779</td><td>0.861</td><td>0.782</td><td>0.813</td><td>0.707</td><td>0.758</td><td>0.860</td><td>0.645</td><td>0.804</td></tr><tr><td>ARGO</td><td><strong>0.881</strong></td><td><strong>0.712</strong></td><td><strong>0.821</strong></td><td><strong>0.833</strong></td><td><strong>0.873</strong></td><td><strong>0.787</strong></td><td><strong>0.846</strong></td><td><strong>0.723</strong></td><td><strong>0.811</strong></td><td><strong>0.883</strong></td><td><strong>0.703</strong></td><td><strong>0.834</strong></td></tr></tbody></table>

https://doi.org/10.1371/journal.pone.0250890.t008
<table><caption>Table 9. Two-week ahead estimate—RMSE and PCC for ARGO models including only historical data (AR(52)) and only Twitter data, for the period starting from January 2015 to March 2017.</caption><thead><tr><th></th><th>Auv.</th><th>Bour.</th><th>Bre.</th><th>Cen.</th><th>Gd Est</th><th>Ht Fra.</th><th>Ile Fra.</th><th>Norm.</th><th>Aqui.</th><th>Occi.</th><th>Loi.</th><th>Pro.</th></tr></thead><tbody><tr><td><strong>RMSE</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>AR(52)</td><td><strong>144.22</strong></td><td>69.89</td><td>116.89</td><td>155.47</td><td><strong>111.85</strong></td><td>124.06</td><td><strong>129.82</strong></td><td>132.51</td><td>145.67</td><td>128.13</td><td>157.68</td><td>164.00</td></tr><tr><td>ARGO</td><td>144.56</td><td><strong>65.52</strong></td><td><strong>114.06</strong></td><td><strong>150.85</strong></td><td>115.04</td><td><strong>118.94</strong></td><td>143.15</td><td><strong>130.71</strong></td><td><strong>143.62</strong></td><td><strong>124.49</strong></td><td><strong>153.54</strong></td><td><strong>160.30</strong></td></tr><tr><td><strong>PCC</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>AR(52)</td><td>0.731</td><td>0.522</td><td>0.665</td><td>0.609</td><td>0.733</td><td>0.658</td><td>0.688</td><td>0.539</td><td>0.610</td><td>0.731</td><td>0.528</td><td>0.699</td></tr><tr><td>ARGO</td><td><strong>0.746</strong></td><td><strong>0.615</strong></td><td><strong>0.753</strong></td><td><strong>0.696</strong></td><td><strong>0.765</strong></td><td><strong>0.685</strong></td><td><strong>0.712</strong></td><td><strong>0.559</strong></td><td><strong>0.685</strong></td><td><strong>0.790</strong></td><td><strong>0.563</strong></td><td><strong>0.744</strong></td></tr></tbody></table>

https://doi.org/10.1371/journal.pone.0250890.t009

and two-week ahead estimates. Fig 1 displays the ranking of each model for each time horizon of prediction across regions during the out-of-sample evaluation time period (January 2015 to March 2017). If a model is ranked in the first position, it means that it led to the best prediction results in terms of error (RMSE) and in most cases this was also the case in terms of correlation. As displayed in Fig 1, ARGONet is the most accurate model, ranking either first or second in all regions in both, real-time estimates and the one-week prediction horizon, and ranking first for the two-week prediction horizon.

**Real-time estimate.** Table 10 summarizes results obtained with the baseline model, AR (52), ARGO, Net and ARGONet models for the period starting from January 2015 to March 2017, for the twelve regions. Over this time period, the 90% confidence interval (CI) of the best correlation is [0.915;0.971] with a median value equal to 0.950. The 90% CI of the blackbest RMSE is [42.97;63.08] with a median value equal to 56.08 These values are mostly obtained with ARGONet model which implies a reduction of the error from 15% to 41% compared to the baseline. The lowest values in term of correlation and the highest values in term of error are obtained with AR(52) and the baseline models. Because public health officials care more about the flu season starting from week 40 to week 15 in the next year, we presented a similar S7 Table in S1 File but calculating correlation and errors only during this time period. We obtained comparable results, with ARGONet model giving the lowest errors and the highest correlations.

Fig 2 is a visualization of results obtained in Table 10. It confirms, region by region, that the best PCC and RMSE are mostly obtained with ARGONet. Nevertheless, for real-time estimate, ARGO shows good performance. For seven regions ARGO is the model with the second lowest

<figure><img src="image_5.png" /><figcaption>Fig 1. Ranks obtained by each model over the 12 French regions for PCC and RMSE.</figcaption></figure>

https://doi.org/10.1371/journal.pone.0250890.g001
<table><caption>Table 10. PCC and RMSE for real-time estimate for all french regions for the period starting from January 2015 to March 2017.</caption><thead><tr><th></th><th>Auv.</th><th>Bour.</th><th>Bre.</th><th>Cen.</th><th>Gd Est</th><th>Ht Fra.</th><th>Ile Fra.</th><th>Norm.</th><th>Aqui.</th><th>Occi.</th><th>Loi.</th><th>Pro.</th></tr></thead><tbody><tr><td colspan="14"><strong>RMSE</strong></td></tr><tr><td>Baseline</td><td>75.89</td><td>39.77</td><td>65.49</td><td>88.96</td><td>61.56</td><td>70.67</td><td>66.72</td><td>77.10</td><td>74.25</td><td>73.99</td><td>106.72</td><td>95.31</td></tr><tr><td>AR(52)</td><td><strong>59.02</strong></td><td>36.24</td><td>65.41</td><td>77.00</td><td>50.05</td><td>67.52</td><td>61.34</td><td>75.29</td><td>71.55</td><td>58.48</td><td>102.38</td><td>82.49</td></tr><tr><td>Argo</td><td>65.14</td><td>33.88</td><td><strong>56.39</strong></td><td>63.74</td><td>52.30</td><td>79.54</td><td><strong>57.10</strong></td><td><strong>63.08</strong></td><td>65.04</td><td>49.51</td><td>96.04</td><td>64.89</td></tr><tr><td>Net</td><td>64.46</td><td>37.65</td><td>70.84</td><td>65.35</td><td>45.90</td><td><strong>58.76</strong></td><td>68.22</td><td>76.19</td><td>64.45</td><td>57.15</td><td>105.28</td><td>66.06</td></tr><tr><td>K = 1</td><td>60.06</td><td>33.34</td><td>62.10</td><td>61.16</td><td>49.65</td><td>62.58</td><td>63.21</td><td>66.32</td><td>56.31</td><td>50.74</td><td>100.28</td><td>62.86</td></tr><tr><td>K = 2</td><td>59.58</td><td>32.97</td><td>61.33</td><td>66.30</td><td>46.22</td><td>65.27</td><td>62.50</td><td><strong>65.08</strong></td><td>56.87</td><td>49.46</td><td>99.33</td><td>61.23</td></tr><tr><td>K = 3</td><td>59.74</td><td>33.41</td><td>60.25</td><td>66.77</td><td><strong>43.79</strong></td><td>63.96</td><td>63.36</td><td>72.47</td><td><strong>60.46</strong></td><td>49.92</td><td><strong>90.21</strong></td><td>64.72</td></tr><tr><td>K = 4</td><td>64.77</td><td>33.49</td><td>63.97</td><td>63.51</td><td>49.07</td><td>70.37</td><td>60.07</td><td>66.10</td><td><strong>61.07</strong></td><td>49.12</td><td><strong>92.07</strong></td><td>66.28</td></tr><tr><td>Mean</td><td>61.42</td><td><strong>31.77</strong></td><td>59.27</td><td><strong>59.62</strong></td><td>44.86</td><td>60.47</td><td>60.07</td><td><strong>65.20</strong></td><td>61.83</td><td><strong>47.06</strong></td><td><strong>92.05</strong></td><td><strong>55.77</strong></td></tr><tr><td>Lm</td><td><strong>55.16</strong></td><td><strong>31.02</strong></td><td><strong>52.72</strong></td><td><strong>60.96</strong></td><td><strong>42.97</strong></td><td><strong>55.34</strong></td><td><strong>54.64</strong></td><td>78.06</td><td>63.55</td><td><strong>44.55</strong></td><td>100.11</td><td><strong>58.91</strong></td></tr><tr><td colspan="14"><strong>PCC</strong></td></tr><tr><td>Baseline</td><td>0.936</td><td>0.884</td><td>0.917</td><td>0.897</td><td>0.930</td><td>0.907</td><td>0.927</td><td>0.879</td><td>0.918</td><td>0.925</td><td>0.830</td><td>0.911</td></tr><tr><td>AR(52)</td><td>0.958</td><td>0.898</td><td>0.916</td><td>0.919</td><td>0.952</td><td>0.915</td><td>0.935</td><td>0.879</td><td>0.919</td><td>0.946</td><td>0.815</td><td>0.929</td></tr><tr><td>Argo</td><td>0.970</td><td><strong>0.930</strong></td><td><strong>0.951</strong></td><td><strong>0.958</strong></td><td><strong>0.966</strong></td><td>0.939</td><td><strong>0.951</strong></td><td><strong>0.915</strong></td><td>0.941</td><td><strong>0.979</strong></td><td>0.857</td><td>0.964</td></tr><tr><td>Net</td><td>0.964</td><td>0.892</td><td>0.906</td><td>0.942</td><td>0.963</td><td>0.933</td><td>0.928</td><td>0.881</td><td>0.936</td><td>0.956</td><td>0.830</td><td>0.957</td></tr><tr><td>K = 1</td><td><strong>0.971</strong></td><td>0.922</td><td>0.932</td><td>0.952</td><td>0.968</td><td>0.935</td><td>0.939</td><td>0.909</td><td><strong>0.952</strong></td><td>0.971</td><td>0.844</td><td>0.961</td></tr><tr><td>K = 2</td><td><strong>0.971</strong></td><td>0.922</td><td>0.934</td><td>0.946</td><td>0.969</td><td>0.931</td><td>0.941</td><td><strong>0.911</strong></td><td><strong>0.952</strong></td><td>0.971</td><td>0.845</td><td><strong>0.966</strong></td></tr><tr><td>K = 3</td><td>0.970</td><td>0.920</td><td>0.936</td><td>0.946</td><td><strong>0.970</strong></td><td>0.933</td><td>0.940</td><td>0.887</td><td>0.946</td><td>0.971</td><td><strong>0.874</strong></td><td>0.961</td></tr><tr><td>K = 4</td><td>0.967</td><td>0.919</td><td>0.929</td><td>0.948</td><td>0.964</td><td>0.912</td><td>0.945</td><td>0.905</td><td>0.944</td><td>0.975</td><td>0.869</td><td>0.956</td></tr><tr><td>Mean</td><td><strong>0.971</strong></td><td>0.923</td><td>0.937</td><td><strong>0.955</strong></td><td><strong>0.969</strong></td><td><strong>0.946</strong></td><td>0.944</td><td>0.909</td><td>0.943</td><td>0.974</td><td><strong>0.870</strong></td><td><strong>0.970</strong></td></tr><tr><td>Lm</td><td>0.966</td><td><strong>0.926</strong></td><td><strong>0.948</strong></td><td>0.952</td><td>0.969</td><td><strong>0.940</strong></td><td><strong>0.948</strong></td><td>0.878</td><td>0.938</td><td><strong>0.979</strong></td><td>0.841</td><td>0.965</td></tr></tbody></table>

https://doi.org/10.1371/journal.pone.0250890.t010

RMSE and for 5 regions, ARGO is the model with the first highest PCC. In comparison, for the one- and two-week time horizons, Figs 3–6 confirm that ARGONet outperforms all other models.

Fig 7 is a visualization of estimates obtained with ARGONet and AR(52) models.

To assess the statistical significance of the improved prediction power of ARGONet, we constructed a 95% confidence interval for the relative efficiency of ARGONet compared to the autoregressive model AR(52) (the error of ARGONet is in the denominator). Table 11 shows that in real-time, the improvement obtained thanks to the ARGONet model compared to the

<figure><img src="image_6.png" /><figcaption>Fig 2. Visualization of correlation and errors obtained for real-time estimate with each model.</figcaption></figure>

https://doi.org/10.1371/journal.pone.0250890.g002
<figure><img src="image_7.png" /><figcaption>Fig 3. One-week ahead estimate obtained with ARGONet and AR(52) models from January 2015 to March 2017.</figcaption></figure>

https://doi.org/10.1371/journal.pone.0250890.g003

autoregressive model AR(52) is statistically significant for all regions. Depending on the region, ARGONet allows to reduce the error by 8% to 32%.

Fig 8 shows a typical example of plot obtained for estimates in real time with ARGONet and AR(52) models. This plot allows also us to visualize the forecast error. On this plot, we can see that for the autoregressive model, there is a time lag more important. Fig 9 is a heatmap showing the coefficients used in ARGO model. On the heatmap, we can see that ARGO model uses mostly five variables including two variables from Google Data, two variables from Hospital Data and one variable from Historical Data. Similar plots are presented in Supplementary for all the other regions.

**One-week ahead estimate.** Table 12 shows results for one-week ahead estimates for the time period January 2015-March 2017. Over this time period, the 90% CI of the best correlation is [0.852;0.970] with a median value equal to 0.936. The 90% CI of the best RMSE is [43.53;89.45] with a median value equal to 62.435. All these values are mostly obtained with

<figure><img src="image_8.png" /><figcaption>Fig 4. Two-week ahead estimate obtained with ARGONet and AR(52) models from January 2015 to March 2017.</figcaption></figure>

https://doi.org/10.1371/journal.pone.0250890.g004
<figure><img src="image_9.png" /><figcaption>Fig 5. Error distribution.</figcaption></figure>

https://doi.org/10.1371/journal.pone.0250890.g005

ARGONet model which implies a reduction of the error from 22% to 67% compared to the baseline. In comparison to the best results, AR(52) and the baseline are the models giving the highest errors and lowest correlations. In contrast to real-time estimates, ARGO and Net models are comparable. Indeed, for seven regions, Net model is the model having the second lowest
<figure><img src="image_10.png" /><figcaption>Fig 6. Correlation distribution.</figcaption></figure>

https://doi.org/10.1371/journal.pone.0250890.g006

<figure><img src="image_11.png" /><figcaption>Fig 7. Real-time estimate obtained with ARGONet and AR(52) models from January 2015 to March 2017.</figcaption></figure>

https://doi.org/10.1371/journal.pone.0250890.g007

<table><caption>Table 11. Real-time estimate—relative efficiency being higher than one suggests increased predictive power of ARGONet compared to the autoregressive model AR(52).</caption><thead><tr><td>Region</td><td>Relative efficiency</td><td>95% CI</td></tr></thead><tbody><tr><td>Auv.</td><td>1.10</td><td>[1.06;1.14]</td></tr><tr><td>Bour.</td><td>1.20</td><td>[1.16;1.23]</td></tr><tr><td>Bre.</td><td>1.11</td><td>[1.06;1.15]</td></tr><tr><td>Cen.</td><td>1.29</td><td>[1.25;1.33]</td></tr><tr><td>Gd Est</td><td>1.17</td><td>[1.14;1.21]</td></tr><tr><td>Ht Fra.</td><td>1.27</td><td>[1.22;1.32]</td></tr><tr><td>Ile Fra.</td><td>1.07</td><td>[1.04;1.09]</td></tr><tr><td>Norm.</td><td>1.13</td><td>[1.10;1.17]</td></tr><tr><td>Aqui.</td><td>1.22</td><td>[1.19;1.24]</td></tr><tr><td>Occi.</td><td>1.40</td><td>[1.33;1.47]</td></tr><tr><td>Loi.</td><td>1.16</td><td>[1.13;1.19]</td></tr><tr><td>Pro.</td><td>1.45</td><td>[1.38;1.52]</td></tr></tbody></table>

https://doi.org/10.1371/journal.pone.0250890.t011
<figure><img src="image_12.png" /><figcaption>Fig 8. Nouvelle-Aquitaine real time estimate.</figcaption></figure>

https://doi.org/10.1371/journal.pone.0250890.g008

errors. These results can be observed on Fig 10 and on the distribution of correlation and error (Figs 5 and 6). Similar results are also observed for the flu season period (S8 Table in S1 File).

Fig 3 is a visualization of results obtain with ARGONet and AR(52) models for one-week ahead estimates.

Table 13 shows that the improvement obtained with ARGONet model compared to the autoregressive model AR(52) is statistically significant for all regions for one-week ahead estimates. Depending on the region, ARGONet allows to reduce the error by 18% to 59%.

Fig 11 shows one-week ahead estimate obtained for the french region Nouvelle-Aquitaine. On this plot, we can see that AR(52) still have a lag of one or two weeks in contrast to ARGONet model. The heatmap on Fig 12, shows that ARGO model uses mostly seven variables including three variables from Google Data, two variables from Hospital Data, one variable from Climatic data and one variable from Historical data.

**Two-week ahead estimate.** Table 14 shows results for two-week ahead estimates for the time period January 2015-March 2017. Over this time period, the 90% CI of the best correlation is [0.825;0.935] with a median value equal to 0.885. The 90% CI of the best relative error is [59.28;105.77] with a median value equal to 79.53. As for real-time and one-week ahead estimates, ARGONet is the model giving the best results in terms of correlation and error whereas AR(52) and the baseline model give the least accurate results. ARGONet allows a reduction of the error from 37% to 67% compared to the baseline. For most of the french regions, the method giving the highest correlation and lowest error for ARGONet is the method using the mean between estimates obtained from ARGO and Net models.

Fig 13, allows to visualize that ARGONet is the best model for all regions in term of correlation and error. These results are confirmed with the distribution of correlation and error of each model obtained by calculating the PCC and RMSE for each flu season and each region. (Figs 5 and 6).

Fig 4 allows to visualize estimates obtained with ARGONet and AR(52) models.

Table 15 shows that the improvement obtained with the ARGONet model compared to the AR(52) model is statistically significant for all regions for two-week ahead estimate. Depending on the region, ARGONet allows to reduce the error by 27% to 57%.

Fig 14 shows two-week ahead estimates for the region Nouvelle-Aquitaine. As for one-week ahead estimate, we can see that estimates obtained with AR(52) is still delayed. It is not the
<figure><img src="image_13.png" /><figcaption>Fig 9. Coefficients Nouvelle-Aquitaine real-time estimate.</figcaption></figure>

https://doi.org/10.1371/journal.pone.0250890.g009
<table><caption>Table 12. PCC and RMSE for one-week ahead estimate for all french regions for the period starting from January 2015 to March 2017.</caption><thead><tr><th></th><th>Auv.</th><th>Bour.</th><th>Bre.</th><th>Cen.</th><th>Gd Est</th><th>Ht Fra.</th><th>Ile Fra.</th><th>Norm.</th><th>Aqui.</th><th>Occi.</th><th>Loi.</th><th>Pro.</th></tr></thead><tbody><tr><td colspan="14"><strong>RMSE</strong></td></tr><tr><td>Baseline</td><td>136.56</td><td>66.90</td><td>108.59</td><td>150.66</td><td>106.89</td><td>115.98</td><td>116.45</td><td>120.85</td><td>129.93</td><td>130.65</td><td>150.85</td><td>166.71</td></tr><tr><td>AR(52)</td><td>105.36</td><td>59.41</td><td>99.54</td><td>122.54</td><td>83.21</td><td>103.89</td><td>101.55</td><td>110.63</td><td>117.00</td><td>102.94</td><td>139.49</td><td>134.43</td></tr><tr><td>Argo</td><td>115.70</td><td>56.25</td><td>84.90</td><td>96.91</td><td>88.17</td><td>94.86</td><td>96.91</td><td><strong>89.45</strong></td><td>106.28</td><td>91.96</td><td>137.27</td><td>105.82</td></tr><tr><td>Net</td><td>120.74</td><td>42.07</td><td>93.78</td><td>91.97</td><td>76.19</td><td>86.38</td><td>113.57</td><td>112.50</td><td>98.63</td><td>84.34</td><td>154.18</td><td>87.44</td></tr><tr><td>K = 1</td><td>93.47</td><td>42.92</td><td>77.12</td><td>89.46</td><td>72.52</td><td>83.44</td><td><strong>84.34</strong></td><td>108.51</td><td>77.74</td><td>86.23</td><td><strong>98.04</strong></td><td>89.48</td></tr><tr><td>K = 2</td><td>109.39</td><td>46.51</td><td>78.70</td><td>92.20</td><td>80.97</td><td>92.23</td><td>104.61</td><td>110.08</td><td>84.42</td><td>86.52</td><td>121.87</td><td>84.05</td></tr><tr><td>K = 3</td><td>111.03</td><td>43.67</td><td>75.76</td><td>94.88</td><td>79.74</td><td>89.66</td><td>96.62</td><td>110.53</td><td>89.80</td><td>85.08</td><td><strong>117.30</strong></td><td>99.06</td></tr><tr><td>K = 4</td><td>112.71</td><td>40.34</td><td>69.04</td><td>103.34</td><td>81.70</td><td>89.91</td><td>94.96</td><td>110.37</td><td>79.24</td><td>85.63</td><td>121.59</td><td>105.17</td></tr><tr><td>Mean</td><td><strong>75.95</strong></td><td><strong>39.93</strong></td><td><strong>64.33</strong></td><td><strong>59.84</strong></td><td><strong>48.54</strong></td><td><strong>71.84</strong></td><td><strong>72.44</strong></td><td><strong>90.96</strong></td><td><strong>60.54</strong></td><td><strong>51.23</strong></td><td><strong>117.64</strong></td><td><strong>54.75</strong></td></tr><tr><td>Lm</td><td><strong>70.84</strong></td><td><strong>37.46</strong></td><td><strong>67.18</strong></td><td><strong>70.46</strong></td><td><strong>43.53</strong></td><td><strong>70.23</strong></td><td>85.18</td><td>105.60</td><td><strong>62.65</strong></td><td><strong>61.20</strong></td><td>132.59</td><td><strong>60.61</strong></td></tr><tr><td colspan="14"><strong>PCC</strong></td></tr><tr><td>Baseline</td><td>0.793</td><td>0.672</td><td>0.771</td><td>0.704</td><td>0.788</td><td>0.749</td><td>0.777</td><td>0.702</td><td>0.748</td><td>0.765</td><td>0.660</td><td>0.729</td></tr><tr><td>AR(52)</td><td>0.868</td><td>0.692</td><td>0.782</td><td>0.779</td><td>0.861</td><td>0.782</td><td>0.813</td><td>0.707</td><td>0.758</td><td>0.860</td><td>0.645</td><td>0.804</td></tr><tr><td>Argo</td><td>0.881</td><td>0.803</td><td>0.877</td><td>0.903</td><td>0.904</td><td>0.890</td><td>0.856</td><td><strong>0.816</strong></td><td>0.839</td><td>0.917</td><td>0.747</td><td>0.913</td></tr><tr><td>Net</td><td>0.864</td><td>0.861</td><td>0.828</td><td>0.886</td><td>0.906</td><td>0.859</td><td>0.824</td><td>0.716</td><td>0.867</td><td>0.916</td><td><strong>0.654</strong></td><td>0.928</td></tr><tr><td>K = 1</td><td>0.919</td><td>0.868</td><td>0.899</td><td>0.899</td><td>0.919</td><td>0.875</td><td>0.907</td><td>0.735</td><td>0.926</td><td>0.928</td><td><strong>0.852</strong></td><td>0.921</td></tr><tr><td>K = 2</td><td>0.908</td><td>0.854</td><td>0.902</td><td>0.896</td><td>0.910</td><td>0.859</td><td>0.865</td><td>0.727</td><td>0.921</td><td>0.957</td><td>0.787</td><td>0.930</td></tr><tr><td>K = 3</td><td>0.909</td><td>0.862</td><td>0.901</td><td>0.891</td><td>0.914</td><td>0.863</td><td>0.876</td><td>0.727</td><td>0.903</td><td>0.963</td><td><strong>0.801</strong></td><td>0.909</td></tr><tr><td>K = 4</td><td>0.905</td><td><strong>0.888</strong></td><td><strong>0.927</strong></td><td><strong>0.927</strong></td><td><strong>0.927</strong></td><td>0.885</td><td>0.864</td><td>0.884</td><td>0.730</td><td>0.919</td><td>0.955</td><td>0.895</td></tr><tr><td>Mean</td><td><strong>0.947</strong></td><td>0.882</td><td><strong>0.922</strong></td><td><strong>0.953</strong></td><td><strong>0.966</strong></td><td><strong>0.913</strong></td><td><strong>0.922</strong></td><td><strong>0.816</strong></td><td><strong>0.945</strong></td><td><strong>0.970</strong></td><td>0.783</td><td><strong>0.971</strong></td></tr><tr><td>Lm</td><td><strong>0.944</strong></td><td><strong>0.894</strong></td><td>0.915</td><td><strong>0.936</strong></td><td><strong>0.965</strong></td><td><strong>0.904</strong></td><td><strong>0.909</strong></td><td>0.752</td><td><strong>0.943</strong></td><td><strong>0.966</strong></td><td>0.728</td><td><strong>0.965</strong></td></tr></tbody></table>

https://doi.org/10.1371/journal.pone.0250890.t012

case for ARGONet model. On the heatmap, Fig 15, we can see that ARGO model uses mostly nine variables, including six variables from Google Data, one variable from Climatic Data, two variables from Historical data.

## Discussion

We have introduced a machine learning ensemble methodology that combines multiple data sources and multiple statistical approaches to accurately track flu activity in the twelve continental regions of France. To the best of our knowledge, this is a spatial resolution for which no

<figure><img src="image_14.png" /><figcaption>Fig 10. Visualization of correlation and errors obtained for one-week ahead estimate with each model.</figcaption></figure>

https://doi.org/10.1371/journal.pone.0250890.g010
<table><caption>Table 13. One-week ahead estimate—relative efficiency being higher than one suggests increased predictive power of ARGONet compared to the autoregressive model AR(52).</caption><thead><tr><td>Region</td><td>Relative efficiency</td><td>95% CI</td></tr></thead><tbody><tr><td>Auv.</td><td>1.47</td><td>[1.42;1.52]</td></tr><tr><td>Bour.</td><td>1.70</td><td>[1.62;1.78]</td></tr><tr><td>Bre.</td><td>1.54</td><td>[1.48;1.60]</td></tr><tr><td>Cen.</td><td>2.07</td><td>[1.95;2.19]</td></tr><tr><td>Gd Est</td><td>1.94</td><td>[1.85;2.04]</td></tr><tr><td>Ht Fra.</td><td>1.51</td><td>[1.47;1.55]</td></tr><tr><td>Ile Fra.</td><td>1.42</td><td>[1.37;1.46]</td></tr><tr><td>Norm.</td><td>1.38</td><td>[1.31;1.45]</td></tr><tr><td>Aqui.</td><td>2.05</td><td>[1.94;2.15]</td></tr><tr><td>Occi.</td><td>1.97</td><td>[1.86;2.08]</td></tr><tr><td>Loi.</td><td>1.45</td><td>[1.35;1.54]</td></tr><tr><td>Pro.</td><td>2.42</td><td>[2.30;2.55]</td></tr></tbody></table>

https://doi.org/10.1371/journal.pone.0250890.t013

forecasting approaches have been explored before in France. Our methodology provides real-time estimates as well as one- and two-week ahead forecasts.

The success of our approach comes from the ability to dynamically identify the appropriate method and data sources to produce the best disease activity estimates for a given location and time horizon in a prospective way (out-of-sample). Specifically, we show that the ARGO model alone (one that does not incorporate flu activity from neighboring regions) yields accurate results for real-time estimates but fails to produce optimal predictions for longer-term time-horizons. We find that the Net model (one that leverages information from neighboring regions alone) leads to reasonable flu predictions but tends to overestimate epidemic peaks. The proposed ensemble approach, named ARGONet (that combines information from both ARGO and the Net model), an extension of a model proposed in the USA [9], produces forecasts with the lowest errors and highest correlation as captured by Fig 1. Particularly, the most reliable longer-term forecasts are obtained with ARGONet's method using the mean of estimates from ARGO and Net models. This machine-learning ensemble approach displays both

<figure><img src="image_15.png" /><figcaption>Fig 11. Nouvelle-Aquitaine one-week ahead estimate.</figcaption></figure>

https://doi.org/10.1371/journal.pone.0250890.g011
<figure><img src="image_16.png" /><figcaption>Fig 12. Coefficients Nouvelle-Aquitaine one-week ahead estimate.</figcaption></figure>

https://doi.org/10.1371/journal.pone.0250890.g012

accuracy and robustness to estimate ILI activity up to two-weeks ahead of time at the french regional level. Our methodological approach was inspired by Lu et al. [9] using ARGO, Net and ARGONet methods to track flu activity at state level in United States. However, Rangara-jan et al. [40] have shown that potential improvements can be achieved in data-driven

<table><caption>Table 14. PCC and RMSE for two-week ahead estimate for all french regions for the period starting from January 2015 to March 2017.</caption><thead><tr><th></th><th>Auv.</th><th>Bour.</th><th>Bre.</th><th>Cen.</th><th>Gd Est</th><th>Ht Fra.</th><th>Ile Fra.</th><th>Norm.</th><th>Aqui.</th><th>Occi.</th><th>Loi.</th><th>Pro.</th></tr></thead><tbody><tr><td colspan="14"><strong>RMSE</strong></td></tr><tr><td>Baseline</td><td>183.37</td><td>83.10</td><td>142.04</td><td>198.19</td><td>143.26</td><td>148.72</td><td>155.39</td><td>154.69</td><td>173.17</td><td>178.29</td><td>175.17</td><td>218.07</td></tr><tr><td>AR(52)</td><td>144.22</td><td>69.89</td><td>116.89</td><td>155.47</td><td>111.85</td><td>124.06</td><td>129.82</td><td>132.51</td><td>145.67</td><td>128.13</td><td>157.68</td><td>164.00</td></tr><tr><td>Argo</td><td>140.92</td><td>59.58</td><td>116.14</td><td>116.91</td><td>104.27</td><td>123.72</td><td>122.35</td><td>107.48</td><td>134.90</td><td>119.40</td><td>133.39</td><td>120.45</td></tr><tr><td>Net</td><td>161.43</td><td>46.06</td><td>113.19</td><td>114.30</td><td>122.32</td><td>112.73</td><td>140.13</td><td>129.32</td><td>177.22</td><td>81.00</td><td>144.54</td><td>124.58</td></tr><tr><td>K = 1</td><td>130.99</td><td>44.73</td><td><strong>107.81</strong></td><td><strong>66.95</strong></td><td>104.91</td><td>96.25</td><td>115.71</td><td>113.29</td><td>144.06</td><td><strong>62.30</strong></td><td>124.25</td><td>105.48</td></tr><tr><td>K = 2</td><td>143.37</td><td>43.62</td><td>115.40</td><td><strong>76.26</strong></td><td>110.01</td><td>94.20</td><td>120.63</td><td>116.80</td><td>144.78</td><td>67.12</td><td>124.46</td><td>108.19</td></tr><tr><td>K = 3</td><td>139.22</td><td>43.53</td><td>113.27</td><td>86.15</td><td>116.06</td><td>106.26</td><td><strong>110.86</strong></td><td>120.85</td><td>151.50</td><td>64.75</td><td><strong>113.65</strong></td><td>106.03</td></tr><tr><td>K = 4</td><td>139.90</td><td>43.86</td><td>117.70</td><td>91.28</td><td>119.35</td><td>105.56</td><td>114.64</td><td>113.98</td><td>154.33</td><td>68.86</td><td>113.83</td><td>125.38</td></tr><tr><td>Mean</td><td><strong>107.89</strong></td><td><strong>41.48</strong></td><td><strong>80.62</strong></td><td>87.88</td><td><strong>71.24</strong></td><td><strong>88.21</strong></td><td><strong>86.90</strong></td><td><strong>91.60</strong></td><td><strong>102.77</strong></td><td>68.05</td><td><strong>111.15</strong></td><td><strong>78.44</strong></td></tr><tr><td>Lm</td><td><strong>105.77</strong></td><td><strong>38.58</strong></td><td>120.10</td><td>87.78</td><td><strong>91.78</strong></td><td>77.39</td><td>128.37</td><td>119.14</td><td>173.22</td><td><strong>59.28</strong></td><td>120.66</td><td><strong>82.49</strong></td></tr><tr><td colspan="14"><strong>PCC</strong></td></tr><tr><td>Baseline</td><td>0.627</td><td>0.494</td><td>0.608</td><td>0.484</td><td>0.616</td><td>0.584</td><td>0.603</td><td>0.508</td><td>0.550</td><td>0.563</td><td>0.538</td><td>0.536</td></tr><tr><td>AR(52)</td><td>0.731</td><td>0.522</td><td>0.665</td><td>0.609</td><td>0.733</td><td>0.658</td><td>0.688</td><td>0.539</td><td>0.610</td><td>0.731</td><td>0.528</td><td>0.699</td></tr><tr><td>Argo</td><td>0.823</td><td>0.753</td><td>0.766</td><td>0.829</td><td>0.835</td><td>0.840</td><td>0.779</td><td>0.727</td><td>0.735</td><td>0.829</td><td>0.712</td><td>0.860</td></tr><tr><td>Net</td><td>0.759</td><td>0.832</td><td>0.786</td><td>0.814</td><td>0.781</td><td>0.775</td><td>0.775</td><td>0.677</td><td>0.700</td><td>0.905</td><td>0.631</td><td>0.847</td></tr><tr><td>K = 1</td><td>0.869</td><td>0.844</td><td><strong>0.831</strong></td><td>0.838</td><td><strong>0.865</strong></td><td>0.838</td><td>0.850</td><td>0.740</td><td><strong>0.826</strong></td><td>0.897</td><td>0.688</td><td>0.868</td></tr><tr><td>K = 2</td><td>0.856</td><td><strong>0.861</strong></td><td>0.829</td><td>0.846</td><td>0.862</td><td>0.856</td><td>0.857</td><td>0.722</td><td>0.814</td><td>0.904</td><td>0.749</td><td>0.891</td></tr><tr><td>K = 3</td><td>0.860</td><td>0.845</td><td>0.828</td><td>0.847</td><td>0.853</td><td>0.848</td><td>0.857</td><td>0.723</td><td>0.820</td><td>0.879</td><td><strong>0.765</strong></td><td>0.855</td></tr><tr><td>K = 4</td><td>0.857</td><td>0.814</td><td>0.826</td><td>0.832</td><td>0.850</td><td>0.853</td><td><strong>0.858</strong></td><td>0.741</td><td>0.767</td><td>0.901</td><td>0.750</td><td>0.860</td></tr><tr><td>Mean</td><td><strong>0.901</strong></td><td>0.857</td><td><strong>0.876</strong></td><td><strong>0.895</strong></td><td><strong>0.909</strong></td><td><strong>0.873</strong></td><td><strong>0.893</strong></td><td><strong>0.825</strong></td><td><strong>0.858</strong></td><td><strong>0.938</strong></td><td><strong>0.802</strong></td><td><strong>0.935</strong></td></tr><tr><td>Lm</td><td><strong>0.886</strong></td><td><strong>0.876</strong></td><td>0.795</td><td><strong>0.852</strong></td><td>0.833</td><td><strong>0.865</strong></td><td>0.850</td><td><strong>0.745</strong></td><td>0.754</td><td><strong>0.906</strong></td><td>0.744</td><td><strong>0.917</strong></td></tr></tbody></table>

https://doi.org/10.1371/journal.pone.0250890.t014
<figure><img src="image_17.png" /><figcaption>Fig 13. Visualization of correlation and errors obtained for two-week ahead estimate with each model.</figcaption></figure>

https://doi.org/10.1371/journal.pone.0250890.g013
<table><caption>Table 15. Two-week ahead estimate—relative efficiency being bigger than one suggests increased predictive power of ARGONet compared to the AR(52) model.</caption><thead><tr><td>Region</td><td>Relative efficiency</td><td>95% CI</td></tr></thead><tbody><tr><td>Auv.</td><td>1.39</td><td>[1.35;1.43]</td></tr><tr><td>Bour.</td><td>1.91</td><td>[1.84;1.98]</td></tr><tr><td>Bre.</td><td>1.44</td><td>[1.38;1.49]</td></tr><tr><td>Cen.</td><td>2.36</td><td>[2.25;2.46]</td></tr><tr><td>Gd Est</td><td>1.54</td><td>[1.51;1.58]</td></tr><tr><td>Ht Fra.</td><td>1.62</td><td>[1.58;1.66]</td></tr><tr><td>Ile Fra.</td><td>1.49</td><td>[1.44;1.53]</td></tr><tr><td>Norm.</td><td>1.49</td><td>[1.45;1.53]</td></tr><tr><td>Aqui.</td><td>1.44</td><td>[1.39;1.49]</td></tr><tr><td>Occi.</td><td>2.24</td><td>[2.15;2.33]</td></tr><tr><td>Loi.</td><td>1.41</td><td>[1.35;1.47]</td></tr><tr><td>Pro.</td><td>1.99</td><td>[1.90;2.08]</td></tr></tbody></table>

https://doi.org/10.1371/journal.pone.0250890.t015

forecasting methods by exploiting sparsity structures in the predictors. Future studies could explore the efficacy of these techniques for flu prediction in France.

Prediction error reductions are observed when using ARGONet over its autoregressive counterpart (AR(52)) (up to 32% across regions) in real-time predictions. As the time-horizon of prediction increases, the improvements of predictions are more evident, leading to up to 60% error reductions when comparing ARGONet with AR(52), and up to 70% error reduction when comparing ARGONet with the baseline (Tables 12 and 14). S2 through S13 Figs in S1 File show these results graphically. As expected, autoregressive approaches show “within-range” prediction values that consistently lag behind the observed disease activity and lead to under-predictions close to peak activity.

We find that all external data sources contribute to improving local flu estimates, when compared to the autoregressive model (AR(52)), specially for longer-term forecasts. Indeed, for the two-week ahead estimates, the combination of EHR data and Google data lead to correlation improvements of up to 30% and decreases in error of up to 25%. For Climatic data, this improvement reaches 20% for correlation and 11% for the error. For Twitter data, it reaches

<figure><img src="image_18.png" /><figcaption>Fig 14. Nouvelle-Aquitaine two-week ahead estimate.</figcaption></figure>

https://doi.org/10.1371/journal.pone.0250890.g014
<figure><img src="image_19.png" /><figcaption>Fig 15. Coefficients Nouvelle-Aquitaine two-week ahead estimate.</figcaption></figure>

https://doi.org/10.1371/journal.pone.0250890.g015

20% for correlation and 7% for the error. By analyzing heatmaps (S5, S9 and S13 Figs in S1 File) obtained for ARGO models, we can see that the contribution of different predictors (data sources) changes over time and time-horizon of prediction, but all data sources appear to possess predictive power. Indeed, the most important data sources are EHR data and Google data in real-time and for longer-term forecasts. Historical data is consistently used in real-time, but less used for longer-term forecasting. Conversely, Climatic data and Twitter data are used more prominently for longer-term forecasts than for real-time estimate.

The fact that we could only access EHR data from Rennes University Hospital, and thus from the Brittany region, prevented us from being able to quantify the added value of region-specific EHR information on flu predictions in their respective region. This should be evaluated in future research efforts. On the other hand, we find interesting the fact that data from a hospital in Rennes can improve flu forecasting in other regions. Indeed, S4-S6 Tables in S1 File show that forecasts that include EHR information from Rennes, up to two weeks, are more accurate for all the regions when compared to the autoregressive model (AR(52)). EHR data appears to be more relevant for some regions than others. For example, it appears to be an important predictor in the Brittany region (which contains Rennes) as expected, as well as in Normandy, which shares a border with Brittany. For Occitanie, EHR data from Rennes improves predictions, which is in alignment with the fact that historical information shows that flu activity tends to occur synchronously (with a correlation of 0.93) as seen in S1 Fig in S1 File. We hypothesize that having access to region-specific EHR data, from all the french regions, will lead to prediction improvements across the board.

Twitter data was collected at the National level given the sparsity of relevant flu-related Tweets at the regional level. This was the case as we only had access to the publicly available data shared by Twitter's API that only allows users to view up to 5% of all Geo-coded Tweets (themselves a small fraction of about 5% of the total corpus of all Tweets). We also suspect that gaining access to higher volumes of Tweets at the regional level could improve our forecasts.

For climatic data, we only had access to weekly local temperature and precipitation. Future studies may explore incorporating other climatic indicators known to be more directly related to the transmission of the virus, such as humidity [32].

For historical data, the variables with highest predictive power include lags or 52 weeks (one year). However, some other long-term lags show up as important in predictions (as 42
and 43 lags). Given the short time period of our study, we suspect that the flu seasons that we studied may have had specific trends (an early season combined with a late season) that could cause our methods to identify a meaningful influences of lags that are shorter than the intuitive 52 week lag.

Data retrieved from Google Correlate is normalized by Google in a (frequently) distinct sample and over different time periods depending on the data request. This pre-normalization can affect our results, but as shown in [2] the process of dynamic training minimizes the impact of this instability.

Our methods were designed to produce point estimates and decision-makers who are potential end-users of the output of our models would benefit from a quantification of the confidence we have in our predictions. For such purposes, by conducting a historical analysis on the errors between out-of-sample predictions and subsequent observations, we find that for a forward looking prediction, the bracket ($\hat{y}_t - RMSE, \hat{y}_t + RMSE$) can be thought of as a 95% confidence interval for each region at every point in time (See S80-S82 Figs in S1 File). This is consistent with previous work by Yang et al [2] where the collection of observed errors between (out-of-sample) predictions and subsequent observations were fitted using a Gaussian distribution (over a moving window of about 2 years, 104 observations) and the RMSE was found to be comparable to the standard deviation of such distribution (See S83-S85 Figs in S1 File). This is an empirical result and suggests that in 100 out-of- sample observations, 95 will fall within the suggested bracket around the point prediction. As shown in S80-S82 Figs in S1 File most observations fall within the proposed confidence intervals prospectively, confirming the validity of our approach.

To conclude, we have shown that Internet-based data sources can yield accurate influenza estimates in the twelve continental regions in France. Operational implementations of these methods may prove to be useful for public health officials in the face of public health threats. Our regional-level flu estimates may contribute to better management of patients' flow in general practitioners' offices and in hospitals, particularly emergency departments.

**Supporting information**

S1 File.

(PDF)

**Author Contributions**

**Conceptualization:** Canelle Poirier, Mauricio Santillana.

**Formal analysis:** Canelle Poirier.

**Investigation:** Canelle Poirier.

**Methodology:** Canelle Poirier, Mauricio Santillana.

**Resources:** Yulin Hswen, Guillaume Bouzillé, Thomas Brewer.

**Supervision:** Mauricio Santillana.

**Validation:** Mauricio Santillana.

**Writing – original draft:** Canelle Poirier.

**Writing – review & editing:** Yulin Hswen, Guillaume Bouzillé, Marc Cuggia, Audrey Lavenu, John S. Brownstein, Thomas Brewer, Mauricio Santillana.
References

1. Ferguson N. M. et al. Strategies for mitigating an influenza pandemic. *Nature* 442, 448–452, (2006).
https://doi.org/10.1038/nature04795 PMID: 16642006

2. Yang S., Santillana M. & Kou S. C. Accurate estimation of influenza epidemics using google search data via ARGO. *Proc. Natl. Acad. Sci. United States Am.* 112, 14473–14478, (2015). https://doi.org/10.1073/pnas.1515373112 PMID: 26553980

3. Yang W., Lipsitch M. & Shaman J. Inference of seasonal and pandemic influenza transmission dynamics. *Proc. Natl. Acad. Sci. United States Am.* 112, 2723–2728, (2015). https://doi.org/10.1073/pnas.1415012112 PMID: 25730851

4. Kalimeri K. et al. Unsupervised extraction of epidemic syndromes from participatory influenza surveillance self-reported symptoms. *PLOS Comput. Biol.* 15, e1006173, (2019). https://doi.org/10.1371/journal.pcbi.1006173 PMID: 30958817

5. Fleming D.M., P. W., van der Velden J. The evolution of influenza surveillance in europe and prospects for the next 10 years. *Vaccine* 21, 1749–1753 (2003). https://doi.org/10.1016/S0264-410X(03)00066-5 PMID: 12686088

6. Santillana M. et al. Cloud-based electronic health records for real-time, region-specific influenza surveillance. *Sci. Reports* 6, 25732, (2016). https://doi.org/10.1038/srep25732 PMID: 27165494

7. Nsoesie E. O., Brownstein J. S., Ramakrishnan N. & Marathe M. V. A systematic review of studies on forecasting the dynamics of influenza outbreaks. *Influ. Other Respir. Viruses* 8, 309–316 (2014).
https://doi.org/10.1111/irv.12226 PMID: 24373466

8. Yang W., Karspeck A. & Shaman J. Comparison of filtering methods for the modeling and retrospective forecasting of influenza epidemics. *PLOS Comput. Biol.* 10, e1003583, (2014). https://doi.org/10.1371/journal.pcbi.1003583 PMID: 24762780

9. Lu F. S., Hattab M. W., Clemente C. L., Biggerstaff M. & Santillana M. Improved state-level influenza nowcasting in the united states leveraging internet-based data and network approaches. *Nat. communications* 10, 1–10 (2019). https://doi.org/10.1038/s41467-018-08082-0 PMID: 30635558

10. Zhang Y., Bambrick H., Mengersen K., Tong S. & Hu W. Using google trends and ambient temperature to predict seasonal influenza outbreaks. *Environ. Int.* 117, 284–291, (2018). https://doi.org/10.1016/j.envint.2018.05.016 PMID: 29778013

11. Lampos V., Miller A. C., Crossan S. & Stefansen C. Advances in nowcasting influenza-like illness rates using search query logs. *Sci. Reports* 5, 1–10, (2015). Number: 1 Publisher: Nature Publishing Group.
https://doi.org/10.1038/srep12760 PMID: 26234783

12. Zou, B., Lampos, V. & Cox, I. Multi-task learning improves disease models from web search. *Proc. 2018 World Wide Web Conf.* 87–96, (2018).

13. Santillana M., Zhang D. W., Althouse B. M. & Ayers J. W. What can digital disease detection learn from (an external revision to) google flu trends? *Am. journal preventive medicine* 47, 341–347 (2014).
https://doi.org/10.1016/j.amepre.2014.05.020 PMID: 24997572

14. Paul M. J., Dredze M. & Broniatowski D. Twitter improves influenza forecasting. *PLoS Curr. 6*, (2014).
https://doi.org/10.1371/currents.outbreaks.90b9ed0f59bae4ccaa683a39865d9117 PMID: 25642377

15. Santillana M. et al. Combining search, social media, and traditional data sources to improve influenza surveillance. *PLoS Comput. Biol.* 11, (2015). https://doi.org/10.1371/journal.pcbi.1004513 PMID: 26513245

16. Mowery J. Twitter influenza surveillance: Quantifying seasonal misdiagnosis patterns and their impact on surveillance estimates. *Online J. Public Health Informatics* 8, (2016). https://doi.org/10.5210/ojphi.v8i3.7011 PMID: 28210419

17. Nagar R. et al. A case study of the new york city 2012-2013 influenza season with daily geocoded twitter data from temporal and spatiotemporal perspectives. *J. medical Internet research* 16, e236 (2014).
https://doi.org/10.2196/jmir.3416 PMID: 25331122

18. Signorini A., Segre A. M. & Polgreen P. M. The use of twitter to track levels of disease activity and public concern in the us during the influenza a h1n1 pandemic. *PloS one* 6 (2011). https://doi.org/10.1371/journal.pone.0019467 PMID: 21573238

19. Sharpe J. D., Hopkins R. S., Cook R. L. & Striley C. W. Evaluating google, twitter, and wikipedia as tools for influenza surveillance using bayesian change point analysis: A comparative analysis. *JMIR Public Health. Surveillance* 2, (2016). https://doi.org/10.2196/publichealth.5901 PMID: 27765731

20. McIver D. J. & Brownstein J. S. Wikipedia usage estimates prevalence of influenza-like illness in the united states in near real-time. *PLOS Comput. Biol* 10, e1003581, (2014). https://doi.org/10.1371/journal.pcbi.1003581 PMID: 24743682
21. Generous N., Fairchild G., Deshpande A., Valle S. Y. D. & Priedhorsky R. Global disease monitoring and forecasting with wikipedia. *PLOS Comput. Biol* 10, e1003892, (2014). https://doi.org/10.1371/journal.pcbi.1003892 PMID: 25392913

22. Hickmann K. S. et al. Forecasting the 2013–2014 influenza season using wikipedia. *PLOS Comput. Biol* 11, e1004239, (2015). https://doi.org/10.1371/journal.pcbi.1004239 PMID: 25974758

23. Carneiro H. A. & Mylonakis E. Google trends: A web-based tool for real-time surveillance of disease outbreaks. *Clin. Infect. Dis.* 49, 1557–1564, (2009). https://doi.org/10.1086/630200 PMID: 19845471

24. Ginsberg J. et al. Detecting influenza epidemics using search engine query data. *Nature* 457, 1012–1014, (2009). Number: 7232 Publisher: Nature Publishing Group. https://doi.org/10.1038/nature07634 PMID: 19020500

25. Butler D. When google got flu wrong. *Nature* 494, 155–156, (2013). https://doi.org/10.1038/494155a PMID: 23407515

26. Santillana M., Nsoesie E. O., Mekaru S. R., Scales D. & Brownstein J. S. Using clinicians' search query data to monitor influenza epidemics. *Clin. Infect. Dis. An Off. Publ. Infect. Dis. Soc. Am.* 59, 1446–1450, (2014). https://doi.org/10.1093/cid/ciu647 PMID: 25115873

27. Smolinski M. S. et al. Flu near you: Crowdsourced symptom reporting spanning 2 influenza seasons. *Am. J. Public Health.* 105, 2124–2130, (2015-10). https://doi.org/10.2105/AJPH.2015.302696

28. Biggerstaff M. et al. Results from the second year of a collaborative effort to forecast influenza seasons in the united states. *Epidemics* 24, 26–33, (2018). https://doi.org/10.1016/j.epidem.2018.02.003 PMID: 29506911

29. Poirier C. et al. Real time influenza monitoring using hospital big data in combination with machine learning methods: Comparison study. *JMIR Public Health. Surveillance* 4, e11361, (2018). https://doi.org/10.2196/11361 PMID: 30578212

30. Bouzillé G. et al. Leveraging hospital big data to monitor flu epidemics. *Comput. Methods Programs Biomed.* 154, 153–160, (2018). https://doi.org/10.1016/j.cmpb.2017.11.012 PMID: 29249339

31. Viboud C. et al. Demonstrating the use of high-volume electronic medical claims data to monitor local and regional influenza activity in the us. *PloS one* 9, e102429 (2014). https://doi.org/10.1371/journal.pone.0102429 PMID: 25072598

32. Lowen A. C. & Steel J. Roles of humidity and temperature in shaping influenza seasonality. *J. Virol.* 88, 7692–7695, (2014). https://doi.org/10.1128/JVI.03544-13 PMID: 24789791

33. Lowen A. C., Mubareka S., Steel J. & Palese P. Influenza virus transmission is dependent on relative humidity and temperature. *PLOS Pathog.* 3, e151, (2007). https://doi.org/10.1371/journal.ppat.0030151 PMID: 17953482

34. Tamerius J. D. et al. Environmental predictors of seasonal influenza epidemics across temperate and tropical climates. *PLOS Pathog.* 9, e1003194, (2013). https://doi.org/10.1371/journal.ppat.1003194 PMID: 23505366

35. Lawrence. The relationship between relative humidity and the dewpoint temperature in moist air. *Am. Meteorol. Soc.* (2005-02). https://doi.org/10.1175/BAMS-86-2-225

36. Tibshirani R. Regression shrinkage and selection via the lasso. *J. Royal Stat. Soc.* 58, 267–288 (1996).

37. R Core Team. *R: A Language and Environment for Statistical Computing* (R Foundation for Statistical Computing, 2015).

38. Kuhn, M. caret: *Classification and Regression Training* (2018). R package version 6.0-80.

39. Trapletti, A. & Hornik, K. *tseries: Time Series Analysis and Computational Finance* (2015).

40. Rangarajan P., Mody S. K. & Marathe M. *Forecasting dengue and influenza incidences using a sparse representation of google trends, electronic health records, and time series data.* 15, e1007518.