# Enhanced surveillance of hospitalized COVID-19 patients in Europe: an evaluation of the I-MOVE-COVID-19 surveillance network

Johanna Young, Heather Mutch, Angela Mc Rose, Josie Evans, Jim Mcmenamin, Josie Murray, Ausenda Machado, Ana Paula Rodrigues, Irina Kislaya, Giedre Gefenaite, et al.

<figure><img src="image_1.png" /></figure>

## To cite this version:

Johanna Young, Heather Mutch, Angela Mc Rose, Josie Evans, Jim Mcmenamin, et al. "Enhanced surveillance of hospitalized COVID-19 patients in Europe: an evaluation of the I-MOVE-COVID-19 surveillance network." European Journal of Public Health, 2024, 34 (1), pp.181-189. 10.1093/eur-pub/ckad185. hal-04885955

HAL Id: hal-04885955

https://hal.science/hal-04885955v1

Submitted on 14 Jan 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# Enhanced surveillance of hospitalized COVID-19 patients in Europe: an evaluation of the I-MOVE-COVID-19 surveillance network

Johanna J. Young ID ¹, Heather Mutch ID ¹, Angela M.C. Rose², Josie M.M. Evans ID ¹ and the European COVID-19 Hospital Evaluation Writing Group*

1 Clinical and Protecting Health Directorate, Public Health Scotland, Glasgow, UK

2 Epiconcept, Paris, France

**Correspondence:** Josie M.M. Evans, Public Health Scotland, 2 Central Quay, 89 Hydepark Street, Glasgow G3 8BW, UK, Tel: 0345 646 0238, e-mail: josie.evans@phs.scot

*The members of the European COVID-19 Hospital Surveillance Evaluation Writing Group are listed in the Acknowledgements.

**Background:** A pre-existing, well-established European influenza surveillance network known as I-MOVE enabled the rapid implementation of a European multi-country COVID-19 hospital surveillance network for surveillance of hospitalized COVID-19 cases in early 2020. This network included 257 hospitals in 11 surveillance sites across nine countries. We aimed to identify whether the surveillance objectives were relevant to public health actions, whether the surveillance system met its objectives, where and how shortcomings could be improved, and whether the system was sustainable. **Methods:** We identified six key attributes (meeting objectives, usefulness, timeliness, data quality, simplicity and sustainability) to assess, using Centers for Disease Control and Prevention's evaluation framework. We analyzed pooled datasets, held interviews and group discussions with 10 participating and coordinating sites and gathered feedback through web surveys. **Results:** There was overall agreement that the surveillance objectives had been met and being involved in a network of European partners had additional important benefits for stakeholders. While the publication of the outputs was not always sufficiently timely, data submission processes were considered straightforward and the key surveillance variables (age, sex, hospital admission and mortality data) were complete. The main challenges were identified as the collection of the large number of variables, limited available human resources and information governance and data protection laws. **Conclusions:** I-MOVE-COVID-19 delivered relevant and accurate data supporting the development and implementation of COVID-19 surveillance. Recommendations presented here identify learning opportunities to support preparedness and surveillance response for future pandemics. The applied evaluation framework in this study can be adapted for other European surveillance system evaluations.

## Introduction

In response to the emergence of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), many European countries established comprehensive multi-level surveillance systems for COVID-19 cases, to provide critical information for public health decision-making.¹

The I-MOVE-COVID-19 hospital surveillance network was established by the I-MOVE-COVID-19 Consortium (founded in February 2020 and coordinated by Epiconcept²,³). This was an expansion of the multi-country Influenza-Monitoring Vaccine Effectiveness (I-MOVE) network founded in 2007 to measure influenza vaccine effectiveness (VE) in Europe.⁴ Having this pre-existing, well-established European platform facilitated the development and implementation of COVID-19 hospital surveillance across surveillance sites, as they were able to build on existing common protocols, expand existing information governance procedures and re-train staff already familiar with the surveillance activities.

The surveillance objectives were to (i) describe clinical and epidemiological characteristics of hospitalized COVID-19 SARI patients, (ii) describe the virological characteristics of SARS-CoV-2 in these patients, (iii) improve understanding of severe disease progression to guide patient management and public health response, (iv) strengthen COVID-19 preparedness through hospital surveillance,

(v) describe severe COVID-19 cases by sex, age group and risk/protective factors; and (vi) describe in-hospital COVID-19 deaths.⁵ These objectives were established in the early days of the COVID-19 pandemic in a rapidly changing context with much uncertainty as to how the situation would develop. It was envisaged that pooling data from each country, thereby providing a greater sample size, would allow for a more accurate and representative description of hospitalized COVID-19 cases. Surveillance bulletins were prepared quarterly, and regular network meetings were held to share experiences on the implementation processes and developments across each participating site's surveillance systems.

Any surveillance system, and particularly new systems, should be evaluated to identify possible improvements in performance and the overall public health response. Our evaluation objectives were (i) to identify whether the surveillance objectives were relevant to public health action, (ii) to assess the surveillance objectives and areas for improvement, (iii) to evaluate the sustainability of the surveillance system and finally, (iv) to consider more efficient or alternative routes to achieve the surveillance objectives. We report the results of this comprehensive evaluation of the European COVID-19 hospital surveillance system and provide recommendations to improve the current system and to support the planning and implementation of future enhanced surveillance activities.
# Methods

## COVID-19 hospital surveillance network

Results were collected from 11 surveillance sites, i.e. groups of hospitals and public health institutes within specific European regions that submitted the surveillance data collectively. A total of 257 participating hospitals across nine European countries were included in I-MOVE-COVID-19 (Albania, Belgium, England, France, Lithuania, Portugal, Romania, Scotland and Spain). Spain and France had two participating sites in different regions. With the exceptions of hospitals in England and Scotland—where surveillance was nationwide and register-based via data linkage of routine hospital and infection datasets—an average of three hospitals were included per site. All 23 sentinel hospitals collected COVID-19 hospital data through questionnaire-based surveillance, either on paper or electronically. Every quarter, the participating sites securely submitted their case-based surveillance data to Epiconcept. There were 105 separate variables requested for data submission covering patient demographics, hospital records, severity indexes, risk factors, SARS-CoV-2 presenting symptoms, laboratory test results and clinical information. The data were cleaned and pooled before sharing with Public Health Scotland (PHS), who together with Epiconcept led on analysis (figure 1). Analytical results were presented in quarterly surveillance bulletins that were published on the I-MOVE-COVID-19 website³ and used to improve understanding of I-MOVE's objectives. No ethical approval was required for this evaluation as existing non-identifiable data were used.

## Evaluation framework

The framework for this evaluation was adapted from the 'Updated Guidelines for Evaluating Public Health Surveillance Systems' of the Centers for Disease Control and Prevention (CDC).⁶ The surveillance system was assessed against six attributes tailored to suit the scope and the specific evaluation objectives including the achievement of surveillance objectives, usefulness, timeliness, data quality, simplicity and sustainability. Indicators to measure these attributes were defined using adapted CDC and European Centre for Disease Prevention and Control (ECDC) guidelines⁶,⁷ (table 1).

## Data collection

The evaluation was performed unblinded by three internal evaluators from PHS, part of the I-MOVE-COVID-19 hospital surveillance

network, and data were collected through a mixed-method approach using quantitative and qualitative data collection methods, including document review and stakeholder consultation (survey and semi-structured group discussions and interviews). Table 1 shows the data collected for each indicator, reviewed between October and December 2021.

## Document review

All documents and systems related to the surveillance system were identified and reviewed. These included the protocol for I-MOVE-COVID-19 hospital phased surveillance, documents describing standard operating procedures, surveillance bulletins, scientific posters presented at scientific conferences, network meeting agendas and meeting minutes.

## Survey

The indicators of each attribute were used to formulate survey questions that were administered online via LimeSurvey.⁸ The survey was piloted and adapted accordingly prior to its administration. The survey link was sent to all network members from the participating sites. Respondents were asked to identify their role in the surveillance system but otherwise remained anonymous.

## Group discussions/interview

Depending on participant availability, either semi-structured interviews or group discussions were held with the sites, to allow a blend of closed- and open-ended questions, accompanied by follow-up questions. The questions were prepared in advance using the attributes and indicators under surveillance and the key discussion points were shared with the interviewees in advance of the meeting. The sessions were held virtually by the authors using Microsoft Teams between November 2021 and January 2022. Interview notes were recorded on a structured interview template according to each attribute and coded into the most commonly mentioned themes within each indicator by two researchers.

## Analysis of the datasets

Pooled datasets from six data collections with data up to 31 December 2021 were used to derive quantitative indicators (table 1). Timeliness of the surveillance system was measured by calculating the number of days between key steps in the surveillance process;

<figure><img src="image_3.png" /><figcaption>**Figure 1** The process for cleaning, pooling and analysing data from 11 sites</figcaption></figure>
<table><caption>Table 1 Methods to evaluate the surveillance system attributes and its set key indicators</caption><thead><tr><td>Attribute</td><td>Indicator</td><td>Type of data</td><td>Data collection method(s)</td></tr></thead><tbody><tr><td>Attribute 1: The achievement of surveillance objectives</td><td>Number, percentage of participating site representatives who think written objectives have been met by the system</td><td>Quantitative and qualitative</td><td>Web survey, semi-structured interviews/group discussions, document review</td></tr><tr><td>Attribute 2: Usefulness</td><td>Data collection<br/>Number and list of variables reported which are required but considered not useful at the European level<br/>Number and list of additional variables not being collected but which would be useful to be collected at European level<br/>Use of European level data for decision-making processes<br/>Use of European level data for decision-making or to improve surveillance (e.g. has this European level data been used to guide policy at national level)<br/>Added value of participation in a European surveillance network<br/>Usefulness of being part of European COVID-19 hospital surveillance network (e.g. added value of European network, networking meetings and surveillance bulletins)</td><td>Quantitative and qualitative<br/>Qualitative<br/>Qualitative</td><td>Web survey, semi-structured interviews/group discussions<br/>Web survey, semi-structured interviews/group discussions</td></tr><tr><td>Attribute 3: Timeliness</td><td>Number of days between key steps<br/>Number of days between the dataset received by PHS from Epiconcept completed for analysis and publication date<br/>Median number of days between mean admission date across all sites and date of each surveillance bulletin publication<br/>Median number of days that data was submitted across all sites prior to agreed submission deadline<br/>Balance between timeliness and information needed<br/>Timeliness and frequency of system reporting (data dissemination) to meet surveillance objectives</td><td>Quantitative<br/>Qualitative</td><td>Document review, analysis of final dataset<br/>Web survey, semi-structured interviews/group discussions</td></tr><tr><td>Attribute 4: Data quality</td><td>Variable completeness<br/>Number of excluded cases that did not meet the case definition<br/>Stakeholder opinion of data quality (whether they feel it is sufficient to meet objectives)</td><td>Quantitative<br/>Qualitative</td><td>Analysis of final dataset<br/>Web survey, semi-structured interviews/group discussions</td></tr><tr><td>Attribute 5: Simplicity</td><td>Opinion on simplicity of the surveillance system<br/>Opinion on the data collection process<br/>Opinion on the data collation<br/>Data submission<br/>Opinion on the data collation<br/>Person-days for data preparation—need of additional resources (on top of usual workload) done by routine services.<br/>Feasibility of reporting deadlines for data collection</td><td>Qualitative<br/>Qualitative<br/>Qualitative and quantitative</td><td>Semi-structured interviews/ group discussions<br/>Web survey, semi-structured interviews/group discussions<br/>Web survey, semi-structured interviews/group discussions</td></tr><tr><td>Attribute 6: Sustainability</td><td>Plans for continuation of data collection and/or expansion</td><td>Qualitative</td><td>Semi-structured interviews/ group discussions<br/>Semi-structured interviews/ group discussions</td></tr></tbody></table>

specifically, the time between the initial date of analysis and the median date of admission, and the publication date of the surveillance bulletins. The data quality of the surveillance system was measured by calculating the level of completeness of the collected variables, the proportion of reported cases that fit the case definition⁹ and the number of unanalyzed variables. All analyses were performed using Microsoft Excel.

## Results

### Document review

The I-MOVE-COVID-19 surveillance protocol and the I-MOVE-COVID-19 hospital surveillance evaluation protocol were identified and reviewed, as well as all six pooled datasets submitted during the data collection points, six I-MOVE-COVID-19 surveillance bulletins

and two scientific poster presentations presented at The European Scientific Conference on Applied Infectious Disease Epidemiology (ESCAIDE) 2021 conference.⁵,¹⁰⁻¹⁶ All network meeting agendas and meeting minutes of the hospital surveillance system were identified and included in the document review.

### Semi-structured group discussions and interviews and web survey

There were two interviews with individuals from two sites and four group discussions. Overall, all countries and 9 out of 11 participating sites were represented. The link to the web survey was sent to all stakeholders in the network and ten completed surveys were received. The roles of the respondents were varied (diverse and multiple roles could be selected): regional and national public health institute-based network members (n = 3), co-ordinators (n = 3),
data managers (n=2), hospital-based network members (n=2), university-based network members (n=2) and laboratory expert (n=1).

## Evaluation of system attributes

### The achievement of surveillance objectives

Respondents generally indicated that the surveillance system met all six of its objectives. From the 10 completed surveys received, 8 responded to six parts of this question. Most questions (42/48) were answered by the respondents in (strong) agreement that the surveillance system met its objectives. Five responses were neutral, and only one answer disagreed that the surveillance system met all its objectives.

## Usefulness

### Data collection

Forty-two percent (61/145) of the requested variables as defined in the study protocol and collected for surveillance purpose (table 3) were selected by at least one respondent as unnecessary for surveillance purposes. They also commented on variables which could be omitted for other reasons, such as postcode which could lead to patient identification, clinical characteristics that did not align with typical epidemiological surveillance, and hospital ward and patient test/scan results as these were free text complicating comparability.

Six out of nine interviewed sites reported that too many variables were required, resulting in high levels of missing data for some variables that were not considered core. It was suggested that the number of variables should be determined according to variable completeness levels, and from both the interviewees and respondents, a common view was that if timely data collection is desired then only essential variables, such as patient demographics, hospital information (e.g. hospital admission and discharge dates) and severity indicators should be collected (table 3).

## Added value of participation in a European surveillance network

Stakeholders identified that a key strength of this surveillance system was that it involved a network of European partners. Having a network supported the understanding of developments across different sites, strengthened relationships with stakeholders, supported advocacy for national COVID-19 hospital surveillance and helped to attract funding from other sponsors to support the surveillance activities. Collaboration within the network allowed sites to automate data flow processes which was recognized and highly praised at the national governmental level.

The quarterly publication of the surveillance data and network meetings were generally perceived positively. This allowed sites to

raise awareness of the importance of collecting this data, thereby supporting the implementation and development of their surveillance.

## Timeliness

### Number of days between key steps in the surveillance process

An average of 39 days (range: 15–63 days) was recorded between the pooled dataset being sent for analysis and publication date (table 2). Across surveillance bulletins, the median number of days between mean admission date and publication was 132 days (table 2). On average, the median date of dataset submission was within 1 day of the agreed submission deadline (table 2).

### Timeliness and frequency of system reporting (data dissemination) to meet surveillance objectives

Half of the respondents considered the frequency of the publications sufficient to meet the surveillance objectives, although it was suggested that more regular, short, and rapid communications could have been combined with lengthier in-depth surveillance bulletins published less frequently. However, it was also acknowledged that more frequent data submissions would have caused an increased burden on an already-strained workforce.

## Data quality

Key surveillance variables (age, sex, hospital admission and mortality data) had completion rates of 100%. Patient's type of residence, healthcare worker status, symptom onset date and most chronic conditions had completion rates between 30% and 69%, whereas patient's smoking status, pregnancy status, three chronic conditions (anaemia, stroke and tuberculosis), and all symptom data had completion rates <25% (table 3). The overall completion rate of the variables varied by site and decreased over time. Data from England and Scotland were the main contributors to this decrease as they supplied more data and their data completion for non-essential surveillance variables decreased over time. An average of 14% (n=9611/69 734) of cases that did not meet the case definition were excluded. The proportion of analyzed variables in the surveillance bulletins varied between 43% and 63%.

## Stakeholder opinion of data quality

While most interviewees performed regular data management activities to ensure better data quality, such as variable recoding, deduplication, cross-validation and translation to English at a national level, several sites indicated that these were completed by the hospital sites directly. Generally, interviewees indicated that data quality improved over time. One interviewee suggested introducing

<table><caption>**Table 2** Number of days between key steps to assess timeliness in the surveillance system, by surveillance bulletin, I-MOVE-COVID-19 hospital surveillance network, Europe, 2020–21</caption><thead><tr><th></th><th>Number of days between the dataset received by PHS from Epiconcept and publication date</th><th>Median number of days between mean admission date across all sites and date of each surveillance bulletin publication (IQR)</th><th>Median number of days that data were submitted across all sites prior to agreed submission deadline (IQR)</th></tr></thead><tbody><tr><td>Surveillance bulletin 1</td><td>15</td><td>160 (141–169)</td><td>—</td></tr><tr><td>Surveillance bulletin 2</td><td>51</td><td>98 (143–293)</td><td>10 (6 to 25)</td></tr><tr><td>Surveillance bulletin 3</td><td>38</td><td>106 (139–345)</td><td>5 (–4 to 15)</td></tr><tr><td>Surveillance bulletin 4</td><td>41</td><td>160 (204–458)</td><td>0 (–8 to 1)</td></tr><tr><td>Surveillance bulletin 5</td><td>63</td><td>111 (248–362)</td><td>–11 (–3 to 13)</td></tr><tr><td>Surveillance bulletin 6</td><td>26</td><td>158 (180–475)</td><td>0 (–4 to 1)</td></tr><tr><td>Overall (mean)</td><td>39</td><td>132 (176–350)</td><td>1 (–4 to 8)</td></tr></tbody></table>

Notes: No formal deadline was given for the first data submission; therefore, the number of days that data were submitted prior to the agreed submission deadline could not be calculated for surveillance bulletin 1. Minus figures refers to days submitted after the agreed submission deadline.
<table><caption>**Table 3** Average completion rates across all participating sites of the I-MOVE COVID-19 hospital surveillance network and data submissions of variables analyzed and included in the surveillance bulletins, by category</caption><thead><tr><th>Category</th><th>Variable</th><th>Definition</th><th>Average variable completion rates across all sites and data submissions (%)</th></tr></thead><tbody><tr><td rowspan="3">Patient characteristics</td><td>Age</td><td>Age of patient</td><td>100</td></tr><tr><td>Sex</td><td>Sex of patient</td><td>100</td></tr><tr><td>Smoking</td><td>Never, former (stopped smoking at least 1 year before inclusion in the study), current smoker</td><td>9</td></tr><tr><td rowspan="2">Hospital information</td><td>Pregnant</td><td>Whether patient is pregnant</td><td>24</td></tr><tr><td>Residence</td><td>Patient residence at time of SARI onset. Whether patient was living at home or was institutionalized, or had pre-hospital dependence on home support/care</td><td>68</td></tr><tr><td rowspan="6">Severity indicators</td><td>Hcw</td><td>Whether the patient is a healthcare worker</td><td>48</td></tr><tr><td>Admitdate</td><td>The hospital admission date of each patient</td><td>100</td></tr><tr><td>Icuadmitdate</td><td>Date first admitted ICU/HDU</td><td>100</td></tr><tr><td>Swabdate</td><td>Respiratory specimen collection date</td><td>94</td></tr><tr><td>Dischargedate</td><td>Date of hospital discharge</td><td>63</td></tr><tr><td>Icu</td><td>Admission to ICU or HDU</td><td>96</td></tr><tr><td rowspan="2">Symptoms at admission</td><td>Icudischargedate</td><td>Date last discharged from ICU/HDU</td><td>96</td></tr><tr><td>Vent</td><td>Patient's level of mechanical ventilation. Note that option 1 is for respiratory support level ECMO, option 2 includes any high-flow (6 l/min or higher, including OptiFlow), and option 3 includes any noninvasive, positive pressure ventilator</td><td>84</td></tr><tr><td rowspan="10">Underlying chronic conditions</td><td>Outcome</td><td>Indicate the outcome of the patient known at the time of data collection</td><td>94</td></tr><tr><td>Deathdate</td><td>Date of death</td><td>100</td></tr><tr><td>Deathcause</td><td>Cause of death</td><td>100</td></tr><tr><td>Onsetdate</td><td>Date of onset of symptoms</td><td>65</td></tr><tr><td>Malaise</td><td>Malaise</td><td>8</td></tr><tr><td>Headache</td><td>Headache</td><td>9</td></tr><tr><td>Cough</td><td>Cough</td><td>10</td></tr><tr><td>Sob</td><td>Shortness of breath</td><td>10</td></tr><tr><td>Sorethroat</td><td>Sore throat</td><td>8</td></tr><tr><td>Myalgia</td><td>Myalgia</td><td>9</td></tr><tr><td>Vomit</td><td>Vomiting</td><td>8</td></tr><tr><td>Diarr</td><td>Diarrhoea</td><td>9</td></tr><tr><td>Abdopain</td><td>Abdominal pain</td><td>9</td></tr><tr><td>General_deter</td><td>Deterioration of general condition (asthenia or loss of weight or anorexia or confusion or dizziness)</td><td>4</td></tr><tr><td>Suddenonset</td><td>Sudden onset</td><td>2</td></tr><tr><td>Ageusia</td><td>Loss of sense of taste</td><td>6</td></tr><tr><td>Anosmia</td><td>Loss of sense of smell</td><td>6</td></tr><tr><td>Dysgeusia</td><td>Distortion of the sense of taste</td><td>1</td></tr><tr><td>Fever</td><td>History of fever</td><td>9</td></tr><tr><td>Feverish</td><td>Sub-febrility (37-38°C)</td><td>2</td></tr><tr><td>Chills</td><td>'Chills', shivering</td><td>2</td></tr><tr><td>Coryza</td><td>Coryza</td><td>4</td></tr><tr><td>Dizzy</td><td>Dizziness</td><td>3</td></tr><tr><td>Tach</td><td>Tachypnoea or signs of low oxygen saturation</td><td>4</td></tr><tr><td>Palp</td><td>Palpitations</td><td>2</td></tr><tr><td>Nausea</td><td>Nausea</td><td>3</td></tr><tr><td>Nausea_vomit</td><td>Nausea and vomit</td><td>3</td></tr><tr><td>Conjunct</td><td>Conjunctivitis</td><td>7</td></tr><tr><td>Dermato</td><td>Rash or other dermatological manifestations of COVID-19</td><td>8</td></tr><tr><td>Confusion</td><td>Confusion</td><td>9</td></tr><tr><td>Chest</td><td>Chest pain</td><td>6</td></tr><tr><td>Liverdis</td><td>Chronic liver disease (excluding cancer)</td><td>69</td></tr><tr><td>Diabetes</td><td>Dementia</td><td>70</td></tr><tr><td>Heartdis</td><td>Heart/cardiac disease (excluding hypertension)</td><td>72</td></tr><tr><td>Cancer</td><td>Cancer (any)</td><td>49</td></tr><tr><td>immuno</td><td>HIV (including other immunodeficiency, organ transplantation)</td><td>49</td></tr><tr><td>Lungdis</td><td>Lung disease (excluding asthma)</td><td>66</td></tr><tr><td>Rendis</td><td>Renal disease (excluding cancer and acute renal failure)</td><td>72</td></tr><tr><td>Dement</td><td>Dementia</td><td>48</td></tr><tr><td>Stroke</td><td>Stroke</td><td>12</td></tr><tr><td>Rheumat</td><td>Rheumatologic disease</td><td>49</td></tr><tr><td>Anaemia</td><td>Anaemia/chronic haematologic disease</td><td>12</td></tr><tr><td>Tuberc</td><td>Tuberculosis</td><td>6</td></tr><tr><td>Asplenia</td><td>Asplenia (absence of/damage to spleen)</td><td>37</td></tr><tr><td>Asthma</td><td>Asthma</td><td>71</td></tr><tr><td>Hypert</td><td>Hypertension</td><td>58</td></tr><tr><td>Neuromusc</td><td>Neuromuscular disorder</td><td>61</td></tr><tr><td>Obese</td><td>Obesity (only if height, weight and BMI not collected; can be calculated)</td><td>60</td></tr></tbody></table>

Notes: BMI, body mass index; ECMO, extracorporeal membrane oxygenation; HDU, high-dependency unit; ICU, intensive care unit.
conditions and rules during automated data collection to facilitate
high data quality and improved completeness levels.

Simplicity

Data collection

Several challenges were identified by respondents around the simpli-
city of data collection, including information governance processes
and following data protection laws, obtaining clearance from data
protection bodies or patients, and developing automated data collec-
tion approaches. Respondents and interviewees suggested that more
frequent updates of the surveillance protocols and training sessions
for new staff would be helpful.

Data collation

The collation of data was deemed straightforward, although respond-
ents expressed challenges such as the lack of automation in some
sites, high workload and the large number of requested clinical var-
iables (e.g. chest X-ray findings, ECG findings).

Data submission

Preparing the data for the submission process was considered straightforward by respondents and interviewees. This process took between 2 and 30 days with an average of 11.5 days. Four out of nine sites reported during the interviews that data submission deadlines were challenging due to competing deadlines, lack of human resources, compliance with information governance procedures or delayed data collection in hospitals.

Sustainability

Sites' stances differed regarding the continuation of COVID-19 sur-
veillance post the I-MOVE-COVID-19 project. Sites intending to
continue their COVID-19 hospital surveillance will do so either
through their national surveillance programme or through other
integrated European respiratory surveillance networks such as the
'Vaccine Effectiveness, Burden and Impact Studies' (VEBIS),¹⁷ which
also includes E-SARI-NET, the European SARI surveillance network.
For sites not part of these networks, continuity of the surveillance
will depend on the development of the pandemic.

Discussion

This evaluation of the multi-country European hospital surveillance
system for COVID-19, focused on the description of the system and
assessment of the surveillance attributes. Survey respondents were in
overall agreement that the surveillance system met its objectives.

Usefulness

This rapidly deployed European surveillance network was an important enhancement to existing surveillance mechanisms used in Europe. Sharing and pooling of European data allowed questions to be answered that individual countries could not answer efficiently alone, particularly where data is scarce (e.g. information on chronic conditions).¹⁸ Being involved in a European network was also considered extremely valuable, as it assisted individual sites to strengthen national surveillance and supported relationship building with different national and international stakeholders. Cross-country co-operation and data sharing have been recognized to support the centralization of efforts, disseminate information and better prepare and respond to global health challenges.¹⁹,²⁰ Combining the expertise and resources of I-MOVE influenza, an existing European multidisciplinary network, to include COVID-19 has shown that European surveillance systems can be adapted in a timely and flexible way suggesting potential for adaptability to also include other pathogens. While the experience and infrastructure from this surveillance

network could form a foundation for future emerging pathogens, it
is acknowledged that the adaptability of the surveillance system
would still depend on various factors such as the nature of the patho-
gen, the availability of data, and coordination between different
stakeholders.

Many hospitals and clinical staff were overburdened during the pandemic, so the collection of the large number of variables required at the start of the pandemic to better understand this novel virus in a hospital setting was challenging. Data collection should be designed to meet the information needs of public health decision-makers, the public and health workers.²¹ It is key that the benefits of collecting additional variables are balanced against the potentially increased burden on clinical staff. A minimum dataset for integrated influenza/SARS-CoV-2 sentinel surveillance is recommended to support the data quality and sustainability of future surveillance systems.²¹,²²

Timeliness

The publication of the bulletins was not timely enough to influence public health actions. However, most sites shared their own data prior to publication to inform decision-makers and ensure an adequate response. It is acknowledged that improved timeliness of the reporting of pooled and therefore more powerful analyses could have influenced wider public health actions but it was also important not to compromise the validity and quality of the data, and to burden sites with additional data collection and submissions. The development of automated data submission systems linked to electronic databases would address this problem, but the development of such systems across Europe is heterogeneous.²³

Data quality

Basic demographic data generally had high completeness levels, whereas more specific patient characteristics were less complete. While some issues stemmed from clinicians being overwhelmed by the pressures of the pandemic, other reasons for incomplete data collection included difficulty obtaining ethical permission for certain variables, patients' unwillingness to disclose sensitive information²¹ and symptom data not always being routinely reported in hospital settings. While sites that collected data through a sentinel questionnaire-based approach generally reached higher levels of data completeness, sites that collected their data through linkages of national registers accrued more cases. These sites supplying more data submitted fewer non-essential surveillance variables, resulting in lower variable completeness. The World Health Organization (WHO) suggests that good quality data with timely reporting, even from fewer sites, are more useful than a large volume of poor-quality data not reported in a timely manner.²² It is therefore key that sites collecting data on more cases assess whether they can be effectively managed, monitored and sustained, while considering the increased human and financial resources, technical and operational assistance that may be required.²²

Simplicity

While the operation of the COVID-19 hospital surveillance was gen-
erally perceived as straightforward, the main challenges were asso-
ciated with data collection processes. The collection of COVID-19-
specific hospital data has been previously identified as an additional
burden for clinical staff.²⁴ To facilitate the collation and analysis of
data at the national level, countries may consider establishing or
strengthening electronic data platforms that link epidemiological
and virologic data to sentinel and non-sentinel surveillance systems
and are accessible to stakeholders.²² Flexible electronic systems
should support data collection and transfer in the changing
COVID-19 pandemic situations as well as in other emerging
outbreaks.²⁵
## Sustainability

The sites took different positions on the continuation of COVID-19 hospital surveillance, with some planning to continue and others depending on the pandemic's development. ECDC and WHO recommend integrated respiratory surveillance of COVID-19, influenza and other respiratory pathogens.²¹,²² While most European countries have established comprehensive surveillance systems for COVID-19 with a large proportion reporting all positive cases regardless of indication for testing, European countries are now being encouraged to transition from emergency surveillance to more sustainable and objective-driven surveillance systems.²¹ As a result, integrated European respiratory surveillance networks that include COVID-19 hospital surveillance have been developed.¹⁷

## Limitations

This evaluation has some limitations. The variance of the different types of surveillance systems (sentinel vs. national surveillance), as well as the discrepancies in datasets, number of collected variables and completion rates, made comparisons challenging. The evaluation was performed during an emergency response, which may have reduced stakeholder participation, thereby reducing representativeness from participating sites. For example, due to staffing and resource issues, it was not possible to arrange separate one-to-one interviews with all stakeholders from all participating sites which would have provided the most comprehensive picture, but we were able to arrange either group discussions or interviews with representatives from all countries and nine out of 11 sites. In contrast, the response rate of the survey was low, despite reminder e-mails being sent to the network in attempts to improve the response rate. Responses to certain questions of the survey may also have reflected the respondent's perception of the current system within their site or specific area of work and may not necessarily provide a general view of the overall participating site. Although facilitators attempted to mitigate dominance bias, responses may have been biased towards participants with strong opinions. The evaluation was performed by internal evaluators, which enabled a strong understanding of the system, an extended period of participant observation, and the ability to make ongoing improvements throughout the implementation. However, this may have introduced confirmation bias and inhibited their ability to act purely as objective outsiders.

## Recommendations

Our evaluation has generated the following recommendations specific to any future deployment of an integrated respiratory surveillance system:

### Surveillance methods

* Increase the timeliness of reporting data to support public health actions in a timely fashion.

* Reduce the minimum mandatory dataset in collaboration with participating sites to enable increased data completeness and improve data quality.

* Consider introducing a uniform template which is consistent between data submissions to harmonize the order and coding of variables.

* Further reduce the substantial inter- and intra-country differences in surveillance methods to ensure homogeneity of the scope, focus, objectives, methodology, resources and reporting across different regions and countries.

### Communication

* Continue further data analysis to identify preventive and risk factors with the available data collected through the surveillance and share results with the network and scientific community.

## Sustainability

* Monitor outbreak-related workload issues to ensure the sustainability of the surveillance.

* Consider lessons learned from the I-MOVE-COVID-19 hospital surveillance network when data submission continues under other networks (VEBIS) to ensure sustainability.

## Conclusion

This evaluation found that the COVID-19 hospital surveillance system met its surveillance objectives by describing COVID-19 cases with severe disease and outcomes. There were clear-cut added benefits for stakeholders from being involved in a collaborative European network. The COVID-19 pandemic, an unprecedented event, put an immense pressure on respiratory surveillance systems. The proposed recommendations presented here identify learning opportunities to support preparedness and surveillance response for future pandemics. Finally, the evaluation framework applied in this paper can be instrumental for other European surveillance system evaluations, particularly for those established as a result of the COVID-19 pandemic.

## Acknowledgements

We thank Kamil Orzel from Public Health Scotland for his support with the development of the evaluation survey. We also acknowledge all colleagues in the hospital sites and all partners of Work Package 3 (Hospital Surveillance Network) of the I-MOVE COVID-19 consortium that supported the data collection throughout the pandemic: Instituto de Salud Carlos III (Spain), Organismo Autonomo Instituto de Salud Publica y Laboral de Navarra (Spain), Public Health Scotland (Scotland, UK), Instituto Nacional de Saude Dr. Ricardo Jorge (Portugal), United Kingdom Health Security Agency (England, UK), Lietuvos Sveikatos Mokslu Universitetas (Lithuania), Vienna Vaccine Safety Initiative (Germany) and Université Bourgogne Franche-Comté (France), Institutul National de Cercetare-Dezvoltare Medico-Militara Cantacuzino (Romania), Ministerio de Saude—Republica Portuguesa (Portugal), Institute of Public Health (Albania), Vrije Universiteit Brussel (Belgium), Assistance Publique Hopitaux de Paris (France), Centre Hospitalier Universitaire Montpellier (France), Centre Hospitalier Universitaire de Rennes (France), Hospices Civils de Lyon (France). We would also like to thank our colleagues at Epiconcept (France), and the World Health Organization and the European Centre for Disease Prevention and Control for their support throughout this project.

**Collaborators:** Members of the European COVID-19 hospital surveillance evaluation writing group:

Jim McMenamin<fnref n="1" />, Josie Murray<fnref n="1" />,<fnref n="2" />, Ausenda Machado (0000-0002-1849-1499)<fnref n="3" />, Ana Paula Rodrigues(0000-0003-2264-4723)<fnref n="3" />, Irina Kislaya (0000-0001-5772-2416)<fnref n="3" />, Giedre Gefenaite (0000-0001-9952-9446)<fnref n="4" />,⁵, Monika Kuliese (0000-0002-4601-4585)<fnref n="4" />, Ligita Jancoriene (0000-0001-6488-6312)⁶,⁷, Ieva Kubiliute (0000-0002-4220-6053)⁶,⁷, Indrė Jonikaitė<fnref n="4" />, Roberta Vaikutytė (0000-0002-5992-7746)<fnref n="4" />, Birute Zablockiene (0000-0001-8519-6265)⁶,⁷, Aukse Mikiene<fnref n="4" />, Amparo Larrauri (0000-0002-1704-2245)⁸,⁹, Clara Mazagatos (0000-0002-6909-6923)⁸,⁹, Thomas Demuyser (0000-0002-6410-9246)¹⁰,¹¹, Els Van Nedervelde¹², Lucie Seyler¹², Mihaela Lazar (0000-0002-8226-6026)¹³, Florence Galtier (0000-0002-0720-6277)¹⁴ and Philippe Vanhems¹⁵,¹⁶

<footnote n="1">¹Public Health Scotland, Glasgow, UK</footnote>

<footnote n="2">²School of Medicine, University of St Andrews, UK</footnote>

<footnote n="3">³Department of Epidemiology, National Institute of Health Doutor Ricardo Jorge, Lisbon, Portugal</footnote>

<footnote n="4">⁴Department of Infectious Diseases, Lithuanian University of Health Sciences, Kaunas, Lithuania</footnote>
⁵Department of Health Sciences, Faculty of Medicine, Lund University, Box 157, 22100 Lund, Sweden

⁶Center of Infectious Diseases, Vilnius University Hospital Santaros Klinikos, Vilnius, Lithuania

⁷Clinic of Infectious Diseases and Dermatovenerology, Institute of Clinical Medicine, Faculty of Medicine, Vilnius University, Vilnius, Lithuania

⁸National Centre for Epidemiology, Institute of Health Carlos III, Madrid, Spain

⁹Consortium for Biomedical Research in Epidemiology and Public Health (CIBERESP), Institute of Health Carlos III, Madrid, Spain

¹⁰Department of Microbiology and Infection Control, Vrije Universiteit Brussel (VUB), Universitair Ziekenhuis Brussel (UZ Brussel), Laarbeeklaan 101, 1090 Brussels, Belgium

¹¹Center for Neurosciences, Faculty of Medicine and Pharmacy, Vrije Universiteit Brussel (VUB), Laarbeeklaan 103, 1090 Brussels, Belgium

¹²Vrije Universiteit Brussel (VUB), Universitair Ziekenhuis Brussel (UZ Brussel), Internal Medicine and Infectious Diseases Department, Laarbeeklaan 101, 1090 Brussels, Belgium

¹³"Cantacuzino" National Military-Medical Institute for Research and Development, Bucharest, Romania

¹⁴Inserm CIC 1411, CHU de Montpellier, France; F-CRIN IREIVAC, Paris, France

¹⁵Hospices Civils de Lyon, Service Hygiène, Épidémiologie, Infectiovigilance et Prévention, Lyon, France

¹⁶Centre International de Recherche en Infectiologie, (CIRI), Université de Lyon, Inserm, U1111, Université Claude Bernard Lyon 1, CNRS, UMR5308, ENS de Lyon, F-69007 Lyon, France

## Funding

This project (I-MOVE-COVID-19) received funding from the European Union's Horizon 2020 Research and Innovation action funding scheme (grant agreement ID: 101003673).

Conflicts of interest: None declared.

## Data availability

The data underlying this article can be shared upon reasonable request to the corresponding author.

## Key points

* This rapidly deployed European surveillance network was an important enhancement to existing surveillance mechanisms used in Europe and participation in this collaborative European network was considered beneficial for stakeholders.

* The surveillance system could easily be adapted to include additional respiratory pathogens and be used by countries with less surveillance capability to make them resilient and agile for addressing global and national respiratory surveillance needs.

* The lessons we learned during this evaluation are critical to the successful implementation of novel long-term respiratory surveillance projects being carried out across Europe in the aftermath of the COVID-19 pandemic.

## References

1. Mutch H, Young JJ, Sadiq F, et al.; European COVID-19 Hospital Surveillance Analysis Writing Group. Enhanced surveillance of hospitalised COVID-19 patients in Europe: i-MOVE-COVID-19 surveillance network, February 2020 to December 2021. *Euro Surveill* 2023;28:2200669. Jun 29

2. Epiconcept. I-MOVE-COVID-19. Paris: Epiconcept. https://www.epiconcept.fr/en/epidemio-project/i-move-covid-19/ (16 October 2023, date last accessed).

3. I-MOVE. I-MOVE-COVID-19. European primary care and hospital networks. https://www.imoveflu.org/i-move-covid-19/ (16 October 2023, date last accessed).

4. I-MOVE. I-MOVE | Influenza - Monitoring Vaccine Effectiveness in Europe. https://www.imoveflu.org/ (16 October 2023, date last accessed).

5. Public Health Scotland. COVID-19 European Hospital Surveillance: Draft Generic Protocol. https://www.imoveflu.org/wp-content/uploads/2021/03/14jun2020_draft generichandleresearch/I-MOVE-COVID-19_hosp_survl_v8.1.pdf (16 October 2023, date last accessed).

6. German RR, Lee LM, Horan JM, et al. Updated guidelines for evaluating public health surveillance systems: recommendations from the Guidelines Working Group. *MMWR Recomm Rep* 2001;50:1–35.

7. European Centre for Disease Prevention and Control. Field Epidemiology Manual. https://eva.ecdc.europa.eu/course/view.php?id=728 (16 October 2023, date last accessed).

8. Limesurvey GmbH. LimeSurvey: An Open Source Survey Tool. Hamburg, Germany: Lime Survey GmbH. https://www.limesurvey.org/ (16 October 2023, date last accessed).

9. Public Health Scotland. Multidisciplinary European Network for Research, Prevention and Control of the COVID-19 Pandemic. COVID-19 European Hospital Surveillance: Final Bulletin. https://www.imoveflu.org/wp-content/uploads/2022/03/DL-3.7-I-MOVE-bulletin_6th_10_03_2022_FINAL.pdf.

10. I-MOVE-COVID-19 Network. I-MOVE-COVID-19 Network: COVID-19 European Hospital Surveillance. First Surveillance Bulletin including data from all sentinel sites. Paris: Epiconcept, 2020

11. I-MOVE-COVID-19 Network. COVID-19 European Hospital Surveillance: Second Bulletin. Paris: Epiconcept, 2021.

12. I-MOVE-COVID-19 Network. COVID-19 European Hospital Surveillance: Third Bulletin. Paris: Epiconcept, 2021.

13. I-MOVE-COVID-19 Network. COVID-19 European Hospital Surveillance: Fourth Bulletin. Paris: Epiconcept, 2021.

14. I-MOVE-COVID-19 Network. COVID-19 European Hospital Surveillance: Fifth Bulletin. Paris: Epiconcept, 2021.

15. I-MOVE-COVID-19 Network. COVID-19 European Hospital Surveillance: Sixth Bulletin. Paris: Epiconcept; 2022.

16. I-MOVE-COVID-19 and Public Health Scotland. I-MOVE-COVID-19 Hospital Surveillance System: Surveillance Monitoring and Evaluation Report. https://www.imoveflu.org/wp-content/uploads/2022/03/D.3.8.-Evaluation-report-I-MOVE-hospital-surveillance_10_03_2022_FINAL.pdf (16 October 2023, date last accessed).

17. Epiconcept. Vaccine Effectiveness, Burden and Impact Studies (VEBIS) of COVID-19 and Influenza. Paris: Epiconcept, 2022. https://www.epiconcept.fr/en/epidemio-project/vaccine-effectiveness-burden-and-impact-studies-vebis-of-covid-19-and-influenza/ (16 October 2023, date last accessed).

18. European Centre for Disease Prevention and Control. Managing Heterogeneity When Pooling Data from Different Surveillance Systems. Stockholm: ECDC, 2019. https://www.ecdc.europa.eu/en/publications-data/managing-heterogeneity-when-pooling-data-different-surveillance-systems. Paris: Epiconcept.

19. Haldane V, Jung A-S, Neill R, et al. From response to transformation: how countries can strengthen national pandemic preparedness and response systems. *BMJ* 2021; 375:e067507.

20. Chua AQ, Al Knawy B, Grant B, et al. How the lessons of previous epidemics helped successful countries fight COVID-19. *BMJ* 2021;372:n486.

21. European Centre for Disease Prevention and Control. COVID-19 Surveillance Guidance - Transition From COVID-19 Emergency Surveillance to Routine Surveillance of Respiratory Pathogens. 2021. https://www.ecdc.europa.eu/en/publications-data/covid-19-surveillance-guidance (16 October 2023, date last accessed).

22. World Health Organization. End-to-End Integration of SARS-CoV-2 and Influenza Sentinel Surveillance: Revised Interim Guidance, 31 January 2022, 31 January 2022. https://www.who.int/publications/i/item/WHO-2019-nCoV-Integrated_sentinel_surveillance-2022.1 (16 October 2023, date last accessed).

23. European Centre for Disease Prevention and Control. Assessment of Electronic Health Records for Infectious Disease Surveillance. https://www.ecdc.europa.eu/en/
publications-data/assessment-electronic-health-records-infectious-disease-surveillance (16 October 2023, date last accessed).

24 Van Goethem N, Vilain A, Wyndham-Thomas C, et al. Rapid establishment of a national surveillance of COVID-19 hospitalizations in Belgium. *Arch Public Health* 2020;78:121.

25 Köster AM, Bludau A, Devcic SK, et al. Infection surveillance measures during the COVID-19 pandemic in Germany. *GMS Hyg Infect Control* 2021;16:Doc27.