# “Goldmine” or “big mess”? An interview study on the challenges of designing, operating, and ensuring the durability of Clinical Data Warehouses in France and Belgium

Sonia Priou, Emmanuelle Kempf, Marija Jankovic, Guillaume Lamé

<figure><img src="image_1.png" /></figure>

## To cite this version:

Sonia Priou, Emmanuelle Kempf, Marija Jankovic, Guillaume Lamé. “Goldmine” or “big mess”? An interview study on the challenges of designing, operating, and ensuring the durability of Clinical Data Warehouses in France and Belgium. Journal of the American Medical Informatics Association, 2024, 31 (11), pp.2699-2707. 10.1093/jamia/ocae244. hal-04697903

HAL Id: hal-04697903

https://hal.science/hal-04697903v1

Submitted on 17 Sep 2024

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# ‘Goldmine’ or ‘big mess’?

An interview study on the challenges of designing,
operating, and ensuring the durability of Clinical Data
Warehouses in France and Belgium

Sonia Priou (MSc)¹, Emmanuelle Kempf (MD, PhD)²,³, Marija Jankovic (PhD)¹, Guillaume Lamé (PhD)¹

¹Université Paris-Saclay, CentraleSupélec, Laboratoire Génie Industriel, France

²Université Paris Est Créteil, AP-HP, Department of medical oncology, CHU Henri Mondor and Albert Chenevier, Créteil, France

³Sorbonne Université, Inserm, Université Sorbonne Paris Nord, Laboratoire d'Informatique Médicale et d'Ingénierie des Connaissances pour la e-Santé, LIMICS, Paris, France

Corresponding Author: Sonia Priou, Centrale Supélec, Laboratoire de Génie industriel, 3 rue Joliot Curie, 91190 Gif-sur-Yvette, France.

E-mail: sonia.priou@centralesupelec.fr

Phone: +33 (0) 1.75.31.63.83

**Keywords:** Electronic Health Records, Clinical Data Warehouse, Real-world Data, Medical Informatics, Qualitative Research, Interviews

Word count: 4556/4000
# ABSTRACT

**Objective:** Clinical Data Warehouses (CDW) are the designated infrastructures to enable access and analysis of large quantities of Electronic Health Record (EHR) data. Building and managing such systems implies extensive "data work" and coordination between multiple stakeholders. Our study focuses on the challenges these stakeholders face when designing, operating, and ensuring the durability of CDWs for research.

**Materials and Methods:** We conducted semi-structured interviews with 21 professionals working with CDWs from France and Belgium. All interviews were recorded, transcribed verbatim, and coded inductively.

**Results:** Prompted by the AI boom, healthcare institutions launched initiatives to repurpose data they were generating for care without a clear vision of how to generate value. Difficulties in operating CDWs arose quickly, strengthened by the multiplicity and diversity of stakeholders involved and grand discourses on the possibilities of CDWs, disjointed from their actual capabilities. Without proper management of the information flows, stakeholders struggled to build a shared vision. This was evident in our interviewees' contrasting appreciations of what mattered most to ensure data quality. Participants explained they struggled to manage knowledge inside and across institutions, generating knowledge loss, repeated mistakes, and impeding progress locally and nationally.

**Discussion & Conclusion:** Management issues strongly affect the deployment and operation of CDWs. This may stem from a simplistic linear vision of how this type of infrastructure operates. CDWs remain promising for research, and their design, implementation, and operation require careful management if they are to be successful. Building on innovation management, complex systems, and organizational learning knowledge will help.
BACKGROUND AND OBJECTIVE

To enable easier access and analysis of large quantities of Electronic Health Record (EHR) data,
healthcare institutions have developed Clinical Data Warehouses (CDWs). CDWs are infrastructures
dedicated to extracting clinical data from local data sources such as Hospital Information Systems (HIS),
their transformation into a suitable format (e.g., Observational Medical Outcome Partnership —
OMOP v5.0), [1] and their access by researchers. [2]

The stories of CDW development resemble clear cases of “data-push” innovation:[3] starting
from data already being generated for operational purposes, enthused by the boom of AI and data
science, hospitals set out to generate new value streams. However, data-push projects classically
exhibit ‘unknown time to market, a high level of unknown and technology uncertainty, as well as a
high level of required R&D efforts’.[3] This is not unfamiliar in healthcare, where the dynamics of
innovation adoption or failure are notoriously complex.[4,5] Information technology (IT) projects have
sometimes been compared to “mega-projects”,[6] characterized by ambitious goals, long time frames,
over budget, and conflicting results.[7] In the case of CDWs, the goals also include a scientific
dimension. Researchers have developed frameworks to evaluate data quality,[8] and proposed
guidelines to secure high-quality research on EHR data.[9–12] There is also room for guidelines on the
management and operation of CDWs beyond technical aspects.[13] For example, some have suggested
that deeper collaboration and exchange of learning between healthcare institutions on IT projects,
particularly for CDWs, could support improvement.[14,15]

Building and managing large data systems like CDWs implies extensive “data work” in
processing and giving access to data,[16] work that is often invisible and underestimated.[17,18]
Multiple stakeholders, from data engineers to clinical researchers and data scientists, are involved in
repurposing EHR data through CDWs.[19] Although some institutions have published their experience
implementing and operating their own CDWs,[15] we still know little about the experience of the
professionals who design, operate, and exploit CDWs. In response to this gap, we ask the following
questions: what challenges do stakeholders face when designing, operating, and ensuring the
durability of CDWs, and why do these challenges emerge? Through interviews, we described and analyzed the experiences and the difficulties encountered by stakeholders involved with CDWs.

# MATERIALS AND METHODS

We sought to develop a qualitative description,[20] through semi-structured interviews with experts in EHR data reuse from France and Belgium (Box 1).

**Box 1.** CDWs and relevant health data policies in France and Belgium.

France has various types of infrastructure for data-enabled health research, including the anonymized national claims database,[21] hospital CDWs and specialized registries. Based on a study from 2022, 19 out of the 32 French regional and university hospital trusts (large, urban, public, tertiary care institutions) had an operational or experimental CDW, and five others were planning to develop one.[22] These figures can be explained by the drive of the French Ministry of Health to advance in e-health, materialized by the creation in 2019 of a national Health Data Hub that aims to simplify access to healthcare data, and by dedicating funding to build and consolidate hospital CDWs.[23]

Belgium has several infrastructure aggregating health data, including a national cancer registry and population-based reimbursement data.[24] In 2012, a national electronic health action plan was launched to improve the secondary use of healthcare data for innovation and research, resulting in a national primary care data warehouse.[25] The will to repurpose health data was strengthened by the 2018 "Data for Better Health" policy, [26] and the setup in 2023 of a federal Health Data Agency.[27]
We interviewed professionals working on the design and operation of CDWs, be they involved
in the development and maintenance of the technical infrastructure (e.g., data engineers), the
management of CDWs as organizations (e.g., CDW directors), or the exploitation of CDWs for research
(e.g., data scientists, clinical researchers). Participants had to have at least two years of experience
working with health data, and we did not exclude any type of institution (public or private hospitals,
national authorities, software providers, consultancies). Participants were recruited purposively based
on their institution, current position (e.g., clinician, data scientist, data engineer, or clinical researcher),
seniority, and initial background (medical or engineering degree); we sought diversity across these
dimensions to obtain a wide range of profiles. We identified prospective candidates through
publications and conference presentations, involvement with institutions working with EHR data, and
snowball sampling (whereby participants suggest other interesting candidates to interview).
Participants were contacted via email. We stopped recruiting participants when we reached inductive
thematic saturation (i.e., no new themes arose when analyzing new transcripts, Supplementary File 1).
All interviewees provided written informed consent to participate in the study.

The interview guide (Supplementary File 2) started with the participant's background and current position, after which we asked them to think about a project they had either conducted or been a part of that included secondary use of healthcare data. This project would be the starting point for the interview, allowing participants to focus on their experience with CDWs and data analysis, and to provide more concrete and specific answers. As in all semi-structured interviews, the conversation could stray away from the guide; in particular, interviewees could discuss other projects than the one used as the initial focus, as well as more general issues with CDWs.

The first author (SP) conducted face-to-face or online videoconferencing interviews in French or English between October 2022 and December 2023 for her doctoral research. SP has a biomedical engineering background specializing in data science and, at the time of the study, had two years of professional experience working with EHR data in France and abroad. The interviews lasted between
forty-five minutes and two hours; they were recorded and transcribed verbatim. The analysis followed conventional content analysis.[28] The first author familiarized herself with the data by listening to the tapes, and validated and anonymized the transcripts. She coded the transcripts inductively, using sensitizing concepts drawn from research into healthcare innovation, data quality assessment of real-world health data, and the principles of learning organizations. After coding five transcripts, another author (GL) checked the coding to validate the process. SP then regrouped codes into higher-level themes, which were reviewed collectively during two three-hour discussion meetings among all authors. Where relevant, information was transcribed into simple graphical models (i.e., boxes-and-arrows). Interim results were regularly presented and discussed outside the authorship team in seminars, to obtain feedback from domain experts.

Citations were translated to English by the first author. Results were reported following established guidelines for qualitative research.[29,30] This study was approved by the Paris-Saclay University Research Ethics Committee (CER-Paris-Saclay-2022-033).

## RESULTS

We interviewed 21 professionals from 15 institutions (Table 1). Participants referenced three main phases of the lifecycle of CDWs: designing a CDW, operating a CDW, and ensuring the durability of CDWs (Figure 1). They described the challenges encountered in each phase and the determinants that triggered their emergence.
<figure><img src="image_3.png" /><figcaption>**Figure 1.** Determinants of the challenges faced by participants in the three phases of CDW development. In the design phase, the CDW is not structured but many uncertainties exist on its relationships with its environment: funding, technologies, stakeholders. In the operational phase, the CDW is designed, but different types of barriers between stakeholders constrain information flows and hinder CDW functionality. When seeking to ensure the durability of the CDW, the lack of appropriate approaches for internal and cross-institutional learning and knowledge management is detrimental</figcaption></figure>
<table><caption>**Table 1.** Profile of the interviewees.</caption><thead><tr><th></th><th>Number of interviewees</th></tr></thead><tbody><tr><td><strong>Initial background</strong></td><td></td></tr><tr><td>Medical degree</td><td>7</td></tr><tr><td>Engineering degree</td><td>14</td></tr><tr><td><strong>Current position</strong></td><td></td></tr><tr><td>Clinical or epidemiological researcher<br>Participants analyzing data from epidemiological or<br>clinical perspective, including clinicians undertaking<br>research with CDWs</td><td>4</td></tr><tr><td>Medical informatics expert<br>Postgraduates in medical informatics working with<br>hospital information systems, including medical billing<br>and coding analysts</td><td>4</td></tr><tr><td>Clinical data warehouse director</td><td>2</td></tr><tr><td>Data scientist<br>Participants analyzing data with a machine learning<br>approach</td><td>7</td></tr><tr><td>Data engineer</td><td>2</td></tr><tr><td>Employee of CDW consultancy<br>In charge of product development and partnership with<br>hospitals</td><td>2</td></tr><tr><td><strong>Seniority - years working on health data</strong></td><td></td></tr><tr><td>2 - 5 years</td><td>9</td></tr><tr><td>> 5 years</td><td>12</td></tr><tr><td><strong>Country</strong></td><td></td></tr><tr><td>France</td><td>18</td></tr><tr><td>Belgium</td><td>3</td></tr><tr><td><strong>Type of clinical data warehouse</strong></td><td></td></tr><tr><td>Hospital database</td><td>14</td></tr><tr><td>National database</td><td>4</td></tr><tr><td>Regional population-based cancer registries</td><td>3</td></tr></tbody></table>

## Designing a CDW

According to participants, the development of CDWs was often conducted as an exploratory project, without a reliable business model connecting stakeholders, infrastructure, and use cases. Interviewees described the emergence of CDWs fifteen years ago, when, with increased digitalization of patient records and developments in data management, hospitals started thinking about exploiting care data.

> In 2007, with the computerization of hospitals and the instauration of electronic health records, we thought that these data, that were already being generated, we could reuse them for more than care.

Medical informatics expert (13)
CDWs seemed natural to develop and a possible strategic asset for conducting large-scale studies with
limited effort on data collection. However, at this stage, neither the use cases nor the business model
were clear.

*We gather data super quickly. We put them in this big mess. And then there are millions of data scientists that have to figure it out.*

Medical informatics expert (5)

Participants reported that hospitals would invest their own resources and obtain one-off funding through national calls for projects to build the infrastructure to aggregate all their HIS data into a single data warehouse.

Interviewees described how CDW administrators shared the ambitious vision of larger, quicker, high-quality observational studies on readily-available EHR data, and how this was often disconnected from what CDWs could deliver. For participants, having CDW development supervised by business or management graduates, with little knowledge of data science, medical research, or medicine, led to a very optimistic vision of what would be possible, and created high expectations. The 'official', corporate narrative pictured CDWs as a tool to give clinicians access to 'their' data in a structured environment.

*Our first target is not the data scientist, not the data expert. Our first target is the clinician or the researcher who doesn't really know anything about data mining but who will be able to generate the correct hypothesis and ask the right questions.*

Data engineer (12)

In practice, participants described complex regulatory procedures and questionable data quality. The number of regulatory hoops to go through before accessing data was often a first disillusion. Then, once data was accessed, interviewees reported frequent disappointments with the capabilities of CDWs. The format, quality, and effort needed to understand the data were off-putting. More often than not, clinicians were disillusioned by what CDWs and data science could accomplish. Some participants stated that projects were abandoned or that clinicians would return to old-school manual data extraction.

*When you are told "log in on this platform, you will see, it's amazing, the data is perfect, you will only need to click and then you can have a publication" and it doesn't work – especially for someone who is with patients every day and who has 10 minutes tops in their week [to do*
research], and you told them they would be able to do it in five minutes, but when it doesn't work - well they do it once, maybe twice but then it is sure that they won't come back to it.
Data engineer (12)

Data engineers and data scientists interviewed had to manage this disappointment.

It is quite difficult to shut down a research project with an investigator after having bragged about the CDW. They were very excited, spent several hours discussing the medical research question with you but in the end, you have to tell them that you won't be able to complete the study because of data quality.
Data scientist (2)

The data scientists, whose mission was to transform large quantities of clinical data into insights to help answer research questions, were also disappointed by what was available. They explained that the data did not always prove to be as "big" as anticipated, and that the content was sometimes questionable. Despite the difficulties, participants all mentioned successful research projects. Yet, they all insisted on the poor quality of the available data.

## Operating a CDW

Participants identified four main categories of stakeholders involved in operating CDWs, divided by disciplinary background (engineers and physicians) and functions (users and providers) (Table 2). Interviewees noted that one person could fit more than one profile, e.g., a clinician could also be a clinical researcher. Participants reported challenging collaboration between these stakeholders, because of several levels of barriers and misaligned expectations.

<table><caption>**Table 2.** Four main stakeholders involved in operating CDWs, according to participants.</caption><thead><tr><th rowspan="2" colspan="2"></th><th colspan="2">Disciplinary background</th></tr><tr><th>Engineers</th><th>Physicians</th></tr></thead><tbody><tr><th rowspan="2">Role</th><th>Users</th><td>Data Scientist</td><td>Clinical Researcher</td></tr><tr><th>Providers</th><td>Data Engineer<br>provider of IT infrastructure</td><td>Clinician<br>provider of data</td></tr></tbody></table>

Managing information flows across disciplines and profiles
Making a CDW work involves “[getting] together people who normally don’t work together and help[ing] them realize the perspectives they need to have over each other’s world” (Medical informatics expert (16)). In practice, participants explained that managing information flows was not always easy.

*The problem is usually that there is a lack of communication between the people in Artificial Intelligence, primarily informaticians or mathematicians, and the clinical world with medical knowledge.*

CDW director (3)

Because of different backgrounds and significant knowledge gaps, each group did not always know what to expect or ask of the other.

*For us clinicians, it is a problem. [Data science] is a black box. We have no background, no experience, no training on what research [using big data] is or what an algorithm is.*

Clinical/epidemiological researcher (14)

Participants mentioned the difference in the scientific culture between disciplines as a barrier to understanding each other. Data engineers cited the agile methodology for developing CDWs, where a product (here, data delivered to researchers) is not perfect on the first try, but quality improves with iterations and feedback from users. However, getting data that is not fully qualified was antinomic with the expectations of the clinical researchers we interviewed, who built on the experience of data obtained in the controlled environment of trials and cohorts.

*It's a very engineering way to operate, mostly a data engineer way. So, there are lots of clinicians who are not at all used to this kind of operation.*

Data engineer (21)

Organizational structures sometimes made cooperating efficiently across disciplines and functions difficult. For example, some participants explained that data engineers - who built and operated CDW infrastructure - and the people responsible for data science applications were sometimes in separate departments, with little information getting across. Interviewees explained that interfaces were not always clearly managed, and the needs and expectations of different stakeholders did not seem coordinated. For example, one clinical researcher expressed frustration that researchers did not have input into which data flows were prioritized for integration into the CDW, when they were
supposed to be the key beneficiaries of these infrastructures. Some participants attributed the lack of
a common vision to senior hospital management.

> The head managers have a literature background [...]. The problem with the people that manage us is that they don't have this [scientific] culture so they can't understand our needs and the way it should be done and how we should all interact.
> 
> CDW director (10)

Without coordination, interviewees mentioned that it was common for teams to develop and pursue their own objectives and bemoan others' lack of cooperation.

Misaligned expectations: the case of data quality

Data quality was a clear example of misalignment between stakeholders during the interviews.

Although all interviewees complained about issues with data quality, differences were evident in what they were expecting.

> [The data engineers] needed someone that uses the clinical data warehouse because they integrate the data, but they don't have the same notion of data quality [...] because they don't use the data.
> 
> Data scientist (19)

The first divergence was on what counts as ‘correct’ data. Among the physicians interviewed, diverse criteria for correct data were described: researchers measured ‘correct’ data through bias, whereas clinicians expected data to represent the patient’s condition truly.

> When somebody says: "I've got lab data on a diabetic patient", [I say:] "Ok very interesting. How much home monitoring data do you have?" [and the answer is:] "0". So, actually, what you've got is the very imperfect hospital data. [...] It's only perfect if you're looking at accuracy, calibration, and reliability. If you're looking at faithfulness to the life of the patient, it's terrible.
> 
> Medical informatics expert (16)

We also observed unaligned expectations between data engineers and data scientists. For example, the importance of the technical location of the data (i.e., the field where it is entered) was interpreted differently. The data scientists said they needed the correct value in the right box. In contrast, the data engineers focused on transferring whatever value was in the HIS to the CDW, without modifying the information in the process. Clinicians argued that they could interpret data entry mistakes, but data scientists reported that this became challenging when working with large quantities of data.
Participants also disagreed on what ‘completeness’ meant. Clinicians explained that they could
sometimes deliberately omit to include information in the patient record, possibly to wait until the
time is right to discuss it with the patient. This was not seen as poor reporting for the clinician as it was
done consciously and with the patient’s best interest in mind. However, this counted as poor data
quality for clinical researchers and data scientists. For data engineers, once again, the challenge was
not in the content of the data but in being as transparent as possible when transferring it from the HIS
to the CDW. This sometimes involved converting data into suitable formats, which is not without
problems. For example, the terminologies in common data models, such as OMOP, were not
systematically used in the original data. Mapping each local terminology to the common model’s
terminology could create completeness issues.

I keep on explaining that OMOP is not perfect, why? because it uses two terminologies that are
RxNorm for drugs, which is not used in France so not suitable for France and SNOMED-CT. We
have translated 55% of SNOMED-CT to French.

Medical informatics expert (5)

A complex information network

Aside from highlighting differences in expectations, participants also identified the knowledge held by
each stakeholder and the interactions they put in place - or *wished* to put in place – in order to share
this knowledge and make collaboration more effective. We synthesized these interactions graphically
in Figure 2.
<figure><img src="image_4.png" /><figcaption>**Figure 2.** Interactions and knowledge and information flows between CDW stakeholders</figcaption></figure>

Interviewees explained how interactions between the clinical researcher and the data scientist are crucial for the good progress of a research project. The clinical researcher initiates the research question and the research hypothesis, provides essential clinical knowledge, and validates the trends of the results based on their knowledge of the field; then, data scientists are essential to implement the right data analysis methods.

However, participants were adamant that the research team cannot function on its own. It needs to interact with the clinicians who generate the data and the data engineers who integrate it. Data engineers manage the data integration process and, therefore, are a precious source of knowledge on what is available in the CDW.

> When I can't find the data, I interact with the data manager who integrates the data into the CDW because they have the vision of what has been incorporated or not in the warehouse.
## Data scientist (19)

Clinicians know better than anyone what can be found in the data they generate. For the projects described by the interviewees, clinical input was crucial to understanding and interpreting data.

> *The coordinating clinician knows very well the data and the way the information is reported. [...] So we have this contact person who knows very well their own data and we will be able to exploit it directly with them.*

## Data engineer (12)

This was all the more true for unstructured data such as clinical notes, where the flourishing of synonyms and the evolution of terminologies beat data scientists' best efforts.

> *I was working on pathology reports and I had the impression that all [the clinicians] had literature degrees and that they enjoyed changing the formulation in each conclusion. [...] Each formulation was different, and when it comes to the negative form, they are very inventive.*

## Clinical or epidemiological researcher (6)

Participants told us how data engineers could also learn from the research team. Data engineers need researchers to understand which data should be integrated in priority, and can benefit from the feedback of data scientists to pinpoint potential data quality issues. Data engineers said they could also benefit from clinicians' knowledge of how the HIS is used in practice to integrate the data needed by the research team. Moreover, clinicians are best placed to validate the data integration flows computed by data engineers. Yet, information exchange was often suboptimal.

> *The problem is that I never hear about the needs [of data users]. We are cut off from all of it. So I hear only a few feedbacks, and it is usually in an informal way.*

## Data engineer (21)

Finally, data engineers depend entirely on HISs, but they reported getting little to no support from HIS software providers, who showed little interest in helping data engineers. Engineers are then left to explore the data structure of the HIS themselves. Care always comes first, and the needs of the CDW are treated as secondary.

## Ensuring the durability of CDWs
For participants, interactions between stakeholders were not managed systematically, and knowledge was not shared effectively within their institutions. Despite participants' interest in collaborating between institutions, most explained that cross-institution learning was still hesitant.

The first blockage stated by interviewees to managing knowledge was that it was very time-consuming to convert tacit knowledge, held by individuals who gained it through personal experience, into explicit knowledge, easily expressed and transferred between individuals.[31] Data engineers explained during the interviews that they were caught up by time and could not always document how data was processed as much as they would have liked.

*One of the problems is that documentation ideally takes longer than anybody ever has. So, you say the first compromise is time. Everybody is in some way documenting less completely, less thoroughly, less thoughtfully than they would really like to do.*

Medical informatics expert (16)

Participants described situations where the tacit knowledge was not transferred to the organization, which could not capitalize on this tacit knowledge and risked losing it with individual departures. Unfortunately, most of the institutions interviewed had difficulties securing experts in the long term.[19] Job offers were not always aligned with the market, making recruiting and keeping good candidates difficult. With limited knowledge management, every time someone left, considerable knowledge was lost.

*Experience shows that the maintenance of the algorithms is done by experts, but these experts have an average work span within the institution of 3 years, and that they leave on a day's notice, without any legacy, without transmission of their knowledge. Then, the algorithms disappear.*

Clinical or epidemiological researcher (14)

Some participants also deplored the lack of collaboration between institutions. They said there were currently few initiatives for knowledge sharing on CDWs. Every team seemed to re-start from scratch and then faced the same difficulties. As institutions usually only build one CDW, interviewees highlighted how sharing past experiences with other institutions would seem helpful. To enable this, stakeholders in some of the institutions interviewed had created networks with regular meetings to share knowledge and learn from others.
> We installed workshops for data engineers within our network and now engineers outside the network are joining. We talk about data integration process, for drugs, pharmacy, images etc. Everyone comes with their problem, and we reflect together on the best way to integrate the data.

Medical informatics expert (13)

The data engineers we interviewed really appreciated this environment of mutual help, although they highlighted that differences in IT systems got in the way of transferring approaches. Even with the same software providers, implementations could differ.

Aside from data infrastructure, knowledge sharing was also important for the data scientists interviewed. Participants cited initiatives built on routine data science practices: sharing code and algorithms, and reusing others' work. Yet, there was a general feeling of missing the tools to collaborate between institutions. Participants argued that creating and sharing guidelines, safeguards, and algorithms could benefit the community.

Finally, it is worth noting that some institutions seemed reluctant to open their systems. As one interviewee put it, "it is seen as "my" goldmine and so I protect it with everything I've got, and nobody can touch it" (clinical or epidemiological researcher (14)). For participants, these organizations believed that data was a valuable resource for them to exploit, and they used all possible tactics to prevent external access, e.g., using strict data regulations as an excuse. This behavior put CDWs at the heart of political games, which could ultimately hinder their use.

> Hospitals see data as their own [...]. There is this mentality of "it's my data, my clinical data warehouse" instead of saying "we want to build a research environment".

Consultancy (18)

# DISCUSSION

## Findings

In this study, we aimed to identify the main challenges faced by different stakeholders when designing, operating, and ensuring the durability of CDWs for research and to understand the determinants that enhance these challenges. During the design phase, CDWs faced the risk of 'mega-projects' at the scale of the institutions that launch them: high ambition, but over time, over budget, and under
expectations.[6,7] When CDWs were sold as game-changers for research but delivered much less, stakeholders often felt frustrated, and research projects could prematurely come to an end. Then, during the operating phase, CDWs suffered from miscommunication between stakeholders, leading to challenges with information flows. Finally, CDWs lacked proper knowledge management to ensure durability. Within an institution, limited internal learning could threaten the durability of a CDW. Cross-institutional learning was presented as a major asset to ensure the durability of CDWs but remained under-used. In some interviews, the implicit idea that CDWs were built around a linear model of projects, where clinical researchers would write up research questions, data engineers would provide data, and data scientists would analyze it, was evident. This was at odds with the picture of interactions, interdependencies, and feedback loops painted by interviewees.

## Interpretation

More generally, our results identify CDWs as complex sociotechnical systems that, like other digital innovations, require the alignment of many factors to reach durability;[5] in particular, a clear identification of beneficiaries, use cases, and benefits derived from the system (which frameworks like the business model and value proposition canvasses can help think through [32,33]), an organization capable of implementing and adapting, and a supportive wider environment.

Participants described that stakeholder groups struggled to build shared visions, such as in our interviewees' discussion of data quality. Participants only referred to two data quality dimensions, 'completeness' and 'correctness', the most frequently encountered in the literature.[8] Yet, participants had different conceptions and expectations even on these two dimensions. Siloed organizations hampered good communication between clinical and data experts, and even inside expert groups, preventing a shared vision from emerging—a phenomenon already documented in healthcare information technology research.[14,15,19] It was clear in the interviews that the extent to which clinical expertise is required to structure, design and operate CDWs did not receive proper attention.[17] This 'invisible work' is crucial if clinicians are to benefit from data systems.[18] However,
symmetrically, clinicians alone cannot do much without extensive methodological support.[34] These issues may stem from a misconception of research project management.

From an early stage in their development, CDWs need to be supported by organizational learning strategies. So-called ‘learning organizations’ are apt at turning tacit knowledge, primarily acquired through individual experience, into explicit knowledge that can be shared between individuals or organizations.[31] To do so, they build on systematic problem-solving, experimentation, learning from past experience, learning from others, and transferring knowledge.[35] In our interviews, CDWs typically struggled to keep a record of past experience, learn from others, and transfer knowledge between stakeholders and institutions. Improving the climate towards organizational learning is crucial if CDWs wish to prosper, but doing so will require, among other things, an adequate human resource strategy (including cross-disciplinary training),[36] internal and external communication capabilities, and a long-term vision and commitment.[37] Given funding uncertainties on CDWs in France,[22] government support will likely be necessary to support learning.

## Limitations

Our qualitative design was well-suited to the exploratory nature of this study. We interviewed a broad range of interviewees, and the semi-structured interviews allowed new topics to emerge. We focused on the challenges and difficulties of implementing and operating CDWs by interviewing intentionally diverse profiles. However, this qualitative study is not intended to be a statistically representative sample of CDW users or a representation of the overall experience of multiple stakeholders, and the interview guide encouraged participants to discuss challenges more than facilitators. This choice is in line with our research question: we sought to highlight challenges and to identify their underlying mechanisms, in order to inform improvement. Then, we only interviewed participants in two countries, and the situation may be different in countries with other healthcare systems and data environments (although similar issues, e.g. staff retention and unstable funding, are noticeable).[6,13] Finally, as the
same interviewer conducted and coded all the interviews, their past experience with EHR data could
have influenced data collection and analysis, despite our efforts to ensure trustworthiness.

## Perspectives and future work

The complex information network (Figure 2) extracted from the interviews can serve as a starting point to improve information flows between stakeholders. We hope that future research can help better understand the ‘data work’ behind care data re-use infrastructure, e.g., through in-depth case studies of specific institutions, including using the ‘positive deviance’ approach to understand what works well, [38] as opposed to laying out challenges as we did in the present study.

## CONCLUSION

Using qualitative methods, our results highlight that a linear vision of CDW design, operation and maintenance is not enough to ensure durability. A more system-oriented theoretical framework needs to be built, taking into account the specificities of CDW. CDWs remain promising infrastructures for research, and their design, implementation, and operation require careful knowledge management if they are to be successful.

## CONFLICT OF INTEREST STATEMENT

The authors have no competing interests to declare.

## FUNDING

This study did not benefit from specific funding.

## ACKNOWLEDGMENTS
The authors thank the Associate Editor and the Reviewers for their insightful comments on earlier versions of this manuscript.

REFERENCES

1 OHDSI – Observational Health Data Sciences and Informatics. https://www.ohdsi.org/ (accessed 2 October 2023)

2 Pavlenko E, Strech D, Langhof H. Implementation of data access and use procedures in clinical data warehouses. A systematic review of literature and publicly available policies. *BMC Medical Informatics and Decision Making*. 2020;20:157. doi: 10.1186/s12911-020-01177-z

3 Bordas A, Le Masson P, Weil B. Data-Push Projects and their Unique Feature: Managing with Anomalies. *2023 IEEE International Conference on Big Data (BigData)*. IEEE 2023:2377–85.

4 Greenhalgh T, Robert G, Macfarlane F, et al. Diffusion of Innovations in Service Organizations: Systematic Review and Recommendations. *The Milbank Quarterly*. 2004;82:581–629. doi: 10.1111/j.0887-378X.2004.00325.x

5 Greenhalgh T, Wherton J, Papoutsi C, et al. Beyond Adoption: A New Framework for Theorizing and Evaluating Nonadoption, Abandonment, and Challenges to the Scale-Up, Spread, and Sustainability of Health and Care Technologies. *Journal of Medical Internet Research*. 2017;19:e8775. doi: 10.2196/jmir.8775

6 Price C, Green W, Suhomlinova O. Twenty-five years of national health IT: exploring strategy, structure, and systems in the English NHS - PubMed. *Journal of the American Medical Informatics Association*. 2019;26:188–97.

7 Flyvbjerg B. What you Should Know about Megaprojects and Why: An Overview - Bent Flyvbjerg, 2014. *Project management journal*. 2014;45:6–19.

8 Declerck J, Kalra D, Stichele RV, et al. Frameworks, Dimensions, Definitions of Aspects, and Assessment Methods for the Appraisal of Quality of Health Data for Secondary Use: Comprehensive Overview of Reviews. *JMIR Medical Informatics*. 2024;12:e51560. doi: 10.2196/51560

9 Hersh WR, Weiner MG, Embi PJ, et al. Caveats for the Use of Operational Electronic Health Record Data in Comparative Effectiveness Research. *Medical Care*. 2013;51:S30–7. doi: 10.1097/MLR.0b013e31829b1dbd

10 Kohane IS, Aronow BJ, Avillach P, et al. What Every Reader Should Know About Studies Using Electronic Health Record Data but May Be Afraid to Ask. *J Med Internet Res*. 2021;23:e22219. doi: 10.2196/22219

11 Safran C, Bloomrosen M, Hammond WE, et al. Toward a National Framework for the Secondary Use of Health Data: An American Medical Informatics Association White Paper. *Journal of the American Medical Informatics Association*. 2007;14:1–9. doi: 10.1197/jamia.M2273

12 Konrad R, Zhang W, Bjarndóttir M, et al. Key considerations when using health insurance claims data in advanced data analyses: an experience report. *Health Systems*. 2020;9:317–25. doi: 10.1080/20476965.2019.1581433

13 Campion TR Jr, Craven CK, Dorr DA, et al. Understanding enterprise data warehouses to support clinical and translational research. *Journal of the American Medical Informatics Association*. 2020;27:1352–8. doi: 10.1093/jamia/ocaa089

14 Parks R, Wigand RT, Othmani MB, et al. Electronic health records implementation in Morocco: Challenges of silo efforts and recommendations for improvements. *International Journal of Medical Informatics*. 2019;129:430–7. doi: 10.1016/j.ijmedinf.2019.05.026

15 Jannot A-S, Zapletal E, Avillach P, et al. The Georges Pompidou University Hospital Clinical Data Warehouse: A 8-years follow-up experience. *International Journal of Medical Informatics*. 2017;102:21–8. doi: 10.1016/j.ijmedinf.2017.02.006
16 Bossen C, Pine KH, Cabitza F, et al. Data work in healthcare: An Introduction. *Health Informatics J*. 2019;25:465–74. doi: 10.1177/1460458219864730

17 Passi S, Sengers P. Making data science systems work. *Big Data & Society*. 2020;7:2053951720939605. doi: 10.1177/2053951720939605

18 Green S, Hillersdal L, Holt J, et al. The practical ethics of repurposing health data: how to acknowledge invisible data work and the need for prioritization. *Med Health Care and Philos*. 2023;26:119–32. doi: 10.1007/s11019-022-10128-6

19 Knosp BM, Craven CK, Dorr DA, et al. Understanding enterprise data warehouses to support clinical and translational research: enterprise information technology relationships, data governance, workforce, and cloud computing. *Journal of the American Medical Informatics Association*. 2022;29:671–6. doi: 10.1093/jamia/ocab256

20 Sandelowski M. What's in a name? Qualitative description revisited. *Research in Nursing & Health*. 2010;33:77–84. doi: 10.1002/nur.20362

21 Boudemaghe T, Belhadj I. Data Resource Profile: The French National Uniform Hospital Discharge Data Set Database (PMSI). *International Journal of Epidemiology*. 2017;46:392–392d. doi: 10.1093/ije/dyw359

22 Doutreligne M, Degremont A, Jachiet P-A, et al. Good practices for clinical data warehouse implementation: A case study in France. *PLOS Digital Health*. 2023;2:e0000298. doi: 10.1371/journal.pdig.0000298

23 HDH: Health Data Hub [Online] Available: https://www.health-data-hub.fr/.

24 Depoorter V, Vanschoenbeek K, Decoster L, et al. Linking clinical and population-based data in older patients with cancer in Belgium: Feasibility and clinical outcomes. *Journal of Geriatric Oncology*. 2023;14:101428. doi: 10.1016/j.jgo.2023.101428

25 Delvaux N, Aertgeerts B, Van Bussel JC, et al. Health Data for Research Through a Nationwide Privacy-Proof System in Belgium: Design and Implementation. *JMIR Med Inform*. 2018;6:e11428. doi: 10.2196/11428

26 Heijlen R, Crompvoets J. Open health data: Mapping the ecosystem. *DIGITAL HEALTH*. 2021;7:205520762110501. doi: 10.1177/20552076211050167

27 Saelaert1 M, Mathieu1 L, Van Hoof W, et al. Expanding citizen engagement in the secondary use of health data: an opportunity for national health data access bodies to realise the intentions of the European Health Data Space. *Arch Public Health*. 2023;81.

28 Hsieh H-F, Shannon SE. Three Approaches to Qualitative Content Analysis - Hsiu-Fang Hsieh, Sarah E. Shannon, 2005. *Qualitative health research*. ;15:1277–88.

29 O'Brien BC, Harris IB, Beckman TJ, et al. Standards for Reporting Qualitative Research: A Synthesis of Recommendations. *Academic Medicine*. 2014;89:1245–51. doi: 10.1097/ACM.0000000000000388

30 Ancker JS, Benda NC, Reddy M, et al. Guidance for publishing qualitative research in informatics. *J Am Med Inform Assoc*. 2021;28:2743–8. doi: 10.1093/jamia/ocab195

31 Nonaka I, Konno N. The Concept of “Ba”: Building a Foundation for Knowledge Creation. *California Management Review*. 1998;40:40–54. doi: 10.2307/41165942

32 Osterwalder A, Pigneur Y, Bernada G, et al. *Value Proposition Design: How to Create Products and Services Customers Want*. John Wiley&Sons. 2015.

33 Osterwalder A, Pigneur Y. *Business Model Generation: A Handbook for Visionaries, Game Changers, and Challengers*. John Wiley & Sons 2010.

34 Ioannidis JPA, Greenland S, Hlatky MA, et al. Increasing value and reducing waste in research design, conduct, and analysis. *The Lancet*. 2014;383:166–75. doi: 10.1016/S0140-6736(13)62227-8

35 Garvin DA. Building a learning organization. *Harv Bus Rev*. 1993;71:78–91.

36 Hersh WR, Hoyt RE, Chamberlin S, et al. Beyond mathematics, statistics, and programming: data science, machine learning, and artificial intelligence competencies and curricula for clinicians, informaticians, science journalists, and researchers. *Health Systems*. 2023;12:255–63. doi: 10.1080/20476965.2023.2237745
Sonia Priou, Emmanuelle Kempf, Marija Jankovic, Guillaume Lamé, "Goldmine" or "big mess"? An interview study on the challenges of designing, operating, and ensuring the durability of Clinical Data Warehouses in France and Belgium, *Journal of the American Medical Informatics Association*, 2024;, ocae244, https://doi.org/10.1093/jamia/ocae244

37 Alerasoul SA, Afeltra G, Hakala H, et al. Organisational learning, learning organisation, and learning orientation: An integrative review and framework. *Human Resource Management Review*. 2022;32:100854. doi: 10.1016/j.hrmr.2021.100854

38 Bradley EH, Curry LA, Ramanadhan S, et al. Research in action: using positive deviance to improve quality of health care. *Implementation Sci*. 2009;4:25. doi: 10.1186/1748-5908-4-25
**Supplementary File 1. Thematic saturation**

<figure><img src="image_5.png" /><figcaption>**Figure S1.** Evolution of the number of theme codes according to the encoding of interviews</figcaption></figure>
**Supplementary File 2. Semi structured interview guide**

**Introduction**

- Presentation of the interviewer's background

- Presentation the project and its main objectives

In recent years, there has been an increase interest in repurposing Real-World Data (RWD) for research, leading to the multiplication of Clinical Data Warehouses (CDWs) in France and in Belgium. As a former data-scientist researcher working on RWD extracted from a French hospital, my team and I have encountered numerous challenges when analyzing RWD stored in a CDW. In this study, we asked ourselves if these challenges were common to multiple institutions and/or to the multiple actors involved in CDWs and RWD repurposing and if so, what were their cause. To that aim, we have elaborated a qualitative study to widen our knowledge on the subject. Today, we will be following an interview guide composed of a series of open questions. These questions aim to let you (the participant) express yourselves on your personal experience of research on RWD and utilization of CDWs.

**I. Presentation**

I.1. Can you state your position within the institution you are working for? How long have you been working on data analysis in healthcare?

**II. Feedback on a research project**

*The interviewee is asked to choose a specific project he/she has been a part of. The project must imply working with real-world health data. The project can either be finished, abandoned or still in progress.*

II.1. General presentation of the project

Could you explain the project:

- What was the initial scientific question?

- What were the expected results?

- Who initiated the research (doctor, researcher, management, certification program...)?

- What is the population of interest?

- If the study is over, what are the findings?

- Were the findings useful? Did they lead to a publication?

- How long took the study?

II.2. Organisation of the project

- Could you summarise the main steps of the project, from the elaboration of the scientific question to the results?

- Who was involved in this project?

- Did you feel the need to interact with other *experts* outside of the initial collaborators?

- Did you face any challenges during the study? How did you overcome these challenges?
Examples:

- Regulatory

- Understanding the role of each partner

- Delay to gain access to the data

- Data quality: who is responsible for data quality?

- Data processing methods
- Was any part of the study abandoned?

If so, what were the reasons (technical, data, time, human...)?

- Was there a change in the initial research question because of the difficulties encountered?

## II.3. Technical aspect of the project

- What data was used for this project?

- Type

- Format (i2b2? OMOP? International standard?)

- Data sources (unique source, multiple source, national)

- How/where was the data available?

- Was data updated during the study?

- Was there a data-quality-validation phase during the project?

- If so, what KPI were used?
    (If needed: completeness, representativeness, coding quality...)

- If not, why?

- Lack of time

- Technical difficulties

- Confidence in the data

- Governance? Whose responsibility is it to guarantee data quality?

- What methods were applied on the data?

- Statistical methods

- Algorithm development

- Use of previously developed algorithms

If so, was an evaluation of the performances of the algorithm performed?

- Did you identify biases due to the provision of the data?

- If so, at which step of the project did you identify them?

- Before the project: Was any anticipated measure of correction taken?

- During the project: How were these biases taken into account? How did they affect the study?