RESEARCH ARTICLE

# Multiple criteria decision analysis for therapeutic innovations in a hemophilia care center: A pilot study of the organizational impact of innovation in hemophilia care management

Karen Beny<sup>1,2,*</sup>, Amélie Dubromel<sup>2,*</sup>, Benjamin du Sartz de Vigneulles<sup>1,*</sup>, Valérie Gay<sup>3</sup>, Florence Carrouel<sup>1</sup>, Claude Negrier<sup>4</sup>, Claude Dussart<sup>1,2</sup>

1 Health Systemic Process (P2S), Research Unit 4129, University Claude Bernard Lyon1, University of Lyon, Lyon, France, 2 Central Pharmacy, Lyon Public Hospices, Lyon, France, 3 Hemophilia Care Center, Métropole Savoie Hospital, Chambery, France, 4 Reference Center on Hemophilia and other Constitutional Hemorrhagic Diseases, Groupement Hospitalier Est, Lyon Public Hospices, Lyon, France

* These authors contributed equally to this work.

* karen.beny@chu-lyon.fr

<figure><img src="image_1.png" /></figure>

<figure><img src="image_2.png" /></figure>

OPEN ACCESS

**Citation:** Beny K, Dubromel A, du Sartz de Vigneulles B, Gay V, Carrouel F, Negrier C, et al. (2022) Multiple criteria decision analysis for therapeutic innovations in a hemophilia care center: A pilot study of the organizational impact of innovation in hemophilia care management. PLoS ONE 17(9): e0273775. https://doi.org/10.1371/journal.pone.0273775

**Editor:** Wolfgang Miesbach, University Hospital Frankfurt, GERMANY

**Received:** December 16, 2021

**Accepted:** August 16, 2022

**Published:** September 9, 2022

**Peer Review History:** PLOS recognizes the benefits of transparency in the peer review process; therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. The editorial history of this article is available here: https://doi.org/10.1371/journal.pone.0273775

**Copyright:** © 2022 Beny et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

**Data Availability Statement:** All relevant data are within the paper.

## Abstract

### Background

Several innovative drugs liable to lead to changes in healthcare organization are or soon will be available for the management of hemophilia. Analyzing their implementation can shed further light on healthcare decision-making, to anticipate changes and risk of breakdown in the patient's care pathway.

### Methods

Multiple criteria decision analysis (MCDA), based on ISPOR recommendations, was used to assess the organizational impact of innovation in hemophilia care management. The MCDA process designed for this specific context involved ten French experts in hemophilia care management (physicians, nurses, pharmacist, physiotherapist and psychologist) in the hemophilia care center of Chambéry, in the Rhône-Alpes Region of France. This pilot study involved seven steps: (i) defining the decision problem; (ii) selecting and structuring criteria; (iii) assessing the relative weight of each criterion with software-assisted simulation based on pairwise comparisons of different organizational change scenarios; (iv) measuring the performance of the selected innovations; (v) scoring alternatives; (vi) calculating aggregate scores; (vii) discussion. The endpoint was to determine the expected overall organizational impact on a 0–100 scale.

## Results

Seven organizational criteria were selected. "Acceptability for patient/caregiver/association" was the most heavily weighted. Factor VIII by subcutaneous route obtained the highest
**Funding:** The author(s) received no specific funding for this work.

**Competing interests:** The authors have declared that no competing interests exist.

aggregate score: i.e., low impact on care organization (88.8 out of 100). The innovation with strongest organizational impact was gene therapy (27.3 out of 100).

## Conclusion

This approach provided a useful support for discussion, integrating organizational aspects in the treatment decision-making process, at healthcare team level. The study needs repeating in a few years' time and in other hemophilia centers.

## Introduction

Until the turn of the 21st century, hemophilia treatment basically consisted in repeated intravenous replacement of the deficient coagulation factor [1]. For some years now, new therapies have been emerging with different, non-replacement action mechanisms, less burdensome administration route (subcutaneous), longer action duration and, in the case of gene therapy, a curative aim [2–5]. Some are real technological innovations liable to disrupt care organization and/or patients’ care pathways. Management of hemophilia, which is a rare condition, is highly structured and governed by guidelines for treatment and care organization. Patients are managed in specialized structures ensuring global multidisciplinary care. In France, the structures are hospital-based and the clotting factor concentrates are available only in hospital pharmacies. The patient’s pathway is thus highly hospital-centered; but this is expected to change, as, in France, new treatments become available in community pharmacies, closer to the patient. These less burdensome treatments and accompanying clinical improvements are liable to move care away from specialized structures and to involve new actors in patient management [6]. To maintain optimal secure care, these organizational impacts need to be studied to accompany the changes in pathway.

Introducing an innovation impacts both the organization governing it and its environment [7]. The organizational aspect is still not fully taken into account, but is emerging as one of the key aspects of the decision-making process, alongside the clinical and medico-economic aspects. Assessment of therapeutic innovations (healthcare products, acts, care organization) does not at present take account of all aspects of healthcare organization, and this can lead to irrational decisions and non-optimal use of resources. A study with 53 European hospital managers showed that organizational aspects of new treatments were considered among the most relevant information for decision-making, with safety, clinical and economic aspects [8].

This finding has led to a need for effective decision support tools to enable a systemic approach to decision-making issues [9]. According to a special task force, to identify, collect and structure information required by those making judgments to support the deliberative process, particularly in the field of health, using a multiple criterion decision analysis (MCDA) is a correct route to go down [10]. Developed in the 1970s and widely used in non-medical fields such as agriculture, energy and marketing [11–13], MCDA approaches have spread to the health field since the 2000s [14]. MCDA is especially appropriate to questions requiring a range of relevant but often heterogeneous criteria to be taken on board. It facilitates strategic and operational decision-making in fuzzy or unstable settings, identifying the decision-making problem and shedding useful light on complex issues by implementing a number of alternatives. It also has the advantage of taking account of all relevant aspects (systemic approach), in a multidisciplinary setting, stimulating exchange in a structured decision-making process and enhancing the transparency and rationality of the decisions. To cite different examples of the
value of using this methodology: the Latin American Federation of the Pharmaceutical Industry (FIFARMA) stated that MCDA should strongly be considered as a tool to support Health Technology Assessment (HTA) [15]; and MCDA has been recognized as a relevant approach by the independent expert advisory committee of the Canadian Health Ministry in the context of drug reimbursement decision-making [16]. Recently, the French National Authority for Health (HAS) published, in December 2020, a methodology guide about organizational impact map for health technology assessment based on an MCDA approach [17]. In the MCDA method, the revelation of preferences can be obtained via the discrete choice approach (DCE) or via the Potentially all pairwise rankings of all possible alternatives (PAPRIKA) approach. The DCE method compares scenarios composed of four to six criteria on average [18], whereas the PAPRIKA method compares pairs of scenarios by evaluating two criteria at a time [19]. In the DCE method, the comparison of two alternatives by evaluating more than two criteria can lead to a bias if the user concentrates on one part of the criteria. The analysis conducted is then not exhaustive and can lead to uncertain results [18]. Conversely, the PAPRIKA method allows to simplify the choice of the users. Implicit pairs are automatically eliminated, which reduces the number of pairs to be compared and avoids inconsistencies in classification and the risk of redundancy [18]. This PAPRIKA method is integrated with an online software 1000minds® [20] which has the advantage of generating a random order of comparisons thus reducing the risk of bias related to the order. Moreover, the system does not involve any additional calculation and generates the global (mean or median) and individual weights for each criterion.

In the case before us, the MCDA approach has already been used in the evaluation of the organizational impacts of a therapeutic innovation, such as in acute pain management in an emergency department [9]. Despite some limitations of modeling, this demonstration provided conclusive results. To our knowledge, only one study has used MCDA in the setting of hemophilia. Gourzoulidis et al. (2021) determined the value of prophylaxis versus on-demand treatment in hemophilia A and emicizumab versus replacement therapy in the Greek health-care setting [21].

Hemophilia management requires complex structured organization comprising specialized multidisciplinary teams with numerous interfaces with the various actors involved in patient management. Many parameters are relevant to treatment decision-making: type of hemophilia, severity, comorbidities, age, and the patient's personal experience [22]. Introducing several innovations in this context is liable to induce changes in care management organization.

This context appears appropriate to develop the MCDA methodology for the assessment of the organizational impacts of a therapeutic innovation. In our view, this seems relevant because the more complex the system, the greater the uncertainty, especially in biopsychosocial models [23]. Moreover, working to reduce uncertainty will also improve the accompaniment of innovation-induced change, and make greater use of the benefits for improved health.

Thus, we conducted a pilot study in a single hemophilia care center to evaluate the organizational impact of innovation, using MCDA.

## Materials and methods

The Scientific and Ethical Committee of Hospices Civils de Lyon IRB n° 00013204 (Lyon, France) examined the protocol registered under file number 21_322 and waived the requirement for ethics approval. According to French regulations and General Data Protection Regulation, all participants are informed and given the right to object. Since no medical or personal data was collected, informed oral consent was collected during the first meeting and documented in the meeting report.
The study methodology conformed to that of Lvovschi et al. [9]. The good practice guide-
lines of the International Society for Pharmacoeconomics and Outcomes Research (ISPOR)
for the implementation of MCDA by health professionals were followed. Quantitative MCDA
was used, as its methodology is well described, it is mainly used in the health field, and it has
the advantage of providing aggregate scores that facilitate decision-making.

Seven steps, following the ISPOR quantitative MCDA approach, were followed: (i) defining the decision problem, (ii) selecting and structuring criteria, (iii) weighting criteria, (iv) measuring performance of alternatives, (v) generating a performance score, (vi) calculating aggregate scores, (vii) and deliberation [24, 25].

Defining the decision problem

The main study objective was to assess the impact of an innovation on the organization of care management of hemophilia patients. Secondary objectives were to define the main criteria underlying this organization and to collect the opinions of those involved regarding the interest of the MCDA approach. The analysis was performed from the point of view of health professionals specializing in hemophilia. Through this procedure organizational impact scores were produced to determine the impact of an innovation on the organization of care: weak impact, requiring minor changes, or strong impact requiring considerable changes.

A multidisciplinary expert group was set up. Inclusion criteria comprised (i) involvement in patient management in the center, (ii) familiarity with care organization, and (iii) being a medical or non-medical health professional. Individuals recruited were identified following an initial study on the care pathway of patient with hemophilia [26]. Among the 11 health professionals previously identified, only the biologist declined to participate. This led to 10 inclu- sions: 3 physicians, 3 nurses, 2 pharmacists, 1 physiotherapist and 1 psychologist. The study was conducted by a research team with experience in MCDA.

The study took place in the hemophilia treatment center of Chambéry in the Auvergne-
Rhône-Alpes area of France, from January to June 2021, with 2 face-to-face meetings of 2
hours each and a remote individual work session (Fig 1).

In the first meeting, the MCDA approach and its interest for assessing the organizational impact of innovations were presented to the group, which then chose to score the organizational impact of the 6 therapeutic innovations shown in Table 1 [2, 3, 5, 27, 28].

Selecting and structuring criteria

Organizational criteria were selected by the group from a list based on a review of the litera-
ture, and grouped in 5 domains: task, surrounding, peoples, structure and technology [29].
Each expert consulted the criteria for each domain to select those that seemed relevant with
respect to the predetermined issue. If the proposed criteria did not seem adequate, they could
add one in a “free text” field. They were asked to select at least 1 per domain and not more
than 8 criteria per domain.

Criteria were validated by consensus after this individual selection stage in the first meeting.
The performance levels for each criterion was then established by consensus.

Weighting criteria

Weighting used the PAPRIKA method (*Potentially all pairwise rankings of all possible alternatives*), a method to reveal preferences integrated in an on-line software (1000minds Ltd, Dunedin, New Zealand) [19]. The principle is based on pairwise comparison between scenarios, assessing two criteria at a time and assuming all others to be equal. Before weighting a criterion, particular attention should be paid to performance level: low, intermediate, high. For the
<figure><img src="image_3.png" /><figcaption>Fig 1. Description of the MCDA process assessing the organizational impact of innovations.</figcaption></figure>

https://doi.org/10.1371/journal.pone.0273775.g001

study, each expert chose the situation they would prefer, answering the following question: "In the context of a novel hemophilia treatment, which situation would you prefer (in terms of

<table><caption>Table 1. Alternatives selected by the expert group.</caption><thead><tr><td>THERAPY</td><td>TYPE OF THERAPY</td><td>INNOVATIVE CHARACTERISTIC</td><td>ACCESS TO THERAPY</td></tr></thead><tbody><tr><td>BIVV001</td><td>Replacement (recombinant factor VIII)</td><td>Fusion protein with XTEN polypeptides to extend half-life</td><td>Clinical trial (phase III) (hemophilia A)</td></tr><tr><td>OCTA101</td><td>Replacement (recombinant factor VIII)</td><td>Subcutaneous anti-hemophilia factor</td><td>Clinical trial (phase I/II) (hemophilia A)</td></tr><tr><td>emicizumab</td><td>Non-replacement (bispecific antibodies mimicking factor VIII action)</td><td>Action mechanism: enhancing coagulation. Long half-life. Subcutaneous route</td><td>Marketed (hemophilia A including patients with inhibitors)</td></tr><tr><td>concizumab</td><td>Non-replacement (anti-TFPI monoclonal antibodies)</td><td>Action mechanism: rebalancing coagulation. Long half-life. Subcutaneous route</td><td>Clinical trial (phase III) (hemophilia A and B including patients with inhibitors)</td></tr><tr><td>fitusiran</td><td>Non-replacement (RNA interference against antithrombin)</td><td>Action mechanism: rebalancing coagulation. Long half-life. Subcutaneous route</td><td>Clinical trial (phase III) (hemophilia A and B including patients with inhibitors)</td></tr><tr><td>volactocogene roxaparvovec girooctocogene fitelparvovec fidanacogene elaparvovec etranacogene dezaparvovec</td><td>Gene therapy</td><td>Healthy gene introduced in a single administration</td><td>Clinical trial (phase III) (hemophilia A and B including patients with inhibitors)</td></tr></tbody></table>

https://doi.org/10.1371/journal.pone.0273775.t001
impact on the organization of care)?" The upper limit thus corresponded to the lowest-impact situation and the highest weighting.

An explanatory notice was drawn up to optimize use of the tool. The experts provided their individual preference via the software. The mean weighting was used to calculate the aggregate score.

## Measuring performance of alternatives

The second meeting reviewed the main characteristics of the innovations according to the current state of knowledge. The performance level of the alternatives was then assessed by consensus after a step of individual assessment.

## Scoring alternatives and calculating aggregate scores

The performance scores and aggregate score of each innovation were calculated automatically by the PAPRIKA software after entering the performance levels validated by the expert group.

The method for aggregating weightings and performance levels most often used in the health field is the weighted sum method:

$$S(a) = \sum_{i=1}^{n} w_i s_i(a)$$

where S(a) is the aggregate score, w the relative weight of the criterion, and s_i(a) the performance score for alternative A for criterion i.

# Results

## Choice and description of criteria

Eight criteria were identified as priorities in the first meeting: 2 in the “task” domain, 1 in “surrounding”, 1 in “peoples”, 1 in “structure” and 2 in “technology”.” The “structure” criterion (innovation matching network objectives) was eliminated in the assessment meeting; at care-unit level, it is rather a precondition for introducing an innovation rather than a criterion that varies with department organization depending on the innovation in question. Performance levels per criterion ranged between 3 and 4; Table 2 shows criteria and performance levels.

## Weightings of criteria

The experts made between 46 and 84 explicit comparisons in a median 14 minutes (range, 9–120 min). Those who took longest (55 and 120 minutes) did not complete the analysis, due to interruption: 1 nurse (who began again), and 1 physiotherapist; the physiotherapist’s partial analysis was not taken into account in weighting the criteria.

The 3 criteria with the strongest weightings were “acceptability for patient/care-giver/association” (weight: 31.4%), “number of emergency or unscheduled consultations” (weight: 20.3%) and “complexity of the innovation and informed choice” (weight: 16.6%). Fig 2 shows criterion weightings.

## Assessment of organizational impact of innovations

Alternatives were assessed by 4 of the experts: 1 physician, 2 nurses and 1 pharmacist. Responses varied according to profession and individual factors of understanding the criteria and knowledge of the innovation. After compiling individual results, discrepancies were discussed and consensus was reached for all innovations. The “number of emergency or
<table><caption>Table 2. Criteria for evaluation of the organizational impact.</caption><thead><tr><th>Field</th><th>Criteria</th><th>Items</th><th>Item definitions</th></tr></thead><tbody><tr><td rowspan="5">Task</td><td rowspan="4">Number of emergency or unscheduled consultations</td><td>Strong increase</td><td>Increase in number of emergency and unscheduled consultations</td></tr><tr><td>Increase</td><td>Increase in number of unscheduled consultations</td></tr><tr><td>No change</td><td>No change</td></tr><tr><td>Decrease</td><td>Decrease in number of emergency consultations</td></tr><tr><td>Strong decrease</td><td>Decrease in number of emergency and unscheduled consultations</td></tr><tr><td rowspan="3">Change in patient recruitment</td><td>Complexification</td><td>Need for additional decision criteria</td></tr><tr><td>No change</td><td>No change</td></tr><tr><td>Simplification</td><td>Innovation meeting non-covered therapeutic need</td></tr><tr><td rowspan="3">Number of pathway or interface actors</td><td>Increase</td><td>New actors along pathway</td></tr><tr><td>No change</td><td>No change</td></tr><tr><td>Decrease</td><td>Fewer actors along pathway</td></tr><tr><td rowspan="4">Medical and non-medical personnel training</td><td>Considerable</td><td>Very complex theoretical and practical training requiring development of several supports; long training (several days)</td></tr><tr><td>Classical or standard</td><td>Complex theoretical and practical training requiring development of a single support accessible to all types of health professional; duration &lt;1 day.</td></tr><tr><td>Lightened</td><td>Simple theoretical and practical training requiring no new support; duration negligible (&lt;1h)</td></tr><tr><td>None</td><td>No training required</td></tr><tr><td rowspan="4">Acceptability for patient/care-giver/ association</td><td>None</td><td>No contribution in terms of efficacy, tolerance, practicality, adherence</td></tr><tr><td>Weak</td><td>Improves 1 criterion of efficacy, tolerance, practicality, adherence</td></tr><tr><td>Moderate</td><td>Improves 1 or 2 criteria of efficacy, tolerance, practicality, adherence</td></tr><tr><td>Strong</td><td>Improves several criteria of efficacy, tolerance, practicality, adherence</td></tr><tr><td rowspan="4">Supply (product, associated devices)</td><td>Very complex</td><td>Several ordering procedures for products and devices, different supply procedures and supply times</td></tr><tr><td>Complex</td><td>One additional step, easy to include in current procedures</td></tr><tr><td>Simple</td><td>No change</td></tr><tr><td>Complex</td><td>Action mechanism hard to understand even for professionals; difficult to popularize</td></tr><tr><td rowspan="3">Complexity of innovation and informed choice</td><td>Very complex</td><td>Action mechanism hard to understand even for professionals; difficult to popularize</td></tr><tr><td>Complex</td><td>Action mechanism more difficult to popularize</td></tr><tr><td>Simple</td><td>Action mechanism easy to explain</td></tr></tbody></table>

https://doi.org/10.1371/journal.pone.0273775.t002

unscheduled consultations” criterion had the same score for all innovations (13.3 points) except for gene therapy (18.7 points). This was accounted for by the small number of emergency or non-scheduled consultations in everyday practice, leaving little room for improvement on this criterion. Treatments were presumed to be effective in the light of the state of knowledge at the time of the study. Thus, no deterioration in this criterion was expected. Responses to the “supply” criterion revealed a need for better definition, taking account of the laboratory supplying the treatment. For treatments still at the stage of clinical trials, supply modalities were difficult to determine. Obviously, the experts assessed this criterion taking account of the laboratory’s expertise: hindsight on the knowledge of the pathology and patients’ needs in terms of associated devices and the ability to set up emergency supply routes. One innovation could not be assessed on all criteria, as the experts lacked knowledge of the treatment. The 2 innovations with the highest aggregate scores (i.e., low impact on care organization) were subcutaneous recombinant anti-hemophilia factor (87/100) and very long action recombinant anti-hemophilia factor (88.8/100). Low scores (i.e., strong impact) were attributed to gene therapy (27.3/100) and anti-antithrombin RNA (21.9/68.3); their acceptability was deemed low and the innovation was considered difficult to explain in securing informed patient choice. Table 3 shows aggregate scores per innovation.
<figure><img src="image_4.png" /><figcaption>Fig 2. Mean and individual weighting per criterion.</figcaption></figure>

https://doi.org/10.1371/journal.pone.0273775.g002

Discussion

This study was conducted in the field of hemophilia care. And as it took place in France, some local conditions and standards of care must be taken into account. For example, while France has implemented all 10 principles of hemophilia care (while other European countries have not), it is recognized that various aspects of the French hemophilia patient's care need to be improved (aging, psychosocial support, etc.) [27, 30, 31]. All the more so with regard to the therapeutic innovations that are coming.

Therefore, this study shows that the use of a structured analysis method helps to formalize the essential organizational criteria in the management of a pathology and to become aware of the possible impacts in the event of a breakthrough innovation in therapies. Thus, therapeutic innovations can induce organizational innovations, and recognizing this early and precisely can make it possible to prepare adequate support for change for care teams.

Then, the MCDA method, used in this study, allowed an expert group to attribute organizational impact scores for each study innovation. The method assessed the organizational impact

<table><caption>Table 3. Organizational impact scores per innovation.</caption><thead><tr><th></th><th>BIVV001</th><th>OCTA101</th><th>emicizumab</th><th>concizumab</th><th>fitusiran</th><th>gene therapy</th></tr></thead><tbody><tr><td>Acceptability</td><td>31.4</td><td>31.4</td><td>31.4</td><td>19.3</td><td>8.6</td><td>8.6</td></tr><tr><td>Number of consultations</td><td>13.3</td><td>13.3</td><td>13.3</td><td>13.3</td><td>13.3</td><td>18.7</td></tr><tr><td>Complexity</td><td>16.6</td><td>16.6</td><td>8.1</td><td>8.1</td><td>0.0</td><td>0.0</td></tr><tr><td>Supply</td><td>11.7</td><td>11.7</td><td>6.1</td><td>11.7</td><td>0.0*</td><td>0.0</td></tr><tr><td>Training</td><td>7.7</td><td>7.7</td><td>7.7</td><td>7.7</td><td>0.0*</td><td>0.0</td></tr><tr><td>Recruitment</td><td>3.9</td><td>5.7</td><td>0.0</td><td>0.0</td><td>0.0*</td><td>0.0</td></tr><tr><td>Number of actors</td><td>2.4</td><td>2.4</td><td>0.0</td><td>0.0</td><td>0.0*</td><td>0.0</td></tr><tr><td>Aggregate score</td><td>87</td><td>88.8</td><td>66.6</td><td>60.1</td><td>21.9</td><td>27.3</td></tr></tbody></table>

https://doi.org/10.1371/journal.pone.0273775.t003

<footnote>* In the absence of sufficient data in the current state of knowledge</footnote>
of innovations and identified useful points for reflection for a subsequent more qualitative
analysis of the nature of such organizational changes. An innovation that is complex to explain
may require setting up new explanatory tools, impacting care-unit organization. In contrast,
innovations consisting in replacement showed the least organizational impact, although this is
to be read in the context of the study situation and cannot be directly extrapolated to other
hemophilia centers. Although hemophilia management is highly structured and coordinated,
there may be organizational differences between centers according to their geographic situa-
tion and history. It would be useful to repeat the present study in other centers, to analyze vari-
ations in choice of relevant organizational criteria and possible differences in impact
according to care organization and geographical region. This could complement pathway anal-
ysis, with optimization according to organizational specificities, and also a greater consider-
ation of complementary actors, like city caregivers. For example, in France, since June 15,
2021 (after this study), the pharmaceutical product emicizumab has had a dual dispensing cir-
cuit: hospital pharmacy and city pharmacy. This is the first hemophilia treatment available to
patients in private practices. This dispensing is accompanied by additional risk-reduction mea-
sures, like a new organizational chart [32], an optimized information flow and a full training
on the disease and treatments.

A better understanding of the nature and strength of the organizational impact of a thera-
peutic innovation should allow for a better match between therapeutic choice and individual's
health determinants (biological characteristics, family environment, local health system, cul-
tural context, etc.), to understand the most suitable patient support, and to ensure an optimal
healthcare pathway, reducing the risk of dropouts.

From the point of view of the experts, the acceptability for patients/caregiver/association,
the number of emergency and unscheduled consultations and the complexity of the innova-
tion were the main criteria determining organizational impact. Gene therapy and anti-anti-
thrombin RNA were the innovations with the greatest organizational impact, notably due to
poor acceptability and high complexity hindering shared informed decision-making. Accept-
ability is a multifactorial criterion that may be related more to the patient and their experience
and life than to the innovation itself. Thus, for example, not taking into account a good match
between individual capacity for acceptability and the complexity of the innovation, which may
lead to difficulties in understanding its usefulness or usability, could lead to a risk of poor ther-
apeutic adherence. In this case, having already planned a specific work to increase the patient's
level of health literacy could be a key success factor. Poor therapeutic adherence could also
result from supply difficulties inherent to a therapeutic innovation if these were not clearly
envisaged beforehand.

At patient level, individual variation in organizational impact score can be expected due to
the strong weighting of this criterion. In a qualitative study in 20 hemophilia patients, 40% were
very favorable and 35% favorable toward gene therapy, after information delivered ahead of the
interview [33]. This study underlined similarities in the organizational impact from gene ther-
apy and anti-antithrombin RNA related to their mechanism of action. It was estimated as very
complex by this expert group. The organizational impact of these two therapeutic innovations
differed in terms of the number of emergency or unscheduled consultations. This criterion was
considered favorable only for gene therapy because of its effectiveness in treating hemophilia.
New tools or process need to be implemented in order to bring appropriate information at
patient for shared-decision. This need is not specific to this team since other studies stressed the
need to develop means of training or communication to facilitate shared decision-making [34-
37], in agreement with the present expert group's opinion regarding the importance of informa-
tion for shared decision-making in hemophilia management. In this sense, it has already been
demonstrated elsewhere that gene therapy brings a different therapeutic value chain than
traditional pharma, and involves new capabilities and requirements [38]. It calls for a rethink of current collaboration: additional roles of a hemophilia treatment centers could be implemented, and ‘hub-and-spoke’ models are thought out (prescription and management in national hubs; monitoring in spokes centers) [39]. This makes it crucial to educate patient and health care personnel, essential to define new quality and safety criteria [38].

The present study contributes to the early phase of assessment of MCDA in organizational impact in the health field. The MCDA method aims to facilitate the identification of the optimal solution [40]. It is particularly appropriate for problems that require taking into account a set of relevant and often heterogeneous criteria. It facilitates strategic or operational decision-making in an imprecise or unstable environment, objectifies a decision-making problem, and analyzes complex problems by implementing several possible alternatives. This method has several advantages. It allows (i) to take into account all relevant aspects (systemic approach), (ii) to be part of a multidisciplinary approach, (iii) to stimulate exchanges within the framework of a structured decision-making process, and (iv) to improve the transparent and reasoned character of decision-making. However, the MCDA method has several drawbacks [24, 41]. First, sufficient material and human resources (time, expertise, commitment) must be available and planned. In addition, training and familiarization of professionals are prerequisites for the efficient implementation of these methods. On the other hand, the limitations of multi-criteria methods are mainly related to reliability (accurate and complete data), variability and level of evidence of the input data, and structural uncertainty, such as disagreement on the weighting method. In this regard, several limitations are to be borne in mind.

The composition of the group immediately follows the starting point of any multi-criteria method, the definition of the problem may present biases in this study. It has an impact on the choice of alternatives, criteria and performance levels that structure the problem. Moreover, the group, composed of individuals from different backgrounds, is at the origin of the weighting of the criteria and their performance levels. Variations are expected; the aggregation of individual results with or without (average value) group consensus allows participants to express their point of view. Its composition is a crucial element of the multi-criteria method [10]. It is a source of heterogeneity which must be controlled via sensitivity analysis (study of subgroups, exploration of uncertainties and disagreements with other groups) [24].

A single workshop with participants may not be sufficient to ensure a representative evaluation, and multiple workshops or surveys may be necessary [24]. In this exploratory work, two workshops were held and the individual preference revelation step was conducted remotely.

Regarding the number of experts in the group, two other studies were performed with the same method, comprising 8 and 7 experts respectively [9, 40]. However, studies need repeating to test the method in larger groups so as to resolve issues of heterogeneity and assess varied alternatives.

No patients were included in the group, although patients play a growing role in decisions affecting their health. Including them in the decision-making process is coherent with the development of pathway-based approaches and of interactions with professionals to achieve transverse dialogue. The present study will be followed up by a qualitative study in patients regarding their perception of the impact on their care organization [42]. Finally, training participants is essential, and in the present study the experts received no specific training, although the MCDA method was presented at the outset of the study.

Regarding the PAPRIKA weighting method, several authors reported that users could find it difficult to decide about certain scenarios. They find it difficult to choose between two alternatives while setting aside important criteria not involved in the proposed scenarios. The present expert group did not raise this issue. Goetghebeur et al. also reported that some users found it hard to express their own point of view rather than projecting into an institutional or
social perspective. The perspective in the present study was that of the treatment center. PAPRIKA is an ordinal method, more robust than those used in grading scales for decision aid such as EVIDEM (Evidence and Value: Impact on Decision Making) or Matrix4Value® [43-45]. It avoids several biases inherent to grading scales and makes comparison of weightings more reliable. The problem of individual perception of the scale is circumvented. And comparing two alternatives strictly in terms of two criteria prevents the user concentrating on only one part of the criteria.

The individual preferences expressed by the experts were seen as homogeneous by the group as a whole. However, individual weightings were closer for the criterion with the greatest weight (acceptability). Other authors likewise reported comparable weightings between assessors for the criteria judged more important; in the present study, individual weightings for criteria judged to be of intermediate importance (training and supply) were more divergent. This was also found by Goetghebeur et al. and Martelli et al. According to Goetghebeur et al., divergences in weighting corresponded to varying individual perceptions and representations, and were not an obstacle; rather, they contribute to the exchanges generating consensus. These methods are interesting as they reveal disagreements between participants.

The same experts determined the weightings of the criteria and assessed the alternatives on these criteria. Other authors proceeded in the same way; however, Sampietro-Colom. et al. suggested that this could introduce bias. The Matrix4value® method uses two groups of experts; but this requires having sufficient human resources.

A question arises concerning maintaining non-discriminating criteria in assessing alternatives. In the present study, number of consultations was non-discriminating between the innovations with the exception of gene therapy. However, these criteria still have an impact. Increasing the number of performance levels for these criteria could confirm or discount their non-discriminating character. Moreover, in case of doubt regarding a difference in performance between several alternatives for a given criterion, the same score was attributed, and the criterion became non-discriminating, without, however, being canceled. Opinions differ on this point. Martelli et al. and Sampietro-Colom et al. attributed a score of zero in case of doubt, whereas Goetghebeur et al. attributed a low score in the EVIDEM model.

No sensitivity analysis was performed in the present study. Our MCDA model had several sources of uncertainty, affecting all components [24, 41]. Structural uncertainty was presumed to be low, as the weighting method and criterion value tree were established consensually. Uncertainty mainly came from using expert opinions as entry data. If a performance measure had been used, the choice between introducing an uncertainty criterion or an uncertainty analysis could have been discussed [24]. Oliveira et al. reported that different approaches to uncertainty analysis are used in MCDA to assess health technologies [46]. Moreover, the present low level of structural uncertainty could be confirmed by scenario analysis [24]. Finally, heterogeneity of preference could be studied by calling upon other expert groups [24].

In the absence of any prior assessment of the impact of the selected innovations on the organization of hemophilia care, no comparison could be made between a previous assessment process and the present method, as proposed by Martelli et al. and Goetghebeur et al. [43, 44]. Simulations could be set up to this end.

## Conclusion

The interest of studying organizational impact is now recognized and including it in health technology assessment is a goal for the coming years. The idea of cost-effectiveness is now well understood by decision-makers. However, what choice should be made about an innovation of clear clinical efficacy but requiring changes in pathway organization and relations between
actors? The distinction between medico-economic and organizational evaluation needs to be
retained; but the study of organizational impact can enrich medico-economic evaluation. The
question of level of evidence arises for the organizational dimension. Moreover, taking organi-
zational impact into account complexifies the evaluation of innovations. Methods must be
chosen that respect a balance between simplicity of implementation and methodological rigor.
It is also important to decide at what point in the cycle of an innovation organizational impact
can most usefully be assessed.

Acknowledgments

We acknowledge the members of the hemophilia treatment center from Chambery, especially Isabelle, Laurence, Odile, V. Favre, B. Steng, Dr Sylvestre, Dr Jailler and Dr Meyer for their participation in this study. We would like to thank Mr Iain McGill for his help in manuscript preparation.

Author Contributions

**Conceptualization:** Karen Beny, Amélie Dubromel, Benjamin du Sartz de Vigneulles, Claude Dussart.

**Data curation:** Karen Beny, Amélie Dubromel, Benjamin du Sartz de Vigneulles.

**Formal analysis:** Karen Beny, Amélie Dubromel, Benjamin du Sartz de Vigneulles.

**Investigation:** Karen Beny, Amélie Dubromel, Benjamin du Sartz de Vigneulles, Valérie Gay.

**Methodology:** Karen Beny, Amélie Dubromel, Benjamin du Sartz de Vigneulles, Claude Dussart.

**Project administration:** Benjamin du Sartz de Vigneulles.

**Resources:** Valérie Gay, Claude Dussart.

**Supervision:** Claude Dussart.

**Validation:** Claude Negrier.

**Writing – original draft:** Karen Beny, Amélie Dubromel, Benjamin du Sartz de Vigneulles.

**Writing – review & editing:** Valérie Gay, Florence Carrouel, Claude Negrier, Claude Dussart.

References

1. Srivastava A, Brewer AK, Mauser-Bunschoten EP, Key NS, Kitchen S, Llinas A, et al. Guidelines for the management of hemophilia. Haemophilia. 2013; 19(1):e1–47. https://doi.org/10.1111/j.1365-2516.2012.02909.x PMID: 22776238

2. Croteau SE, Wang M, Wheeler AP. 2021 clinical trials update: Innovations in hemophilia therapy. American Journal of Hematology. 2021; 96(1):128–44. https://doi.org/10.1002/ajh.26018 PMID: 33064330

3. Butterfield JSS, Hege KM, Herzog RW, Kaczmarek R. A Molecular Revolution in the Treatment of Hemophilia. Molecular Therapy. 2020 Apr 8; 28(4):997–1015. https://doi.org/10.1016/j.ymthe.2019.11.006 PMID: 31843450

4. Mahlangu J, Cerquiera M, Srivastava A. Emerging therapies for haemophilia—Global perspective. Haemophilia. 2018 May; 24 Suppl 6:15–21. https://doi.org/10.1111/hae.13510 PMID: 29878661

5. Mancuso ME, Mahlangu JN, Pipe SW. The changing treatment landscape in haemophilia: from standard half-life clotting factor concentrates to gene editing. The Lancet. 2021 Feb 13; 397(10274):630–40. https://doi.org/10.1016/S0140-6736(20)32722-7 PMID: 33460559

6. Hart DP, Kessler CM, Aledort L. Re-personalization and stratification of hemophilia care in an evolving treatment landscape. Hematology. 2019 Dec; 24(1):737–41. https://doi.org/10.1080/16078454.2019.1687798 PMID: 31711380
7. Fery-Lemonnier E. [Research and innovation: nine ways for the successful introduction of medical innovations]. Rev hosp Fr (Paris). 2003;(493):25-31. French.

8. Kidholm K, Ølholm AM, Birk-Olsen M, Cicchetti A, Fure B, Halmesmäki E, et al. Hospital managers' need for information in decision-making-An interview study in nine European countries. Health Policy. 2015 Nov 1; 119(11):1424-32. https://doi.org/10.1016/j.healthpol.2015.08.011 PMID: 26362086

9. Lvovschi VE, Maignan M, Tazarourte K, Diallo ML, Hadjadj-Baillot C, Pons-Kerjean N, et al. Multiple criteria decision analysis approach to consider therapeutic innovations in the emergency department: The methoxyflurane organizational impact in acute trauma pain. PLOS ONE. 2020 Apr 15; 15(4):e0231571. https://doi.org/10.1371/journal.pone.0231571 PMID: 32294125

10. Thokala P, Duenas A. Multiple Criteria Decision Analysis for Health Technology Assessment. Value in Health. 2012 Dec 1; 15(8):1172-81. https://doi.org/10.1016/j.jval.2012.06.015 PMID: 23244821

11. Gésan-Guiziou G, Alaphilippe A, Aubin J, Bockstaller C, Boutrou R, Buche P, et al. Diversity and potentiality of multi-criteria decision analysis methods for agri-food research. Agron Sustain Dev. 2020 Nov 16; 40(6):44.

12. Steinhilber S, Geldermann J, Wietschel M. Renewables in the UE after 2020: a multi-criteria decision analysis in the context of the policy formation process. EURO J Decis Process. 2016; 4:119-155.

13. Denguir-Rekik A, Montmain J, Mauris G. A possibilistic-valued multi-criteria decision-making support for marketing activities in e-commerce: Feedback Based Diagnosis System. European Journal of Operational Research. 2009; 195 (3):876-888.

14. Diaby V, Campbell K, Goeree R. Multi-criteria decision analysis (MCDA) in health care: a bibliometric analysis. Operations research for health care. 2013; 2(1-2):20-4.

15. Drake JI, Hart JCT de, Monleón C, Toro W, Valentim J. Utilization of multiple-criteria decision analysis (MCDA) to support healthcare decision-making FIFARMA, 2016. Journal of Market Access & Health Policy. 2017 Jan 1; 5(1):1360545. https://doi.org/10.1080/20016689.2017.1360545 PMID: 29081919

16. Laba T-L, Jiwani B, Crossland R, Mitton C. Can multi-criteria decision analysis (MCDA) be implemented into real-world drug decision-making processes? A Canadian provincial experience. International Journal of Technology Assessment in Health Care. 2020 Aug; 36(4):434-9. https://doi.org/10.1017/S0266462320000525 PMID: 32762789

17. Organisational impact map for health technology assessment [Internet]. Haute Autorité de Santé; 2020 [cited 18 juill 2021]. Available from: https://www.has-sante.fr/upload/docs/application/pdf/2021-04/organisational_impact_map_for_health_technology_assessment.pdf

18. Sullivan T. Using MCDA (Multi-Criteria Decision Analysis) to prioritise publicly-funded health care. [Thèse de docorat]. [Dunedin, New Zealand]: University Of Otago; 2012.

19. Hansen P, Ombler E. A New method For scoring Additive multi-attribute value models using pairwise rankings of alternatives. J Multi-Criteria Decis Anal. 2008; 15(3-4):87-107.

20. 1000minds [Internet]. [cité le 9 août 2017]. Disponible à: https://www.1000minds.com

21. Gourzoulidis G., Stefanou G., Economou M. et al. Application of Multicriteria Decision Analysis to Determine the Value of Prophylaxis Relative to On-Demand Treatment in Hemophilia A and Emicizumab versus Replacement Therapy in the Greek Healthcare Setting. Clin Drug Investig. 2022; 42:75-85. https://doi.org/10.1007/s40261-021-01108-4 PMID: 34874542

22. Srivastava A, Santagostino E, Dougall A, Kitchen S, Sutherland M, Pipe SW, et al. WFH Guidelines for the Management of Hemophilia, 3rd edition. Haemophilia. 2020; 26(S6):1-158. https://doi.org/10.1111/hae.14046 PMID: 32744769

23. Mick G, Gandon T, Poutout G. [Chronic pain, a complex situation model]. Douleur analg. 2017 July 11. https://doi.org/10.1007/s11724-017-0497-4

24. Thokala P, Devlin N, Marsh K, Baltussen R, Boysen M, Kalo Z, et al. Multiple Criteria Decision Analysis for Health Care Decision Making - An Introduction: Report 1 of the ISPOR MCDA Emerging Good Practices Task Force. Value Health J Int Soc Pharmacoeconomics Outcomes Res. 2016; 19(1):1-13. https://doi.org/10.1016/j.jval.2015.12.003 PMID: 26797229

25. Marsh K, Lanitis T, Neasham D, Orfanos P, Caro J. Assessing the Value of Healthcare Interventions Using Multi-Criteria Decision Analysis: A Review of the Literature. PharmacoEconomics. 2014 Apr 1; 32(4):345-65. https://doi.org/10.1007/s40273-014-0135-0 PMID: 24504851

26. Beny K., du Sartz de Vigneulles B., Carrouel F., Bourgeois D., Gay V., Negrier C., et al. (2022). Haemophilia in France: Modelisation of the Clinical Pathway for Patients. Int J Environ Res Public Health. 2022 Jan 6; 19(2):646. https://doi.org/10.3390/ijerph19020646 PMID: 35055467

27. Franchini M, Marano G, Pati I, Veropalumbo E, Vaglio S, Pupella S, et al. Investigational drugs to treat hemophilia. Expert Opinion on Investigational Drugs. 2020 Mar 3; 29(3):295-301. https://doi.org/10.1080/13543784.2020.1722999 PMID: 32008381
28. Nogami K, Shima M. New therapies using nonfactor products for patients with hemophilia and inhibitors. Blood. 2019 Jan 31; 133(5):399-406. https://doi.org/10.1182/blood-2018-07-820712 PMID: 30559263

29. Dubromel A, Duvinage-Vonesch M-A, Geffroy L, Dussart C. Organizational aspect in healthcare decision-making: a literature review. Journal of Market Access & Health Policy. 2020 Jan 1; 8(1):1810905. https://doi.org/10.1080/20016689.2020.1810905 PMID: 32944200

30. Noone D.; O'Mahony B.; Peyvandi F.; Makris M.; Bok A. Evolution of Haemophilia Care in Europe: 10 Years of the Principles of Care. Orphanet J. Rare Dis. 2020, 15, 184. [CrossRef] https://doi.org/10.1186/s13023-020-01456-y PMID: 32660500

31. O'Mahony B.; Noone D.; Giangrande P.L.F.; Prihodova L. Haemophilia Care in Europe: A Survey of 19 Countries. Haemophilia 2011, 17, 35-40. [CrossRef] https://doi.org/10.1111/j.1365-2516.2010.02362.x PMID: 20722746

32. Chamouard V. Negrier C. Cabon M, et al. Schéma de coordination entre les intervenants pour l'accompagnement de la dispensation d'emicizumab en pharmacie d'officine. MHEMO. 2021 Jun 8. Available from: https://mhemo.fr/actualites/accompagnement-de-linstauration-dun-double-circuit-dispensation-demicizumab-en-pharmacie-de-ville-et-a-lhopital/. French.

33. van Overbeeke E, Michelsen S, Hauber B, Peerlinck K, Hermans C, Lambert C, et al. Patient perspectives regarding gene therapy in haemophilia: Interviews from the PAVING study. Haemophilia. 2021 Jan; 27(1):129-136. https://doi.org/10.1111/hae.14190 PMID: 33161616

34. Hart DP, Branchford BR, Hendry S, Ledniczky R, Sidonio RF, Négrier C, et al. Optimizing language for effective communication of gene therapy concepts with hemophilia patients: a qualitative study. Orphanet J Rare Dis. 2021 Dec; 16(1):1-16.

35. Miesbach W, O'Mahony B, Key NS, Makris M. How to discuss gene therapy for haemophilia? A patient and physician perspective. Haemophilia. 2019 Jul; 25(4):545-57. https://doi.org/10.1111/hae.13769 PMID: 31115117

36. van Balen EC, Krawczyk M, Gue D, Jackson S, Gouw SC, van der Bom JG, et al. Patient-centred care in haemophilia: Patient perspectives on visualization and participation in decision-making. Haemophilia. 2019 nov; 25(6):938-45. https://doi.org/10.1111/hae.13830 PMID: 31531924

37. Srivastava S. How Cell And Gene Therapy Is Transforming Healthcare. Cellandgene. 2020 Feb 4. Available from: https://www.cellandgene.com/doc/how-cell-and-gene-therapy-is-transforming-healthcare-0001

38. Miesbach W, Chowdary P, Coppens M, et al. Delivery of AAV-based gene therapy through haemophilia centres-A need for re-evaluation of infrastructure and comprehensive care: A Joint publication of EAHAD and EHC. Haemophilia. 2021; 27: 967-73. https://doi.org/10.1111/hae.14420 PMID: 34553460

39. Athanasopoulos A. Statement on Gene Therapies Hub-and-Spoke Model. EAHAD-EHC. 2020 May 20. Available from: https://eahad.org/ehad-ehc-statement-on-promoting-hub-and-spoke-model-using-gene-therapies/

40. Dubromel A. [Taking into account the organizational impact of a health innovation: example of a clinical pathway for a diabetic patient with macular edema] [dissertation]. Lyon (FR): Claude Bernard Lyon 1 University; 2017. Available from: https://www.sudoc.fr/233290486. French.

41. Marsh K, IJzerman M, Thokala P, Baltussen R, Boysen M, Kaló Z, et al. Multiple Criteria Decision Analysis for Health Care Decision Making—Emerging Good Practices: Report 2 of the ISPOR MCDA Emerging Good Practices Task Force. Value in Health. 2016 Mar 1; 19(2):125-37. https://doi.org/10.1016/j.jval.2015.12.016 PMID: 27021745

42. Beny K, du Sartz de Vigneulles B, Chamouard V, Guilloux R, Gay V, Negrier C, et al. Patients' Perception of the Impact of Innovation on Hemophilia Care Management Organization: A Qualitative Study Protocol (INNOVHEMO Study). Patient Prefer Adherence. 2021 Aug 18; 15:1807-15. https://doi.org/10.2147/PPA.S322531 PMID: 34434044

43. Martelli N. [Assessment of innovative medical devices in university hospitals with a view to their acquisition: inventory and development of a decision support tool]. Thesis, University Paris Sud—Paris XI; 2015. Available from: https://tel.archives-ouvertes.fr/tel-01158923

44. Goetghebeur MM, Wagner M, Khoury H, Levitt RJ, Erickson LJ, Rindress D. Evidence and Value: Impact on Decision-Making—the EVIDEM framework and potential applications. BMC Health Services Research. 2008 Dec 22; 8(1):270. https://doi.org/10.1186/1472-6963-8-270 PMID: 19102752

45. Sampietro-Colom L, Morilla-Bachs I, Gutierrez-Moreno S, Gallo P. Development and test of a decision support tool for hospital health technology assessment. Int J Technol Assess Health Care. 2012 Oct; 28 (4):460-5. https://doi.org/10.1017/S0266462312000487 PMID: 23062518

46. Oliveira MD, Mataloto I, Kanavos P. Multi-criteria decision analysis for health technology assessment: addressing methodological challenges to improve the state of the art. Eur J Health Econ. 2019 Aug; 20 (6):891-918. https://doi.org/10.1007/s10198-019-01052-3 PMID: 31006056