# User Feedback on the MSF Tele-Expertise Service After a 4-Year Pilot Trial – A Comprehensive Analysis

Laurent Bonnardot, Elizabeth Wootton, Joanne Liu, Olivier Steichen, Jean-Hervé Bradol, Christian Hervé, Richard Wootton

► To cite this version:

Laurent Bonnardot, Elizabeth Wootton, Joanne Liu, Olivier Steichen, Jean-Hervé Bradol, et al. User Feedback on the MSF Tele-Expertise Service After a 4-Year Pilot Trial – A Comprehensive Analysis. Frontiers in Public Health, 2015, 3, pp.257. 10.3389/fpubh.2015.00257. hal-01290118

HAL Id: hal-01290118

https://hal.sorbonne-universite.fr/hal-01290118

Submitted on 17 Mar 2016

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# User Feedback on the MSF Tele-Expertise Service After a 4-Year Pilot Trial – A Comprehensive Analysis

Laurent Bonnardot<fnref n="1" />,²*, Elizabeth Wootton³, Joanne Liu⁴,⁵, Olivier Steichen⁶,⁷,⁸, Jean-Hervé Bradol⁹, Christian Hervé² and Richard Wootton¹⁰,¹¹

## OPEN ACCESS

**Edited by:**
Harshad Thakur,
Tata Institute of Social Sciences, India

**Reviewed by:**
Stephan Theiss,
Heinrich Heine University Düsseldorf,
Germany
Charles D. Treser,
University of Washington School of
Public Health, USA
Deborah Elaine Seale,
Saint Louis University, USA

*Correspondence:
Laurent Bonnardot
lbonnardot@gmail.com

**Specialty section:**
This article was submitted to Public Health Education and Promotion, a section of the journal Frontiers in Public Health

**Received:** 12 July 2015
**Accepted:** 02 November 2015
**Published:** 20 November 2015

**Citation:**
Bonnardot L, Wootton E, Liu J, Steichen O, Bradol J-H, Hervé C and Wootton R (2015) User Feedback on the MSF Tele-Expertise Service After a 4-Year Pilot Trial – A Comprehensive Analysis. *Front. Public Health* 3:257.
doi: 10.3389/fpubh.2015.00257

We surveyed all users of the Médecins Sans Frontières (MSF) tele-expertise service, approximately four years after it began operation. The survey contained 50 questions and was sent to 294 referrers and 254 specialists. There were 163 responses (response rate 30%). There were no significant differences between the responses from French and English users, so the responses were combined for subsequent analysis. Most of the responders were doctors (133 of 157 who answered that question), and most had completed field missions for MSF, i.e., both specialists and referrers. The majority stated that the system was user friendly and that they found it self-explanatory (i.e., they did not need to be shown how to use it). Almost all the referrers found that the telemedicine advice that they received was helpful, changed diagnosis and management, and/or reassured the patient. Similar feedback came from the specialists, who also felt that there was educational value for the field doctor. Although there was general satisfaction with the service, the survey identified various problems. The main concerns of the referrers were the lack of promotion of the system at headquarters' level, and the main concerns of the specialists were the lack of feedback about patient follow-up. Nonetheless, both referrers and specialists recognized the benefits of telemedicine in improving patient management, providing education, and reducing isolation in the field.

**Keywords:** telemedicine, teleconsultation, tele-expertise, humanitarian, low-resource setting

## INTRODUCTION

In 2009, Médecins Sans Frontières (Doctors Without Borders, MSF) began a pilot trial of a store-and-forward medical tele-expertise service. “Tele-expertise” is defined in the French Public Health Code (1), as one of the five main areas of telemedicine (teleconsultation, tele-assistance, tele-monitoring, medical emergency call center, and tele-expertise). The MSF system uses electronic message exchange between field clinicians working in resource-limited settings without access

<footnote n="1">¹Fondation Médecins Sans Frontières, Paris, France, ²EA4569, Department of Medical Ethics and Legal Medicine, Paris Descartes University, Paris, France, ³University of Edinburgh, Edinburgh, UK, ⁴Médecins Sans Frontières, Geneva, Switzerland, ⁵McGill University, Montréal, QC, Canada, ⁶Internal Medicine Department, Tenon Hospital, AP-HP, Paris, France, ⁷Faculty of Medicine, Sorbonne Universités, University Pierre et Marie Curie, Paris, France, ⁸U1142, Laboratoire d'Informatique Médicale et d'Ingénieurie des Connaissances en e-Santé (LIMICS), INSERM, Université Paris, Paris, France, ⁹Crash, Fondation Médecins Sans Frontières, Paris, France, ¹⁰Norwegian Centre for Integrated Care and Telemedicine, University Hospital of North Norway, Tromsø, Norway, ¹¹Faculty of Health Sciences, University of Tromsø, Tromsø, Norway</footnote>
to specialized medical expertise, and specialists working in relatively well-resourced settings who offer *pro bono* specialist advice to their field colleagues. In collaboration with the Swinfen Charitable Trust (2), three separate tele-expertise networks (French, English, and Spanish) were established initially and then combined into a single multilingual network in the late 2013, using web-based technology based on the Collegium Telemedicus system (3).

The MSF tele-expertise service provides specialized advice to field clinicians across a very wide range of medical and surgical specialties. At present more than 350 MSF field clinicians have access to the system, and there are 300 volunteer specialists from all over the world who provide tele-expertise; the median delay in providing the first specialist response to the referrer is less than 6 h (4).

To ensure that the allocation of resources is appropriate in health care, pilot projects require careful evaluation. In the clinical field, such an evaluation can also be considered as a necessary reflective practice, carried out in order to improve both the efficiency of the system and provider practice (5). We therefore surveyed the users of the MSF pilot tele-expertise service to obtain their feedback in terms of satisfaction and benefit, to draw conclusions about the usefulness of the service, and to identify potential areas of improvement. In a previous paper, we presented the main results of the survey (4). The present study provides the raw data and a further and more detailed analysis. In order to understand the results and put them in perspective, we compared them with the (limited) data which has been published on other, similar telemedicine networks (6, 7).

# MATERIALS AND METHODS

We carried out a survey of all users with an account on the tele-expertise network in January 2014. The questions were established after conducting a literature search combined with qualitative data collection:

1. in-depth interviews with three referrers and three specialists;

2. participant observation performed by two of the authors during several field missions in MSF settings (Democratic Republic of the Congo, Central African Republic, Yemen, Somalia, Syria, and Haiti).

The survey contained 50 questions. They were closed-ended, multiple choice and scale type questions, and open-ended questions.

The survey was tested on three referrers and three specialists, in English and in French. After the pilot testing, the survey was sent separately to all referrers (French and English) and specialists (French and English) with accounts on the system, regardless of whether they were active (i.e., irrespective of whether they had logged in and submitted cases or answered them). The questions for referrers and specialists were almost identical. Versions of the survey were made available in French and English. Web-based survey software (https://www.surveymonkey.com/) was used to collect the data. Two reminders were sent by email to the users, after 1 and 3 weeks. Data collection was closed after 1 month.

Ethics permission to conduct the survey was not required because patient consent to access the data had been obtained previously, and the work was a retrospective chart review conducted by the organization's staff in accordance with its research policies (8).

Survey responses were examined with the usual methods for quantitative analysis, while the results of the open-ended questions were processed in a qualitative way based on a systematic content analysis.

# Differences Between Language Groups

Responses were received to both the English and French versions of the survey. To examine potential differences between the two language groups, the English and French responses were compared using chi-squared tests for certain key variables:

1. Age. The respondents' ages were recorded in five age categories. Since the age categories were ordered, a chi-squared test for trend was carried out.

2. Qualifications. The respondents' qualifications were recorded in three categories. These were unordered categories, so Fisher's exact test was used with *P*-values calculated according to the Freeman-Halton extension (9, 10).

3. User-friendliness of the system interface. The respondent's opinion about the user-friendliness of the system was categorized as yes/no, so Fisher's exact test was used.

4. Value of telemedicine. The respondent's opinions about whether telemedicine improved patient management was recorded in three unordered categories, so Fisher's exact test was used with *P*-values calculated according to the Freeman-Halton extension (9, 10).

# Differences Between Referrers and Specialists

Responses from referrers and specialists were compared, irrespective of language, using chi-squared tests. Where multiple ordered responses were possible, the chi-squared test for trend was employed.

# RESULTS

## Users and Response Rate

A total of 294 referrers and 254 specialists had accounts on the MSF network. From an examination of the system database, a total of 104 referrers (35%) had submitted cases and 120 specialists (47%) had replied to queries, i.e., 224 of the 548 account holders could be considered as to be active users (41%).

The survey was sent to all referrers and specialists. Of the 548 people who were sent the survey, 163 (30%) responded: the group details are summarized in **Table 1**. The survey was completed reasonably promptly. Non-linear regression showed that during the first week, the numbers of responses received doubled every 1.8 days ($R^2$ = 0.94). Seventy percent of the questionnaires were completed within 6 days (**Figure 1**).

By matching answers to some questions (such as whether they found the system easy to use and how long it took to write
a referral or answer a case), the survey showed that approximately 40 of the 64 referrers who responded (62%) and 70 of 99 specialists who responded (70%) could be considered as active users. Although there were more referrers (54%) than specialists (46%) in the population sampled, specialists provided significantly more answers than referrers – 39 versus 22%. This was consistent when considering the individual item response rates.

## Differences Between Main Categories

There were no significant differences between the responses from French and English users (Table 2), so the responses were combined for subsequent analysis. The survey questions were divided into five main categories (see Table 3):

### General User Profile (Personal, Professional)

The average age of the specialists was significantly higher than that of the referrers (P < 0.01). The responders were multinational, predominantly European. Most responders were medical doctors, and the majority had previously completed one or more MSF field missions (including the specialists). Further details are provided in Table 3A.

### IT Profile (Telemedicine Experience, Habits)

Most responders had access to email, usually via a laptop computer or a mobile phone. Three-quarters of the responders had used social media, such as Facebook and Twitter, for general

<table><caption>TABLE 1 | Responses from referrers and specialists.</caption><thead><tr><th></th><th>Total number in group</th><th>Number of responses in English</th><th>Number of responses in French</th><th>Total number of responses (%)</th></tr></thead><tbody><tr><td>Referrers</td><td>294</td><td>52</td><td>12</td><td>64 (22)</td></tr><tr><td>Specialists</td><td>254</td><td>68</td><td>31</td><td>99 (39)</td></tr><tr><td>Total</td><td>548</td><td>120</td><td>43</td><td>163 (30)</td></tr></tbody></table>

<figure><img src="image_2.png" /><figcaption>FIGURE 1 | Cumulative number of responses to the survey (n = 163).</figcaption></figure>

communications, with a small difference (P = 0.04) between specialists (67%) and referrers (83%). Further details are provided in Table 3B.

## Use of Telemedicine (Interface/Platform and Interaction)

Only about half of the responders (44% of the referrers and 55% of the specialists) had received a prior briefing about telemedicine. However, the majority (more than three-quarters) stated that the system was user friendly, and they found it self-explanatory (i.e., they did not need to be shown how to use it). The average time reported by referrers for writing a case was 14.7 min, and the average time reported by specialists for replying was 17.4 min.

Half of the referrers (51%) reported having connection problems, either sometimes or always, compared to 24% of specialist; this difference was significant (P < 0.01). Less than half of the referrers (45%) stated that they had provided feedback about the patient for the specialist; however, 92% of the specialists reported that they had not received any feedback about the patient. Further details are provided in Table 3C.

## Outcomes (Satisfaction, Usefulness)

The majority of referrers (91%) stated that they found the advice received via telemedicine to be useful. The most common reasons for this were in providing reassurance for the referrer or patient, in changing the management of the patient and in changing the diagnosis. The majority of referrers (79%) felt that advice from the specialist improved their management of the patient. The referrers' mean score for their satisfaction with telemedicine was 7.6 (on a scale from 1 = very unsatisfied to 10 = very satisfied).

The majority of specialists who provided an opinion (94%) thought that their advice had improved the management of the patient. The majority of specialists (99%) felt that there was educational benefit for the field doctor. The specialists' mean score for their satisfaction with telemedicine was 6.6 (on a scale from 1 = very unsatisfied to 10 = very satisfied). Further details are provided in Table 3D.

## Other (Referrals, Opinions)

Both referrers and specialists indicated the time in which it was desirable to receive (or provide) the response, by choosing from a series of time intervals. Using the midpoint of these time intervals to summarize the frequency distributions showed that

<table><caption>TABLE 2 | Differences between the responses from those completing the French and English versions of the questionnaire.</caption><thead><tr><th></th><th>Possible responses</th><th>Referrers</th><th>Specialists</th></tr></thead><tbody><tr><td>Age (years)</td><td>25–34, 35–44, 45–54, 55–64, &gt;65</td><td>P = 0.14</td><td>P = 0.90</td></tr><tr><td>Qualifications</td><td>Medical doctor, nurse, and others</td><td>P = 0.61</td><td>P = 0.31</td></tr><tr><td>System is user friendly</td><td>Yes and no</td><td>P = 0.17</td><td>P = 0.10</td></tr><tr><td>Telemedicine is valuable</td><td>Yes, no, and unknown</td><td>P = 0.73</td><td>P = 0.51</td></tr></tbody></table>

The P-values shown are for the tests described in the text.
<table><caption>TABLE 3 | Survey results.</caption><thead><tr><th>Question</th><th>Response (majority response in bold with %)</th><th>Total answered</th><th>Skipped</th></tr></thead><tbody><tr><td colspan="4"><strong>(A) GENERAL USER PROFILE</strong></td></tr><tr><td colspan="4"><strong>Referrers</strong></td></tr><tr><td>1. How old are you?</td><td>Mean 41 years</td><td>63</td><td>1</td></tr><tr><td>2. What is your nationality?</td><td>Spain 6, Canada 6, France 6, Belgium 4, Netherlands 3, Kenya 3, Colombia 3, 8 other countries (2 referrers each) 16, and 13 other countries (1 referrer each) 13</td><td>60</td><td>4</td></tr><tr><td>3. What is your qualification and/or field of expertise?</td><td>Doctors (73%) 46, nurses 12, and others 5</td><td>63</td><td>1</td></tr><tr><td>4. How many missions (MSF or other field experiences) have you undertaken?</td><td>&lt;1 1, 1-2 14, 3-5 13, and &gt;5 (54%) 33</td><td>61</td><td>3</td></tr><tr><td>5. How long is the total (cumulative) duration of these missions?</td><td>&gt;5 years, &lt;1 year 5, 1-2 years 16, 3-5 years 12, and &gt;5 years (46%) 28</td><td>61</td><td>3</td></tr><tr><td>6. What was your job position when a system user?</td><td>Medical team leader (44%) 21, medical coordinator 20, regular volunteer 7, first mission 4, and others 16</td><td>48</td><td>16</td></tr><tr><td colspan="4"><strong>Specialists</strong></td></tr><tr><td>1. How old are you?</td><td>Mean 47 years</td><td>99</td><td>1</td></tr><tr><td>2. What is your nationality?</td><td>France 20, Canada 17, Spain 10, US 8, UK 6, Netherlands 5, Argentina 4, Colombia 3, 4 other countries (2 specialists each), and 16 other countries (1 specialist each)</td><td>97</td><td>2</td></tr><tr><td>3. What is your qualification and/or field of expertise?</td><td>Doctors (93%) 87, nurses 2, and others 5</td><td>94</td><td>5</td></tr><tr><td>4. Where is your work location?</td><td>Public health service 28, teaching hospital (45%) 44, private 13, NGO 28, retired 3, and others 6</td><td>98</td><td>1</td></tr><tr><td>5. How many missions (MSF or other field experience) have you undertaken?</td><td>&lt;1 33, 1-2 14, 3-5 12, and &gt;5 (38%) 37</td><td>96</td><td>3</td></tr><tr><td>6. How long is the total (cumulative) duration of these missions?</td><td>&lt;1 year 28, 1-2 years (37%) 19, 3-5 years 16, and &gt;5 years 12</td><td>75</td><td>24</td></tr><tr><td>7. In which year was your last mission?</td><td>1971-1980 1, 1981-1990 1, 1991-2000 2, 2001-2010 23, and 2011-2014 (52%) 36</td><td>69</td><td>30</td></tr><tr><td colspan="4"><strong>(B) IT PROFILE</strong></td></tr><tr><td colspan="4"><strong>Referrers</strong></td></tr><tr><td>7. Are you involved in any other telemedicine project?</td><td>Yes 12 and no (80%) 49</td><td>61</td><td>3</td></tr><tr><td>8. During your mission, how many times per day do/did you usually check your emails?</td><td>&lt;1/day 6, 1/day 8, &gt;2/day (58%) 34, and continuously (e.g., smartphone) 13</td><td>59</td><td>4</td></tr><tr><td>9. When off mission, how many times do you check your emails during the day?</td><td>&lt;1/day 2, 1/day 10, &gt;2/day (57%) 35, and continuously (e.g., smartphone) 14</td><td>61</td><td>3</td></tr><tr><td>10. Do you have access to Internet at work?</td><td>None 0, not easily 10, easily (65%) 40, and continuously (e.g., smartphone) 11</td><td>61</td><td>3</td></tr><tr><td>11. How was the Internet connection quality and speed?</td><td>Very low 2, low 10, middle (52%) 32, and high 17</td><td>61</td><td>3</td></tr><tr><td>12. Are you able to send files attached to emails?</td><td>Easily whatever the attachment size 15, easily if small attachment (59%) 36, with difficulty 10, and impossible 0</td><td>61</td><td>3</td></tr><tr><td>13. What type of connection do you mainly use?</td><td>Wifi (60%) 36, Ethernet 14, modem 14, mobile 6, and Vsat 5</td><td>60</td><td>4</td></tr><tr><td>14. What is usually the duration of your Internet connection?</td><td>&lt;2 min 3, 2-5 min 2, 6-20 min 11, and &gt;20 min (74%) 45</td><td>61</td><td>3</td></tr><tr><td>15. What type of equipment do you have?</td><td>Mobile phone with email 18, laptop (95%) 58, tablet 7, and others 2</td><td>61<sup>a</sup></td><td>3</td></tr><tr><td>16. What other networks do you use?</td><td>Facebook (70%) 37, Twitter 14, professional medical network 27, and others 3</td><td>53<sup>a</sup></td><td>11</td></tr><tr><td colspan="4"><strong>Specialists</strong></td></tr><tr><td>8. Are you involved in any other telemedicine project?</td><td>Yes (74%) 25 and no 73</td><td>98</td><td>1</td></tr></tbody></table>

(Continued)
<table><caption>TABLE 3 | Continued</caption><thead><tr><th>Question</th><th>Response (majority response in bold with %)</th><th>Total answered</th><th>Skipped</th></tr></thead><tbody><tr><td>9. How many times do you check your emails during the day?</td><td>&lt;1/day 5, 1/day 18, &gt;2/day 34, and <strong>continuously (44%)</strong> (e.g., smartphone) 42</td><td>96<sup>a</sup></td><td>3</td></tr><tr><td>10. Do you have access to the Internet at work?</td><td>None 0, not easily 3, easily 45, and <strong>continuously (51%)</strong> (e.g., smartphone) 49</td><td>97</td><td>2</td></tr><tr><td>11. How was the Internet connection quality and speed?</td><td>Very low 1, low 2, middle 25, and <strong>high (71%)</strong> 68</td><td>96</td><td>3</td></tr><tr><td>12. Are you able to send files attached to emails?</td><td><strong>Easily whatever the attachment size (52%)</strong> 51, easily if small attachment 46, with difficulty 1, and impossible 0</td><td>98</td><td>1</td></tr><tr><td>13. What type of connection do you mainly use?</td><td><strong>Wifi (66%)</strong> 64, Ethernet 30, modem 12, and mobile 14</td><td>97</td><td>2</td></tr><tr><td>14. What is usually the duration of your Internet connection?</td><td>&lt;2 min 3, 2–5 min 2, 6–20 min 11, and &gt;20 min (83%) 79</td><td>95</td><td>4</td></tr><tr><td>15. What type of equipment do you have?</td><td>Mobile phone with email 56, <strong>laptop (87%)</strong> 85, tablet 32, and others 17</td><td>98</td><td>1</td></tr><tr><td>16. What other networks do you use?</td><td>Facebook 33, Twitter 7, <strong>professional medical network (70%)</strong> 46, and others 10</td><td>66</td><td>33</td></tr><tr><td colspan="4"><strong>(C) USE OF THE TELEMEDICINE NETWORK</strong></td></tr><tr><td colspan="4"><strong>Referrers</strong></td></tr><tr><td>17. Did you receive any briefing about the system prior to departure?</td><td>Yes 21 and <strong>no (56%)</strong> 27</td><td>48</td><td>16</td></tr><tr><td>18. Do you find the design user friendly?</td><td><strong>Yes (84%)</strong> 36 and no 7</td><td>43</td><td>21</td></tr><tr><td>19. Did you find it self-explanatory (i.e., there was no need to be shown how it works)?</td><td><strong>Yes (58%)</strong> 24 and no 17</td><td>41</td><td>23</td></tr><tr><td>20. Did you encounter any problems with the username or password?</td><td><strong>Never (45%)</strong> 19, sometimes (45%) 19, regularly 3, and always 1</td><td>42</td><td>22</td></tr><tr><td>21. Did you encounter any problems of connection while using the interface?</td><td><strong>Never (49%)</strong> 20, sometimes 16, regularly 5, and always 0</td><td>41</td><td>23</td></tr><tr><td>22. If a problem occurred, was it easy to solve?</td><td><strong>Yes (45%)</strong> 20, no 5, and not applicable 19</td><td>44</td><td>20</td></tr><tr><td>23. If a problem occurred, was it easy to contact the system operator for support?</td><td>Yes 20, no 4, and <strong>not applicable (47%)</strong> 21</td><td>45</td><td>19</td></tr><tr><td>24. How long on average does it take to write a clinical case?</td><td>&lt;5 min 4, 5–10 min 11, <strong>10–20 min (32%)</strong> 13, and &gt;20 min 12</td><td>40</td><td>24</td></tr><tr><td>25. How did you write your referral?</td><td>Online 13 and <strong>offline (and then copy and paste it in) (67%)</strong> 26</td><td>39</td><td>25</td></tr><tr><td>26. Did you find it easy to send an attached file?</td><td><strong>Yes (55%)</strong> 23, no 10, and not applicable 9</td><td>42</td><td>22</td></tr><tr><td>27. Have you ever given any information about the system to a patient before sending a case?</td><td><strong>Yes (70%)</strong> 26 and no 11</td><td>37</td><td>27</td></tr><tr><td>28. How did you get his/her consent?</td><td>Orally (78%) 28, written 0, and no consent 9</td><td>36</td><td>28</td></tr><tr><td>29. Which useful documents do you think should be linked on the website?</td><td>MSF clinical guidelines 15, essential drugs 11, <strong>medical report template (36%)</strong> 30, technical advice (e.g., taking pictures and sending attachments) 26, and others 2</td><td>84 responses and 38 responders</td><td>26</td></tr><tr><td>37. Did you give back to the specialist any feedback about this patient?</td><td>Yes 15 and <strong>no (55%)</strong> 18</td><td>33</td><td>31</td></tr><tr><td>38. If No, was it because (tick any that apply)</td><td>Patient lost to follow-up 5, <strong>lack of time (30%)</strong> 11, forgotten to update 9, feeling it was not necessary 6, worse outcome of patient died 1, and difficulties with Internet access 5</td><td>37 responses and 21 responders</td><td>43</td></tr><tr><td>40. In your opinion, is the patient likely to be available for follow-up in 2–4 months?</td><td>Yes 8, <strong>no (46%)</strong> 17, and do not know 12</td><td>37</td><td>27</td></tr><tr><td colspan="4"><strong>Specialists</strong></td></tr><tr><td>17. Did you receive any briefing about the system?</td><td><strong>Yes (55%)</strong> 48 and no 40</td><td>88</td><td>11</td></tr><tr><td>18. Do you find the design user friendly?</td><td><strong>Yes (77%)</strong> 63 and no 19</td><td>82</td><td>17</td></tr><tr><td>19. Did you find it self-explanatory (i.e., there was no need to be shown how it works)?</td><td><strong>Yes (79%)</strong> 64 and no 17</td><td>81</td><td>17</td></tr></tbody></table>

(Continued)
<table><caption>TABLE 3 | Continued</caption><thead><tr><th>Question</th><th>Response (majority response in bold with %)</th><th>Total answered</th><th>Skipped</th></tr></thead><tbody><tr><td>20. Did you encounter any problems with the username or password?</td><td>Never (48%) 38, sometimes 30, regularly 9, and always 2</td><td>79</td><td>20</td></tr><tr><td>21. Did you encounter any problems of connection while using the interface?</td><td>Never (76%) 60, sometimes 17, regularly 1, and always 1</td><td>79</td><td>20</td></tr><tr><td>22. If a problem occurred, was it easy to solve?</td><td>Yes 29, no 8, and not applicable (53%) 42</td><td>79</td><td>20</td></tr><tr><td>23. How long on average does it take to write your answer to a clinical case?</td><td>&lt;5 min 5, 5-10 min 11, 10-20 min 24, and &gt;20 min (33%) 32</td><td>72</td><td>27</td></tr><tr><td>24. How did you write your referral answer?</td><td>Online (68%) 50 and offline (and then copy and paste it) 25</td><td>74</td><td>25</td></tr><tr><td>25. Did you find it easy to send an attached file?</td><td>Yes (48%) 37, no 6, and not applicable 34</td><td>77</td><td>22</td></tr><tr><td>27. How many cases did you answer?</td><td>0-1 9, 1-5 (49%) 38, 6-10 14, and &gt;10 7</td><td>78</td><td>21</td></tr><tr><td>28. Was the information (including any images) supplied by the referrer adequate?</td><td>Yes (66%) 44 and no 22</td><td>66</td><td>33</td></tr><tr><td>29. Was information about the hospital available on the website (e.g., information about the staff and facilities)?</td><td>Absent 18, insufficient 21, sufficient (35%) 22, and easily accessible and complete 1</td><td>62</td><td>37</td></tr><tr><td>30. Was the question asked by the referrer clear?</td><td>Yes (92%) 59 and no 5</td><td>64</td><td>35</td></tr><tr><td>31. Was it difficult to find the time required to answer this case?</td><td>Yes 22 and no (66%) 42</td><td>64</td><td>35</td></tr><tr><td>37. Did you receive any follow-up information about this patient?</td><td>Yes 5 and no (92%) 58</td><td>63</td><td>36</td></tr><tr><td colspan="4" style="background-color: #E0E0E0; text-align: left; font-weight: bold;">(D) OUTCOMES</td></tr><tr><td colspan="4" style="text-align: left; font-weight: bold;">Referrers</td></tr><tr><td>30. Did you find the advice helpful?</td><td>A lot (42%) 14, moderately 13, a little 3, and not at all 3</td><td>33</td><td>31</td></tr><tr><td>31. If yes, did it. Please tick any that apply<sup>a</sup>: change your diagnosis, change your management of the patient, improve the patient's symptoms, improve the patient's function, and provide reassurance to you or the patient</td><td>Change your diagnosis 12, change your management of the patient 21, improve the patient's symptoms 10, improve the patient's function 6, and provide reassurance to you or the patient (34%) 25</td><td>74 responses and 31 responders</td><td>33</td></tr><tr><td>32. Was the answer appropriate and adapted to your field environment?</td><td>A lot 10, moderately (61%) 20, a little 1, and not at all 2</td><td>33</td><td>31</td></tr><tr><td>33. Do you think that the advice you received from the specialist improved the management of the patient?</td><td>Yes (79%) 26, no 2, and unknown 5</td><td>33</td><td>31</td></tr><tr><td>36. Was there any educational benefit to you in the reply?</td><td>No 2, a little 8, moderately (39%) 13, and a lot 10</td><td>33</td><td>31</td></tr><tr><td>42. Do you have any concerns about the telemedicine process?</td><td>No (76%) 28 and yes 9 (see Table 4)</td><td>37</td><td>27</td></tr><tr><td>43. Would you recommend using the system to colleagues?</td><td>Yes (97%) 33 and no 1</td><td>34</td><td>30</td></tr><tr><td>44. Overall, how would you rate your user satisfaction on a scale from 0 to 10?</td><td>Average 7.61</td><td>36</td><td>28</td></tr><tr><td>45. In which specialty do you think that the system is the most useful?<sup>a</sup></td><td>All specialties (28%) 20, medical specialties (e.g., infectious diseases) 17, surgical specialties (e.g., orthopedics) 4, radiology 11, pediatrics 12, subspecialties (e.g., pediatrics) 7, and others 1</td><td>72 responses and 40 responders</td><td>24</td></tr><tr><td>46. In a low-resource setting, do you think that access to a specialist doctor can help the field doctor? (visual scale from 0 to 10)</td><td>Average 8.27</td><td>41</td><td>23</td></tr><tr><td>47. Do you think that field doctors are isolated in their practice in the field? (visual scale from 0 to 10)</td><td>Average 6.76</td><td>41</td><td>23</td></tr><tr><td>48. Do you think that this system of assistance can help the referring doctors? (four-point scale from not at all to a lot)</td><td>(a) Overall: average 3.65<br>(b) To feel less isolated: average 3.67</td><td>37<br>35</td><td>27<br>29</td></tr></tbody></table>

(Continued)
<table><caption>TABLE 3 | Continued</caption><thead><tr><th>Question</th><th>Response (majority response in bold with %)</th><th>Total answered</th><th>Skipped</th></tr></thead><tbody><tr><td colspan="4"><strong>Specialists</strong></td></tr><tr><td>34. Do you think the advice you provided improved the management of the patient?</td><td>Yes (43%) 30, no 2, and unknown 35</td><td>69</td><td>30</td></tr><tr><td>35. Do you think that there is any educational benefit for the field doctors when receiving the reply?</td><td>No 1, a little 9, medium (52%) 40, and a lot 26</td><td>76</td><td>23</td></tr><tr><td>36. Did the consultation have any value for you personally?</td><td>No 11 and yes 56 [mainly clinical 11, mainly educational 7, both clinical and educational (42%) 30, and others 12]</td><td>71 responses and 67 responders</td><td>32</td></tr><tr><td>39. Do you have any concerns about the telemedicine process?</td><td>No (78%) 59 and yes 17 (see Table 5)</td><td>76</td><td>23</td></tr><tr><td>41. Generally speaking, how would you rate your satisfaction of the system on a scale from 0 to 10?</td><td>Average 6.63</td><td>73</td><td>26</td></tr><tr><td>43. In a low-resource setting, do you think that access to a specialist doctor can help the field doctor? (visual scale from 0 to 10)</td><td>Average 8.04</td><td>82</td><td>17</td></tr><tr><td>44. Do you think that field doctors are isolated in their practice in the field? (visual scale from 0 to 10)</td><td>Average 7.21</td><td>81</td><td>18</td></tr><tr><td>45. Do you think that this system of assistance can help the referring doctors? (six-point scale from Not at all to A lot)</td><td>(a) Overall: average 3.63<br>(b) To feel less isolated: average 3.65</td><td>71<br>75</td><td>28<br>24</td></tr><tr><td colspan="4"><strong>(E) OTHERS</strong></td></tr><tr><td colspan="4"><strong>Referrers</strong></td></tr><tr><td>34. In your opinion, what is the acceptable amount of time for receiving the expert's answer?</td><td>&lt;6 h 4, 6-12 h 8, 12-24 h (41%) 16, 24-48 h 9, and 48-72 h 2</td><td>39</td><td>25</td></tr><tr><td>35. In your opinion, what is the desirable amount of time for receiving the expert's answer?</td><td>&lt;6 h 15, 6-12 h (39%) 15, 12-24 h 7, 24-48 h 1, and 48-72 h 0</td><td>38</td><td>26</td></tr><tr><td>39. Do you think that feedback about the patient is?</td><td>Optional 5, desirable (43%) 16, necessary 11, and mandatory 5</td><td>37</td><td>27</td></tr><tr><td>41. In your opinion, when would it be relevant to give follow-up information? (e.g. providing a progress report)</td><td>After 1 week (53%) 20, after 2 weeks 9, after 1 month 7, after 3 months 2, and after 6 months 0</td><td>38</td><td>28</td></tr><tr><td>49. Please add any other comments about the service in general, or any suggestions for improvement</td><td>See Table 4</td><td>16</td><td>48</td></tr><tr><td colspan="4"><strong>Specialists</strong></td></tr><tr><td>26. Which useful documents do you think should be linked on the website?<sup>a</sup></td><td>MSF clinical guidelines (28%) 54, essential drugs 43, medical report template 34, technical advice (taking pictures and sending attachments) 44, and others 18</td><td>193 responses and 77 responders</td><td>22</td></tr><tr><td>32. In your opinion, what is the acceptable amount of time in which to provide an answer?</td><td>&lt;6 h 5, 6-12 h 10, 12-24 h (42%) 33, 24-48 h 24, and 48-72 h 6</td><td>78</td><td>21</td></tr><tr><td>33. In your opinion, what is the desirable amount of time in which to provide an answer?</td><td>&lt;6 h 23, 6-12 h (26%) 20, 12-24 h 23, 24-48 h 12, and 48-72 h 0</td><td>78</td><td>21</td></tr><tr><td>38. Do you think that feedback about the patient is?</td><td>Optional 1, desirable (52%) 35, necessary 19, and mandatory 12</td><td>67</td><td>32</td></tr><tr><td>40. Are you happy to provide consultations for another non-MSF network in the future?</td><td>Yes (81%) 63 and no 15</td><td>78</td><td>21</td></tr></tbody></table>

(Continued)
<table><caption>TABLE 3 | Continued</caption><thead><tr><th>Question</th><th>Response (majority response in bold with %)</th><th>Total answered</th><th>Skipped</th></tr></thead><tbody><tr><td>42. In which specialty do you think that the system is the most useful?<sup>a</sup></td><td>All specialties (45%) 54, medical specialties (e.g., infectious diseases) 18, surgical specialties (e.g., orthopedics) 7, radiology 17, pediatrics 8, subspecialties (e.g., neuropediatrics) 10, and others 7</td><td>121 responses and 79 responders</td><td>20</td></tr><tr><td>46. What is your main motivation to act as a volunteer for the system?<sup>a</sup></td><td>Taking part in MSF action (30%) 56, keeping in touch with MSF 17, interest in telemedicine 23, treating patients from low-resource settings 53, taking part in the field doctors' training 27, and others 7</td><td>183 responses and 81 responders</td><td>18</td></tr><tr><td>47. Do you think that volunteering is the right status for the expert?</td><td>Yes (94%) 75 and no 5</td><td>80</td><td>19</td></tr><tr><td>48. How many cases in routine do you think you could reasonably answer?</td><td>1/day 8, 3/week 24, 1/week (36%) 29, 1/fortnight 9, 1/month 9, and others 2</td><td>81</td><td>18</td></tr><tr><td>49. Do you think that the expert should receive payment?</td><td>Yes 4 and no (95%) 76</td><td>80</td><td>19</td></tr><tr><td>50. Please add any other comments about the service in general or any suggestions for improvement</td><td>See Table 5</td><td>37</td><td>62</td></tr></tbody></table>

the referrers felt that it was desirable to receive the specialist's response within 9 h, and that the specialists thought that it was desirable to provide a response within 14 h, a slightly longer delay. Similarly, almost all responders felt that receiving feedback (specialists) or providing feedback (referrers) was necessary or obligatory; only 14% of referrers and 1% of specialists felt that it was optional. Further details are provided in Table 3E.

## Free-Text Comments Analysis

The comments made by referrers and specialists in response to the open-ended questions are summarized in Tables 4 and 5, respectively. Both referrers and specialists used adjectives such as “excellent” and phrases expressing their gratitude that the system had been set up. It is worth noting that these comments emphasized the users’ involvement in the pilot and reflected their desire to improve the system where appropriate.

The main comments from referrers focused on the lack of support, promotion, and predeparture briefing at headquarters’ level, whereas the principal concern expressed by the specialists was the lack of feedback about patient follow-up. There were some negative comments made by referrers concerning the

<table><caption>TABLE 4 | Comments from referrers in response to open-ended questions (numbers 42 and 49).</caption><thead><tr><th></th><th>Number of comments</th></tr></thead><tbody><tr><td>Lack of headquarters' support in using the system</td><td>5</td></tr><tr><td>Satisfaction (e.g., "excellent," "congratulations," "thank you," "well done," "outstanding," "wonderful," and "very good")</td><td>4</td></tr><tr><td>Lack of promotion of the system (internally and externally)</td><td>4</td></tr><tr><td>Reduced isolation of field doctors (the more remote is the setting, the more telemedicine helps)</td><td>2</td></tr><tr><td>Briefing to field staff should be improved ("part of the package" and "keep fighting for promoting the service")</td><td>2</td></tr><tr><td>Platform "not interactive enough": proposal to use other technology (e.g., video and SMS)</td><td>2</td></tr><tr><td>Difficulties in getting access to the websites</td><td>2</td></tr><tr><td>Specialist advice:</td><td></td></tr><tr><td>"needs to be more field specific"</td><td>1</td></tr><tr><td>"need to make specialist well aware of field limitations"</td><td>1</td></tr><tr><td>"providing solutions for specialized management if not available in the field"</td><td>1</td></tr><tr><td>issue of delay in getting specialist reply</td><td>1</td></tr><tr><td>Field observation that "there is low use of the system"</td><td>1</td></tr><tr><td>Not all specialties covered (e.g., multidrug resistant tuberculosis)</td><td>1</td></tr><tr><td>Reluctance to use the system by expatriate doctors (fearing that specialized treatment recommended by specialist would be not available)</td><td>1</td></tr><tr><td>Creating a "link to telemedicine" in MSF clinical guidelines</td><td>1</td></tr><tr><td>Fear of bypassing the headquarters, medical referent if using the system</td><td>1</td></tr><tr><td>Decision to give access to the system only to medical coordinator (and not bedside doctors) since field medical staff should not "be service dependant" (no need of the system to take medical decision)</td><td>1</td></tr></tbody></table>

<footnote><sup>a</sup>Respondents could select more than one choice.</footnote>
<table><caption>TABLE 5 | Comments from specialists in response to open-ended questions (numbers 39 and 50).</caption><thead><tr><th></th><th>Number of comments</th></tr></thead><tbody><tr><td>Lack of feedback about patient follow-up</td><td>9</td></tr><tr><td>No case received ("frustration" and "disappointment" leading to loss of motivation/disengagement)</td><td>7</td></tr><tr><td>Satisfaction (e.g., "congratulations," "excellent," and "merci")</td><td>3</td></tr><tr><td>Importance of field experience for giving a well-adapted answer (feeling of lack of experience)</td><td>2</td></tr><tr><td>Reasons of difficulties in giving their opinion</td><td></td></tr><tr><td style="padding-left: 2em;">lack of time</td><td>1</td></tr><tr><td style="padding-left: 2em;">difficulties in giving advice without performing their own clinical/physical examination</td><td>1</td></tr><tr><td style="padding-left: 2em;">lack of knowledge about local diseases and hospital setting (facilities, investigations, and drugs available)</td><td>1</td></tr><tr><td>Negative points about the referral received</td><td></td></tr><tr><td style="padding-left: 2em;">poor quality of images</td><td>1</td></tr><tr><td style="padding-left: 2em;">improving the quality of the case reports</td><td>1</td></tr><tr><td style="padding-left: 2em;">system not well adapted for emergency case</td><td>1</td></tr><tr><td>Negative points about the platform design</td><td></td></tr><tr><td style="padding-left: 2em;">design "unpersonal, cold"</td><td>1</td></tr><tr><td style="padding-left: 2em;">needs "to be polished"</td><td>1</td></tr><tr><td style="padding-left: 2em;">password forgotten: suggestion that system sends systematically a login reminder after 6 months without logging in</td><td>1</td></tr><tr><td>Difficulties in getting access to the websites</td><td>1</td></tr><tr><td>Service provides "moral (psychological?) support" and "field doctor reassurance"</td><td>1</td></tr><tr><td>Long-term benefit: "telemedicine can be the future" especially when finishing up an MSF project (to give assistance to local staff after the project has closed)</td><td>1</td></tr><tr><td>Specialist happy to volunteer for MSF, but financial incentives should be considered for those who are sent cases very frequently</td><td>1</td></tr><tr><td>Technical issues</td><td></td></tr><tr><td style="padding-left: 2em;">setting up a video link such as Skype, Facetime (comment from an ophthalmologist)</td><td>1</td></tr><tr><td style="padding-left: 2em;">possibility of using SMS for some communications</td><td>1</td></tr></tbody></table>

characteristics of the expert advice received (delayed, not field specific, etc.) and the platform itself (not enough interactive communication). Such remarks may explain the reluctance of some potential users to use the system (e.g., fear of not being able to implement the expert advice received).

Anticipating a possible loss of motivation, specialists expressed their frustration in not receiving cases, which can also be seen as another consequence of the lack of promotion and the low use of the system. However, the specialists provided constructive comments about the system. One pointed out the long-term benefit of such a network, which can continue at a health care facility even after MSF support in the field has ended.

## DISCUSSION

We conducted a comprehensive survey of the users of the MSF tele-expertise service during its first four years of operation.

Our previous paper (4) summarized the main results, and the present paper provides a detailed description and analysis. There was high satisfaction with the service, among both referrers and specialists. All users recognized the benefits of providing access to specialist advice in low-resource settings where there is usually no alternative way of obtaining specialist expertise.

Although there was general satisfaction with the service, the survey identified some specific problems. For example, some referrers reported poor connectivity in the field, which affected their ability to communicate by email and to upload files via the telemedicine network. This problem is amenable to technical solution. However, the main concerns – raised by both referrers and specialists – were the lack of promotion of the system at headquarters' level, and the lack of feedback about patient follow-up. These problems are less easy to address.

Another issue uncovered during our field participant observations was the use of uncontrolled social media for communicating confidential information in remote health care. The survey confirmed that most referrers and specialists used Facebook and/or Twitter for general communications, and personal communications from certain users confirmed that some MSF staff also used these media for obtaining second opinions about medical cases. For example, a Facebook group was created to provide access to a videolink and obtain feedback from expert friends. In contrast, the MSF tele-expertise service represents a safe and secure method of facilitating such consultations at a distance, and this aspect has been an important factor in convincing some reluctant headquarters' stakeholders to adopt the system.

## Limitations of the Study

The study had certain limitations. For example, some of the information solicited in the survey was based on the opinions of the responders, which may itself introduce bias. Despite the methodological weakness of seeking personal opinions, it is important to conduct such surveys in order to identify areas for service improvement.

About two-thirds of the responses were from active users. The results therefore provide a view from users with practical experience of the tele-expertise service. This was supplemented by the views of the inactive users, i.e., people who were registered account holders but at the time of the survey were still awaiting an opportunity to submit a case or provide an answer. Unfortunately, the detailed characteristics of the 385 non-responders are unknown, which represents a potential source of bias. This study limitation was a consequence of the strict anonymization, which made it impossible to link the answers in the survey to individual user profiles in the system database. This in turn prevented the follow-up of the non-responders, so that we were unable to hypothesize about their reasons for non-response. One way to improve the design of a future study would be send different surveys to users who had previously been identified as active or inactive, and perhaps to differentiate the latter into those who had never logged in and those who had yet to submit or answer a case.

Another weakness of the study was the low response rate (30%). In principle, this might have been improved by extending the period for which the survey was open, but the trajectory of the responses received (Figure 1) suggests that 99% of responses
<table><caption>TABLE 6 | Benefits identified by referrers.</caption><thead><tr><th></th><th>No of responses</th><th>%</th></tr></thead><tbody><tr><td><strong>Patterson 2013 (67 replies)</strong></td><td></td><td></td></tr><tr><td>Happy to use the service again</td><td>59</td><td>88</td></tr><tr><td>Advice was helpful</td><td>58</td><td>87</td></tr><tr><td>Educational benefits</td><td>50</td><td>75</td></tr><tr><td>Cost savings</td><td>31</td><td>46</td></tr><tr><td><strong>Zolfo 2011 (73 replies)</strong></td><td></td><td></td></tr><tr><td>Helpful in establishing a diagnosis</td><td>30</td><td>41</td></tr><tr><td>Educational benefits</td><td>20</td><td>27</td></tr><tr><td>Literature collection</td><td>10</td><td>14</td></tr><tr><td>Reassurance to physician</td><td>9</td><td>12</td></tr><tr><td>Others (e.g., avoided patient transfer)</td><td>4</td><td>5</td></tr><tr><td><strong>Wootton 2004 (106 replies)</strong></td><td></td><td></td></tr><tr><td>Positive opinion of the service</td><td>106</td><td>100</td></tr><tr><td>Happy to use the service again</td><td>106</td><td>100</td></tr><tr><td>Advice was helpful</td><td>99</td><td>93</td></tr><tr><td>Advice changed or confirmed patient management</td><td>69</td><td>65</td></tr><tr><td><strong>Present study (64 replies)</strong></td><td></td><td></td></tr><tr><td>Educational benefits</td><td>31</td><td>93</td></tr><tr><td>Advice was helpful</td><td>30</td><td>91</td></tr><tr><td>Reassurance to physician or patient</td><td>25</td><td>81</td></tr><tr><td>Advice improved patient management</td><td>26</td><td>79</td></tr></tbody></table>

were received within 2 weeks, so that the cutoff at 30 days was, if anything, conservative. Nonetheless the response rate was lower than those observed in previous surveys of doctors in industrialized countries, which have been up to twice as high. On the other hand, a survey of the users of a telemedicine network in a low-resource setting had a response rate of 19% (7).

## User Feedback

It is clear that the MSF tele-expertise service was easy to use and provided clinically useful diagnostic and management advice to clinicians in the field. Most referrers reported that the advice received via the service improved their management of the patient (30/33). Similarly, the majority of specialists who provided an answer felt that their advice had improved the management of the patient (30/32). However, a similar number of specialists did not know about the value of their advice – presumably these were specialists who had not yet answered a case. The lack of feedback about patient follow-up was the main concern expressed by the specialist. This feedback can be considered as an implied contract and is crucial in keeping our experts motivated. Well-motivated experts answer promptly, and the feedback allows them to improve and fine-tune their advice. For this purpose, the system now sends progress reports automatically (from the referrer to the specialist) that includes follow-up data.

The specialists were asked “Was the information (including any images) supplied by the referrer adequate?” Of those who responded, 33% said no. Poor quality information submitted in a referral is a common problem in store-and-forward telemedicine networks generally, and in particular it is very difficult to ensure that good quality images are supplied (11, 12). Methods for improving the quality of the information in referrals include user education and structured referral templates.

## Specialist Workforce: Cornerstone of the System

Comments made by the volunteer specialists suggested that they were highly motivated, demonstrating a high level of user participation, despite the questionnaire being rather long (50 questions). Their comments suggest that being able to show solidarity with their colleagues in the field, together with the positive image of MSF worldwide, were important factors in recruiting and keeping them motivated. Indeed, by virtue of their commitment, they played a key role in the development and the implementation of the system. All the experts who were asked to join the MSF network accepted the invitation to do so, and after four years only one specialist decided to quit because of pressure of other work. It is clear that their support and the quality of their service were vital for the system development. Their answers and free-text comments demonstrate that they see the service as more than just a way of providing a second opinion. They are fully aware that they are also providing education, support, reassurance, and reducing the isolation of the field staff.

However, in their survey responses, some of them expressed frustration about not receiving enough cases. Part of the art in managing a telemedicine network is to maintain a large enough pool of volunteer specialists to be able to answer the range of queries occurring on the network, but to keep individual specialist workloads at a reasonable level. Too many cases are likely to lead to “consultant fatigue,” whereas too few cases may lead to specialists abandoning the network. It is worth noting that the majority of specialists considered that their status (i.e., acting as unpaid volunteers) was appropriate. In terms of desirable workload, three-quarters of them mentioned that they could routinely answer several cases per week.

## Barriers to Service Adoption

Both the analysis of the free comments and the large number of inactive users among the account holders emphasize that the successful implementation of telemedicine is fraught with difficulties, including technical, cultural, legal, financial, organizational, and political barriers (13). It is clear from the literature on technology adoption that one reason for the reluctance to adopt telemedicine is that it is sometimes perceived as a threat (14) with the risk of managers losing control. In contrast, it should more properly be viewed as a complementary tool, and people should be invited to take full advantage of it, even if this necessitates changes in working practices.

The high proportion of inactive users among those registered to use the service (59%) could have several explanations, such as the rapid turnover of field staff or specialists still waiting to receive a case. In our experience, it is not unusual in store-and-forward telemedicine networks to have a substantial proportion of inactive users.

Studies of technology adoption show that there are three main reasons for under-use (15). The results of the survey suggest that service use may have been reduced for two of these reasons:

1. Lack of knowledge about the existence of the service. This seems plausible because half of potential users were not
briefed before going to the field, and many users mentioned
poor communication or lack of promotion of the system.

2. Lack of interest in the service by potential users. This also seems plausible because the field doctors often have very high workloads, so that access to specialized advice for non-critical cases may be considered as a luxury. In addition, the turnover of staff is high: field missions commonly last only a few months.

3. Non-delivery when the service is requested. This seems unlikely because published performance indicators show that in fact the service is delivered rapidly and at high quality (4).

Comparison with Other Networks

A recent review identified seven store-and-forward telemedicine networks which were reasonably well-established (i.e., had been in operation for periods of five years or more) and which delivered teleconsultations to health care staff in low-resource settings (16, 17). Some of these networks have conducted user surveys, the results of which can be used as a comparator for the present study. In particular, previous surveys have asked referrers to identify the benefits of telemedicine (2, 6, 7). The results are summarized in Table 6. It is clear that referrers find these telemedicine services helpful in managing patients and in other ways, e.g., providing reassurance for patient and doctor. Nonetheless, it must be recognized that this represents low-level evidence and formal studies of the clinical and cost-effectiveness of telemedicine in low-resource settings are still awaited.

CONCLUSION

The MSF tele-expertise service is highly regarded by field users,
and the majority of specialists are satisfied with it. All users
recognize the benefits of providing access to specialist advice

in low-resource settings, where there is usually no other way of
obtaining specialist expertise.

Both referrers and specialists recognize the benefits of telemedicine in terms of better patient management, the provision of education, and the reduction of isolation in the field. Although access to specialist advice might not be considered as a priority in low-resource settings, it is clear from the user feedback that the MSF tele-expertise service provides an answer to a real need. Much of the success of the service rests on the efforts of the volunteer specialists, who not only provide the benefits of their special expertise, but demonstrate solidarity toward their colleagues in the field in supporting their education, reducing their sense of isolation, and improving the management of their patients.

Although there was general satisfaction with the service, the survey identified some specific problems. Some of these could be solved relatively easily, such as poor connectivity in certain locations, while others, such as the lack of feedback on patients, are more difficult. Perhaps the main challenge for building a sustainable service lies in the political dimension, since improving the adoption of the system requires a strong organizational commitment. The evidence base for the tele-expertise service now seems to be irrefutable, so it is to be hoped that further diffusion will occur more rapidly.

ACKNOWLEDGMENTS

We thank all the referrers and specialists who took part of
the study. We are particularly grateful for support from Isabel
Amoros, Eric Comte, Sophie Delaigue, Bertrand Draguez,
Kathrin Gottwald, Jarred Halton, Annette Heinzelmann, Jean-
Paul Jemmy, Daniel Martinez Garcia, David Olson, Pedro Pablo
Pablo, Jean Rigal, Micaela Serafini, Leslie Shanks, Saskia Spijker,
Raghu Venugopal, Sidney Wong, and Will Wu.

REFERENCES

1. Code de la Santé Publique – Article R6316-1. *Code de la Santé Publique*. (2015). Available from: http://www.legifrance.gouv.fr

2. Patterson V, Wootton R. A web-based telemedicine system for low-resource settings 13 years on: insights from referrers and specialists. *Glob Health Action* (2013) *6*:21465. doi:10.3402/gha.v6i0.21465

3. Wootton R, Wu W-I, Bonnardot L. Nucleating the development of telemedicine to support healthcare workers in resource-limited settings: a new approach. *J Telemed Telecare* (2013) *19*(7):411-7. doi:10.1177/1357633X13506511

4. Bonnardot L, Liu J, Wootton E, Amoros I, Olson D, Wong S, et al. The development of a multilingual tool for facilitating the primary-specialty care interface in low resource settings: the MSF tele-expertise system. *Front Public Health* (2014) *2*:126. doi:10.3389/fpubh.2014.00126

5. Mann K, Gordon J, MacLeod A. Reflection and reflective practice in health professions education: a systematic review. *Adv Health Sci Educ* (2007) **14**(4):595-621. doi:10.1007/s10459-007-9090-2

6. Wootton R, Youngberry K, Swinfen P, Swinfen R. Prospective case review of a global e-health system for doctors in developing countries. *J Telemed Telecare* (2004) *10*(Suppl 1):94-6. doi:10.1258/1357633042614177

7. Zolfo M, Bateganya MH, Adetifa IM, Colebunders R, Lynen L. A telemedicine service for HIV/AIDS physicians working in developing countries. *J Telemed Telecare* (2011) *17*(2):65-70. doi:10.1258/jtt.2010.100308

8. MSF Ethics Review Board. *MSF Research Ethics Framework – Guidance Document*. (2015). Available from: http://fieldresearch.msf.org/msf/handle/10144/305288

9. Soper DS. *Fisher's Exact Test Calculator for a 2x3 Contingency Table [Software]*. (2015). Available from: http://www.danielsoper.com/statcalc

10. Freeman GH, Halton JH. Note on an exact treatment of contingency, goodness of fit and other problems of significance. *Biometrika* (1951) **38**:141-9. doi:10.2307/2332323

11. Jakowenko J, Wootton R. An analysis of the images attached to referral messages in an email-based telemedicine system for developing countries. *J Telemed Telecare* (2006) *12*(Suppl 3):S3:49-53. doi:10.1258/135763306779380066

12. Delaigue S, Morand JJ, Olson D, Wootton R, Bonnardot L. Teledermatology in low-resource settings: the MSF experience with a multilingual tele-expertise platform. *Front Public Health* (2014) *2*:233. doi:10.3389/fpubh.2014.00233

13. LeRouge C, Garfield MJ. Crossing the telemedicine chasm: have the U.S. barriers to widespread adoption of telemedicine been significantly reduced? *Int J Environ Res Public Health* (2013) **10**(12):6472-84. doi:10.3390/ijerph10126472

14. World Health Organization. *Telemedicine: Opportunities and Developments in Member States: Report on the Second Global Survey on eHealth* 2009. World Health Organization (2011). Available from: http://www.who.int/goe/publications/ehealth_series_vol2/en/
15. Bradol J-H, Vidal C, editors. *Medical Innovations in Humanitarian Situations*. (2015). Available from: http://www.doctorswithoutborders.org/sites/usa/files/msf-medical-innvoations-book.pdf

16. Wootton R, Geissbuhler A, Jethwani K, Kovarik C, Person DA, Vladzymyrskyy A, et al. Comparative performance of seven long-running telemedicine networks delivering humanitarian services. *J Telemed Telecare* (2012) **18**(6):305–11. doi:10.1258/jtt.2012.120315

17. Wootton R, Geissbuhler A, Jethwani K, Kovarik C, Person DA, Vladzymyrskyy A, et al. Long-running telemedicine networks delivering humanitarian services: experience, performance and scientific output. *Bull World Health Organ* (2012) **90**(5):341D–7D. doi:10.2471/BLT.11.099143

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2015 Bonnardot, Wootton, Liu, Steichen, Bradol, Hervé and Wootton. *This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*