Summary of Terminal Evaluation
I. Outline of the ProjectCountry: Kingdom of Cambodia

Project title: The Project For Strengthening of Medical
Equipment Management at Referral Hospitals

Issue/Sector: Health

Cooperation scheme: Technical Cooperation

Division in charge:
JICA Cambodia OfficeTotal cost: 409,519,000 Japanese Yen (=US$ 3,992,191
US$1=JPY102.58, June 2014, JICA rate)

Period of
Cooperation

Partner Country’s Implementing Organization:
Ministry of Health(R/D): 2009/11/22-2014/11/21

Supporting Organization in Japan: - Estrella Inc. andJapanese Organization for International Cooperation in
Family Planning (JOICFP)
1. Background of the Project
Based on the Health Sector Strategic Plan, the Ministry of Health (MOH) is currently endeavouring to
improve the health service delivery in Cambodia, as one of six key areas of priority works. As a strategy,
MOH is committed to “further improve coverage and access to health services especially for the poor andother vulnerable groups through the planning and location of health facilities and strengthening the outreachservices.
As one of the measures to implement the strategy, MOH launched the Health Sector Support Project(HSSP, currently pursuing HSSP2 2008-2015) since 2003. HSSP includes soft-components and renovation
and/or construction of new hospital buildings with medical equipment (ME). At the same time, it was also
widely recognized that due consideration to management and maintenance of medical equipment aimed at
keeping them in good working condition for many years, was critical.
However, existing ME at public hospitals suffer from multiple difficulties; many of them are old, broken,
users do not know how to operate appropriately, there is no human resources that can diagnose the failure ofME, spare parts are not easily available, and financing for replacement is also difficult to obtain. Many of the
ME of the public referral hospitals have been donated by development partners, which may make it difficult
to implement standardized ME management across the country. In addition to the technical capacity for ME
maintenance and repair, management capacity for identifying and solving problems had not been sufficiently
developed in the country.
JICA-Cambodia Technical Cooperation project on promotion of Medical Equipment Management system(MEDEM-1) enhanced the cooperation between technical and management departments and their capacities attarget Complimentary Package of Activities 3 / National Hospital (CPA3/NH) and introduced ME management
system, by focusing on ME preventive maintenance and minor repair. This has been done by successfully
exploiting resources and facilities which had been strengthened by Japanese Official Development Assistance,
namely the medical engineering section of National Maternal and Child Health Centre (NMCHC). The project
also strengthened Hospital Services Department’s (HSD) administrative capacity for supervision and support
through report submitted from hospitals and monitoring visits. ME Management Networking Group (MEMWG) consisting of management and technical staff was established at the target hospitals by MEDEM-1.
They pursued to identify ME conditions and reﬂect ME maintenance budget into Annual Operation Plan
(AOP) to solve problems as their ME management activities. These activities resulted in outputs such asxi

implementation of the ME maintenance and inventory at hospitals, and establishment of a system for reportingand monitoring/instruction and for sharing the understanding on ME maintenance matters between hospitals
and MoH.
While recognizing these plausible achievement, the Preparatory Survey Mission Team for the Project forStrengthening Medical Equipment Management at Referral Hospitals (MEDEM-2 ) re-identified that there
are still a number of challenges at CPA3/NH; lack of technician and maintenance workshops, difficulties
in motivating MEM-WG to commit in preventive measures while their expectations are rather for moretangible results such as repairing ME, difficulties in securing sufficient fund for spare parts, and some target
beneficiaries of the Project are too busy to engage in ME management activities. At the same time, it wasobserved that good ME maintenance performers have favorable relationship among hospital staffs and with
Provincial Health Department (PHD). The difference among hospitals’ progress in ME maintenance actives
can be explained by these various factors.
This Project is to extent the outputs of MEDEM-1 into CPA2 hospitals. ME of CPA2 hospitals are in moredifficult conditions than the CPA3/NH because they lack; (1) human resource that can understand and carryout ME preventive maintenances, (2) finance for purchasing spare parts and new ME, (3) facilities including
ME maintenance workshop, (4) basic information necessary to solve ME related problems, and (5) leadershipof the management team.
2. Project Overview
The Overall goal of the Project is to institutionalize the Cambodian Medical Equipment Management
System to expand to all of the CPA2. The project aims NWT, CPA3 and National Hospitals (NH) and CPA2
to implement ME management in cooperation.
(1) Overall Goal
Cambodian Medical Equipment Management System is institutionalized and expanded to non-target CPA2
(2) Project Purpose
Cambodian ME Management System is established, and National Workshop Team (hereinafter referred
to as “NWT”), the target CPA3/NH and CPA2 implement medical equipment management activities incooperation.
(3) Outputs(1)	(1)ME management system covering maintenance, minor repair and ME allocation and budget
planning is institutionalized at the target CPA3/NH.
(2)	(2)ME management activities at Lead-CPA3 (L-CPA3)/NH as the core of the Cambodia ME
management system are further strengthened and their capacities to instruct CPA2 are enhanced.
(3)	(3)Basis of the network for consulting, Supporting and supervising ME management activities among
NWT- L-CPA3/NH- the target CPA2 is formed.
(4) NWT’s capacity to supervise ME management is strengthened.

xii

(4) Inputs (at the time of evaluation)
Japanese side:
No. of long-term Experts: 4
No. of trainees at technical exchange in Sri Lanka : 40
No. of short-term Experts: 8
Equipment: 39,818US$
No. of trainees received: 3
Local cost: 37,122 thousand Japanese Yen (365,087US$)
Cambodian side
No. of counterparts: 4 Project management team, 9 National Workshop Team (NWT). At the hospital level,
98 personnel from 22 CPA3/NH, 59 personnel from 26 CPA2, and three from a CPA1 as Medical Equipment
Management Working Group (MEM-WG) of each hospital.
Land and Facilities: Office Spaces
Local Cost:
II. Evaluation Team
Member of
Evaluation
Team
Period of
Evaluation

Mr. Hiroshi Takeuchi

Leader, Senior Representative, JICA Cambodia

Ms. Aya Mizusawa

Evaluation Planning, Senior Program Officer, JICA Cambodia

Ms. Kyoko Sakurai

Health Planning, Program Officer, JICA Headquarters

Ms. Chie Tsubone

Evaluation Analysis, Global Link Management
Type of Evaluation: Terminal Evaluation

From May 26 to June 13, 2014

III. Results of Evaluation
3．Project Performance
3-1. Outputs(1)	Output 1: ME management system covering maintenance, minor repair and ME allocation andbudget planning is institutionalized at the target CPA3/NH ⇒ Mostly achieved
The indicator 1-1 and 1-4 have been fully achieved, and 1-2 and 1-3 have been mostly achieved. It is
possible to achieve these indicators fully by the end of the Project if appropriate follow-up activities are
conducted. The Joint Evaluation Team also found many improvements at the hospital level such as that thecondition of each ME is more properly recorded and understood by MEM-WG, ME are more properly andregularly maintained, ME users use ME more properly, and life spans of ME became longer.
● Indicator 1-1: All CPA3/NH prepare ME replacement and maintenance plan, and reﬂect the required cost
into AOP.
⇒In 2014, all the 18 CPA3 hospitals budgeted these activities in AOP 2014. As National Hospitals now
have financial autonomy, they do not develop an AOP anymore, but plan the activities in their own annual
plan.
● Indicator 1-2: ME management monitoring score of the target CPA3/NH exceeds 80%⇒Regarding MEM Report Score, all the 22 hospitals have reached the target level. As for MET
Monitoring Score, 20 hospitals reached the target in the first semester of 2014 while one failed to reachthe target. Siem Reap CPA3 has not been visited for this semester, but reached the target in the second
semester of 2013. With regard to MEM Monitoring Score, 21 hospitals have achieved the target as of thefirst semester of 2014. Siem Reap CPA3 has not been visited for this semester, but it reached the target
in the second semester of 2013.The average scores of 22 CPA3/NH have already exceeded 80% for all
the monitoring aspects.

xiii

●
●

Indicator 1-3: Unnecessary ME is identified and recorded in the disposal list.
⇒As of the second semester of 2013, 16 hospitals out of 22 sent their ME disposal list to NWT.
Indicator 1-4: Meeting with departments/sections/wards related to ME management at each CPA3/NH isheld at least 2 times/year⇒All the 22 target CPA3/NH have been conducting a meeting twice per year.

(2)	Output 2: ME management activities at L-CPA3/NH as the core of the Cambodia ME management
System are further strengthened and their capacities to instruct CPA2 are enhanced.⇒Mostly
Achieved
All the indicators are achieved. However, some challenges still remain in the quality and accuracy ofactivities conducted by some L-CPA3/NH. Therefore, it is still important to continuously improve the quality
and accuracy of activities conducted by L-CPA3/NH.
● Indicator2-1: MEM-WG of all L-CPA3/NH receive TOT (consultation, support) training.
⇒The project has conducted 15 TOT courses for CPA3/NH. As a result, the capacities of six L-CPA3/
NH have been strengthened; six L-CPA3/NH’s monitoring scores are higher than the other 16 CAP3
hospitals.
● Indicator 2-2: ME management monitoring score of the target L-CPA3/NH exceeds 90%.
⇒As of June 2014, all the six CPA3/NH reached the target level for all the monitoring aspects except for
Siem Reap CPA 3, which has not been monitored in 2014. However, it reached the target in the second
semester of 2013.The average scores of the three monitoring aspects have exceeded 90%.
● Indicator 2-3: Score to measure L-CPA3's consultation ability for the target CPA2 is improved.
⇒The comparison of pre- and post-test scores of ME management network training forL-CPA3 as
well as the progress of ME monitoring scores and ME operable rate of CPA2 demonstrate that theconsultationabilityconsultation ability of L-CPA3 has been developed.
(3)	Output 3: Basis of the network for consulting, supporting and supervising ME management
activities among NWT-L-CPA3/NH-the target CPA2 is formed.⇒ Mostly Achieved
The indicators were fully achieved. Meanwhile, the scores of newly participated 12 CPA2 are relatively
weak. Also, the answers from CPA2 MEM-WG members to the questionnaire of the Joint Evaluation Teamshow that around half of the 43 members think that supports from CPA3 are still insufficient. Similarly, around
one third of them do not think that the ME management network is functioning sufficiently.
● Indicator 3-1: MEM network guidebook was developed and NWT-L-CPA3/NH-target CPA2 activates its
network functions by March 2013
⇒The operation manual of ME management network finalized in August 2013. The contents were revisedin January 2014. The pilot network started its function in December 2011.
● Indicator 3-2: All target CPA2 submit MEM report including ME inventory to respective L-CPA3, and all
L-CPA3 submits MEM report of their CPA2 and correspondence record to NWT.
⇒As of June 2014, all the 22 CPA2 hospitals submitted their MEM report and correspondence record forthe first semester of 2014 to respective CPA3, and these were submitted to NWT.
(4) Output4: NWT capacity to supervise ME management is strengthened.⇒ Mostly Achieved
Output 4 has been mostly achieved. Meanwhile, some NWT members hardly participated in the project
activities. This fact limited the achievement level of Output 4.

xiv

●
●

●

Indicator 4-1: Necessary governmental documents are developed by March 2012
⇒Seven documents including manuals, guidelines and policy and strategic documents were developed.
Indicator 4-2: Budget for activity of NWT is allocated after 2012
⇒In total, 270,475.43 USD was requested by HSD through AOP for the project activities, and 45,320
USD was made available.
Indicator 4-3: NWT regular meeting is held at least 4 times/year.
⇒The meeting was conducted five times in 2011, four times in 2012, and four times in 2013. In 2014,
two meetings were held so far in January and April.

3.2 Project Purpose
Project Purpose: Cambodia ME management System is established, and NWT, the target CPA3/NH and
CPA2 implement medical equipment management activities in cooperation⇒ Mostly Achieved
Indicator 1 and 3 have been fully achieved. Though Indicator 2 is achieved, answers to the questionnaire
distributed by the Joint Evaluation Team show that almost half of CPA2 are not satisfied with supports from
CPA3, and one third do not think that the network is functioning sufficiently. This is because the network
activity has started only in 2013, and most of the CPA3 do not have sufficient human and financial resources
to visit and assist CPA2.
● Indicator 1: ME Operable rate at the all target CPA3/NH exceeds 90% and target CPA2 exceeds 80%⇒Regarding CPA3/NH, all the hospitals reached the level in the second semester of 2013. As for CPA2,
all the 10 original target CPA2 have attained the required level. The average ME operable rate of 22
CPA3/NH in the second semester of 2013 was 95.6%, and that of 22 CPA2 was 87.7% in the first semesterof 2014.
● Indicator 2: Ratio of appropriate responses by the L-CPA3/NH's to requests from the selected CPA2 forsupport exceeds 60%⇒While the average score of correspondence record of the second semester 2013 was 46.9%, it was
improved to 71.1% in the first semester of 2014. Among the original 10 target CPA2, one hospital did notreach the attainment level. Meanwhile, the Joint Evaluation Team found that almost half of CPA2 are notsatisfied with consultation, supports and supervision of CPA2.
● Indicator 3: Training for remained CPA3/NH to be L-CPA3/NH starts.
⇒ The Project started to include all the CPA3 and CPA2 in September 2013, and two training courses
were provided on ME management network in September 2013 and January 2014.
3-2. Summary of Evaluation Results(1) Relevance: HighThe Project is relevant to the Cambodian policy as increasing investment in medical equipment andimprovement of support services including management and maintenance is one of the policy directions of
the Health Strategic Plan (2008-2015). Also, “developing procedures and follow-up systems which ensure
a placement of appropriate medical supplies, basic equipment and infrastructure” is one of the strategicinterventions in the Plan.
The Project is relevant to the Japanese aid policy for Cambodia as it includes promotion of social
development as one of the assistance areas. “Enhancement of Health and Medical Care” is one of thedevelopment issues in the area. JICA’s Rolling Plan places the project under this development issue.
The Project has been relevant to the needs of the government and hospitals because the country did not

xv

have an ME management system, and there were no other donors assisting in this field.
Establishing a network among MOH, CPA3 and CPA2 was appropriate as knowledge accumulated in
NWT and CPA3can be shared with CPA2. Also, the involvement of stakeholders from the central, provincial/
district and hospital levels was appropriate to ensure provision of mutual feedback.
(2) Effectiveness: Moderately highThe prospective to achieve the Project Purpose is high. Also, the Joint Evaluation Team confirmed many
improvements at the hospital level through interviews and questionnaires.
The logic of the Project was mostly clear and effective. Meanwhile, the contribution of Output2 to theachievement of the Project Purpose was relatively limited because some of the six L-CPA3 had challenges in
the quality of activities, there were other CPA3 which were more active than the six L-CPA3, and the Project
included all the CPA3 and CPA2 from 2013.
Regarding training, seminars and workshops, the effectiveness seems to have varied from one participant
to another. The training was sometimes too difficult for the technicians who do not have sufficient preliminary
knowledge and skills. Also, several people indicated that the teaching methodology used in the sessions was
not very interactive, thus not conducive to learning.
The project team developed various manuals and guidelines. They were helpful to disseminate information.
Meanwhile, as many documents were developed and revised, some target hospitals were confused about which
version they should refer to. Moreover, due to the revisions, some documents have become longer and more
complex. Therefore, they were sometimes difficult to understand for ME technicians with limited capacities.
(3) Efficiency: Moderately high with a few issues in inputs
As many as 1,800 persons have benefitted from various training and seminars conducted by the Project.
Also, though the initial target was 32 hospitals (86 personnel), the Project successfully reached to 50 hospitals(128 personnel) without budget increase.
Meanwhile, due to the four-month interval in 2012 and increase of target hospitals, many activities wereconcentrated in 2013, and it made the project team difficult to conduct regular monitoring.
Human resources of MOH and CPA3/NH, and experiences strengthened during MEDEM-1 were optimized
for the Project. Meanwhile, because of the increase of target hospitals and decrease of active NWT members,
the workload of remaining NWT members increased.
Two different Japanese short-term experts were dispatched under the same task in 2013. Due to different
opinions of the two consultants, the project team needed to revise and reprint the manual, which had been
finalized only a few months before.
The training preparation, arrangement and communication had room for improvement. Meanwhile, themembers learned advanced ME management in Japan, and it was beneficial to develop the draft policy
document and long-term strategy. 5S training in Sri Lanka was eye-opening to the participants, and theylearned what they can apply in their hospitals immediately.
Regarding technical transfer, NWT members learned from Japanese experts not only the contents of ME
management, but the procedure to revise manuals and other related documents.
The Project has made a Memorandum of Understanding (MOU) with HSSP2. The Chief Advisor assisted
HSSP2 in developing specifications of ME. Moreover, this MOU facilitated obtaining budget from the HSSP2
fund. Also, there were some cases in which the Project assisted JOCVs dispatched to target hospitals. The
Project also had collaboration with JICA’s grant scheme, which was targeted to the target hospitals of thexviProject.
(4) Impact: Moderately HighOverall Goal: Cambodia Medical Equipment Management System is institutionalized, and it is also
expanded to non-target CPA2
The achievement level is already high because the Project has scaled up and included all the CPA2. The
Overall Goal is likely to be achieved in three to five years after the termination of the Project if appropriate
supports to continue the activities are provided by MOH.
● Indicator 1: MEM-monitoring score of all CPA2 exceeds 60%⇒21 out of 22 hospitals reached the level for both monitoring aspects as of the first semester of 2014. The
average MEM Report Score was 90.2% and that of MEM Monitoring Score was 80.0%.
● Indicator 2: ME Operable rate at the target CPA3/NH exceed 95% and all CPA2 exceed 80%⇒Regarding CPA3, 15 out of 22 hospitals reached the level as of the second semester 2013. As for CPA2,
18 out of 22 hospitals reached the level as of the first semester of 2014. The average ME operable rate of
CPA3 was 95.6% in the second semester of 2013 and that of CPA2 as of the first semester of 2014 was
87.7%.
● Indicator 3: Operable mean time of essential ME becomes longer⇒The project team utilized two alternative indicators, which are (1) comparison of CPA3/NH’s average
operable rates of essential 10 ME of 2010 with those of 2013, and (2) comparison of actual ME operation
condition of each hospital with “standard index”, developed by the project team referring to the ME
guideline for CPA3 set by MOH. As for (1), all the hospitals increased the average operable rate except
for one hospital. Regarding (2), in 2010, 70 ME were below the index in 2010 while 56 ME out of 220
are below the standard index in 2013. Meanwhile, it is more reasonable to conclude that the improvement
is attributable not only to the Project but also to the provision of more equipment by HSSP2 during the
project period.
As an unexpected positive impact, it was reported that a CPA 3 has been assisting CPA1 even though CPA1
is not included as a target of the Project. Moreover, having been inspired by the 5S/KAIZEN/TQM, another
CPA3 constructed a laundry room for patients in order to improve quality of service by asking for donation.
(5)	Sustainability: Medium but if MOH retains NWT, and allocates appropriate budget for theiractivities, higher sustainability can be expected.
Policy Aspect: The project team drafted an ME policy and long-term strategy document. The project teamaims to finalize it and obtain authorization by MOH by the end of the project period.
Financial Aspect: In Total, 45,320 USD was made available from the Cambodian side for training and seminars.
Though MOH expressed the will to make efforts to secure budget for future activities, it is necessary to makemore concrete plans to secure future budget. Meanwhile, it is expected that securing budget will be facilitatedonce the policy and strategic documents are approved. Moreover, it is a big challenge for hospitals to secure
funds for ME repair and transportation fee to visit other hospitals for guidance and consultation.
Institutional Aspect: It has not been confirmed yet if the mandate of NWT will be effective after the terminationof the Project. MOH will seek to re-establish NWT so that activities can be continuously implemented. Also,
it will be important to seek more supports from PHD and OD so that PHD and OD can supervise the activities
properly after the project period, and provide hospitals with necessary assistance and guidance. Moreover,xvii

one third of ME technicians targeted are not civil servants; they are contract or temporary staff. Therefore,
there is a possibility that they quit the current position once they find another stable job. Another issue is about
broken ME. In order to dispose of broken ME, it is necessary for hospitals to contact MOH and Ministryof Economics and Finance (MEF), but the procedure is complicated. Therefore, the hospitals have been
accumulating broken ME in storage for several years, and they do not have any more space to store them.
Technical Aspect: NWT will be able to conduct monitoring and follow-up activities. They will be able torevise manuals when necessary. It is expected that their training methodology is improved further. It is likelythat most of the CPA3/NH can continue ME management activities. As for CPA2, newly involved hospitals
demonstrate relatively low capacities as we can observe from their monitoring scores.
3-3. Factors that promoted realization of effects(1) Planning
・Relatively longer duration of JICA assistance since the first phase(2) Implementation Process
・Regular monitoring of the project activities and indicators by scoring ME management situation of hospitals
and PCM workshops
3-4. Factors that constrained realization of effects(1) Planning
・Change of the contract with Japanese experts during the project period(2) Implementation Process
・Difficulty to uniformly address ME technicians whose capacities differ greatly through group training or
seminars
3-5. Conclusion
The Project mostly achieved all the Outputs and the Project Purpose. It will be possible to fully achieve
the Project Purpose by the end of the project period if the project team provides appropriate follow-up
activities. Therefore, the project will be terminated as planned by the end of November 2014. Regarding anevaluation by five criteria, relevance is high, and effectiveness, efficiency and impact are moderately highwhile sustainability is medium.
3-6. Recommendation(1)	MoH to work with Ministry of Economy and Finance (MEF) to proceeds on disposal of the broken ME
from the state property list.
(2)	MoH to work with PHD and OD officials to strengthening their commitment on ME management for
reinforcing the ME network system at provincial level.
(3)	JICA to take more active role in sharing the progress of the project with other partners through channels
such as health partners meeting, TWG-Health.

xviii

3-7. Lessons Learned(1)	Effective use of existing resources such as NTW framework for smooth take-off and rapid expansion of
the network system.
(2) Collaboration with other development partners’ scheme such as, HSSP2 for staff training, procurementof medical equipment and tools under official agreement for collaboration.
(3) Create a system to encourage participation and ownership of the counter parts through; (a) developmentof reporting format, (b) periodical submission of the reports, (c) review and follow-up of the reports by
NWT and provincial network, (d) publicize the ranking of target hospitals using report scores.

xix