Summary of the Evaluation Study
１．Outline of the ProjectCountry: Indonesia

Project title: Tuberculosis Control Project in
the Republic of Indonesia

Issue/Sector: HealthCooperation scheme:
Technical Cooperation Project

Division in charge: Human Development 310 million yen
Department, JICA
Period(Three hundred and ten million yen)of (R/D)：October, 2008 - October, Partner country’s Implementing Organization:

Cooperation

2011

NTP,

Airlangga

University/Dr.Soetomo

Hospital, West Java Provincial Health Office
Supporting Organization in Japan: Japan
Anti-Tuberculosis Association (JATA)
1-1 Background of the ProjectIndonesia ranks fifth on the list of 22 high-burden tuberculosis (TB) countries in theworld. According to the World Health Organization’s (WHO’s) Global Tuberculosis
Control Report 2010, there were an estimated 292,754 new and relapse cases and an
estimated incidence rate of 189 cases per 100,000 population. TB is responsible for 27 of
the mortality (excluding HIV) per 100,000 in Indonesia in 2009.
DOTS (Directly Observed Treatment, Short course chemotherapy) strategy adopted in
1992 and has been expanded to 98 percent of the country by 2005. In 2007, DOTS
coverage reached 100 percent in Indonesia. Indonesia has exceeded WHO’s target of 85
percent for DOTS treatment success, reaching 91 percent. The DOTS case detection rate
has more than doubled over the past six years, from 30 percent in 2002 to 68 percent in2007 (in 2006, Indonesia surpassed WHO’s target of 70 percent for DOTS case detection
for the first time, reaching 73 percent). This was due to the collaboration of both public andprivate health care providers and the expansion of community-based TB care.
However, there remain challenges in providing quality DOTS in the country. With theexpansion of the DOTS coverage, it has been recognized that the diagnostic system for TB
case detection should be improved. In order to improve the TB case detection, quality oflaboratory services needs to be strengthened. In this backdrop, the Indonesian Strategic
Plan to Stop TB 2006-2010 maintains the necessity to improve laboratory services at alllevels through better internal and external quality control mechanisms, training andsupervision and updating of guidelines on TB control.
Based on the request of the Indonesian government, the Project started October 2008 withthree years cooperation period. The objective of the Project is to assure the quality of TB
laboratory services through strengthening of laboratory network. To achieve this purpose,
the Project aims to enhance the training capacity, assuring QA as well as the monitoringand supervision capabilities of laboratories at all levels. Considering the geographicalaspect of Indonesia, cascading system of training is adopted for enhancing the capacity oflaboratory personnel and improving the diagnostic system.

1-2 Project Overview＜Overall Goal＞
Quality National Tuberculosis Program (NTP) is sustainably managed.
＜Project Purpose＞
Quality laboratory service for TB is assured through strengthening of laboratory network atthe Project site for nationwide expansion.
＜Outputs＞
1) National TB Reference Laboratory on Human Resource Development (the National Training
Center of TB Microscopy Laboratories) is established at the Department of Microbiology,
School of Medicine Airlangga University/Dr. Soetomo Hospital Surabaya.

2) The Provincial TB Reference Laboratory functions well as a model of functional quality
assurance and training laboratory network in Project site (West Java Province).
3) Quality Assurance, Recording and Reporting amongst the diagnostic centers and District
Health Department including intermediate laboratories is strengthened as a model offunctional laboratory network in the Project site (West Java Province).
＜Inputs＞(1) Japanese side:
Short-term Expert: 6 persons
Trainees received in Japan: 5 persons
Trainees received in the Philippines: 6 persons
Equipment: 1,127,362,471IDR (approx. 10 million yen)
Local cost: 65,353(thousand yen)(2) Indonesian side:
Assignment of counterpart personnel
Land, building, and facilities necessary for the Project
Local cost included expenses for training, regular meeting and supervision
2. Evaluation TeamMembers

Team Leader: Dr. Mitsuo ISONO, Senior Advisor (health sector), JapanofInternational Cooperation Agency (JICA)

Evaluation

Evaluation Planning: Mr. Seiji KATO, Advisor, Human Development Department,TeamJICA
Evaluation Analysis: Ms. Junko SATO, TAC International Inc., Japan

Periodof 12th June 2011 – 23rd June, 2011

Type of Evaluation: Final evaluation

Evaluation
3. Results of Evaluation
3-1 Achievement
3-1-1 Achievement of the Project Purpose“Quality laboratory service for TB is assured through strengthening of laboratory
network at the Project site for nationwide expansion.”Project Purpose has been mostly achieved as shown in Indicator 1 and 2.
As for Indicator 1 “more than 70% of the laboratories of diagnostic centers
participate in the EQA under the SOP for EQA with LQAS sampling method including
monitoring /supervision,” 471 health centers (93%) currently participate on new EQA.
Regarding Indicator 2 “more than 70% of above mentioned laboratories report nomajor error,” the target has been almost achieved.

Out of 471 health centers, 324

health centers (69%) reported no major error as of the current month ofimplementation (Feb 2011).

If the denominator is 400 (health centers that weredirectly supported by the Project), the rate reached 81%.
3-1-2 Achievement of Outputs(1) Output 1 “National TB Reference Laboratory on Human Resource Development (theNational Training Center of TB Microscopy Laboratories) is established at theDepartment of Microbiology, School of Medicine Airlangga University/Dr.Soetomo
Hospital Surabaya.”Output 1 has been accomplished as all activities were conducted as planned and 4indicators were all achieved.
The core group was organized in the NTRL Training Unit, which is located in
Airlangga University, East Java Province, to manage capacity building activities
within the laboratory network. Among its major achievements was the developmentof training materials and curriculum, which are effectively utilized in the training
for master trainers for WASORs and laboratory technicians at the field level.
It has been observed that the core group was effective in the follow-up of training
held by the master trainers, however, the group’s capacity needs to be further
strengthened especially those concerning supervision activities to be able to sustain
their function, which is noted from the fact that some core members do not fully
practice course evaluation using training tools.
(2) Output 2 “The Provincial TB Reference Laboratory functions well as a model offunctional quality assurance and training laboratory network in Project site (WestJava Province)”
It is confirmed that planned activities have been implemented towards achievementof Output 2.
EQA Task Force Group formulated by staff from West Java Provincial Health
Laboratory and Provincial Health Office contributed to promoting EQA activities
such as trainings and supporting supervisory visits for and with the district level.
The Project assisted in developing SOP, which is utilized as a training material.

InWest Java, a total of 364 laboratory technicians, 176 WASORs and 33 controllers
were trained using SOP.

In addition, EQA coverage reached 100%(26/26) at districtlevel, 100% for intermediate laboratories (24/24) and 93% (471/508) at health centerlevel.

The Project has been trying to increase opportunities to communicate with theIndonesian counterparts as well as other international partners for the effective
implementation of project activities.

For example, EQA annual report meeting

serves as a venue to give feedback on the progress of EQA in pilot and expansion
areas and discuss relevant issues with Indonesian counterparts and otherstakeholders.
(3) Output 3 “Quality Assurance, Recording and Reporting amongst the diagnostic
centers and District Health Department including intermediate laboratories isstrengthened as a model of functional laboratory network in the Project site (WestJava Province)”
Output 3 has been largely accomplished.
The Project has provided technical guidance in the form of supervisory visits andregular meeting to achieve sustainable technical proficiency at the field level.
Steady progress for quality assurance can be seen in an increasing trend of healthcenters without major errors.
It has been also observed that an increasing number of intermediate laboratories
which are responsible for cross-checking conduct supervision for EQA under theSOP.

Twenty one (21) out of 24 intermediate laboratories (88%) conductedsupervision for 187 health center from February 2010 April 2011, while only one outof 18 intermediate laboratories (5.6%) conducted supervision under the SOP at thetime of mid-term review.

The Project will prioritize supervision for districts oncross-checking to further strengthen their capacity in the remaining period.
3-1-3 Implementation ProcessMajor promoting factors that promoted realization of effects are 1) technicaltransfer method and 2) commitment to the Project from the Indonesian andJapanese sides.
As for technical transfer method, the Projcet has successfully operated on capacity
building for WASORs and laboratory technicians at different levels by enhancing
thecascades effects of training system.

It has been also achieved by ensuring theirenabling environment in the combination with the development of SOP/training
manuals and the provision of equipments.

In addition,the Project has provided

opportunities to share the knowledge and experience on new EQA amongstakeholders.
The Project conducted EQA annual report meetings based on the“Annual/semi-annual EQA Report on TB Microscopy in West Java”, which wasinitially published in 2010.

The publication aimed to give feedback, shareproject
experiences and serve as reference for stakeholders as they tackle EQA

implementation in their respective areas.
Both the Indonesian side and the Japanese side have strong commitment to theProject. Particularly, the strong initiative taken by West Java provincial healthoffice/provincial health laboratory accelerated the EQA expansion in the entire
province in the second half of the Project.
On the other hand, suspension of approval on the TB Laboratory Network waspointed out as factor that inhibited realization of effects at the time of mid-term
review, and the statement regarding the Regional Reference Laboratory (RRL)
was deleted from PDM.

Shortage of budget allocation for operational

expenditure at provincial and district level was also mentioned as constraints atthe time of mid-term review.

Although budget for operational expenditure atprovincial, district and peripheral levels remains unstable at the time of final
evaluation, positive indication for sustainability has been observed from the factthat budget allocation for training and supervision by the Indonesian governmenthas been steadily increased.
3-2 Evaluation by Five Criteria(1) Relevance
Relevance of the Project is still high at the time of final evaluation.

1) The Project is in line with global health policies such as the Global Plan to Stop TB
2006-2015, which emphasizes the quality TB control services.
2) The Project is also consistent with national health plan of Indonesia.
prioritized in the Strategic Plan of MOH.

TB control isIn particular, it has been conducted in linewith “Five Years Strategic Plan for TB Control (2006-2010),” which addresses the needto improve laboratory services through internal and external quality control.
3) Project Purpose and Overall Goal have also consistency with Japan’s ODA policies forIndonesia, which prioritize the infectious disease control including TB.

4) The Project approach focusing on capacity building for laboratory personnel andestablishment of an external quality assurance (EQA) model for TB microscopy
responds to the needs for strengthening case detection by improving laboratory services.
(2) Effectiveness

Effectiveness of the Project is very high as three Outputs have been successfully achieved,
which contributed to the realization of Project Purpose.
1) Effectiveness has been recognized particularly in capacity building for WASORs andlaboratory technicians by enhancing the cascade effects.

With consultation from theproject, the core group developed the training materials/curriculum and initiated related
activities to improve the training environment.
2) The Project successfully established an effective external quality assurance (EQA)
model for TB microscopy in West Java.

Standard Operational Procedures (SOP)

developed by the Project gave clear instruction and proved to be very effective topractice new EQA activities.
3) Another remarkable achievement by the Project is conducting EQA annual reportmeetings based on the “Annual/semi-annual EQA Report on TB Microscopy in WestJava.”
4) Intensifying the monitoring and supervision is also contributing factor to expand newEQA in the entire province.

Active involvement of controllers in supervision led tosteady progress for quality assurance, which can be seen in the increasing trend of healthcenters without major errors.
5) For nationwide expansion of EQA, developing national guideline and institutionalizing
training system are needed.
(3) Efficiency

Overall, the Project has been efficient in terms of quality, quantity and timing of theprovision of most inputs being adequate and utilized for the achievement of the Outputs.
1) Inputs by the Japanese side
Japan Anti-Tuberculosis Association (JATA) was assigned as an implementing
institution of the Project. Research Institute of Tuberculosis (RIT)/JATA has a wide
experience in human resource development in the field of TB control through its
international cooperation, which makes it easier to dispatch technical experts to theProject. RIT has also conducted trainings and accepted many participants from abroad. It
is very meaningful for the Project to make full use of resources in JICA TB control
Project in the Philippines by dispatching a lecturer from the Philippines and sending
Indonesian counterparts to the “Study Tour to the Philippines for TB LaboratoryNetwork.”

Indonesian counterparts are also satisfied with training and equipments

granted by the Japanese side.
2) Inputs by the Indonesian sideThe Indonesian side allocated experienced counterparts and provided the office space/necessary equipment to the Project.

In addition, the Indonesian side shared the costsof training, quarterly meeting and supervision, which are essential for the Projectimplementation.
3) Cooperation with other partners
Cooperation with other partners enabled effective and efficient implementation of theProject.

For example, project experts participated in district level regular meeting

supported by KNCV, and gave lecture on new EQA.

EQA annual report meeting

serves as a venue to give feedback on the progress of EQA and discuss relevant issues
with Indonesian counterparts and other stakeholders.
(4) Impact

Several positive impacts have been observed.
1) The Provincial TB Reference Laboratory in West Java is in the process of being
designated National TB Reference Laboratory.

2) Training materials were adopted as national standards for TB microscopy training byNTP, MOH.
3) Training materials for laboratory technicians are made available nationwide through
MOH homepage.
(5) Sustainability

Sustainability is expected to be high if NTP and other relevant organizations continue
quality laboratory service for TB by utilizing trained human resources, transferredtechnology, developed system and equipment provided by the Project.
1) Indonesian government commitment to TB control is expected to be sustained, which isin line with Next Five Year Strategic Plan 2010-2014 with the theme “Breakthroughtoward Universal Access.”2) Organizational sustainability is high.

If the Provincial TB Reference Laboratory inWest Java is officially designated National TB Reference Laboratory, it is important forit and national Training Center for TB Microscopy at Airlangga University to cooperate
to provide quality laboratory service for TB.
3) Financial sustainability is not very high but expected to some extent.

Although

NTP/MOH receives the support for TB activities from Global Fund, budget for
operational expenditure at Provincial, District and Peripheral levels remains unstable.
It is important that NTP work for proper management of the budget to support EQA
expansion nationwide.
4) For technical sustainability, roles of skilled supervisors are essential in providing
technical support and supportive supervision through regular supervision.

In thatsense, active involvement of controllers in supervision should be continuouslyencouraged.

It is also expected that training system, training materials, SOP developedby the Project are utilized for expanding EQA nationwide.
3-3 Conclusion
By the strong commitment of both the Indonesian and Japanese sides, the Project activities
have been accelerated after the mid-term review upon mutual understandings for the Project
purpose. As a result, the Project has for successfully implemented all planned activities, andhas not only produced visible outcomes by means of the effective Project approach andefficient inputs, but also brought certain impacts in the related fields of TB control programin Indonesia. The team has also observed the strong initiatives and ownership of theIndonesian side those are crucial for sustaining continuous implementation of the Projectoutputs after the end of the Project.
The Project has worked in the limited area of the country, but now, the Indonesian side isplanning to apply the achievements of the Project to nationwide. Thus, despite short term of
the implementation, the team is sure that the Project might bring maximum effects by theideal way of technical transfer.
3-4 Recommendation
1) Although there has been significant and rapid expansion of the new EQA system in Westjava Province, still there are certain PRM/PPM to be covered by this system. Thus, TheProject is recommended to expand the current EQA system to PRM/PPM as much aspossible in the remaining period. Considering sustainability, the West Java Provincial
health laboratory is recommended to take initiatives in this expansion and the Japaneseside is recommended to support technically.
2) To enhance significance of the new EQA system in West Java province, there are several
points to be considered. Among them, it is crucial that there are high incidences ofmajor errors in some districts. Thus, the Project is recommended to analyze the situationin those districts and develop plans to reduce major errors. Also, reporting system on
EQA should be strengthened and the Project is recommended to improve timely
reporting.
3) Now, NTP and the West Java Provincial health laboratory, along with stakeholders,started to develop the SOP for the new EQA system aiming at nationwide application.
Thus, the Japanese side is recommended to support technically so that the SOP will befinalized within the remaining period.
4) Now, the West Java Provincial health laboratory is planned to be up-graded to theNational Reference Laboratory (NRL) for AFB sputum smear examinations. Thus, it isrecommended to start to prepare for the functional organization as NRL including
human resources. The Japanese side is recommended to provide advices from technicalpoints of view in this regard.
5) The School of Medicine Airlangga University is recommended to utilize equipment
provided by the Project. Now, the training mechanism for personnel who belongs to
public health sector is fully managed by the Ministry of Health. Thus, it is recommended
to develop the plan to work as the training institute for personnel in other sectors such asprivate hospitals.
3-5 Lessons Learned
Despite short term of the implementation, the Project has made significant achievement andimpacts. Among several components of the successful implementation, following
points are extracted as positive lessons learned.
 Developing the cascade system for trainings was very useful to develop huge number ofhuman resources in the short term and establish sustainable training system. Also, as
judge the facts that all trained personnel at the bottom of this cascade passed theexamination, the quality of the training was also ensured by this system.
 The Project has tried to provide opportunities to share the knowledge and experience on
new EQA among stakeholders, as much as possible, including EQA annual reportmeetings and publishing the “Annual/semi-annual EQA Report on TB Microscopy inWest Java”. These activities helped to increase motivation among related staffs in WestJava provinces, resulting expansion of new EQA by their own ownership.