Summary of Final Evaluation
1. Outline of the ProjectCountry:
Islamic Republic of Pakistan

Project title:
The Tuberculosis Control Project in the Islamic
Republic of PakistanIssue/Sector:
Health and Medical Sector

Cooperation Scheme:
Technical Cooperation Project

Division in charge:
Infectious Disease Control Division, Health
Human Resources and Infectious Disease
Control Group, Human Development
Department

Partner country’s implementing organization:a. National Tuberculosis Program (NTP) , Ministry of
Healthb. Provincial Tuberculosis Program (PTP), Directorate
of Health Services, Punjab Province

Period ofcooperation

Suｂ-contracted Organization:
Japan Anti-Tuberculosis Association(R/D):
April 1, 2006 – March 31, 2009

1.1 Background of the Project

After WHO’s declaring TB as global emergency in 1993, the Government of Pakistan endorsed the
DOTS strategy and revised its national TB control policy in 1994, and in 1995 DOTS pilot Project was
launched in five sites. Through decentralization of DOTS implementation in 1998, each province
started to take responsibility of DOTS implementation under the federal NTP guideline.
In 2002, according to the Pakistani official request, JICA started to dispatch experts to NTP for
effective DOTS implementation in 4 districts of Punjab province (Faisalabad, Gujrat, Lahore andMultan) where DOTS was launched in 2004. Achieving full DOTS coverage in 2005, in front of thechallenge which is to assure the DOTS quality, the Government of Pakistan has implemented the “TB
Control Project” for three years from 2006 in cooperation with JICA.
In this Project, NTP, PTP Punjab and JICA have been working for the capacity building which aims todevelop technical and managerial capacity of NTP and PTP Punjab through strengthening TB program(quality DOTS) in model districts (Faisalabad, Gujrat, Lahore and Multan). The Project disseminates
the quality DOTS for all Punjab Province as well as in NTP in the light of the Project Objective“Quality National TB Control Program (NTP) is systematically implemented in close collaboration
with provincial and district TB units.”
The Project started on April 2006 and will be completed on March 31st, 2009.
1.2 Project overview
The Project has been conducted based on the PDM (Annex 1) approved in the R/D on 1st April 2006. Its
main points are as follows:(1) Overall Goal

Mortality, morbidity and transmission of the tuberculosis are reduced.
(2) Project Purpose

Quality National TB Control Program (NTP) is systematically implemented in close collaboration
with provincial and district TB units.

i(3) Outputs

1)

Technical and managerial capacity of Punjab Provincial TB Control Program (PTP) unit isstrengthened.

2)

Technical and managerial capacity of National TB Control Program (NTP) unit and National
Reference Laboratory is strengthened.

(4) Project Sites and Model Facilities

NTP unit in Islamabad, PTP unit of Punjab Province in Lahore and 4 model districts of Punjab
Province namely Faisalabad, Gujrat, Lahore and Multan Districts.
(5) Inputs
Japanese side:
Long/Short-term expert:

Trainees received in Japan:
Provision of equipment:
Pakistani side:
Counterparts
Land and facilities:
Operational cost:

11 persons, 92.0 man-month(MM) in total(project manager 3.9MM, team leader/TB control
30.2MM, laboratory management 26.1MM, Drugmanagement 5.9MM, advocacy/data management
23.4MM, operational research 2.4MM)
6 persons
30 million yen (approx.)
35 persons
Provision of spaces for the Project offices in Islamabadand Lahore
13 million Rs (approx.)

2. Evaluation Team

Membersof
Evaluation
Team

Assignment

Name

Organization/Institution

Team leader

Dr. Mitsuo
Isono

Visiting Senior Advisor Human Development
Department, JICATB Control/Public
HealthDr. Tsuneo
Masui

Director General,
Mental Health and Welfare Center in Aichi Prefecture

Evaluation
Planning

Mr. Tomoyuki
Odani

Programme Officer, Infectious Disease Control
Division, Human Development Department, JICA

Evaluation
Analysis

Ms. Masako
TanakaTAC International, Ltd.

Period of evaluation: November 2-27, 2008

Type of evaluation: Final evaluation

3. Results of Evaluation
3.1 Summary of Performance(1) Project Purpose

In terms of the treatment success rate of 90%* and the case detection rate (ss+) of 70%, the Project
Purpose will be achieved although the cure rate of 85% has not been achieved yet in Punjab
province.

ii

Quality National TB Control Program (NTP) is systematically implemented in close collaboration
with provincial and district TB units.
(2) Outputs
Output 1：
Except the indicator No.18 “Indicators on TB drug management in 2007 are improved compared withthose of 2006.” all indicators for Output 1 have been achieved or are expected to be achieved by theend of the Project.
Quality DOTS has became to be implemented in all Punjab province through project activities and byproject inputs although the global target of TB control has not been achieved in some model districts
yet. The EQA system of Punjab province has been established after Nishitar provincial TB referencelaboratory’s setting up.
Technical and managerial capacity of PTP Punjab has been greatly strengthened through all Project
activities, such as introduction and implementation of EQA system, counterparts training in Japan,
implementation and improvement of monitoring and supervision, conducting of various operational
researches, and analysis of quarterly reports, although Pakistani side (NTP, PTP Punjab) initiallyshowed little interest to JICA CD (Capacity Development) concept, which has been resolved by
Pakistani managerial and technical capacity strengthened in the second half of the Project.
On the other hand, regarding the points which have not been achieved yet in model districts,
appropriate measures must be clarified by analyzing results of on-going operational researches and its
challenges.
Regarding the TB drug management the Project achieved significant results in development of thenational guideline for TB drug management based on operational researches. The Punjab PTP will be
responsible for improvement of drug management according to the national guideline.
Output 2：
The all indicators have been achieved. However, technical assistance for revision of national guidelines
and training modules were limited.
The Project has achieved significant results: EQA system has been introduced nationwide according to
the EQA system introduced by the Project and the national guideline for TB drug management wasdeveloped.
However, concrete action at national level were not adequate comparing to provincial level to bring theresult of “Technical and managerial capacity of National TB Control Program (NTP) unit and National
Reference Laboratory is strengthened.”
3.2 Summary of Evaluation(1) RelevanceFor the following reasons, relevance of the Project is high.( The relevance evaluated by the ex-anteevaluation study still correct.)(a) The project overall goal and project purpose have consistency with Pakistani health anddevelopment policy and needs in health sector.
(b) Project purpose is also consistent with the aid policy of the Japanese government for Pakistan.

iii

(c) The project approach focusing on human resource development, capacity building,monitoring and supervision, establishment of EQA system and improvement of drug
management is appropriate for improvement of quality of TB DOTS that was the issues to besolved in Pakistan.
(d) Punjab province is the most populated province in the country and the half of TB patients arefrom this province.

(2) Effectiveness
Although it is difficult to evaluate effectiveness of the Project on this achievement of the global target
which is result of technical assistance by various donors, effectiveness of the Project is evaluated to behigh from the following reasons.
(a) Quality TB DOTS has been steadily expanded in Pakistan, and Punjab province achieved theglobal targets of 90% of TSR and 70% of CDR in the first quarter of 2008 and have kept its good
performance level until now.
(b) The achievement level of Outputs is adequate at the point of the evaluation, and the Project
Purpose is expected to be achieved.
(c) The Project achieved the following significant results:
1) Establishment of EQA system in Punjab province including the establishment of the second
provincial reference laboratory in Nisitar Medical College and national expansion of the EQA
system.
2) Improvement of quality in DOTS in model districts and its expansion to the whole provinceof Punjab.
3) Development of the National Guideline for Anti-TB Drug Management based on theoperational research.
4) Development of a guideline for Hospital DOTS linkage and patients referral forms.
(3) Efficiency
Judging from the achievement level of the Project and inputs conducted, efficiency was reasonable.
(a) JICA experts
Expertise and skills, timing of dispatch and the number of JICA experts were appropriate.
Performance of JICA experts and their technical inputs have been appreciated by counterparts.
However, the followings are pointed out by both sides (JICA experts and counterparts).
- There occurred delay in contract procedure and dispatch of Japanese experts in 2007, which
didn’t affect the achievement of the outputs.
- Replacement of the JICA experts in the field of TB laboratory caused some difficulty to theconsistency of the Project, but the activities planned are being developed by an expert newly
assigned and it didn’t affect the achievement of the outputs.
(b) Counterparts
The number of counterparts and their technical field were reasonable in general. Quick turnover oftrained staff at the district level needed more training than expected.

iv

Initial absence of a counterpart at national level in development of the national guideline for TB
drug management has been solved by assignment of a focal person by NTP after the mid-term
evaluation.
(c) Counterparts training in JapanCounterparts training had greatly contributed to promotion of the TB control program in Punjab
province. However the knowledge obtained in Japan was not shared widely in theirorganizations. The counterparts trained in Japan are expected to make more effort todisseminate what they learned for their institutional sustainability.
(d) Provision of equipment
The quality and quantity of the equipment were appropriate and well maintained and utilized.

For utilization of computers which have been provided to diagnostic centers in model districts,
doctors and operators should develop their skills on data entry.
(e) Budget allocation
Operational cost borne by both sides was appropriate. As mentioned by the mid-term evaluation,
establishment of a provincial reference laboratory in Nishtar Medical College for southern Punjab
significantly delayed since the budget was not secured on time. It’s appropriate that in the final
year Japanese experts strengthen their technical orientation for its solid capacity building.
(f) Project offices
The necessary spaces for project offices have been provided in NTP building, Islamabad. In
Lahore, an office for a JICA expert for TB laboratory was also provided in IPH.
(4) Impact

The followings are impacts identified by this final evaluation mission. There is no negative impact
or influence due to the Project.
(a) Establishment of EQA system at national level
For establishment EQA, the activity of the Project in Punjab has given further impact and NTP
started to adapt the EQA system established by the Project to the whole nation. NTP alreadyfinished the pilot studies in 40 districts and is planning to extend to other 90 districts.
(b) Expansion of study area of Hospital DOTS linkage
The pilot study and operational researches have been aimed to conduct in 4 model districts after
the initial study in Lahore. However, due to importance and high quality of this study, PTP Punjab
decided to implement this study in all districts simultaneously.
(c) Improvement in managerial and general capacity
Managerial and general capacity of counterparts at all levels has been improved through various
activities of the Project. These include, though by subjective observation, organization of
meetings, capacity for presentation, active participation to meetings, and well motivatedatmosphere for workings, etc.
(5) Sustainability
For the following reasons, sustainability of the activities is considered high.

v

(a) Strong policy support and sufficient financial resources
TB control is one of the important issues in health and social sectors and also one of the MDG
goals as indicated in Pakistani development plan and policy papers. Through the interview by the
mission member, it is confirmed that the Government of Pakistan continue provide policy support
to TB control.
For the financial sustainability, NTP has succeeded to get enough budgets from GFATM R6 andR8. Thus, financial sustainability seems to be high, though NTP needs to work for proper
management of the budget and improve capacity to develop successful proposal for other grants.
(b) Strong leadership of NTP and PTP
The mission member confirmed that both NTP and PTP Punjab have strong leadership andmanagerial capacity for TB control, resulting in developing capable staffs. Although both NTP and
Punjab PTP should continue to develop human resources to cope with staff turnover and newchallenges, both have established the functional organizational structure for sustainable
development.
(c) Technical transfer has been done in Punjab province and NTP
Although NTP and PTP Punjab needs to develop technical capacity to cope with further
challenges, such as MDR-TB or TB/HIV, both already have obtained enough technical capacity to
implement basic TB control program through the Project activities and their own efforts. Thus, it isjudged that both NTP and PTP Punjab have sustainability which has been aimed by the scope of
this Project.
3.3 Conclusion
As the conclusion, all five criteria can be reasonably evaluated. This is the results of tremendous efforts
by NTP and PTP Punjab and certain contributions by the Project, as the Project, in addition to thepreceded JICA activities, made significant achievement according to the PDM.

However, as pointed above, controversies remained for the evaluation, especially for achievement at
the national level. As the results, achievement of the Project at the national level is not optimal. Several
factors are attributable to this and decentralized system in Pakistan might have partly influenced.
However, as pointed out, there might have been additional or alternative approaches to bring moreachievement at the national level upon utilizing the experiences of preceded technical assistance by
JICA.
Also, it has been controversial about the method of capacity building and there has been discrepancy inthe concept of capacity building between Pakistani and Japanese sides, as pointed out by the JICA
consultation mission and the mid-term evaluation mission. Thereafter, mutual understanding has beenpromoted thorough series of discussion and efforts by the Project, resulting finally in deepened
understanding for capacity building by the Pakistani side and favorable achievement of the Project.
For better approaches for capacity buildings, there also might have been alternatives or modification
upon mutual understandings for the design of the Project.
3.4 Recommendation
The team was impressed by the efforts and commitment and ownership by NTP and PTP Punjab in allviaspects of the TB control program. Also, the activities of Japanese side were outstanding despite of limited
resources. The Team would be very grateful if recommendations described below will eventually bringcertain additional development in TB control program in Pakistan.

(1) Review of the project activities
Since the Project has worked in four districts which have various social situations, importantlessons will be expected from the review and analyze of the Project activities, resulting in possible
adaptation to other districts not only in Punjab , but also in other provinces. Thus, it isrecommended that the Project develop the extensive review of the Project activities in four
districts to feed back to Punjab PTP and NTP. Punjab PTP and NTP are recommended todisseminate and utilize the review for future management of TB control program in other districts.
(2) Operational researches
It is recommended to finish all on-going researches and the Project and Punjab PTP areencouraged to submit as research articles. Some research will continue after the end of the Project
period, thus both Pakistan and Japanese sides are required to allocate necessary human andfinancial resources to complete researches.
(3) EQAThe Project made certain contribution to develop EQA not only in Punjab, but also in whole
Pakistan. Punjab PTP haven been utilizing the SOP for EQA which was developed by the Projectfor practical use. The contents of the SOP have been incorporated into the National Guideline.
Thus, it is recommended that NTP, too, starts practical use of the SOP upon full endorsement assoon as possible to feed back the results to the Project.
For EQA in Punjab, the reference laboratory in Multan still needs to develop its capacity as thereference laboratory. Thus, the Project should continue full support to develop the capacity in theremaining period.
(4) The National Guideline for Drug Management
The Project made significant contribution to develop this guideline. Thus, it is recommended that
NTP should print and publish it upon full authorization as soon as possible.
(5) Continuous utilization of the training courses supported by JICA
Training courses abroad supported by JICA have been effective to develop human resources. Thus
it is recommended that NTP and Punjab PTP would continue to utilize those training courses andshould take necessary action for this purpose. It is also recommended that NTP and Punjab PTP
develop the plan of human resource development to maximize effects of those training courses.
3.5 Lessons Learned(1) Capacity Development (CD)

Difficulty of Pakistani’s understanding about CD has been pointed out in previous missions.
JICA’s policy which is to establish institutional capacity through technical cooperation transferredto the counterparts, and the Pakistani stance which is to establish their organization by dividing
their rolls with development partners showed distance each other. Also, Pakistani understanding
about the Project design, which is to spread out the results generated in model districts to all other
districts, was relatively low. Fortunately this gap of understanding has been resolved by Japaneseviiexperts efforts, as well as the Pakistani positive effort to disseminate the Project’s good practice toall district, in some cases at national level. Thus following lessons were learned from thisperspective.
(a) It’s necessary to draw project design considering the in-site situation of the target country instead of taking conventional approach of CD.
(b) In situation like Pakistan, it may be necessary to consider another measure in stead of
spreading out good practice from smaller site to wider which has faced to some difficulty.
(c) Considering these points above pointed out, capacity development for which the project target
shall be a province, tackling common issues laid in major districts, may be effective. Bymaking this approach, direct capacity development will be possible.
(d) Regarding financial support, measures shall be considered to utilize effectively the Global
Fund support, and each project has to convince Pakistani side the difference between technicalsupport and financial one.
(e) Also it’s necessary for each project to make coordination more actively with otherorganizations counting on if necessary JICA support.
(2) Activities at NTP Level
In countries like Pakistan where decentralization has made progressively, if activities at NTP levelis included in Project’s results, more intervention is necessary such as making guidelines andmodules etc. and a expert will be required who can play as advisor at national level. If so, under
the advisor expert the project design in which necessary activities at provincial level are planned
may be effective.
(3) Activities at Model Area
Through this project, activities to improve quality DOTS has been implemented selecting 4 model
districts. The question whether the project activities have been effective or not should not be
determined only by confirming if the indicators regarding 4 districts have been achieved or not.
But taking into consideration the in-site TB control situation, the project design would have beenreconsidered during the project period.
Especially the involvement of tertiary hospital into DOTS implemented thorough operational
research by the Project made great contribution to the TB control program. Although the priorityof this type of activity tends to be low considering that the basic design of this project is toimprove the basic DOTS, the mentioned activity should have been implemented from its initialstage.
From this point of view, and from the reason that this Project has been implemented for 6 years intotal including the previous period, deliberation on the Project activities with mid-tem evaluation
perspective should have been made before starting this Project period in 2006.

viii