Summary of Terminal Evaluation Study of the ProjectI. Outline of the ProjectCountry：Islamic Republic of Pakistan

Project Title：
District Health Information System Project for
Evidence‐Based Decision Making and Management
Issue/Sector：HealthCooperation Scheme:
Technical Cooperation Project
Division in charge：Pakistan OfficeTotal Project Cost: Approx. 320 million JPY(R/D): April 25, 2009
Partner Country’s Implementing Organization:
Period of*Project period: July 15, 2009 – July National Health Information Resource Center
Cooperation 14, 2012
(NHIRC) (up to June 2011)
National Institute of Health (NIH) (after July 2011)
Supporting Organization in Japan：
System Science Consultants Inc.
1. BackgroundIn 1992, Health Management Information System (HMIS) was developed by USAID, however, there
were emerging requirements in HMIS to make it compatible with the information needs from theprovincial and district levels, particularly in the context of health systems devolution from the central
government to the local government in 2001. Under this situation, JICA supported the Development
Study on Improvement of Management Information Systems in Health Sector from January 2004 toMarch 2007, upon a request of the Government of Pakistan. District Health Information System (DHIS)
was developed as a result of the study.
NHIRC, the national authority on information system, formulated a plan for nation‐wide scale‐up ofDHIS, DHIS has not been kept in a proper use mainly due to prolonged mixing of HMIS and DHIS atprimary and secondary medical facilities.
Taking the above situation into consideration, the Government of Pakistan, through NHIRC, requested
JICA to extend technical cooperation for District Health Information System Project for Evidence‐BasedDecision Making and Management, with the purpose that routine operation and budget planning areto be practiced in an evidence‐based manner, through DHIS, nationwide in Pakistan.
2. Project Overview
Through the project, the Government officers in PHD and DHO is expected to be able to use andmaintain DHIS software in a proper manner after the installation of DHIS software, training on DHIS
such as data collection, data input and data use, and monitoring activities conducted by the Japaneseexperts.
(1) Overall Goal
Policy and strategies for health services are developed in an evidence‐based manner, throughsustainable DHIS, nationwide in Pakistan.
(2) Project Purpose of the Project
Routine operation and budget planning are practiced in an evidence‐based manner, through newly
introduced DHIS, at the selected districts in Pakistan.

(3) Outputs
1) Project implementation plan in the target districts is approved at JCC.
2) PHDs / DHOs staff is adequately trained on the DHIS operation.
3) The DHIS data are collected in a complete, precise and timely manner from health facilities to DHOs.
4) The DHIS data are entered into the DHIS software, processed and analyzed at PHDs and DHOs.
5) By using the results of analysis of the DHIS data, the items for resource reallocation and budgetingare identified respectively at PHDs and DHOs.
6) The DHIS is adequately coordinated among the stakeholders.
(4) Inputs
Japanese Side: 5 Long‐term Experts, Operational Cost
Pakistani Side: Counterpart, Office Space for Japanese Experts, Local Cost
II. Evaluation Team
Members of Team Leader: Mr. Toshiya Sato, Senior Representative, JICA Pakistan OfficeEvaluation
Evaluation and Analysis: Mr. Kunio Nishimura, ICONS Inc.
Team
Cooperation Planning; Mr. Tomoyuki Nagita, JICA Pakistan Office
Periodof June 4, 2012~ June 15, 2012
Type of Evaluation:Evaluation
Terminal Evaluation
III. Results of Evaluation
1. Confirmation of Outputs(1) Project Purpose: Almost achieved
In 87 DHOs of 100 target districts, routine operation (resource allocation) and budget planning havebeen practiced based on the analysis of DHIS data which has been collected for more than 3
consecutive months.
(2) Outputs
Output 1: Achieved
Strategic planning for scaling up DHIS was approved at the first JCC meeting held on June 2010, as aresult of collection of basic information through base‐line survey and interviews with relevant
stakeholders.
Output 2: Achieved
At all PHDs (including AJK and FATA) and 100 target DHOs, the revised DHIS software, a platform for
DHIS data input, was installed. Training on collection of information related to medical facilities wasconducted through Cascade Training Method and 173 master trainers at the district level 9,586 staff at
the primary and secondary level medical facilities has been trained. Moreover, for the purpose of DHIS
data input, analysis and use at PHD and DHO, 81 provincial master trainers and 129 district mastertrainers were trained. Through these trainings, each province successfully established the training
system involving master trainers at the provincial and district levels.
Output 3: Partially achieved
According to each PHD, new DHIS tools and instruments have been properly distributed. Even though
the Japanese experts found through their monitoring activities that the distribution was not done in aproper manner in some areas, it can be said that the goal of proper distribution has almost beenachieved.
Compliance rates of DHIS monthly report from public primary and secondary level medical facilities toDHOs were kept more than 90% at the last 6 months of the project in 39 districts (39 %) out of 100
target districts. The low compliance seems to be caused by shortage of DHIS tools and instruments due
to delay in release of the required budget for printing, lack of coordination with stakeholders and delayof date input.
Output 4: Achieved

As a result of training on data input/process/analysis, one or more staff in each PHD and each target
DHO has been able to enter the collected DHIS data, process, analyze, and make necessary tables andcharts for analysis.
Output 5: Almost achieved
Staff in 99 DHOs out of 100 target DHOs as well as all PHDs has been trained on use of DHIS data. By
using the results of analysis of the DHIS data, the items for resource reallocation and budgeting havebeen identified in 87 DHOs.
Output 6: AchievedFederal level meetings (1 S/C, 4 JCC, 2 TAG), provincial level meetings (3 PMC, 12 WG) and 2 Partner
Meetings were conducted for discussion for scaling up DHIS.
2. Summary of Evaluation Results(1) Relevance: Almost highIt has been found that Project Purpose and Overall Goal are consistent with National Health Policy
2009 (Final Draft) as well as Japan’s Assistance Policy for Pakistan; therefore, their level of coherencewith development policy seems to be high. As the devolution of the Ministry of Health was executed inJune 2011, scaling up DHIS will be continued in each province based on provincial health strategy,
Moreover, as a project in the field of health information in which Japan has maintained advantage
through the Study on Improvement of Management Information Systems, the project was implemented
keeping in close touch with donors and NGOs involved in health issues, which contributed to the highlevel of Relevance in the evaluation.
(2) Effectiveness: Almost highAs 6 Outputs were prepared in a way that the achievement of each output leads smoothly to theachievement of Project Purpose, Taking Output 1 as a base of formulation of consensus among theProject stakeholders, installation of software (output2), training staff of PHD and DHO on data input,
aggregation, analysis (output 3 and 4), and utilization of analyzed data (outpu5) have been
implemented and contributed to achievement of the Project Purpose. Output 6 aimed at coordination
with relevant government authorities and development partners, it facilitated smooth implementationof the Project.
Project Purpose has almost been achieved in spite of such difficulties as the postponement of activities
by flood disaster 2010, the devolution of the Ministry of Health, and the restriction of the project
members’ activities in particular areas due to security reasons.
(3) Efficiency: Moderate
Inputs by Japanese side were almost appropriate, while those of Pakistani side were not always
appropriate, the reason of which includes the decrease of number of the targeted districts following thefact that some of the districts have not secured sufficient budget for the project activities.
(4) Impact: Moderate
As there is no organization responsible for scaling up DHIS nationwide, the achievement of Overall
Goal, which is to use DHIS directly as a tool for formulating a federal health policy, will be difficult tobe realized. However, the fact that DHIS data has been used for budget planning and decision‐making
in targeted provinces and districts, indicates that it is expected that DHIS data will be utilized in other
districts in case DHIS software is installed in those districts in the future. No negative impact will beforeseen at the moment.
(5) Sustainability: Federal level‐ Low, Provincial level‐ Moderate
Even though it is desirable that a federal organization keeps involvement in DHIS activities, there is no
single organization that is responsible for health administration in an integrated fashion at the federal

level as due to the devolution of the Ministry of Health in June 2011, which keeps the level ofsustainability at the federal level low at the moment. However, PHDs show their intention to formulate
their health policy and scale up DHIS by using their own budget after the termination of the project,which will be a positive implementation for sustainable use of DHIS.
3. Factors that promoted realization of effects(1) Factors concerning to Planning‐ As a way to achieve the Project Purpose, training was planned and implemented in a logical manner(1. Data Collection, 2. Data Analysis, 3. Data Use) to ensure the effectiveness of the project.
(2) Factors concerning to the Implementation Process‐ Information related to DHIS was exchanged among PHDs through monthly‐held WG meetings,
which contributed to fostering awareness among PHDs considering the prevalent friendly rivalry
and improving the possibility of the sustainable use of DHIS.
‐ Cascade training system was adopted as a sustainable system so that PHD staff who got trainings
from Japanese Experts can conduct the same trainings for DHO staff. The PHD and DHO staff wastrained in an efficient and effective manner by the training system.
4. Factors that impeded realization of effects(1) Factors concerning to Planning‐ As some districts have failed to allocate adequate budget for the implementation of the project, thenumber of target districts was decreased.
(2) Factors concerning to the Implementation Process‐ NHIRC’s uncooperative attitude throughout the project period of three years affected the achievementof the project including the delay of the project.
‐ C/P at the federal level has not been officially appointed for some months since the devolution of theMinistry of Health, due to which the federal government was not involved with DHIS during thepost devolution period and Japanese Experts made efforts to maintain a good relationship withPHDs to ensure the sustainability.
‐ Due to the security constraint, the project activities in some provinces and districts, including FATA
and AJK were restricted. The visits to those restricted areas have been replaced with inviting thosepeople living in the area to their neighboring area thought to be security‐wise safe in order tomitigate the risk against efficient implementation of the project.
5. Conclusion
Project Purpose has almost been achieved by efforts by staff in Provinces/Districts/Health facilities andflexible judgment for change of plan by Japanese experts, including the decision that those districts
which failed to secure budget enough for the implementation of the project shall be excluded from thetarget districts.
Though it is not clear whether national health policy will be finalized due to the devolution of theMinistry of Health, it is expected that PHDs can scale up DHIS if they secure budget necessary for
DHIS because each province definitely requires DHIS data for their policy and planning at the districtlevel.
6. Recommendations(1) Sustainable use of DHIS
As mentioned in 2 (3), insufficient budget, including cost for DHIS software maintenance, will
negatively affect the implementation of the project; therefore, even after the completion of the projecteach province is requested to secure the budget, including cost for DHIS software maintenance,necessary for the operation of DHIS, for the sustainable use of DHIS.

(2) DHIS in the non‐targeted districts
As mentioned in 2 (4), it is expected that DHIS data will be utilized nationwide in case DHIS softwareis installed in all the districts; therefore, DHIS software should be installed in those districts in which
DHIS software was not installed during the project period as soon as possible after concerned PHDs
arrange hardware required for each DHO.
(3) Involvement of Federal Government on DHIS
As mentioned in 2 (5), there is no federal level organization that is clearly designated as anorganization responsible for DHIS due to the devolution of the Ministry of Health in June 2011, which
will negatively affect the level of sustainability. It is strongly hoped that the involvement of federal
government should be strengthened in order to utilize DHIS as a nationwide unified health system inthe long term.
7. Lessons Learned(1) Conditions necessary for the start of the project
As mentioned in 4. (1), the number of target districts was decreased because some districts did notallocate adequate budget for the implementation of the project. We need to be noted that all theprerequisites for the commencement of the project shall be clarified and shared with stakeholders inorder to avoid any unnecessary change in the scope of the project during the project implementation
period.
(2) Appropriate Response to Problematic Implementation Organization
As mentioned in 4. (2), NHIRC’s uncooperative attitude led to the serious delay of the implementation
schedule of the project. Any issue over Implementation Organization’s attitude should have been
resolved through the involvement of high‐ranking officers of their parent ministry as well as ODA
counterpart organization (e.g. Ministry of Health and EAD in Pakistan).
8. Follow‐up Situation
Following the request from the Government of Pakistan, JICA will plan to make a contract with asoftware company, asking them for the maintenance of the latest DHIS software (Ver. 1.03) up to March2013 at the longest.