Summary of Terminal Evaluation
I. Outline of the ProjectCountry： Indonesia

Project title：The Project for the Support on Forest
Resources Management through Leveraging Satellite
Image Information

Issue/Sector:

ForestryandForest Cooperation scheme：Technical Cooperation

Conservation
Division in charge：Forestry and Nature Total cost： Conservation Division 1, Forestry andNatureConservation

Group,

GlobalEnvironment Department(R/D):
PeriodPartnerCountry’s

Implementing

Organization ：of Sep. 11, 2008～Sep.10, 2011 Directorate general of Forestry Planning (DJP),

Cooperation

Ministry of Forestry (MOF)
Supporting Organization in Japan： Forest Agency

Related Cooperation： Nothing
1. Background of the ProjectIndonesia's tropical and subtropical forests and wetlands have the third largest area (over 120
million ha) after Brazil and the Democratic Republic of Congo. On the other hand, forest has been
decline by 2% annually (FAO, 2005), and forest conservation and restoration has been recognized
as an urgent issue. Causes of deforestation include forest fires, illegal logging, timber processing
and unplanned land conversion to agriculture. Background of these causes are, ①the accuracy ofmonitoring forest resources, law reliability caused by non-integrated information ② lack of③a lack of legal andcoordination with related government agencies on land use permits,
institutional arrangements which results from rapid decentralization.
As effective measures to deal with issues above, it is necessary to obtain forest resources
information with a certain degree of accuracy and reliability, to share such information among
relevant organizations (various ministries , government agencies under decentralization
condition and private companies), and to develop and implement appropriate forest resource
management plan based on information above. In forest resources management in Indonesia,
remote sensing techniques utilizing satellite image is needed.
The Project for the Support on Forest Resources Management through Leveraging Satellite
Image Information (“The Project”) is transferring forest remote sensing techniques for forest
resources management by using PALSAR (microwave sensor the ALOS satellite in JAXA) and
MODIS (optical sensors on satellites AQUA / TERRA U.S. which can conduct wide-area survey offorest resources)，and strengthen current forest resources monitoring system. Moreover, theProject was formulated for capacity development for central- and local-level staff. JICA
implemented Ex-ante evaluation in October, 2007 and February, 2008, and the Project started in
September, 2008.

xvii

The Project conducted technical instruction on PALSAR image interpretation and developed
interpretation manual. In addition, the Project carried out activities related to REDD (reducing
emissions from deforestation and forest degradation) such as provision of information and donor
coordination. In March, 2010, JICA monitoring mission was dispatched to understand current
situation and to discuss future perspective.
Considering the project is scheduled for completion in September 2011, terminal evaluation ismade this time.
2. Summary of the Project(1) The Project Purpose: BAPLAN’s capacity to conduct more reliable forest resources
monitoring and assessment is upgraded through transfer of technology and training(2) The Overall Goal: Sustainable Forest Management (SFM) is promoted in Indonesia
through the upgraded forest resources monitoring and assessment(3) The Outputs
1) Output 1: Accuracy of forest resources monitoring and assessment data utilizing
satellite image information is improved2) Output2: Capacity of BAPLAN and its UPTs id upgraded(4) Inputs:
1) Indonesian side
Allocation ofproject
personnel:

Management
personnel:2 persons
Technical
personnel:5persons

Running
Expenses:n/a2)Japanese side
DispatchofExperts: (as ofJun 2011)
Project
personnel
TrainedinJapan:

7persons in total

Provision
Equipment:

of

n/a

18 persons in total

Local
Cost:

Activity

n/aII. Evaluation Teamof Title
Leader
Evaluation

Name

Position

Mr.Shigeki

Executive Technical Advisor to the Director

Team

Hata

General, Global Environment Department

Dr.Gen Takao

Department of Forest Management, Forestry

Membersand Forests Products Research Institute
Mr.Kazunobu

Advisor, Forestry and Nature Conservation

Suzuki

Division I, Forestry and Nature Conservation
Group, Global Environment Department

Evaluation
Analysis

Ms.Yasuyo

Permanent Expert, International Development

Hirouchi

Associates, Ltd.

xviii

PeriodTypeof 5/ 15/ 2011- 1/ 6/ 2011

Evaluationof

Evaluation ： Terminal

evaluation

III. Results of Evaluation
3-１ Accomplishment of the Project.
3-1-1 Accomplishment of the Outputs(1) Output 1: The cloud cover appeared in the land cover map is eliminated by using PALSAR
data. An interpretation manual for PALSAR data as well as a guideline for the use of
PALSAR data for land cover mapping (as main satellite images) has been developed.
Technically achievable accuracy (i.e. matching ratio between the land cover types appearedon the sample land cover maps formulated based on the above manual/guideline, using
only PALSAR 50-m Resolution Ortho Mosaic data, and the land cover types identified
through field surveys) based on the methods developed by the Project was more than 85%.
Land cover mapping based on the developed methods, however, has not been
operationalized yet: DJP has not yet reached an organizational decision on substituting
PALSAR for LANDSAT because of discontinuation of free-of-charge provision of PALSAR
data by JAXA from 2010 data due to change in a data provision policy of JAXA andunexpected termination of ALOS operation in April 2011. In addition, the final draft of the
PALSAR data interpretation manual was not available until March 2011. At present,
DJP is planning to use the PALSAR data as complement to the LANDSAT data in the land
cover mapping process in 2011, but DJP is uncertain about utilization of PALSAR data in
the future especially in light of uncertainties related to continuous provision of PALSAR
data and JAXA’s data provision policy in the future.
(2) Output 2: With regard to DJP, the officers trained by the Project are expected to havegained skills and knowledge necessary for supervision of land cover mapping using
PALSAR data. As for UPTs, there is a gap in capacity building. From 9 UPTs at least
one officer has participated in advanced-level training: these UPTs are presumably readyto formulate land cover maps using PALSAR data through leadership of the officers withadvanced skills and knowledge. In contrast, the other 8 UPTs have the officers who weretrained only in basic-level (or less-advanced) courses. Presumably, these 8 UPTs are notready to formulate land cover maps using PALSAR data for themselves. It should also be
noted that the officers trained by the Project have had little opportunities to familiarize
themselves with the skills and knowledge through practical experiences.

Thoughon-the-job training (OJT) is scheduled in the remaining period, the details, including
target officers/UPTs and modality of OJT, are yet to be finalized.
3-1-2 Accomplishment of the Project Purpose
Improvement of reliability of forest resource information was not able to be confirmed

xixbecause the land cover maps, utilizing PALSAR data, from which the information would beacquired from, are yet to be formulated by DJP. As stated already, DJP has not been able
to decide on the use of PALSAR in their land cover mapping especially in light of
uncertainties related to continuous provision of PALSAR data and JAXA’s data provision
policy in the future. For reference, such information is likely to be made available forother Departments of the Ministry of Forestry or other Ministries in early-mid 2012 since
DJP is planning to use PALSAR in their land cover mapping process in 2011 as complement
to LANDSAT
3-2 Summary of Evaluation
3-2-1

Relevance

The Overall Goal is relevant with the needs of Indonesia.
consistent with the organizational needs of DJP.

The Project Purpose is

The Overall Goal and the Project

Purpose are relevant with the latest Mid-term Development Plan (2010-2014) of Indonesia
as well as Official Development Assistance (ODA) policies of Japan. Technical advantage
of Japan is confirmed.
Overall, the Project is considered to be still relevant.
3-2-2

Effectiveness

Progress has been made towards achievement of the Project Purpose but the exact degree ofachievement was not able to be assessed because of unavailability of the relevant data.
Judging from the overall achievement level of the Outputs, achievement level of the Project
Purpose is presumed to be lower than expected due to adverse effects posed by the external
condition beyond the control of the Project (i.e. discontinuation of free-of-charge provision of
PALSAR data by JAXA and unexpected termination of ALOS operation in April 2011).
Taken together, the Project is considered moderately effective.
3-2-3

Efficiency

In general, the Inputs from the both Indonesian and Japanese sides have been mostly
appropriate in producing the Outputs in terms of timing, quality and quantity. Output 1
has been mostly produced but is not likely to be fully produced due to external factors thatare beyond control of the Project (i.e. discontinuation of free-of-charge provision of PALSAR
data by JAXA and the unexpected termination of ALOS operation). Output 2 has beenmoderately produced. It is uncertain whether Output 2 would be fully produced by the
Project end mainly because the Project is yet to come up with a clear capacity buildingxxstrategy for the remaining period.
On the whole, the Project has been mostly efficient.
3-2-4

Impacts

Impacts at the Overall Goal level: It is unclear if the Overall Goal would be achieved inthree years after the termination of the Project due to adverse effects posted by the external
factors that cannot be controlled by the Project (i.e. discontinuation of free-of-charge
provision of PALSAR data by JAXA and the unexpected termination of ALOS operation)
Other impacts: Some positive impacts have been observed, including the increased
knowledge and skills of lectures and students of IPB on PALSAR through participation in
the works contracted out to Faculty of Forestry of IPB, and increased awareness on the useof PALSAR through publicity activities of the Project. Negative impacts have not been
observed. They are not foreseen, either.
3-2-5 Sustainability
Institutional and organizational aspects: The legal and policy support for utilization ofsatellite images in forest resource monitoring is likely to continue. DJP has not been able
to come up with a clear post-project strategy on the use of PALSAR data in land cover
mapping and associated capacity building especially because (i) discontinuation of
free-of-charge provision of PALSAR data by JAXA from 2010 data due to change in a data
provision policy of JAXA; (ii) provision of the PALSAR data itself has been discontinued due
to power anomaly experienced by ALOS in April 2011; and (iii) there should be a time-gap
of a few years until PALSAR data becomes available again. Meanwhile, DJP would be able
to manage land cover mapping process, using PALSAR, given such a decision is reachedbecause it has managed land cover mapping, using LANDSAT, through coordination with
relevant organizations.
Financial aspects: DJP has formulated land cover maps, using LANDSAT, with their own
budget. Necessary budget is likely to be allocated for land cover mapping, using PALSAR,
given a decision on the use of PALSAR is made by DJP.
Technical aspects:

It is presumed that, the Indonesian Project Personnel will haveacquired practical knowledge, skills, and experience on the use of PALSAR, which arenecessary for continuation of the relevant activities in the post-project period.

It isuncertain whether the knowledge and skills transferred to the officers of DJP/UPTs as wellxxi

as the project deliverables, including the PALSAR data interpretation manual, Guideline
for using PALSAR data for land cover mapping, training modules and training guidelines,
etc. would be actually utilized in the future, because DJP has not been able to come up witha post-project strategy on the use of PALSAR as noted above. In addition, the capacity
development strategy to fill-up a gap between the UPTs with advanced skills and those onlywith basic skills in the remaining period is not clear, which also raises concern overutilization
Taken together, It seems possible, but is not certain whether the sustainability of the
Project would be ensured because DJP has not been able to nail down a post-project
strategy, especially on the use of PALSAR in its forest resource monitoring and assessment
system in light of uncertainties related to continuous provision of PALSAR data and JAXA’sdata provision policy in the future, which are beyond the control of the Project.
3-3 Factors that promoted realization of effects
3-3-1 Factors concerning to Planning
Nothing special
3-3-2 Factors concerning to Implementation
Technical Advisory Group, consisting of experts from University of Indonesia, Indonesian
National Institute of Aeronautics and Space (LAPAN), and Center for International Forestry
Research (CIFOR), has been formulated in order to take most advantage of the expertiseavailable in Indonesia, in particular in the development process of PALSAR Data
Interpretation Manual.

Valuable technical advice made by the Members of the Group, whowere invited to internal workshops and meetings, have been duly reflected in the Manual.
3-4．Factors that impeded realization of effects
3-4-1 Factors concerning to Planning
The project design was not clear enough. For example, descriptions of some of the Activities
and Outputs were vague and most of the Objectively Verifiable Indicators were not welldefined. In addition, many of the Indicators lack criteria to judge their achievement level sothat they were not objectively verifiable. Some Indicators were placed at a wrong level of the
Narrative Summary. Furthermore, Indicators of the PDM were not sufficient to assess theachievement of the Project properly. As result, the concerned personnel had to invest thetime and effort in understanding the contents of the Project described in the PDM.
3-4-2 Factors concerning to Implementation
Project staff of DJP has been occupied with routine work and other tasks so that they couldxxii

not participate in the Project Activities fully.

It was often difficult to find time formeetings/discussions.
As for the PDM, during the visit of the JICA Consultation Team in March 2008, a discussion
paper on interpretation on PDM, prepared by the Japanese Expert Team, was reviewed andthe necessity of further elaboration of the paper was agreed. The paper, however, has notbeen discussed within the Project as it has been difficult to find convenient meeting time.
Meanwhile, the issues other than definition of the Indicators have been left unnoticed. Thishas made it difficult for all those concerned to have common understanding on the expected
achievement level and progress of the Outputs and the Project Purpose of the PDM.
3-5 Conclusion
The Project is still relevant and moderately effective. And the Project has been mostly efficient.
It is unclear if the Overall Goal would be achieved in three years after the termination of the
Project due to adverse effects posted by the external factors that cannot be controlled by the
Project. In the meantime, some positive impacts have been observed. Negative impacts havenot been observed. They are not foreseen, either.
It seems possible, but is not certain whether the sustainability of the Project would be ensured
because DJP has not been able to nail down a post-project strategy, especially on the use of
PALSAR in its forest resource monitoring and assessment system in light of uncertainties
related to continuous provision of PALSAR data and JAXA’s data provision policy in the future,
which are beyond the control of the Project.
3-6 Recommendations
3-6-1

By the end of the ProjectThe Evaluation Team recognized that implementation of OJT was approved at the
JCC (April, 2011), and that the Project is in the process of planning OJT.
Considering that remaining time is very limited, it is recommended that theTraining contents such as place and target participants be discussed and
immediately agreed by the Project. In addition, recognizing that there is a gap incapacity development to formulate land cover maps using PALSAR data, it isrecommended that the Project develop a clear capacity development strategy.



The Evaluation Team found that “Guideline for using PALSAR data for land cover
mapping” which has been developed by the Project described solely the PALSAR
image interpretation with no relevance to the conventional Landsat image
interpretation that MoF had adopted. Therefore, it is recommended that theinterpretation guideline further describe the complementary use of PALSAR

xxiii

image interpretation to the Landsat image interpretation so that the developed
method be integrated in the conventional land cover mapping. The Team also
recommends changing the title of the guideline so as to explicitly indicate thecomplementary use of PALSAR to the conventional mapping scheme.
The Evaluation Team found the method developed by the Project very innovativewith high accuracy. To promote a better understanding of this achievement by theexternal stakeholders, the Team recommends to the Project the additional actions
as follows;

Calibrate the accuracy table and correct the bias in area estimation of each class asrecommended by Card (1982).



Present the aggregated accuracy tables by different levels of land cover classes so as
to demonstrate the higher accuracies for simpler (or higher level) classification
demands.

Compare the resulted accuracies with the accuracies by the conventional Ministry ofForest land cover map, which is supposed to be released for the first time as a part of
the “Land Cover Map of Indonesia, 2011 ” in coming June or July, as a baseline.

Publish the summary of the Project products in English and proactively present the
Project results in international arenas.

3-6-2 After the end of the Project

It is recommended that Ministry of Forestry utilize PALSAR images as a complementary
data source for land cover mapping as suggested in the guideline and manuals by adopting
those products as the materials for the training which HQ regularly conducts every yearwith participation of HQ staff and all UPT’s staff in Indonesia. For those limited staff withexcellent abilities, it is recommended to provide knowledge and skills in ALOS PALSAR
pre-processing for their possible use in further application.



The Evaluation Team found that the PALSAR image interpretation developed by the
Project would be potentially superior to the conventional Landsat image interpretation
adopted by MoF in terms of the temporal resolution especially when the mapping interval
becomes shorter. Usually 3 to 4 or more Landsat TM/ETM+ images acquired over a periodof 2 years are necessary for interpretation of an area so as to minimize the cloud cover bymosaicking cloudless parts of the images. This image acquisition period limits the updatinginterval of the map to 2 years or longer, otherwise wider cloud coverage in the resultant
maps should be accepted in return for a shorter updating interval. Thus, MoF’s plan toshorten the interval of land cover mapping from 3 years to merely 1 year would result most
likely in a series of maps with lower quality if they continue the same interpretation
procedure. In contrast, PALSAR and the scheduled ALOS2 SAR can provide cloudlessxxivimages with a precise temporal resolution which would allow more frequent mapping witha narrow time window. Thus the Team recommends for MoF to evaluate the feasibility of
the PALSAR image interpretation applying in any given area in Indonesia in response to
the shorter mapping interval requirement.


The Evaluation Team found that the Project had accomplished development of the PALSAR
image interpretation at an excellent level of accuracy, which demonstrated the modality ofusability of PALSAR for land cover mapping. It was recommended that MOF further
explore possible applications of PALSAR data for its policy implementation, e.g.
deforestation detections and/or carbon stock estimation, in combination with other data
sources and methods so as to make wider and robust options of forest monitoring in future.

3-7 Lessons learned

It could be a good idea to have a target of technical transfer outside of the government.
The Project showed that working with staff of IPB contributed to developing their skill for
dealing with PALSAR data. Educational institution like national university has a potential
to be a source of providing human resources to the government.

3-8 Follow-up situation
Nothing.
It is suggested that outputs/outcome of the Project be utilized in REDD+ project in future inIndonesia.

xxv