Paraguay ECT Final Report

i

U.S. Department of the Treasury 

Office of Technical Assistance

Jamaica Budget and Financial 

Accountability Project

Evaluation Report 

Final Submission

July 24, 2023
Jamaica Budget and Financial Accountability Project Evaluation Report

i

TABLE OF CONTENTS

I. EXECUTIVE SUMMARY .................................................................................................. 1

JAMAICA BUDGET AND FINANCIAL ACCOUNTABILITY PROJECT.................................................. 1

EVALUATION ............................................................................................................................... 1

SUMMARY OF FINDINGS............................................................................................................... 2

CONCLUSIONS AND RECOMMENDATIONS..................................................................................... 3

II. BACKGROUND AND PROJECT OVERVIEW .......................................................... 5

III. EVALUATION METHODOLOGY AND PURPOSE.................................................. 6

EVALUATION QUESTIONS ............................................................................................................ 7

EVALUATION METHODOLOGY AND PROCESS............................................................................... 8

IV. PROJECT FINDINGS ................................................................................................... 10

PROJECT OBJECTIVE 1—CASH MANAGEMENT .......................................................................... 13

PROJECT OBJECTIVE 2—PENSION REFORM ............................................................................... 16

PROJECT OBJECTIVE 3—DAILY RECONCILIATION ..................................................................... 18

PROJECT OBJECTIVE 4—TRAVEL CARD .................................................................................... 20

PROJECT OBJECTIVE 5—GOVERNMENT ACCOUNTING & REPORTING UNIT............................... 22

V. CONCLUSIONS ................................................................................................................. 23

VI. RECOMMENDATIONS................................................................................................ 25

VII. ANNEXES ....................................................................................................................... 29

ANNEX 1: DATA SOURCE MATRIX ............................................................................................. 30

ANNEX 2: EVALUATION FINDINGS MATRIX ............................................................................... 32

ANNEX 3: OTA PROJECT DOCUMENT LIST ................................................................................ 35

ANNEX 4: CONTEXT DOCUMENT LIST........................................................................................ 36

ANNEX 5: DATA COLLECTION INSTRUMENTS ............................................................................ 38

ANNEX 6: LIST OF KEY INFORMANT INTERVIEWS (KIIS) AND RESPONDENTS............................ 44

ANNEX 7: ASSUMPTIONS AND RISKS IDENTIFIED BY EVALUATION TEAM.................................. 46

ANNEX 8: PEFA ASSESSMENT INDICATORS AND SCORES.......................................................... 48

ANNEX 9: EVALUATION TEAM QUALIFICATIONS ....................................................................... 50
Jamaica Budget and Financial Accountability Project Evaluation Report

ii

ACRONYMS & ABBREVIATIONS 

AGD Accountant General’s Department

BFA Budget and Financial Accountability

BOJ Bank of Jamaica

EQ Evaluation Question

FATAA Foreign Aid Transparency and Accountability Act

FMIS Financial Management Information System

GAR Government Accounting and Reporting Unit

GOJ Government of Jamaica

IMF International Monetary Fund

KII Key Informant Interview

Logframe Logic Framework

M&E Monitoring and Evaluation

MCC Millennium Challenge Corporation

MEL Monitoring, Evaluation, and Learning

MOFPS Ministry of Finance and Public Service

NGO Nongovernmental Organizations

OTA Office of Technical Assistance

PAS Pension Administration System

PEFA Public Expenditure Financial Accountability

PFM Public Finance Management

TA Technical Assistance

TOC Theory of Change

TSA Treasury Single Account

U.S. United States

USAID United States Agency for International Development

USG United States Government
Jamaica Budget and Financial Accountability Project Evaluation Report

1

I. EXECUTIVE SUMMARY 

Jamaica Budget and Financial Accountability Project

Department of Treasury’s Office of Technical Assistance’s (OTA) 

Jamaica Budget and Financial Accountability (BFA) project

provided technical assistance (TA) in public financial management to 

the Government of Jamaica (GOJ) from 2017–2021. OTA advisors 

worked closely with their counterparts in Jamaica’s Accountant 

General’s Department (AGD) of the Ministry of Finance and Public 

Service (MOFPS) with a goal of strengthening the management of 

public finances to help ensure the availability of sufficient funds to 

meet government service delivery needs by improving cash 

management and internal controls of the AGD. This work was 

guided by the five project objectives detailed in the text box. 

Evaluation

In September 2022, OTA engaged Bixal to undertake a summative 

evaluation of the project in order to answer the following evaluation 

questions (EQs):

EQ 1: To what extent were each of the anticipated project

outcomes achieved? 

EQ 2: Which of these outcomes have been sustained over time 

and why? 

EQ 3: Which of the BFA’s project outcomes were not achieved 

and why?

Bixal’s qualitative evaluation methodology included a desk review of 

OTA’s project documents; interviews with a total of 16 respondents 

from six organizations, including OTA as well as AGD, Bank of 

Jamaica (BOJ), MOFPS, and the International Monetary Fund (IMF). 

This evaluation did not assess OTA’s projects focused on IT 

governance or International Public Sector Accounting Standards 

(IPSAS). The evaluation team had some difficulty securing 

interviews with the GOJ and based much of the evaluation on OTA

project documentation, external reference documents, and interviews 

with OTA staff.

BFA Objectives 

1. Improve cash 

management through 

creation of a bank 

account inventory 

with signatories and 

visibility to all 

accounts.

2. Review pension 

process and internal 

controls to ensure 

accurate 

distributions to 

pensioners.

3. Implement a daily 

reconciliation 

process for 

Consolidated Fund 

operations.

4. Assist with 

implementation of

the Travel Card 

program.

5. Enhance the capacity 

of the Government 

Accounting and 

Reporting Unit 

(GAR) through a 

review of job 

descriptions and 

standardization of 

information flow.
Jamaica Budget and Financial Accountability Project Evaluation Report

2

Summary of Findings

Underlying the five key objectives was OTA’s desire to improve public financial management 

for sustainable fiscal consolidation. Overall achievement of the project objectives was partial 

and mixed (Table 1). 

Table 1: BFA Project Overview: Fully Achieved, Partially Achieved, and Not Achieved

objectives and outcomes.

Objective Area Outcome Focus 

1. Cash 

Management 

GOJ bank account visibility and tracking

Consolidate cash resources (Treasury Single Account (TSA))

Regular resource reports for active cash management 

2. Pension 

Reform

Plan for/enhance administration of pension benefits and payments 

Enhanced plan for functionality/systems of pension benefits and 

payments

3. Daily 

Reconciliation 

Automated reconciliations to improve cash management and internal 

controls

4. Travel Card Travel Card program for government travel expenditures 

Reconciliation process to ensure compliance of travel expenditures 

5. Government 

Accounting and 

Reporting Unit 

Effective staffing, protocol use, and capacity assessments

Timely and accurate financial reports for enhanced decision-making

PROJECT OBJECTIVE 1—CASH MANAGEMENT

OTA provided guidance and assistance on the Central Treasury Management System and 

the TSA to improve the predictability of cash flow, cash management, and the budget 

process. The target was to increase the share of payments recorded and performed through 

electronic fund transfer to the TSA. Project Objective 1 was partially achieved because not all 

relevant accounts were incorporated into the TSA.

PROJECT OBJECTIVE 2—PENSION REFORM 

The passage of the Pension Law in 2017 and implementation of related regulations in 

2018 facilitated progress in pension reforms. OTA provided process flow maps, an action 

plan for implementing system enhancements, processes for disbursement, and validation to the 

AGD. The planned user testing of the BOJ Financial Management Information System (FMIS) 

was not completed and the system was not yet operationalized. The follow-on training for the
Jamaica Budget and Financial Accountability Project Evaluation Report

3

completed pension reforms was partially achieved. There was insufficient time to implement all 

reforms during the OTA project. Project Objective 2 was partially achieved because time 

constraints limited full implementation.

PROJECT OBJECTIVE 3—DAILY RECONCILIATION 

OTA supported the AGD to automate daily account reconciliations and limit the need for 

engaging in manual adjustments. All relevant accounts, however, are not in the TSA, so 

this improvement is not instituted across the GOJ. Additionally, MOFPS has only limited digital 

access to the TSA due to incomplete FMIS operations. Project Objective 3 was partially 

achieved; OTA assistance helped the BOJ automate and standardize reconciliation of

accounts in the TSA.

PROJECT OBJECTIVE 4—TRAVEL CARD

The OTA and its counterparts identified the goal of establishing a travel credit card system 

for government employees. This would allow for better control of expenses and easy 

reconciliation of travel-related expenses. Despite significant efforts by OTA toward this

objective, Project Objective 4 was not achieved because the GOJ decided to pause the Travel 

Card program for reasons not made clear to the evaluation team.

PROJECT OBJECTIVE 5—GOVERNMENT ACCOUNTING & REPORTING UNIT 

The evaluation team found that the objective of enhancing the capacity of the Government 

Accounting and Reporting (GAR) unit was not achieved. Project Objective 5 included 

outcomes focused on updating job descriptions and using them to better staff the unit (Objective 

5.1) and building capacity for accurate and timely financial reporting (Objective 5.2). At the 

conclusion of OTA’s engagement, the job descriptions were updated, but no other activities were 

completed. Although Outcome 5.2 was partially achieved, Project Objective 5 was not 

achieved.

Conclusions and Recommendations

The evaluation team found that the five objectives were highly relevant to country 

conditions at the time of implementation and closely aligned with the government’s strategy and 

priorities. Of the five objectives, three were partially achieved, and two were not achieved. 

Under the fiscal consolidation objectives, the project targeted progress on pension reform and 

improved cash management leading to a consolidation of accounts within the TSA. Reporting 

on the various outcomes was complicated by a lack of monitoring data. The OTA End of Tour 

Report described outcomes, but the lack of AGD data made it difficult to quantify change. Some 

reforms, such as cash management and pension reform, took longer than originally envisioned. 

Cumulative evidence suggests that the reforms supported by the OTA related to cash and 
Jamaica Budget and Financial Accountability Project Evaluation Report

4

treasury management have been sustained. The evaluation team reached the following main 

conclusions:

1. Project design should be more holistic and data driven. The project documentation 

did not demonstrate adequate consideration of assumptions, behavior change, risks, or the

theory of change (TOC) at the project design and planning phase. 

2. In designing and sequencing complex reform projects such as the Jamaica BFA, it is 

important for OTA to demonstrate quick and important wins. This helps to ensure 

support for reforms (e.g., pension reform and TSA development) that require more time 

to build consensus and to prepare and implement modifications to legal and policy 

frameworks. However, the need for quick results should not occur at the expense of 

project design and planning.

3. TA was well received, but not fully effective at achieving the project aims. 

Significant progress was made in cash management and pension reforms, but not fully 

operationalized during the project period. The Travel Card program and building 

capacity of the GAR largely were not achieved. Continued staffing changes within the 

GOJ, which impacted continuity and high-level buy-in, also limited progress. The 

workplan appears overly ambitious given how complex and multidisciplinary the 

activities were.

4. Achieving objectives and sustaining outcomes requires a complex set of actions by a 

number of stakeholders. The involvement of a range of stakeholders (including OTA, 

AGD, BOJ, MOFPS and other GOJ institutions, and donors) continues to be necessary to 

move reforms forward and critical coordination could be strengthened. 

The evaluation team found that the OTA workplan was well aligned to the needs of the GOJ, but

the workplan may have included too many objectives requiring multidisciplinary skills for the 

timeframe, especially given interruptions due to COVID-19, staffing changes, and necessary 

sequencing of activities.

RECOMMENDATIONS

The evaluation team makes the following key recommendations:

Engaging Stakeholders

1. Deepen engagement across stakeholders and staff levels. Intentional engagement 

with stakeholders (e.g., relevant agencies and ministries, and donors) coupled with 

working with staff across all levels, can increase understanding of the local context, 

further counterpart engagement, and better direct the project activities. Consider 

convening objective-specific working groups in future engagements. 
Jamaica Budget and Financial Accountability Project Evaluation Report

5

2. Engage collaboratively with the donor community. Develop more formalized, 

higher-level coordination between donor agencies to ensure efficiencies and 

streamline synergies.

3. Ensure counterpart governments have necessary infrastructure in place to 

support and sustain recommended reforms. Assess the status of extant systems 

needed to support activities when developing objectives and activities to help 

determine if major infrastructure overhauls are necessary to achieve project activities

and outcomes.

4. Engage counterpart in greater communication, monitoring, and evaluation.

Create agreements on realistic activities and proposed outputs, review output 

integration with other international or donor activities, and establish a regular 

quarterly/semiannual monitoring and review process. Without reliable data and a 

clear M&E plan it is not possible to accurately evaluate certain outcomes and the

broader objectives of the OTA BFA Team in Jamaica.

Programmatic Planning & Management

5. Incorporate a comprehensive programmatic risk assessment into project

development. Identify risks that may impact achievement of outcomes during the 

project design—use a systems approach to identify and map potential risks at all 

levels, determine the risk severity, and develop mitigation measures.

6. Develop proposed reform strategy in a sequenced manner. Use the risk 

assessment to inform project design (e.g., status, system and capacity constraints, and 

opportunities for reform) in order to plan reforms and activities in the appropriate 

sequence.

7. Develop a context-aware long-term strategic plan. This long-range plan is an 

essential element for designing project-level, long-term, holistic strategic plans, and 

corresponding performance indicators. 

8. Develop a comprehensive logic framework at the planning stage and update 

throughout the project lifecycle. This is an essential tool for planning and 

implementing project activities that can be used to track progress and subsequently be 

evaluated to assess outcomes and impact—the basis for the monitoring and evaluation 

(M&E) plan. 

II. BACKGROUND AND PROJECT OVERVIEW

Although the GOJ has made progress stabilizing the economy and reducing reliance on external 

debt, there remains a need to strengthen governance and improve public financial management to 
Jamaica Budget and Financial Accountability Project Evaluation Report

6

create the fiscal space for needed investments in infrastructure and social services and to promote 

inclusive growth.

The GOJ has historically had weak oversight of its cash and treasury operations, resulting in 

inefficient use of the government's financial resources to support its projects and priorities, as 

well as increasing the risk of accounting and payment errors and misuse or loss of financial 

records. This was due in part to a lack of access to and knowledge of available government bank 

accounts. The AGD, located in the MOFPS, made progress in modernizing and integrating 

treasury systems. However, the AGD continued to lack the capacity to compile timely and 

accurate financial information that undermined the GOJ’s ability to make optimal financial 

decisions. The AGD also relied too heavily on manual accounting operations including 

operational payment processes (government travel), payments to pensioners, and the 

reconciliation of cash transactions, all of which hampered expenditure tracking and cash 

management operations. Reliance on manual processes also presented internal control risks, 

including the increased likelihood of inaccurate benefit calculations to pensioners and loss of 

records.

The goal of this OTA project was to strengthen the management of public finances to help ensure 

the availability of sufficient funds to meet government service delivery needs by improving cash 

management and internal controls of the AGD. Project objectives included: 1) enhancing the 

oversight and management of cash resources; 2) improving and automating pension benefit 

calculations and payments; 3) enhancing the effectiveness of the Government Financial 

Management System to support automated reconciliation of the Central Processing Account; 4) 

implementing a Travel Card payment program to improve controls on government travel 

expenditures; and 5) enhancing the ability of the AGD reporting unit to provide timely and 

accurate financial information to management.

OTA addressed the project components by providing TA using resident-based and intermittent 

expert advisors who have worked closely with counterparts in the AGD, BOJ, and MOFPS. The 

project began in April 2017 and ran through May 2021. During the project, there was significant 

turnover at the top levels of the MOFPS, including the Minister of Finance, Financial Secretary, 

and the Accountant General, which posed challenges to continuity. The final year of the project

(2021) coincided with the COVID-19 pandemic. Although the Resident Advisor remained in the 

country, work during the period of March 2020 to May 2021 was conducted remotely from the 

advisor’s residence. The requirement for remote work, coupled with attention diverted to 

pandemic-related health and economic issues, reduced the impact the project was able to 

achieve.

III. EVALUATION METHODOLOGY AND PURPOSE

The evaluation of OTA’s BFA project serves two overarching purposes:
Jamaica Budget and Financial Accountability Project Evaluation Report

7

1. To ensure compliance with the Foreign Aid Transparency and Accountability Act 

(FATAA) of 2016,1 which includes requirements to evaluate the outcomes and impacts of 

OTA’s projects whose dollar value equals or exceeds the median project size and apply the 

findings and conclusions of such evaluations to proposed project design.

2. To provide OTA with in-depth, third-party, retrospective insights of its BFA project activities 

in Jamaica. The evaluation will focus on the key evaluation questions related to its 

contributions to Jamaica’s public financial management sector development. Some aspects 

of OTA’s experience in Jamaica may be extrapolated to current or future projects in Jamaica 

or other countries to inform adaptation, improve projects, discover ways to reduce the time to 

achieve project outcomes, identify new ways to effect positive change, and document project

achievements and success.

This evaluation did not assess OTA’s Jamaica projects focused on IT governance or IPSAS.

Evaluation Questions

Bixal’s evaluation team collaborated with OTA’s M&E team and project team to develop three 

EQs to better understand the OTA project in Jamaica. The EQs provided the foundation for the 

evaluation review. Bixal then wrote interview guides and other evaluation instruments 

predicated on the EQs and related areas of interest previously defined with OTA. The following 

overarching EQs were deployed:

EQ 1: TO WHAT EXTENT WERE EACH OF THE ANTICIPATED PROJECT

OUTCOMES ACHIEVED? 

This question evaluated which project objectives were achieved by the BFA project. 

EQ 2: WHICH OF THESE OUTCOMES HAVE BEEN SUSTAINED OVER 

TIME AND WHY? 

This question focused on the sustainability of OTA’s contributions to Jamaica’s 

public financial management practices and institutions.

EQ 3: WHICH OF THE BFA’S PROJECT OUTCOMES WERE NOT 

ACHIEVED AND WHY? 

This question evaluated both enabling and constraining factors to project

achievements, including external factors outside OTA’s influence.

1 Foreign Aid and Transparency and Accountability Act of 2016, Public Law 114-191 

https://www.congress.gov/114/plaws/publ191/PLAW-114publ191.pdf
Jamaica Budget and Financial Accountability Project Evaluation Report

8

In the evaluation inception report, a fourth EQ was considered: Did OTA allow sufficient time to 

achieve full implementation of project objectives? The answer to this question is incorporated 

into answers to EQ 1, EQ 2, and EQ 3.

Annex 2 presents a matrix of the project overview; a matrix of outcomes achieved and sustained 

(EQ 1 and EQ 2) and objectives not achieved (EQ 3).

Evaluation Methodology and Process

The evaluation team consisted of a team leader with an extensive background in program

evaluation; a subject matter expert in public financial management; an M&E specialist working 

remotely and one Jamaican M&E specialist, both with extensive qualitative evaluation 

experience. 

Primary evaluation methods included a desk review of project documents provided by OTA and 

context documents; key informant interviews (KIIs) and/or group interviews; and results 

mapping. The combination of these methods provided multiple perspectives and sources that 

support findings related to the EQs. Annex 1 includes a data source matrix.

DATA COLLECTION 

Document Review 

The evaluation team members collaborated on the document review process through a multiphased process that included the organization of the entire suite of OTA documents into 

categories; the selection and prioritization of high-value sources; the identification of preliminary 

findings for deeper probing and verification; the identification of context (non-OTA) sources to 

fill knowledge gaps and triangulate/validate findings from other sources; and a data coding 

process using the ATLAS.ti qualitative analysis application that enables categorization of 

document sources, text excerpts, themes, key words, etc. 

KIIs 

The evaluation team created questionnaires based on the EQs. All interviewers closely followed 

prepared interview guides and instructions to ensure that interviews were unbiased and focused 

on the EQs. In total, there were sixteen respondents, including representatives from AGD, BOJ, 

MOFPS, IMF, as well as OTA. Most interviews lasted approximately one hour; some lasted 90 

minutes. One limitation of this evaluation was that all interviews were conducted virtually, at 

the request of the GOJ. Annex 6 contains a list of KIIs conducted.

All KIIs were recorded and transcribed. Qualitative analysis of KII data was conducted on an 

ongoing basis concurrent with fieldwork to ensure quality and the identification of emerging 

findings and trends as soon as possible during the evaluation’s data collection phase. KII notes 

were coded and analyzed with the ATLAS.ti application by grouping similar data under themes 
Jamaica Budget and Financial Accountability Project Evaluation Report

9

or categories so the evaluation team could easily locate, retrieve, and combine the textual data 

that corresponds to an EQ, concept, or theme of interest. 

Results Mapping

Evaluation team members created one matrix for EQ 1 and EQ 2 and a separate matrix for EQ 3, 

which can be found in Annex 2. The matrix for EQ 1 and EQ 2 contains data related to 

achievement and sustainability of project outcomes. The matrix for EQ 3 contains data related to 

non-achievement of project objectives.

DATA VALIDATION

Evaluation team members convened an extended analysis workshop. During this workshop, the 

team filled in findings for each achieved/sustained outcome in the matrix for EQ 1 and EQ 2 and 

findings for the unachieved objectives in the separate matrix for EQ 3. Preliminary findings and 

emerging trends were debated among the team. The team identified knowledge gaps, exchanged 

insights, sought clarification, and proposed next steps for triangulation and validation. 

On May 12, 2023, insights and ideas from the analysis process were presented to the OTA M&E 

team as preliminary findings. This presentation provided the OTA M&E team an opportunity to 

vet and discuss the findings before the final evaluation report was drafted. 

QUALITY ASSURANCE

The evaluation team followed standard quality assurance best practices that include the 

following:

1) Notetakers uploaded their notes and transcripts after each KII. The team leader/Bixal 

staff reviewed notes and transcripts for accuracy, completeness, precision, and quality.

2) Three members of the evaluation team met daily while onsite in Kingston, Jamaica. 

Other members of the team contributed to data analysis virtually and through written 

comments. The data analysis meetings provided an opportunity for evaluation team 

members to check each other’s assumptions, identify knowledge gaps, brainstorm about 

next steps for data triangulation/validation, and cross-check data quality.

3) The ATLAS.ti qualitative data analysis software enabled the team to check for accuracy 

and completeness of qualitative data as they were entered and coded. Standard crosschecking protocols were followed to ensure consistency and accuracy.

4) The ATLAS.ti application enabled the team to efficiently compile quotes from 

documents and KIIs to support findings, inform results mapping, and complete the matrix 

for EQ 1 and EQ 2 and the matrix for EQ 3.
Jamaica Budget and Financial Accountability Project Evaluation Report

10

LIMITATIONS

Lack of TOC or logframe in project planning

A very useful tool for planning, evaluation, and learning is a TOC and is the cornerstone of best 

practices for practical M&E. In the project planning phase, OTA did not establish and document 

a TOC or note the behavior change required for objectives to be achieved and outcomes to be 

sustained. OTA also did not document its assumptions or anticipate the various additional 

numerous risks that hindered achievement of the project outcomes. Finally, the project lacked a 

comprehensive M&E plan to measure and assess achievement. The lack of typical project

design elements in addition to TOCs, including planning stage original logic framework 

(logframe) or results frameworks, performance indicators, and baseline metrics, limited the 

ability of the evaluation team to design an evaluation focused on project performance using bestpractice evaluation methods. 

Limited monitoring data

The OTA End of Tour Report describes project outcomes, however, monitoring data are lacking 

to quantify achievements.

Limited number of respondents 

It was difficult to get the necessary individuals within the GOJ to commit to interviews. Four 

representatives of the AGD participated in interviews, but the evaluation team was not able to 

talk to senior executives at either the BOJ or the MOFPS. The findings are therefore based on 

this limited number of interviews with the GOJ, as well as the OTA documentation and 

interviews.

All KIIs were conducted virtually

Due to adverse timing issues outside of the evaluation team’s control, the GOJ was conducting 

its annual budget discussions at the time of the evaluation; therefore, many government 

employees and BOJ counterparts would not be available for discussions. All interviews were 

conducted virtually, which the evaluation team found challenging. Technology—particularly 

connectivity issues—also proved to be an impediment. 

IV. PROJECT FINDINGS 

The overarching goal of the OTA BFA project was to improve public financial management for 

sustainable fiscal consolidation. The project focused on five key areas: (i) government cash 

management, (ii) pensions, (iii) Travel Card efficiency, (iv) automated reconciliation, and (v)

improving government reporting processes. The evaluation team evaluated progress against each 

of the five objectives in the following sections. 
Jamaica Budget and Financial Accountability Project Evaluation Report

11

The evaluation team found some inconsistencies in the intended outcomes in the logic 

framework (logframe) used to describe OTA’s project and those cited in the End of Tour Report 

prepared by advisors. Table 2 presents the objectives and outcomes as presented at the end of 

the project as they appear in the final advisor report, along with a high-level assessment of 

achievement by the evaluation team.

Table 2: Summary Review of OTA’s Jamaica Budget and Financial Accountability Project.

Objective Status Outcome Status

Objective 1:

Improve cash 

management through 

creation of a bank account 

inventory with signatories 

and visibility to all 

accounts 

AGD has full visibility into GOJ 

bank accounts. The AGD effectively 

utilizes the bank accounts to track all 

accounts on an ongoing basis.

The government achieved 

inventory of all accounts. 

Sustained. 

AGD uses its procedures to close 

unnecessary accounts and consolidate 

government cash resources under its 

control.

The government rolled in some 

but not all ministries, 

departments, and agencies

accounts into a TSA. Transfer 

procedures and regulations 

operationalized.

AGD produces and uses liquidity 

reports to actively manage government 

cash resources. The GOJ has improved 

understanding of its cash position and 

uses this information to inform borrowing 

and payment decisions.

Government cash resource reports 

are submitted on a regular basis. 

Procedures operationalized.

Sustained.

Objective 2:

Review pension 

process and internal 

controls to ensure accurate 

distributions to pensioners

The GOJ uses process flow maps, 

concept papers, and action plans to 

integrate and enhance system functionally 

to administer pension benefits and 

payments efficiently and accurately. 

Process maps, concept map, and 

action plan developed. Pension 

reforms are difficult to achieve, 

requiring significant buy-in from 

many actors. Not all 

functionalities were integrated 

into the pension system.

Enhanced pension administration 

reduces risks of ineligible payments and 

inaccurate benefit calculations.

One output was partially

achieved—training for pension 

staff employees. 

Objective 3:

Implement a daily 

AGD Cash Management Unit uses 

automated reconciliations to timely, 

Cash management reconciliations 

were operationalized and 
Jamaica Budget and Financial Accountability Project Evaluation Report

12

Objective Status Outcome Status

reconciliation process for 

Consolidated Fund 

operations

efficiently, and effectively manage cash. 

Automated reconciliations improve 

internal controls, reducing the opportunity 

and need for manual adjustments.

automated for accounts that were 

in the TSA. Sustained.

Objective 4:

Assist with 

implementation of the 

Travel Card program

AGD implements a Travel Card

program to improve the efficiency 

of travel expenditures and improve 

internal controls over government travel, 

including through preauthorized limits 

and increased availability of supporting 

documents.

Decision-making at GOJ on hold.

AGD implements a reconciliation 

process that improves internal 

control and transparency of travel 

expenditures by ensuring that 

expenditures are authorized and comply 

with regulations.

Travel Card program not 

implemented. GOJ requested

postponement.

Objective 5:

Enhance the 

capacity of the GAR 

through a review of job 

descriptions and 

standardization of 

information flow

AGD uses updated job 

descriptions to effectively staff the 

unit and uses recommended protocols and 

operational capacity assessment to 

enhance GAR.

Personnel descriptions were 

generated. With ongoing change 

management processes, these 

were not routinely upgraded or 

updated. 

New GAR unit consistently 

develops accurate and timely 

financial reports to respond to internal and 

external queries and facilitates enhanced 

decision-making by management within 

GOJ ministries.

The government is implementing 

new FMIS processes with the 

World Bank. These systems were 

not in operation.
Jamaica Budget and Financial Accountability Project Evaluation Report

13

Project Objective 1—Cash Management 

IMPROVE CASH MANAGEMENT THROUGH CREATION OF A BANK ACCOUNT 

INVENTORY WITH SIGNATORIES AND VISIBILITY TO ALL ACCOUNTS

EQ 1: To what extent were each of the anticipated project outcomes achieved? 

Outcomes Status: 

• Outcome 1.1: Achieved

• Outcome 1.2: Partially achieved

• Outcome 1.3: Partially achieved

For Outcome 1.1, reporting on cash resources has been automated and routinized 

within the AGD to provide better information for decision-making. 

EQ 2: Which of these outcomes have been sustained over time and why? 

Outcomes Sustained: 

• Outcome 1.1: Achieved

• Outcome 1.2: Partially 

• Outcome 1.3: Partially 

Bank reconciliation practices for the accounts in the TSA were formalized, used 

regularly, and integrated into routine operations. The BOJ FMIS was developed and had 

begun user acceptance testing—but was not yet operationalized.

EQ 3: Which of the BFA’s project outcomes were not achieved and why? 

Outcomes Not Achieved: 

• None

All outcomes under Objective 1 were at least partially achieved, but the outstanding 

accounts to be added to the TSA prevent full achievement of this objective. 

Outcome 1.1: The AGD has full visibility into GOJ bank accounts. The AGD effectively 

utilizes the Banking Platform to track all accounts on an ongoing basis.

Speaking to EQ 1, the evaluation team found that this outcome was fully achieved. OTA 

provided guidance and TA to AGD focused on the Central Treasury Management System to 

consolidate accounts within the TSA. The evaluation team, through an interview with the AGD, 
Jamaica Budget and Financial Accountability Project Evaluation Report

14

found that the department now has full visibility into GOJ bank accounts. The AGD effectively 

utilizes the bank account balances to track all accounts on an ongoing basis.

It is not uncommon for a country to lack a complete inventory of bank accounts with government 

cash resources. However, a government accounting system should be designed to record all 

transactions and capture relevant information independently of the cash flows in specific bank 

accounts. At the beginning of the OTA BFA project, information on government bank accounts 

was incomplete. The GOJ had approximately 3,000 bank accounts dispersed across several 

banks in the country according to project documents. This complicated the initial inventory of 

government bank accounts and weakened the oversight of MOFPS. 

According to respondents, with the support of OTA, the GOJ was able to complete at least a 

basic inventory of accounts for all ministries, departments, and agencies. From the inventory, 

the GOJ was able to identify unused accounts or accounts that could be consolidated, resulting in 

an overall reduction in accounts. The AGD did not provide monitoring data on the number of 

accounts closed or account inventory, which would have been useful to document the extent of 

the consolidation process. This finding relates to EQ 2 on sustainability, as it is unclear how 

much progress the AGD has made on closing accounts or how many more might need closing, as 

is discussed further under Outcome 1.2.

Outcome 1.2: AGD uses its procedures to close unnecessary accounts and consolidate 

government cash resources under its control.

Speaking to EQ 1, the evaluation team found that this outcome was partially achieved. 

Consolidation of cash resources in a TSA is critical for effective cash management. According 

to the IMF, the consolidation of cash resources was a priority reform of the GOJ.

Based on project documents and an interview Bixal conducted with the AGD, the evaluation 

team found that the GOJ, with assistance from OTA, consolidated many (but not all) of its bank 

accounts into its TSA and developed procedures for closing/transferring accounts. GOJ also 

developed and implemented a phased-in approach for progressive cash consolidation through a 

TSA system. The phased approach was used to alleviate concerns regarding the banking sector’s 

liquidity while transferring balances. However, according to an interview with the BOJ, a 

considerable number of bank accounts remain outside the TSA. The specific number was not 

quantifiable due to the lack of data. 

In terms of EQ 2 focused on sustainability, the evaluation team also found, based on an interview 

that the bank reconciliation practice for the accounts in the TSA has been formalized, is 

undertaken on a regular basis, and is now a routine part of operations. The BOJ FMIS is updated 

with the new procedures and is currently in user acceptance testing. The FMIS is not yet 

operationalized, but it is a critical step in ensuring that cash management and reporting 

operations are fully implemented and routinized.
Jamaica Budget and Financial Accountability Project Evaluation Report

15

In terms of EQ 3, due to a lack of monitoring data from the AGD on the number or portion of 

accounts closed, it is not possible to discern how close OTA and its counterparts came to 

achieving Outcome 1.2.

Outcome 1.3: AGD produces and uses liquidity reports to actively manage government 

cash resources. The GOJ has improved its understanding of its cash position and uses this 

information to inform borrowing and payment decisions.

In terms of EQ 1, the evaluation team found that this outcome was partially achieved. As the 

coverage of TSA and treasury oversight of cash resources expands, the government should 

require full reporting by banks (to be defined in formal agreements) and ensure that bank 

reconciliations are duly conducted in a timely manner—at least monthly. To this end, the 

evaluation team found that the cash reporting structures are being undertaken digitally, whereas 

before OTA interventions, the cash management reporting was undertaken via paper submission 

and reporting. In an interview the BOJ confirmed they provide real time information for three of 

the eight accounts in the TSA. A digital format creates efficiencies by allowing data to be 

aggregated and analyzed more easily. It also increases data quality by decreasing manual 

manipulation. Digital reporting has allowed the GOJ to use data for cash management. While 

progress has been made, the reporting does not include all accounts. The BOJ confirmed that a 

substantial number of ministry, department, and agency accounts are outside of the TSA, leaving 

gaps in information for decision-making.

In terms of EQ 2 around sustainability, the evaluation team found that government cash resource 

reports are used on a regular basis and that procedures are operationalized. As noted previously 

for EQ 3, the process does not yet include all accounts, which hinders full achievement.

CONCLUSION AND SUMMARY

Objective 1 focused on increasing the share of ministry, department, and agency financial 

activity recorded and performed through electronic fund transfer by the AGD. Based on the data 

collected, the evaluation team found that OTA provided guidance and assistance to the AGD to 

enhance the efficiency and predictability of cash management by introducing reforms to the 

FMIS and the TSA. During the OTA project, the GOJ was able to create an inventory of a 

substantial number of ministry-, department-, and agency-level accounts, and based on this

inventory, the GOJ consolidated and closed accounts, reducing the total number to reduce 

management and increase visibility of its cash flow. Reporting on cash resources has been 

digitized and routinized within the AGD to provide better information for decision-making. 

While much progress has been made on Objective 1, the evaluation team found that not all 

accounts had been included in the consolidation process, resulting in continued gaps in 

information. 

In terms of EQ 2, the relative sustainability of the outcomes under this objective varied. Many of 

the operational changes have happened to ensure sustainability, the implementation of standard 

operating procedures and reporting, digitization of reports, and updates to the BOJ FMIS. 
Jamaica Budget and Financial Accountability Project Evaluation Report

16

However, the FMIS is still in user testing, so it is not fully operational. In addition, not all 

accounts have been consolidated, leaving information gaps. These issues will need to be 

addressed to ensure full functionality into the future.

In terms of EQ 3, better monitoring data would help quantify what portion of accounts remain to 

be consolidated. The evaluation team recognizes that this process is complex and takes time. 

The GOJ did prioritize work under Objective 1; however, high-level staffing changes within the 

AGD was a limiting factor in maintaining momentum and leadership. 

Project Objective 2—Pension Reform 

REVIEW PENSION PROCESS AND INTERNAL CONTROLS TO ENSURE 

ACCURATE DISTRIBUTIONS TO PENSIONERS

EQ 1: To what extent were each of the anticipated project outcomes achieved?

Outcomes Status: 

• Outcome 2.1: Partially achieved

• Outcome 2.2: Partially achieved

The support provided was lauded by the AGD as a significant accomplishment—this 

included development of guiding documents advising on a process to validate pensioners 

to cut down on erroneous payments and ensure accuracy of legitimate disbursements. 

There are still areas for improvement, such as comprehensively validating pensioners to 

ensure accuracy of payments. 

EQ 2: Which of these outcomes have been sustained over time and why? 

Outcomes Sustained: 

• Outcome 2.1: Partially sustained

• Outcome 2.2: Partially sustained

The completed processes and reforms were sustained but the outcomes were not fully 

achieved. The complexity of changes will require continued attention by the AGD to 

ensure future sustainability.

EQ 3: Which of the BFA’s project outcomes were not achieved and why? 

Outcome Not Achieved: 

• These outcomes were partially achieved

As noted above, there was only partial achievement in this objective area. Limiting

factors included a lack of high-level buy-in and lack of routine counterpart input. Full 
Jamaica Budget and Financial Accountability Project Evaluation Report

17

implementation of the pension administration system was hampered by the lack of user 

accessibility testing and staff training for the new application and validation of pensioners 

to ensure accuracy of payments remains outstanding.

Outcome 2.1: Review pension process and internal controls to ensure accurate distributions 

to pensioners. The GOJ uses process flow maps, concept paper and action plan to integrate 

and enhance system functionally to administer pension benefits and payments efficiently 

and accurately. 

In terms of EQ 1, the evaluation team found that Outcome 2.1 was partially achieved. OTA did 

provide TA to the GOJ to implement pension reforms, including development of guiding 

documents such as process flow maps and an action plan to integrate and enhance functionality

into the pension administration system (PAS), and staff training on the PAS upgrades. At the 

beginning of the OTA project, the PAS was manually updated, without sufficient staff to 

maintain the volume of transactions. The process was also not integrated with the payroll system 

and validation of pensioners was not done consistently. Respondents noted that the 

recommended upgrades to the PAS have been made; however, gaps remain. User acceptance 

testing has remained incomplete for two years. Additionally, the validation of overseas 

pensioners has not been addressed, resulting in the GOJ erroneously paying pensioners who have

died.

In terms of sustainability, collaboration between OTA and the GOJ resulted in needed reforms to 

the PAS. According to interviews with the AGD, the help OTA provided with the PAS system 

was its most substantial achievement of the project and that the transition remains a major 

priority for the GOJ. It is unclear how sustainable the work will be given continued delays in 

user testing and training. AGD turnover may also decrease sustainability over time.

In terms of EQ 3, the evaluation team noted that the project lacked high-level buy-in, which was 

exacerbated by turnover. The evaluation team also found that consistent formal counterpart 

input was not routinely collected during this intervention, possibly due to the buy-in issues.

Outcome 2.2: Enhanced pension administration reduces risks of ineligible payments and 

inaccurate benefit calculations.

In terms of EQ 1, the evaluation team found that Outcome 2.2 was partially achieved. Under 

the IMF Standby Loan Credit Arrangement (2017), pension reform passage has paved the way 

for early retirement and should eventually stabilize the GOJ’s macroeconomic fiscal framework. 

The evaluation team concluded that this progress was aided by assistance OTA provided to the 

GOJ. 

In terms of the more operational level, as addressed in the discussion of Outcome 2.1, OTA 

assisted the GOJ with implementing system enhancements and updating their processes for 

disbursement and pension validation. While respondents told the evaluation team that the OTA 
Jamaica Budget and Financial Accountability Project Evaluation Report

18

interventions led to some improvements in the process of validating pensioners were still alive, 

others noted that some problems persist. 

In terms of EQ 2, the sustainability of this outcome remains uncertain. Full implementation will 

require completion of the training on the procedures and system, and it is unclear if the GOJ will 

commit those resources and time. 

The project was not able to achieve full implementation of the PAS system (EQ 3), noting 

specifically the lack of user accessibility testing and training of staff on the new application. 

Validation of pensioners to ensure accuracy of payments remains outstanding. The evaluation 

team found that more time may have facilitated further progress. The lack of or inconsistent 

high-level buy-in is also a key factor in not fully achieving outcomes. Support from senior 

leaders is necessary to establish priorities for staff and dedicate necessary resources.

CONCLUSION AND SUMMARY

Pension reform passage in 2017 provided the legal framework for OTA to make further progress 

on Objective 2. OTA provided specific TA over the five-year period to reform the PAS. 

Specific activities included the development of guiding documents such as process flow maps 

and an action plan to integrate and enhance functionally. OTA also advised on a process to 

validate pensioners to cut down on erroneous payments and ensure accuracy of legitimate 

disbursements. This support was lauded by the AGD as a significant accomplishment. 

However, gaps remain in terms of complete implementation. The PAS has been updated, but 

user acceptability testing has not been implemented to ensure the system meets the needs of the 

organization. Training on the new PAS functionality was partially implemented according to an 

interview with the AGD and the End of Tour Report, but the evaluation team did not find 

evidence that the training was ongoing. Finally, while progress has been made, the GOJ was still 

not comprehensively validating pensioners to ensure accuracy of payments. 

While GOJ has stated that pension reform is a priority, continued turnover of leadership within 

the AGD has hampered progress. The complexity of changes will also take time and continued 

attention by the AGD to ensure sustainability.

Project Objective 3—Daily Reconciliation 

IMPLEMENT A DAILY RECONCILIATION PROCESS FOR CONSOLIDATED FUND 

OPERATIONS

EQ 1: To what extent were each of the anticipated project outcomes achieved? 

Outcomes Status: 

• Outcome 3.1: Partially achieved
Jamaica Budget and Financial Accountability Project Evaluation Report

19

The objective was achieved for the accounts within a consolidated account, in this case 

the TSA. OTA’s support contributed to the adoption of daily reconciliations and

automation of the reconciliations process for the TSA. Not all accounts were in the TSA, 

so this objective was only partially achieved. 

EQ 2: Which of these outcomes have been sustained over time and why? 

Outcomes Sustained: Partially

The automated cash management procedures and reconciliations for accounts in the TSA 

were sustained. As noted in EQ 1, the new procedures and reconciliation processes 

adopted only cover accounts that were within the TSA. 

EQ 3: Which of the BFA’s project outcomes were not achieved and why? 

Outcomes Not Achieved: 

All outcomes under Objective 3 were at least partially achieved; full achievement was 

hindered because not all bank accounts are in the TSA.

Outcome 3.1: AGD Cash Management Unit uses automated reconciliations to facilitate 

more timely, efficient, and effective cash management. Automated reconciliations improve 

internal controls, reducing the opportunity and need for manual adjustments.

In terms of EQ 1, the evaluation team found that Outcome 3.1 was partially achieved. 

Industry best practice calls for the BOJ/MOFPS to establish a daily, weekly, and monthly 

reconciliation process for all bank/cash accounts. The evaluation team found that OTA support 

has helped BOJ apply daily reconciliations to all accounts consolidated within the TSA and 

automate the reconciliations process, limiting the need for manual adjustments. The evaluation 

team concluded that the outcome was only partially achieved because not all bank accounts 

are in the TSA.

In terms of sustainability (EQ 2), the AGD stated that they have continued to implement the 

current protocols and begun to broaden the scope. The evaluation team believes that the 

MOFPS, through its FMIS, would need additional functionality to track and control specific 

categories of cash flows within the TSA. This functionality, for example, would allow the 

Ministry of Finance to control or monitor spending ceilings given to ministry, departments, and 

agencies instead of advancing physical cash to their bank accounts, whether in the central bank 

or a commercial bank. This was beyond the current scope but would significantly aid in the 

sustainability of reconciliations.

EQ 3 does not apply as this objective was achieved beyond the limitation already noted related to 

accounts that are not yet consolidated within the TSA.
Jamaica Budget and Financial Accountability Project Evaluation Report

20

CONCLUSION AND SUMMARY

Objective 3 was partially achieved. The evaluation team found that OTA support helped BOJ

apply daily reconciliations to all accounts consolidated within the TSA and automate the 

reconciliations process, limiting the need for manual adjustments. The evaluation team 

concludes that Objective 3 was only partially achieved because a substantial number of

bank accounts are still not in the TSA.

Project Objective 4—Travel Card

ASSIST WITH IMPLEMENTATION OF THE TRAVEL CARD PROGRAM 

EQ 1: To what extent were each of the anticipated project outcomes achieved? 

Outcomes Status: 

• Outcome 4.1: Not achieved

• Outcome 4.2: Not achieved

The implementation of the Travel Card program is on hold at the GOJ level, so it has 

not been implemented. Without the implementation of the Travel Card program, 

outcome 4.2 could also not be achieved. 

EQ 2: Which of these outcomes have been sustained over time and why? 

Outcomes Sustained: 

• None

None of the outcomes (4.1 or 4.2) were achieved; therefore, none were sustained.

EQ 3: Which of the BFA’s project outcomes were not achieved and why? 

Outcomes Not achieved

• Outcome 4.1: Not achieved

• Outcome 4.2: Not achieved

Objective 4 was not achieved and neither objective was achieved as the Travel Card 

program was on hold at the GOJ level. 
Jamaica Budget and Financial Accountability Project Evaluation Report

21

Outcome 4.1: AGD implements a Travel Card program to improve the efficiency of travel 

expenditures and improve internal controls over government travel, including through 

preauthorized limits and increased availability of supporting documents. 

The evaluation team found that Outcome 4.1 was not achieved. Using the logframe developed 

by the OTA advisor, the evaluation team assessed the five outputs specified under Outcome 4.1, 

including: i) Providing policies and procedure documents (including authorization levels and 

credit limits) for a Travel Card program, ii) Developing a request process for proposals and 

bidding documents, iii) Developing a cardholder authorization forms, iv) Providing training to 

AGD staff, and v) Providing procedures for Travel Card expenditure reports. The evaluation 

team found that OTA committed a substantial amount of work toward achieving Outputs 1–3; 

however, part way through the OTA project, the GOJ decided not to implement a Travel Card

program. 

The AGD indicated that they still intend to automate travel expenses, but that there are internal 

policies that need to be followed before they can move forward. They want to ensure that all 

proper procedures are followed to ensure compliance, which is routinely checked through 

internal and external audits.

Outcome 4.2: AGD implements a reconciliation process that improves internal control and 

transparency of travel expenditures by ensuring that expenditures are authorized and 

comply with regulations.

The evaluation team found that Outcome 4.2 was not achieved. With the Travel Card program 

on hold, AGD could not move forward with improving internal control and transparency of 

travel expenditures.

CONCLUSION AND SUMMARY

Objective 4 was to assist with implementation of the Travel Card program. The GOJ has put the 

Travel Card program on hold, so Objective 4 was not achieved. Further, neither Outcomes 4.1 

or 4.2 were achieved, despite substantial work on OTA’s part to set up the GOJ to implement the 

Travel Card program, including provision of policy and procedure documents, developing 

bidding and procurement documents, and provision of cardholder authorization forms. The 

AGD indicated that it does intend to automate travel expenditures in the future but wants to 

ensure all proper procedures are followed in doing so.
Jamaica Budget and Financial Accountability Project Evaluation Report

22

Project Objective 5—Government Accounting & Reporting Unit

ENHANCE THE CAPACITY OF THE GOVERNMENT ACCOUNTING AND 

REPORTING UNIT (GAR) THROUGH A REVIEW OF JOB DESCRIPTIONS AND 

STANDARDIZATION OF INFORMATION FLOW 

EQ 1: To what extent were each of the anticipated project outcomes achieved? 

Outcomes Status: 

• Outcome 5.1: Partially achieved

• Outcome 5.2: Not achieved

AGD generated some job descriptions but not the ongoing change management 

processes to support the changes. 

EQ 2: Which of these outcomes have been sustained over time and why? 

Outcomes Sustained: 

• None

Only one of the outcomes was achieved (5.1), and the evaluation team found

evidence that this outcome was sustained.

EQ 3: Which of the OTA’s project outcomes were not achieved and why? 

Outcomes Not Achieved: 

• Outcome 5.2: Not achieved

A limited number of job descriptions are being used, but it is unclear the proportion 

that have been updated. 

Outcome 5.1: AGD uses updated job descriptions to effectively staff the unit and uses 

recommended protocols and operational capacity assessment to enhance GAR. The new 

GAR unit consistently develops accurate and timely financial reports to respond to internal 

and external queries and facilitates enhanced decision-making by management within GOJ 

ministries.

The evaluation team found that Outcome 5.1 was partially achieved. AGD has updated several 

job descriptions with OTA’s support. While multiple respondents reported this outcome as 

achieved, the AGD was unable to provide data to the evaluation team on the current scope or 

percentage of the updated job descriptions compared to the total. The MOF also reported that 

some of the job descriptions were being used currently for recruiting and performance 

management, which indicates they are sustaining this achievement (EQ 2). 
Jamaica Budget and Financial Accountability Project Evaluation Report

23

Outcome 5.2: The new GAR unit consistently develops accurate and timely financial 

reports to respond to internal and external queries and facilitates enhanced decisionmaking by management within GOJ ministries.

The evaluation team found that Outcome 5.2 was not achieved. The government is 

implementing new processes within the FMIS in collaboration with World Bank, but the systems 

are not yet in operation.

Specifically, under Outcome 5.2, the GOJ requested that OTA help: i) update job descriptions, ii) 

improve financial reporting templates, and iii) provide an update to operational protocols and the 

operational assessment review. The evaluation team concluded that only one of these outputs 

was achieved—a limited number of job descriptions were updated. As Outcome 5.2 was not 

achieved, sustainability could not be assessed. 

CONCLUSION AND SUMMARY

The final objective was to enhance the capacity of the GAR through a review of job descriptions 

and standardization of information flow. Based on the data collected, the evaluation team found 

that OTA supported the GAR to generate some job descriptions (5.1) but was unable to support 

them on the standardization of information flow (5.2). Additionally, in terms of EQ 2, there was 

evidence that job descriptions were being actively used and sustained, but Outcome 5.2 was not 

achieved; therefore, it was not sustained. Overall, objective 5 was not achieved.

V. CONCLUSIONS 

Objective Area

1. Improve cash management through creation of a bank account inventory with signatories and 

visibility to all accounts

2. Review pension process and internal controls to ensure accurate distributions to pensioners

3. Implement a daily reconciliation process for Consolidated Fund Operations

4. Assist with implementation of the Travel Card program

5. Enhance the capacity of the Government Accounting and Reporting Unit through a review of 

job descriptions and standardization of information flow
Jamaica Budget and Financial Accountability Project Evaluation Report

24

The evaluation team found that the five objectives were highly relevant to country 

conditions at the time of implementation and spoke to the need to avoid fiscal insolvency and 

support stabilization and reform. The five objectives were closely aligned with the government’s 

strategy and priorities and they were relevant to PFM challenges in Jamaica, with policy 

priorities identified in consultation with GOJ counterparts. 

Of the five objectives, the evaluation team concluded that three were partially achieved, 

and two were not achieved. Under the fiscal consolidation objectives, the project targeted 

progress on pension reform and improved cash management leading to a consolidation of 

accounts within the TSA. Reporting on the various outcomes was complicated by a lack of 

monitoring data. The OTA End of Tour Report describes outcomes, but the lack of AGD data 

makes it hard to quantify change. 

Some reforms, such as cash management and pension reform, took longer than originally 

envisioned. Cumulative evidence suggests that the reforms supported by the OTA related to 

cash and treasury management have been sustained. This is reflected in the new development 

policy financing series supported by the IMF Stand-By Arrangement.

2

The evaluation team reached the following main conclusions: 

1. Project design should be more holistic and data driven. Project documentation did

not demonstrate adequate consideration of assumptions, behavior change, risks, and TOC 

at the project design and planning phase. OTA project documents did not reference 

context indicators, which are critical to understanding the operating environment. The 

absence of more robust design documentation made it difficult to assess why certain 

activities were undertaken.

2. In designing and sequencing complex reform projects such as the Jamaica BFA 

project, it is important to demonstrate quick and important wins. This helps to 

ensure support for reforms (e.g., pension reform and TSA development) that require more 

time to build consensus and to prepare and implement modifications to legal and policy 

frameworks. However, the need for quick results should not occur at the expense of 

project design and planning.

3. TA was well received, but not fully effective at achieving the project aims. While the 

OTA project was well received by both the AGD and the MOFPS, the individual building 

blocks of TA provided by OTA did not fully achieve all five objectives. Significant 

progress was made in terms of pension reforms and cash management operations; 

however, neither were fully operationalized within the timeframe of the project. The 

2

2022 "IMF Reaches Staff-Level Agreement with Jamaica on a Precautionary and Liquidity Line (PLL) and the Resilience and 

Sustainability Facility (RSF) and conducted the 2022 Article IV Consultation" No 22/435. 
Jamaica Budget and Financial Accountability Project Evaluation Report

25

Travel Card program and building capacity of the GAR largely were not achieved. The 

evaluation team concluded that the workplan was overly ambitious given the complexity 

of the activities. Progress was also limited by continued staffing changes within the GOJ, 

which impacted continuity and high-level buy-in.

4. Achieving objectives and sustaining outcomes required a complex set of actions by a 

number of stakeholders. OTA, AGD, BOJ, MOFPS, and other GOJ institutions, as well 

as donors, all had to be involved to move reforms forward. Coordination among the GOJ

and other donors, such as the IMF and World Bank, is critical; the evaluators found that 

coordination could have been strengthened. Within the GOJ, more work in the design 

phase may have benefited internal communication. While the OTA seemed to have 

informal coordination with the IMF, they could have benefitted from utilizing the more 

formal donor coordination meetings already in place. 

The evaluation team gathered evidence that confirmed the strength of government ownership of, 

and commitment to, the overall reform project, indicating that a more realistic time frame for 

some reforms (e.g., a phased approach aligned to the capacity of the GOJ) could have increased 

the achievements of the project. While OTA’s workplan was well aligned to the needs of the 

GOJ, it may have been overly ambitious for the time frame, especially given interruptions due to 

COVID-19.

VI. RECOMMENDATIONS

The evaluation team makes the following key recommendations, many of which are relevant for 

future interventions:

Engaging Stakeholders

1. Better engage all key counterpart stakeholders. More consistent and context-specific 

engagement with the GOJ. Stakeholders can include the treasury and other relevant 

agencies of the Ministry of Finance, line ministries/spending agencies, revenue agencies, 

the central bank, commercial banks involved in government payment and revenue 

collection, others in the financial market, and donors. 

One important missing step in creating engagement was working with staff across the 

agency. OTA should have formally requested GOJ’s permission to engage staff at all 

levels—not just leadership—to assist with understanding the day-to-day challenges and 

facilitate implementation of reforms. Formally engaging different stakeholders to 

understand obstacles would have enabled OTA to: i) better direct their intervention, ii) 

document various risk factors more appropriately, and iii) develop a more targeted 

intervention logframe—this would have further increased counterpart engagement. 

To the extent that specific objectives are developed in future demand-driven OTA 

interventions, future advisor(s) should establish and convene objective-specific working 
Jamaica Budget and Financial Accountability Project Evaluation Report

26

groups made up of relevant counterparts and technical staff to maximize buy-in and 

engagement. Developing this counterpart engagement and feedback loop during 

implementation would have hopefully led to increased counterpart engagement in a 

midterm or final evaluation. 

2. Engage more collaboratively with fellow donor community. Better coordination 

between OTA and major donors in the financial and economic sector—such as the World 

Bank and IMF—should have occurred to assign more complementary roles and activities 

from the beginning, specifically related to cash management activities. The evaluation 

team found that OTA did not regularly attend meetings in Jamaica sponsored by partner 

international donors. Attending the meetings would have facilitated coordination and 

collaboration for overlapping projects, such as the TSA, FMIS, accountability, and 

reporting. Although the OTA’s advisor had an informal working relationship with the 

IMF’s country director, the evaluation team recommends more formalized, higher-level 

coordination between donor agencies to ensure efficiencies and streamline synergies.

3. Ensure that counterpart governments have proper infrastructure for reforms. In the 

Jamaican case, OTA agreed to objectives that could not be achieved without major 

infrastructure overhauls. For example, the FMIS, operated by the MOFPS, should include 

additional functionality to track and control specific categories of cash flows. This 

functionality, for example, would allow the MOFPS to control or monitor spending 

ceilings given to ministries, departments, and agencies instead of advancing physical cash 

to their bank accounts. 

4. Engage counterparts in greater communication, monitoring, and evaluation. An 

inadequate M&E plan at the outset made it difficult to evaluate performance. The AGD 

and other counterpart organizations provided the evaluation team with minimal

monitoring data. Without concrete and reliable information and a clear M&E plan at the 

outset, it was ultimately impossible to evaluate how close the intervention came to 

achieving certain outcomes and broader objectives. OTA should: i) obtain a fundamental 

agreement with the government counterpart on realistic activities and proposed outputs, ii) 

review these outputs to determine how they are integrated with other 

international/stakeholder donor activity, iii) establish a regular quarterly/semiannual 

monitoring and review process, and iv) develop a memorandum of understanding with 

counterpart agencies to allow for data sharing for the development of indicators and for 

future project evaluations.

Programmatic Planning & Management

5. Conduct a risk assessment that identifies and accounts for relevant programmatic 

risks. The evaluation team concluded that OTA did not fully account for the economic 

and political environment in Jamaica during the design and planning phase of the BFA 

project. Completing a comprehensive risk assessment could help the team anticipate the 

numerous risks that hinder achievement of project outcomes. Table 3 provides a common 
Jamaica Budget and Financial Accountability Project Evaluation Report

27

set of risk categories that OTA should review and implement in future project

developments.3 OTA should use a systems approach to identify and map potential risks at 

all levels, determine the risk severity, and develop mitigation measures.

Table 3: Project Risk Categories.

Risk Categories

Project Risk Analysis Broader (outside) Project Risk

Public Accountability 

▪ Political will for sound Public Financial 

Management (PFM) reforms 

▪ Governance, including horizontal and vertical 

accountability and anti-corruption stance 

Political Economy 

▪ Macroeconomic stability 

▪ Resilience to shock (natural 

and manmade) 

Public Accountability 

▪ Sustainability 

Governance 

▪ Institutional capacity 

▪ Project oversight quality 

▪ PFM reform and capacity building agenda 

Governance 

▪ Project selection and design 

▪ Implementation modality 

Fiduciary and Accountability Risk Group

Accounting and Reporting 

▪ Financial report accuracy 

▪ Project/project accounting 

Procurement 

▪ Contract management 

▪ Inventory management 

Financial Management Operations 

▪ Budget execution, cash management 

▪ Internal and external audit 

Context 

▪ Legal and regulatory framework 

▪ Domestic financial sector capacity

▪ Domestic professional service capacity (legal) 

Project Management

▪ Meeting the challenges of 

OTA’s management structure 

and creating effective wholeof-mission teams 

▪ Value for money 

▪ Project management 

▪ Relationship management 

Reputational Risk

Partner Country 

▪ Development interest, managing expectations 

▪ Whole-of-government support 

▪ Political support 

3 USAID defines the following key categories of risk: fiduciary, human capital, information technology, legal, 

operational, programmatic/development outcome, reputational and security. See USAID Risk Appetite Statement, 

August 22, 2022. URL: https://www.usaid.gov/sites/default/files/2022-12/596mad.pdf 
Jamaica Budget and Financial Accountability Project Evaluation Report

28

Risk Categories

Project Risk Analysis Broader (outside) Project Risk

▪ Shared risk 

▪ Community and partner relations

▪ Relationships with other incountry donors 

6. Develop any proposed reform strategy in a sequenced manner. The risk assessment 

should inform the project design and help the team to understand the starting point, system 

and capacity constraints, and opportunities for reform. The reforms and activities should 

then be planned in the appropriate sequence. For example, if changes are required in the 

FMIS to operationalize a change, these must be prioritized, allowing sufficient time for the 

work to be completed before beginning the next phase of the project. This may help OTA

make their workplans more efficient for the project timeframe.

7. Develop a long-term strategic plan. OTA’s demand-driven TA approach, while 

responsive to the GOJ’s requests, did not follow a predetermined, long-term strategic plan. 

Project activity plans were frequently scaled back, apparently due to limited political will 

or government buy-in on the part of counterparts. Absent a coherent TOC, it appears to 

have been difficult for OTA to design project-level, long-term, holistic strategic plans, and 

corresponding performance indicators. As a result, the project strategy OTA arrived at did 

not seem to account for actual Jamaican institutional capacity.

8. Develop a comprehensive logic framework. OTA did not develop a clear logframe until 

well into the project cycle, after project implementation began. Logframes are essential 

tools for planning and implementing project activities that can be used to track progress 

and subsequently be evaluated to assess outcomes and impact. The logframe also serves 

as a basis for the M&E plan. Developing a logframe at the planning stage, as a living 

document to update as circumstances change through the project cycle, would serve OTA 

as a useful planning tool and also provide an invaluable reference for internal and thirdparty evaluations.
Jamaica Budget and Financial Accountability Project Evaluation Report

29

Y

VII. ANNEXES

Annex 1: Evaluation Process

Annex 2: Evaluation Question Matrices

Annex 3: OTA Project Document List

Annex 4: Context Document List

Annex 5: KII Guide

Annex 6: List of Key Informant Interviews and Respondents

Annex 7: Assumptions and Risks Identified by Evaluation Team

Annex 8: PEFA Assessment Indicators and Scores

Annex 9: Evaluation Team Qualifications 
Jamaica Budget and Financial Accountability Project Evaluation Report

30

Annex 1: Data Source Matrix

Table A-1: Evaluation Data Source Matrix.

Evaluation 

Question

Research Questions (indicative) Sources of 

Evidence

Research 

Method

1. To what extent 

were each of the 

anticipated project

outcomes achieved?

In designing this project and its 

objectives, were assumptions 

made? Did the project design 

consider risks?

Was your institution (AGD, BOJ, 

MOFPS) engaged in the design 

and planning of the OTA project?

Did OTA request feedback from 

the AGD, BOJ or MOFPS? In 

which form? When?

How was progress towards each 

objective monitored/tracked?

Comparing the OTA project with 

other projects, was the quantity 

and quality (scale) of objectives 

similar?

Would training have contributed 

to achieving project outcomes?

What were some key challenges

encountered?

OTA documents

Financial sector 

assessments (IMF

and World Bank)

AGD

BOJ

IMF

MOFPS

OTA

U.S. Embassy, 

Kingston, Jamaica

Document 

review

Results 

mapping

Key Informant 

Interview

2. Which of these 

outcomes have been 

sustained over time 

and why?

Did your institution (AGD, BOJ 

or MOFPS) provide feedback to 

OTA? In which form? When?

What is the legacy of OTA’s work 

in Jamaica? Which aspects of 

OTA’s TA continue to have a 

sustained impact? 

OTA documents

Financial sector 

assessments (IMF 

and World Bank)

AGD

BOJ

IMF

Document 

review

Results 

mapping

Key Informant 

Interview 
Jamaica Budget and Financial Accountability Project Evaluation Report

31

Would training have contributed 

to sustainability of project

outcomes?

What were some key challenges 

encountered?

MOFPS

OTA

U.S. Embassy, 

Kingston, Jamaica

3. Which of the 

BFA’s project

objectives were not 

achieved and why?

What were your expectations at 

the beginning of the project? Did 

they change over time?

How would you describe the 

technical assistance provided by 

OTA?

What were some key challenges 

encountered?

Would training have contributed 

to project objectives?

Which recommendations would 

you give to OTA considering 

future programming in Jamaica?

OTA documents

Financial sector 

assessments (IMF 

and World Bank)

AGD

BOJ

IMF

MOFPS

OTA

U.S. Embassy, 

Kingston, Jamaica

Document 

review

Results 

mapping

KII
Jamaica Budget and Financial Accountability Program Evaluation Report

32

Annex 2: Evaluation Findings Matrix 

• Evaluation Question 1: To what extent were each of the anticipated project outcomes achieved?

• Evaluation Question 2: Which of these outcomes have been sustained over time and why?

Table A-2: Evaluation Findings Matrix EQ 1 and EQ 2.

Objective Outcome Status

Objective 1: Improve 

cash management 

through creation of a 

bank account inventory 

with signatories and 

visibility to all accounts.

AGD has full visibility into GOJ bank 

accounts. The AGD effectively utilizes the 

bank accounts to track all accounts on an 

ongoing basis.

Achieved The government achieved inventory 

of all accounts. Sustained. 

AGD uses its procedures to close unnecessary 

accounts and consolidate government cash 

resources under its control.

Partially 

Achieved

The government rolled in Ministries, 

Departments, Agencies’ accounts 

into a TSA. Transfer procedures 

and regulations operationalized.

AGD produces and uses liquidity reports to 

actively manage government cash resources. 

The GOJ has improved understanding of its 

cash position and uses this information to 

inform borrowing and payment decisions.

Partially 

Achieved

Government cash resource reports 

are submitted on a regular basis. 

Procedures operationalized.

Objective 2: Review 

pension process and 

internal controls to 

ensure accurate 

GOJ uses process flow maps, concept papers,

and action plans to integrate and enhance 

system functionally to administer pension 

Partially 

achieved

Process flow maps, concept papers,

and action plans developed. Pension 

reforms are difficult to achieve, 

requiring significant buy-in from a 
Jamaica Budget and Financial Accountability Program Evaluation Report

33

Objective Outcome Status

distributions to 

pensioners.

benefits and payments efficiently and 

accurately. 

number of actors. Not all the 

functionalities were integrated into 

the pension system.

Enhanced pension administration reduces risks 

of ineligible payments and inaccurate benefit 

calculations.

Partially 

achieved

One output was partially achieved—

training for pension staff employees. 

Objective 3: Implement 

a daily reconciliation 

process for Consolidated 

Fund operations.

AGD Cash Management Unit uses automated 

reconciliations to timely, efficiently, and 

effectively manage cash. Automated 

reconciliations improve internal controls, 

reducing the opportunity and need for manual 

adjustments.

Partially 

achieved

Sustained. Procedures 

operationalized. Cash management 

reconciliations were automated for 

accounts that were in the TSA.
Jamaica Budget and Financial Accountability Program Evaluation Report

34

• Evaluation Question 3: Which of the BFA’s project objectives were not achieved and why? 

Table A-3: Evaluation Findings Matrix EQ 3—Project objectives not achieved.

Outcome Outcome Outcome Outcome

Objective 4: Assist with 

implementation of the 

Travel Card program.

AGD implements a Travel Card program to 

improve the efficiency of travel expenditures 

and improve internal controls over government 

travel, including through preauthorized limits 

and increased availability of supporting 

documents.

Not achieved Decision-making at GOJ on hold.

AGD implements reconciliation process that 

improves internal control and transparency of 

travel expenditures by ensuring that 

expenditures are authorized and comply with 

regulations.

Not achieved Travel Card program not 

implemented. Government 

requested postponement.

Objective 5: Enhance 

the capacity of the 

Government Accounting 

and Reporting Unit 

(GAR) through a review 

of job descriptions and 

standardization of 

information flow.

AGD uses updated job descriptions to 

effectively staff the unit and uses 

recommended protocols and operational 

capacity assessment to enhance GAR.

Partially 

achieved

Some personnel descriptions were 

generated. With ongoing change 

management processes, these were 

not routinely updated. 

New GAR unit consistently develops accurate 

and timely financial reports to respond to 

internal and external queries and facilitates

enhanced decision-making by management 

within GOJ ministries.

Not achieved The government is implementing 

new FMIS system processes with 

the World Bank. These systems 

were not in operation.
Jamaica Budget and Financial Accountability Program Evaluation Report

35

Annex 3: OTA Project Document List

OTA documents, by type:

Assessment Report - 1

Briefings and Memos - 20

Budgets - 5

End of Project Report -1

End of Tour Report - 1

Monitoring and Evaluation (Assessments, Logframes, Narratives, Project Performance, Q 

Reports) - 28

Trip Reports: (2016 - 3; 2017 - 15; 2018 - 13; 2019 - 15; 2020 - 12; 2021 - 13; 2022 - 1) - 72

Workplans - 5

End of Tour Report - 1

Total OTA documents - 132
Jamaica Budget and Financial Accountability Program Evaluation Report

36

Annex 4: Context Document List

Jamaica Public Expenditure and Financial Accountability (PEFA). International Monetary Fund 

(IMF) Caribbean Regional Technical Assistance Centre (CARTAC). February 28, 2017. 

URL: https://www.pefa.org/node/571 

Jamaica Strategic Public Sector Transformation (P146688). Implementation Status & Results 

Report. The World Bank. June 28, 2018. URL: 

https://documents1.worldbank.org/curated/en/162351607574528284/pdf/Disclosable-Version-ofthe-ISR-Jamaica-Strategic-Public-Sector-Transformation-P146688-Sequence-No-12.pdf

IMF (International Monetary Fund). 2010. “Jamaica Article IV Consultation.” IMF Country 

Report 10/267, IMF, Washington, DC. 

———. 2013a. “IMF and Jamaican Authorities Reach a Staff-Level Agreement on Key 

Elements of EFF-Supported Program.” IMF press release, February 15. 

———. 2013b. “Jamaica: First Review Under the Extended Arrangement Under the Extended 

Fund Facility and a Request for Modification of Performance Criteria.” IMF Country Report 

13/304, IMF, Washington, DC. 

———. 2013c. “Jamaica Request for an Extended Arrangement under the Extended Fund 

Facility.” IMF Country Report 13/126, IMF, Washington, DC. 

———. 2013d. “Jamaica Second Review under the Extended Arrangement under the Extended 

Fund Facility and Request for Modification of Performance Criterion and Modification of 

Performance Criteria.” IMF Country Report 13/378, IMF, Washington, DC. 

———. 2014a. “Jamaica Article IV Consultation and Fourth Review under the Arrangement 

under the Extended Fund Facility and Request for Modification of Performance Criteria—Staff 

Report; Press Release; and Statement by the Executive Director for Jamaica.” IMF Country 

Report 14/169, IMF, Washington, DC. 

———. 2014b. “Jamaica Fifth Review under the Extended Fund Facility and Request for 

Modification of Performance Criteria—Staff Report; Press Release.” IMF Country Report 

14/295, IMF, Washington, DC. 

———. 2014c. “Jamaica Sixth Review under the Extended Fund Facility and Request for 

Modification of Performance Criteria—Staff Report; Press Release; and Statement by the 

Executive Director for Jamaica.” IMF Country Report 14/359, IMF, Washington, DC. 

———. 2014d. “Jamaica Third Review under the Extended Arrangement under the Extended 

Fund Facility and Request for Modification of Performance Criteria—Staff Report; Press 

Release; and Statement by the Executive Director for Jamaica.” IMF Country Report 14/85, 

IMF, Washington, DC.
Jamaica Budget and Financial Accountability Program Evaluation Report

37

———. 2015a. “Jamaica Eighth Review under the Arrangement under the Extended Fund 

Facility and Request for Waiver for the Nonobservance of Performance Criterion and 

Modification of Performance Criteria—Press Release; and Staff Report.” IMF Country Report 

15/150, IMF, Washington, DC. 

———. 2015b. “Jamaica Ninth Review under the Arrangement under Extended Fund Facility 

and Request for Modification of Performance Criteria—Press Release; Staff Report; and 

Statement by the Executive Director for Jamaica.” IMF Country Report 14/270, IMF, 

Washington, DC. 

———. 2015c. “Jamaica Seventh Review under the Extended Fund Facility and Request for 

Modification of Performance Criteria—Staff Report; Press Release; and Statement by the 

Executive Director for Jamaica.” IMF Country Report 15/95, IMF, Washington, DC. 

———. 2015d. “Jamaica Tenth Review under the Arrangement under the Extended Fund 

Facility and Request for Modification of Performance Criteria—Press Release; Staff Report; and 

Statement by the Executive Director for Jamaica.” IMF Country Report 15/343, IMF, 

Washington, DC. 

———. 2016a. “Jamaica Article IV Consultation, Eleventh and Twelfth Reviews under the 

Extended Fund Facility and Request for Modification of Performance Criteria—Press Release; 

Staff Report; and Statement by the Executive Director for Jamaica.” IMF Country Report 

16/181, IMF, Washington, DC. 

———. 2016b. “Jamaica Request for Stand-By Arrangement and Cancellation of the Current 

Extended Arrangement under the Extended Fund Facility—Press Release and Staff Report.” IMF 

Country Report 16/350, IMF, Washington, DC. 

———. 2016c. “Jamaica Thirteenth Review under the Arrangement under the Extended Fund 

Facility – Press Release and Staff Report.” IMF Country Report 16/297, IMF, Washington, DC. 

———. 2018. “Jamaica Article IV Consultation.” IMF Country Report 18/103, IMF, 

Washington, DC. 

———. 2019. “Jamaica Fifth Review Under the Stand-By Arrangement Staff Report.” IMF 

Country Report 19/105, IMF, Washington, DC. 

———. 2022 "IMF Reaches Staff-Level Agreement with Jamaica on a Precautionary and 

Liquidity Line (PLL) and the Resilience and Sustainability Facility (RSF) and conducted the 

2022 Article IV Consultation," No 22/435. 

International Monetary Fund and the Government of Jamaica. 2016. Economic Program 

Oversight Committee: Extended Fund Facility 2013–2016. Kingston, Jamaica: Sagicor Group 

Jamaica Limited. 

World Bank. 2011. Jamaica Country Economic Memorandum: The Road to Sustained Growth. 

Washington, DC: World Bank.
Jamaica Budget and Financial Accountability Program Evaluation Report

38

Annex 5: Data Collection Instruments

Introduction (to be read to interview subject)

The evaluation of OTA’s Budget and Financial Accountability project in Jamaica will serve two 

purposes: 

The first purpose is to ensure compliance with the Foreign Aid Transparency and Accountability 

Act (FATAA) of 2016, which includes requirements to evaluate the outcomes and impacts of 

OTA’s projects whose dollar value equals or exceeds the median project size and apply the 

findings and conclusions of such evaluations to proposed project and project design. 

The second purpose is to provide OTA with in-depth, third-party, retrospective insights of its 

Budget and Financial Accountability project activities in Jamaica. The evaluation will focus on 

the key evaluation questions related to OTA’s contributions to Jamaica’s public financial 

management sector development. Some aspects of OTA’s experience in Jamaica may be 

extrapolated to other current or future projects in Jamaica or other countries to inform adaptation, 

improve projects, discover ways to reduce the time to achieve project outcomes, identify new 

ways to effect positive change, and document project achievements and success.

Informed Consent (to be read to interview subject)

We anticipate the discussion to take 60 to 90 minutes. We will be asking you a series of questions 

about the OTA project in Jamaica. Your participation in this discussion is voluntary and you are 

free to not respond to some questions or discontinue with the discussion at any point. We also 

request your consent to record the interview to ensure that our notes and transcripts are complete 

and accurate. At any time, you may request that you not be quoted, or your interview responses 

be off the record and not included in the interview notes or transcripts. The interview recordings, 

notes, and transcriptions will not be shared with OTA or others outside the evaluation team and 

will be deleted after the evaluation report is complete. 

If you have understood these points and consent to continue the discussion, can you please state, 

“Yes, I consent to be interviewed and recorded.” 
Jamaica Budget and Financial Accountability Program Evaluation Report

39

EVALUATION QUESTIONS 

(For reference use by interviewers during interview)

1. To what extent were each of the anticipated project outcomes achieved? 

Objective 1: Improve cash management through creation of a bank account inventory with

signatories and visibility to all accounts.

Outcomes

AGD has full visibility into GOJ bank accounts. The AGD effectively utilizes the bank accounts 

to track all accounts on an ongoing basis.

AGD uses its procedures to close unnecessary accounts and consolidate government cash 

resources under its control.

AGD produces and uses liquidity reports to actively manage government cash resources. The 

GOJ has improved understanding of its cash position and uses this information to inform 

borrowing and payment decisions.

Objective 2: Review pension process and internal controls to ensure accurate distributions 

to pensioners.

Outcomes

GOJ uses process flow maps, concept papers, and action plans to integrate and enhance system 

functionally to efficiently and accurately administer pension benefits and payments. 

Enhanced pension administration reduces risks of ineligible payments and inaccurate benefit 

calculations.

Objective 3: Implement a daily reconciliation process for Consolidated Fund operations.

Outcomes

AGD Cash Management Unit uses automated reconciliations to timely, efficiently, and 

effectively manage cash. Automated reconciliations improve internal controls, reducing the 

opportunity and need for manual adjustments.

Objective 4: Assist with implementation of the Travel Card program

Outcomes

AGD implements a Travel Card program to improve the efficiency of travel expenditures and 

improve internal controls over government travel, including through preauthorized limits and 

increased availability of supporting documents.
Jamaica Budget and Financial Accountability Program Evaluation Report

40

AGD implements reconciliation process that improves internal control and transparency of travel 

expenditures by ensuring that expenditures are authorized and comply with regulations.

Objective 5: Enhance the capacity of the Government Accounting and Reporting Unit 

(GAR) through a review of job descriptions and standardization of information flow.

Outcomes

AGD uses updated job descriptions to effectively staff the unit and uses recommended protocols 

and operational capacity assessment to enhance GAR.

New GAR unit consistently develops accurate and timely financial reports to respond to internal 

and external queries and facilitates enhanced decision-making by management within GOJ

ministries.

2. Which of these outcomes have been sustained over time and why? 

3. Which of the BFA’s project objectives were not achieved and why?

4. Did OTA allow sufficient time to achieve full implementation of project objectives?

INTERVIEW QUESTIONS

Planning:

1) How were the five objectives chosen and planned? 

2) In designing this project and its objectives, were assumptions made? Did the project

design consider risks?

3) What were your expectations at the beginning of the project? Did they change over time?

Communication and Feedback:
Jamaica Budget and Financial Accountability Program Evaluation Report

41

4) Did OTA request feedback from you? In which form? When?

5) Did you provide feedback to OTA? In which form? When?

Results:

6) How would you describe the technical assistance provided by OTA?

7) What were some of the successful outcomes from OTA’s work? (Evaluation Team, 

Please scroll up the list under evaluation question one and type notes under each 

objective/outcome.)

BELOW ARE SOME FOLLOW-UP QUESTIONS

8) Have you tracked progress with objective 1?

For example, which percentage of bank accounts are now “visible” to AGD? (Objective 1)

9) Have you tracked progress with objective 2?

For example, has the percentage of errors decreased?

10) Have you tracked progress with objective 3?

For example, to what extent does the Cash Management Unit use automated reconciliations?

11) Have you tracked progress with objective 4?

For example, does the government use a Travel Card program?

12) Have you tracked progress with objective 5?

For example, how does the GAR unit operate?
Jamaica Budget and Financial Accountability Program Evaluation Report

42

Training:

13) Did OTA provide training – formal, in-house, or other?

14) Would training have contributed to achieving project objectives?

Would training have contributed to project outcomes?

Would training have contributed to sustainability of project outcomes?

15) Did the project period allow sufficient time to achieve full implementation of the various 

project objectives? Which ones (that were not achieved) do you think could have been 

achieved with more time?

Other Issues:

16) Are there other issues that the evaluation team should consider?

Challenges Encountered:

17)What were some key challenges encountered? (counterpart capacities, resource 

limitations, leadership/management, political context, disruptive individuals, economic 

externalities, staff attrition, OTA/Washington support, unrealistic expectations, other OTA 

advisor issues...)

18) Comparing the OTA project with other projects, was the quantity and quality (scale) of 

objectives similar?

19) What is the legacy of OTA’s work in Jamaica? Which aspects of OTA’s TA continue to 

have a sustained impact? 
Jamaica Budget and Financial Accountability Program Evaluation Report

43

20)Which recommendations would you give to OTA considering future programming in 

Jamaica?

21) Do you have anything else that you would like the evaluation team to know?
Jamaica Budget and Financial Accountability Program Evaluation Report

44

Annex 6: List of Key Informant Interviews (KIIs) and 

Respondents

Between March 9 and April 26, 2023, the Bixal evaluation team conducted KIIs.

The Bixal evaluation team comprised the following persons: 

1. Joseph Kotun, Technical Lead, Monitoring and Evaluation

2. Michael Tatone, Project Manager

3. Geoffrey Mazullo, Team Lead

4. Michael Schaeffer, Subject Matter Expert, Public Financial Management (PFM)

5. Genta Agaj, Monitoring, Evaluation and Learning (MEL) Expert

6. Renee Johnson, MEL Expert

From March 19–25, 2023, three members of the evaluation team (Geoffrey Mazullo, Michael 

Schaeffer, and Renee Johnson) were in Kingston, Jamaica. 

A total of seven KIIs were conducted, with 16 respondents. The gender composition was eight 

men and eight women. 

Respondents represented the following institutions (in alphabetical order):

• Accountant General Department (four respondents)

• Bank of Jamaica (two respondents)

• IMF (three respondents)

• Ministry of Finance and Public Service (one respondent)

• OTA (three respondents)

• U.S. Embassy (three respondents)

Table A-4 presents the KIIs and respondents.

Table A-4: Key Informant Interview Participants.

KII ID Name

Redacted

Position Agency Date Format Gender

1 1 Former Advisor OTA March 9, 

2022

Online Male

2 2 Acting Associate 

Director and Senior 

Advisor

Budget and Financial 

Accountability

OTA March 

16, 2023

Online Female

3 Senior Advisor Female
Jamaica Budget and Financial Accountability Program Evaluation Report

45

KII ID Name

Redacted

Position Agency Date Format Gender

3 4 Political and 

Economic Counselor

U.S.

Embassy in 

Kingston, 

Jamaica

March 

20, 2023

In-person Female

5 Economics Officer Male

6 Political and 

Economic Specialist

Male

4 7 Accountant General Accountant 

General’s 

Department 

(AGD)

April 4, 

2023

Online Female

8 Deputy Accountant 

General

Female

9 Consultant Male

10 Director, Information 

Technology (IT)

Male

5 11 Director IMF 

Caribbean 

Regional 

Technical 

Assistance 

Centre 

(CARTAC)

April 6, 

2023

Online Male

12 Public Financial 

Management Resident 

Advisor

Male

13 Public Financial 

Management Resident 

Advisor

Female

6 14 Acting Principal 

Director, Accounting 

and Financial Policy 

Branch

Ministry of 

Finance & 

Public 

Service

April 12, 

2023

Online Male

7 15 Head, Banking 

Department

Bank of 

Jamaica

April 26, 

2023

Online Female

16 Assistant Head, 

Banking Department

Female
Jamaica Budget and Financial Accountability Program Evaluation Report

46

Annex 7: Assumptions and Risks Identified by Evaluation Team

Table A-5 presents the evaluation team’s review of OTA Jamaica assumptions and risks. 

Although OTA conducted a standard needs assessment at the start of the intervention, it did not 

identify any obvious risks in the planning phase of the project. OTA did identify and document 

risks to the project during implementation. The evaluation team, however, identified several 

distinct risk factors that should have been identified prior to the implementation of the project 

that impacted the OTA BFA project in Jamaica. These risks should have been documented and 

identified at the project planning/design stage, reviewed throughout implementation, and 

described at closeout (in the End of Tour Report). As noted above, this lack of risk management 

negatively impacted project outcomes and precipitated a significant disconnect between M&E 

and implementation.

Completion of the risk assessment requires answers to fundamental questions. What is OTA’s 

risk appetite? Which are the identified relevant risks? How should the identified risks be 

assessed, scored, and appropriately mitigated? The answers to these questions represent major 

milestones in the development of PFM projects.

Table A-5: Jamaica OTA Assumptions and Risks.

Assumptions 

and Risks:

Design Phase4

Assumptions and Risks 

Envisioned During 

Implementation5

Possibility Impact Assumptions and Risks 

that Materialized in the 

Closeout Phase6

Impact

During the 

design/

planning 

phase, risks 

were not 

Counterparts are 

committed to reform in 

the areas in which they 

have requested 

assistance and in which 

OTA is engaged.

Medium High Counterparts’ 

commitment to reform 

limited. Travel Card

program lacked 

sufficient government 

commitment.

Medium

4 Source: 2017 Jamaica Budget 01 Narrative states, “There are no obvious risks to this project. While Jamaica had undergone a 

couple of recent elections the staff of the MOF and AGD has remained mostly unchanged and the commitment to PFM reform 

and the IMF program has continued. In a project like this one, the most significant risks is lack of funding or resources to make 

improvements to information technology systems. This risk is mostly mitigated by the fact that the upgrades to the new systems 

seem to be sufficiently funded by donors like the IDB. While there is no guarantee that delays might occur having outside donors 

fund new systems implies at least a solid funding source even if the implementation of the systems is not done smoothly. 

Coordination with other donors will be an ongoing issue but we plan to mitigate this by having the advisor participate in project 

coordination meetings within the MOF and donor coordination meetings outside the MOF.” The evaluation team noted a 

disconnect between the first sentence and the following three sentences in the paragraph. 

5 Source: Jamaica Logframe Final, 2021.

6 Source: KIIs and document review.
Jamaica Budget and Financial Accountability Program Evaluation Report

47

Assumptions 

and Risks:

Design Phase4

Assumptions and Risks 

Envisioned During 

Implementation5

Possibility Impact Assumptions and Risks 

that Materialized in the 

Closeout Phase6

Impact

adequately 

addressed. Governance 

environment in the 

country conducive to 

development and 

reform (also includes 

key nongovernmental 

stakeholders).

Low High Coordination of 

multiple government 

stakeholders 

challenging. 

Medium

Project is consistent 

with United States 

Government (USG) and 

Treasury policies and 

applicable laws.

Medium High Project consistent with 

USG and Treasury 

policies and applicable 

laws

No 

impact

Advisor with the 

necessary skillset will 

be available when 

required.

Low High Advisor was available. 

Advisor tasked with a 

diverse number of 

objectives. 

Medium

Counterparts available 

to engage with OTA 

advisors with 

frequency.

Low High Counterparts slow to 

provide support to 

advisor. 

High

Minimum level of 

embassy support 

through NSDD-38 

approval or country 

clearances.

Low High Embassy support as 

well as OTA support to 

the advisor available.

Low

Travel to and within 

counterpart countries 

can occur as needed for 

project implementation.

Low High Due to COVID-19, 

travel as well as inperson activities were 

limited.

Medium

Sufficient funds will be 

available when they are 

required.

Low High Funds sufficient Low
Jamaica Budget and Financial Accountability Program Evaluation Report

48

Assumptions 

and Risks:

Design Phase4

Assumptions and Risks 

Envisioned During 

Implementation5

Possibility Impact Assumptions and Risks 

that Materialized in the 

Closeout Phase6

Impact

Banking system 

capable to support 

electronic payments

Medium Medium Banking system 

capability as well as 

legal framework 

needed to be updated to 

implement the Travel 

Card program.

High

Annex 8: PEFA Assessment Indicators and Scores

The five objectives that guided the project were aligned to problems identified by the 2017 

Public Expenditure Financial Accountability (PEFA) Assessment (Table A-6), a useful tool that 

was not yet available during OTA’s assessment stage. The 2012 PEFA was reviewed and 

referenced during the planning process in both the assessment report and the budget proposal for 

the project. 

However, OTA does not reference PEFA context indicators. OTA objectives 1, 3, and 5 tie more 

closely to PEFA indicators PI 19, 20, 21, and 22, and the poor scores associated with this 

independent and credible review. The objectives of OTA appear to have been closely aligned to 

resolving issues identified by PEFA (2017) and government priorities, which indicates that OTA 

missed an opportunity to take advantage of an internationally recognized assessment tool to 

indicate gaps and risks in the Jamaican context at the outset of its intervention.

Table A-6: Relevant PEFA indicators and scores/comments.

PEFA Indicator Score/Comment (2017)

PI-19 Revenue 

administration

Score “D” Cash forecasting and progress towards treasury single account

PI-20 Accounting 

for revenue

Score “D” This indicator provides an analysis for accounting for collected 

revenue; and assesses the procedures for recording and reporting revenue 

collections, consolidating revenues collected, and reconciling the tax revenue 

accounts.

PI-21 accounting 

for collected 

revenue

Score “D+” This indicator assesses the procedures for recording and reporting 

revenue collections, consolidating revenues collected, and reconciling the tax 

revenue accounts. It covers both tax revenues and non-tax revenues collected by 

the central government. 
Jamaica Budget and Financial Accountability Program Evaluation Report

49

PI-21 

Predictability of 

in-year Resource 

Allocation

Score “C” This indicator assesses the extent to which the ministry of 

finance can forecast commitment, cash requirements, and provide reliable 

information on the availability of funds to budgetary units for service 

delivery.

PI-22 provides 

stocktaking of all 

government 

accounts

Score “D” provides that stocktaking of all government accounts has been 

launched, initially covering accounts with BOJ. The intention is to close all 

dormant and non-essential accounts, but this is a slow process as it requires 

collection of information for each account on its purpose as well as 

reconciliation of its balance.
Jamaica Budget and Financial Accountability Program Evaluation Report

50

Annex 9: Evaluation Team Qualifications

Team Leader Geoffrey Mazullo has 28 years of technical experience as an evaluator, corporate 

analyst, subject matter expert, and trainer. His expertise includes corporate governance, socially 

responsible investment, and environmental, social, and governance reporting in emerging market 

environments. He has extensive experience as an evaluation Team Leader, a program Chief of 

Party, technical expert, and expert trainer in MEL, governance, and other subjects for United 

States Agency for International Development (USAID), the U.S. Department of State, and 

numerous other government, private sector, and nongovernmental organization (NGO) clients. 

He was the Team Leader for the evaluation of OTA’s Kosovo program. He has an M.Sc. in 

Political Science from the London School of Economics and Political Science, a graduate 

certificate from Jagiellonian University in Poland, and a B.Sc. in Foreign Service from 

Georgetown University. He has lived and worked extensively throughout Eastern Europe, is a 

native English speaker, and is proficient in Serbo-Croatian, German, French, Polish, and 

Russian.

Michael Schaeffer is a project-oriented public financial and investment management specialist 

with expertise gained from assignments in the United States, Central and Eastern Europe, Middle 

East and North Africa, Africa, Latin America, and Southeast and East Asia. Mr. Schaeffer’s 

areas of specialization include public financial management (budgeting, treasury, financial 

management systems, public investment management, etc.) state and local public finances in 

emerging economies, municipal project development (project financing, specifically limited 

recourse financing); municipal credit market development, privatization of municipal and 

corporate assets, national and subnational budgeting and public financial management, and 

private sector participation in service delivery sectors (e.g., water, sanitation, electric power, 

solid waste management). Mr. Schaeffer has worked with the World Bank and IMF in various 

capacities.

Genta Agaj is an M&E Specialist, working across several sectors including governance, energy, 

education, and community development. Currently, she serves as an M&E Specialist for the 

Millennium Challenge Corporation (MCC) with the Kosovo Threshold Program. Her expertise 

includes impact evaluation, M&E design and planning, quantitative and qualitative methods and 

analysis, survey design, advising on performance indicators, and data collection strategies for 

USAID, contractors, and NGOs. She has a Bachelor of Applied Science degree in Economics 

from the Rochester Institute of Technology, and a master’s degree in applied economics from the 

University of Maryland. She is a native Albanian speaker and is fluent in English.

Renee Johnson is an independent consultant with over 15 years’ experience in fields related to 

public health, evaluation, youth, and development. Ms. Johnson earned her master’s degree in 

public health from the University of Birmingham in the United Kingdom (2004) and holds a 

certificate in Project Management from the Mona School of Business, University of the West 

Indies (2009). Ms. Johnson recently completed a certificate course in Economic Evaluation 
Jamaica Budget and Financial Accountability Program Evaluation Report

51

through the University of York and the London School of Tropical Health (2022). Over the last 

10 years, she has led and been part of teams that conduct evaluations, surveys, baseline analysis 

and other forms of assessments. Most of Ms. Johnson’s work has been qualitative, with specific 

technical expertise in health, HIV, and youth. She has contributed significantly to the evaluation 

and development of strategic plans and policies over the years, working with a number of 

multisectoral teams. She has worked in several Caribbean countries including Jamaica, 

Montserrat, St. Kitts, and Barbados.

Michael Tatone is an International Evaluation Project Manager at Bixal. He has 13 years of 

technical experience in evaluation and research, managing and providing technical support for 

evaluations and impact studies in conflict and violence in Colombia, Brazil, and Chicago. 

Previous roles have included Research Manager at the University of Chicago and various Project 

Manager and Researcher roles in academia and NGOs. He has a master’s degree in political 

science from Universidad de Los Andes in Colombia, and a bachelor’s degree in political science 

from University of Wisconsin-Eau Claire. He is a native English speaker and is fluent in 

Spanish and Portuguese.

Joseph Kotun is a Director of MEL at Bixal. He has more than 25 years of technical experience 

in international development, focused on evaluation. Before his role at Bixal, he worked 

extensively as an evaluation team leader for contractors with USAID, MCC, U.S. Department of 

Agriculture, U.S. Department of State, World Bank, Department for International Development, 

Asian Development Bank, and other private sector and NGO clients. His expertise includes 

evaluation design; mixed, qualitative, and quantitative methods; program evaluations; impact 

evaluations; and systems-based evaluations as well as MEL frameworks, strategies, and learning 

agenda development. His evaluation work includes financial inclusion and microfinance subject 

areas, as well as a broad range of other international development sectors. He has a master’s 

degree in international development from the School for International Training Graduate 

Institute in Vermont and a bachelor’s degree in science from McGill University. He is a native 

English speaker and proficient in French.