# Presentation: 736285

## Monitoring Grants and Cooperative Agreements (PY222)

- Office of the Procurement Executive (A/OPE)
- Mr. Corey M. Rindner

## A/OPE/FA Staff

- Georgia K. Hubert, Director, _[hubertgk@state.gov](mailto:hubertgk@state.gov)_, 703-812-2526
- Kimberly S. Butler, Grants Policy Analyst,
-    _[butlerKS2@state.gov](mailto:butlerKS2@state.gov)_,  703-516-1754
- Jennifer K Ceriale , Grants Policy Analyst, _[cerialejk@state.gov](mailto:cerialejk@state.gov)_,  703-516-1686

## A/OPE/FA Staff

- Phyllis A. Swann, Grants Policy Analyst, _[swannpa@state.gov](mailto:swannpa@state.gov)_, 703-516-1685
- Gregory M. Young, Grants Policy Analyst, _[younggm@state.gov](mailto:younggm@state.gov)_,  703-516-1988

## Goals and Objectives

- Develop a monitoring plan.
- Evaluate hypothetical grant scenarios to determine whether prior approval is required.
- Identify weaknesses in recipient internal controls through a mock-site visit.
- Determine appropriate special conditions for high-risk recipients.
- Analyze scenarios illustrating recipient waste, fraud and abuse in federal programs and determine appropriate measures to pursue.
- Proactive negotiating resolution of problems that arise during grant performance.

## Module 1

- Introduction and Course Overview

## What is effective grants management?

-    Effective grants management is a      process (or result) of adequate overall oversight and monitoring of federal assistance awards that includes project resources, activities, and results.

## What are the purposes of effective grants management?

- Accountability
- Program goals accomplished
- Agency performance goals

## When does it begin?

- 	Effective grants management begins before the award is signed!

## Which type are you?

## Specific Purposes

- Confirm compliance of grant terms and conditions
- Determine the extent the proposal matches activities
- Detect early warning signs of trouble and non-compliance

## Basic Process of Monitoring

- Collect and review data
- Analyze
- Take action

## Formal or Informal

- Monitoring can be formal or informal
- Factors that determine monitoring level
- Experience – grantee and grants officer
- Dollar amount
- Program visibility

## Government Wide Concerns in Grant Management	

- Lack of monitoring
- No site visits
- Lack of progress reports
- Reports not reviewed
- Lack of expertise
- Understaffed

## Government Wide Concerns in Grant Management Cont.

- Questionable costs
- Lack of proper procurement
- Prospective grantees not being treated fairly
- Results not being measured
- Lack of closeout

## Module 2

- Objectives and Performance Measures

## Performance Measures

- Process of regularly measuring the outputs and outcomes produced by the program
- Inputs – dedicated resources for outputs
- Outputs – activities
- Outcomes – results or changes that occur

## Performance Measures

- An attempt to assess the impact and value of programs
- To demonstrate program accountability
- To address the question of “What are programs trying to accomplish?”

## Techniques

- Evaluations
- Data collection
- Analysis of data
- Questionnaires
- Reports

## Why Conduct Performance Measurement?

- Spends limited resources on most effective programs
- Clarifies program purpose
- Documents program results
- Improves program performance
- Identifies successes
- Identifies areas for improvements

## Government Performance and Results Act (GPRA)

- What does it do?
- Requires agency strategic plans, performance plans, program performance reports

- What is the process?
- Identify outcomes
- Identify programs to achieve outcome
- Assess program performance

## Program Assessment Rating Tool (PART)

- Formal system to rate effectiveness of federal programs
- Series of questions designed to provide consistent approach to rate all programs on performance

## Purpose of PART

-    PART presents an opportunity to inform and improve agency GPRA plans and reports, and establish a meaningful systematic link between GPRA and the budget process.

## Definition of Results

- Significant, intended, measurable change in the condition or change in recipient country or institutions
- Results oriented assistance instrument – achieves results that contribute to the performance goals

## Planning Results Oriented Assistance

- Develop a vision
- MPP
- BPP
- Develop measurement system
- Determine how to achieve specific programs

## Consideration

- What lends itself to being measured?
- What are are the resources needed to achieve program results?
- Political, economic, geographic factors

## Specific Assistance Programs

- Define the problem
- Results oriented program description – express as tangible, measurable objective against which actual achievement can be compared
- Establish system for monitoring and evaluating performance and reporting

## Howling Wolves and Dead Horses

- How to Survive and Thrive in a Culture of Measurement

- EAP PAO Conference

## Tribal Wisdom Vs. Government Approach

- Buying a stronger whip.
- Asserting “This is the way we always have ridden this horse.”
- Arranging to visit other sites to benchmark how they ride dead horses.
- Providing additional training to increase riding ability.
- Outsourcing to  private contractors to see if they can ride the dead horse cheaper.
- Harnessing several dead horses together to increase the speed.
- Declaring that, as the dead horse does not have to be fed, it is less costly, carries lower overheads, and therefore contributes substantially more to the bottom line of the economy than do some other horses.
- Commissioning a study to identify ways to improve dead horses through incremental enhancements, such as adding wheels.
- Rewriting the expected performance requirements for all horses.
- Promoting the dead horse to a supervisory position.

**Notes:**
How do we evaluate as a function or as an organization?

The tribal wisdom of the Dakota Indians, passed on from generation to generation, says that when you discover that you are riding a dead horse, the best strategy is to dismount. 

In government however, a whole range of far more advanced strategies is often employed, such as 

Do we have any dead horses in PD?

## PD Dead Horses 

- The results happen in the long-term so you can’t measure.
- You can’t measure outcomes of public diplomacy.
- This is just another management fad that will die off.
- Our programs are about unique experiences and are too diverse to quantify.
- Our alumni don't have time to answer surveys, they're too busy.
- An evaluation won’t tell us anything we don’t already know.
- Mutual Understanding can’t be quantified.  It’s too soft.
- Measurement is something posts don’t have to worry about—it’ll be handled by [Washington, R, that Kniker guy, etc.]

**Notes:**
If you or someone you know is saying any of these things, chances are:  They are riding a dead horse.

## What To Take Away

**All programs can be measured in some way**
**Outcomes and results mean “change”**
**Build upon the successful measurement you are already doing**
**Plan evaluation when you are planning programs**
**Measure in your sphere of influence**
**Quantitative data (numbers) about results is important to stakeholders**
**There are no perfect measures or evaluations**
**Value from measures comes from how they are used, not in the measure itself**

**Notes:**
10 minutes of undivided attention   

Measure – Good evaluation answers 3 questions

We often measure outputs – the tangibles of what we do, 
	outputs link our resources to our outcomes

Planning – what gets measured gets done, compass and map
	Plan Do Check Act

Don’t have to measure world peace – knowing who you are trying to reach is critical, especially in activities like PD

Numbers – much of what we do is best captured by anecdotes, 
	plural of anecdote is not data, 
Mantra in God we trust

## Why Measure and Evaluate?

**To justify the existence (or budget) of the program and how it contributes to the organization’s goals and objectives**
**To understand how the program is working**
**To make real improvements to the program**
**To drive program and policy outcomes**
**To fulfill legislative requirements**
**To keep federally funded programs accountable to the American taxpayer**

**Notes:**
Good evaluation answers 3 questions

You can’t manage it, if you can’t measure it
  -- Provides specific information
  -- Provides balanced information (not just the vocal minority)
Measurement helps pinpoint areas for process improvement, program innovation and people involvement.

-- Justification – Funding is tied to how well we can demonstrate results, and the only way we can demonstrate results consistently and effectively is through measurement and evaluation

-- Evaluation shows relationships

-- Just the implementation of an evaluation can make changes

## The Government Performance and Results Act of 1993

The President’s Management Agenda

The Office of Management and Budget (OMB)’s Program Assessment Rating Tool (PART) 

- The Government Performance and Results Act of 1993
- The President’s Management Agenda
- The Office of Management and Budget (OMB)’s Program Assessment Rating Tool (PART)

- Government Context

**Notes:**
Five Government-Wide Initiatives
Strategic Management of Human Capital
Competitive Sourcing
Improved Financial Management
Expanded Electronic Government
Budget and Performance Integration
Score Card Approach
Red, Yellow, Getting to Green------------------------------------------------------
A set of questions that evaluates program performance in four critical areas:
Program Purpose and Design
Strategic Planning
Program Management
Program Results and Accountability
A tool to assess performance using evidence.
Provides a consistent approach to evaluating programs across the Federal government.
Evaluation every 5 years
PART results inform the budget process, but do not by themselves determine budget decisions
-----------------------
The gist of all of these initiatives is that we must quantify what is inherently qualitative and demonstrate in a simple way that exchange programs have results.

## Is This Familiar?

**Notes:**
Sometimes you may feel that you are howling at the moon

## PD Evaluation

- Work in progress
- Coordinated by R/PPR through the Public Diplomacy Evaluation Council
- Long-term strategy
- Reduce the number of indicators
- Articulate more clearly defined goals
- Establish evaluation project schedule
- Revise measures
- Build evaluation capacity (guidance, training)

## Definitions

- Output
- A measure of the number of units produced, goods and services provided, or people served by an agency or program.
- Examples include:
- Number of youth exchange participants or visitors to American Corners
- Number of post-written op-eds placed
- Number of unique website users, page views

**Notes:**
Before we go further, I think it might be useful for us to take a minute and make sure that we are sharing the same understanding of some terms.

Simply – outputs are the number of people reached or the number of activities completed.

Outputs are often where most “measurement” stops, primarily because it is the usually the easiest thing to measure with the least amount of effort or resources.  Outputs are important because they give a sense of scope to whatever the program or project is doing, but in and of themselves they are not results.  Outputs 

Outputs help us link inputs or resources to our outcomes.  We can’t always budget for outcomes, but we can budget for outputs.

We need to go beyond measuring only outputs.

## Definitions continued

- Outcome
- A measure of the extent to which a service or program has achieved its goals and objectives or desired impact on its clients or society.
- Examples include:
- Change in knowledge or skills (e.g. better understanding of the U.S., more favorable impression of the U.S.)
- Change in behavior or condition of the participant
- Change in an institution, community or society (new legislation, changed curricula, more favorable editorials, etc.)

**Notes:**
Outcomes, or results, mean change.  Ask yourself, what must change in the status quo to create the conditions necessary for goal attainment?  Who are the targets of the change and what must they do?

If an IV goes to the United States and speaks highly of her program, this is not really an outcome.  If she draws on her U.S. experience to establish a small business incubator, and businesses using her services create wealth, boost employment, and provide needed services at a rate of 50% above the national average, that is a result.

The formal evaluations administered by ECA and soon to be by R/PPR try to measure these kinds of things.

What is the bottomline of your program?
How will you know you have been successful that you can stop the activity?
If you had to defend your programs’ value/benefit before a grand jury, what 2-3 pieces of evidence would prove the program was success rather than a failure?
What is the end benefit to the taxpayer?

## Levels of Outcomes

- What were the results?

- Did they use it?

- Did they learn?

- Did they like it?

**Notes:**
As I mentioned, we use four levels of outcomes.  Some of you may recognize this as the four-level scale developed by Donald Kirkpatrick for training results.  
REACTION -- ACSI
LEARNING
--This is where most measurement stops.
BEHAVIOR
BUSINESS RESULTS
There are several reasons we like this conceptual framework.  One, it is used internationally for learning and training programs, which better fits ECA’s business lines than health or research indicators which are well developed.  It is currently being used by organizations like Cisco Systems, Microsoft, Eaton, the Department of Commerce

Second, it provides easy to use timeframes.  For example, we can demonstrate immediate outcomes at the first and second levels.  We’ve also found that we can demonstrate results at the third level within a year after the exchange experience – so we don’t have to wait 15-20  years before we see results.
Finally, the levels have helped us formulate performance indicators.

## Definitions continued

- Impact
- The cumulative or net effect of the outcomes or results.
- Examples include:
- Better relations between the United States and other countries
- Increased level of education of youth in developing countries

## How to Achieve Your Dreams Through Evaluation

- Send
- $19.95
- To
- Ted Kniker
- U.S. Department of State
- 301 4th Street, SW
- Washington, DC 20547
- www.teds_early_retirement.com

**For this **_**incredible**_** information and *****more***

**Create **
**Winning**
**MPPs!**

**Notes:**
A man in a hot air balloon realized he was lost. He reduced altitude and spotted a woman below. He descended a bit more and shouted, "Excuse me, can you help me? I promised a friend I would meet him an hour ago, but I don't know where I am."

The woman below replied, "You are in a hot air balloon hovering approximately 30 feet above the ground. You are between 40 and 41 degrees north latitude and between 59 and 60 degrees west longitude."

"You must be an evaluator," said the balloonist.  "I am," replied the woman, "How did you know?"

"Well," answered the balloonist, "everything you told me is technically correct, but I have no idea what to make of your information, and the fact is I am still lost. Frankly, you've not been much help so far."

The woman below responded, "You must be a manager.“  "I am," replied the balloonist, "but how did you know?"

"Well," said the woman, "you don't know where you are or where you are going. You have risen to where you are due to a large quantity of hot air. You made a promise which you have no idea how to keep, and you expect me to solve your problem. The fact is you are in exactly the same position you were in before we met, but now, somehow, it's my fault."

## Use A Logic Model

- Inputs

- Activities

- Outputs

- Outcomes

- Impact

- Your planned work

- Your intended results

- Funding
- Staff
- Equipment

- Websites
- Outreach
- Media placements
- Exchanges
- Partnership development

- # of web readers
- # of articles
- # of audience members
- # of media placements
- # of exchange participants
- # of institutions reached

- Target audience more aware
- Target audience with increased knowledge
- Media runs more balanced or positive stories about U.S.
- Local institution develops American Studies program

- Better relations
- International support
- Cooperation

**Notes:**
Input – Certain resources are needed to operate your program

Activities – IF you have access to the resources, THEN you can use them to accomplish your planned activities

Outputs – IF you accomplish your planned activities, THEN you will hopefully deliver the amount of program, product, service that you intended

Outcomes – IF you accomplish your planned activities to the extent intended, THEN your target audience will benefit/change in certain ways

Impact – IF these benefits to participants are achieved, THEN certain changes in organizations, communities or systems might be expected to occur.

## Performance Metrics

- Inputs

- Activities

- Outputs

- Outcomes

- Impact

- Your planned work

- Your intended results

- Can you measure and track

- Theme
- or
- Topic

- Target
- Audience

- Time Frame

**Notes:**
SMART measures – specific, measurable, Accountable, Results oriented, time-bound
Funding by activity, theme/topic – can you answer how much goes to what?
Target audience – can you answer how much funding and which activities are targeted to what audiences?
Can you measure and track:  unique web users, circulation, number of op-eds, visitors to American Corners, US speaker audiences, 
Timeframe – when do you expect these things to happen

Logical progression Examples
Inputs – FTE time devoted to an activity, Funding devoted to an activity
Activity – number of speakers, number of outreach activities, number of HIV articles on embassy website
Efficiency – cost per delivery of activity; time cycle to deliver
Output – Size of audiences reached by post, number of media placements on HIV/AIDS
Efficiency – unit cost per output
Outcome – change in awareness, knowledge, behavior of recipients, adoption of practices by people or organizations, establishment of organization, partnership
Input – reduction of HIV/AIDS

## Aligning Outputs to Outcome Goals and Measures

- _Outputs_

- Intermediate _Outcomes_

- End _Outcome_

- _Goals_

- 35,000
- Exchange Visitors

- Participant satisfaction
- Increased aptitude
- Increased understanding
- Increased trust
- Increased leadership
- Concrete actions
- Increased explanation
- Increased collaboration

- More accurate and balanced understanding of U.S. or host country
- Sustainable partnerships between U.S. and host country institutions and people
- Individuals or institutions effect positive change in society

- International exchanges increase mutual understanding and build trust between Americans and people and institutions around the world

**Notes:**
ECA example

## What to Evaluate/Report

**Individual Results**
**Change in**
**Knowledge**
**Understanding**
**Attitudes, Perceptions**
**Behavior**
**Condition, Status, Direction**

**Institutional Results**
**Change in**
**Procedures**
**Programs**
**Policies**
**Partnerships**

- Recommendations

## We Have To DoWHAT?!

- We Have To DoWHAT?!

**Notes:**
Based on previous experience of giving these kinds of presentations, at least 60 percent of you are probably thinking:

But I want to assure you that I and those I work with do understand the burdens that you are under.  We understand evaluation of public diplomacy is not easy. We are on your side and I promise that in our thinking and development about our future evaluation strategies that we make it a top priority to not add more work to your load.  

We also want to make you aware that you are not alone in this and there are several resources available to you. 

## Post Measurement Assistance

- R/PPR – Evaluation
- Public Diplomacy Evaluation Council
- RESULTS Database
- E-GOALS Survey Tool (ECA)
- Professionals in Residence
- Local Social Science Research Organizations

## Last Thoughts

- For qualitative measures, choose process steps
- Limit to critical 2 or 3 measures
- Can mix activity, output and outcome measures
- Resource increases are not performance measures

## Linking Goals and Performance Measurement

**Performance Indicator**

- % participants who improve their understanding of the U.S.

**Initiative/Program/PART**

- Exchanges in NEA and SA

**Performance Goal**

- Exchanges increase mutual understanding and build trust

**Strategic Goal**

- Increase understanding for American values, policies and initiatives to create a receptive int’l environment

**Strategic Objective**

- Promote international understanding

**DoS Mission**

- More secure, democratic, and prosperous world for the benefit of the American people and the international community

**JPP**
**BPP**
**PART**
**MPP**

**Notes:**
This slide represents how we tie it all together from a metric we use all the way to the Dept. of State mission.

Responses to 10 standardized questions in evaluation are aggregated and used as data for performance measurement
Building an online performance measurement system that will cover the standardized areas
It’s not impossible, therefore, to go from some quantifiable measure to showing our contribution.  And it is only that we demonstrate evidence, not proof.
One of the common mistakes that people make is to think they have to measure the end outcome – now it might make sense if you were able to have a more tangible and definable goal like reducing the mortality rate in Uganda by 10% over the next five years.  
The Department is committed to utilizing the funds it receives from taxpayers through the Congress to produce successful results. To assess performance, the Department employs a performance management methodology depicted in the pyramid below. Each of the six components of the pyramid is defined below: 
Strategic Objectives High level, broad categories of action through which the Department will achieve its strategies and performance goals. 
Strategic Goals The Department's long-term goals as detailed in the Strategic Plan. 
Performance Goals The desired outcomes the Department is planning to achieve in order to attain its strategic goals. The Department has thirty-eight performance goals. 
Initiatives/Programs Specific functional and/or policy areas, including programs as defined by the OMB Program Assessment Rating Tool (PART), to which the Department of State devotes significant attention. 
Performance Indicators Values or characteristics that the Department utilizes to measure progress achieved towards stated annual performance goals. The indicators are drawn from bureau and mission performance plans. 
Performance Targets Expressions of desired performance levels or specific desired results targeted for a given fiscal year. Achievement of targets defines success. Where possible, targets are expressed in quantifiable terms. The FY 2003 Performance and Accountability Report reports on how well the Department achieved its targets.

## Module 3

- Monitoring

## What is monitoring?

-    Ongoing assessment of the progress of the activities being funded, with the objective of determining if the terms and conditions of the award are being met and the goals are being met.

## What purpose does monitoring serve?

- Provides oversight as required by agency – and covers you!
- What are they doing?
- What are they planning to do?
- What did they say they would do?

## Techniques

- Pre-award risk assessment
- Site visits
- Telephone calls
- Enforcement of prior approvals

- Analysis of reports
- Provide technical assistance
- Review reports (program, financial, audit)

## Risk Assessment

**Types of Risk**
- Recipient stability
- Geographic problems
- Amount of award
- Susceptible to fraud

**Types of Risk**
- New program (HIV/AIDS)
- Public or congressional scrutiny
- Lack of discretion in making award

## Documentation of Monitoring

- Trip reports
- Memos to the file
- Detail action taken
- Recommended actions

## Module 4

- Developing a Monitoring Plan

## Designing a Monitoring Plan

- A monitoring plan should be –
- Part of an overall comprehensive approach
- To recipients to be monitored

## Considerations

- Are grantees grouped by types of programs?
- Are grantees grouped by geographic location?
- Do the grantees have similar characteristics?

## Considerations (Cont’d)

- Are the grantees interchangeable or unique in capabilities?
- How do you choose grantees?  Is there an adequate available pool?
- What is the impetus for the programs to be funded?
- Apply consistent approach to recipients
- Customize accordingly

## Elements of a Monitoring Plan

- Techniques to be used
- Inherent risks with the grantee or the program
- Resources available for monitoring

## Monitoring Techniques

- Site visits
- Telephone calls
- Reports
- Additional prior approvals
- Review of audit reports
- A-133 or OIG

## Evaluate Inherent Risk

- Risk can be program or financial
- What is the dollar amount of the award?
- Is the program susceptible to abuse?
- Is the grantee susceptible to abuse?
- Long-term grantees may be in charge of the program – too much authority and control
- What potential pitfalls are out there?
- What about geographic proximity of the grantee?
- Prior problems with grantee?

## Determine Agency Resources

- What is necessary and desirable as opposed to what is practical?
- Corporate culture
- What is the level of importance given to monitoring of programs?
- Available staff
- Travel funds

## Designing the Monitoring Plan

- Resources
- Methods

## Module 5

- Budgets and Costs

## Definition of Budget

-    A detailed plan of income and expenses expected over a certain period of time.

## Definition of Costs

-    The outlay made to achieve or acquire an objective.

## Total Costs

-   Composition of all direct and indirect   costs.

## Direct Costs

- Costs identified with a particular award
- Direct salaries
- Travel
- Equipment
- Supplies

## Cost Requirements

- Reasonable – the “prudent person” test.
- Are the costs necessary for performance of the award?
- Do the costs reflect sound business practices – leasing vs. purchase?
- Market prices – what is the going rate on the economy?

## Cost Requirements

- Allocable
- Are the costs incurred specifically for the award?
- Costs cannot be shifted from another award
- Costs must be in the budget

## Cost Requirements

- Allowable
- Are the costs prohibited by statute,  regulation, or decision – OMB, GAO?
- In accordance with the Fly America Act?
- Within the authorizing legislation?

## Indirect Costs

- Administrative costs (running the organization that cannot be specifically identified for a program)
- Facility costs (rent, utilities, insurance, taxes)
- Legal
- General/administrative salaries (accounting, administrative, executives, etc.)

## Indirect Cost Rates - Domestic

- GPD 22 - Must be approved by cognizant agency
- For State – A/LM/AQM/IP

## Indirect Cost Rates - Overseas

- Allowable cost – A/OPE suggests 10% of the budget for indirect costs
- Make sure indirect costs are not also listed as direct items
- Ask recipient for breakout of the indirect charge

## Indirect Cost Pool

- Definition
- Total administrative costs to be allocated to projects by the use of an indirect cost rate
- Direct cost base
- All costs incurred in the direct performance of a project
- Salaries and wages
- Benefits
- Total direct costs

## OMB Circular A-122

- Cost Principles for Non-Profit                 Organizations
- Establishes uniform and consistent principles for determining costs
- Rules for indirect cost rates
- Direct costs – 56 items listed

## Budget Analysis

- Are costs reasonable, allocable and allowable?
- Do not accept summary budgets
- Review budget detail – how many employees, hours, salary rates
- Are items to purchase really necessary for the award?
- Are you helping build a business or just conducting a project?

## Budget 

- Negotiate items
- Are costs “wish list” items?
- Ask questions
- Ask for more detail

## Budgeted Costs 

- Review with a skeptical eye...

## Module 6

- Cost Sharing

## Cost Sharing

- Definition
- Recipient provides a share of non-federal cash or “in-kind” services in support of the federal award.
- OMB A-110 – the portion of project or program costs not borne by the federal government.

## Cost Sharing Methods

- Cash – from individuals or other organizations
- In-kind – non-cash such as donated time and effort
- Program income – generated by program activities

## Why Require Cost Sharing?

- Buy-in from the recipient
- Raise responsibility level of recipient
- Adds to appearance of active monitoring and oversight

## Cost Sharing

- Must be documented and verifiable from records
- Other federal funds cannot be used for cost sharing
- Must be necessary for the program
- Must be included in the approved budget

## Valuing In-Kind Contributions

- Volunteer or donated services – rates consistent with rates paid for similar work
- Donated Property – fair market value at time of the donation

## Module 7

- Site Visits

## Reasons

- Familiarization visit
- Staff, business practices
- High dollars
- Program visibility
- Prior negative history

- Complex compliance requirements
- When requested by recipient
- Documentation review

## Prior to Visit . . .

- Notify recipient POC
- Discuss objectives for visit, including documents to review
- Notify program and financial officials
- Review grant file
- Determine issues to discuss

## While on Site . . .

- Conduct entrance discussion with officials
- Observe project activities
- Review program and financial records
- Compare to submitted reports
- Property records

- A checklist can be very useful!

## After the Visit . . .

- Prepare report
- Document findings
- Negative and positive
- Review and adjust monitoring plan, if necessary

## Module 8

- Substantial Involvement

## What is Substantial Involvement?

- No formal government-wide definition
- Agency participation – shared responsibility for project success
- Collaboration
- Consultation
- Intervention
- Control
- Difference between grant and cooperative agreement

## What Isn’t Substantial Involvement?

- Site visits
- Monitoring
- Report review

- Under what circumstances would we use
-  a cooperative agreement?

## Considerations/Issues

- Substantial involvement vs. excessive control
- Control of organization’s operations
- Decision-making for organization
- To avoid high-risk designation
- Use of agency seal
- Agency considerations
- Resources
- Tort liability

## Module 9

- Sub-Awards and
- Pass-Throughs

## Sub-Award	

- Definition
- Award of assistance made under a grant to eligible sub-grantees
- A procurement contract under a grant is _not_ a sub-award

## Sub-Award

- History
- Devolution of control of federal funds in the 1980s to the states
- “New Federalism”
- Single Audit Act of 1984
- One audit required regardless of funding
- Recipients must monitor each program, activity, and sub-awards

## Pass Through

- Essentially assume the role of the agency
- Must have sub-award agreements
- Must ensure that all funds it receives, including sub-awards, are used properly

## Sub-Award Agreements

- Primary tool the primary recipient has for monitoring sub-grantees
- Should be detailed and describe responsibility of the sub-grantee, including reporting requirements
- Primary recipient should have a written monitoring plan for the sub-awards
- Plan approved by the agency

## Pros and Cons

- Pros
- Ease
- Shifts burden of monitoring to pass through

- Cons
- Need larger dollar amounts to justify
- Lack of direct control
- Large organization to deal with

## Sub-Awards – Recipient Responsibilities

- Compete sub-awards, if possible
- Have a method of evaluating sub-grantees
- Ensure sub-grantee is not suspended or debarred
- Monitor sub-awards
- Provide technical assistance, if necessary

## Module 10

- Problem Resolution and Prior Approvals

## Recipient Problems

- Problems require different levels of agency concern and action
- Material problems
-  program or financial
- Immaterial problems
- program or financial

## Recipient Problems

- Intentional problems
- disregard for grant terms, laws and regulations, fraud
- Unintentional problems
- lack of experience and knowledge

## Agency Problems

- Poor award instruments
- “Hands off” for too long
- Waiting too long to intervene
- Allowing problems to grow

## When Problems Arise . . .	

- What do you do?
- Phone call
- Determine grantee solution
- Recommend solution
- Provide assistance
- Make high-risk distinction

- At what point do you become aware of problems?

## Don’t let this be you!

## Changing Recipient Institutions

- Reasons
- Loss of primary project manager
- Grantee ceases operations
- Grant terminated
- Replacement grants

## Prior Approvals

- Form of monitoring
- Circular A-110 has guidance
- Grantee risks
- Non-payment
- Suspension
- Termination
- . . . if it disregards the requirement

## What Needs Prior Approval?

- Additional funds
- Change in scope of the project or objectives
- Change in key personnel if identified

- Budget revisions over 10%
- Pre-award costs
- Sub-awards
- Time extensions

## Module 11

- High Risk Recipients

## OMB Circulars

- A-110 and A-122
- Both have requirements regarding high risk recipients
- Process for correction
- Overseas
- Less restrictive
- Can add requirements to the agreements

## Indicators

- Inexperience
- Poor prior performance
- Program behind schedule
- Fraud – suspicion or actual

- Questioned costs from audit
- Violations of grant terms
- Management problems
- Financial instability

## Solutions

- Special terms and conditions
- Cost reimbursement
- More frequent reports
- More report detail
- Additional prior approvals
- Program phase approvals

## Definitions from A-122

- Suspension
- Temporary withdrawal of Federal sponsorship, pending corrective action
- Termination
- Cancellation of Federal sponsorship, in whole or in part, at any time prior to the date of completion
- Debarment

## Terrorism Requirements

- GPD 19 – consult Department of the Treasury Excluded Parties List System  www.epls.gov

## Module 12

- Audits

## What does “audit” mean?

- Examination of financial operations and work
- Review of compliance with applicable laws and regulations
- Evaluation of economy and efficiency of operations
- Evaluation of effectiveness in achieving program results

## “Audit” means . . .

- In the assistance context, it means -
- A look at the past performance of an entity, program or function to determine whether funds and property were properly administered and whether the projects have met or fallen short of program intent and expectations.

## Audit Process

- Objectives
- What are we doing and why?
- Scope
- Program, time period
- Methodology
- How
- Findings
- Results

## Internal Controls

-    The plan of organization,  methods
-    and procedures adopted by management to ensure that resource use is consistent with laws, regulations, and policies; that resources are safeguarded against waste, loss and misuse, and that reliable data is obtained, maintained, and fairly disclosed in reports. (General Accounting Office)

## Audit Terms

- Materiality
- Matter of professional judgment
- Influenced by auditor’s perception of the needs of a reasonable person will rely on the assertion audited
- Reasonable Assurance
- Negative Assurance

## Audit Opinions

- Unqualified
- In our opinion, the F/S present fairly, in all material respects
- Qualified
- Except for . . . the F/S present fairly
- Adverse
- Do not present fairly, in all material respects
- Disclaimer
- No opinion; evidence not available (not auditable) to make a determination

## Other Audit Concerns

- Going concern
- Company will not be able to continue to operate indefinitely (one year)
- Separation of duties
- Authorizing, processing, recording, approving

## Office of Inspector General

- Overview of the OIG
- Audits, Inspections and Investigations

## AUDITS

- Performance
- Assessment of performance to improve public accountability and facilitate decision-making.
- Financial
- Reports on whether an entity’s financial information is presented fairly, implementation of proper internal controls and compliance with laws and regulations.

## AUDIT PLANNING PROCESS

- Develops annual plan
- Plan based on:
- Legal requirements
- OIG knowledge and experience
- Requests (Bureaus and Congress)
- Governmentwide interest
-  Unanticipated audits

## AUDIT TEAM AT POST

- Narrow focus – in-depth review of program or issue
- Specific post involvement
- Short duration
- Systemic or post-specific issues

## INSPECTIONS

- Systematic evaluation of Department operations – posts and bureaus
- Evaluate all post operations
- Includes assessment of post management, security issues, intelligence/law enforcement
- Broad focus
- Issue post-specific reports

## INSPECTION FOCUS

- Policy Implementation
- Achieving policy goals and objectives
- Representing US interests
- Coordinating all elements of a post
- Resource Management (Diplomatic Readiness)
- Effectively and efficiently uses and manages resources
- Accurate reporting of financial transactions and accounts

## INSPECTION FOCUS Cont.

- Management Controls
- Complies with applicable laws/regs.
- Develops and implements internal controls
- Identifies instances of fraud, waste and abuse

## INSPECTION SCHEDULE

- Inspection cycle
- Every post every 5 years
- Requests or specific issues
- Composition of Team
- Broad range of skills and background – primarily FSOs
- Team size depends on post and assignment

## INSPECTION PROCESS

- Advance questionnaires (OIG website)
- Survey
- Fieldwork
- Reporting
- Compliance

## INSPECTION EVOLUTION

- Increased focus on management and leadership
- Focus on treatment of FSNs
- More resources on security and IT
- Highlight best practices

## FMO INSPECTION

- Inspection of financial management covers all section operations
- Budgeting
- Fund management
- Payments
- Payroll
- Allowances
- Cashiering

## INVESTIGATIONS

- Criminal, civil, and administrative misconduct related to organizational programs and operations
- Examine allegations, reports or other information indicating possible violations of laws or regulations
- Violations of federal laws – DOJ
- Administrative violations – HR, DS

## EMPLOYEE RESPONSIBILITIES

- Report all information with the appearance of waste, fraud, abuse, or mismanagement to the OIG
- Executive Order 12674 (April 12, 1989) Federal Principles of Ethical Conduct for Government Officers and Employees

## INVESTIGATORS AT POST

- Minimal notification
- Focused on specific allegation
- Support RSO at post, including local law enforcement
- Officials kept informed of status of investigation, but written reports are rarely provided

## Module 13

- Fraud, Waste, Abuse and Mismanagement

## Fraud

- Definition
- Intentional misstatements or omissions
- Failure to disclose
- Misappropriation
- Theft, defalcation (lying)
- Irregularity
- Intentional non-compliance
- Collusion
- Two or more working to conceal

## Red Flags

- Too much white-out
- Excessive photocopies instead of originals
- Checks written to cash/employees
- Splitting payments to avoid dollar thresholds
- Sales of assets to employees or relatives

- Loans to staff/board members
- Too much cash lying around
- Excessive transactions (adjusting entries) in financial system
- Excessive bank accounts
- Strange alliances – subletting office space, off-the-book deals

## Considerations	

- Need evidence to pursue
- Difficult to recover stolen assets
- Overseas – rarely prosecute or investigate

## Waste, Abuse and Mismanagement

- Lower level than fraud
- Can be difficult to pursue – must confront management
- Whistleblower
- Difficult course of action
- Department Notice
- Investigations – do not guarantee anonymity

## Module 14

- Financial Systems and Records

## OMB Circular A-110

- Uniform Administrative Requirements
- Establishes uniform administrative requirements
- Pre-award
- Post-award
- Financial and program management
- Reports and records
- Termination and enforcement
- Closeout

## Financial System

- 	Financial system should provide accurate, current and complete disclosure of all financial transactions.

## Financial System

- 	Adequate system lies somewhere between the “shoebox” system and electronic system.

## Policies and Procedures

- Policies and procedures for authorizing purchases, payments, sub-awards, contracts, hiring
- Written and well documented
- Is staff trained and does staff follow?

## Elements of Financial System

- Accounting “books” – general ledger, double entry system
- Business records
- Banking – signatories, blank checks secured
- Purchase orders
- Contracts

## Elements of Financial System

- Vendor invoices
- Employment agreements
- Personnel manual
- Time sheets – signed and allocated
- Cash – petty cash ledger, cash activity reports.  Is cash secured?

## Elements of Financial System

- Support documentation – records that track and support expenditures and provide an audit trail
- Audit trail – records that provide the means to trace a transaction from inception through the books to payment.

## Module 15

- Reports
- (Program and Financial)

## Purpose of Reports

- May serve as only visible indicators of program activity and expenditure of funds
- Read and analyze!
- Program and financial reports should encompass the same period of time for comparison and analysis purposes

## Program Reports

- Format
- No standard government-wide format
- Frequency
- Not more than quarterly or less than annually
- Content
- Comparison of actual results with stated goals and objectives
- Reasons goals and objectives were not met
- Other information as appropriate
- Significant events that occurred

## Report Review	

- Do the reports –
- Provide sufficient information?
- Show that adequate progress is being made?
- Raise red flags?
- Contain information that is useful for GPRA and performance measures?

## Financial Reports

- Purpose
- To certify expenditure of funds
- Evaluate project progress
- Account for funds
- Show cash needs

- Compare to progress report.
- Does progress correlate to amounts expended?
- Are they consistent?

## Sample Reports

- ECA Sample Report
- Use for overseas

- SF-269
- OMB requirement

## Module 16

- Grant Closeout

## Definition of Closeout

- Process of determining that all required work has been completed, all reports are submitted and approved, and any pending actions are completed (agency or recipient).

## Do Not Close Grants If...

- Refunds not submitted
- Actions not completed
- All costs not paid to recipient
- Excess funds not deobligated

## OMB Circular A-110

- Within 90 days after completion . . .
- Submit all financial and performance reports as required
- Liquidate all obligations incurred, unless an extension is authorized by the agency
- Recipient promptly refunds all balances

## OMB Circular A-110

- Agency makes prompt payment for allowable costs
- Note - closeout does not mean that agency cannot claim questioned or disallowed costs at a later time during a review or an audit

## Collection of Amounts Due

- Must collect amounts owed to USG
- Federal Claims Collection Standards
- Aggressive action
- Make administrative offset against other requests for reimbursement
- Withhold advance payments due the recipient
- Other action as permitted by statute

## Module 17

- Miscellaneous

## Forms

- DS-1909 – Federal Assistance Award
- DS-1909A – Amendment to Federal Assistance Award
- SF-424 – Application (OMB)
- SF-269 – Financial Report (OMB)