OFFICE OF 

INSPECTOR GENER!L !udit Report 

Farm Credit !dministration’s 

Commissioning Program 

!-14-04 

!uditor-in-Charge 

Tori Kaufman 

Issued March 31, 2015 

F!RM CREDIT !DMINISTR!TION 
 

 

 

 

     

              

             

              

   

       

  

   

                     

                        

                           

   

                               

              

                      

           

                              

                   

                         

                 

       

                       

         

                       

                   

Farm Credit Administration	 Office of Inspector General 

1501 Farm Credit Drive 

McLean, Virginia 22102-5090 

March 31, 2015

The Honorable Kenneth A. Spearman, Board Chairman

The Honorable Dallas P. Tonsager, Board Member

The Honorable Jeffrey S. Hall, Board Member

Farm Credit Administration

1501 Farm Credit Drive

McLean, Virginia 22102‐5090

Dear Board Chairman Spearman and FCA Board Members Tonsager and Hall:

The Office of Inspector General (OIG) completed an audit of FCA’s Commissioning Program.

The objective of this audit was to determine whether FCA is effectively managing the

Commissioning Program.

During our audit, we found that the Office of Examination (OE) has a robust training and

development program. Each pre‐commissioned examiner attends formal training courses,

receives extensive on‐the‐job training, and completes certification testing. Progress is tracked

and documented continuously throughout the multi‐year program.

We identified nine opportunities for improvement. In response to our audit, OE and OMS have

agreed to take actions to improve the Commissioning Program, including:

1.	 Identify and track specific commissioning costs to evaluate the cost of the program

and identify cost‐saving opportunities and consider timekeeping code revisions,

with OMS assistance in implementation.

2.	 Establish a process to verify time charged by Associate Examiners complies with

work performed and timekeeping guidance.

3.	 Analyze the costs and benefits of streamlining and consolidating current testing and

assessment milestones through the elimination of the final Commissioning Test

simulations.
                     

                           

                   

           

                     

                       

             

                         

               

                 

           

          

                     

           

                            

                                   

     

   

   

4.	 Establish a plan to compete Commissioning Program contractor services to manage

risks of reliance on one source and ensure the best value to the Agency.

5.	 Ensure current Commissioning Program contracts are well‐defined in regards to

general and administrative and hourly rates.

6.	 Ensure the invoice approval process for the Commissioning Program covers the

requirements of the contract and review by all Agency personnel necessary to

verify work performed before approval and payment.

7.	 Assess strategies to identify the cause of hiring shortfalls and employee attrition to

meet commissioned examiner goals and maximize Agency investments.

8.	 Evaluate opportunities to implement Service Agreements or another type of

comparable reimbursement arrangement to protect Agency investments in

Commissioning Program training and certification.

9.	 Revise processes to provide feedback to every Associate Examiner on Technical

Evaluations and Commissioning Test multiple‐choice test performance.

We appreciate the courtesies and professionalism extended by FCA personnel to the OIG staff.

If you have any questions about this audit, I would be pleased to meet with you at your

convenience.

Respectfully,

Elizabeth M. Dean
 

Inspector General
 

Enclosure
                       

          

                 

                    

              

      

                     

                    

                         

             

                   

             

               

              

              

           

       

                   

                   

           

                   

         

                 

                 

   

   

                     

OBJECTIVE:

To determine whether

FCA is effectively

managing the

Commissioning Program.

BACKGROUND:

A commission is a

designation to signify that

a Farm Credit

Administration examiner

is qualified to examine

Farm Credit System

institutions.  The

Commissioning Program

consists of specific

training, on‐the‐job work

experience, and testing in

order to cover necessary

Office of Examination

(OE) knowledge, skills,

tasks, and performance

attributes.  Program

requirements are

completed over a multi‐

year, two‐tier process,

which culminates with a

commission and is the

foundation for OE’s

Career Path Program.  

Commissioning Program

requirements,

responsibilities, policies,

and procedures are

described in Policies and

Procedures Manual 503.     

    

The Farm Credit Administration (FCA or Agency) has a robust training and

development program to commission examiners. Each pre‐commissioned

examiner (Associate Examiner) attends formal training courses, receives extensive

on‐the‐job training, and completes certification testing. Progress is tracked and

documented continuously throughout the multi‐year program. Our review

revealed opportunities for improvement. Specifically:

	 Overall costs of the Commissioning Program are not monitored, despite the

significance of the program and the level of resources committed.

Although there is a budget detailing employee staff days and travel days for

the Commissioning Program and contract services, the Agency’s

timekeeping and budgeting systems do not differentiate time or dollars

charged for commissioning from recruiting/hiring to commission

certification.

	 Potential efficiencies could be gained by streamlining and consolidating

Commissioning Program assessment milestones. Associate Examiners must

demonstrate specific competencies throughout a variety of continuous

comprehensive assessments to earn a commissioned examiner

certification.

	 Contracts were single‐sourced to one contractor thereby, relying on one

source to provide services that are critical to operating the Commissioning

Program puts the Agency at risk.

	 Staffing shortfalls and attrition impacted OE’s ability to maximize training

investments in the Commissioning Program.

	 Providing feedback to candidates for all components of Technical

Evaluations and the Commissioning Test will support Associate Examiners’

development.

There are 9 agreed‐upon actions to improve the Commissioning Program.
     

   

       

             

         

           

         

       

         

         

         

         

   

Table of Contents

BACKGROUND _______________________________________________________________________ 1
 

AUDIT RESULTS ______________________________________________________________________ 3
 

Opportunities for Improved Cost Monitoring ____________________________________________ 4
 

Agreed‐Upon Actions 1‐2 ____________________________________________________________ 5
 

Streamlining Testing and Evaluation ___________________________________________________ 5
 

Agreed‐Upon Action 3 _______________________________________________________________ 8
 

Contractor Oversight ________________________________________________________________ 8
 

Agreed‐Upon Actions 4‐6 ___________________________________________________________ 10
 

Other Areas for Improvement _______________________________________________________ 10
 

Agreed‐Upon Actions 7‐9 ___________________________________________________________ 11
 

OBJECTIVE, SCOPE, AND METHODOLOGY ________________________________________________ 12
 

ACRONYMS ________________________________________________________________________ 13
 
 

1

BACKGROUND 

The core mission of the Farm Credit Administration (FCA or Agency) is to regulate and supervise the

Farm Credit System (FCS) in order to ensure a safe, sound and dependable source of credit and related

services for agriculture and rural America.  FCA’s Office of Examination (OE) plays a critical role in

accomplishing this mission by examining and supervising FCS institutions.  Examiners are the essential

link with institutions through on‐site examinations, interactions with FCS boards and management, and

written reports and correspondence.  For this reason, one of OE’s top priorities is to maintain and train

highly skilled examiners who understand the unique risks of agriculture, retain sufficient regulatory and

financial experience, and communicate effectively.           

A commission is a designation to signify an FCA examiner has fulfilled professional, educational, and

experience requirements and is qualified to examine FCS institutions.  Specifically, only a commissioned

examiner can act as an Examiner‐in‐Charge.  The commission designation is used by other financial

regulators, which makes it a valuable industry standard for the Agency.   

OE’s Commissioning Program consists of extensive formal training courses, targeted on‐the‐job work

experience, and testing covering required knowledge, skills, tasks, and performance attributes (referred

to as competencies).  Pre‐commissioned examiners (Associate Examiners) achieve core competencies

through training, and accomplishment is confirmed through testing.  These required competencies are

the entry way to promotion into OE’s Career Path Program and a career as an FCA examiner.

The Commissioning Program is a multi‐year, two‐tier progression, which culminates with a commission.   

Commissioning Program

Tier 1‐ Basic Tier 2‐Foundational

•Years 1‐2 •Years 3 and beyond

•Proficiency validated by •Commissioning Test certification

Technical Evaluations completed

                        

Tier 1 includes years 1 and 2 of the Commissioning Program, and tier 2 includes years 3 and beyond.  

During tier 1, all Associate Examiners are assigned to OE’s Staff Development Division (SDD).  Within

SDD, Associate Examiners are assigned to trainers, experienced senior examiners who ensure work is

consistent with quality standards and who provide extensive feedback.   

During tier 2, Associate Examiners are assigned to either the Market Risk Division or the Association

Examination Division within OE.  As part of their division assignments, they shadow a seasoned

Examiner‐in‐Charge who is assigned as a coach.  In addition to on‐the‐job training, many training courses

must be completed at each level.  This basic structure may be adjusted based on each Associate

Examiner’s progress.
 

                     

                    

                       

                         

                        

                   

       

                       

                                

                          

         

   

   

   

   

   

   

                          

                             

                                

                       

                            

                     

                    

                               

                                

                              

                               

             

                               

                

                     

                          

                      

                                

                             

                                

The Commissioning Program incorporates various tools to determine whether an Associate Examiner

can demonstrate required competencies. Trainers and Examiners‐in‐Charge use the Associate Examiner

Competency Evaluation System (AECES), a computer application developed in coordination with FCA’s

Office of Management Services (OMS), to capture feedback and document successful completion of

critical on‐the‐job tasks for each candidate. In addition, Associate Examiners’ progress is continuously

evaluated through simulated training exercises that recreate work experiences, post‐training quizzes,

and formal testing.

Associate Examiners must pass three tests (described below) to advance through the Commissioning

Program and achieve promotions. Technical Evaluation tests are administered at the end of years 1 and

2, before candidates are converted to permanent positions. These tests cover competencies developed

during each respective year.

Technical

Evaluation 1

•Multiple‐choice test

•Loan Simulation

Technical

Evaluation 2

•Multiple‐choice test

•Loan Simulation

•Financial Institution

Rating System

Simulation

Commissioning

Test

•Multiple‐choice test

•Examination Scoping

and Planning

Simulation

•Loan Simulation

(conducted offsite)

•Examination

Reporting Simulation

(conducted offsite)

Both Technical Evaluations include a multiple‐choice test and simulation tests. Technical Evaluation 1

includes a loan simulation, and Technical Evaluation 2 includes a loan simulation and a Financial

Institution Rating System (FIRS) simulation. The objective of the loan simulations is to test the Associate

Examiner’s ability to independently analyze risks, document support, and formulate interview questions

for loan officers. The FIRS simulation measures the candidate’s skill in independently identifying an

institution’s strengths and weaknesses, developing targeted questions for institution management, and

drawing conclusions on the safety and soundness of an institution.

Year 1 and 2 Technical Evaluations may also assist in identifying Associate Examiners who are not

progressing as anticipated. Prior to year 3 a candidate falling short of anticipated progress can be

released from the program in order to minimize further investments in their training and development.

This assessment is increasingly important given the Agency’s small size and the resulting higher cost per

candidate to administer formal training courses.

The Commissioning Test is the final element of the program and serves as a standardized metric to

evaluate candidates’ knowledge and demonstration of required performance attributes. Final

Commissioning Test simulations test performance attributes including: planning exam activities, drawing

conclusions, written and oral communication skills, and teamwork. Various controls are utilized to

create an independent testing environment for these simulations. Multiple experienced FCA examiners

(assessors) score each candidate and role play as loan officers and institution officials. Assessors are not

assigned to candidates they have supervised or worked closely with, and a contractor oversees each

assessor to evaluate objectivity. In addition, the composite score necessary to achieve a passing rate is

2
 
 

not disseminated as a further measure to ensure valid scoring.  Following the final multiple‐choice test

and three days of final simulation testing, if passed, a candidate is certified as a commissioned FCA

examiner.

      

In 2014, OE hired a contractor to analyze and update the required competencies for newly

commissioned examiners.  The same contractor had completed a prior review and update as part of

OE’s 2008 Commissioning Program redesign project.  The redesign project also allocated responsibilities

to OMS, including:

 Assisting in administration of the Commissioning Test twice a year,

 Grading the multiple‐choice and reviewing the simulation Commissioning Test results, and

 Funding a contractor to monitor each final offsite Commissioning Test simulation.  

Responsibilities and requirements of the Commissioning Program are described within OE’s Policies and

Procedures Manual (PPM) 503, Examiner Commissioning.

  

AUDIT RESULTS 

OE has a robust training and development program to ensure Associate Examiners obtain the necessary

competencies to examine FCS institutions.  Formal training courses are administered over about a 3‐year

period and cover areas such as credit, exam management, finance, and operations.  Commissioning

Program requirements are aligned with essential OE knowledge, skills, tasks, and performance

attributes, which are updated periodically to reflect current examination responsibilities.  OE also

reviews testing and training materials each year for comprehensiveness.  Staff levels and the number of

commissioned examiners are continuously tracked through OE quarterly reports, yearly operating plans,

briefings to the FCA Board, and the Agency Human Capital Plan.  

Several controls are in place to ensure Associate Examiners meet developmental milestones.  

Performance of critical on‐the‐job activities is rated and tracked for each candidate throughout the

program.  The Commissioning Program’s tiered approach with year 1 and 2 testing is also designed to

assess the level at which Associate Examiners can demonstrate required competencies.  Promotion to

higher grade levels is contingent on passing each test.  This structure limits Agency investment based on

specific progress.       

Our review revealed opportunities for improvement and efficiency in the Commissioning Program.  

Specifically, we identified opportunities to:  

  

 improve cost monitoring,  

 streamline testing and evaluation,

 improve contractor oversight,  

 reduce staffing shortfalls, and  

 provide additional feedback to candidates.  

3
 

	 	 	 	

     

                           

                      

                       

                       

                        

                      

                             

                          

                     

                                       

                         

                            

                                    

                       

                       

                          

           

   

   

     

   

                                 

                            

                                 

                            

                               

                                  

                    

	 	

	 	 	 	 	

	 	 	 	

	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	

                                                            

                                      

                                

                    

                                         

Opportunities for Improved Cost Monitoring

Commissioning Cost Estimate

The significance of the Commissioning Program, and the resources committed to its success, warrant

increased cost monitoring. The Government Accountability Office (GAO) created A Guide for Assessing

Strategic Training and Development Efforts in the Federal Government (GAO‐04‐546G) as a framework

to address how agencies plan, design, implement, and evaluate effective training and development

programs. The guide highlights the importance of knowing costs to improve planning,

design/development, implementation, and evaluation of training programs. By capturing the costs of

these programs, an agency can determine the appropriate level of investment and prioritize funding to

address the most important needs first. Cost information also informs managerial decisions by

evaluating the cost‐effectiveness of different options for achieving learning objectives.

The cost of the FCA Commissioning Program is not tracked in a way that could be utilized to identify the

cost of the program from recruiting/hiring to obtaining the commission certification and evaluate cost‐

saving opportunities. OE formulates budgets and tracks costs based on employee staff days and the

cost of travel, its main cost drivers. Total OE staff days and travel costs are monitored in general,

activity‐based categories that correspond to timekeeping codes, and the Agency’s timekeeping and

budgeting systems do not sufficiently allow the identification and differentiation of various

commissioning activities. Currently, the cost to hire, train, and commission an Associate Examiner is

included in five different timekeeping categories:

 Recruiting,

 Training Administration and Development,

 Training Taken,

 On‐the‐Job Training, and

 Commissioning Program

For the purposes of this audit, OE prepared a general estimate of the total cost of the Commissioning

Program and the cost to commission an individual Associate Examiner. OE applied percentages of

budgeted OE staff days to total dollar costs for each timekeeping category and used these totals to

calculate the average total cost for the Commissioning Program from FY 2011 through 2014. This

average total cost per year was divided by the average number of participating Associate Examiners per

year and multiplied by four to reflect the general time period to be commissioned. OE estimated the

cost to commission an Associate Examiner to be about $416,323.

OE Estimated Commissioning Costs

Average Commissioning Program Cost per year $4,163,235

Average Number of Associate Examiners 40

Average Total Cost Per Year/Average Number of Associate Examiners $104,081

Four Year Estimated Average Cost per Associate Examiner $416,3231

1 This OE estimate was based on a number of assumptions and manual adjustments. OIG did not validate its

reliability. This estimate does not include OMS costs, contractor costs, or hotel space rental expenses associated

with the Commissioning Program. In addition, OMS performance reports did not differentiate “On‐The‐Job

Training” time, so actual dollar costs in this category were assumed to be the same as what was budgeted.

4
 
 

                       

                

   

                             

                    

                             

                        

                         

            

                           

                           

                    

                        

                   

        

	 	 	

             

                             

                   

                           

   

	 	 	 	

          

                           

                  

               

    

           

   

               

             

                          

                   

                  

                            

                     

Timekeeping

We also noted disconnects in timekeeping, timekeeping guidelines, and budgeting that further limited

the accuracy of commissioning cost monitoring. For example:

	 For the first two years of the Commissioning Program (tier 1) Associate Examiners were directed

by timekeeping guidance to code all institution examination hours as “On‐The‐Job Training.”

However, for FY 2013 and 2014, 6 tier 1 Associate Examiners charged about 190 onsite

examination staff days incorrectly. OE officials stated that despite the direction in timekeeping

guidance, a percentage of tier 1 Associate Examiners’ time directly contributed to actual

institution examination versus on‐the‐job training.

	 In regards to budgeting, OE’s Guide for Recording Time and Travel directed Associate Examiners

to charge time spent studying for Technical Evaluations and the Commissioning Test to the

“Commissioning Program” timekeeping code. However, this time was consistently budgeted as

“Training Taken.” OE officials stated that this issue stemmed from an ongoing disconnect

between timekeeping guidance and the way OE budgeted for study time.

Agreed‐Upon Actions 1‐2

To improve cost monitoring, OE agreed to:

1.	 Identify and track specific commissioning costs to evaluate the cost of the program and identify

cost‐saving opportunities and consider timekeeping code revisions, with OMS assistance in

implementation.

2.	 Establish a process to verify time charged by Associate Examiners complies with work performed

and timekeeping guidance.

Streamlining Testing and Evaluation

Opportunities for Streamlining and Consolidating

OE utilizes the Commissioning Test as a standardized metric to certify Associate Examiners can

demonstrate competencies of a commissioned examiner. Before the Commissioning Test, each

candidate must successfully complete various assessment milestones, including:

 simulated exercises as part of training,
 

 post‐training quizzes,
 

 year 1 and 2 Technical Evaluation tests, and
 

 fully‐effective ratings for all AECES on‐the‐job tasks.
 

OE’s training program is designed to assist Associate Examiners in reaching these milestones. OE

developed its competency‐based Commissioning Program to cover knowledge, skills, tasks, and

performance attributes necessary to be a commissioned examiner. These competencies include

numerous aspects of credit, exam management, finance, and operations. In total, the Agency identified

more than 400 interrelated competencies required to be a commissioned examiner.

5
 
 

                         

                           

                              

                                   

                      

                                      

                         

     

                           

                        

                       

                           

                         

                           

     

                         

                      

                          

                                 

                            

	 	 	 	

	 	

     

   

   

	 	 	 	 	 	

   

     

     

         

         

       

             

           

     

    ‐ ‐      

         

Throughout the Commissioning Program, each candidate is continuously evaluated to ensure they have

the capacity to demonstrate these competencies. Of the more than 400 competencies, OE identified

nearly 150 critical on‐the‐job tasks. Each time an Associate Examiner attempts one of these on‐the‐job

tasks it is tracked in AECES, and it is rated as ineffective, moderately effective, or effective based on

specific definitions related to adequate performance. Trainers and Examiners‐in‐Charge also provide

feedback and comments, and must do so if a task is rated ineffectively. In order to sit for the

Commissioning Test, an Associate Examiner must have effectively completed all on‐the‐job tasks tracked

in AECES.

We noted potential efficiencies could be gained by streamlining the Commissioning Program, perhaps by

condensing and consolidating assessment milestones. A large portion of the Commissioning Test

consists of three simulations (Examination Scoping and Planning, Loans, and Examination Reporting)

graded by independent assessors, selected, trained and observed to create a controlled environment.

Because other existing assessment channels may provide assurance that an Associate Examiner is

proficient in mastering these competencies, OE could potentially revise the program to decrease costs

and reallocate resources.

OE considers simulations to be a necessary standardized metric to assess and certify performance

attributes. While the final Commissioning Test simulations create a standardized metric, standardization

is also accomplished through the competency‐based training model utilized by OE. For example,

completion of the same 150 tasks is tracked for each Associate Examiner in AECES, and effective ratings

are based on specific definitions. The chart below depicts examples of potential areas for consolidation:

Performance Attributes Evaluated in

Final Commissioning Test Simulations

Planning oversight and examination

activities

Analyzing information, drawing

conclusions, researching, and investigating

Oral communication

Written communication

Teamwork

Examples of Critical Tasks Evaluated in

AECES

Delegate examination responsibilities to

examiners and develop examination

scope, strategies, and schedules

Evaluate underlying causes of capital

trends and evaluate the amount,

composition, and stability of earnings

Interview loan officers to obtain or verify

loan information

Produce written products that adhere to

the general writing protocol

Serve as Examiner in Charge for institution

examinations or lead examiner for

examination segments

6
 
 

          

                       

                                  

                              

                               

                         

                                    

                       

                             

                           

	 	 	 	

	 	

   

   

   

   

	 	

                  

      

                             

                            

                   

	 	 	 	 	 	

	 	 	 	 	

	 	 	 	 	 	

	 	 	

	 	 	 	 	

                                                            

                       

Final Commissioning Test Simulation Resource Requirements

Planning and conducting final Commissioning Test simulations is very resource‐intensive. Final

Commissioning Test simulations, held twice a year, are a weeklong process each time. Two of the three

final simulations are conducted at an offsite hotel, attended and monitored by a contractor. In

addition to the Associate Examiners undergoing the simulation, for the years in review, each final offsite

Commissioning Test simulation was attended by 2‐4 monitors, and 6‐12 experienced FCA examiners

assigned to role play and grade each test (assessors). All assessors are also trained by a contractor, both

when selected to be an assessor, and before the actual simulations.

For FY 2011 through 2014, about $289,303 was spent to conduct final offsite Commissioning Test

simulations, which included travel, contractor, and hotel expenses, as noted in the chart below.

Final Offsite Commissioning Test Simulation Costs

Test Date Travel

Costs

Contractor

Cost

Hotel

Costs

Total

Cost Monitors Assessors CandidatesTested

October

2010 $18,295 $7,725 $3,229 $29,248 4 6 5

October

2011* $28,328 $8,529 $14,742 $51,599 3 9 8

April 2012 $19,886 $11,429 $9,194 $40,509 3 6 5

October

2012 $39,254 $11,179 $9,019 $59,452 3 12 10

April 2013 $20,903 $10,261 $7,922 $39,087 3 6 6

October

2013 $29,174 $10,874 $1,536 $41,584 3 9 7

April 2014 $17,595 $7,622 $2,608 $27,825 2 6 4

Total $173,434 $67,620 $48,250 $289,303

Average

Cost Per

Test

$24,776 $9,660 $6,893 $41,329

*Commissioning Test simulations were not held in April 2011.

In addition to reducing costs by eliminating final offsite Commissioning Test simulations, staff time could

be re‐directed to other mission‐based activities. In order to quantify this, we calculated the percentages

of budgeted staff days to administer final Commissioning Test simulations:

Staff Time for Final Commissioning Test Simulations

FY 2015 Budget FY 2016 Budget

Staff Days for Final Commissioning Test Simulations 216 130

Overall “Commissioning Program” Staff Days 772 590

Simulation Percentage of “Commissioning Program” Staff Days 28% 22%

Average Simulation Percentage 25%2

2 The actual average simulation percentage is 25.0066% and was rounded for simplification.

7
 
 

                           

       

	 	 	

	 	 	 	 	 	 	 	

	 	 	

	 	 	

                                 

                               

                          

                           

                            

                       

                       

                                 

                               

                          

                          

                    

                          

                         

                                

                         

       

	 	 	

               

                         

                   

					

                           

                        

                               

   

                                                            

                                 

                                    

                       

We then applied this percentage to actual “Commissioning Program” dollar costs, which tracked overall

testing costs for OE.

Costs for Staff Time

FY 2011 FY 2012 FY 2013 FY 2014 Total

“Commissioning Program” Dollar

Costs $231,392 $311,207 $369,379 $345,592 $1,257,570

Average Simulation Percentage $57,863 $77,822 $92,369 $86,421 $314,475

Average Simulation Cost Per Year $78,619

Considering $41,329 as the average cost of final offsite simulations and the simulation is given twice a

year, we calculated that about $82,658 per year in expenses could be eliminated by removing final

Commissioning Test simulations. In addition, the calculated value of time spent administering these

final simulations that could be re‐directed under a revised testing structure would average about

$78,619 per year. In total, we conservatively estimated3 that the Agency could recognize about

$161,277 in savings each year by removing final Commissioning Test simulations.

Although there would be resource requirements associated with revising existing assessments and

substituting the same quality of oversight and testing to be given at earlier time junctures, a review

process is already in place during 2015 to update the Commissioning Program based on the September

2014 revised competencies. In addition to decreased costs and resource requirements, a condensed

testing structure could also decrease the timeframe to commission an examiner. Competencies are

developed and demonstrated throughout the Commissioning Program. The majority of Associate

Examiners pass the multiple‐choice portion of the Commissioning Test after their first attempt.

Additionally, the multiple‐choice portion of the Commissioning Test is administered in the Associate

Examiner’s field office location, so it would be easier and less costly to re‐schedule, if necessary.

Although these costs and benefits represent examples, OE expertise will provide additional substantial

opportunities to improve program effectiveness.

Agreed‐Upon Action 3

To improve the Commissioning Program, OE agreed to:

3.	 Analyze the costs and benefits of streamlining and consolidating current testing and assessment

milestones through the elimination of the final Commissioning Test simulations.

Contractor Oversight

OMS and OE both used the same contractor for consulting services related to the Commissioning

Program. This contractor has provided consulting services on the Agency’s training and commissioning

certification programs for more than 20 years. We found that contractor oversight could be improved.

Specifically:

3 We believe our estimate is conservative because in calculating the staff days for final Commissioning Test

simulations, we only included the days budgeted for administering the test. We did not include time for reviewing

final Commissioning Test simulations, planning new exam exercises, or initial training for assessors.

8
 
 

                            

                        

                             

                              

                     

                         

                                

                         

       

                           

                    

                                 

                           

                         

                                

                          

                       

                            

                           

                          

                 

                           

                        

             

  

      

   

                               

                                

                                        

                         

                              

   

                        

                                  

                                

                             

                   

                           

                             

	 All work was awarded to the contractor on a single‐source basis. A single‐source selection is the

award of a contract without competition and requires a documented justification. FCA’s single

source justification required an informal market survey or a description of why it was not

conducted. OE and OMS stated a market survey was not conducted due to the contractor’s

“successful history as a consultant on the Agency’s Commissioning Program.”

	 The OMS contract involved OE work on the Commissioning Program; however, many invoices

were only reviewed by OMS. Given the contractor’s involvement with both OE and OMS for the

Commissioning Program, it is important to ensure invoices are routed to the appropriate

individuals for review.

	 We identified a few instances where the contractor was reimbursed outside of specific contract

allowable charges. For instance, the contractor was reimbursed for general and administrative

expenses for one trip that were not allowed because a rate had not been specified in the

contract. Also, the contractor increased hourly rates for two individuals without prior approval.

The contract identified estimated overall costs based on amounts quoted in the contractor’s

proposal but did not identify hourly rates for each staff member. The contractor stated that the

revised hourly rates were based on merit and market adjustment increases. However, the

contractor should have contacted FCA’s Contracting Officer and requested the increase or

charged the same rates throughout the period of performance. Lastly, in three instances, the

contractor was reimbursed for meals and incidentals included on lodging receipts in addition to

receiving per diem allotments. The contractor was also reimbursed for lodging expenses that

exceeded the allowable per diem maximum for one trip.

OE and OMS justified the single‐source selection based on “logical follow‐on” and the contractor’s

longstanding history and experience with the Commissioning Program. This contractor was awarded 6

contracts for a wide variety of services:

 test monitoring,
 

 analyzing required competencies, and
 

 developing hiring assessments.
 

This brings into question whether each award is a “logical follow‐on” for a continuation of previous

services. In addition, relying on a single contractor, and actually one individual for certain services, puts

the Agency at risk if this contractor is not available. It is important to ensure there is more than one

source capable of providing services that are critical to ongoing operation of the Commissioning

Program. Also, competition ensures the best value for the Agency and avoids the appearance of

preference.

Each contract had the same Contracting Officer’s Technical Representative (COTR). The COTR sent

invoices to OE and OMS personnel to ensure that the work was performed. Upon receiving an email

that they were satisfied with the invoice, the COTR approved invoices for payment. OE and OMS

officials who reviewed the contractor’s invoices stated that their review was “high‐level” and they did

not receive training specific to invoice review or approval.

PPM 840, Role and Responsibility of the Contracting Officer’s Technical Representative, states that the

COTR may solicit assistance of other agency personnel during the performance period of the contract,

9
 
 

                                 

   

	 	 	

                      

                         

                     

                       

     

                         

                           

   

	 	 	

     

                               

                          

                               

                             

                                  

             

	 	 	 	 	 	

	 	 	 	 	

	 	 	

	 	 	

	 	 	 	

	 	 	 	

	 	

	 	

                            

                                 

                                

  

but the COTR is responsible for monitoring contractor performance to ensure it is in line with the

contract.

Agreed‐Upon Actions 4‐6

To improve contract oversight and administration, OE and OMS agreed to:

4.	 Establish a plan to compete Commissioning Program contractor services to manage risks of

reliance on one source and ensure the best value to the Agency.

5.	 Ensure current Commissioning Program contracts are well‐defined in regards to general and

administrative and hourly rates.

6.	 Ensure the invoice approval process for the Commissioning Program covers the requirements of

the contract and review by all Agency personnel necessary to verify work performed before

approval and payment.

Other Areas for Improvement

Maximizing Training Investments

FCA’s pending wave of retirements and OE’s position as the primary human capital conduit for the

Agency make the success of the Commissioning Program increasingly important. In FCA’s 2013‐2017

Human Capital Plan, OE projected staff levels would build to 180 over the five‐year period. OE

determined a need for about 125‐130 of this total to be commissioned examiners and technical

specialists. OE originally aimed to hire 10 Associate Examiners each year, but this goal was increased to

account for hiring shortfalls and increased attrition.

Planned vs. Actual Hiring, Attrition and Commissioning

FY 2011 FY 2012 FY 2013 FY 2014

Planned Associate Examiner Hiring 13 10 15 15

Actual Associate Examiner Hiring 13 6 11 9

Planned Associate Examiner Attrition Rate 11% 8% 5% 10%

Actual Associate Examiner Attrition Rate 15% 13% 5% 13%

Planned Newly Commissioned Examiners 5 12 12 9

Actual Newly Commissioned Examiners 3 9 8 8

As a smaller financial regulator, it is important for FCA to maximize training investments. Because

training course costs are mostly fixed each year, hiring shortfalls result in a higher per capita training

cost. In other words, 9 Associate Examiners or 15 require relatively the same cost for classroom

instruction.

10
 
 

                           

                              

                             

                              

                                    

          

                             

                            

                           

                     

                       

                                     

                              

                            

                           

                             

  

                        

                 

                       

                               

                          

                                     

                           

                          

                             

                               

                                 

           

	 	 	

                           

                           

               

                   

                 

  

                         

       

OE largely attributed hiring shortfalls to challenges in using the OPM‐required Pathways Program, a

hiring program for students and recent graduates implemented in 2012. OE addressed this challenge by

revising its hiring assessment questionnaire and online tests for Associate Examiners to further limit the

pool of applicants to qualified candidates. Nine of the planned 15 Associate Examiners were hired

during FY 2014 after implementing the new method. As a result, more hiring will be necessary in the

future to meet planned projections.

If an Associate Examiner leaves the Agency during the Commissioning Program, it also affects training

investments. Currently, OE does not have a reimbursement opportunity or other method to protect

FCA’s investments if an Associate Examiner or newly commissioned examiner leaves the Agency.

By contrast, FCA’s training policy, PPM 843 Training, Development, Professional Certifications, Licenses,

and Membership Fees, requires employees to sign a 1‐year Service Agreement when receiving more

than 80 hours of training for one course or $15,000 tuition for a particular subject matter in a 12‐month

period. If an employee voluntarily leaves the Agency before completing this service, the employee will

reimburse FCA for tuition (unless the obligation is waived). Although the Commissioning Program does

not specifically fit this definition, this type of arrangement with required reimbursement could be

extended to the Commissioning Program to limit the risk of costly attrition after training and

certification.

Feedback

Feedback is another important aspect of an Associate Examiner’s testing and development. OE’s

Performance Feedback Process directive emphasizes the importance of continuous, constructive

feedback to address individual performance and identify progress towards career goals.

Through interviews, we learned that OE did not administer feedback to all candidates for components of

Technical Evaluations and the Commissioning Test. For the Technical Evaluations, there is a pass/fail

score. Associate Examiners receive feedback only if they do not pass any portion of the test. For the

Commissioning Test, candidates receive simulation feedback regardless of whether or not they pass, but

feedback is not provided for the multiple‐choice tests. Because the Technical Evaluations are designed

to represent condensed versions of the Commissioning Test and serve as a preparation tool, feedback

would be useful to focus on strengths and weaknesses regardless of whether or not the candidate

passes. Candidates informed the OIG that additional feedback would be put to good use in preparing

for the next phase of their certification.

Agreed‐Upon Actions 7‐9

To support human capital requirements and Associate Examiner development, OE and OMS agreed to:

7.	 Assess strategies to identify the cause of hiring shortfalls and employee attrition to meet

commissioned examiner goals and maximize Agency investments.

8.	 Evaluate opportunities to implement Service Agreements or another type of comparable

reimbursement arrangement to protect Agency investments in Commissioning Program training

and certification.

9.	 Revise processes to provide feedback to every Associate Examiner on Technical Evaluations and

Commissioning Test multiple‐choice test performance.

11
 
 

12

OBJECTIVE, SCOPE, AND METHODOLOGY 

The objective of this audit was to determine whether FCA is effectively managing the Commissioning

Program.  We conducted fieldwork at FCA’s headquarters in McLean, VA from September 2014 through

March 2015.  We limited our scope to Fiscal Years 2011 – 2014.   

We completed the following steps to accomplish the objective:

 Reviewed applicable laws, regulations, policies and procedures related to the Commissioning

Program.  

 Interviewed OE and OMS officials on Commissioning Program plans, programs, and processes.

 Obtained and analyzed Human Capital Plans, Operating Plans, Quarterly Reports, and

presentations to FCA’s Board.

 Evaluated FY 2015‐2016 Commissioning Program budget documentation and how costs and

resources are monitored.

 Reviewed estimated Commissioning Program costs prepared by OE and the cost to commission

an Associate Examiner over four years.

 Analyzed the costs of final Commissioning Test simulations.

 Analyzed and reviewed contractor invoices pertaining to Commissioning Program administration

services and reviewed single‐source justifications for the associated contracts.

 Interviewed a judgmental sample of six Associate Examiners in different stages of the

Commissioning Program.

This audit was performed in accordance with Generally Accepted Government Auditing Standards.  

Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence

to provide a reasonable basis for our findings and conclusions based on our audit objective.  We

assessed internal controls and compliance with laws and regulations to the extent necessary to satisfy

the objective.  Our review would not necessarily have disclosed all internal control deficiencies that may

have existed at the time of our audit.  We assessed the computer‐processed data relevant to our audit

objective and identified issues included in the body of the report and footnotes.  Overall, we believe the

evidence obtained provides a reasonable basis for our conclusions based on our audit objective.
 

13

             

         

       

         

       

       

           

           

           

   

ACRONYMS 

AECES

COTR

FCA

FCS

GAO

OE

OIG

OMS

PPM

SDD

Associate Examiner Competency Evaluation System

Contracting Officer’s Technical Representative

Farm Credit Administration

Farm Credit System

U.S. Government Accountability Office

Office of Examination

Office of Inspector General

Office of Management Services

Policies and Procedures Manual

Staff Development Division
                             

   

       

              

      

  

      

       

       

  

REPORT

Fraud | Waste | Abuse | Mismanagement

FARM CREDIT ADMINISTRATION
 

OFFICE OF INSPECTOR GENERAL
 

Phone: Toll Free (800) 437‐7322; (703) 883‐4316
 

Fax: (703) 883‐4059
 

E‐mail: fca‐ig‐hotline@rcn.com

Mail: Farm Credit Administration
 

Office of Inspector General
 

1501 Farm Credit Drive
 

McLean, VA 22102‐5090