Many of the estimates made by the agency, including the four “workload unit” estimates
presented in the preceding paragraph, are driven by factors that are outside the control of the
agency. The examples are not meant to be critical of the agency’s estimation techniques or
performance. Rather they are there to illustrate how important it is that the public be informed
these are estimates (i-e., nine months of actuals combined with three months of projections),
otherwise it would be misled.

Testing Workload Units

In addition to identifying differences between the estimates and actuals in the PAR
caused by reliance on estimated figures, the OIG also checked, on a limited basis, the source
documentation behind the estimated and final numbers. In other words, we reviewed how
reliable select systems are when producing workload data for reporting purposes.

We reviewed the source documentation behind the workload units for three
bureaus/offices: the Office of the Secretary, the Office of Administrative Law Judges and the
Bureau of Trade Analysis — Office of Service Contracts. Within the OS, we reviewed the count
of Federal Register Notices; in the Office of the ALJ’s, we reviewed (i) formal proceedings
settled and dismissed and (ii) decisions issued but not settled by the ALJ; within BTA, we
reviewed tallies of new service and amendment contracts. We found that two of the three
processes reviewed produced inaccurate workload units.

Office of the Secretary

The Federal Register (FR or Notice) is a daily publication containing rules, proposed
rules, Federal agency and organization notices, and executive orders. In the FR, the FMC reports
on mission-related activities, to include filed agreements, applications submitted for casualty and
performance certificates and ocean transportation intermediary licenses, among other things.

The OIG visited the Government Printing Office’s (GPO) website to verify OS’s
compilation of Federal Register Notices. The GPO’s website is not organized to abstract the
information by agency. Therefore, we reviewed all daily Notices published in FY 2006 to
identify FMC Notices. We then compiled and scheduled each pertinent Notice.

Our analysis of Federal Register Notices issued in FY 2006 identified 357 Notices
pertaining to FMC activities. By way of comparison, OS reported 260 Notices in the PAR
(roughly nine months of FY 2006 data) and 323 Notices in the FY 2008 Congressional Budget
submission. While the tally in the PAR is the result of relying on nine months of actual data, it
appears that the Notices were simply miscounted in the Congressional Budget request. As a
result, agency-related Notices were understated in both documents.


Moreover, the importance and value of now imposing such requirements should
not be viewed as critical to any external reviews, as the FMC already has a process in
place for reporting its updated “actual” workload statistics. The FMC currently produces
two documents subsequent to the issuance of the OMB budget submission and the PAR
that contain final, actual workload statistics: the Annual Report and the President’s
Budget. These documents are created early in the new fiscal year, allowing agency staff
additional time to adequately prepare and double-check the required statistical data
shortly after the conclusion of the fiscal year. OA does not believe that the existing
practice of preparing final workload statistics for incorporation into the President’s
Budget and the Annual Report has, at any time, given rise to confusion regarding the
agency’s workload statistics. Most significantly, whenever FMC is asked for actual
workload statistics by Congressional or OMB staff, formally or informally, the latter
documents are the ones from which all inquiries regarding FMC workload are answered.
If deemed appropriate to assuage OIG’s concern for the potential for confusion regarding
the agency’s practice of preparing workload statistics, OA would have no objection to
expressly reflecting this information in the PAR.

The Draft Report contains two recommendations. We are providing corrective
action advice concerning them herein:

Recommendation #1: The OIG recommends that the Office of
Administration ensure that estimates are updated prior to their public release in the
PAR or that the reader be advised that the performance measures are based on nine
months of actual data and three months of estimated industry trends.

The Director, OA, will ensure that the FY 2007 PAR contains a reference to the
fact that the workload statistics are estimates, and that final FY 2007 workload statistics
will appear in the agency’s Annual Report for FY 2007 and the agency’s FY 2009
President’s Budget submission. Since the FY 2007 PAR will be made public on
November 15, 2007, corrective action will be completed on that date.

Recommendation #2: The OIG recommends that Office of Administration
remind offices whose workload unit statistics are dependent on manual counts to
institute controls to guard against miscounts. Once such control is to have two
individuals perform the count and to reconcile any differences between the two
numbers if they exist.

The Director, OA, will ensure that a reminder is incorporated into the Call Letter
for the next President’s Budget submission to advise those who do manual counts to
institute controls against miscounts. The corrective action completion date will be
December 31, 2007.

If you have any questions concerning this memorandum, please let me know.

A ban Ss K.

Peter J. King 7


Appendix A

Comparison between PAR Estimates and Actual (FY 2006)

Estimated Workload
Units Reported in Actual 2006 +/-
No. | Office/Workload Units the FY2006 PAR Workload Units | Percentage
Secretary
1 | Orders/Notices Issued 11 15 36%
2 | Agenda Items 58 73 26%
3 | Minutes (pages) 90 85 5%
4 | Federal Register Notices 260 323 24%
5 | FOIA Request 13 28 115%
CADRS
6 | Inquiries (phone, e-mail, fax) 3,500 3,000 -14%
7 | Non Docketed Informal Complaints 650 500 -23%
8 | Docketed Informal Complaints 6 5 -16%
General Counsel
9 | Speeches/Articles Produced 15 15 0%
10 | Interagency & Int'l Group Sessions 30 30 0%
11 | Request for Information 275 275 0%
International Affairs
12 | Reports - Produced 35 35 0%
Legislative Activity
13 | Bills, Proposals - Referred 120 120 0%
Office of Administrative Law Judges
14 | Formal Proceedings (settled) 2 2 0%
15 | Formal Proceedings (dismissed) 2) 5 0%
16 | Initial Decisions Not Settled by ALU 7 a 0%
Bureau of Enforcement
17 | Enforcement Cases - Beginning of FY 2r 27 0%
18 | Enforcement Cases - Received Actions 20 21 5%
19 | Enforcement Cases - Compromised & Settled 15 14 6%
20 | Enforcement Cases - End of FY. 32 34 6%
21 | Formal Proceedings - Beginning of FY 7 4 -43%
22 | Formal Proceedings - New 4 4 0%
23 | Formal Proceedings - Completed 4 3 -25%
24 | Formal Proceedings - End of FY 4 5 25%
25 | Matters Monitoring - Beginning of FY 71 val 0%
26 | Matters Monitoring - Received 51 53 4%
27 | Matters Monitoring - Completed 49 57 16%
28 | Matters Monitoring - End of FY 73 67 -8%



Office of Administrative Law Judges

We reviewed the source documents provided by the ALJ behind workload units
consisting of (i) formal proceedings settled and dismissed, and (ii) decisions issued but not
settled by the ALJ. The units were consistently reported in both the FY 2006 PAR and the FY
2008 Congressional budget request, and tied to the source documents that we reviewed.

Bureau of Trade Analysis — Office of Service Contracts

The OIG determined that source documents from the Office of Service Contracts
reconciled to final numbers posted in the FY 2008 Congressional Budget request. However, the
data did not agree with what was reported in the PAR. The information for these workload units
are abstracted monthly from SERVCON.? However, the projections for new service agreements
understated the actual number of new service agreements and overstated the number of amended
agreements processed.

Recommendations

1. The OIG recommends that the Office of Administration ensure that estimates are updated
prior to their public release in the PAR or that the reader be advised that the performance
measures are based on nine months of actual data and three months of estimated industry
trends.

2. The OIG recommends that Office of Administration remind offices whose workload unit
statistics are dependent on manual counts to institute controls to guard against miscounts.
One such control is to have two individuals perform the count and to reconcile any
differences between the two numbers if they exist.

> SERVCON is an automated system containing service contracts filed with the FMC by select members of the
maritime industry. Ocean common carriers, either individually or through agreements with other carriers, can
negotiate and execute service contracts with one or more shippers or shipper associations. Shippers make a
commitment to provide a certain volume or portion of cargo over a fixed period of time and carriers commit to a
specified rate and a defined service level. The FMC is charged with reviewing these contracts for anticompetitive
effects. The OIG reviewed the number of service contracts filed with the agency.


Form FMC-2 (Rev. 7-89)

UNITED STATES GOVERNMENT FEDERAL MARITIME COMMISSION

Memorandum
TO : Inspector General DATE: June 28, 2007
FROM : Director of Administration

SUBJECT —: Comments on Draft Report OR-07-02 - Data Accuracy of
FMC’s FY 2006 Performance and Accountability Report

Comments on the Draft Report

The Draft Report may leave the reader with the impression that the FMC does not
produce accurate information regarding workload statistics. This is not the case.

The Office of Administration (“OA”) asks all agency programs to provide their
PAR and OMB budget submissions concurrently, in order to save effort and time on the
part of the agency’s small staff. Inasmuch as the OMB budget submission must be
submitted prior to the end of a fiscal year, OA collects and incorporates estimates of
workload statistics to meet applicable OMB budget requirements and submission
deadlines. The PAR statistics necessarily are estimates also, as is accurately pointed out
in the Draft Report.

There are many activities clustered at the end of each fiscal year, including
evaluating the agency’s progress in meeting performance goals. This process is
exceedingly important, as our budget and goals are linked in both our OMB and
President’s budget submissions. While the November 15 PAR due date might appear to
allow adequate time after the end of a fiscal year in which to develop final workload
statistics, one must take into account the process leading up to the submission of the
PAR. OMB in fact requires a draft PAR to be submitted no later than November 1.
Within that thirty-day time frame ending November 1, the FMC must undertake and
complete an internal review process of collecting, aggregating, editing and obtaining
senior management approval of the draft PAR so that any changes can be timely made
prior to submission of the PAR to OMB; additionally, certain substantive portions of the
PAR must be provided annually to the agency’s financial auditor even earlier, usually by
mid-October. (This year’s due date is October 15.) In fact and in practice, any internal
due date for compilation of final workload statistics would have to fall very close to the
actual end of the fiscal year in order to allow time for creation of the final PAR and the
PAR review process, injecting yet another demand on the program offices at a time when
significant demands have already been placed on them.


Data Accuracy of FMC’s
FY 2006 Performance and
Accountability Report (PAR)
ORO7-02

A\TIME
si

June 2007


ce:

Director, Operations

Director, BCL

Director, BTA

Director, BOE

Clay Guthridge, Office of ALIs
General Counsel

Secretary

Director, CADRS

Director, OFM


Actual 2006

= Workload Units Workload Unit 5
Reported in 2006 Reported in '08
No. | Office/Workload Units PAR Congressional ar
Bureau of Trade Analysis
Office of Agreements
29 | Carrier Agreements Filings Received 265 256 -3%
30 | Terminal Agreement Filings Received 18 28 55%
Office of Economic & Comp Analysis
31_| Set of Minutes Received 873 886 1%
32_| Quarterly Monitoring Reports Expected 231 164 -29%
Office of Service Contracts & Tariffs
33 | New Service Contracts Received 50,000 46,682 6%
34 | Contract Amendments 242,000 252,566 4%
35 | NSA Filed 414 557 35%
36_| NSA Amendments Filed 342 448 31%
37 | Active/Current Tariff - Posted 3,996 3,942 -1%
38 | Inactive/Cancel Tariff - Posted 1,856 1,902 2%
Bureau of Certification & Licensing
Transportation Intermediaries
39 | Applications - New 500 420 -16%
40 | Applications - Amended 300 227 -24%
41 | Licenses Issued - New 475 343 -28%
42 | Licenses Issued - Amended 300 169 44%
43 | Licenses - Revoked 250 278 11%
44 | Licenses - Cancelled 55 63 15%
Passenger Vessels & Information Processing
Applications for Performance Certificates -
45 | Received 30 29 3%
Applications for Performance Certificates -
46 | Processed 30 29 -3%
Applications for Casualty Certificates -
47 | Received 30 30 0
Applications for Casualty Certificates -
48 | Processed 34 24 -29%



Although we identified differences in both directions (i.e., overstating and understating program
outcomes), in all cases the differences occurred because the agency combined three-month
estimates with data from nine-months of actual performance.

Table 1, below, summarizes the outcome of our comparison of workload units in the FY
2006 PAR and the FY 2008 Congressional Budget Request.

Table 1. Summary Statistics
2006 PAR-Reported Workload Units vs. 2006 Actuals

Number of PAR Discrepancies
Reporting Unit Workload Units | Between Estimates
Sampled & Actuals

Secretary 5 5
Consumer Affairs & Dispute Resolution Services 3 3
Administrative Law Judges 3 0
General Counsel 5 0
Enforcement 12 9
Trade Analysis 10 10
Certification and Licensing 10 9

TOTAL 48 36

A detailed comparison between the individual data elements presented in the PAR and
actual outputs presented in the FY 2008 Congressional Budget submission is made in
Appendix A to this report.

Although the table above indicates that discrepancies were seen in 75 percent of the
measures we reviewed, a closer examination of Appendix A shows wide variation in the
magnitude of these differences. Some of the estimates missed actual totals by only a few
percentage points, while other estimates were significantly off their mark. For example, the
Bureau of Enforcement estimated in July that it would have 32 enforcement cases pending at year
end. In the FY 2008 Budget Submission, the Bureau reported that it had 34 cases pending, or a 6
percent difference. Similarly, the Bureau of Trade Analysis’s Office of Agreements estimated
that it would receive 265 Carrier Agreement Filings, but reported 256 in the FY 2008 Budget
Submission — a 3 percent decline.

On the other hand, other estimates differed from actual totals by larger margins. For
example, CADRS overestimated informal complaints by 23 percent; the Bureau of Trade
Analysis’s Office of Economic and Competition Analysis “Quarterly Monitoring Reports” were
overestimated by 29 percent; BCL overestimated the number of amended licenses it processed by
44 percent; and the actual number of Freedom of Information (FOIA) requests was 115% higher
than OS originally estimated.


The agency begins to compile data for the PAR report in July, nine months into the fiscal
year. At the same time, the agency also requests that program offices provide budget narratives
and resource requests for future budgets. Both the PAR and budget narratives present identical
performance information for the current year. Rather than making requests to program offices at
different times for the same information, OA consolidated the requests into one, with a reporting
deadline in July of each year. While reducing overlap, this common deadline has required
program staff to estimate performance for the last three months of the fiscal year, based on past
experience and current maritime industry trends.

The PAR is due to the Office of Management and Budget (OMB) on November 15 of
each year, giving the agency 45 days after the close of the fiscal year to update its estimates from
the preceding July and to prepare the document. If these estimates are not updated, the agency
may, depending on the accuracy of its forecasting methodology and the impact of industry forces
outside its control, publicly report inaccurate performance data.

Data Accuracy

Based on discussions with senior agency staff and OA managers, the OIG learned that the
agency did not update PAR performance data estimates before publicly releasing the information.
Nor did it indicate in the PAR that the performance data was based on a combination of actual
and estimated information. As a result, a significant amount of performance data appearing in
the PAR is inaccurate.

The OIG sought to compare estimated data reported in the PAR with actual FY 2006
performance information to assess the extent to which randomly selected estimates deviated from
actual performance data and to illustrate the variation between estimates and actuals. As our
benchmark, we used actual FY 2006 performance information contained in the FY 2008
Congressional Budget submission.

The FMC’s performance management system includes both strategic goals and
performance goals and measures. The strategic goals taken together represent the FMC’s
mission. The performance goals focus on outcomes which contribute to the agency achieving its
strategic goals. The performance measures associated with each performance goal provide
benchmarks for measuring how effectively the FMC is achieving its goals. Performance data

- provided by program offices provides a barometer to assess how well the agency is meeting its
goals.

To test the accuracy of information reported in the FY 2006 PAR, the OIG selected 48
workload units from seven FMC program areas for review.’ We then compared these workload
units to the corresponding data elements in the FY 2008 budget submission. We identified 36
discrepancies (75 percent) when comparing identical workload units in the two documents.

' Workload units are output statistics used by the FMC to quantify and assess performance. For example, workload
statistics include number of (i) licenses processed, (ii) service contracts received, and (iii) enforcement actions
brought against alleged violators. The OIG selected all workload units in the FY 2006 PAR for review.

aQie


FEDERAL MARITIME COMMISSION
800 North Capitol Street, N.W.
Washington, DC 20573

June 29, 2007

Office of Inspector General
Tel: (202) 523-5863
Fax: — (202) 566-0043
E-mail: oig@fme.gov

TO: Peter J. King, Director
Office of Administration

—_—
FROM: Adam R. Trzeciak 7p
Inspector General : 3 ~

SUBJECT: Data Accuracy of FMC’s FY 2006 Performance and Accountability Report
(OR-07-02)

The Office of Inspector General (OIG) has completed a limited review of data accuracy in
the FY 2006 Performance and Accountability Report (PAR). The objectives of this review were
to determine, on a sample basis, whether (i) information externally reported in the FY 2006 PAR
is accurate, and (ii) processes used to compile the data are reliable. To meet our objectives, we
interviewed and/or obtained relevant information from the Office of the Secretary (OS), Office of
Consumer Affairs and Dispute Resolution Services (CADRS); Office of Administration (OA);
Office of the General Counsel (OGC); Office of Administrative Law Judges (ALJ); and the
Bureaus of Enforcement (BOE), Trade Analysis (BTA) and Certification and Licensing (BCL).

Background

The PAR provides program and financial information that enables the President,
Congress and the public to assess Agency performance relative to its mission and resources. The
report is divided into three sections (chapters) and six appendices. Chapter One, Management’s
Discussion and Analysis, provides an overview of the agency, its missions and responsibilities, as
well as its financial performance. Chapter Two, Program Performance, presents information on
the performance of the FMC, the FMC’s performance measurement system, resource
requirements to meet strategic goals and a summary of performance relative to specific program
goals. Chapter Three, Auditor's Report and Financial Statements, contains the auditor’s opinion
of the agency’s financial statements and the statements themselves.