APPENDIX 3
Page 1 of 1

MAJOR CONTRIBUTORS TO THIS REPORT

OFFICE OF INSPECTOR GENERAL

Benny W. Lee, Director, Program Audits
Linda G. Darby, Audit Manager

Joseph K. Eom, Auditor-in-Charge
Garrett W.J. Gee, Auditor

Gerard C. Vroomman, Auditor

OIG-00-061 OTS’ DATA RELIABILITY AND VALIDITY Page 17
UNDER THE RESULTS ACT


limitations in their performance plan. OTS will also attempt to find
better ways of collecting data through their outreach process.

Customer Satisfaction Survey

One performance measure OTS uses to assess its goal related to
contributing to a safe and sound industry is based on thrift responses
to an OTS survey (i.e., measure 6, Table 1, page 3). We found that
reported performance has been based on a 60 percent response rate.
By not assessing the significance of the 40 percent non-responses,
OTS’ reported performance might have introduced a sampling bias,
which in turn, resulted in a misleading measure of actual
performance.

We found that of the 1,316 surveys distributed to the thrift
institutions, OTS received 801 completed surveys or about 60
percent. OTS used the 801 surveys as its population. However,
non-responses create a potential for bias that cannot be overlooked
after the data has been collected. The effects of non-response bias
can be substantial because a portion of the population is under-
represented. The impact of non-responses is analogous to the impact
of omissions in the sampling framework, whereby a proportion of the
population has been omitted from the sample.

In light of the 40 percent non-response rate, we suggest that OTS
assess whether there is a potential sampling bias with the reported
performance under this measure. We also suggest that OTS assess
the need for additional disclosures in the Performance Report as to
the survey response rate.

In response to our draft report, OTS management acknowledges there
may be a potential bias in light of the 40 percent non-response rate.
However, they consider a 60 percent response rate for such a survey
to be very good. OTS will provide additional disclosures in their
performance report related to other OTS measures that reinforce such
customer satisfaction data.

OIG-00-061 OTS’ DATA RELIABILITY AND VALIDITY Page 11
UNDER THE RESULTS ACT


Management System (CSMS). Using these systems, OTS was able
to capture, store, and retrieve the performance data collected when
needed.

To better address Results Act requirements, OTS has taken steps to
strengthen the review process. Among other things, OTS recently
set up a new Internal Review Office and hired a Director. The new
office is responsible for ensuring OTS’ effectiveness in complying
with all applicable laws including the Results Act. In addition, the
recently hired Director is responsible for reviewing all of OTS’
financial management and data collection systems for accuracy,
planning, directing, and overseeing the development of internal
control and risk management policies, and ensuring that OTS
operates with strong internal controls.

Objective, Scope, and Methodology

Our audit objective was to determine whether OTS systems and
control processes are in place to reasonably ensure the reliability and
validity of its performance measurement data. Table 1 shows OTS’
17 performance measures as of February 1999, and the measures
included in our review. In selecting OTS performance measures to
be reviewed, we emphasized those performance measures related to
OTS strategic goals of ensuring thrift’s safety and soundness (S&S)
and fair access to financial services for all Americans.

OIG-00-061 OTS’ DATA RELIABILITY AND VALIDITY Page 2
UNDER THE RESULTS ACT


APPENDIX 4
Page 1 of 1

REPORT DISTRIBUTION

U.S. Department of the Treasury
Office of the Assistant Secretary for Financial Institutions
Office of Strategic Planning and Evaluations, Departmental Offices
Office of Accounting and Internal Control, Departmental Offices
Office of Budget, Financial Management Division, Departmental Offices
Office of Thrift Supervision

Regional Director, West Region
Regional Director, Northeast Region

Office of Management and Budget

OIG Budget Examiner

OIG-00-061 OTS’ DATA RELIABILITY AND VALIDITY Page 18
UNDER THE RESULTS ACT


APPENDIX 2
Page 2 of 3

MANAGEMENT RESPONSE

Mr. Dennis S. Schindel
Page 2

surveys would not be included in calculating the performance measure due to a time
lag, or selecting a different starting point to capture 1 year’s worth of data for
customer survey responses.

We will select a different starting point to capture 1 year’s worth of data for
customer survey purposes.

Other Matters -- Examination Schedules

An enhancement to this measure might be to incorporate timeliness into the
measure directly given that OTS’s EDS currently captures examination data, such
as the beginning and completion date of examinations. Another possible option
might be to supplement the performance report with a cautionary note about the
measure not reflecting the number of scheduled examinations starting on schedule.

We will incorporate timeliness into the performance measure and disclose the
related information in future reports.

Other Matters - Community Affairs Program (CAP)

We believe as currently presented that the community affairs performance
measure has limitations that need to be disclosed in the performance plan. These
disclosures would cover the time lag and the existence of external factors affecting
the reported outcomes from OTS CAP efforts. If OTS must rely on additional data
from the industry, we suggest that OTS assess the possible reporting burden
according to the Paper Reduction Act of 1980, as amended in 1995.

Your report clearly outlines the difficulties in establishing a performance
measure for the CAP. We will continue to refine the performance measure and disclose
the limitations in our performance plan.

‘We do not wish to increase the industry’s burden by mandating data collection,
however, we will attempt to find better ways of collecting data through our examination
outreach process.

Other Matters — Customer Satisfaction Survey

In light of the 40 percent non-response rate, we suggest that the OTS assess
whether there is a potential sampling bias with the reported performance under
this measure. We also suggest that the OTS assess the need for additional
disclosures in the Performance Report as to the survey response rate.

OIG-00-061 OTS’ DATA RELIABILITY AND VALIDITY Page 15
UNDER THE RESULTS ACT


the survey is to obtain feedback from the thrift on OTS’ performance
during its on-site examination. OTS officials told us that surveys are
usually sent 4 to 6 weeks after the report has been transmitted to the
thrifts. It is not mandatory for the thrifts to respond to the survey.
However, for those that do respond to the survey, OTS enters the
results into the CSMS. The CSMS is an automated system that
captures and tabulates the survey responses.

The CSMS accurately identified the number of examinations
conducted during the year, captured the survey responses received
from the thrifts, and tabulated the total number of positive responses
from review of the surveys. However, some thrift surveys were not
received in time to be included in the tabulation of the performance
report. Surveys were usually mailed several weeks after OTS issued
the examination reports. For those exams transmitted at year-end,
the surveys were not actually sent until some time in January or
February of the following year. In February 1999, when OTS
tabulated the final survey count, the CSMS showed the 2 regions had
received 189 surveys. However, the 2 regions received 31 additional
surveys after the tabulation date. As a result, 14 percent of the
surveys received were not included in the final data.

Recommendations
Recommendation 1-1:

The Director of OTS should increase regional oversight to ensure
performance data is reported consistently (e.g., EOP performance
measure) especially when data is collected and reported manually.

nm n I nt

OTS management concurred with the OIG recommendation. EOP
data will be reported separately in future reports. In addition, the
director of Internal Review will independently review financial
management and data collection systems for accuracy, planning,
directing and overseeing the development of internal controls prior to
report submission.

The OIG believes OTS' actions will address the issues surrounding
this recommendation.

OIG-00-061 OTS’ DATA RELIABILITY AND VALIDITY Page 7
UNDER THE RESULTS ACT



We reviewed OTS’ systems and processes for collecting data for 13
of 17 performance measures. Five of the 13 measures did not have
data because they were not measures in 1998. Consequently, for
these five measures, we discussed with OTS officials how they were
collecting data for these measures. For the eight measures carried
forward from 1998, we tested the reliability and validity of
performance data reported in the performance plan. We tested the
data for consistency, accuracy, and completeness. We also
determined whether OTS identified and adequately disclosed potential
data limitations for these eight measures in its performance plan and
reports.

To determine the adequacy of OTS systems and controls used for
capturing and tracking the performance measurement data, we
identified the necessary data and source documents for each
measure’s data element and judgmentally sampled the data to verify
the data’s accuracy. We also reviewed the data collection system and
control processes for each measure to ensure that errors did not affect
data reliability and validity.

We performed audit work at 2 of OTS' 5 regional offices. We did
not project our observations nationwide. It should be noted,
however, all five OTS regional offices would have been reporting the
same performance measures and the component data elements, and
all five regions would have been subject to the same national
reliability and validity reporting standards. We also did not test the
data that OTS obtained from external sources such as the Federal
Financial Institutions and Examination Council. Although we did not
evaluate the appropriateness of the measures that OTS was using to
gauge performance, we identified three related issues warranting
OTS’ attention.

Our fieldwork was performed between March and June 1999. We
performed audit work at OTS headquarters in Washington D.C., and
OTS regional offices in San Francisco, California and Jersey City,
New Jersey. We conducted this audit in accordance with
Government Auditing Standards issued by the Comptroller General of
the United States and included such audit tests as were determined
necessary.

OIG-00-061 OTS’ DATA RELIABILITY AND VALIDITY Page 4
UNDER THE RESULTS ACT


sec eT TE aS CS A ESSE cacrnRCOEE SESE
Table 1
OTS Performance Measures'
Strategic Goal/Performance Measure Included in Our
Review
1. Percentage of thrifts that received each type of examination. x
2. Percentage of thrifts rated “4” or “5” during the measure period that have an x
enforcement action, supervisory action, or waiver in place.
3. Percentage of examination reports deemed generally consistent with OTS x
examination policies and procedures.
4. Percentage of thrifts receiving Year 2000 (Y2K) conversion reviews.
5. Percentage of thrifts with a satisfactory Y2K rating.
6. Percentage of thrifts with “satisfactory” or “better than satisfactory” ratings x
responding to an OTS survey.
7. Percentage of thrifts with a “4” or “5” compliance rating that have an x
enforcement action, supervisory action, or waiver in place.
8. Percentage of outreach, training, or partnership building events held. x
9. Number of partnerships formed, activities commenced, or investment made as a
result of OTS’ outreach, training, or partnership building activities. x
10. Percentage of thrifts with less than satisfactory CRA rating that OTS offers one-
to-one community outreach assistance to within 60 days of the completed exam. x
11. Percentage of community development related regulatory barriers addressed. x
12. Percentage of speeches or presentations given by OTS senior management on
community development related topics.
Maintain competitiveness of the thrift industry to ensure its safety and soundness.
13. Ensure that 100 percent of thrifts are at least “adequately capitalized” or are x
under a Prompt Corrective Action after becoming undercapitalized.
14. Percentage of outreach project completed.
Conduct operations efficiently 10 keep regulatory burdens.on the thrift svstem at the
minimum level consistent with effective supervision.
15. Percentage of applications meeting processing timeframes. x
16. Percentage of OTS regulatory reinvention projects completed in question and
answer plain language format.
Provi: ‘ional i | major with which OTS i
ve x nsivs
17. Percentage of service plans that met their standards. x
Source: OTS Performance Plan, as of February 11, 1999
' See Appendix 1 for the full text of the 13 measures we reviewed.
OIG-00-061 OTS’ DATA RELIABILITY AND VALIDITY Page 3

UNDER THE RESULTS ACT



APPENDIX 2
Page 3 of 3

MANAGEMENT RESPONSE

Mr. Dennis S. Schindel
Page 3

OTS acknowledges there may be a potential bias in light of the 40 percent non-
response rate. However, a 60% response rate for any passive survey is generally
considered to be very good. OTS remains confident in and pleased with the high level of
satisfied responses.

We will provide additional disclosures in the performance report related to other
OTS measures that reinforce customer satisfaction data obtained through town meetings,
the examination outreach program, the Ombudsman’s office, contacts with trade groups
and direct meetings with me and other senior level management.

Our staff appreciated the courtesies extended by your staff. We look forward to

cooperation in future audits.
Please contact me or Lynnwood Campbell, 202-906-5713, if you or your staff

have additional questions.

Sincerely,
Slew Ne ito

Ellen Seidman
Director

Enclosure

OIG-00-061 OTS’ DATA RELIABILITY AND VALIDITY Page 16

UNDER THE RESULTS ACT


found, however, that the West and the Northeast regions conducted
post-transmittal reviews, whereas, according to Headquarters, the
Southeast, Midwest, and Central regions conducted reviews before
the ROE was transmitted.

We believe the inconsistency of when the ROEs were reviewed exists
mainly due to lack of OTS Headquarters oversight of the
performance measure. By performing reviews after the ROEs were
sent to the thrifts, the Northeast and the West regions were unable to
meet the intent of this measure, which was to ensure examination
reports were consistent with OTS examination policies and
procedures before they were given to the thrifts.

The data reported for the EOP measure was also inaccurate. OTS
reported that 896 EOP examination reports were reviewed during
1998, including 286 for the 2 regions we visited. Of the 286, we
found that OTS actually completed 206 EOP reviews. The numbers
were overstated because one region included QA reviews with EOP
reviews when reporting performance results to Headquarters.

According to OTS Headquarters personnel, the QA reviews should
not have been combined with EOP reviews. The region submitting
both types of reviews assumed that since the same regional group
performed both types of reviews, OTS Headquarters would want
both data. The document sent to Headquarters explicitly indicated
that the total reflected both EOP and QA reviews.

The consistency and accuracy issues for this performance measure
could be resolved by better oversight on the type of EOP data that
should be submitted, especially since this data is collected and
reported manually by the regions.

Completeness in Reporting Customer Satisfaction Survey Responses

We found that the number of customer satisfaction survey responses
reported by OTS (i.e., measure 6, Tables 1, page 3) does not include
responses to surveys sent out at the end of the year. Because these
surveys were not returned to OTS before the annual performance
report was compiled, OTS’ reported performance, for example, in 2
regions omitted responses from at least 14 percent of those thrifts
sent a survey.

OTS sends each thrift a customer satisfaction survey after a S&S, a
compliance, or an information systems examination. The purpose of

OIG-00-061 OTS’ DATA RELIABILITY AND VALIDITY Page 6
UNDER THE RESULTS ACT


Office of Thrift Supervision's
Data Reliability and Validity
Under the Results Act

OIG-00-061 March 06, 2000

Office of Inspector General

United States Department of the Treasury



measure as meeting the implied performance given that both events
occurred in the same calendar year. Although OTS has information
systems and controls in place to identify these timeliness situations,
this situation would not be reflected as an exception in the
performance report or the measure.

Another possible misinterpretation might be OTS’ performance in
meeting the examination frequency requirements under the Federal
Deposit Insurance Corporation Improvement Act of 1991 (FDICIA).
Again, the reference to scheduled examinations in the measure’s
denominator could be mistakenly associated to FDICIA given that
safety and soundness examinations are subject to legislatively
mandated timeframes.

An enhancement to this measure might be to incorporate timeliness
into the measure directly given that OTS’ EDS currently captures
examination data, such as the beginning and completion dates of
examinations. Another possible option might be to supplement the
performance report with a cautionary note about the measure not
reflecting the number of scheduled examinations starting on schedule.

In response to our draft report, OTS management indicated that they
will incorporate timeliness into the performance measure and disclose
the related information in future reports.

Community Affairs

As a refinement to the CY 1998 Performance Plan, OTS added five
Community Affairs Program (CAP) performance measures in 1999.
Because there was no data for us to review and test for these
measures, we only discussed with OTS officials how they plan to
collect data for these measures.

One of the five measures is intended to measure outcomes rather than
outputs. The outcome-oriented measure (i.e., measure 9, Table 1,
page 3) attempts to identify the number of partnerships formed (i.e.,
partnerships may involve either OTS or the thrift and a community
based organization), activities commenced, or investments made as a
result of OTS’ outreach, training, or partnership building activities.
As of July 1999, OTS had yet to find ways to isolate OTS’
contribution to the program’s outcomes from external factors that
also affect program outcomes. In fact, OTS noted in its
“Methodology Used to Verify Measured Values” section of the CY

OIG-00-061 OTS’ DATA RELIABILITY AND VALIDITY Page 9
UNDER THE RESULTS ACT


Recommendation 2-1:

The Director of OTS should refine the customer satisfaction survey
performance measure by either disclosing data limitations in its
performance report since all customer satisfaction surveys would not
be included in calculating the performance measure due to a time lag,
or selecting a different starting point to capture 1 year’s worth of data
for customer survey responses.

Management Response and OIG Comment

OTS management concurred with the OIG recommendation. OTS
will select a different starting point to capture annual data for
customer survey purposes.

The OIG believes OTS' actions will address the issues surrounding
this recommendation.

Other Matters for Consideration

During our audit, we observed three related issues that fell outside
our original audit objective. Nonetheless, we are bringing these to
OTS’ attention with the belief that further assessment of these issues
will improve the usefulness of the associated performance measures.
These issues relate to the need for added disclosures and/or
reassessment of how the measure might be derived.

Examination Schedules

To contribute to a safe and sound thrift industry, one OTS goal is to
maintain and enhance a risk-focused approach to thrift supervision.
One of the performance measures for this goal is “the number of
thrifts that received this type of examination divided by the number
of thrifts scheduled to receive this type of examination” (i.e.,
measure 1, Table 1, page 3). We believe the usefulness of this
measure could be enhanced with additional disclosures or by revising
it.

Given how this measure is derived, we believe it would be useful to
note that the measure does not reflect examination timeliness.
Because of the reference to scheduled examinations in the measure’s
denominator, the results might be mistaken as specifically meeting an
examination schedule. For example, an examination scheduled for
February but not started until November would be reflected by this

OIG-00-061 OTS’ DATA RELIABILITY AND VALIDITY Page 8
UNDER THE RESULTS ACT


Audit Results

For most of the performance measures, we found OTS had
reasonably adequate systems and processes in place to ensure that the
underlying data was reliable and valid. However with four
performance measures, we found (1) consistency and accuracy
problems with the data reported for conducting Examination
Outreach Program (EOP) reviews, and (2) a completeness problem in
reporting data for customer satisfaction surveys. We also found
issues relating to the clarity of the performance measures on
examination schedules, outcome-oriented community affairs
program, and customer satisfaction survey data.

Consistency and Accuracy of EOP Data

We found consistency and accuracy issues with the EOP review
performance measure data (i.e., measure 3, Table 1, page 3). In
terms of consistency, we found the regions conducted the EOP
reviews of the examination reports at different points in the review
process. Two regions conducted post-transmittal reviews while the
other three regions conducted EOP reviews before the Report Of
Examinations (ROE) were sent out.” In terms of accuracy, 1 of the 2
regions we visited included the quality assurance (QA) reviews when
reporting the EOP performance results to Headquarters.

OTS policy states that examination reports should effectively
communicate S&S concerns and that OTS' supervisory efforts should
be responsive to thrift management’s concerns. To assess these
requirements, OTS developed a performance measure that focuses on
EOP examination reviews. EOP reviews determine whether
examinations were generally consistent with OTS policies and
procedures. The regional offices conduct reviews of at least 20
percent of the ROEs annually. As indicated in the CY 1999
Performance Plan, each regional office is to conduct EOP reviews on
S&S, trust, and information systems examination reports. Each
region manually tracks the number of EOP reviews performed and
reports that number to Headquarters quarterly.

The performance measure is supposed to reflect EOP reviews
conducted before the ROEs are transmitted to the institution. We

? Post-transmittal reviews are defined as reviews of examinations performed by the regional EOP group after
the ROEs have been sent to the thrift. Pre-transmittal reviews are performed before the ROEs are sent to the
thrift.

OIG-00-061 OTS’ DATA RELIABILITY AND VALIDITY Page 5
UNDER THE RESULTS ACT


Overview

Background

This report summarizes our audit of the Office of Thrift
Supervision’s (OTS) systems and controls to ensure that performance
information reported under the Government Performance and Results
Act of 1993 (the Results Act) is reliable and valid. This audit is one
in a series of planned audits designed to assist Treasury bureaus in
meeting the requirements of the Results Act. We recognize that OTS
is continuing to refine its performance measures in an evolving
process to better meet the Results Act goals for the Year 2000. As
such, the audit conclusions presented in this report are intended to
help OTS develop a systematic process for ensuring that performance
data are reliable and valid when the March 2000 reporting
requirement becomes effective.

The Results Act holds Federal agencies systematically accountable
for achieving program results. Specifically, Federal managers are
required to set strategic goals, measure performance, and report on
their progress. Federal agencies were required to develop strategic
goals before Fiscal Year (FY) 1998 and prepare annual plans setting
performance goals beginning with FY 1999. They are also required
to report annually on actual performance compared to goals with the
first report due March 2000. Finally, agencies are to report on
actual program performance results beginning in FY 2002.

A fundamental underpinning of the Results Act is that the
performance information is reliable and valid. Additionally, since
errors may occur at any point in the collection, maintenance,
processing, and reporting of data, an agency should have procedures
in place to address data reliability and validity at each step of the
process. To ensure completeness, accuracy, and consistency of
performance data, an agency is to implement verification and
validation procedures, identify data limitations, and disclose in the
performance plans and reports how the limitations may affect the
credibility of the performance information.

OTS uses data generated by both manual and automated systems to
collect and report on the performance measures. Most systems were
already automated and integrated before the Results Act. For
instance, OTS' established automated systems include the
Examination Data System (EDS), Regulatory Action Data (RAD)
System, Thrift Financial Report (TFR), and the Customer Service

OIG-00-061

OTS’ DATA RELIABILITY AND VALIDITY Page 1
UNDER THE RESULTS ACT


APPENDIX 1
Page 1 of 2

PERFORMANCE MEASURES REVIEWED

The full text of the 13 performance measures we reviewed is as
follows.

1,

The number of thrifts that received each type of examination
divided by the number of thrifts scheduled to receive this type of
examination.

The number of thrift institutions rated “4” or “5” during the
measurement period that have an enforcement action, supervisory
action or waiver in place or that have received a decision for
waiver, enforcement action or supervisory action divided by the
total number of thrift institutions that were rated “4” or “5” with
examinations that were completed more than 60 days ago.

. The number of examination reports deemed generally consistent

with OTS examination policies and procedures divided by the
number of examination reports reviewed.

The number of times thrifts rated the value of the examination
process as being “satisfactory” or “better than satisfactory”
divided by the total number of thrifts that responded to the
examination survey.

Number of thrift institutions with a compliance rating of “4” or
“5” during the measurement period and an enforcement action,
supervisory action or waiver in place, or that have received a
decision for waiver, enforcement action or supervisory action
divided by the total number of thrift institutions rated “4” or “5”
with examinations completed more than 60 days ago.

The number of outreach, training, or partnership building events
held (or participated in) divided by the number of events planned
(90 percent); the number of new areas or groups reached divided
by the number targeted (85 percent); and the number of thrift
participants divided by the number targeted (50 percent).

The number of new partnerships formed, activities/programs
commenced or investments made in 1999 as a result of OTS’
outreach, training, or partnership building activities conducted.

OIG-00-061 OTS’ DATA RELIABILITY AND VALIDITY Page 12
UNDER THE RESULTS ACT


APPENDIX 2

Page 1 of 3
MANAGEMENT RESPONSE
Office of Thrift Supervision Ellen Seidman
Department of the Treasury Director

1700 G Street, N.W., Washington, D.C. 20552 + (202) 906-6590

January 12, 2000

Mr. Dennis S. Schindel

Assistant Inspector General for Audit
Department of the Treasury

Office of Inspector General
Washington, D.C. 20220

Dear Mr. Schindel:

We are pleased to review and comment on the Inspector General’s draft report
dated December 22, 1999 entitled, Draft Report on Audit of Office of Thrift
Supervision’s Data reliability and Validity Under the Results Act.

The Office of Thrift Supervision supports the recommendations and will comply
with the other matters for consideration.

Recommendation 1-1

Increase regional oversight to ensure performance data is reported
consistently (eg. EOP performance measure) especially when data is collected and
reported manually.

EOP data will be reported separately in future reports. The director of Internal
Review will independently review financial management and data collection systems for
accuracy, planning, directing and overseeing the development of internal controls prior to
report submission.

We do not believe that there is an inconsistency of when EOP reviews were
performed. The decision to perform post-transmittal reviews of ROEs versus pre-
transmittal reviews rests with the individual regions. Post-transmittal reviews and pre-
transmittal reviews have provided us with the necessary assurances that examination
reports are consistent with OTS examination policies and procedures. We will refine the
performance measure to reflect regional choices.

Recommendation 1-2

Refine the customer satisfaction survey performance measure by either
disclosing data limitations in its performance report since alll customer satisfaction

OIG-00-061

OTS’ DATA RELIABILITY AND VALIDITY Page 14
UNDER THE RESULTS ACT



Page 2

to address the three related issues. The complete text of the
response is presented at the end of the report.

We appreciate the courtesies and cooperation provided to our
staff during the audit. Should you have any questions or
require any further assistance, please call me at

(202) 927-5400, or a member of your staff may contact

Benny W. Lee, Director, Program Audits, in San Francisco,
California at (415) 977-8810 ext. 222.

Attachment
cc: Lynnwood Campbell

Director, Internal Review
Office of Thrift Supervision


DEPARTMENT OF THE TREASURY
WASHINGTON, D.c. 20220

OFFICE OF

INSPECTOR GENERAL MAR 6 2000

MEMORANDUM FOR ELLEN SEIDMAN, DIRECTOR

OFFICE OF THRIFT SUPERVISION
FROM: Dennis S. Schindel
Assistant Inspector General’ for Audit

SUBJECT: Office of Thrift Supervision's Data Reliability
and Validity Under the Results Act

This memorandum transmits the final report on the Office of
the Inspector General's (OIG) audit of the Office of Thrift
Supervision's (OTS) systems and controls to ensure that
performance information reported under the Government
Performance and Results Act of 1993 (the Results Act) is
reliable and valid.

During our audit, we identified two weaknesses relating to the
validity and accuracy of the data used in performance
reporting. Specifically, we found (1) inconsistent and
inaccurate data reported for conducting Examination Outreach
Program (EOP) reviews and (2) incomplete data reported from
the customer satisfaction surveys. To better ensure data
reliability and validity, we recommend that OTS (1) increase
oversight of performance data reporting and (2) refine the
customer satisfaction survey measure and provide additional
informational disclosures.

We also observed three related issues that fell outside our
original audit objective. Nonetheless, we are bringing these
to OTS' attention with the belief that further assessment of
these issues will improve the usefulness of the associated
performance measures. These issues relate to the need for
added disclosures and/or reassessment of how measures might be
derived. The three measures include examination schedules,
the outcome-oriented community affairs program, and customer
survey data.

In response to our draft report, OCC concurred with the two
recommendations and proposed suitable corrective action

to address the three related issues. The complete text of the
response is presented at the end of the report.


1999 Performance Plan that data could not be collected to support
this performance measure.

Because data cannot be systematically tracked, OTS plans to provide
anecdotal evidence on whether its CAP activities resulted in new
partnerships formed. OTS community affairs staff told us they plan
to solicit periodic feedback from the thrift industry. Additionally,
OTS plans to evaluate ways to better collect this data through the
examination outreach process.

OTS officials cited three reasons why it might be difficult to collect
data for this measure. First, a precise cause and effect relationship
does not exist between OTS’ programs and the desired outcomes.
OTS officials told us that many factors lead to the formation of
partnerships and some are outside of OTS’ control. They further
said that the outcome-oriented measure creates data limitation issues
because greater dependence is placed on external data sources, such
as the thrift’s community groups or other regulatory agencies.

Second, there is usually a time lag from when OTS provides
outreach, training, and partnership building activities to thrifts to
when the partnerships are formed or activities commenced. OTS
officials told us that the time lag could be as much as a year. As
such, they said it would be difficult to measure the immediate
benefits of OTS’ community affairs activities.

Third, OTS officials indicated that the results-oriented performance
measure might place an administrative burden on OTS and the
industry by having them collect the outcome data. They further went
on to say that requiring OTS and industry personnel to track and
report on all partnerships formed, programs commenced, or
investments made could be time and labor intensive.

We commend OTS for attempting to establish an outcome-oriented
measure. However, we believe as currently presented, this
performance measure has limitations that need to be disclosed in the
performance plan. These disclosures would cover the time lag and
the existence of external factors affecting the reported outcomes from
OTS CAP efforts. If OTS must rely on additional data from the
industry, we suggest that OTS assess the possible reporting burden
according to the Paper Reduction Act of 1980, as amended in 1995.

In response to our draft report, OTS management indicated that they
will continue to refine this performance measure and disclose the

OIG-00-061 OTS’ DATA RELIABILITY AND VALIDITY Page 10
UNDER THE RESULTS ACT


APPENDIX 1
Page 2 of 2

PERFORMANCE MEASURES REVIEWED

10.

11.

12.

13.

. The percentage of thrifts with less than satisfactory Community

Reinvestment Act (CRA) rating assigned in 1999 that OTS offers
or provided one-on-one community development related outreach
and technical assistance to within 60 days of completion of the
examination.

The number of community development related regulatory
barriers addressed divided by the number identified (investment
authority of CRA).

The number of speeches or presentations given by senior
management on community development related topics divided by
the number planned.

The number of OTS regulated thrift institutions that are at least
“adequately capitalized” or are under Prompt Corrective Action
Directive or are recapitalized to at least the “adequately
capitalized” level within 150 days of becoming undercapitalized
divided by the total number of OTS regulated thrift institutions
minus those that have been undercapitalized for less than 150
days.

The number of applications meeting the processing timeframes
except when an application contains an issue or law or policy.

The number of service plans that met their standards divided by
the number of service plans. (The goal is that all service plans
will meet their standards at least 80 percent of the time.)

OIG-00-061 OTS’ DATA RELIABILITY AND VALIDITY Page 13
UNDER THE RESULTS ACT