Foreclosure Review: Lessons Learned Could Enhance Continuing Reviews and Activities under Amended Consent Orders
Published: 2013-03-26T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Since April 2011, OCC and the Federal Reserve had been overseeing the foreclosure review, a requirement of consent orders entered into by 14 mortgage servicers. This undertaking involved a review of loan files by thirdparty consultants to identify errors in servicing and foreclosure practices. More than 4 million borrowers were eligible for reviews. In January 2013, the regulators announced agreements with 11 of the servicers that replaced the reviews with a broad payment process to compensate borrowers in a more timely manner. Reviews continue for three remaining servicers. GAO has been reviewing various aspects of the foreclosure review process. This report addresses: (1) challenges to the achievement of the goals of the foreclosure review, (2) transparency of the process, and (3) lessons that could be useful for carrying out activities under the amended consent orders and continuing reviews. GAO analyzed third-party consultants' sampling plans, reviewed regulatory guidance and other documents, and interviewed representatives of third-party consultants and law firms, consumer groups, and regulators.
section_title: What GAO Found
paragraphs: Complexity of the reviews, overly broad guidance, and limited monitoring for consistency impeded the ability of the Office of the Comptroller of the Currency (OCC) and the Board of Governors of the Federal Reserve System (Federal Reserve) to achieve the goals of the foreclosure review--to identify as many harmed borrowers as possible and ensure similar results for similarly situated borrowers. Regulators said that coordinating among foreclosure review participants was challenging, and consultants said that the reviews were complex. In spite of regulators' steps to foster consistency, broad guidance and limited monitoring reduced the potential usefulness of data from consultants and increased risks of inconsistency. For example, GAO found that guidance was revised throughout the process, resulting in delays. Other guidance did not specify key sampling parameters for the file reviews and regulators lacked objective monitoring measures, resulting in difficulty assessing the extent of borrower harm. Good planning and collecting objective data during monitoring provide a basis for making sound conclusions. Without using objective measures to assess sampling or comparing review methods across consultants, regulators' ability to monitor progress toward achievement of foreclosure review goals was hindered.
Although regulators released more information than is typically associated with consent orders, limited communication with borrowers and the public adversely impacted transparency and public confidence. To promote transparency, regulators released redacted engagement letters and guidance on remediation. In addition, OCC released two interim progress reports. However, some stakeholders perceived gaps in key information and wanted more detailed information about how the reviews were carried out. Regulators stated they considered publicly releasing additional information, but expressed concerns that releasing detailed information risked disclosure of confidential or proprietary information. Further, borrowers who requested reviews experienced gaps in communication. For example, borrowers who submitted requests when the submission period opened waited nearly a year before receiving an update.
The foreclosure review activities to date highlight key lessons related to planning, monitoring, and communication. GAO's prior work shows that assessing and using lessons learned from previous experience can benefit the planning of future activities. The foreclosure review produced lessons in advanced planning and establishing mechanisms to monitor progress toward goals. Without assessing and applying relevant lessons learned, regulators might not address challenges in the continuing reviews or similar challenges in activities under the amended consent orders. In particular, regulators announced the agreements that led to the amended consent orders without a clear communication strategy. Although the regulators plan to release reports on the results of the amended consent orders and the continuing foreclosure reviews, neither regulator had made decisions about what information to provide to borrowers. GAO's internal control standards and best practices indicate that an effective communication strategy and timely reporting can enhance transparency and public confidence. Absent a clear strategy to guide regular communications with individual borrowers and the general public, regulators face risks to transparency and public confidence similar to those experienced in the foreclosure review.
section_title: What GAO Recommends
paragraphs: OCC and the Federal Reserve should improve oversight of sampling and consistency in the continuing reviews, apply lessons in planning and monitoring from the foreclosure review, as appropriate, to the activities of the continuing reviews and amended consent orders, and implement a communication strategy to keep stakeholders informed. In their comment letters, the regulators agreed to take steps to implement the recommendations.

section_title: Letter
paragraphs: In 2011, the Office of the Comptroller of the Currency (OCC), the Board of  Governors of the Federal Reserve System (Federal Reserve), and the  Office of Thrift Supervision (OTS) issued consent orders against 14  mortgage servicers after a review of these servicers’ controls over their  foreclosure processes. The consent orders require the servicers to  engage third-party consultants to review the servicers’ loan files to identify  borrowers who suffered financial injury through errors,  misrepresentations, or other deficiencies in foreclosure processes in 2009  and 2010. The regulators also directed servicers to conduct an outreach  process to enable eligible borrowers to request a review of their loan  files. Together, both processes comprise the Independent Foreclosure  Review (foreclosure review). According to regulators, the goals of the  foreclosure review were for consultants to identify as many harmed  borrowers as possible, to treat similarly situated borrowers across all 14  servicers similarly, and to help restore public confidence in the mortgage  market. The consent orders require the servicers to remediate the  financial harms suffered by borrowers as determined by consultants.  Roughly 4.3 million borrowers of the 14 servicers were in some stage of  foreclosure in 2009 and 2010 and, as of December 2012, consultants had  more than 800,000 loans slated for review. In a turn of events, in January  2013, the regulators announced agreements that discontinued the  foreclosure review with 11 of the 14 servicers. These agreements  replaced the foreclosure review with a compensation framework that does  not rely on determinations of whether borrowers suffered financial harm.  The servicers participating in the agreements cover nearly 90 percent of  borrowers who were eligible for the foreclosure review. In late February  2013, the regulators publicly released amended consent orders for the  servicers participating in the agreements that formalized the provisions in  the agreements. The remaining three servicers, covering 450,000  borrowers (10 percent), are continuing with the foreclosure review work.
This report represents the second phase of our examination of the  foreclosure review process. In a previous report, we reviewed the  servicers’ outreach efforts to inform borrowers of the foreclosure review.  We were in the process of reviewing other aspects of the foreclosure  review when OCC and the Federal Reserve announced the agreements.  We note that this report does not assess the regulators’ rationale for  accepting the agreements. Further, we do not assess any trade-offs  involved in the regulators’ choice to amend the consent orders with the  servicers. We abbreviated our review plan and developed this report to  address: (1) challenges to the achievement of the goals of the foreclosure  review, (2) the extent of transparency in the foreclosure review process,  and (3) lessons that could be useful for the activities under the amended  consent orders and continuing reviews.
To review third-party consultants’ file review processes, we used a data  collection instrument and gathered information on third-party consultants’  sampling plans, as contained in engagement letters between the  servicers and consultants. We analyzed information in the consultants’  plans for selecting loan files, analyzing file review results, and conducting  additional sampling of harmed borrowers, if needed. We confirmed key  observations of our analysis in interviews and site visits with officials  responsible for developing the sampling plans at five third-party  consultant engagement teams, interviews with regulator staff, and with  regulators’ examination teams that reviewed the plans. We selected these  consultants based on the size of their respective servicer and the identity  of the servicer’s regulator to ensure a range of perspectives. We reviewed  relevant documents from regulators, third-party consultants, and law firms  describing steps taken to foster consistency and accountability among file  reviews. We also reviewed regulators’ guidance issued to third-party  consultants, law firms, and examination teams and compared this  information and these parties’ actions to criteria such as the regulators’  standard practices, regulators’ goals for the foreclosure review, and our  previous work. We reviewed press releases and documents from  regulators related to the foreclosure review, enforcement action  documents available on the regulators’ websites, speeches and  testimonies by agency officials, and agency policies regarding public  disclosure of enforcement action information. We conducted interviews  with regulator staff, selected third-party consultants and law firms, and  consumer groups. For those third-party consultants we did not interview,  we obtained written information from them to address our objectives. See  appendix I for additional information on our scope and methodology.
We conducted this performance audit from July 2012 through March 2013  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Mortgage servicers are the entities that manage payment collections and  other activities associated with mortgage loans. Servicing duties can  involve sending borrowers monthly account statements, answering  borrowers’ inquiries, collecting monthly mortgage payments, and  maintaining escrow accounts for property taxes and insurance. In the  event that a borrower becomes delinquent on loan payments, servicers  also initiate and conduct foreclosures. Errors, misrepresentations, and  deficiencies in foreclosure processing can result in a number of harms to  borrowers ranging from inappropriate fees to untimely or wrongful  foreclosure. A number of federal regulators share responsibility for  regulating the banking industry in relation to the origination and servicing  of mortgage loans. OCC has authority to oversee nationally chartered  banks and federal savings associations. The Federal Reserve oversees  insured state-chartered banks that are members of the Federal Reserve  System, bank and thrift holding companies, and entities that may be  owned by federally regulated depository institution holding companies but  are not federally insured depository institutions.
In September 2010, allegations surfaced that several servicers’  documents accompanying judicial foreclosures may have been  inappropriately signed or notarized. In response to this and other  servicing issues, federal banking regulators directed servicers to  complete self-assessments of their foreclosure processes. In addition,  banking regulators conducted a coordinated on-site review of 14 of the  largest mortgage servicers to evaluate the adequacy of controls over  servicers’ foreclosure processes and to assess servicers’ policies and  procedures for compliance with applicable federal and state laws. On the  basis of their findings from the coordinated review, OCC, OTS, and the  Federal Reserve issued in April 2011 formal consent orders against the  14 servicers under their supervision. To comply with the consent orders,  each of the 14 servicers is required to, among other things: enhance its  vendor management, training programs and processes, and compliance  with all applicable federal and state laws, rules, regulations, court orders,  and servicing guidelines. In addition, the consent orders required each  servicer to retain an independent firm to review certain foreclosure  actions on primary residences from January 1, 2009, to December 31,  2010. Third-party consultants were permitted to retain outside counsel to  provide necessary legal expertise in completing the foreclosure review.
Through the foreclosure review, consultants were to identify borrowers  who suffered financial injury as a result of errors, misrepresentations, or  other deficiencies in foreclosure actions, and recommend remediation for  harms suffered by borrowers, as appropriate. In general, the consent  orders identified seven areas for consultants to review: (1) whether the  servicer had proper documentation of ownership of the loan; (2) whether  the foreclosure was in accordance with applicable state and federal laws;  (3) whether a foreclosure sale occurred while a loan modification was  under consideration; (4) whether nonjudicial foreclosures followed the  terms of the loan and state law requirements; (5) whether fees charged to  the borrower were permissible, reasonable, and customary; (6) whether  loss-mitigation activities were handled in accordance with program  requirements and policies; and (7) whether any errors,  misrepresentations, or other deficiencies resulted in financial injury to the  borrower. Servicers proposed third-party consultants to conduct the  foreclosure reviews. After regulators reviewed the independence of  proposed third-party consultants, servicers and third-party consultants  submitted engagement letters outlining their foreclosure review processes  to the regulators for their review and approval. OCC and the Federal  Reserve posted approved engagement letters between the servicers and  third-party consultants on their respective websites. Regulators also  posted on their websites a financial remediation framework that provided  examples of errors covered by the consent orders and the corresponding  compensation or other assistance that consultants could recommend  based on their findings.
The foreclosure review had two components: a process for eligible  borrowers to request a review of their particular circumstances (referred  to as the borrower outreach process) and a review of categories of files  (referred to as the look-back review). The borrower outreach process was  intended to complement the look-back review and help identify borrowers  who may have suffered financial injury. The regulators required the  servicers to inform borrowers who believed they might have been  financially harmed due to inappropriate foreclosure that they could submit  a request for review of their particular circumstances. The servicers  conducted this outreach through advertising and direct mail. After several  extensions, the final deadline for submission of these requests was  December 31, 2012.
For the look-back review, the consent orders allowed third-party  consultants to use statistical sampling techniques to select samples of  files for review from various categories of loans, pursuant to regulators’  guidance and approval as well as 100 percent review of certain loan  categories. Regulators established minimum sampling requirements for  third-party consultants to conduct statistically valid sampling of the target  population and expectations for additional sampling methods, if  warranted, to identify as many harmed borrowers as possible for  remediation. Bank examiners often use sampling methods to review the  files of a financial institution’s operations. The use of sampling by  examiners has been a regularly accepted practice and is commonly used  as a means to review a bank’s files when a full review of all files is not  practicable. OCC states in its handbook on sampling methodologies,  which is geared toward sampling loan portfolios, that it is impractical or  impossible for bank examiners to review all items or files when examining  an area of a bank’s operations, especially if the volume of information is  large. Examiners use sampling to identify a random subset of files to  learn about the multitude of items from which those files are drawn. Upon  drawing appropriate statistical inferences from this subset, they can state  with a certain level of confidence that the inferences apply to the  population as a whole. The benefits of using statistical sampling include  the ability to generalize results to the sampled populations and to quantify  uncertainty in estimates attributable to sampling.
In our prior report on the borrower outreach component of the foreclosure  review, we found that regulators and servicers had gradually improved  the communication materials for borrowers but that regulators could make  further enhancements to the outreach efforts. First, we found that  regulators did not consider best practices, such as using tests or focus  groups, to assess the readability of the outreach materials and did not  solicit input from consumer groups when reviewing initial communication  materials. As a result, we reported that the materials at that time might  have been too complex to be widely understood. Second, we found that,  although the communication materials included information about the  purpose, scope, and process for the foreclosure review and noted that  borrowers may be eligible for compensation, the materials did not provide  enough specific information about remediation, which best practices  suggest could have encouraged more borrower responses. Third, we  found that the outreach planning and evaluation targeted all eligible  borrowers with limited analysis conducted to tailor the outreach to specific  subgroups within the population. Therefore, we noted that some  underrepresented borrowers may not have been apparent to regulators  without further analysis of the characteristics of respondents compared to  nonrespondents. To help ensure that all borrowers had a fair opportunity  for review, we recommended that OCC and the Federal Reserve enhance  the language on the foreclosure review website, include specific  remediation information in the outreach, and require servicers to analyze  trends in borrowers who have not responded and, if warranted, take  additional steps to reach underrepresented groups.
In response to the recommendations that we made in our previous report,  OCC and the Federal Reserve took steps to enhance the independent  foreclosure review website and communication materials and conducted  more targeted outreach. We determined that regulators implemented our  first recommendation by including on the foreclosure review website a  help sheet for the request-for-review form that provides tips in plain  language, an explanation of key terms, and additional instructions to help  borrowers fill out the form. In response to our second recommendation,  OCC and the Federal Reserve publicly released a financial remediation  framework and included ranges of potential payment amounts or  categories in their communication materials and other outreach. For  example, the regulators released 60-second radio and television  advertisements stating that if consultants find errors, homeowners may be  eligible for compensation or other remedies, such as refunding fees,  stopping a foreclosure action, or making payments that could range from  $500 to $125,000. In addition, at the regulators’ instruction, servicers  included similar language on ranges of potential payment amounts or  categories in other outreach materials. In response to our third  recommendation, OCC and the Federal Reserve tailored their recent  outreach actions to target communities based on audience  characteristics, response data, and consumer research. Regulators  included a wide range of minority media and a broad mix of media  formats, including print advertisements, radio, television, and Internet. To  tailor this outreach, a market analysis was completed to identify areas  and ethnic groups with the greatest opportunity for increased awareness.
Outreach materials also were made available in eight different languages.  Regulators solicited numerous community groups for their input and  assistance on the outreach and identified effective messengers by using  leaders of community groups that represent minorities to deliver radio and  television public service announcements.
In January 2013, OCC and the Federal Reserve issued joint press  releases stating that they had reached agreements with 11 of the 14  mortgage servicing companies subject to the April 2011 consent orders to  discontinue the foreclosure review conducted by third-party consultants  and to provide almost $8.8 billion in cash payments and foreclosure- prevention assistance to borrowers. With this change in direction from the  foreclosure review to an agreed-upon payment process, regulators and  servicers moved away from identifying the types and extent of harm  borrowers may have experienced and focused instead on assigning  borrowers into categories based on objective criteria and issuing  payments in what they expect will be a shorter amount of time than would  have occurred under the foreclosure review. To explain their rationale for  pursuing the agreements, OCC and Federal Reserve staff said they  considered a variety of factors, including delays in payments to harmed  borrowers, the total remediation payments expected to be made to  borrowers, and costs of the reviews. Under the agreements, servicers will  provide approximately $3.4 billion in direct payments to eligible  borrowers.
Using a framework provided by regulators and characteristics of  borrowers’ loans, servicers will categorize borrowers, and regulators will  develop a distribution plan and direct a payment administrator to  distribute cash payments. As a result, all borrowers who were eligible for  foreclosure reviews under the consent orders are expected to receive  payments ranging from hundreds of dollars up to $125,000, depending on  the borrower’s category. Under the agreements, servicers will also  provide approximately $5.4 billion in foreclosure-prevention assistance to  borrowers, such as loan modifications. To the extent practicable,  servicers are to prioritize such assistance for borrowers eligible for the  foreclosure review. Eligible borrowers are expected to receive notice  about the payments by the end of March 2013, whether or not they filed a  request-for-review form, and borrowers will not need to take further action  to be eligible for compensation. Nearly 4 million borrowers, or about 90  percent of the eligible borrower population, are covered by the servicers  that signed the agreements. In late February 2013, regulators released  amendments to the April 2011 consent orders that incorporated the  provisions of the agreements. The amended orders are publicly available  on the regulators’ respective websites. Consultants for the servicers that  did not reach agreements with the regulators—Ally Financial (GMAC  Mortgage), Everbank, and OneWest—continue their foreclosure review  activities. More than 450,000 borrowers fall under the three servicers that  do not have amended consent orders.
In April 2011, regulators entered into consent orders with 14 mortgage servicers.  The consent orders require the servicers to conduct foreclosure reviews by  engaging third-party consultants to review the servicers’ loan files to identify and  remediate the financial injuries suffered by borrowers through errors,  misrepresentations, or other deficiencies in their foreclosure processes in 2009 and  2010.
In January 2013, the regulators announced agreements in principle with 11 of the 14  servicers subject to the April 2011 consent orders. The parties agreed to replace the  foreclosure review with a compensation framework that does not rely on  determinations of whether borrowers suffered financial harm, instead requiring  participating servicers to provide cash payments and foreclosure-prevention  assistance to borrowers.
In February 2013, the regulators publicly released amended consent orders that  formally replaced the requirements related to the foreclosure review for the servicers  participating in the agreements.
section_title: Complexities of the Foreclosure Review Process and Limitations in Regulators’ Guidance and Monitoring May Have Hindered Achievement of Goals
paragraphs: According to regulators, the goals of the foreclosure review were for  consultants to identify as many harmed borrowers as possible, to treat  similarly situated borrowers across all 14 servicers similarly, and to help  restore public confidence in the mortgage market. However, regulators  faced various challenges in accomplishing these goals.
subsections: section_title: Size, Scope, and Complexity of the Foreclosure Review Process Posed Challenges
paragraphs: According to regulator staff and third-party consultants, coordinating the  foreclosure review process was challenging because of the large number  of actors directly involved in the review process and the large number of  borrowers eligible for the review. Specifically, the foreclosure review  process included 14 servicers; 14 third-party consultant teams from 7  different consulting and accounting firms, some with subcontractors; and  more than 10 third-party law firms. Further, the reviews were overseen by  local examination teams for OCC or the Federal Reserve as well as  headquarters staff for both regulators. In addition, there were as many as  4.3 million borrowers whose files could have been reviewed. One  examination team informed us that the size of the review population  presented challenges beyond those that accompany a typical consent  order. Another challenge examiners and consultants noted was that, due  to the deadline extensions for borrowers to submit their requests-for- review, the volume of the file review work kept changing and the total  number of files requiring review was unknown. The original deadline for  requests-for-review was April 2012. Regulators reported that more than  160,000 borrowers had submitted requests-for-review as of April 30,  2012. The deadline was extended three times to December 31, 2012.  Regulators reported that more than 510,000 requests were submitted and  received as postmarked on or before the December 31, 2012 deadline.  According to OCC, roughly 300,000 additional files were selected for  review through the consultants’ look-back review process. OCC staff told  us that more files likely would be selected if errors were found and  consultants needed to conduct additional sampling to identify as many  harmed borrowers as possible.
Third-party consultants and law firms told us that the size of the loan files  and the scope of the file review made the process complicated and time- consuming. Consultants from whom we obtained information told us that  a typical loan file is large with many types of documents. For example,  documents may include servicer notes on communication with the  borrower, documents collected from the state foreclosure attorneys  responsible for the foreclosure activities, records of fees charged and  payments made, and, in many cases, documents assessing a borrower’s  eligibility for loan modification and loss mitigation activities. Consultants  told us that some files may contain as many as 50 documents, potentially  comprising more than 2,000 pages. Consultants also stated that the  reviews were challenging because they covered such a wide variety of  complex issues, including different state foreclosure laws, federal laws  and regulations, and guidelines for federal and servicers’ proprietary loan  modification programs. To assess each of these areas, consultants  developed a series of test questions—generally yes or no questions—to  identify potential errors. The number of test questions used by third-party  consultants to conduct the file reviews varied. For example, one  consultant told us that they had approximately 2,600 test questions with  more than 4,000 discrete steps, while another consultant told us they had  16,000 test questions. Further, third-party consultants from whom we  obtained information stated that their reviewers spent as many as 50  hours to complete a full file review, although review times varied  depending on the issue and type of review.
OCC and the Federal Reserve stated that the uniqueness of each  servicer’s borrower population and process for recording and storing  information on borrowers’ loan files posed challenges for defining the  review parameters and developing a uniform review structure for all the  consultants. For example, Federal Reserve staff told us that servicers’  systems have different ways to identify borrowers who are current on a  loan modification and that the systems have varying capabilities to  provide information about the last payment received from a borrower. As  a result, consultants had to take different approaches to identify  borrowers with the same characteristics. In addition, Federal Reserve  staff told us that servicers had different concentrations of loans in  geographic areas and that, therefore, consultants might select samples  differently based on these concentrations. For example, if a servicer did  not have high numbers of foreclosures in a particular state, choosing a  large sample of loans in that state would not have been appropriate. As a  result, OCC and Federal Reserve staff said that it was not feasible to  design one file review process that would apply to all servicers and that it  was necessary for third-party consultants to tailor their file review  processes, particularly the review questions used to conduct the file  review, to the unique characteristics of a given servicer.
Faced with various complexities and challenges that the foreclosure  review posed, regulators told us that they issued guidance and took a  number of oversight steps. First, the sections of the consent orders  issued to servicers supervised by both OCC and the Federal Reserve  outlining the purpose of the foreclosure review were nearly identical.  According to OCC and Federal Reserve staff, the similarity in the consent  orders was intended to ensure that the reviews covered the same issues  and resulted in similar results for similarly situated borrowers. Consultants  we interviewed said that they designed their reviews to address the  issues as they were identified in the consent orders. Second, regulators  issued guidance to third-party consultants to help frame the file review  process and promote consistency in its implementation. Between May  2011 and October 2012, regulators issued 29 joint pieces of guidance to  third-party consultants on various topics. For example, OCC and the  Federal Reserve jointly issued a financial remediation framework that was  designed to be a unifying factor among all the reviews by helping to  ensure that similarly harmed borrowers received similar remediation.  Regulator staff said that they issued guidance in response to similar  questions they received from multiple consultants or examination teams,  which oversaw the reviews at the local level. For example, one third-party  consultant we interviewed said that the reviews of issues related to the  Servicemembers Civil Relief Act (SCRA) would likely provide fairly  consistent results for borrowers due, in part, to the clear guidance  provided by regulators.
In addition to consistent consent orders and guidance, OCC and Federal  Reserve staff implemented regular communication mechanisms to help  foster consistency in the reviews. Regulators had a robust system of  regular meetings involving third-party consultants, servicers, examination  team staff overseeing the consultants’ work, and OCC headquarters and  Federal Reserve Board staff to discuss challenges with the file review  process and help promote consistency among the reviews. OCC and the  Federal Reserve met with the parties both separately and as a group and  received weekly status reports from the consultants. For example,  regulator staff said that weekly calls with third-party consultants provided  consultants with an opportunity to raise areas of inconsistency that they  had identified. These calls were also used to disseminate new guidance  and discuss comments on any pending guidance. According to third-party  consultants and examination team staff we interviewed, these meetings  were helpful for sharing information among the reviews, developing a  similar understanding of the file review process, and discussing  challenges consultants encountered in reviewing files that may have  affected the consistency of the results for borrowers. In addition to  meeting with consultants and servicers, OCC and Federal Reserve staff  also held a separate weekly meeting, which included only headquarters  staff of each regulator, to discuss new and emerging issues or requests  for clarification on guidance that each regulator received from its  respective local examination teams. In addition to the weekly calls, OCC  headquarters staff and Federal Reserve Board staff visited each of the  consultants to observe the file review processes.
section_title: Lack of Clarity in Regulators’ Sampling Guidance Could Have Limited the Usefulness of the Information Obtained from the Foreclosure Review Process
paragraphs: The April 2011 consent orders expressly allowed third-party consultants  to use sampling techniques to identify harmed borrowers. According to  regulator staff, the large number of loans in the review population as well  as the complexity of the file review process made it difficult for  consultants to review all the eligible loan files for errors. As a result, the  review relied on sampling, a process of selecting units—in this case,  foreclosure files—in a manner that regulators envisioned would allow  consultants to identify patterns in errors. In May 2011 regulator-issued  guidance on sampling, regulators expressly allowed third-party  consultants to use sampling and noted that the consultants’ sampling  methodologies should take into consideration public perception as well as  the need to provide a high degree of certainty that borrowers who were  financially harmed would be identified and obtain remediation.
The May 2011 guidance outlined broad parameters for how third-party  consultants should approach the sampling methodology to identify  harmed borrowers and support a consistent review, such that similarly  situated borrowers would have similar results. As part of sampling,  regulators anticipated that third-party consultants would segment the  review population into loan categories as some loan categories had a  potentially higher likelihood of errors. For example, the guidance identified  some potential high-risk loan categories, including certain states,  foreclosure law firms, and servicing or foreclosure processing activities  (e.g., rescinded foreclosures or foreclosure that occurred after loan  modification) that could be associated with a higher likelihood of servicing  or foreclosure-related errors. Regulator staff stated that differences  among the servicer’s loan portfolios and servicing practices made  requiring specific loan categories for review inappropriate, with the  exception of three categories in which regulators anticipated that  consultants would identify a relatively large number of errors in the files.  For these three categories—bankruptcies, SCRA loans, and agency- referred foreclosure cases—regulators required 100 percent review of  files. The May 2011 guidance also indicated that consultants should be  prepared to conduct a second stage of additional analysis of files with a  certain number of servicing and foreclosure-related errors that were  identified during the initial sampling.
Based on our analysis of the sampling plans developed by third-party  consultants, we found that the sampling methodologies used by  consultants varied among the reviews.
Expected population error rate. The sampling plans varied in their  expected population error rate from 0 percent—that is, consultants  expected to find few or no servicing or foreclosure processing errors  in the sampled loan category—to 10 percent, with 8 of the 14 reviews  assuming an error rate of zero, three assuming an expected  population error rate of 3 percent, and three a rate of 10 percent.
Loan category sample size. As a result of different expected  population error rates, the plans varied in the size of the samples for  analyzing various loan categories identified by the consultant. For  example, some consultants selected approximately 100 loans in a  sampled loan category for a review, and some consultants selected  approximately 370 loans in the sampled loan categories.
Loan categories. Based on our analysis, the loan categories used by  consultants for their analysis varied from review to review. For  instance, although all the consultants analyzed the files for errors  related to loan modifications, some categorized loans by the loan  modification program (e.g., Home Affordable Modification Program  (HAMP) or proprietary) and others categorized loans for reasons  modifications were denied.
Review parameters: For similar loan categories, some consultants  anticipated conducting 100 percent review of all files in that category  whereas other consultants planned to sample files. For example,  some third-party consultants planned to review all rescinded  foreclosures, whereas others proposed reviewing a sample of those  loans.
According to regulator staff, differences in the sampling plans reflected  differences in the size and characteristics of the servicers’ loan portfolios  and data systems. Regulator staff explained that they reviewed each  proposed sampling plan to help ensure it met the parameters outlined in  the guidance and would result in statistically valid results. However,  according to OCC staff, they recognized that some consultants had not  fully implemented the sampling approach as expected, and OCC is taking  steps to address these differences for one of the servicers that is not  subject to an amended consent order and must continue its review.
Our analysis of the May 2011 sampling guidance provided by regulators  found that the guidance was ambiguous about a key parameter that  affected consultants’ sampling methodologies and contributed to  differences in those methodologies. Specifically, in their May 2011  guidance, regulators did not indicate whether consultants should explicitly  set an expected population error rate or provide consultants with direction  on the factors they should consider when setting an appropriate expected  population error rate to determine the size of the sample used for their  analysis. GAO’s Financial Audit Manual and the standards of the  American Institute of Certified Public Accountants (AICPA) have found  that a key element of effective sampling is the determination of an  appropriate sample size based on specified precision and reliability levels  and an expected population error rate or frequency of errors.
Regulators’ May 2011 guidance on sampling specified that consultants  should use 3 percent precision and 95 percent reliability levels to  determine their sample size. According to regulator staff, these precision  and reliability levels were selected to provide a high-level of confidence in  the sampling results. However, the guidance did not specify an expected  population error rate for consultants to use in determining sample size.  Generally, the expected population error rate, like precision and reliability  levels, is determined based on professional judgment and includes  consideration of factors such as results of prior reviews and knowledge  about any potential risks in servicing and foreclosure processing errors.  According to regulators, they expected consultants to find errors in their  sampled files, because the consent orders that required the foreclosure  review process arose out of assessments conducted by regulators that  identified the potential for servicing and foreclosure-related errors. Similar  to the precision and reliability levels, there is a relationship between the  expected population error rate and sample size needed to attain a  specific precision level (margin of error), where the size of the sample  generally expands as the expected population error rate approaches 50  percent. When an expected population error rate is not specified, it can  be interpreted as implying that the error rate is low or even zero—that is,  that few or no errors are expected to be found in the sample regardless of  whether the expected population error rate of zero is appropriate for the  sample goals. OCC staff told us that their handbook on sampling  methodologies—a reference their staff use for sampling and one that both  regulators suggested consultants consider in designing their sampling  approaches—generally assumes few or no errors will be found in a  sample.
Variations among the sample sizes used to analyze the various loan  categories identified by consultants result from differences in the  consultants’ expected population error rates and led consultants to use  different triggers to determine when to conduct additional analysis of an  error or errors found in a loan category. As shown in figure 1, Consultant  A would conduct additional analysis of their sampled loan categories if  one or more errors were found. In contrast, Consultant B would conduct  additional analysis only when five or more errors were found in their  sampled categories. Both of these approaches use the precision and  reliability levels specified in the May 2011 guidance, with an expected  maximum threshold of a 3-percent error rate, but relied on different  sampling approaches and different expected population error rates to  calculate sample size. Therefore, with the same number of errors  identified through file reviews, some consultants would conduct additional  analysis and some would not. According to regulator staff, when errors  were found, they expected consultants to conduct additional sampling of  the loan category or to review all of the loans in that category. OCC staff  explained that after approving the original sampling methodologies, they  recognized that their review of the plans had missed some aspects and  that consultants were using various expected population error rates to  calculate sample size. OCC staff told us that they were considering steps  to try and address these differences for one of the servicers that had not  joined the agreements to end the foreclosure review. As a result of this  variation, the reviews could have produced inconsistent results for  similarly situated borrowers, thereby potentially limiting achievement of  one of the goals of the foreclosure review process.
Our analysis also found that the May 2011 guidance on sampling did not  include a discussion of regulators’ expectations for reporting on sampling,  and variations among the sampling plans would have limited the types of  information that regulators could report. For example, the guidance did  not specify if regulators expected consultants, based on their sampling  methodology, to be able to provide information on the error rate for the  servicer’s whole population of eligible borrowers or for certain  characteristics, such as high-risk loan categories (e.g., states). Our  analysis found that 1 of the 14 reviews explicitly designed samples that  would potentially allow consultants to generate reliable estimates of the  error rate for certain states. Other consultants may have been able to  report counts of errors by state, but some consultants sampled as few as  one or five cases per state, and the counts would not allow regulators to  determine whether reported errors were relatively frequent or infrequent  in specific states. As a result, regulators could not have reported results  by state because some reviews did not have enough cases, selected in a  generalizable manner, from individual states to report statistically reliable  estimates. According to regulators, reporting error rates was not a  consideration in developing the sampling approach outlined in the  guidance; rather, the sampling methodology was intended to find as many  harmed borrowers as possible. However, it would be feasible to develop  sampling approaches that could both produce error rates and identify  characteristics of harmed borrowers.
Similarly, differences in the loan categories analyzed by the third-party  consultants would have limited regulators’ ability to easily aggregate  results among the reviews or present comparable servicer-specific  information. For example, differences in how consultants organized the  loan modification category could have impeded regulators’ ability to  aggregate the results of the loan modification analyses conducted by  third-party consultants or to present equivalent information among the  servicers to the public. In addition, these differences in the loan category  definitions as well as other differences in the consultants’ sampling  methodologies would have made it difficult for regulators to aggregate  results to represent the full population of eligible loans. One consultant  told us that they anticipated calculating an error rate based on the number  of files reviewed compared to the number of files with errors. This method  might have provided some information on errors, but could be subject to  bias if the sample of loans reviewed were not representative of the  population, regardless of whether the estimates from the samples were  appropriately weighted for probability of selection or included appropriate  confidence intervals to reflect sampling error. However, this information  would not necessarily provide a statistically valid description of the extent  of errors in the full population that regulators and other stakeholders  would need to understand and assess the results.
section_title: Regulators’ Sampling Approach Hindered Their Ability to Monitor Achievement of a Review Goal
paragraphs: According to regulator staff, they expected consultants to continue  reviewing files until as many harmed borrowers as possible were  identified, a goal of the foreclosure review process. Regulator staff told us  that if third-party consultants’ initial analyses of sampled loans identified  errors within a loan category, consultants were expected to analyze the  characteristics of loans with errors to identify any patterns and use this  analysis as the basis for a second sampling phase or to review all the  files in a loan category. Although not explicitly stated in the May 2011  guidance, according to OCC staff, additional analysis was warranted  when the errors in the sampled loan category exceeded 3 percent with a  95 percent confidence level. Although the May 2011 guidance on  sampling did not specify the characteristics to consider for this analysis,  Federal Reserve staff told us that identifying the characteristics to include  would require consultants to use their judgment and they anticipated it  would include things like the loan category itself—such as an error related  to a certain foreclosure attorney that might warrant additional review of  other loans that had used the same attorney—or some other  characteristic about the loan.
According to a few consultants, they were considering characteristics  such as the loan product or date of the loan modification solicitation or  foreclosure sale, as potential characteristics. Regulators told us that  based on the results of the consultants’ analyses of the initially sampled  loans with errors, consultants were to develop a methodology—including  conducting additional sampling or reviewing all files with those shared  characteristics—to identify other loans that had similar characteristics that  could have had similar errors to use as a basis for a second review  phase. According to Federal Reserve staff, this process would have been  iterative—where consultants would have found errors, analyzed those  errors, resampled and then repeated the process until as many files with  errors as possible had been found. Third-party consultants were expected  to discuss the results of their initial file reviews, their analysis of error  patterns, and their proposed approach for conducting additional file  reviews with regulators prior to conducting additional reviews.
Our analysis found that the regulators’ sampling approach did not include  mechanisms to facilitate their oversight of the extent to which consultants  would have reached as many harmed borrowers as possible. For  example, the regulators’ sampling approach did not provide an objective  method for regulators to use in determining if consultants had conducted  sufficient reviews and could stop their review activities, except in those  cases where there were few or no errors. As we described earlier,  consultants were using different triggers of the number of errors to  determine if additional analysis of loan categories was required. Without a  mechanism for regulators to use in assessing the extent to which each  consultant had found as many harmed borrowers as possible or to  compare the review results among the consultants, assessing which  consultants had done enough work to identify a sufficient portion of  harmed borrowers and which consultants needed to conduct additional  analysis would have been difficult for regulators. According to Federal  Reserve staff, they anticipated that consultants would continue reviewing  files until the sampling found no additional errors. However, unless a  large majority of the population is examined for errors, reviewing files until  sampling finds no more errors does not necessarily imply that all or most  errors would be identified. OCC staff told us that at the time of the  agreements that led to the amended consent orders, regulators were  considering developing such mechanisms.
Additional sampling activities—specifically, the use of baseline samples— could have provided regulators with a mechanism to use in monitoring the  extent to which consultants had identified as many harmed borrowers as  possible. The Office of Management and Budget (OMB) standards for  statistical surveys state that where sampling is used, it should include  protocols to monitor activities and provide information on the quality of the  analyzed data. A baseline sample could have been used to establish an  estimate of the number of harmed borrowers among the servicer’s full  population as part of their sampling methodology and would have  provided an objective and consistent measure for regulators to use to  gauge if a third-party consultant had identified a sufficient portion of  harmed borrowers through their reviews. Specifically, regulators could  have compared a statistically valid estimate of the number of harmed  borrowers in the servicer’s total population of eligible borrowers with the  number of harmed borrowers the consultant identified through file  reviews. Discrepancies between the estimated number of harmed  borrowers and the number found in the review would have helped to  indicate the extent to which there may have been additional harmed  borrowers who had not been identified. Without this type of comparison,  regulators did not have an objective measure to help determine when a  consultant had completed sufficient review of the servicers’ files.
In addition, the regulators’ sampling approach did not provide a clear  mechanism for regulators to assess the extent to which consultants had  identified the appropriate high- and low-risk loan categories to confirm  that those categories were accurate and to signal if there were additional  potential high-risk loan categories that had not been identified, but  warranted additional sampling and review. As we noted earlier, the  regulators’ sampling guidance suggested a number of high-risk loan  categories consultants should consider in designing their sampling  approach—such as rescinded foreclosure or foreclosures following loan  modification—that were potentially associated with a higher likelihood of  error and, depending on the servicer, warranted targeted sampling to  verify the extent to which errors had occurred among the loans in those  categories. However, there is evidence to suggest that other loan  categories not included in the guidance, such as certain loan categories  based on demographic characteristics may also be associated with higher  likelihood of servicer errors. Without a mechanism to assess the  accuracy and sufficiency of each consultant’s high- and low-risk loan  categories, regulators would not have been able to assess the extent to  which consultants were targeting the appropriate high- and low-risk  categories to find as many harmed borrowers as possible.
An estimate of the overall error rate for the population—a rate that could  have been generated using information from the baseline sample—could  have been compared with the error rates consultants found for high- and  low-risk loan categories to confirm that those categories were accurate  and to signal if there were additional potential high-risk loan categories  that had not yet been identified. OMB has suggested that where  different characteristics are being compared, agencies should conduct  additional testing to help ensure appropriate statistical conclusions are  derived from the data. In our prior work we have used this type of  statistical testing to compare the performance (such as error rate) in a  total population with the performance for subgroups—such as loan  categories—to help distinguish among those groups. This can be part of  conducting an overall risk assessment of the population to enable users  to better focus on high- and low-risk areas. Without a mechanism to  gauge the extent to which high-risk loan categories had been identified,  regulators could have had difficulty assessing whether the consultants’  proposed activities for additional analysis were targeted at the appropriate  high-risk loan categories to identify as many harmed borrowers as  possible.
The May 2011 sampling guidance did not include a requirement that  consultants develop a baseline sample, but most consultants included  one in their sampling methodology. According to regulator staff, they had  not considered requiring consultants to include a baseline sample in their  methodology because the purpose of sampling was to help find harmed  borrowers by identifying concentrations of servicing and foreclosure- related errors as described in the consent orders, not to establish an error  rate for the servicer. Our analysis found that most consultants included a  baseline sample for the full review population in their sampling  methodology. However, due to differences among the consultants’  baseline samples, the ability of regulators to use consultants’ baseline  sampling results to assess consultants’ activities varies. For example, the  samples were not designed to estimate actual population error rates and  regulators’ guidance did not direct consultants to statistically test for  differences in the servicer’s total eligible population and among high- and  low-risk loan categories.
Our analysis also found that the regulators’ approach to conducting  repeated additional analyses to find as many harmed borrowers as  possible potentially involved timeliness trade-offs. OMB’s standards for  statistical analysis state that the sampling design should be appropriate to  achieve the sampling goals; in this case regulator staff told us that the  phase-one sampling was designed to identify patterns from among loans  with errors to facilitate a second phase of analysis to find as many  harmed borrowers as possible. Where the goal of sampling is to identify  patterns from among loans with errors in a sample, using a larger sample  size has the potential to provide more information to use in analyzing  patterns and determining the appropriate next steps. For those  consultants that used smaller sample sizes in their phase one analysis,  for example, a sample size of 100 loans where one error would trigger  additional analysis, finding patterns among the characteristics of those  loans with errors may have been difficult in cases where there were few  errors. As a result, these consultants may have had to conduct  potentially time-consuming repeated sampling of certain loan categories  where errors were found to find enough loans with errors to be able to  analyze the findings and identify patterns that would have allowed them to  conduct additional sampling or review. OCC staff told us that some  consultants were considering using statistical modeling as part of their  analysis of loans with errors to identify patterns from among those loans.  However, in cases where the sample size is small and the number of  loans with errors is small, the value of statistical modeling to identify  patterns could be limited.
According to Federal Reserve staff, due to concerns about the timeliness  of the sampling process, regulators anticipated that they may have had to  cut short the sampling process and provide remediation based on certain  borrower profiles—that is, borrowers who shared characteristics  associated with loans with a higher likelihood of error—without analyzing  each of the loan files that shared that profile. In contrast, a larger phase  one sample size designed with the goal of identifying patterns among  errors may have provided consultants with more information on loans with  errors to use as the basis for their additional analysis. In particular, this  may have been valuable for loan categories with a relatively large number  of loans—for example, our analysis found that in some cases a loan  category had more than 10,000 loans and, for one review, as many as  50,000 loans—and having more information available to use when  analyzing the results for these categories and determining next steps may  have been helpful. Although a larger sample size does not guarantee a  more timely review, a larger statistical sample may have provided  additional opportunities for analysis even with a similar proportion of  errors. In the case of the foreclosure review, where the goal was to use  sampling to find as many harmed borrowers as possible by analyzing  patterns from among loans with errors, larger sample sizes may have  resulted in a more timely process and potentially may have identified  harmed borrowers more quickly. According to regulators, they designed  their sampling to have a low tolerance for errors so as to find as many  harmed borrowers as possible.
section_title: Broad Guidance and Limited Monitoring of Inconsistencies Increased Risks of Different Treatment for Similar Borrowers
paragraphs: According to third-party consultants, regulators’ guidance did not address  certain aspects of the foreclosure review and consultants had to use  additional judgment and interpretation when applying certain guidance,  increasing the risks of inconsistency among review results. Consultants  also noted that regulators issued critical guidance throughout the review  process and frequently updated guidance, which expanded the scope of  the reviews and contributed to delays. For example, the Federal Reserve  issued three clarifications of loan modification guidance and OCC  provided seven responses to frequently asked questions on reviews of  loan modifications. According to regulator staff, developing the  foreclosure review process was intentionally iterative where they  responded to the most immediate need and used their evolving  knowledge to help refine the guidance. Consultants said that changes to  guidance required them to develop new test questions, re-train reviewers,  and redo file reviews. Although regulators issued numerous pieces of  formal guidance and informally responded to specific questions about  review procedures that examination teams or consultants raised,  consultants said that some of the guidance issued was not specific,  leaving room for their interpretation and potentially contributing to  inconsistent interpretations.
Guidance on fees: Guidance from regulators generally directed  consultants to consider whether the fees servicers charged for actions  such as property inspections or lawn care services were permissible  under the terms of the loan, followed applicable state and federal law,  and were reasonable and customary. However, regulators provided  additional explanation in response to requests from consultants for  clarification and guidance on defining the terms “reasonable” and  “customary.” Consultants for 13 of the 14 reviews indicated that they  used investor guidelines to determine if the fees charged were  reasonable; however, one consultant told us consultants were using  different versions of these guidelines. In some cases, consultants  used additional methods to evaluate whether fees charged to  borrowers were customary. For example, one consultant evaluated  fees against a set of benchmarks created from multiple servicers’  actual fee charges, while others said they benchmarked only against  the investor guidelines.
Guidance on remediation: Although regulators issued guidance to  consultants on how to determine the appropriate remediation for  different financial harms, 4 out of the 13 remediation categories would  have required consultants to make remediation determinations on a  case-by-case basis, risking inconsistent treatment of borrowers. For  example, consultants would have had to determine remediation on a  case-by-case basis if they found that a servicer had initiated  foreclosure or foreclosed on a borrower who was protected by federal  bankruptcy law. Consultants reported taking a range of approaches to  these case-by-case determinations, including awaiting further  guidance, developing their own guidelines, or relying on  recommendations from their third-party law firms. According to  Federal Reserve staff, at the time of the agreements that led to the  amended consent orders, regulators were considering options to  provide additional guidance to consultants for work in these areas,  including determining remediation amounts.
Guidance on missing documentation: Guidance issued to  consultants in March 2012 on how to determine whether a borrower  suffered financial harm when key documents—such as documents  that evidenced that foreclosure actions were taken or loan  modification application documents—were missing indicated that  consultants should treat those instances as errors on the part of  servicers. However, this guidance noted that consultants should defer  decisions on how to determine borrower remediation until regulators  provided further guidance. In the absence of additional guidance,  consultants indicated that they took a variety of approaches, including  waiting to make remediation decisions, making preliminary  considerations of whether the error might have caused financial harm,  or working with their third-party law firm to review any applicable legal  precedents. According to OCC staff, at the time of the agreements  that led to the amended consent orders, they were planning to issue  further guidance to consultants on remediation for borrowers with  missing documents and they had informally provided additional  direction about treatment of files with missing documents during their  regular meetings with consultants.
According to third-party consultants, regulators missed key opportunities  to increase the likelihood of consistent outcomes for borrowers by not  requiring development of common criteria or reference materials that  served as the basis for the foreclosure review process. Third-party  consultants and their respective law firms we interviewed told us that they  each developed their own test questions used by their file reviewers to  determine whether any errors or financial harm occurred. Consultants  developed the test questions based on analyses of state foreclosure laws,  loan modification guidelines, and bank policies, among other references.  According to OCC staff, the state law references were fairly  straightforward and they had confidence that the third-party consultants  and law firms would provide fairly consistent interpretations. However,  according to third-party consultants and law firms we interviewed,  compiling these references and using them to develop review questions  was challenging and time consuming and, in some cases, required  judgment or interpretation of the laws or guidelines. For example, they  noted that certain areas of relevant state law were unsettled and  continued to evolve as courts issued decisions. In addition,  representatives of law firms involved in the reviews told us that each firm  developed its own list of state foreclosure requirements and often came to  interpretations different from those of other law firms involved with the  foreclosure review. Consultants also indicated that some law firms had  different interpretations of which laws were applicable. According to OCC  staff, the scope of the reviews was limited to federal and state laws, but  several consultants reported that they reviewed servicers’ compliance  with certain county- or court-level requirements, whereas other  consultants said they generally did not include these requirements.  Although OCC and Federal Reserve staff told us that law firms were  selected for their independence and capacity to interpret these  documents and make these types of decisions, with multiple law firms  developing their own interpretations of the applicable laws and  requirements, consultants may have based their test questions on  different interpretations of laws, which could have hindered regulators’  ability to achieve one of their goals for the foreclosure review, similar  treatment for similarly situated borrowers.
Our analysis indicates that regulators missed another opportunity to  standardize reference materials in the area of loss mitigation and loan  modification requirements. Consultants noted challenges in compiling  relevant loan modification program guidelines. For example, according to  consultants, they had to compile requirements for multiple loan  modification programs and representatives of one consultant we  interviewed said that they had to compile requirements of 40 different  loan modification programs that the servicer used during the 2009 to 2010  period. In addition to numerous different programs, the guidelines for a  single program may have changed multiple times. One consultant noted  that they had to track and apply 28 program changes the Department of  the Treasury (Treasury) made to HAMP between 2009 and 2010. Our  prior work on HAMP identified instances of servicers interpreting HAMP  guidelines inconsistently, and a consumer advocacy group report noted  similar challenges with servicers implementing HAMP according to the  guidelines. In addition, several consultants from whom we obtained  information told us that they had to make some judgment calls when  interpreting regulatory guidance and program guidelines. For example,  one consultant noted that HAMP guidelines were unclear in critical  respects, particularly regarding the nature and extent of servicers’  obligations to notify borrowers about steps in the HAMP process. OCC  staff said they were aware of Treasury’s HAMP guidance and other  servicing guidelines and made an effort to make their guidance to  consultants consistent with these materials. OCC and Federal Reserve  staff also stated that certain staff members had general discussions with  Treasury staff to understand the HAMP guidelines. Consultants we  interviewed told us that they largely relied on their internal subject-matter  experts to interpret the relevant guidelines and did not consult with  program experts, such as Treasury staff who designed and developed  HAMP. As a result, third-party consultants and law firms may have  applied different interpretations of the legal or program requirements for  the same programs to the reviews, and the use of different reference  materials could have reduced the likelihood of achieving the goal of  treating similarly situated borrowers consistently.
Regulators took steps to monitor potential inconsistencies among the  reviews, but these steps were limited and likely would have resulted in  delays in providing remediation to borrowers. First, according to  regulators, they closely monitored weekly reports provided by consultants  to identify any differences in their progress that may have indicated some  inconsistency in the foreclosure review processes. These initial reports  included information on the number of mailings; requests for review; and  high-level counts of file reviews started, in process, and completed and  the number of files with borrower harm, but they did not include  information on the specific types of errors identified in the reviews or the  test criteria used to review files. Therefore, the usefulness of the reports  for identifying inconsistencies was limited. According to OCC staff, they  planned to begin requiring consultants to report on additional information,  such as the types of errors associated with financial harm found in the  reviews and proposed remediation amounts, which would have helped  identify inconsistencies among reviews.
Second, OCC and Federal Reserve staff said that they would identify  potential inconsistencies among the reviews by having staff from one  examination team assist with another team’s oversight of the foreclosure  review, but these rotations among the staff were not systematically  organized and did not include rotations across regulators. Further, some  examination team members we interviewed said that they had  participated in one or two reviews for other servicers, but examiners  overseeing reviews at a larger servicer noted that they were unable to  participate in multiple reviews because of time constraints. In addition,  OCC and the Federal Reserve did not provide the examination team  members conducting these rotations with guidance on the types of issues  to consider in assessing inconsistencies. This unsystematic approach  limited the extent to which regulators would have been able to identify  trends or inconsistencies.
Third, according to regulators, they reviewed some test questions but did  not compare them across reviews to identify inconsistencies. OCC staff  acknowledged that inconsistencies were inherent in the foreclosure  review because of the large number of actors and decision points and the  subjective nature of some of the decisions consultants had to make.  Similarly, Federal Reserve staff told us that inconsistencies among  reviews were inevitable due to differences among the servicers’ policies,  procedures, and systems, including their loan modification and loss  mitigation programs. Regulator staff said they had planned to conduct  assessments of the extent of inconsistencies affecting the outcomes for  borrowers across the reviews after the reviews and recommendations for  remediation were completed. However, conducting such an assessment  after the reviews were completed could have resulted in delays in  remediation because third-party consultants may have needed to change  their file review questions and redo file reviews if regulators identified  inconsistencies. In addition, regulators would have had to wait until all  consultants had completed their reviews to conduct such an assessment.  Consultants reported that they had estimated completing their reviews in  different time periods and an OCC official estimated that the reviews  would not have been completed until 2014.
section_title: The Foreclosure Review Highlighted the Importance of Planning and Monitoring Progress toward Goals
paragraphs: Our analysis of the foreclosure review process identified challenges in  regulators’ planning for the foreclosure review. These challenges  underscored the importance of the following three aspects of planning  and monitoring: (1) identifying the type and amount of information to  report as final results during the design of data analysis; (2) consulting  with stakeholders; and (3) assessing how well the reviews were tracking  their goals of identifying as many harmed borrowers as possible,  achieving consistent reviews for borrowers, and helping to restore public  confidence in the mortgage market.
As described earlier, variations among the 14 sampling methodologies  used by third-party consultants for analysis of loan files limited the types  of information that regulators would have been able to report. According  to regulator staff, they did not want to make final decisions on the types of  information they may have needed for public reporting before they had  seen the preliminary results from the file reviews. OMB has found that in  designing data analysis activities, including sampling, agencies should  consider the types of data they need to collect to be able to report useful  information on the results of their activities to the intended audience.  The resulting sample design should have these data output requirements  built into the structure. GAO’s internal control standards state that  producing reliable and relevant data is important for oversight and  management, including oversight provided by Congress. In addition, our  prior work has found that public reporting of results can be important for  strengthening public confidence in a process, and the data analysis  strategy should be designed to include information that can be reported  publicly. Providing useful and relevant public reporting of the review  results also was a key element in renewing public confidence in the  mortgage servicing market, a goal of the foreclosure review process.
However, regulators were limited in what they would have been able to  report because they did not plan for reporting in the design of the reviews.
As described earlier, the scope of the foreclosure review was broad and  regulators experienced challenges in issuing guidance on the wide variety  of complex issues covered by the reviews, resulting in delays in  completing file reviews and potentially contributing to inconsistencies in  the file review process. Regulators may have been able to better define  the scope of activities and issue more complete guidance prior to  commencing the foreclosure review process, thereby potentially reducing  the number of revisions to the scope or guidance, by consulting with  organizations directly responsible for or familiar with particular aspects of  the review before initiating the foreclosure review process. For example,  although regulators consulted with Treasury staff for their technical  expertise on the HAMP requirements during their development of the loan  modification and loss mitigation guidance, regulators did not have the  benefit of additional consultations with Treasury compliance officials and  discussions with other agencies responsible for overseeing federal loan  modification and loss mitigation programs to help them more clearly  define the programs covered by the consent order requirements and the  elements to consider in assessing servicers’ evaluation of loan  modification and loss mitigation activities—areas where regulators issued  clarifying guidance to third-party consultants. Regulators issued  additional guidance to clarify that the review of HAMP and proprietary  loan modification programs should include programs overseen by the  U.S. Department of Housing and Urban Development, U.S. Department  of Agriculture, and U.S. Department of Veterans Affairs.
In addition, regulators did not consult with community groups, such as  national organizations representing housing counselors that have worked  with individual borrowers on their loan modification and loss mitigation  applications. These consultations might have provided input on  challenges specific servicers and borrowers experienced with the range  of loss mitigation and loan modification activities which could have  assisted in identifying high-risk loan categories or program elements to  consider. For example, our prior work surveying housing counselors  found that while assisting borrowers with HAMP applications, counselors  experienced challenges with servicers, including missing documentation,  lengthy decision-making processes, and miscalculations of borrowers’  incomes. Our prior work that establishes generally accepted project  planning practices identified consulting with stakeholders as one of seven  generally accepted practices. In addition, our internal control standards  have found that consulting with external stakeholders can have a  significant impact on the achievement of goals. In contrast to the  process used to develop the loan modification and loss mitigation  guidance, OCC and the Federal Reserve consulted with consumer  groups while developing the remediation framework. In addition, they  consulted with the U.S. Department of Justice in developing the  foreclosure review guidelines related to SCRA. Although this consultation  occurred later in the process after third-party consultants had begun  SCRA reviews, one consultant cited the guidance that resulted from the  consultations as evidence of a strong practice that helped promote  consistent results among the reviews.
As we described earlier, regulators’ sampling approach did not include  mechanisms that would allow them to objectively measure the extent to  which consultants were on track to identify as many harmed borrowers as  possible. Similarly, as we previously described, regulators had a limited  process in place to identify inconsistencies among consultants’ file review  processes that could have affected results for borrowers. OCC and  Federal Reserve staff told us that they were aware of some areas where  there may have been inconsistencies, but according to Federal Reserve  staff these areas would not have led to significant differences. However,  in the absence of mechanisms to systematically monitor consistency,  regulators did not have the information to verify their understanding or  identify areas of the review with an increased likelihood of inconsistency.
Our prior work has identified using intermediate activities or measures to  assess progress toward intended results as an effective management  practice to understand the extent to which activities are on track to reach  stated goals. We have also identified practices that can help agencies  successfully implement the Government Performance and Results Act  and related results-oriented management initiatives, such as establishing  activities during program planning and design to monitor performance  toward these goals and using intermediate activities to analyze the gap  between where the performance is and where it needs to be to achieve  desired outcomes. We found that such activities can help management  target areas in need of improvement and select appropriate  methodologies to realize that improvement. In the absence of systematic  processes to monitor the extent to which the foreclosure review was  progressing toward its goals, regulators did not have an early warning  mechanism to help identify problem areas where interventions, such as  the issuance of clarifying guidance or other support, might have helped  reorient activities and address concerns.
section_title: Limited Communication Hindered Transparency for Individual Borrowers and the General Public
paragraphs: Regulators publicly released information on the foreclosure review  process beyond what is typically disclosed in connection with a consent  order, including engagement letters between servicers and consultants  and some guidance provided to the consultants. By law, federal banking  regulators must disclose any formal enforcement actions entered into  under the Federal Deposit Insurance Act. On a case-by-case basis,  banking regulators may consider the release of information beyond the  mandatory disclosures.
subsections: section_title: Transparency on How Foreclosure Errors and Financial Harm Would Be Determined Was Generally Lacking
paragraphs: In an effort to promote transparency in the foreclosure review process,  OCC and the Federal Reserve publicly disclosed some information  related to the April 2011 consent orders. For example, in November 2011,  OCC released redacted engagement letters between the servicers under  its jurisdiction and the consultants contracted to conduct the foreclosure  review. With the exception of one servicer, the Federal Reserve released  by February 2012 redacted engagement letters for servicers under its  jurisdiction. OCC and the Federal Reserve also released the  remediation framework for consultants to use that provided examples of  situations in which compensation or other remediation is required for  financial injury due to servicer errors, misrepresentations, or other  deficiencies.
Despite these disclosures, some stakeholders perceived gaps in key  information about how the file reviews were conducted. Regulators  released documents, such as the redacted engagement letters and  remediation framework, which generally described the design and  intended outcomes of the foreclosure review, but they did not disclose the  more detailed guidance and tests consultants relied on to perform their  reviews. As previously discussed, third-party consultants developed  thousands of test questions to determine error and harm, and regulators  issued a number of guidance documents to promote consistency among  the reviews and clarify issues raised by consultants, such as how  consultants were to construct their sample populations and how to  address issues related to borrowers covered by SCRA. Although  regulators released their remediation framework and provided answers to  frequently asked questions on remediation categories and calculations,  they did not release any additional guidance documents nor did they  publicly disclose consultants’ test questions, which according to OCC,  contained proprietary and supervisory information.
To increase the transparency and credibility of the foreclosure review for  borrowers, policy makers, and the public, among other stakeholders,  consumer groups recommended that regulators release such information.  According to consumer groups, without such information, the public would  have questions and doubts about how the reviews were executed. OCC  and the Federal Reserve staff said that they considered releasing  additional guidance to the public, but both regulators refrained from doing  so because of concerns that releasing detailed information risked  disclosure of confidential or proprietary information. Moreover, test  questions developed by consultants were numerous and complex, and  Federal Reserve staff stated that review processes were too dissimilar to  provide a comprehensive summary.
section_title: Borrowers and the General Public Received Limited Information about the Progress of Reviews
paragraphs: Borrowers who requested reviews under the foreclosure review process  initially received limited information about the status of their individual file  review. Borrowers received a letter acknowledging their request was  received, but some did not receive updates until almost a year after the  outreach program was launched, when they received a letter informing  them of the continuing nature of the review. In letters to OCC and the  Federal Reserve, consumer groups indicated that these borrowers were  frustrated by the lack of information on their particular file review. Eligible  borrowers could submit requests as early as November 2011. According  to OCC staff, borrowers who submitted an accepted request-for-review  through June 30, 2012 received a status update letter in September 2012.  OCC staff said that the letters communicated to borrowers that their  requests were being reviewed but that the results of the review might not  be available for several more months. They said that the letters also  provided a brief summary of the foreclosure review process, an Internet  link to the interagency remediation framework, and notice of other help  available through nonprofit organizations approved by the Department of  Housing and Urban Development. Regulators indicated that additional  status letters would be sent to borrowers with outstanding requests-for- review. Before the regulators halted the foreclosure review, draft letters to  be sent to borrowers on the results of their file reviews were in  development. However, regulators were still uncertain about specific  information they would require be shared with both borrowers who would  receive remediation and those who would not. Regulators have  acknowledged the importance of transparency, but when announcing the  agreements that led to the amended consent orders, they had not yet  determined what information to convey beyond that which was included in  their press releases and public websites, nor had they determined  whether additional information would be provided to borrowers who  submitted a request-for-review.
During the foreclosure review process, OCC released two interim reports  that provide the public with information on the organization and conduct of  the file review process and preliminary results, such as the number of  requests-for-review received, for institutions it supervises. The OCC  reports—issued in November 2011 and June 2012—summarized the  status of actions taken to correct deficiencies in mortgage servicing and  foreclosure processing identified in the April 2011 consent orders,  including activities related to the foreclosure review. For example, the  June 2012 report included the number of files selected for review, the  number of requested reviews, and the number of reviews completed,  among other items. These reports, according to OCC, were intended to  build transparency in the process. The Federal Reserve did not issue  interim reports on the foreclosure review process for institutions it  supervised. According to Federal Reserve staff, they did not do so  because they determined that their public release of servicers’ action  plans provided sufficient information about how servicers were  addressing the requirements of the consent orders and their public  release of servicers’ engagement letters provided sufficient information  about how the foreclosure review would be conducted. Prior to the  announcement of the agreements that led to the amended consent orders  and ended the foreclosure review for most servicers, OCC staff told us  they had planned to release a final report on the results of the foreclosure  review. After the agreements were announced, Federal Reserve staff  indicated they expected to publish additional relevant information related  to the foreclosure review and the agreements. However, as of February  2013, regulators had not decided what specific information will be made  available on the work conducted under the foreclosure review prior to the  agreements.
section_title: The Foreclosure Review Highlights the Importance of Communication to Transparency
paragraphs: While OCC and the Federal Reserve acknowledged the importance of  transparency in the foreclosure review process, the absence of timely and  useful communications at certain stages of the process—for individual  borrowers as well as the general public—hindered transparency and  undermined public confidence in the processes and results. In a  December 2011 letter to consultants, OCC noted the importance of public  confidence in the foreclosure review process. According to the Federal  Reserve, the agency endorsed OCC’s letter. However, as previously  discussed, regulators did not publicly release detailed information that  described how consultants were to determine errors and remediation, and  borrowers and the general public received limited information about the  status of the reviews. As a result, consumer groups raised concerns  about the level of transparency of the foreclosure review process and  indicated that the absence of public information undermined credibility  and public confidence in the process.
Our internal control standards state the importance of relevant, reliable,  and timely communications within an organization as well as with external  stakeholders. As illustrated in Treasury’s implementation of the Troubled  Asset Relief Program (TARP), external communications can include  posting information on its website and regular, public reporting. Our  previous work on TARP described the importance of public reporting as a  means to improve transparency and address potential questions of  whether similarly situated borrowers are being treated fairly. For  example, Treasury periodically issued public reports on the progress and  performance of its TARP housing programs. These reports have provided  information to a range of stakeholders, including Congress and the  general public. Similarly, consumer groups recommended that regulators  provide regular, public reports on the progress and findings of the  foreclosure review to increase transparency. Consumer groups also  suggested that regulators provide additional information to address  transparency-related issues for borrowers eligible for remediation through  the foreclosure review. For example, they indicated that borrowers should  have had access to information about the status of the review of their file  and receive a thorough explanation of how decisions were reached. In  addition, they recommended that the review guidelines issued by the  regulators be made public, similar to Treasury’s release of guidance for  HAMP. According to consumer groups, these actions, among others,  could have increased the public’s understanding of the reviews and  allowed borrowers to ensure their files were properly reviewed. The  foreclosure review, unlike TARP and TARP-funded programs such as  HAMP, was not a government program. However, as OCC described, it  was part of a larger set of government-directed actions in response to the  housing and mortgage crises. More publicly disclosed information about  processes and regular reporting about the status of the reviews would  have increased transparency and thereby public confidence in the  reviews, given that one of the goals regulators articulated for the  foreclosure review was to restore public confidence in mortgage markets.
section_title: The Foreclosure Review Experience Could Offer Lessons for the Amended Consent Order Activities and Continuing Reviews
paragraphs: The foreclosure review revealed three key lessons that could help inform  regulators’ implementation of the amended consent orders: (1) designing  project features during the initial stages of the process to influence the  efficiency of file reviews, (2) monitoring progress to better ensure the goal  of achieving intended results, and (3) promoting transparency to enhance  public confidence. These key lessons on planning and implementation  could help contribute to an effective process for distributing direct  payments and other assistance as prescribed by the amended consent  orders between servicers and regulators. In addition, these lessons could  inform the foreclosure review process that is continuing for the servicers  that did not reach agreements with regulators.
subsections: section_title: Sound Planning of Project Design Features Is Key to Efficiency
paragraphs: The foreclosure review experience suggests that a planning process to  determine key project features, such as guidance and necessary data  elements, for activities conducted under the amended consent orders  could lessen the risk of changes to the planned activities, future delays, or  rework. Our work on designing evaluations, including financial audits, has  found that systematic and comprehensive planning enhances the quality,  credibility, and usefulness of the results and contributes to a more  effective use of time and resources. We found that one of the first steps  in designing a review should be to define the purpose and scope of the  review, including defining what data will be collected and what  comparisons will be made. In addition, evaluation questions should be  clear and specific, and should use terms that can be defined and  measured so that the purpose and scope are readily understood and  feasible. Our prior work establishing a lessons-learned process also has  found that assessing and using lessons learned from previous experience  can provide a powerful method of ensuring that beneficial information is  factored into the planning and work processes of future activities. Key  practices of assessing lessons learned include collecting and analyzing  information on prior activities and applying that information to future  activities.
As the foreclosure review experience suggests, some implementation  challenges were inevitable due to the unprecedented nature of the  review. Nevertheless, the broad and expanding scope of the reviews and  delays in defining key concepts could have been mitigated by more  advanced planning from regulators, resulting in more efficient and  effective reviews. The April 2011 consent orders provide a general  description of the scope of the file reviews consultants were to conduct.  According to third-party consultants, these parameters resulted in a broad  review scope that generally covered all instances of noncompliance with  applicable law and servicing guidelines as they related to the post-default  residential mortgage loan borrower experience, including instances of  noncompliance that resulted in financial harm to borrowers requiring  remediation. Regulators may have missed opportunities to potentially  narrow and refine the project scope—for example, through earlier  definition of a harmed borrower or agreement on errors not resulting in  remediation that may not have warranted additional review. Changes to  guidance also expanded the scope of the reviews. For example, as we  noted previously, changes to loan modification guidance contributed to an  expanded scope for those reviews and delays in completing the work,  including, in some cases, redoing file reviews.
According to regulator staff, developing the file review process was  iterative; they learned as the reviews progressed, and regulators used  that knowledge to help refine the review process. In our work on  designing evaluations and audits, we recognize that a review process can  be iterative and that the scope and activities of the review may change as  work progresses and data limitations or new information arise.  Nevertheless, conducting a planning process that involves all  stakeholders provides an opportunity to examine preliminary information  and pilot-test processes and procedures to help further define the scope  of potential activities and hedge against the risk of future changes. In  addition, assessing lessons learned by using project critiques and  discussions with key participants and stakeholders—such as local  examination team staff, third-party consultants and law firms, and external  groups—could identify the root causes of strengths and weaknesses of  the foreclosure review that could apply to the amended consent order  activities. According to regulator staff, they are meeting with examination  staff and third-party consultants to discuss challenges with the servicers’  data that may make it difficult for servicers to determine borrowers’ direct  payment amounts under the amended consent orders. OCC staff also  said that they had discussions with other federal agencies and consumer  advocacy groups before announcing the agreements that led to the  amended consent orders.
As regulators prepare to implement the amended consent orders and  compensate borrowers, they may encounter delays and inconsistencies  similar to those associated with the foreclosure review if they miss  opportunities to make key project planning decisions. Regulators told us  that they anticipated borrowers would be contacted by the end of March  2013 and that they also expected the issuance of checks to begin in April.  However, regulators still need to make some key decisions about these  activities in order to meet their goal of beginning borrower payments in  April 2013, and clear guidance in several areas will facilitate the process.
According to regulator staff, prior to contacting borrowers each  servicer will assign borrowers to direct payment categories. OCC  staff said that the servicers were instructed to use objective data and  definitions and guidance from regulators in placing borrowers in the  categories and that the examination teams would use the guidance  provided to servicers to conduct their validation process. Regulator  staff said that they agreed that clear guidance on the categorization  and file review process and validation by examination teams are tools  that will help the servicers and examination teams navigate this  complex process.
In most cases, servicers, with regulators’ approval, have engaged the  third-party consultants to review borrowers’ files in two categories  (SCRA and foreclosed borrowers who were not in default) to  determine whether borrowers experienced those specific types of  harm. According to one third-party consultant, at the time of the  agreements that led to the amended consent orders, consultants were  waiting on additional guidance from regulators to complete aspects of  these reviews.
In addition, regulators have not yet determined payment amounts for  the different categories of borrowers. Regulator staff said that they  could not make these decisions until servicers had completed their  categorization process and regulators knew the number of borrowers  in each category to allow them to divide up the total payment amount  among the borrowers.
Regulators also have not determined what to do with any funds that  might be left because borrowers refused payments, did not cash their  checks, or could not be located. According to regulators, any  remaining funds will not be returned to the servicers.
Further, regulators told us that the agreements specify that over the  next 2 years servicers are required to take loss mitigation and  foreclosure prevention actions for which each servicer will earn credit  toward fulfilling a specified obligation. These activities will provide  assistance to borrowers covered under the consent orders and other  borrowers. Regulator staff said they provided a list of eligible activities  to servicers and advised them to prioritize borrowers covered under  the consent orders for assistance. However, according to OCC staff,  they did not provide servicers with criteria for identifying borrowers to  whom they will offer mortgage assistance, which would help ensure  that eligible borrowers have a consistent opportunity to be considered  for these funds.
Without assessing past lessons learned and making decisions on key  features of the amended consent order activities in advance, regulators  risk having to implement changes in the planned activities or publicly  announced timelines, which could decrease the efficiency of the process  to distribute direct payments and other assistance.
section_title: Monitoring Activities May Help Ensure Achievement of Goals
paragraphs: The foreclosure review experience suggests that regulators’ process for  monitoring third-party consultant, servicer, and examination team  activities, including holding regular meetings and reviewing weekly  progress reports, could provide a useful model for monitoring activities  under the amended consent orders. According to regulators, examination  teams will play a critical role in overseeing servicer activities under the  amended consent orders, including testing and validating the results of  the servicers’ categorization of borrowers. Regularly collecting and  reviewing information on the activities and approaches of the examination  teams could provide an opportunity to identify challenges and take steps  to address them as the activities progress. Similarly, instituting a process  to monitor the progress of the servicers’ loan categorization and track the  payment administrator’s distribution of payments could help regulators  assess the extent to which they are on target to reach their goal of  providing notification to borrowers about their payment by the end of  March 2013. OCC staff said that they continue to hold regular meetings  with the servicers and examination teams to answer questions and share  ideas. In addition, OCC staff told us that they intend to conduct site visits  with each servicer, review servicers’ draft categorizations, and implement  a reporting mechanism to oversee servicers’ activities under the amended  consent orders.
Regulators’ experience with the foreclosure review suggests that  identifying comparative oversight mechanisms to promote consistency  could help achieve consistent results among key actors. As discussed  earlier, regulators had limited and unsystematic centralized control  mechanisms to monitor consistency among the foreclosure review  processes and did not have the information to assess the implications of  any differences. According to regulators, achieving consistent results for  borrowers, so that similarly situated borrowers receive similar payment  amounts, is a goal of the amended consent orders, as it was of the  foreclosure review process. OCC staff stated that the direct payments  provided under the amended consent orders will likely be more consistent  than what would have occurred under the foreclosure review because  servicers are using a standard framework and objective criteria to  categorize borrowers and all borrowers in a particular category will  receive the same payment amount. According to regulator staff, the  categorization instructions provided to servicers, regular meetings with  servicers and examination teams, site visits, and examination team  verification processes will provide opportunities to review and discuss the  results of each servicer’s categorization of borrowers. However, whether  there is a similar process that will compare results across OCC and  Federal Reserve supervised servicers is unclear. Furthermore, to what  extent regulators will assess the implications of any inconsistencies  among the reviews that consultants are continuing to conduct for the  servicers that did not sign agreements with regulators is unknown.
Applying lessons from the foreclosure review process and our internal  control standards to the amended consent order activities would suggest  that mechanisms to centrally promote consistency and monitor  agreement activities could help achieve consistent results for borrowers.  GAO’s internal control standards state that agencies should take steps to  comprehensively identify and analyze program operations to determine if  risks exist to achieving goals—such as risks to the regulators’ goal of  providing similar results for similarly situated borrowers. In our prior  work, we found that using a horizontal review mechanism is an option to  help mitigate risks of inconsistent results for activities conducted by  multiple entities. For example, comparing servicer decision-making  processes, including any criteria used by the servicers to categorize  borrowers, could identify any potential differences and the extent to which  these differences may result in different direct payment decisions for  similarly situated borrowers. Similarly, mechanisms to provide clear and  specific guidance—such as guidance on testing and validation of servicer  activities—for local examination teams to use in their oversight of  individual servicer activities could help regulators to more easily monitor  and compare servicer activities and the results for borrowers among the  reviews. Without using mechanisms to centrally monitor the consistency  of servicers’ activities to categorize borrowers, regulators may risk delays  in providing direct payments to borrowers and inconsistent results. In  addition, without monitoring potential inconsistencies in the foreclosure  reviews that are continuing for servicers that are not party to the amended  consent orders, regulators will not have the information to assess whether  those servicers’ borrowers are being treated consistently.
section_title: Communication Is Key to Providing Transparency and Enhancing Public Confidence
paragraphs: Lessons from foreclosure review activities conducted to date suggest that  developing and implementing an effective communication strategy that  includes public reporting goals could enhance the transparency of the  activities under the amended consent orders. GAO’s internal control  standards state the importance of relevant, reliable, and timely  communications within an organization as well as with external  stakeholders. As a means to strengthen communication with external  stakeholders and improve transparency and accountability, our work on  TARP has underscored the importance of a communication strategy.  Moreover, our prior work on organizational transformation demonstrates  that public and private-sector leaders view establishing a communication  strategy as especially crucial in the public sector, where policymaking and  program management demand transparency and stakeholders are  concerned not only with what results are to be achieved, but also with  which processes are to be used to achieve those results. In addition, as  previously discussed, public reporting is also a mechanism for external  communication that can enhance transparency. Experiences with current  government initiatives that are aimed at assisting struggling homeowners  and involve institutions and mortgage-related issues similar to those of  the foreclosure review also highlight the benefits of regular performance  reporting. Specifically, periodic reports on the performance of and  participation in TARP programs and scheduled reports on servicers’  compliance with requirements of the National Mortgage Settlement are  intended to promote transparency and build public confidence. Because  the foreclosure review and the subsequent activities under the amended  consent orders—like TARP and the National Mortgage Settlement—are  part of the larger governmental response to the housing and mortgage  crises, a communication strategy which incorporates plans for periodic  public reporting may enhance transparency in the distribution of direct  payments and other assistance and help restore confidence in the  mortgage market.
Regulators announced the agreements that led to the amended consent  orders without a clear communication strategy. As such, what information  will be provided to individual borrowers and the general public about  processes, progress, and results of activities under the amended consent  orders is unclear. Although OCC and the Federal Reserve have provided  some information on the amended consent orders and have plans to  release additional information, regulators have not made key decisions on  communicating directly with individual borrowers and the extent to which  they will report on activities related to the amended consent orders and  continuing foreclosure reviews. Regulators provided limited information  on the amended consent orders through press releases and updates on  their websites, among other ways. For example, OCC and the Federal  Reserve issued joint press releases announcing the agreements and  related amended consent orders, and OCC and the Federal Reserve  posted answers to frequently asked questions on their websites on the  agreements that led to the amended consent orders.
In addition, regulators plan to release information about the distribution  plan after payment amounts are determined, but staff did not describe  plans to release additional information about the procedures servicers are  using to categorize borrowers. As of January 2013, OCC staff said that  they planned to release two public reports, one in April 2013 to discuss  the direct payment process, and one in the summer of 2013 to discuss  the foreclosure review results of servicers not covered under the  amended consent orders. OCC staff told us, however, that they have not  decided on the specific content of these reports. As of February 2013, the  Federal Reserve also plans to issue public reports about servicers’  activities under the amended consent orders and the results of the  foreclosure reviews at servicers not subject to the amended consent  orders. Regulators had not decided what, if any, information will be made  available on the results of the work conducted under the foreclosure  review prior to the agreements. Further, OCC and Federal Reserve staff  told us they had not made decisions about the form and specific content  of communications, if any, directly to individual borrowers.
While the amended consent orders terminate the foreclosure review for  most of the servicers, transparency of past and current efforts continues  to be important to stakeholders, including Congress and consumer  groups. In particular, members of Congress have expressed concerns  about the lack of public reporting on the foreclosure review and a lack of  information about how the amounts of payments and other assistance in  the amended consent orders were determined. In addition, consumer  groups expressed concerns about transparency and urged regulators to  take additional steps to increase transparency and public confidence in  the implementation of the amended consent orders, including robust data  collection and reporting. In the absence of a clear communication strategy  to direct external communications, including public reporting and direct  communication with individual borrowers, regulators face risks to  transparency and public confidence similar to those experienced in the  foreclosure review process.
section_title: Conclusions
paragraphs: The foreclosure review was intended to identify as many harmed  borrowers as possible, ensure that similarly situated borrowers received  similar results, and help restore public confidence in the mortgage  market. Ultimately, the complexity of the foreclosure reviews and  limitations in regulators’ guidance and monitoring of the foreclosure  review challenged their ability to achieve the stated goals. OCC and the  Federal Reserve took a number of steps to foster consistency of this  unprecedented review, including issuance of nearly identical consent  orders and joint issuance of guidance documents for third-party  consultants. However, other efforts related to guidance and monitoring  may have exacerbated the challenges and complexities inherent in the  process. In particular, existing guidance on sampling was ambiguous,  leading to inconsistent sampling methodologies used by consultants, and  did not include key oversight mechanisms to facilitate assessment of the  extent to which consultants had identified as many harmed borrowers as  possible. In addition, regulators’ limited monitoring of consistency of the  consultants’ sampling methodologies and review processes increased  risks that similarly situated borrowers would receive different results. As a  result, the regulators risked not achieving the intended goals of identifying  as many harmed borrowers as possible and treating similarly situated  borrowers similarly, and this remains a challenge for the servicers  continuing the foreclosure review. Our prior work has identified practices,  such as assessing progress toward goals and designing such monitoring  during the planning stage of a project, as effective management  practices. In addition, OMB has found that in planning data analysis  activities, such as sampling, agencies should take necessary steps to  ensure that they have collected the appropriate data from which to draw  conclusions. Assessing the review processes of the continuing reviews  for consistency and implementing a mechanism to assess the sufficiency  of additional sampling activities could help mitigate risk of similarly  situated borrowers receiving different results and facilitate oversight of the  extent to which consultants have reached as many harmed borrowers as  possible.
Although the regulators have terminated activity related to the foreclosure  review for the servicers with amended consent orders, the foreclosure  review process offers an opportunity for the regulators to leverage this  experience to help ensure that similar difficulties are better addressed in  future efforts. In general, identifying, assessing, and using lessons  learned can help ensure that beneficial information is factored into the  work processes of future activities, among other things. Therefore,  consideration of lessons from the foreclosure review activities, such as  advance design of data analysis activities and assessment of  inconsistencies in the file review processes, is one way to help regulators  improve the clarity of their guidance and ensure effective monitoring of  progress toward results for the activities under the amended consent  orders and the three remaining servicers still subject to the foreclosure  review requirement that will cover 450,000 eligible borrowers. Specific  activities include the following.
Sound planning and additional oversight mechanisms can enhance  project design and help ensure achievement of program goals. Our  internal control standards on monitoring, risk assessments, and  consultations with stakeholders, as well as generally accepted project  management practices for agencies to use in implementing the  Government and Performance Results Act and related management  initiatives, emphasize the importance of planning, monitoring, and  assessing lessons learned. Assessing the strengths and weaknesses  of the foreclosure review process by using project critiques and  discussions with key participants and stakeholders—such as local  examination team staff, third-party consultants, law firms, and external  groups—could help ensure better design and more efficient  implementation of activities under the amended consent orders. As  such, consideration of lessons from the foreclosure review processes  would enhance the design, implementation, and oversight of the  activities under the amended consent orders.
The foreclosure review activities to date also highlight the importance  of effective communication. We found that limited communication with  individual borrowers and the general public hindered transparency  and public confidence in the reviews. Further, regulators announced  agreements that led to the amended consent orders and ended the  foreclosure review with 11 servicers without a clear communication  strategy, and the information that will be provided to borrowers and  the general public remains unclear. Our internal control standards  state the importance of external communication, and our key practices  in implementing transformations that were identified by public and  private sector leaders underscore the importance of a communication  strategy. As such, the development and implementation of an  effective communication strategy could enhance regulators’ efforts to  ensure transparency and public confidence in the results of the  foreclosure review as well as processes to implement the activities  under the amended consent orders.
section_title: Recommendations for Executive Action
paragraphs: We are making three recommendations to the regulators:  (1) To better ensure that the goals of the foreclosure review are realized  for servicers that are not subject to amended consent orders, we  recommend that the Comptroller of the Currency and the Chairman of the  Board of Governors of the Federal Reserve System, as appropriate,  improve oversight of sampling methodologies and mechanisms to  centrally monitor consistency, such as assessment of the implications of  inconsistencies on remediation results for borrowers in the remaining  foreclosure reviews.  (2) To better ensure that the goals of the amended consent orders related  to the distribution of direct payments and other assistance are realized,  we recommend that the Comptroller of the Currency and the Chairman of  the Board of Governors of the Federal Reserve System identify and apply  lessons from the foreclosure review process, such as enhancing planning  and monitoring activities to achieve goals, as they develop and implement  the activities under the amended consent orders.  (3) To better ensure transparency and public confidence in the activities  under the amended consent orders and results of the continuing  foreclosure reviews, we recommend that the Comptroller of the Currency  and the Chairman of the Board of Governors of the Federal Reserve  System develop and implement a communication strategy to regularly  inform borrowers and the public about the processes, status, and results  of the activities under the amended consent orders and continuing  foreclosure reviews.
section_title: Agency Comments and Our Evaluation
paragraphs: We requested comments on a draft of this report from OCC and the  Federal Reserve. OCC and the Federal Reserve provided written  comments that are presented in appendixes II and III, respectively. The  regulators also provided technical comments, which we have  incorporated into the report, as appropriate. In commenting on the report,  OCC and the Federal Reserve both identified actions that they have  taken or planned to implement the recommendations.
Specifically, OCC stated that it plans to continue to ensure that its  sampling guidance is used in the continuing reviews and that the agency  plans to monitor for consistency. As we discussed in the report,  developing and using objective measures to monitor and assess  consistency among the continuing reviews is also important. With respect  to the second recommendation, OCC stated that it plans to apply lessons  learned from the foreclosure review to the activities under the amended  consent orders. In response to the third recommendation, OCC noted that  it recognized the importance of providing additional information to the  public about the processes, status, and results of the continuing reviews  and activities under the amended consent orders. As such, OCC said it  plans to issue at least two public reports, as we noted in the draft report.
The Federal Reserve stated that it plans to continue to coordinate with  OCC to provide consistent guidance during the continuing reviews. As we  discussed in the report, developing and using objective measures to  monitor and assess consistency among the continuing reviews is  important. With respect to the second recommendation, the Federal  Reserve stated that it has expanded its planning and monitoring efforts  during the course of the foreclosure review and plans to continue to  devote resources to planning and monitoring as it implements the  amended consent orders. In responding to the third recommendation, the  Federal Reserve outlined a number of steps that it and OCC are taking to  communicate about the continuing foreclosure reviews and activities  under the amended consent orders. These steps, such as developing a  letter to explain why borrowers are receiving a payment and updating  borrowers covered under the continuing reviews who submitted a  request-for-review, will help provide information to affected borrowers.  Moreover, the Federal Reserve and OCC said that they have committed  to providing public reports that detail the implementation of the  agreements. They also said that they anticipate these reports will include  information about the findings of completed reviews, the number of  requests for review, and the status of other activities under the consent  orders.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to interested congressional  committees, the Board of Governors of the Federal Reserve System,  Office of the Comptroller of the Currency, and other interested parties. In  addition, the report will be available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-8678 or evansl@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. Key contributors to this report are listed in appendix  IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The objectives of this report were to assess: (1) challenges to the  achievement of the goals of the foreclosure review, (2) the extent of  transparency in the foreclosure review process, and (3) lessons that could  be useful for activities under the amended consent orders and continuing  reviews. The scope of our work was limited to the foreclosure review at  the 14 servicers that are subject to the April 2011 consent orders. We  were in the process of reviewing various aspects of the foreclosure review  when the Office of the Comptroller of the Currency (OCC) and the Board  of Governors of the Federal Reserve System (Federal Reserve)  announced agreements with 11 of the 14 servicers to discontinue the  foreclosure review and replace it with a broad payment process.
To assess challenges to achieving the goals of the foreclosure review  process, we identified three areas of the review process to use as a basis  for our analysis. Specifically, we analyzed consultants’ application of  guidance to conduct the reviews and recommend remediation;  consultants’ development of test questions related to loan modifications,  state foreclosure laws, and fee reasonableness; and consultants’  sampling methodologies for selecting files to review. Based on our prior  work on the foreclosure review, these were areas we identified as  particularly challenging for ensuring consistent reviews. To obtain  information on consultants’ development of test questions and application  of regulatory guidance, we reviewed the guidance and conducted site  visits and in-person interviews with five consultant engagement teams. To  identify third-party consultants to interview, we selected consultants that  were engaged to conduct reviews for servicers that are overseen by each  regulator. In addition, we selected consultants engaged by servicers with  a range in sizes of eligible population for the review, including some of the  largest servicers. During the site visits we also observed demonstrations  of the file review process and systems. In addition, we obtained  responses to a standardized questionnaire from all of the consultants. We  compared consultants’ processes for compiling reference materials,  developing test questions, and applying guidance. We reviewed  documents regulators used to monitor the file reviews, such as status  reports and meeting agendas. We also interviewed regulator staff and  examination teams on steps taken to promote and assess consistency in  the reviews and the effects of any differences. To obtain information on  third-party consultants’ sampling plans, we used a data collection  instrument created by our statisticians. We analyzed information  contained in engagement letters between the servicers and consultants  for the sampling parameters consultants used to select files. We  confirmed key observations of our analysis in interviews and site visits  with officials responsible for developing the sampling plans at five third- party consultant engagement teams, interviews with regulator staff, and  through examination teams that reviewed the plans. In addition, we  obtained information on consultants’ plans for additional analytical  methods and confirmed other observations of our analysis through a  standardized written questionnaire to consultants. We compared the  information and these parties’ actions to criteria such as the regulators’  standard practices, stated goals for the foreclosure review, and our  internal control standards. In addition, we reviewed our prior work on the  Troubled Asset Relief Program (TARP), Home Affordable Modification  Program, and reports where we established criteria on lessons-learned  procedures, generally accepted project management practices, and  effective management practices. We also referred to our sampling  standards and several other references on sampling standards from  OCC, the American Institute of Certified Public Accountants (AICPA), and  the Office of Management and Budget (OMB). We considered these  practices applicable to the foreclosure review because they concern the  use of data from samples to draw inferences about the populations from  which the samples are drawn. To further evaluate and asses the sampling  plans, we also compiled a list of common sampling terms based on the  work cited above, as well as other analyses, and vetted the definitions for  these terms internally with our methodological staff, including staff  responsible for sampling procedures related to financial audits and  program audits. Any data we obtained on the number of borrowers in the  scope of the reviews or the status of the reviews were used for  background purposes only and were not used to support our findings and  conclusions. As such, we obtained information from regulator staff and  the data administrator about how the data were obtained and compiled,  but we did not assess the reliability of the data.
Statistical sampling can be a powerful tool for drawing inferences about  populations when a full census of cases is infeasible. Well designed and  executed samples allow researchers to make estimates of population  characteristics and to specify uncertainty due to sampling error  associated with those estimates. Parameter estimates and associated  measures of precision (such as confidence intervals or margins of error)  are developed from the sample data using estimation formulas consistent  with the sampling plan actually used. For example, if the sample design  were complex (such as using differing probabilities of selection for  different portions of the population) and if estimation formulas for simple  random sampling were used to produce estimates, those estimates and  the associated confidence intervals would likely be incorrect.  Additionally, estimates projected to a population of interest may be  subject to bias when a sample fails to cover the population of interest,  such as when the list used for sampling (the sampling frame) excludes  relevant units or when data from only a portion of the sample are  collected.
The design of an appropriate sample is intricately linked with the goals of  the sample. These goals can include the importance of establishing the  estimate within a narrow confidence interval, the need to conduct  statistical testing against a threshold or for differences, the ability to make  estimates to subpopulations, and the desire to report specific results. A  sample that is designed to test against a specified tolerance level may  use different sample sizes and a different approach to statistical testing  than a sample that is designed to estimate the level of an attribute in the  population. Some uncertainty is implicit in sampling as a tradeoff for such  factors as the cost and time required to examine all of the data. However,  according to AICPA guidelines, sampling is inappropriate if there is no  tolerance for risk of possible erroneous decisions as a result of examining  only a sample of the data.
For example, figure 2 demonstrates that these two consultants interpreted  the “precision” requirement differently. Consultant A interpreted precision  as being the threshold for the tolerable error rate, whereas Consultant B  interpreted this as requiring a margin of error within plus or minus 3  percentage points of the estimate. Because Consultant A is testing  against a threshold using the methodology specified in the OCC  handbook on sampling methodologies (based on a Poisson distribution),  it uses the upper bound of its one-sided 95 percent confidence interval to  draw inferences about whether the population error rate is less than an  established threshold value. Consultant B, in contrast, uses a two-sided  95 percent confidence interval because it is developing point estimates.  Although both consultants achieve the 95 percent reliability and 3 percent  precision guidance from regulators, the consultants’ different approaches  to sampling resulted in different sample sizes and different decision rules  for when additional sampling or full review of a population would be  required, even when the same number, or even proportion, of errors are  found in the sample. For example, if 2 errors were found in Consultant A’s  sample, one could conclude with 95 percent confidence that the error rate  in the population was below 6.3 percent and that the 3 percent precision  threshold had not been met. Additional sampling would be required in this  case. If 2 errors were found in Consultant B’s sample, one could conclude  with 95 percent confidence that the error rate in the population is between  0.1 and 1.9 percent and no additional sampling would be required. If the  proportion of errors in Consultant B’s sample was similar to Consultant  A’s at approximately 2 percent, or 8 errors, one could conclude with 95  percent confidence that the error rate in the population was between 1  and 4.3 percent, a much narrower estimate than that from Consultant A.
Neither of the samples illustrated in figure 2 was designed with the goal of  estimating the number of harmed borrowers in the population. However,  had samples of this size been drawn with the purpose of estimating the  number of harmed borrowers, the width of the two-sided 95 percent  confidence intervals around each estimate would vary. Assuming a  population size of 100,000 cases and a binomial distribution, and that  each loan with an error corresponds to one harmed borrower, samples  similar to those in figure 2 would result in different ability to concisely  estimate the number of harmed borrowers. If two errors were found in a  sample of 100 files, a 95 percent confidence interval for the estimated  number of harmed borrowers in the population would run between  approximately 240 and 7,040 borrowers. For a sample of 370 cases, a  similar proportion of errors (8 errors in a sample of 370) would result in a  95 percent confidence interval around the estimated number of harmed  borrowers between approximately 940 and 4,220, a much narrower  confidence interval. To effectively use a baseline sample from the full  population of loans to monitor whether the foreclosure reviews have  identified the majority of harmed borrowers would require setting a  realistic expected error rate to calculate sample size and establishing  guidelines for an appropriate margin of error around the estimate to  ensure that sample estimates were sufficiently precise to meet regulators’  goals.
To assess the extent of transparency in the foreclosure review process,  we reviewed press releases and documents from regulators related to the  foreclosure review. In particular, we reviewed what documents related to  the consent orders were available on the regulators’ websites, such as  speeches by agency officials, engagement letters, outreach materials,  and press releases, and analyzed the content of these documents. We  compared this documentation against agency policies on public  disclosure of enforcement action information and our previous work on  transparency in government programs, such as our work on TARP to  identify any similarities and differences. Further, we reviewed reports and  summary documentation from the National Mortgage Settlement and  interviewed the monitor of the settlement to provide the context of a  current example of a large-scale settlement involving similar stakeholders  and issues similar to those of the foreclosure review. We also conducted  interviews with regulator staff, selected third-party consultants, and  consumer groups.
To identify lessons learned that might be useful for the activities under the  amended consent orders, we compared our findings from the first two  objectives with the processes and goals regulators outlined for the  activities under the amended consent orders and identified areas with  similar challenges. We reviewed press releases on the agreements and  amended consent orders from the regulators and interviewed regulator  staff about the goals and rationale for the agreements and their plans for  designing and implementing the activities under the amended consent  orders. We also reviewed our internal control standards, our prior work  identifying key practices during agency transformations, our prior work  covering TARP and other government programs and agencies, and  documents related to the National Mortgage Settlement. In particular, we  focused on information in these reports on project design, oversight of  progress, and communication with stakeholders. Although the foreclosure  review and the activities under the amended consent orders are not  government programs, we considered the steps identified in this work as  applicable because planning, monitoring, and communication are key  principles of any effective process.
We conducted this performance audit from July 2012 through March 2013  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Office of the Comptroller of the Currency
section_title: Appendix III: Comments from the Board of Governors of the Federal Reserve System
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Karen Tremba (Assistant  Director), Bethany M. Benitez, John Karikari, Charlene J. Lindsay,  Patricia MacWilliams, Marc Molino, Jill Naamane, Anna Maria Ortiz,  Robert Rieke, Jennifer Schwartz, Andrew Stavisky, Sonya Vartivarian,  James Vitarello, and Monique Williams made key contributions to this  report.