Federal Reserve: Additional Actions Could Help Ensure the Achievement of Stress Test Goals
Published: 2016-11-15T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Federal Reserve has two stress test programs for certain banking institutions it supervises. DFAST encompasses stress tests required by the Dodd-Frank Act. CCAR comprises a qualitative assessment of firms' capital planning processes and a quantitative assessment of firms' ability to maintain sufficient capital to continue operations under stress. Questions have been raised about the effectiveness and burden of requiring two stress test programs. GAO was asked to review these programs and their effectiveness. This report examines how the stress test programs compare, the CCAR qualitative assessment, and the design of the stress test scenarios and models.
GAO analyzed Federal Reserve documents including rules, guidance, and internal policies and procedures on DFAST and CCAR implementation and assessed practices against federal internal control standards and other criteria. GAO also interviewed Federal Reserve staff and officials of 19 banking institutions selected based on characteristics such as their size, prior stress test participation, and history of CCAR results.
section_title: What GAO Found
paragraphs: The Board of Governors of the Federal Reserve System (Federal Reserve) has two related supervisory programs that involve stress testing but serve different purposes. Stress tests are hypothetical exercises that assess the potential impact of economic, financial, or other scenarios on the financial performance of a company. Stress tests of banking institutions typically evaluate if the institutions have sufficient capital to remain solvent under stressful economic conditions.
The Dodd-Frank Wall Street Reform and Consumer Protection Act (Dodd-Frank Act) implements statutory stress test requirements, known as the Dodd-Frank Act Stress Tests (DFAST) for Federal Reserve-supervised banking institutions with more than $10 billion in total consolidated assets. DFAST projects how banking institutions' capital levels would fare in hypothetical stressful economic and financial scenarios. It applies to a broad range of banking institutions and consists of supervisory- and company-run stress tests that produce capital adequacy information for firms' internal use and for public disclosure.
The Federal Reserve also conducts a Comprehensive Capital Analysis and Review (CCAR), which uses DFAST information to assess the capital adequacy (a quantitative assessment) and capital planning processes (a qualitative assessment) for bank holding companies with total consolidated assets of $50 billion or more. CCAR generally does not require additional stress tests and uses the same data, models, and projections used for DFAST.
While the primary purpose of DFAST is to produce and disclose comparable information on the financial condition of banking institutions (the stress test results), the Federal Reserve uses CCAR to make supervisory assessments and decisions about the capital adequacy plans (including proposed capital actions such as dividend payments) of large bank holding companies. For example, the Federal Reserve may object to a company's plan if stress test results show the company's post-stress capital ratios (regulatory measures that indicate how much capital is available to cover unexpected losses) falling below required minimum levels or if the Federal Reserve's qualitative assessment deems the firm's capital planning and related processes inadequate. An objection can result in restrictions on a firm's capital distributions. Several of the companies GAO interviewed that are subject to Federal Reserve stress tests identified benefits from the tests (such as overall improvements in risk management and capital planning) and also identified costs (including for staff resources and other expenditures necessary to conduct the tests and meet the Federal Reserve's supervisory expectations).
GAO found limitations in the Federal Reserve's stress test programs that could hinder their effectiveness.
Qualitative assessment disclosure and communication. The Federal Reserve uses a framework with multiple levels of review to assess qualitative CCAR submissions that helps ensure consistency, but it has not disclosed information needed to fully understand its assessment approach or the reasons for decisions to object to a company's capital plan. Transparency is a key feature of accountability and such incomplete disclosure may limit understanding of the CCAR assessments and hinder public and market confidence in the program and the extent to which the Federal Reserve can be held accountable for its decisions. Federal internal control standards state the importance of relevant and timely communications with external stakeholders. The Federal Reserve has not regularly updated guidance to firms about supervisory expectations and peer practices related to the qualitative assessment. For example, it has not published observations of leading capital planning practices used in CCAR since 2014. The limited communication can pose challenges to companies that must meet these expectations annually and could hinder the achievement of CCAR goals.
Scenario design. The Federal Reserve has a framework for designing stress test scenarios but its analysis of some key design decisions has been limited. For example, the Federal Reserve has not conducted analyses to determine whether the single severe scenario it uses for the supervisory stress test is sufficient to accomplish DFAST and CCAR goals. While there are advantages to using one scenario—including simplicity and transparency—many different types of financial crises are possible, and the single selected scenario does not reflect a fuller range of possible outcomes. Without additional analysis, the Federal Reserve cannot be reasonably assured that banks are resilient against a range of future risks. The Federal Reserve also has not explicitly analyzed how to balance the choice of severity—and its influence on the resiliency of the banking system—with any impact on the cost and availability of credit, which could limit its ability to avoid undesired economic effects from scenario design choices.
Model risk management. Federal Reserve supervisory guidance for banking institutions states that risk from individual models and also from the aggregate system of models should be managed. The Federal Reserve makes supervisory decisions based on the results of its own stress test models, but its management of model risk—the potential for adverse consequences from decisions based on incorrect or misused model outputs—has not focused on its system of models that produce stress test results. To estimate the effect of stress test scenarios on companies' ability to maintain capital, the Federal Reserve has developed individual component models that predict a company's financial performance in the scenarios. The results of these component models are combined with assumed or planned capital actions of companies and form the system of models used by the Federal Reserve. The Federal Reserve has an oversight structure for developing and using models in the supervisory stress tests but its own risk-management efforts have not targeted the system of models. For example, it has not conducted sensitivity and uncertainty analyses—important elements in the Federal Reserve's model risk management guidance—of how its modeling decisions affected overall results. Without such a focus, the Federal Reserve's ability to effectively evaluate and manage model risk and uncertainty from the entire system of stress test models will be limited. Understanding and communicating this uncertainty is critical because the outcome of the CCAR assessment can have significant implications for a company, including limiting its capital actions (such as dividend payments and share repurchases).
section_title: What GAO Recommends
paragraphs: GAO is making 15 recommendations to help improve the effectiveness of the Federal Reserve's stress test programs, such as improving disclosures and communications to firms, considering the potential consequences of its scenario design choices, and expanding model risk management to include the entire system of models. The Federal Reserve generally agreed with the recommendations and highlighted select ongoing and future efforts.

section_title: Letter
paragraphs: The 2007–2009 financial crisis threatened the stability of the U.S.  financial system and revealed weaknesses in the capital adequacy and  risk-management practices of U.S. banking institutions. Concerns from  market participants and observers about whether banking institutions had  sufficient capital to absorb potential losses and limited information on  institutions’ exposures also contributed to undermining public and market  confidence in the stability of the financial system. Identified shortcomings,  such as a lack of firm-wide stress testing, led banking institutions and  their regulators to reassess capital requirements, risk-management  practices, and other aspects of bank regulation and supervision. A stress  test is a hypothetical exercise for assessing the potential impact of  economic, financial, or other scenarios on a company’s financial  performance.
To help address such concerns and restore market confidence in the  banking system, the Board of Governors of the Federal Reserve System  (Federal Reserve) and other federal banking regulators conducted a  stress test in 2009 to determine whether the 19 largest U.S. banking  institutions had enough capital to survive a further economic shock and  continue lending activities. The Federal Reserve designed the 2009  stress tests—known as the Supervisory Capital Assessment Program  (SCAP)—as a one-time exercise to identify institutions with insufficient  capital. It required firms found to have capital shortfalls to raise specific  amounts of additional capital from private markets or, if they were unable  to do so, to receive capital investments from the Department of the  Treasury (Treasury).
The SCAP exercise was followed by the Federal Reserve’s  Comprehensive Capital Analysis and Review (CCAR), conducted  annually since 2011. The Federal Reserve designed CCAR to assess the  capital adequacy and internal capital planning processes of large and  complex bank holding companies (all the institutions that participated in  SCAP participated in the initial CCAR cycle). CCAR also uses a capital  stress test as input to quantitative assessments of an institution’s capital  adequacy under stress, after making its planned capital actions (such as  dividend payments and stock repurchases), and emphasizes qualitative  assessments of the institution’s capital planning processes.
Between SCAP and the first CCAR, the Dodd-Frank Wall Street Reform  and Consumer Protection Act (Dodd-Frank Act) was enacted in 2010.  Among its regulatory reforms, the Dodd-Frank Act requires the Federal  Reserve to conduct an annual stress test (supervisory stress test) of bank  holding companies with total consolidated assets of $50 billion or more.  The act also requires each of these companies and all other Federal  Reserve-supervised banking institutions with more than $10 billion in total  consolidated assets to conduct their own stress tests (company-run  stress test). In October 2012, the Federal Reserve initially adopted rules  implementing the Dodd-Frank Act Stress Tests (DFAST). The Federal  Reserve completed the initial DFAST exercise in 2013—it conducted  supervisory stress tests of the 18 bank holding companies that had  participated in SCAP, and these companies and their state member bank  subsidiaries performed company-run stress tests. The Federal Reserve  rules delayed implementation for other companies, not part of SCAP, but   subject to DFAST until 2014 or later.
The number of companies participating in CCAR—which only includes  firms with total consolidated assets greater than $50 billion—has grown to  33 bank holding companies for the 2016 CCAR cycle. These institutions’  holdings represent more than 80 percent of the total assets of the U.S.  banking system. Many additional institutions also have been required to  complete stress tests as part of DFAST. As the programs have grown,  some market participants and observers have raised questions about the  programs’ effectiveness and the costs and consequences for institutions  subject to the tests.
You asked us to review the Federal Reserve’s CCAR and DFAST  programs. This report (1) compares DFAST and CCAR and discusses  company and Federal Reserve views about the programs’ costs and  benefits; (2) examines the CCAR qualitative assessment, including the  extent of communication and disclosure; (3) examines how the Federal  Reserve designs the supervisory scenarios for the stress tests; and (4)  examines the Federal Reserve’s modeling process for the stress tests.
To examine how the Federal Reserve’s stress test exercises compare,  we reviewed the Dodd-Frank Act; the final and amended capital plan and  stress test rules; Federal Reserve guidance documents and instructions,  methodology, and results publications related to DFAST and CCAR;  supervisory letters on stress testing and capital planning; public  statements by Federal Reserve officials; and other Federal Reserve  documentation about how it has implemented and used DFAST and  CCAR in its supervision of banking institutions. We collected information  and documentation on stress test extensions and exemptions from the  Federal Reserve, the Office of the Comptroller of the Currency (OCC),  and the Federal Deposit Insurance Corporation (FDIC). We also  interviewed staff from the Federal Reserve, OCC, and FDIC. To obtain  views on the stress tests and their costs and benefits, we judgmentally  selected and interviewed 13 companies that participated in CCAR in 2015  and 6 companies that were subject only to DFAST. The companies were  selected based on their size, industry type, organization type, prior stress  test participation, and history of CCAR results. We also reviewed Federal  Reserve statements on benefits and costs. To examine how the Federal  Reserve designs stress test scenarios and its modeling process, we  collected and reviewed public and nonpublic Federal Reserve  documentation including DFAST- and CCAR-related publications, internal  guidance and procedures, policy statements, model documentation,  model validation reports, and presentations. We interviewed Federal  Reserve staff from across the Federal Reserve System, including those  involved with scenario design and supervisory stress test model  development and validation. We also interviewed officials from the Bank  for International Settlements and International Monetary Fund (IMF) about  general stress testing practices, the use of stress tests in prudential  supervision, and the organizations’ reviews of the Federal Reserve’s  stress tests. We reviewed documentation from these and other  organizations such as the Basel Committee on Banking Supervision.
To examine how the Federal Reserve completes the CCAR qualitative  assessment and the extent of communication and disclosure, we  reviewed the Federal Reserve’s capital plan and stress test rules; CCAR  instructions and results publications; documentation of qualitative  assessments conducted in 2014 and 2015; internal program guidance,  policies, and procedures related to the qualitative assessments; and  guidance, feedback, and other communication with CCAR companies.  We also interviewed the companies we judgmentally selected for our first  objective about their experience with the qualitative assessment and  Federal Reserve communication. For more information on our scope and  methodology, see appendix I.
We conducted this performance audit from December 2014 to November  2016 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: Stress Test Types and Purposes
paragraphs: Stress testing is one of many risk-management tools used by both  financial institutions and regulators. Complex financial institutions need  management information systems, internal controls, and other processes  that can help to identify, assess, and manage a range of risks across the  organization that may arise from both internal and external sources,  including rapid and unanticipated changes in financial markets.
Stress testing has been used throughout the financial industry for several  decades. But as noted in a Federal Reserve Bank of New York staff  report, stress testing before the recent financial crisis was seen as one of  many risk-management tools and was not a major component of banking  regulators’ supervisory programs. Since the recent financial crisis, the  report explains that comprehensive firm-wide stress testing has become  an integral and critical part of firms’ internal capital adequacy assessment  processes and of banking regulators’ supervisory regimes. The expanded  role of supervisory stress testing is discussed later in this report.
IMF has identified four major categories of stress testing, differentiated by  purpose or goals: (1) internal risk management, used by firms to manage  risks from their investment or asset portfolios and as an input for business  planning; (2) crisis management, used by supervisors to assess whether  institutions need additional capital during times of financial sector  distress—such as with SCAP—and as an input for business restructuring  plans; (3) microprudential (supervisory), used by supervisors to assess  the health of an individual institution; and (4) macroprudential  (surveillance), used by central banks and other authorities to analyze  system-wide risks and vulnerability in addition to institution-specific risks.  As discussed later in this report, the Federal Reserve’s CCAR and  DFAST have elements of the internal risk management, microprudential,  and macroprudential approaches.
section_title: Federal Banking Regulators
paragraphs: Federal banking regulators supervise the activities of banking institutions  and require them to take corrective action when the institutions’ activities  or overall performance present supervisory concerns or could result in  financial losses to FDIC’s deposit insurance fund or violations of law or  regulation. See table 1 for an overview of their functions.
section_title: Bank Capital
paragraphs: For banking institutions, capital exists to absorb unexpected losses and  the amount of capital an institution holds is critical to its ability to continue  operating by making loans to businesses and consumers. The Federal  Reserve, FDIC, and OCC require institutions to maintain certain minimum  levels of capital to promote stability across the banking industry and  protect the nation’s financial system. These requirements identify various  types of regulatory capital including common equity Tier 1 capital,  additional Tier 1 capital, Tier 2 capital, and total capital. According to  Federal Reserve staff, common equity tier 1 capital is considered the  most significant capital that a banking institution can have to support its  operations and absorb unexpected financial losses. It consists primarily of  retained earnings (the profits a bank has earned but has not paid out to  shareholders in the form of dividends or other distributions) and common  stock, with deductions for items such as goodwill and deferred tax assets.  Tier 2 capital contains supplementary capital elements such as  subordinated debt, a portion of loan loss reserves, and certain other  instruments. Total capital consists of the sum of Tier 1 and Tier 2 capital.
Regulators establish required capital levels in comparison with various  measures of an institution’s assets and the minimum requirements are  specified as a ratio (regulatory capital ratio). Regulators use different  ratios to assess an institution’s capital adequacy. Among these are the  Tier 1 risk-based capital ratio, which measures Tier 1 capital as a share of  risk-weighted assets, and the Tier 1 leverage ratio, which measures Tier  1 capital as a share of average total consolidated assets. Other  measures include the total risk-based capital ratio (total capital as a share  of risk-weighted assets) and the common equity Tier 1 ratio (common  equity tier 1 capital as a share of risk-weighted assets).
section_title: Federal Reserve’s Stress Test Programs Are Coordinated but Serve Different Purposes
paragraphs: The DFAST and CCAR programs vary in terms of the firms to which they  apply and in their uses. DFAST applies to a broad range of banking  institutions and consists of supervisory- and company-run stress tests to  generate forward-looking information about institutions’ capital adequacy  for the firms’ internal use and for public disclosure. The Federal Reserve  uses CCAR (which builds on information from DFAST) to quantitatively  and qualitatively evaluate the capital adequacy and capital planning  processes of large bank holding companies. The Federal Reserve and  other bank regulators have issued similar rules for DFAST company-run  stress tests as required by the Dodd-Frank Act but have differed in their  use of exemptions. Several of the companies subject to Federal Reserve  stress tests that we interviewed identified a range of benefits from the  tests and many companies described costs as well as factors contributing  to those costs, although several companies did not track costs specifically  related to the tests.
subsections: section_title: Under DFAST, the Federal Reserve and Subject Institutions Project Capital Levels under Common Economic Scenarios and Disclose Comparable Information
paragraphs: The Federal Reserve and a broad range of banking institutions use  DFAST to (1) project how hypothetical adverse scenarios would affect an  institution’s revenues and losses and ultimately its capital levels as  measured by regulatory capital ratios, and (2) disclose comparable  information on test results. The Federal Reserve’s primary goals for  DFAST are the production of capital adequacy information for firms’  internal use and for public disclosure. The Federal Reserve aims to  provide subject companies, the public, and supervisors with forward- looking information to help gauge the potential effect of stressful  economic and financial conditions on the companies’ ability to absorb  losses and continue operations. Federal Reserve staff we interviewed  explained that the purpose of the public stress test disclosures was  transparency and the promotion of market discipline by providing market  participants with comparable information on the financial condition of  banking institutions. The Federal Reserve also intends banking  institutions to incorporate stress testing into their internal capital planning  activities.
subsections: section_title: Subject Firms and Test Components
paragraphs: DFAST consists of supervisory- and company-run stress tests that are  required by the Dodd-Frank Act and are based on a banking institution’s  size and type (see table 2). Federal Reserve-supervised banks and  holding companies with more than $10 billion in total consolidated assets  must perform company-run tests. The Federal Reserve also conducts a  supervisory stress test for bank holding companies with total consolidated  assets of $50 billion or more. Banking institutions with more than $10  billion in total consolidated assets that are supervised by FDIC and OCC  are subject to stress test rules issued by the agencies that also require  the completion of annual company-run stress tests.
For the supervisory tests, the Federal Reserve uses data provided by the  companies and a set of models developed or selected by the Federal  Reserve. Companies use their own data and models to complete the  company-run tests.
The supervisory stress tests performed by the Federal Reserve and the  company-run stress tests conducted by DFAST firms share several key  components necessary for producing post-stress capital ratios. These  ratios, which are an important output of the stress tests, reflect projections  of risk-weighted assets and balance sheet and income statement items  under the stress scenarios and measure the amount of capital a banking  institution has available to cover unexpected losses. For example, the  Federal Reserve’s DFAST stress test rules generally require both types of  tests to include, over a nine-quarter planning horizon, projections of revenues,  regulatory capital ratios.
Other areas in which the tests share some common elements include  data, assumptions, and scenarios. As stated previously, companies use  their own data to complete company-run tests and report similar data to  the Federal Reserve that it uses to perform the supervisory tests. The  Federal Reserve’s stress test rules also prescribe a standard set of  assumptions that are used in both types of tests. The assumptions  involve capital actions, or decisions about transactions affecting capital  levels, such as raising capital by issuing new capital instruments or  returning capital to shareholders through dividend payments or share  repurchases. These transactions can affect the outcome of regulatory  capital ratios measured by the stress tests.
As required under the Dodd Frank Act, the Federal Reserve also annually  defines three stress test scenarios—baseline, adverse, and severely  adverse—that it uses for the supervisory stress test and requires DFAST  firms to use in the annual company-run tests. The scenarios consist of  hypothetical projections for 28 macroeconomic and financial variables.  For instance, the variables include measures of the unemployment rate,  gross domestic product, housing and equity prices, interest rates, and  financial market volatility.
Baseline scenario. Generally reflects economic conditions expected  by economic forecasters.
Adverse scenario. Features mild to moderate economic and financial  stress driven by selected potential risk factors.
Severely adverse scenario. Features severe economic and financial  stress, generally driven by a different set of risk factors than the  adverse scenario.
The Federal Reserve’s rule requires certain elements to be included in  public summaries of DFAST results. Pursuant to the Dodd-Frank Act,  Federal Reserve rules for DFAST require subject banking institutions to  submit their stress test results to the Federal Reserve and also to publicly  disclose a summary of the results of the severely adverse scenario. The  rules require the summary from the severely adverse scenario to include  (1) a description of the types of risks and methodologies included in the  stress test; (2) estimates of aggregate losses, pre-provision net revenue,  provision for loan and lease losses, net income, and projected regulatory  capital ratios; and (3) an explanation of the most significant causes for the  changes in regulatory capital ratios. The Dodd-Frank Act also requires  the Federal Reserve to disclose a summary of its supervisory stress test  results. The Federal Reserve has done so in the aggregate and  individually for each of the banking institutions subject to the supervisory  tests and the disclosures have included similar types of information as  required for DFAST institutions.
The use of standard approaches in the supervisory and company-run  stress tests enhances the comparability and usefulness of public results  disclosures. Specifically, the common capital action assumptions, stress  scenarios, and nine-quarter planning horizon used in DFAST allow for  more consistent capital adequacy assessments, while also allowing for a  focus on the particular characteristics of different institutions. For  example, as the Federal Reserve explained in its 2016 DFAST  supervisory stress test results, differences in loan loss rates across  institutions reflect differences in the risk characteristics of the portfolios  held by each institution (both in relation to the type of lending of each  portfolio and the loans within each portfolio). Stress test rules issued by  FDIC and OCC for their supervised institutions also include the use of  common scenarios and test horizons.
section_title: Uses
paragraphs: The Federal Reserve uses DFAST results to supplement its ongoing  supervision and inform its CCAR evaluations. The Federal Reserve  follows two different approaches that are based on an institution’s size  and type.
The Federal Reserve conducts supervisory stress tests of the largest  bank holding companies (those with at least $50 billion in total  consolidated assets). The Federal Reserve then uses the supervisory  and company-run stress test information as a basis for quantitative  and qualitative CCAR evaluations. As we discuss in more detail later  in the report, CCAR is a comprehensive assessment of a company’s  capital adequacy and capital planning processes.
For all other DFAST institutions (holding companies and banks), the  Federal Reserve does not conduct a supervisory test. Rather, it  uses company-run stress tests to supplement its regular, ongoing  supervision. For example, internal Federal Reserve guidance states  that examiners are expected to assess the quality of a firm’s stress  testing process and overall results as part of the broader assessment  of a firm’s capital adequacy and risk-management process.
In addition, the Federal Reserve performs targeted DFAST examinations  that consider how $10 to $50 billion institutions are completing DFAST  stress tests and using the information they produce as part of their risk- management and capital planning processes. According to Federal  Reserve documentation, in these examinations staff assess whether  institutions that are not subject to CCAR have sound practices for  meeting the Federal Reserve’s requirements and expectations for DFAST  company-run stress tests. The examinations are structured around the  requirements in the Federal Reserve’s DFAST company-run stress test  rules, the standards in its final DFAST supervisory guidance for  institutions with $10 billion–50 billion in assets, and the data reporting  requirements in the reporting form for those companies. Staff from  regional Federal Reserve Banks perform the targeted DFAST  examinations. Federal Reserve staff we interviewed said that examiners  performing DFAST examinations also conducted general supervisory  examinations.
According to Federal Reserve staff and examiner guidance, DFAST does  not represent a separate supervisory assessment, and the Federal  Reserve does not make supervisory decisions based solely on a  company’s stress test processes or results. Instead, for banking  institutions that are not subject to CCAR, the Federal Reserve considers  the results of the DFAST examinations and annual company-run stress  tests as one of several factors influencing an institution’s supervisory examination rating. For example, the Federal Reserve’s internal  guidance instructs examiners to consider the tests as one of many tools  available to assist in the assessment of a company’s capital position and  planning process and not to rely primarily upon a firm’s internal stress test  results in assessing overall capital adequacy or risk management.  Furthermore, Federal Reserve staff we interviewed said that DFAST was  part of the overall supervisory framework for these firms and provided  additional information for examiners to consider when assessing capital  planning and other areas of supervisory focus. Federal Reserve guidance  and staff also indicated that there were no DFAST-specific expectations  for companies to meet minimum capital levels or any associated  regulatory approvals (such as for proposed capital distributions).
section_title: CCAR Uses DFAST Stress Test Results and Plays a Larger and More Direct Role in Supervision
paragraphs: CCAR is a separate exercise in which the Federal Reserve uses  information produced in DFAST as a key input to its supervisory  evaluations of a subset of firms—large bank holding companies with total  consolidated assets of $50 billion or more. The Federal Reserve’s goals  for CCAR are to ensure that large bank holding companies have sufficient  capital to withstand severely adverse economic and financial conditions  and continue operations, and have strong processes for assessing their  capital needs and managing their capital resources.
subsections: section_title: Subject Firms and Test Components
paragraphs: CCAR applies only to a subset of DFAST firms—the largest top-tier bank  holding companies (with total consolidated assets of $50 billion or more)  subject to the DFAST supervisory stress test (see table 3). It does not  affect the other DFAST institutions—that is, certain banks and savings  and loan holding companies with more than $10 billion in total  consolidated assets or bank holding companies with total consolidated  assets greater than $10 billion but less than $50 billion.
CCAR represents a comprehensive and independent supervisory  evaluation and includes a quantitative assessment of a firm’s capital adequacy, and a qualitative assessment of its capital planning processes and capital  policies.  (In this section, we largely focus on the quantitative assessments; we  discuss the qualitative assessment in more detail later in this report.)
Federal Reserve rules promulgated in conjunction with its stress test  requirements call for CCAR institutions to submit annual capital plans to  the Federal Reserve that include detailed descriptions of the company’s  internal processes for assessing capital adequacy, its policies governing  capital actions, and planned capital actions over the nine-quarter planning  horizon.
section_title: Quantitative Assessment
paragraphs: In the quantitative assessment, the Federal Reserve evaluates whether a  company would be able to make its planned capital distributions and meet  minimum capital requirements throughout the stress period based on both  supervisory and company-run stress test results. The Federal Reserve  coordinates key aspects of the DFAST stress tests and the CCAR  quantitative assessment such as the stress scenarios, planning horizon,  and reporting requirements and time frames (see fig. 1). For example,  companies submit information related to the annual DFAST stress tests to  the Federal Reserve at the same time that they provide their capital plan  information for CCAR. The inclusion of capital actions occurs after  projecting revenues, losses, and net income, as one of the last steps in  generating post-stress capital ratios.
For the quantitative assessment, the Federal Reserve uses essentially  the same data, models, and projections from the DFAST supervisory  stress test to calculate post-stress capital ratios for each CCAR firm. But  one key distinction exists—for CCAR, the Federal Reserve uses a  company’s proposed capital actions rather than the standard ones  prescribed for DFAST. As with the supervisory tests, a company can use  the same data, models, and projections from DFAST for the CCAR  company-run stress tests, but with its planned rather than standard  capital actions. As Federal Reserve staff explained, the line items relating  to capital actions are the only difference between the stress tests used for  DFAST and CCAR. According to these staff, standardized capital assumptions are used to project capital ratios in DFAST because the  purpose of the supervisory stress test under DFAST is to estimate and  disclose comparable capital adequacy information, while proposed  actions are used for CCAR, which also evaluates companies’ planning  processes.
In addition to the change in assumptions for capital actions, CCAR firms  also conduct a stress test using at least one company-designed stress  scenario that is specific to CCAR and not publicly disclosed. That is,  companies still use the standard supervisory scenarios (baseline,  adverse, and severely adverse) and develop another scenario that also  would represent stressful conditions. The Federal Reserve requires  CCAR firms to focus this additional scenario on the specific vulnerabilities  of the company’s business activities and exposures. As such, company- run tests using the company-designed scenario can provide greater  insight into firm-specific risks than company-run tests only using the  standard supervisory scenarios. The Federal Reserve has said that it  considers the results of the company-run tests—under both supervisory  scenarios and the company designed scenario—in its quantitative  assessment.
section_title: Inclusion of Company-Run Test Results in the CCAR Quantitative Assessment May Weaken Incentives for Severe Stress Tests
paragraphs: The Federal Reserve has stated that its goals for CCAR include ensuring  that companies have sound capital planning and risk-management  processes. Related to these goals, the Federal Reserve’s capital plan rule  requires companies to use the results of company-run tests—under both  supervisory scenarios and at least one stress scenario they design  (company-designed scenario)—to conduct internal capital adequacy  assessments that support their capital plans. Furthermore, CCAR  requirements call for companies to use the company-designed scenario  to stress the specific vulnerabilities of their risk profile and operations,  including those related to the company’s capital adequacy and financial  condition. The Federal Reserve uses the results of company-run tests in  performing its CCAR quantitative assessments and has stated that a  company will not receive an objection to its capital plan based on the  assessment if it can meet minimum regulatory capital requirements under  the company-run stress tests as well as the supervisory tests. However,  this could weaken incentives for companies to create meaningful and  severe stress tests that are useful for capital planning and risk  management.
Based on data we analyzed for CCAR 2013 through CCAR 2015, post- stress capital ratios for the company-run tests (under both supervisory  severely adverse scenario and the company-designed scenario) were  higher than for the supervisory tests a substantial majority of the time  (see fig. 2). That is, capital ratios declined less under the company-run  tests than under the Federal Reserve’s, indicating that the companies’  results generally were further from breaching minimum capital  requirements and thus less stressful.
Federal Reserve staff acknowledged that the inclusion of the company- run tests—specifically the test using the company-designed scenario—in  the CCAR quantitative assessment may provide an incentive for  companies to create less severe stress tests that would not generate  losses large enough to breach minimum regulatory capital requirements.  But, the staff did not believe these negative incentives warranted the  elimination of the company-run stress test and added that the Federal  Reserve could mitigate this risk through its evaluation of companies’  stress testing practices—including their scenarios and models—in the  CCAR qualitative assessment. However, stress test modeling and  scenario design involve considerable judgment and companies could  make subtle changes that would not indicate manipulation or necessarily  fail to meet Federal Reserve standards. Federal Reserve staff told us that  the company-designed scenario provided an additional view on risk that  increased the variety of stresses to which companies were subject under  CCAR.
In addition, Federal Reserve guidance for the largest and most complex  companies indicates that the firms should have more sophisticated  models for their most material portfolios than for portfolios that are less  significant. However, because the company-run tests typically produced  higher post-stress capital ratios, they may not have meaningfully  contributed to the CCAR quantitative assessment. Furthermore, in  discussing why the Federal Reserve does not require disclosure of results  based on company-designed scenarios with Federal Reserve staff, they  explained that with a requirement to disclose such results, firms might  focus on producing positive results rather than using the scenario to  genuinely identify their most salient risks. Using the company-run test  results (from supervisory and company-designed scenarios) in the  quantitative assessment creates similar conflicting incentives for  companies, which could limit the benefits of the tests and the  achievement of Federal Reserve goals.
section_title: CCAR Qualitative Assessments
paragraphs: The Federal Reserve also uses DFAST-related information in its CCAR  qualitative assessments, which we discuss in additional detail later in this  report. For CCAR firms, the Federal Reserve’s qualitative assessment  represents a dedicated and wide-ranging assessment of their capital  planning processes. A substantial part of the assessment involves  examining how companies perform stress tests—including those required  under DFAST—and incorporate them in overall risk management and  capital planning.
As part of this effort, the Federal Reserve assesses different aspects of  the processes companies use to generate the DFAST company-run and  other stress tests. Among other areas, this includes examining whether  companies meet Federal Reserve expectations related to risk  identification, scenario design, and loss and revenue estimation. For  example, the Federal Reserve considers how a company’s stress testing  practices capture the potential increase in losses or decrease in revenue  that could result from the firm’s risks, exposures, and activities under  stressful scenarios.
section_title: Semiannual Tests
paragraphs: Based on Federal Reserve Dodd-Frank Act requirements, institutions  subject to CCAR must conduct company-run stress tests semiannually.  The semiannual and annual stress tests are similar, but for the  semiannual tests the Federal Reserve requires companies to develop and  use at least three scenarios appropriate for their own risk profile and  operations. The Federal Reserve has not used these tests as part of its  annual CCAR quantitative assessment, and their role in the qualitative  assessment has been limited. According to Federal Reserve staff, the  Federal Reserve has used the semiannual tests significantly less than  other DFAST stress test components and has not done the type of  rigorous assessment of the semiannual stress test that it has done for the  annual tests. However, Federal Reserve staff and internal guidance that  we reviewed indicated that information from the semiannual tests could  be used to help identify areas requiring additional focus in future CCAR  cycles. For example, Federal Reserve staff said they performed limited  reviews of companies’ semiannual stress test submissions—including  information on stress scenarios and test results—to identify anomalies  and other insights such as structural portfolio, modeling, or scenario  changes.
section_title: Uses and Disclosure
paragraphs: CCAR plays a larger and more direct role in the supervision of subject  institutions than DFAST. For example, in its publications of CCAR results  the Federal Reserve has stated that it has made CCAR a cornerstone of  its supervision of the largest and most complex financial institutions. The  Federal Reserve uses the CCAR quantitative and qualitative  assessments to determine whether to object or not object to an  institution’s capital plan (including proposed capital actions such as  dividend payments and share repurchases that affect the firm’s capital  levels). The Federal Reserve can object to a company’s capital plan  based on the quantitative or the qualitative assessment.
Federal Reserve staff also stated that the Federal Reserve has moved  the focal point of its supervisory process for the largest firms toward the  promotion of strong capital adequacy and liquidity planning, in addition to  assessing the firms’ preparedness for recovery and resolution in  bankruptcy. The staff said that the CCAR qualitative assessment is a  primary contributor to overall supervision and a focal point for evaluating  a company’s risk-management and internal controls. They explained that  strong capital and liquidity planning requires companies to identify and  measure risks, understand how risks change in adverse economic  scenarios, and have robust internal controls and governance, all of which  are elements of the qualitative assessment under CCAR.
The Federal Reserve also has been integrating the CCAR qualitative  assessment into its regular, ongoing supervisory activities. According to  the Federal Reserve’s instructions for the 2016 CCAR cycle, it is to  conduct certain supervisory activities throughout the year that inform the  annual CCAR qualitative assessment—which allows the Federal Reserve  to consistently incorporate supervisory findings from all its examination  work into the overall qualitative assessment. For example, Federal  Reserve staff we interviewed said that if their supervision identified  weaknesses in a company’s internal audit functions, these weaknesses  would be relevant to the internal controls and governance assessments  within CCAR. Federal Reserve staff explained that as part of the  integration of the CCAR into year-round supervision, the Federal Reserve  has established teams of subject-matter experts from across the Federal  Reserve System to link their work with CCAR. For example, the Federal  Reserve has already gathered subject-matter experts in modeling loss  revenue to contribute to CCAR throughout the year.
The Federal Reserve publishes firm-specific CCAR results separately  from its DFAST disclosures. Since it initiated CCAR in 2011, the Federal  Reserve has increased disclosure of firm-specific CCAR results, including  its supervisory assessments. After the initial CCAR cycle, the Federal  Reserve published only an overview of its objectives and methodology  and did not reveal whether it had objected or not objected to an  institution’s capital plan. The Federal Reserve currently discloses firm- specific capital ratio projections (based on supervisory stress test results  and firms’ proposed capital actions) and its decision to object or not  object based on the quantitative or qualitative assessment. The Federal  Reserve also discloses the reasons for any qualitative objections.
section_title: Regulators Developed Comparable Stress Test Rules, but OCC Has Made Greater Use of its Supervisory Flexibility
paragraphs: The Federal Reserve and other bank regulators issued similar company- run stress test rules but have used different degrees of supervisory  flexibility to implement them, with OCC granting the most extensions or  exemptions for firms. In addition to the Federal Reserve, the Dodd-Frank  Act called for certain financial regulatory agencies, including OCC and  FDIC, to issue rules requiring the financial companies they supervise with  more than $10 billion in total consolidated assets to conduct stress  tests. OCC and FDIC rules apply to certain banks and savings  associations, some of which may have a holding company subject to the  Federal Reserve’s stress test rules. The act required the agencies to  coordinate their stress test rules so that they are consistent and  comparable. In 2012, the Federal Reserve, OCC, and FDIC each  adopted substantively similar rules implementing this requirement. For  example, the agencies’ rules have similar reporting time frames, public  disclosure requirements, and stress test methodologies and practices.
However, OCC has made greater use of supervisory flexibility in  implementing the stress test requirements. Each agency’s rule allows for  the acceleration or extension of time frame requirements, while OCC’s  and FDIC’s rules also include provisions for them to modify any or all of  the stress test requirements. Furthermore, OCC staff stated that the  agency had interpreted the Dodd-Frank Act as not requiring all banking  institutions within a bank holding company to conduct separate stress  tests. But according to information from the Federal Reserve and FDIC,  banking institutions covered by their rules are required to conduct stress  tests regardless of their status within a holding company structure.
Each of the agencies extended the time frames for a limited number of  firms.  As shown in table 4, the Federal Reserve provided five firms with  extensions of time—ranging from 3 to 12 months—to conduct and report  the results of required company-run stress tests.
FDIC provided a 1-year extension to two firms for the first stress test  cycle in 2013, as shown in table 5.
OCC has granted more stress test extensions than FDIC and the Federal  Reserve and is the only agency to have approved exemptions from stress  test requirements. OCC issued 14 1-year extensions—including one firm  that received 3 consecutive extensions—and 2 shorter extensions for  stress test cycles from 2013 to 2016 (see table 6). Three firms were  granted stress test exemptions for one or more of the stress test cycles  from 2013 to 2016 including a firm that had been exempted for three  cycles. The firms OCC exempted were part of a larger banking  organization in which both the parent holding company and affiliated bank  were required to conduct stress tests.
The reasons cited for granting extensions and exemptions varied among  the regulators. The time extensions issued by the Federal Reserve and  FDIC generally occurred in the initial stress test cycles to allow additional  time for firms to implement effective stress testing systems and compile  necessary data. Most of the Federal Reserve’s extensions were technical  in nature. One firm reduced its total consolidated assets below the $10  billion threshold and represented that it intended to maintain its assets  below the threshold for the foreseeable future. Others involved mergers,  including an institution that had made a business transaction based on  the timing set forth in the Federal Reserve’s original stress test rules,  which were subsequently revised resulting in the institution being subject  to the requirements sooner than expected. Federal Reserve and FDIC  staff stated that they based their determinations for granting extensions  on applicable regulatory provisions and the relevant facts and  circumstances of the particular case. OCC also issued extensions to  provide institutions that had not previously been subject to regulatory  stress test requirements with additional time to construct or enhance their  stress testing frameworks. OCC staff noted that such institutions lacked  the necessary infrastructure to complete a stress test submission at the  time its final rule was implemented.
OCC granted extensions and exemptions from stress test requirements  for different reasons. For example, according to OCC staff and  documents we reviewed, OCC granted extensions and exemptions based  on the extent of an institution’s activities, its relative size within a large  bank holding company, and if the larger entities in the holding company  structure—such as the lead bank and parent holding company—were  subject to stress test requirements. The bases for the extensions and  exemptions also included OCC’s conclusions about the satisfactory  nature of an institution’s financial condition and capital levels and the  nature of its business activities and strategies.
Our prior work on financial regulatory reform identified an important  characteristic of consistent financial oversight—that similar institutions  and risks, among other areas, should be subject to consistent regulation,  oversight, and transparency—to help minimize negative competitive  outcomes while harmonizing oversight. Without a consistent approach to  the implementation of the stress test rules, regulators may not be  regulating financial institutions that pose similar risks similarly, which  could contribute to competitive disadvantages between institutions and  inconsistent oversight of risk management. Additionally, financial  institutions often have options for how to operate their business which  affects who will be their regulator. For instance, banks can change their  charters if such a change will allow them to have a regulator perceived as  having less stringent regulations.
section_title: Firms Generally Indicated That Stress Tests Offered Important Benefits but Also Required Substantial Resources
paragraphs: Several companies subject to DFAST and CCAR that we interviewed  identified a range of benefits but also described significant costs related  to these exercises. We discussed the Federal Reserve’s stress test  exercises with 19 bank holding companies, including 13 of the 31  companies that participated in CCAR in 2015. We also reviewed  Federal Reserve statements and interviewed staff about costs and  benefits.
subsections: section_title: Companies Views on Benefits
paragraphs: The company officials that we interviewed generally identified overall  improvements in risk management and capital planning attributable to  DFAST and CCAR, as the following examples illustrate.
Several firms said that their prior stress test efforts were fragmented  with different business units across the firm assessing risks  independently. The firms said that Federal Reserve stress test  requirements have led to more comprehensive, enterprise-wide, and  forward-looking capital adequacy assessments, including the  identification and measurement of risks.
One firm said the consolidation of the firm’s stress testing and its  integration with capital adequacy assessments had helped the firm to  quantify its risk appetite, which was not accomplished prior to DFAST  and CCAR.
Some firms also identified key benefits from improved data quality and  capabilities, including enhanced data collection, analysis, and  reporting.
Some firms said that the stress test exercises have led to a stronger  focus on the governance of capital adequacy processes and an  increased involvement of senior management and the board of  directors in capital planning decisions.
Firms that distinguished between the benefits of different stress test  components generally said that CCAR has been more beneficial than  DFAST and that company-run tests have been more useful than the  supervisory tests.
Furthermore, several companies explained that the risk management and  capital planning improvements have provided additional benefits for the  company’s business operations. Several firms’ officials said that the  stress tests have helped improve their business decisions, including by  taking a more strategic approach to capital, developing tools to analyze  portfolio risks, and facilitating business planning and budgeting. For  example, according to officials from one CCAR firm, the stress tests have  led to better pricing decisions, strategic and investment focus, and  optimization of investor returns. In addition, one smaller company that has  been subject to DFAST but not CCAR said that it has used stress test  information in its strategic decisions about different markets and  geographical locations and incorporated information about risks from  asset portfolios in its pricing decisions.
Finally, several companies also identified broader, system-wide benefits  related to the stress tests. Some firms said that the stress tests have led  to higher capital levels and improved risk management that have  contributed to the stability of the financial system. Other firms noted that  comparable stress test results provide an industry-wide view of capital  adequacy and comparisons across companies, which can offer a broader  view of relative risks than individual firm assessments.
section_title: Companies’ Views on Costs
paragraphs: While companies we interviewed generally recognized benefits, they also  cited costs in complying with stress test requirements. Officials from many  of the companies we interviewed indicated that DFAST and CCAR have  resulted in significant costs, including for staff resources and other  expenditures.
Most firms said that the stress test-related costs have increased from  year to year or were expected to continue increasing, although some  firms noted that costs have stabilized or declined. Several firms cited  what they viewed as the Federal Reserve’s continually increasing  supervisory expectations—in particular those related to CCAR  qualitative requirements—as a main reason to expect continued  growth in stress test-related costs.
At the same time, the firms stated that they have not collected  information on the specific costs directly attributable to DFAST and  CCAR.
Firms generally said that measuring stress test-specific costs is  difficult because the tests involve many employees from around the  company who have responsibilities beyond DFAST and CCAR.
About half of the companies we interviewed provided estimates of their  stress test-related costs or staff resources, which varied widely. For  example, for the six CCAR firms that provided cost estimates, recurring  annual costs related to both DFAST and CCAR ranged from $4 million to  $7 million to more than $90 million. Half of the estimates were for $15  million to $30 million of annual costs. Some CCAR firms provided  estimates of the amount of staff resources rather than costs. These  estimates varied from about 100 staff for one company to approximately  500 employees with part-time responsibility for the stress tests and an  additional 2,000-plus employees spending part of their time supporting  the stress tests for another company. In addition, more than a third of the  companies said that they used consultants—often to a significant  extent—to help complete the work required for DFAST and CCAR. For  the non-CCAR firms that we interviewed, cost estimates ranged from  around $250,000 to $2 million.
Many of the companies we interviewed identified particular factors behind  what they viewed as the substantial costs required for the stress test  exercises. These included requirements related to the CCAR qualitative  assessment such as documentation of processes and controls and  supervisory expectations for model development and validation. Several  companies noted costs from integrating or upgrading stress test-related  technology and risk-management systems and the vast amounts of  detailed data needed to complete the stress tests. Many firms pointed to  expenditures for consultants to assist with their stress testing efforts.  Officials from one CCAR firm said that they hire consultants for many  tasks including modeling, documentation, and technical writing; another  firm cited the risk-identification process and model risk management as  two main drivers of consulting expenditures. Some firms also stressed  that competition among companies for qualified staff to perform stress  testing had increased the cost of individuals with quantitative skills and  modeling expertise. Two firms said that the supply of qualified employees  has been low compared to demand related to the Federal Reserve’s  stress test requirements.
In addition to direct costs, some firms indicated that they also faced costs  related to holding excess capital to ensure they did not receive an  objection from the CCAR quantitative assessment. In the CCAR  quantitative assessment, the Federal Reserve can object to a company’s  capital distribution plan if stress test results show the company’s post- stress capital ratios falling below required minimum levels. Some of the  companies said that they have held more capital than they otherwise  would to account for differences between their stress test results and the  Federal Reserve’s supervisory test results. The firms stated that limited  transparency about the Federal Reserve’s supervisory stress test models  leads to uncertainty about exactly how much capital they need to hold to  avoid an objection. The companies’ estimates of the amount of additional  capital they have held to avoid a CCAR objection ranged from about $500  million for one firm to $15 billion for another firm.
section_title: Federal Reserve Views on Benefits and Costs
paragraphs: The Federal Reserve has stated that the stress tests have provided  important benefits to the financial system and subject institutions. In  issuing the CCAR results for 2016, the Federal Reserve stated that the  increased capital levels of large bank holding companies since the  financial crisis have at least in part been due to the stress test  exercises. In addition to helping strengthen large firms’ capital positions,  Federal Reserve officials have noted other substantial contributions that  DFAST and CCAR have made to financial supervision in several key  areas.
First, officials identified improved risk management, internal controls,  and governance practices at institutions subject to DFAST and CCAR.
Second, officials noted that the stress test-related exercises have led  to a more forward-looking and stress-based approach to assessing  capital adequacy by the Federal Reserve and institutions, which  represents an improvement over previous practices focused on  traditional regulatory capital ratios (which only reflect past  performance).
Third, officials observed that the horizontal nature of the stress tests— with a simultaneous review across multiple firms—has provided the  Federal Reserve with a more consistent and industry-wide  perspective on potential risks and vulnerabilities.
Fourth, some officials identified greater supervisory transparency  associated with DFAST and CCAR, including disclosure of firm- specific information that could lead to greater market discipline and  information on the Federal Reserve’s framework and methodology  that could contribute to supervisory accountability.
According to Federal Reserve staff we interviewed, the recent financial  crisis revealed that banking institutions—including the largest and most  complex firms—had significant shortcomings and gaps in their risk- measurement and risk-management systems. These deficiencies  included limitations in the collection and use of data for risk identification  and management and in the ability to assess potential risks to the  company during periods of stress, such as a lack of data on firm risks and  exposures. The staff noted that these banking institutions needed to  make investments to improve these fundamental risk-management  capabilities because the largest banking institutions’ financial stability has  implications for the financial system and economy. The staff indicated that  costs to the firms will normalize at some point and the Federal Reserve did not expect that costs would continue to increase. They explained that  initial costs to establish the necessary data, risk management, internal  controls, governance, and stress testing capacity can be high but future  costs will be less once firms establish the needed capabilities and  processes, and required data are available. They noted that although  some of the new costs come from the stress test exercises alone, even  without the explicit DFAST and CCAR requirements, firms still would have  to take these actions and incur costs because of broader supervisory  expectations for enhancing risk management, internal controls, and  governance processes, including around capital adequacy assessments.
section_title: CCAR Qualitative Assessments Include Multiple Levels of Review, but Communication of Methodology and Expectations Was Limited
paragraphs: The Federal Reserve has identified capital adequacy principles and  established an organizational and oversight structure for assessing  qualitative CCAR submissions. The assessment framework includes  processes to help ensure consistency across evaluations. In the  qualitative assessment, the Federal Reserve uses ratings and rankings to  compare firms’ capital planning practices against supervisory  expectations. However, it has not disclosed sufficient information that  would allow for a clear understanding of its methodology. The Federal  Reserve has provided companies with information on supervisory  expectations and peer practices related to the qualitative assessment, but  the infrequent timing of these communications and evolving peer  practices can pose challenges to companies that must meet the  expectations annually.
subsections: section_title: The Federal Reserve Identified Principles and Established an Organizational Structure for Qualitative Assessments
paragraphs: As discussed previously, CCAR qualitative assessments are  comprehensive reviews of the capital planning processes and capital  policies of large bank holding companies. The Federal Reserve has  structured the evaluations for its qualitative assessments around seven  principles of an effective capital adequacy process (see table 7), which it  has identified in public guidance documents. The principles cover  different assessment topics including risk management, stress testing  practices, capital policies, internal controls, and overall governance of  capital planning.
The seven principles each represent distinct aspects of a CCAR  evaluation and, according to Federal Reserve staff, each principle could  influence others (with a deficiency in one principle often highlighting a  deficiency in another principle). For example, weaknesses across  different principles can signal a weakness in effective governance.
subsections: section_title: Organizational Structure for Qualitative Assessments
paragraphs: The Federal Reserve has established a tiered organizational structure for  its CCAR qualitative assessments, with roles and responsibilities  assigned throughout the CCAR program. Federal internal control  standards state the importance of establishing an organizational structure  and clearly assigning responsibility for key roles. According to interviews with Federal Reserve staff and our review of internal agency documents,  the Federal Reserve’s structure for completing CCAR qualitative  assessments is headed by the Director of the Division of Banking  Supervision and Regulation. The entities within the Federal Reserve that  have roles in the process range from Reserve Bank examiners to the  Board of Governors.
Evaluation teams. Teams of designated staff from across the Federal  Reserve System initially review and evaluate companies’ capital plan  submissions for the qualitative assessment. The two types of staff  teams involved are (1) on-site examination teams (supervisory on-site  teams), which consist of staff from the Reserve Bank that oversees  the firm; and (2) subject-matter experts (horizontal evaluation teams)  assigned to assess specific aspects of capital planning and stress  testing for each CCAR firm.
CCAR Executive Committee. The Executive Committee manages the  CCAR program and holds ultimate responsibility for the program’s  design and execution. The Executive Committee is chaired by a  senior officer from the Board of Governors staff and comprises senior  staff from across the Federal Reserve System, including senior staff  from the Large Institution Supervision Coordinating Committee  (LISCC) Operating Committee, Large and Foreign Banking  Organizations (LFBO) Management Group, and the Division of  Financial Stability, which monitors financial markets and analyzes  potential threats to financial stability. The Executive Committee  reviews the evaluation teams’ assessments and provides final ratings  for each principle at each company and consolidated rankings of all  companies to the LISCC Operating Committee and LFBO  Management Group. While the Executive Committee is ultimately  responsible for the CCAR program, the Executive Committee  delegates execution and administration to the CCAR Program  Oversight Group. The CCAR Program Oversight Group works with  evaluation teams during their assessments to maintain consistency  before providing conclusions to the Executive Committee.
LISCC Operating Committee. The LISCC Operating Committee is  responsible for setting priorities for and overseeing the execution of  the LISCC supervisory program—for the largest and most  systemically important financial institutions subject to Federal Reserve oversight. The Operating Committee is chaired by a senior officer  from the Board of Governors Staff and includes senior officials from  various divisions at the Board of Governors and Reserve Banks. The  LISCC Operating Committee chair reports to the Director of Banking  Supervision and Regulation. For the CCAR qualitative assessment,  the Operating Committee provides final recommendations to the  Director of the Division of Bank Supervision and Regulation to object  or not object to the capital plans of companies in the LISCC portfolio.
LFBO Management Group. The Management Group oversees the  supervision of large institutions ($50 billion or more in total assets)— including foreign banking organizations—not included in the LISCC  portfolio. In CCAR, the LFBO Management Group reviews and  provides feedback to supervisory on-site teams on their company- specific object or non-object recommendations for these firms. The  LFBO Management Group does not provide final recommendations to  the Director of the Division of Bank Supervision and Regulation.  Instead, the Reserve Bank responsible for each non-LISCC firm  determines final recommendations in consultation with Board staff.
Director of Banking Supervision and Regulation. Banking Supervision  and Regulation oversees and develops regulations for Federal  Reserve-supervised banking institutions. For CCAR qualitative  assessments, the Director of Banking Supervision and Regulation  makes the final recommendations to object or not object to the capital  plans for each firm to the Board of Governors.
Board of Governors. The Board of Governors has ultimate decision- making authority for CCAR qualitative assessment determinations.  Federal Reserve CCAR staff brief the Board of Governors on the final  recommendations approved by the Director of Banking Supervision  and Regulation. According to Federal Reserve staff, the Board  reviews all assessments but to date has only voted on whether to  implement recommendations to object or conditionally not object to a  company’s capital plan.
section_title: Scope of Qualitative Assessments
paragraphs: Federal Reserve procedures call for staff to adjust the scope of the  qualitative assessment based on a firm’s size, characteristics and the  materiality of risks it poses to the financial system. According to these  procedures, the Federal Reserve varies the scope of its reviews based on  a company’s size and complexity, so that not all companies are assessed  on every aspect of the qualitative assessment each year. However, for  LISCC companies, the Federal Reserve reviews all key aspects of their  capital planning and capital adequacy processes over the course of an  annual CCAR cycle.
For non-LISCC companies the Federal Reserve procedures dictate a risk- focused approach to identify significant aspects of a company’s capital  adequacy process that it will assess in that year’s CCAR cycle. According  to Federal Reserve staff, this approach places priority on reviews of  companies with larger risks and systemic importance by assigning  additional resources based on company characteristics. These staff also  said that risk-focused reviews help to efficiently allocate Federal Reserve  staff resources and that the focus could change from year to year based  on the Federal Reserve’s views on certain risk areas. For example, in  2016 the Federal Reserve horizontal evaluation teams reviewed only  certain areas of their assigned principles for non-LISCC companies. Staff  further explained that, when applying this approach, the Federal Reserve  instructs its staff to avoid focusing on risks that may be immaterial.
Federal Reserve staff said that the risk-focused approach influences the  staffing of evaluation teams for the qualitative assessment. For example,  horizontal evaluation teams involved in the qualitative assessment include  subject-matter experts from across the Federal Reserve System and the  risk-focused approach helps ensure an appropriate number of subject- matter experts can be assigned to the teams.
section_title: Federal Reserve Has Established Processes to Help Ensure Consistency across Qualitative Assessments
paragraphs: According to our examination of Federal Reserve documentation, the  process for qualitative assessments includes procedures, documentation,  and training intended to help ensure consistency across the reviews as  well as multiple levels of review and oversight. A number of teams are  involved in CCAR evaluations (see fig. 3).
As previously discussed, the Federal Reserve assigns two types of teams  to review qualitative submissions, supervisory on-site teams and  horizontal evaluation teams. In CCAR, supervisory on-site teams are to  assess company submissions based on evaluation principles 1, 4, 5, 6,  and 7. Supervisory on-site teams are also responsible for documenting  Federal Reserve staff recommendations as their assigned company’s  submission progresses through the assessment process and making their  own recommendations to the LISCC Operating Committee and the LFBO  Management Group.
subsections: section_title: Cross-Firm Evaluations
paragraphs: To help ensure a comprehensive and consistent evaluation across  companies, the Federal Reserve has divided specific responsibilities  among horizontal evaluation teams. A designated team or a set of teams  are to assess areas corresponding to a specific evaluation principle for all  the CCAR submissions. While both firm-specific and horizontal evaluation  teams are to perform principle-based assessments, in some cases the  horizontal evaluation teams are to provide the assessment in relation to a  specific principle and in other cases to supplement the work of on-site  teams.
The horizontal evaluation teams are broken out into risk evaluation  teams, capital and revenue assessment teams, and the capital adequacy  process review team.
Risk evaluation teams evaluate a company’s ability to measure its risk  exposures under stress and create estimates of potential losses from  these risks. The teams assess and propose ratings for elements  covered by principle 2, which encompasses six different areas of loss- estimation practices.
Capital and revenue assessment teams evaluate and propose ratings  for elements covered by principle 3, which involve a company’s ability  to effectively forecast available capital resources—including revenues  and expenses—during stress periods.
The capital adequacy process review team supplements the work of  the supervisory on-site teams. Specifically, the process review team  provides a cross-firm assessment of supervisory on-site teams’  evaluations of certain areas of capital planning—such as risk  management, capital policy, internal controls, and governance—while  also supporting the on-site teams in making their assessments.  According to CCAR program documents, the process review team  performs an oversight function by providing objective assessments of  some elements of companies’ capital plans and by evaluating the  reasonableness, consistency, and completeness of the supervisory  on-site teams’ assessments. The process review team produces its  own summary assessments of principles 1, 4, 5, 6 and 7 to provide  the Executive Committee and on-site teams with a horizontal  perspective (for instance, the range of practice in key areas of focus  across all CCAR companies).
Another team—the scenario evaluation team—assists supervisory on-site  teams by assessing the completeness and severity of companies’  internally developed stress scenarios for all CCAR firms.
Internal CCAR program documents also establish expectations and  mechanisms for teams to resolve any differences that arise during their  evaluations through communication and collaboration. The program  documents direct teams to send unresolved differences to the Executive  Committee for further and final deliberation, if needed.
section_title: Training
paragraphs: According to CCAR procedures, the CCAR Executive Committee  oversees centralized training for staff participating in the CCAR program,  which helps support a consistent approach to evaluations. Federal  Reserve staff said that all staff involved in developing and executing  CCAR must take annual training and that staff involved in qualitative  assessments participate in additional training throughout the course of  each CCAR cycle.
Training materials have included overviews of the CCAR assessment  framework, decision-making process, review processes, and  documentation requirements. Horizontal evaluation team leads and  subject-matter experts participate in additional technical training on topics  such as new modeling techniques, modeling strengths and weaknesses,  and leveraging information from past CCAR cycles, according to Federal  Reserve staff. In addition, on-site teams and other relevant staff may be  required to participate in training on principles relevant to their work as  well as lessons learned from past CCAR cycles.
section_title: Documentation
paragraphs: Both types of evaluation teams are to record their findings and  conclusions using template forms that also have instructions to further  promote consistency across different teams and evaluations. The  supervisory on-site teams and horizontal evaluation teams each are to  produce memorandums detailing conclusions related to their assigned  capital adequacy process principles. The conclusion memorandum  contains a summary of a company’s practices related to the principle  being evaluated and describes identified weaknesses and shortcomings.  For each principle, the relevant teams are to record a proposed rating (on  a scale of one to four), trend (such as stable, improving, or declining), and  an overall assessment of the team’s conclusions. In addition, the teams’  conclusion memorandums are to highlight trends in a company’s  practices compared to prior CCAR exercises and the observed range of  current industry practice.
Supervisory on-site teams are also to produce recommendation  memorandums for object and non-object decisions, which are to provide  overall conclusions and support for a team’s recommendation. The  memorandums are to describe potential new supervisory findings,  including actions the company should take to remediate issues. They also  are to provide a summary and status of outstanding CCAR-related  supervisory issues (including matters requiring attention and matters  requiring immediate attention).
section_title: Multiple Levels of Review
paragraphs: The review process is to start after the two groups of evaluation teams  have completed their assessments of each CCAR company’s capital plan  and capital adequacy processes and formally documented their  conclusions. According to CCAR procedures, the teams are to submit  their evaluations and conclusions to successive groups of senior  management for additional review, including the CCAR Executive  Committee (see fig. 4). As part of the assessment process, supervisory  on-site teams are to propose object or not object recommendations based  on the CCAR Executive Committee’s review and conclusions.
CCAR procedures call for Executive Committee staff to lead sessions  with evaluation teams to review team findings and ensure evaluations  are conducted in a consistent and comparative manner across all  companies. The Executive Committee is to review findings from these  sessions and develops overall assessments for each company. These  overall assessments are to include findings from the sessions with  evaluation teams and other CCAR information and may differ from  evaluation teams’ original conclusions. In developing its overall  assessments, the Executive Committee is to make adjustments to  reflect different supervisory expectations for companies of various  sizes and levels of complexity, according to the procedures and  Federal Reserve staff.
The supervisory on-site teams are to develop internal  recommendation memorandums based on the review sessions and  Executive Committee assessments. CCAR procedures call for  supervisory on-site teams to provide the recommendation  memorandums to the LISCC Operating Committee or the LFBO  Management Group and to the Reserve Bank responsible for the  company’s supervision.
According to CCAR procedures, the LISCC Operating Committee (for  firms in the LISCC portfolio), LFBO Management Group, and  responsible Reserve Banks (for firms not in the LISCC portfolio) are to  review the supervisory on-site teams’ recommendations, assessment  information provided by the Executive Committee, and other  information from throughout the assessment. The LISCC Operating  Committee and responsible Reserve Banks then are to make final  recommendations for the companies for which they are responsible  and provide the recommendations to the Director of Banking  Supervision and Regulation for approval.
Upon approval by the Director of Banking Supervision and Regulation,  the Board of Governors is to be briefed on all CCAR  recommendations. However, the Board has only voted on whether to  approve objection or conditional non-objection recommendations,  according to Federal Reserve staff and internal documents we  reviewed.
section_title: Qualitative Assessments Use Ratings and Rankings to Reflect Evaluation of Firms’ Capital Planning Practices
paragraphs: In evaluating a company’s capital plan and completing the CCAR  qualitative assessment, the Federal Reserve produces measurements— ratings and rankings—of the extent to which company practices meet  supervisory expectations. As discussed previously, evaluation teams are  to structure their assessments around the Federal Reserve’s seven  principles of an effective capital adequacy process. During the  assessments, teams are to evaluate companies’ current practices and  assign each individual company a numerical rating for each principle and  applicable subcomponents. The ratings are intended to measure the  extent to which a company’s capital adequacy process meets supervisory  expectations.
According to Federal Reserve program documentation, evaluation teams  base their rating assessments on established supervisory guidance and  supervisory expectations specific to capital planning. For example,  when evaluating a company’s modeling practices, Federal Reserve teams  may use prior supervisory guidance on model risk management, which is  also incorporated into the 2015 supervisory and regulation letters.  According to Federal Reserve staff, CCAR reviews have incorporated  CCAR-specific expectations and also include other long-standing  guidance on internal controls, risk management, and corporate  governance, particularly where such guidance is applicable to practices  that support capital planning. For example, these staff said that  supervisory guidance and expectations for internal controls existed before  the implementation of CCAR and that this type of guidance, which was  relevant before the current stress testing regime, is now enhanced by the  CCAR qualitative assessment.
subsections: section_title: Ratings
paragraphs: The Federal Reserve uses evaluation ratings to summarize findings  related to the different review components (organized by principle and  subcomponents) and to develop its overall qualitative assessment for  each company. Federal Reserve program guidance also states that  ratings are intended to help facilitate internal discussions around  deficiencies in a company’s capital adequacy process and serve as the  basis for making qualitative assessment determinations. Supervisory on- site teams develop overall ratings for their assigned capital adequacy  process principles. In contrast, risk evaluation teams and capital and  revenue assessment teams develop ratings for both subcomponents of  their assigned principles and a consolidated rating for the overall  principle.
The Federal Reserve’s rating system comprises four numerical scores:
1 - strong,
2 - satisfactory,
3 - fair, and
4 - unsatisfactory.
The Federal Reserve defines each score by the degree to which a  company’s practices meet supervisory expectations. The top score  reflects company practices that meet expectations and include sound,  transparent, and repeatable processes. Intermediate scores represent  practices that either generally meet or do not meet expectations.  According to Federal Reserve staff, practices rated below satisfactory  may not warrant an objection but would require remediation to avoid  future objections. An unsatisfactory rating is used for practices that do not  meet expectations, have critical deficiencies, and will require significant  corrective action. In addition, Federal Reserve evaluation teams can use  plus or minus modifiers to further differentiate the intermediate scores.  Horizontal evaluation teams can develop further detailed guidance on  rating modifiers but must adhere to general ratings guidelines. Federal  Reserve program guidance instructs evaluation teams to consider the  trend in a company’s practices relative to other CCAR companies and  whether its practices were above, consistent with, or below peer  practices. Federal Reserve internal guidance also instructs evaluation  teams to consider progress towards remediating previously identified  weaknesses when assigning ratings.
section_title: Rankings
paragraphs: The Federal Reserve also develops rankings to compare capital  adequacy practices across CCAR companies and help ensure  consistency across evaluations. According to CCAR procedures, rankings  are developed at various levels of the qualitative assessment (capital  adequacy planning subcomponents, principles, and overall).
The procedures call for horizontal evaluation teams to develop  preliminary rankings directly from assigned ratings by grouping  companies for each principle, and the Executive Committee assigns  the firms to cohorts based on their ratings and rankings.
According to Federal Reserve staff, companies within cohorts are  considered more similar in the overall quality of their practices than  they are to companies in any other cohort. Federal Reserve staff also  said that firms within a cohort are not ranked, but are generally listed  in alphabetical order.
The procedures also state that the rankings’ relative comparisons allow  the Federal Reserve to differentiate among companies that might have  the same ratings but also exhibit differences in their processes that would  allow for meaningful distinctions (that is, different rankings). Figure 5  provides a hypothetical example of the rating and ranking process.  Federal Reserve staff explained that evaluation teams establish rankings  by identifying relative strengths and weaknesses in each company’s  processes.
CCAR procedures call for evaluation teams to provide proposed ratings  and rankings to the Executive Committee based on their respective  evaluations. The Executive Committee is to review and approve the  proposed rankings (in the form of cohorts), including an overall ranking for  each CCAR firm that is to include consideration of the company’s size,  complexity, and systemic importance.
According to Federal Reserve staff, the most important role of the  rankings and cohorts is to help ensure consistency in the Federal  Reserve’s execution of the qualitative assessment. In addition, Federal  Reserve staff stated that cohorts help ensure expectations are properly  tailored for LISCC and non-LISCC firms. The staff noted that the  comparative analysis and rankings help the Federal Reserve to identify  what distinctions exist among individual CCAR firms. The staff explained  that the Executive Committee reviews evaluation teams’ findings and  supporting analyses to help determine which companies have stronger  and weaker practices. Because of the small number of ratings categories,  the staff said that a relative comparison allows Federal Reserve staff to  distinguish which firms have the best practices among those that might  have the same rating. The staff said that while rankings assist the Federal  Reserve in ensuring consistency across CCAR assessments, objection  determinations are made based on an evaluation of a company’s  practices against supervisory expectations—and not in relation to how  they compare to other CCAR firms. According to Federal Reserve staff,  even the lowest-ranked companies may not necessarily have  weaknesses in their capital planning processes that would warrant an  objection. The evaluation teams’ ratings and rankings may change after  deliberations between the evaluation teams and Executive Committee  staff.
section_title: Differentiated Expectations
paragraphs: The Federal Reserve has applied different supervisory expectations to  CCAR companies based on firm characteristics. Specifically, the Federal  Reserve has identified expectations for capital planning and capital  adequacy processes that reflect differences in a company’s size, scope of  operations, activities, and systemic importance. For example, in  December 2015 the Federal Reserve published supervisory letters  explaining how its supervisory expectations differ for two groups of CCAR  companies—the largest and most complex firms, and large and non- complex firms.
In these and other documents, the Federal Reserve has stated that it has  heightened supervisory expectations for the largest and most complex  companies in all areas of capital planning and that it expects such  companies to have leading practices (in terms of sophistication,  comprehensiveness, and robustness) for all of their portfolios and  activities compared to other CCAR companies. It has declared that  smaller and less complex companies will not be held to the same  standard.
CCAR documentation we reviewed indicated that the Federal Reserve  has used differentiated expectations for larger and more complex  companies. For example, the Federal Reserve expects complex  companies to have a formal risk-identification process with quarterly  updates and use quantitative approaches for risk management. In  contrast, the Federal Reserve expects non-complex companies to have a  less formal risk-identification process and use either qualitative or  quantitative approaches for risk management. Moreover, qualitative  assessment results reflected better relative rankings for non-LISCC  companies than LISCC companies, which the Federal Reserve attributed  in part to its higher expectations for LISCC companies. Federal Reserve  staff also explained that successful capital adequacy planning is more  difficult for LISCC companies simply because of their large size and  complexity, which together with the heightened supervisory expectations  explains much of the differences in qualitative assessment results.
According to Federal Reserve staff, similar to the Dodd-Frank Act  requirement for enhanced prudential standards, the Federal Reserve has  higher expectations for the largest and most complex companies because  problems at such firms are more likely to have negative consequences for  the financial system and economy. It has also indicated that new CCAR  companies may need time to build and implement internal systems  necessary to meet CCAR requirements.
section_title: Other Factors Influencing Determinations
paragraphs: Multiple factors influence the Federal Reserve’s final qualitative  assessment determinations. According to Federal Reserve staff,  evaluation teams propose recommendations based on multiple factors  related to (1) weaknesses identified in the qualitative assessment and (2)  the severity of the weaknesses and the likelihood that a company can  remediate weaknesses before the next CCAR review cycle. In particular,  Federal Reserve procedures instruct staff to evaluate the severity,  materiality, quantity, pervasiveness, and duration of identified process  deficiencies when considering whether supervisory findings warrant an  objection determination. In addition, objections may be due to deficiencies  identified in multiple areas or based on a single fundamental weakness if  that weakness is in an area critical to the company’s operations or if the  severity of the deficiency places the reliability of the company’s overall  capital plan into question.
Furthermore, program guidance indicates that, while all capital adequacy  principles are important, some principles reflecting foundational practices  and deficiencies in these areas may be more likely to result in an  objection recommendation than others. Federal Reserve staff said that  the Federal Reserve may decide to apply a conditional non-objection for  various reasons, including that a company has significant deficiencies in  certain areas of their capital planning processes, but would be able to  address identified deficiencies relatively quickly, depending on the nature  of the deficiency. If a company has a significant weakness in a critical  area that is not addressed over time, Federal Reserve staff noted that the  company may receive an objection even if it has tried to address the  deficiency.
section_title: Federal Reserve Has Disclosed Limited Information about Its CCAR Qualitative Methodology and Has Not Provided Timely Guidance on Leading Practices
paragraphs: While the Federal Reserve has communicated certain information about  CCAR qualitative assessments to participating companies and the public,  it has not disclosed more detailed information that would allow for an  understanding of the methodology for the assessments or updated  guidance to firms about leading practices.
subsections: section_title: Limited Disclosures of Information about Methodology and Objections
paragraphs: While the Federal Reserve has communicated some CCAR-related  information to the public and directly to CCAR companies, it has not  provided the level of information necessary for a clear understanding of  its qualitative assessment methodology, including its framework for  evaluating the extent to which companies have met supervisory  expectations and determinations, and for a clear understanding of the  reasons for objection determinations. Federal Reserve communications  about the qualitative assessments have included information about  supervisory expectations and other topics. For example, the Federal  Reserve has issued public documents that include an annual CCAR  instructions and guidance document, which are published at the  beginning of each CCAR cycle. In 2013, the Federal Reserve also issued  a separate document describing its capital planning expectations and  examples of the observed range of practices among CCAR companies.  In December 2015, the Federal Reserve issued supervisory letters to  consolidate previously-issued capital planning expectations and to clarify  differences in expectations based on firm size and complexity. In  addition, at the end of each CCAR cycle, the Federal Reserve has  provided CCAR companies with confidential letters describing  assessment findings and information on any supervisory matters requiring  attention.
The Federal Reserve also has publicly released the identity of firms that  have received an objection or conditional non-objection based on the  qualitative assessment and a general description of the reasons for the  objection or conditional non-objection in its annual CCAR results  document. For example, the Federal Reserve’s published results have  described the capital planning areas in which deficiencies were found,  such as risk management and internal controls.
While these documents are helpful in providing some information about  the CCAR assessment and its results, they do not provide information  about the assessment framework, such as the role of various evaluation  teams and descriptions of ratings and rankings and other associated  processes. The Federal Reserve also has not publicly disclosed  information about the nature of the deficiencies in capital planning areas  leading to its objection determinations, such as why a firm’s risk  management or internal controls were inadequate. For instance, the  Federal Reserve’s 2016 CCAR results disclosure states that the reasons  for qualitative objections for one firm were based on deficiencies in the  risk management and control infrastructure. This includes risk- measurement processes, stress testing processes, and data  infrastructure. It also stated that these deficiencies limited the reliability of  the firm’s capital planning process and its ability to conduct a  comprehensive capital adequacy assessment. The document did not discuss why, or in what ways, these areas did not meet Federal Reserve  expectations. For example, it did not explain what elements or  characteristics of the company’s risk measurement and stress testing  processes and data infrastructure were not reasonable or appropriate.
According to an Office of Management and Budget (OMB) directive on  open government, transparency promotes accountability by providing the  public with information about government activities. While the Federal  Reserve is not required to follow OMB’s guidance, this guidance identifies  a set of actions for all agencies to take to increase transparency in  government operations. Similarly, our prior work has recognized that  transparency—balanced with the need to maintain sensitive regulatory  information—is a key feature of accountability.
Federal Reserve staff told us that they have publicly described some  aspects of the qualitative assessment framework, including factors that  influence objection determinations, but have not published a description  of the evaluation teams and ratings because they consider such aspects  to be internal processes. According to these staff, the Federal Reserve  may consider providing additional information about the qualitative  assessment process in the future through public CCAR documentation or  other communication with companies. However, the Federal Reserve has  published information about its methodology for completing the  supervisory stress test and CCAR quantitative assessment, which also  involves internal supervisory processes, as illustrated in the following  examples.
Although it has not disclosed specific details about its models, the  Federal Reserve’s DFAST results document describes the  supervisory stress test methodology used to produce the post-stress  capital ratios that underlie the quantitative assessment. The  disclosed information includes the analytical framework for the  supervisory stress test, modeling approach, model methodology and  validation, data inputs, and general descriptions of specific models  used to project stressed capital ratios.
Publicly-available Federal Reserve supervision manuals disclose  detailed information about the policies and procedures used in  examinations of banking institutions as part of the Federal Reserve’s  normal, ongoing supervision.
Without disclosing additional information that would allow for a better  understanding of the Federal Reserve’s methodology for completing  qualitative assessments and the reasons for objection determinations,  financial markets and the public may have a limited understanding of this  critical aspect of the CCAR program. The limited transparency could  hinder public and market confidence in the Federal Reserve’s CCAR  assessments and the extent to which the Federal Reserve can be held  accountable for its decisions.
section_title: Federal Reserve Guidance about Practices Not Updated Since 2014 and Companies Cited Concerns about the Communication of Expectations
paragraphs: The Federal Reserve has not updated guidance on current and leading  capital planning practices used by CCAR companies since 2014 and  companies also cited concerns about how expectations were explained.  The Federal Reserve has periodically issued guidance on current and  leading capital planning practices used by CCAR companies. For  example, it included the information in an appendix to the CCAR  instructions for the 2015 cycle. The appendix described common themes  in company practices that the Federal Reserve observed during the prior  year’s review. In the appendix, the Federal Reserve stated that it  included the information to build on expectations outlined in previous  guidance and to provide additional clarification for specific areas in which  companies continued to experience challenges. However, the Federal  Reserve has not provided information on the range of current capital  planning practices (including those it views as leading and lagging  practices) since it issued the instructions document in October 2014.
Although the Federal Reserve’s recently published supervisory letters  consolidated its capital planning expectations, they did not include  information on observed industry practices or the Federal Reserve’s  views on what constituted leading or lagging practices. In contrast, the  expectations and range of practice document that the Federal Reserve  issued in 2013 identified practices among CCAR companies that the  Federal Reserve considered to be stronger (leading practices) as well as  those that it deemed to be weaker for capital planning purposes. The  2013 document also clarified that practices identified as leading or  industry best practices should not be considered a safe harbor and stated  that the Federal Reserve anticipated that leading practices would  continue to evolve as new data became available, techniques advanced,  economic conditions shifted, and new risks emerged.
The Federal Reserve also has communicated its expectations for capital  planning directly to companies through various channels. According to  Federal Reserve staff, the Federal Reserve has communicated specific  expectations to companies through its other supervisory activities and  through discussions with the CCAR Executive Committee during CCAR  reviews. For example, the Federal Reserve has communicated with  companies throughout the year about their progress toward remediating  any supervisory issues identified during the CCAR qualitative  assessment. Staff also said that at the conclusion of a CCAR cycle, the  Federal Reserve has directly discussed CCAR findings with companies  and sent feedback letters describing the results of CCAR reviews and any  related supervisory findings and matters requiring company attention.  Federal Reserve staff further explained that these conversations have  established what actions the company would take to address the contents  of the feedback letter and that additional meetings have been held to  obtain updates on progress toward remediating identified weaknesses  and matters requiring attention. The Federal Reserve has stated that it  generally expects identified weaknesses to be remediated before the next  annual capital plan submission, where appropriate, but recognized that  some efforts may require additional time.
However, many CCAR companies we interviewed expressed concerns  about what they viewed as limited or unclear communication of capital  planning expectations for the qualitative assessment by the Federal  Reserve. While several companies indicated that the Federal Reserve’s  communication and guidance had improved over time, most of the  companies said that a lack of clarity about supervisory expectations  posed challenges for them. Specifically,
Nearly all of the companies we spoke to said that at times feedback  was inconsistent across Federal Reserve teams or lacked clarity.
One company said that ambiguity around Federal Reserve  expectations for the qualitative assessment has made it difficult to  implement necessary changes.
Most CCAR companies we spoke to raised concerns that Federal  Reserve expectations have increased annually while guidance has  sometimes been unclear, insufficient, or outdated. Several of these  companies suggested that the qualitative assessment process could  benefit from increased transparency and more granular and timely  guidance.
Several companies we spoke to noted that Federal Reserve guidance  on industry leading and lagging practices that had been provided—in  particular, the 2013 document highlighting the Federal Reserve’s  views on the range of company practices—has been helpful in  understanding supervisory expectations.
Federal internal control standards state the importance of relevant,  reliable, and timely communications, including with external  stakeholders. Federal Reserve guidance also indicates that the Federal  Reserve expects the largest and most complex CCAR firms to use  leading capital planning practices—those that are the most sophisticated,  comprehensive, and robust—and that these leading practices are  expected to evolve over time.
According to Federal Reserve staff, the Federal Reserve has not decided  whether it will issue additional guidance on company practices or  supervisory expectations beyond the recently published supervisory  letters. However, Federal Reserve staff stated that the Federal Reserve  does not intend to update its 2013 guidance on the observed range of  CCAR companies’ capital planning practices or publish another “common  themes” appendix or similar guidance documents because its recently  issued supervisory letters include consolidated guidance on supervisory  expectations relating to firms’ capital planning processes. The documents  also have technical appendixes containing specific expectations but do  not include information on leading practices. The staff said that the  Federal Reserve intends to use these documents as the primary  mechanism for clarifying expectations. Federal Reserve staff also told us  that communicating company-specific expectations occurs through direct  communication with CCAR companies, including confidential feedback  letters. Yet, without periodically updated guidance on observed capital  planning practices and those that the Federal Reserve considers to be  leading ones—especially as industry practices evolve—some CCAR  companies may have difficulty determining what is necessary to meet  Federal Reserve expectations, which could impede the achievement of  CCAR goals.
section_title: Federal Reserve Has an Official CCAR Communications Channel but Has Not Specified Response Times for Questions from Companies
paragraphs: The Federal Reserve has designated an official communications channel  for CCAR companies to ask questions related to CCAR, but has not  communicated time frames for responding to questions. The Federal  Reserve has designated a general e-mail address—referred to as the  CCAR Communications Mailbox—to field all questions from CCAR  companies and provide all responses and other official communications  on behalf of the Federal Reserve. The Federal Reserve has instructed  firms to submit CCAR-related questions to the mailbox and stated that  only responses received through the mailbox will be considered official  Federal Reserve responses, although meetings and other discussions  with Federal Reserve staff may be arranged. According to Federal  Reserve procedures, staff identify questions as being either broadly  relevant to all CCAR companies or company-specific. The Federal  Reserve distributes broadly relevant questions and responses to all  CCAR firms through a frequently-asked-questions (FAQ) report, while the  procedures call for the company-specific questions to receive a direct  response.
The Federal Reserve has established an internal target for response  times but it has not communicated this or other time frames to CCAR  companies. Federal Reserve procedures for the CCAR mailbox identify  an internal goal of providing companies with a response to submitted  questions within 7 business days of receipt. Federal Reserve staff  indicated that the time frame target is primarily intended to help process  the questions internally and prepare responses. However, according to  Federal Reserve procedures, communication with companies submitting  questions consists only of acknowledging receipt of questions and  providing a direct response after the completion of the internal review and  response process.
However, most companies we interviewed identified limitations in  receiving timely or helpful responses from the CCAR mailbox. For  example, some companies said that the Federal Reserve’s mailbox  responses tended to be general and standardized to apply to all  companies rather than tailored to a company’s specific circumstances. In  addition, two companies said that answers to some questions simply  reiterated the stress test and capital plan rule, which was of limited  usefulness. Several companies also explained that it could take multiple  weeks or even months to receive responses to mailbox questions, which  represented a considerable amount of time in the context of CCAR time  frames. Officials from two companies noted that while 2 to 3 weeks may  not seem like an excessive amount of time, the delays prevent companies  from addressing the capital planning topics covered by the question as  they await guidance from the Federal Reserve. The officials also stated  that they understand that it takes time to research and review answers  before they are sent to companies and that the Federal Reserve wants to  provide considered and consistent responses. Several companies also  identified improvements in communication with the Federal Reserve’s on- site teams, including their responsiveness to company questions.
Federal internal control standards state the importance of relevant,  reliable, and timely communications including with external  stakeholders. The Federal Reserve has stated that it designed the  CCAR mailbox and FAQ process to help ensure that CCAR companies  received timely and consistent responses to all submitted questions.  Federal Reserve staff also explained that many questions require further  research and internal deliberation before a response can be provided,  which makes it difficult to commit to a specific response time frame. For  example, according to Federal Reserve procedures, questions and  responses that set new guidance or involve broad policy implications may  require additional review including discussions with the CCAR oversight  committees. Internal process documentation calls for Federal Reserve  staff with subject-matter expertise in areas such as capital policy, balance  sheet items, or model risk management to draft responses to questions  while other subject-matter experts, a legal reviewer, and management  review and approve the draft responses.
However, failure to establish and communicate time frames for  responding to company inquiries may complicate companies’  management and planning of their CCAR submissions and hinder their  ability to address supervisory concerns in a timely fashion. For example,  due to CCAR deadlines, a company awaiting a response to a question  the Federal Reserve has deemed to involve broad policy implications may  have to proceed with developing its CCAR submission without receiving  the guidance it needs from the Federal Reserve.
section_title: Supervisory Scenario Design Process Integrates Historical Data and Current Risks but Some Limitations Exist
paragraphs: Federal Reserve staff design supervisory scenarios for the supervisory  stress tests in DFAST and CCAR by integrating data from historical  recessions and the recent financial crisis with an assessment of current  risks to financial stability. But limitations exist with some aspects of the  scenario design, including consideration of trade-offs related to the choice  of severity and assessment of the sufficiency of a single severe  supervisory scenario.
subsections: section_title: Scenario Design Process Integrates Historical Data and Current Risks
paragraphs: Federal Reserve staff design supervisory scenarios for the supervisory  stress tests in DFAST and CCAR by integrating data from historical  recessions and the recent financial crisis with an assessment of current  risks to financial stability. Federal Reserve staff design the scenarios  according to Federal Reserve policy outlined in a November 2013 policy  statement.
As previously discussed, the Federal Reserve annually creates multiple  supervisory scenarios for DFAST and CCAR:
Baseline scenario. Generally reflects economic conditions expected  by economic forecasters.
Adverse scenario. Features mild to moderate economic and financial  stress driven by selected potential risk factors.
Severely adverse scenario. Features severe economic and financial  stress, generally driven by a different set of risk factors than the  adverse scenario.
Global market shock and counterparty default components  (applicable to companies with large trading or custodial  operations). These two components, applicable to a subset of  companies, are designed to stress the trading and private equity (in  the case of the global market shock), or counterparty positions (in the  case of the counterparty default component) of bank holding  companies with significant trading activities. These components are  supplemental to both the adverse and severely adverse scenarios.
Overlapping teams of staff from across the Federal Reserve System  simultaneously develop the macroeconomic scenarios (baseline, adverse,  and severely adverse) and the market shock component. The  macroeconomic scenarios feature stress events that evolve over 13  quarters while the global market shock and counterparty default  components take place at a single point in time.
At the start of the process, the macroeconomic scenario design group  meets to discuss salient risks that might be included in the scenarios,  drawing on the Federal Reserve’s quarterly assessments of risks to  financial stability and input from FDIC and OCC staff.
Macroeconomic modelers at the Federal Reserve translate these  identified risks (such as a decline in housing prices) into projections  for each of the 28 economic and financial variables included in the  scenarios. The projections (over 13 quarters) represent the  quantitative output of the scenario design process. The economic and  financial variables include measures of the unemployment rate, gross  domestic product, housing and equity prices, interest rates, and  financial market volatility.
A separate scenario design group develops the global market shock  component, which results in a set of instantaneous shocks to a broad  range of financial market risk factors. These shocks involve large and  sudden changes in asset prices, interest rates, and measurements of  market risk. Price changes in the market shock scenario generally  have been comparable to financial market developments in the  second half of 2008 (the height of the financial crisis) and also  featured larger declines when market valuations have been more  elevated or to emphasize salient risks identified by the Federal  Reserve.
Federal Reserve staff present the proposed scenarios to stress test  oversight groups and division directors in the Federal Reserve and to  officials from FDIC and OCC.
After considering feedback, the scenario design group provides  options for final scenarios to the Board of Governors chair, vice chair,  and other governors involved with bank supervision for their input and  preferences.
Finally, the scenario design group completes the scenarios based on  the principals’ choices, including adding a narrative description of the  key factors driving the scenario, and releases the final scenarios on  the Federal Reserve website.
section_title: Limitations Exist with Some Aspects of the Scenario Design Process
paragraphs: While the Federal Reserve has implemented a framework for designing  supervisory scenarios, some aspects of the process have limitations, in  particular regarding the choice of severity and the sensitivity of results to  alternative severe scenarios.
subsections: section_title: Federal Reserve Has Not Considered Severity outside Postwar U.S. History
paragraphs: The Federal Reserve largely has relied on historical experience to  establish the severity of the severely adverse scenario, operationalized  primarily through the unemployment rate. More severe scenarios (such as  those with higher unemployment rates) generally would have an adverse  impact on loans and other assets, increasing losses, reducing income  and profitability, and hence reducing the projected post-stress capital  ratios for participating companies. IMF principles for supervisory stress  testing suggest that supervisors should focus on tail risks—those risks  associated with very unlikely but extreme events, including events that  have not occurred in the past—and highlight the risks of basing scenario  design decisions solely on historical experience.
The Federal Reserve’s decisions about the severity of its scenarios have  been driven primarily by U.S. postwar historical experience. In designing  stress test scenarios, the Federal Reserve primarily has used the change  in and level of the unemployment rate to determine the severity of the  scenario (with higher unemployment rates associated with more severe  scenarios). According to Federal Reserve policy, the unemployment rate  should rise by 3–5 percentage points and must reach a minimum of 10  percent in the severely adverse scenario. In practice, the Federal  Reserve has increased the unemployment rate in the severely adverse  scenario by 4 percentage points in each year from 2013 to 2015 and by 5  percentage points to reach the 10 percent minimum in the 2016 scenario.  Federal Reserve staff stated that a 3–5 percentage point increase and a  10 percent unemployment rate minimum were consistent with postwar  historical recessions and provided a reasonable basis for determining the  overall severity of the severely adverse scenario. In discussing the 2015  severely adverse scenario, which had a peak unemployment rate of 10  percent, staff indicated that a higher or lower unemployment rate—for  example, 9 percent or 12 percent—would be difficult to justify based on  historical precedent. In particular, staff noted that a 12 percent  unemployment rate has never been seen in postwar U.S. history. Also, it  would have required an unprecedented 7 percentage point increase in the  unemployment rate. Similarly, staff noted that a 9 percent unemployment  rate would be relatively low compared to severe U.S. recessions. In other  aspects of scenario design, in contrast, Federal Reserve policy is  cognizant of the possibility that scenarios may produce risks that appear  together in ways that could be outside of historical experience—and  Federal Reserve Staff noted that supervisory scenarios have featured  combinations of risks that have not occurred historically. Federal Reserve  staff also noted that the cumulative severity associated with multiple,  simultaneously stressed scenario variables could exceed historical  postwar severity. However, the Federal Reserve’s scenario design policy  and process are focused on selecting economic conditions that reflect the  severity of postwar U.S. recessions. Without proactively considering  levels of severity outside postwar U.S. historical experience, the Federal  Reserve could miss opportunities to assess and guard against relevant  but unprecedented risks to the banking system. For example, if scenario  severity decisions had been made in the pre-crisis period based solely on  historical conditions that had prevailed prior to 2006, any associated  stress tests would have dramatically underestimated subsequent events.
section_title: Federal Reserve Has Not Assessed Certain Trade-offs Associated with Scenario Severity
paragraphs: According to Federal Reserve staff and our review of internal documents,  the Federal Reserve has not explicitly analyzed how to balance the  choice of the severity of the severely adverse scenario—and its influence  on the resiliency of the banking system—with any impact on the cost and  availability of credit. The overall severity of a stress scenario will affect  how much capital that participating bank holding companies would need  to hold to avoid an objection from the CCAR quantitative assessment and  make planned capital distributions. Furthermore, a more severe scenario  might induce companies to raise additional capital in the short term—and  potentially pass costs on to borrowers—but increase the resiliency of the  banking system over the long term.
OMB guidance states that regulatory analysis—a tool regulatory agencies  use to anticipate and evaluate the likely consequences of rules—should  be based on the best available scientific, technical, and economic  information. While the Federal Reserve is not legally required to follow  this guidance, it provides a strong set of analytical practices relevant to  significant supervisory and regulatory exercises such as CCAR—and  scenario design is a key element of CCAR. Research by the Basel  Committee on Banking Supervision (Basel Committee)—an international  standard-setting organization for bank regulation—on the potential  economic impact of capital requirements provides an example of how the  Federal Reserve could use available information to help assess the  appropriate degree of severity in stress tests. In its assessments of  post-crisis reforms to strengthen bank regulation, the Basel Committee  assessed how changes to the level of required capital and liquidity would  influence economic growth, the cost and availability of credit, and the  likelihood and severity of future financial crises.
Federal Reserve staff noted that they were aware of a significant amount  of academic literature on the relationship between bank capital and the  economy, including the work of the Basel Committee. Moreover,  Federal Reserve staff said that the scenarios were designed to match the  severity of historical recessions, which assesses the resilience of the  banking system and would allow companies to function through a severe  recession. However, without more careful assessment of scenario  severity, the Federal Reserve cannot be reasonably assured that the  scenario design process balances any improvements in the resiliency of  the banking system with any impact on the cost and availability of credit.  These factors could be especially important when considering a level of  severity without a postwar historical precedent—for example, by helping  ensure that scenarios that might exceed postwar historical severity do not  result in undesired increases in the cost and availability of credit.
section_title: Federal Reserve Has Not Assessed Sufficiency of a Single Severe Supervisory Scenario
paragraphs: The Federal Reserve has not conducted analysis to determine if a single  severe supervisory scenario (that is, the severely adverse scenario) is  sufficiently robust and reliable to promote the resilience of the banking  system against a range of potential crises. The Federal Reserve’s policy  statement on scenario design suggests that at times the stress tests may  require additional supervisory scenarios to capture a large number of  unrelated but significant risks. The CCAR quantitative assessment is based on more than one severe scenario, but the Federal Reserve  designs only one of them. Participating institutions design an additional  severe scenario, which is intended to reflect particular risks that might  differ from the supervisory scenario in substantive ways. The company- designed scenario is implemented with a different capital action  assumption, which limits its comparability to other CCAR stress tests. We  discussed potential incentive problems associated with company-run  CCAR stress tests earlier in the report.
There are advantages and disadvantages associated with reliance on a  single severe supervisory scenario, as the Federal Reserve does with the  supervisory stress tests. Advantages could include simplicity,  transparency, and resource use—that is, using a single severe  supervisory scenario simplifies communication and limits the resources  required to design the scenarios and execute and analyze the supervisory  stress tests. While it may be appropriate to use a single severe  supervisory scenario, there are also potential disadvantages or risks  associated with making the CCAR quantitative assessment based on a  single severe supervisory scenario. For example, many different types of  financial crises are possible, and the single selected scenario does not  reflect a fuller range of possible outcomes. Similarly, IMF principles for  supervisory stress tests note that future stress periods are uncertain and  could be represented by a range of stress factors, each with a different  likelihood of occurrence. Staff at IMF and Bank for International  Settlements with whom we spoke also identified trade-offs associated  with using one or multiple scenarios. For example, Bank for International  Settlements staff noted that firms might hedge against the primary risks in  a single scenario but not others that also might be relevant. Similarly, IMF  staff said that ideally stress tests would use a large and diverse set of  scenarios but also noted that this would increase cost and complexity.
Moreover, it is usually necessary to conduct sensitivity analysis to reveal  whether, and to what extent, the results of an analysis are sensitive to  plausible changes in the main assumptions. For the supervisory stress tests and CCAR quantitative assessment, the design and number of  severely adverse scenarios represent key assumptions affecting results.  However, the Federal Reserve has not conducted sensitivity analysis to  determine whether its single severe scenario is sufficient to accomplish  DFAST and CCAR goals. For example, it has not assessed how a range  of severe scenarios may lead to different judgments about the overall  resiliency of the banking system.
Because it has matched the severity of historical U.S. recessions, Federal  Reserve officials asserted that the severely adverse scenario would  protect against a range of potential crises. Federal Reserve officials also  noted that they perform multiple stress tests using alternative scenarios  outside of DFAST and CCAR, and also conduct a separate stress test of  liquidity. Yet, without assessing the sufficiency of a single severe scenario  in the context of DFAST and CCAR—such as performing sensitivity  analysis involving multiple scenarios—the Federal Reserve is making  CCAR decisions that may not reflect the range of potential crises against  which the banking system would be resilient and the magnitude of the  range of outcomes that might result from different scenarios. The Federal  Reserve therefore may be limited in its ability to understand,  communicate, and manage uncertainty associated with its use of the  supervisory stress test results.
section_title: Federal Reserve Has Not Prospectively Assessed Whether Changes in Scenarios Might Inadvertently Amplify Economic Cycles
paragraphs: The Federal Reserve has not assessed whether or how the year-to-year  changes it makes to the supervisory scenarios over an economic cycle  could inadvertently amplify those cycles until after it has completed and  published the scenarios. Because the supervisory scenarios used in the  stress tests influence a company’s post-stress capital ratios, changes to  the scenarios will affect how much capital participating companies need  to hold to help ensure they do not receive a CCAR objection. Federal  Reserve policy states that supervisory scenarios should feature stressful  outcomes that do not induce greater procyclicality—that is, scenarios  should not amplify cycles (swings in economic activity between expansion  and contraction) in the financial system and economy. Procyclicality  could, for example, result in firms needing to raise additional capital or  reduce lending during a downturn.
After the disclosure of 2015 stress test results, Federal Reserve staff  reported to the Board that the supervisory stress tests produced some  procyclical results. Specifically, losses on portfolios of loans to consumers  (such as credit cards or residential mortgages) in the test were procyclical  partly because scenarios had caused the projected losses to shrink as  actual economic conditions improved. Based on analysis conducted after  the scenarios were finalized, Federal Reserve staff said that most of the  decrease in projected losses (compared with the prior year’s stress test)  resulted from improvements in bank balance sheets from the previous  year, although changes to the scenario also contributed to lower  projected losses. Because the scenario contributed to smaller losses as  the economy improved, the scenario could produce larger losses as the  economy deteriorates, lowering post-stress capital ratios and increasing  the amount of capital required to avoid a CCAR objection. Moreover,  because the analysis of the impact of annual scenario changes on losses  occurs only after scenarios have been developed and made public, the  Federal Reserve might learn of procyclical effects too late to take  effective preventive steps—for example by adjusting relevant scenario  variables before scenarios are made final. As a result, Federal Reserve  stress tests could exacerbate future financial stress by increasing  requirements on stress test participants while economic conditions are  deteriorating.
Such an unintended impact of the supervisory scenario on losses could  emerge because the complexity of the system of models (discussed in  the following section) used by the Federal Reserve makes it difficult to  anticipate the combined effects of changes to 28 scenario variables that  influence the results of multiple supervisory stress test models. Federal  Reserve staff stated that the Federal Reserve’s scenario design policy  attempted to avert procyclicality by instituting the 10 percent  unemployment rate minimum, to prevent the unemployment rate from  falling too much in the scenario when the real economy improved and  otherwise allowing the unemployment rate to increase from 3 to 5  percentage points—increasing the unemployment rate more in the  scenario when actual unemployment is low, and raising it less when  actual unemployment is high. However, the complexity of the system of models the Federal Reserve has used in the supervisory stress test  implies that without additional, supporting analysis the Federal Reserve  cannot be reasonably assured that small adjustments to the  unemployment rate and other variables would produce outcomes that  neither amplify nor dampen economic cycles.
section_title: Federal Reserve Management of Model Risk Has Not Focused on the System of Models
paragraphs: The Federal Reserve’s modeling process for the stress tests includes an  oversight structure and internal reviews, but it has not focused its model  risk management on the system of models that produce stress test  results. To estimate the effect of stress test scenarios on companies’  ability to maintain capital, the Federal Reserve has developed individual  component models that predict companies’ financial performance in the  scenarios. The results of these component models are combined with  assumed or planned capital actions of companies and form the system of  models used by the Federal Reserve. The Federal Reserve has issued  model risk-management guidance that defines model risk as the potential  for adverse consequences from decisions based on incorrect or misused  model outputs and states that it increases with factors such as greater  model complexity and larger potential impact. However, the Federal  Reserve has not focused its model risk-management efforts on the  system of models, including not conducting sensitivity and uncertainty  analyses of how its modeling choices affected the model risk associated  with the overall stress test results (post-stress capital ratios).
subsections: section_title: Federal Reserve Has a Process for Development and Oversight of Models Used for Supervisory Stress Tests
paragraphs: The Federal Reserve has a development process for the models it uses  to predict each institution’s post-stress capital ratios and has an oversight  structure for the process.
subsections: section_title: Overview of Supervisory Stress Test Models
paragraphs: To estimate the effect of supervisory scenarios on companies’ regulatory  capital ratios, the Federal Reserve has developed numerous empirical  models that each predict a component of a company’s balance sheet,  risk-weighted assets, or income statement (component models) for each  of the 9 quarters of the stress test planning horizon. The Federal  Reserve then combines the results of the component models with  assumed or planned capital actions (for the companies) to produce the  post-stress capital ratios. We refer to the combination of the component  models that produces the post-stress capital ratios as the system of  models.
The component models are either predictive or accounting models. The  predictive models use historical data to estimate how economic stress  events might affect an element of an institution’s financial performance,  such as loan losses or revenues. Accounting models apply various  accounting rules to an institution’s financial data or to outputs from the  predictive models to construct aggregate accounting measures, such as  allowances for loan and lease losses or pre-tax net income. Most  component models use estimates produced by other component models  as a source of data to make their projections.
section_title: Overview of Modeling Cycle
paragraphs: The Federal Reserve has implemented a 2-year development cycle for  supervisory stress test models (see fig. 6). According to Federal Reserve  staff and documentation, the overall goal of the development cycle is to  continue refining and developing the models, while simultaneously  producing reliable estimates for the annual DFAST and CCAR  supervisory exercises. The Federal Reserve’s 2-year model development  cycle (which is described in more detail in the following sections) involves  the use of production and development models. Production models are  used to produce annual estimates for the DFAST and CCAR exercises.  Development models allow for more time to validate and evaluate major  model changes before they are incorporated into the actual stress test  exercises. Once a development model has been fully reviewed and  approved, it replaces the corresponding production model.
The Federal Reserve annually refines and continues development of its  models. Modeling approaches and variables can change over time. The  Federal Reserve has stated that revisions to its models generally have  reflected advances in modeling techniques, more detailed data, and  longer histories of performance in different economic environments. In  addition, changes in market or industry risk characteristics and regulatory  or policy changes also may be a reason to make changes to existing  models.
section_title: Responsibilities and Oversight by Process Phase
paragraphs: The Federal Reserve has implemented an organizational structure  focused on model development, oversight, and review (see fig. 6). For  example, according to Federal Reserve staff and program documents, the  Model Oversight Group—a cross-functional group of senior Federal  Reserve managers and stress test experts—coordinates model  development policy and has overall responsibility for the model  development process. According to Federal Reserve staff, the Model  Oversight Group exercises close oversight of the planning and execution  of the supervisory stress tests including changes to stress test models  and the Director of Banking Supervision and Regulation has final  authority to approve models prior to stress test production. The Model  Coordination and Advisory Team assists the oversight group by serving  as the first line of oversight over the modeling teams. According to its  charter, the coordination team is tasked with initial reviews of model  documentation, model changes, and model assessment presentations.  Economists, senior quantitative analysts, and other technical specialists  from across the Federal Reserve System staff the coordination team.
Development. The responsibility for executing model development lies  with 11 supervisory modeling teams staffed with subject-matter experts.  Each of the teams are responsible for developing models to predict  elements of a company’s balance sheet or income statement, which are  ultimately combined to predict post-stress capital ratios. For example,  one modeling team is responsible for 22 separate models that predict  components of net revenue before adjustments for loan loss provisions.  Another is responsible for models that combine output from the loss,  revenue, and other models to produce the regulatory capital ratios. The  Model Oversight Group oversees the development process with  assistance from the Model Coordination and Advisory Team. The Model  Oversight Group also reviews and determines whether to approve  development models to replace prior production models.
Preliminary assessment. According to Federal Reserve documentation,  supervisory modeling teams then conduct a preliminary assessment of  the model production process after the model development phase is  complete. The main purpose of the assessment is to test that all models  and processes perform as expected and remediate problems before the  final production of the annual stress test results. The modeling teams use  loan portfolio and other financial data submitted to the Federal Reserve  by institutions subject to the supervisory stress test as inputs to their  models. They apply the current portfolio data and scenarios to the  production models—including the development models awaiting approval  to become production models—to test the models and provide  information to the Model Oversight Group about the effects on component  model outcomes. The modeling teams analyze any differences from the  previous year’s results, which involves estimating how much of the overall change in a component model’s result can be attributed to model  changes, scenario changes, company portfolio changes, or other factors.  Teams then present the results to the Model Oversight Group and identify  any production or modeling problems, including any problems with the  quality of company-provided data. The modeling teams revise models or  data inputs as necessary in response to problems identified by the Model  Oversight Group or the Model Coordination and Advisory Team during  the preliminary assessment presentations and finalize their model  documentation. Senior Federal Reserve staff approve or reject major  modeling decisions made by the Model Oversight Group.
Validation. In the model validation phase, reviewers in the Model  Validation Unit are to examine the models to identify potential problems,  ranking them by level of concern and threat to model validity. According  to Federal Reserve policy, staff undertaking validation activities are  independent of the model development process for the models they  review. Before 2016, validation reviewers volunteered to leave their  primary duties in the Federal Reserve System to assist with the stress  test model review process. For example, for the CCAR 2015 exercise,  they spent approximately 8 weeks reviewing the models and returned for  an additional 3 weeks to validate model changes made in response to the  most severe findings. According to Federal Reserve staff and program  documents, the Federal Reserve has been transitioning to a validation  program that consists primarily of permanent, full-time staff.
For each modeling team, the Model Validation Unit employs various staff  that are to review (1) model soundness and performance and (2) model  change and implementation controls. Economists and other subject- matter experts from across the Federal Reserve are to evaluate the  model design for conceptual soundness and performance (model  soundness and performance review). Federal Reserve internal control  experts are to evaluate the control processes surrounding model  development and implementation (model change and implementation  control review). Each group of reviewers in the validation unit is to write a  report that summarizes any problems they identified. The reviewers are to  rate problems according to their assessment of the problems’ severity,  materiality, and risk posed to the reliability of the model.
Model finalization. To complete the models, the Federal Reserve has a  policy that defines how the Model Oversight Group oversees the  supervisory modeling teams’ implementation of responses to the findings  of the Model Validation Unit. The oversight group is to review and  approve model changes and the validation unit is to validate them.  According to Federal Reserve staff, modeling teams generally will  address at least those problems identified by validation unit reviewers that  pose the highest risk to the validity of the model.
With the approval of the Director of the Division of Banking Supervision  and Regulation, the oversight group may defer some changes—even  those in response to problems rated in the most-risky category—either  because (1) the changes require structural modifications to models that  the group views as better implemented through the 2-year development  cycle, (2) lack of available data, or (3) other priorities taking precedence.  For example, in the 2015 stress test cycle, 24 high-risk findings of the  validation unit were unaddressed or had associated model changes that  had not yet been validated by the time the models went into production.  Validation unit documentation also indicated 71 repeat findings (at a  lower-risk level) that modeling teams had not addressed for at least a  year. Even if a modeling team is able to address a finding, stress test  model changes may or may not be validated by the Model Validation Unit  during the same cycle in which modeling teams made the change. For  example, in the past some model changes occurring at the end of the  annual modeling process have not been reviewed by the Validation unit  until the following cycle. According to Federal Reserve staff, the new,  permanent staff approach for the Model Validation Unit is designed in part  to increase the unit’s involvement in validating last-minute model  changes. As described later and in appendix II, in a 2015 report the  Federal Reserve Office of Inspector General (OIG) made a  recommendation related to late-stage model changes, among other  things.
Final review and approval and production of results. Supervisory  modeling teams are then to conduct a second run of the production  models to generate results for the final model assessment presentations.  The second run is to use the final company data submissions as well as  the final versions of the models and stress scenarios developed by the scenario design group. Each modeling team is to present its results to a  group that consists of the Model Oversight Group, co-leaders of the  Model Coordination and Advisory Team, and the deputy director of the  Large Institution Supervision Coordinating Committee (LISCC) for final  review and approval. The modeling teams may make certain model  adjustments after the presentation in response to feedback. After  addressing any concerns, the modeling teams are to calculate final  results for all of the models. Federal Reserve staff then are to brief the  Board of Governors on the results.
section_title: Disclosure of Model Methodology
paragraphs: The Federal Reserve has disclosed some information about the models  underlying the supervisory stress test, including in an appendix to its  annual publication of DFAST results. In these documents, the Federal  Reserve has described in broad terms its analytical framework for the  supervisory stress test as well as its modeling approach and some  features of specific models. However, the Federal Reserve has not  disclosed other information about the models it uses to execute the  supervisory stress tests. For example, it has not disclosed a level of  information about the models that would easily allow an external party to  replicate the results of the supervisory stress tests.
Officials from several CCAR companies we interviewed said that limited  transparency about the Federal Reserve’s models impaired their firm’s  capital planning efforts. For example, the company officials explained that  without more detailed information on the Federal Reserve’s model  specifications, they were unable to understand the factors behind the  supervisory stress test outcomes or reconcile them with the results of  their own company-run tests. Several companies’ officials said that this  prevented them from identifying the cause of poor stress test results and  taking appropriate actions in response. These officials said that this  limited transparency could result in companies holding additional capital  as a precaution to better ensure that they do not receive an objection  from the CCAR quantitative assessment.
However, the limited disclosure by the Federal Reserve reflects its  concern about the potential for model information to influence the actions  of covered companies in ways that undermine the purpose of the stress  test exercises, among other potential adverse consequences. Federal  Reserve staff said that more detailed disclosure of the underlying models  would make it easier for companies to manage their capital and asset  decisions in relation to the supervisory stress test (in other words, “game”  the models) without necessarily limiting risk, thus resulting in the potential  for a form of regulatory arbitrage (firms’ circumvention of regulation). In  addition, Federal Reserve staff have noted that fuller disclosure could  reduce the diversity of models used by companies—a problem termed  model monoculture—with companies using models that imitated the  Federal Reserve’s rather than developing internal models that best  reflected their own risks. As Federal Reserve staff explained, companies  need to develop models that they believe are best suited for their unique  business activities and portfolios. Some company officials with whom we  spoke acknowledged these trade-offs and said that they understood the  Federal Reserve’s decision to limit disclosure of model details in light of  such considerations. In our prior work on international standards for  regulatory capital requirements—a regulatory setting analogous to the  supervisory stress test—we also have noted that banks can arbitrage  certain capital requirements by managing their portfolios specifically to  reduce required capital levels without reducing risk.
section_title: Federal Reserve Reviews of Model Risk Management
paragraphs: While the current supervisory stress test modeling process has an  oversight and review system in place as described above, both the Model  Validation Unit and the Federal Reserve Office of Inspector General  (OIG) have conducted reviews of the process and identified areas that  would benefit from improvement. According to the Federal Reserve, it is  committed to continuous assessment and enhancement of the  supervisory models used in the stress testing program. As part of this  commitment, the validation unit has completed multiple internal  assessments, including an evaluation of the Federal Reserve’s  governance of its model risk management activities for supervisory stress  testing completed in December 2014. As described in the OIG report,  the review by the Model Validation Unit determined that certain  governance practices did not fully conform to the Federal Reserve’s  supervisory guidance for banking institutions on model risk-management  practices and exhibited fundamental weaknesses in key areas.
The OIG review was issued in October 2015 and examined the Federal  Reserve’s model risk-management practices for supervisory stress tests,  with a particular focus on the model validation process. The OIG report  found that reviews by the Model Validation Unit to assess the Federal  Reserve’s validation and governance activities had identified  opportunities for improvement, but that additional actions could further  enhance model risk-management practices. Both reviews used the  Federal Reserve’s supervisory guidance for banking institutions on model  risk-management practices as the primary criteria for evaluating the  Federal Reserve’s own processes. See appendix II for more information  on these reviews.
section_title: Federal Reserve Does Not Address Cumulative Risk and Uncertainty from the System of Models That Produce Stress Test Results
paragraphs: The Federal Reserve has not focused its risk-management efforts  (including those relating to model development guidance, documentation,  sensitivity and uncertainty analyses, and risk tolerances) on the system of  models that produce the stress test results—the post-stress capital ratios.  As mentioned previously, the Federal Reserve’s model risk-management  guidance defines model risk as the potential for adverse consequences  from decisions based on incorrect or misused model outputs, which  increases with factors such as greater model complexity and larger  potential impact. The guidance states that organizations should manage  model risk both from individual models and in the aggregate and  establishes a definition of models that encompasses both component  models and a system of models. The guidance also notes that  aggregate model risk is affected by interaction and dependencies among  models; reliance on common assumptions, data, or methodologies; and  any other factors that could adversely affect several models and their  outputs at the same time. However, the Federal Reserve’s organizational  structure for the stress tests does not include a formal process through  which model development or risk management at the aggregate—or  system-of-models—level is implemented.
The connections and relationships between the individual component  models combine to create a system of models that produces the post- stress capital ratios. Figure 7 provides a high-level overview of the  interactions among the component models used in the supervisory stress  tests. The Federal Reserve uses component models to project a  company’s balance sheet assets—the “Asset Balances Models” and  “Trading Assets Models” illustrated in the figure—and risk-weighted  assets. It then calculates changes in the company’s net income using  separate component models to project different parts of an institution’s  revenue, expenses, and losses, as well as changes in its loan loss  allowance. Next, other component models project changes in equity and  regulatory capital by combining projected net income and capital actions.  The final step estimates post-stress capital ratios by joining equity and  regulatory capital projections with total assets and risk-weighted assets  projections.
Based on our review of Federal Reserve documentation and interviews  with staff, the Federal Reserve has not assessed its entire system of  models in relation to the principles that it has applied to individual  component models. The Model Oversight Group has developed a set of  modeling principles to assist in managing the design of the Federal  Reserve’s supervisory stress tests, including managing its risks.  According to its procedures, the oversight group intends for these  principles to guide design and other decision-making in the model  development process.
The level at which the Federal Reserve applies the model development  principles is important because it is the combined system of models  rather than any individual model that generates the relevant stress test  results—the post-stress capital ratios. As with all models, the Federal  Reserve’s models used to produce supervisory stress test results reflect  some amount of uncertainty and are sensitive to the assumptions and  modeling decisions made when developing each model. For example,  model developers must make assumptions about how model inputs  interact, which inputs are relevant, and how or if historical data are  relevant to the outcome that the model seeks to predict. Each of those  decisions can affect a component model’s results. As such, each  component model contributes elements of uncertainty to the overall result  of the system of models—the final post-stress capital ratios in the case of  the supervisory stress tests.
The extent and nature of any interaction among component models in the  supervisory stress tests also will introduce risk to the post-stress capital  ratio estimates. In addition, the overall effects on post-stress capital ratios  of choices about how component models interact may be unclear. For  example, poor decisions about component model interactions could result  in post-stress capital ratios that are insufficiently responsive to economic  changes included in a stress scenario or that respond to economic stress  very differently from the way they would in reality.
In the Federal Reserve model documentation we reviewed, supervisory  modeling teams applied model design principles at the component model  level. For example, the modeling teams justified various component  modeling choices using the principles. When selecting from two versions of a component model, both of which appeared to perform well, a team  cited the principle of simplicity in selecting the version with fewer  variables.
Although Model Oversight Group reviews and oversight are applied  across the system of models, the documentation we reviewed did not  discuss key aspects of the interactions between the component models.  For example, it did not consider how the component models are  combined into the system of models and how and to what extent those  choices may introduce statistical uncertainty to the post-stress capital  ratios. Federal Reserve staff indicated that model development decisions  are closely overseen and approved by the oversight group. The staff  explained that the close oversight provided by the group provides an  adequate assessment of the effect of component model design decisions  on the post-stress capital ratios produced by the system of models. They  noted that the application of the principles at the component level  combined with the role of the oversight group means that the principles  are applied at the system-of-models level. As an example of how that  oversight operates, the Federal Reserve staff provided documentation of  what they described as the consideration of a cross-model issue.  However, the documentation did not discuss the potential effects of  design decisions at the component level on post-stress capital ratios,  including any effects from the interaction of component models. In  addition, the Federal Reserve documentation we reviewed about model  development and implementation and oversight by the Model Oversight  Group did not support that the Federal Reserve has conducted systems- level analyses of the effect of modeling decisions on the post-stress  capital ratios.
By largely focusing the modeling principles on the component models and  not applying those principles to the system of models, the Federal  Reserve has limited its ability to manage the extent to which model risk is  introduced into the supervisory stress test models.
subsections: section_title: Lack of Appropriate Documentation for System of Models
paragraphs: The Federal Reserve has not developed appropriate documentation of  the system of models that would allow for effective management of the  risks posed by component model interactions. Appropriate model  documentation is necessary for assessing, managing, and  communicating model risk, and the Federal Reserve’s supervisory  guidance identifies appropriate documentation as one of the elements of  a sound model risk-management process.
According to Federal Reserve procedures, each supervisory modeling  team is responsible for maintaining its own model documentation and  each review team in the Model Validation Unit is in charge of  documenting its findings related to model limitations and other areas. The  procedures call for documentation to follow guidance from the Model  Oversight Group and the Model Validation Unit in both format and  content. The procedures and guidance indicate that appropriate  documentation for component models include descriptions of model  design, data, and methodology; a quality control plan; and review reports.
The model description documentation is expected to give third parties  the ability to understand the model, evaluate it, monitor its  development, and replicate it as necessary.
The quality control plan provides documentation of modeling team  processes to help ensure that the models have been implemented as  intended by the design specification and to mitigate the potential for  model error or misapplication. The plan specifies roles and  procedures for checking or processing model data, documenting and  approving changes, testing the model, and other control activities.
The review reports record the independent assessment of modeling  teams’ models. The reports document the areas the Model Validation  Unit reviewed, evaluation of the models, issues uncovered and an  assessment of the risk they pose to the reliability and performance of  the model, and an assessment of the modeling team responses to  previous year’s findings.
The Federal Reserve procedures and guidance as well as best practices  suggest that similar documentation would be appropriate for the system  of models.
However, the Federal Reserve does not have a similar set of  documentation for the system of models. Instead, Federal Reserve staff stated that they have used a single document to serve this purpose. The  document has recorded Model Oversight Group decisions including some  that apply to multiple models—which are controlled by more than one  modeling team—over the course of a stress test cycle. Yet, the Model  Oversight Group decisions document that we reviewed did not include  descriptions of model design, data, and methodology sufficient to give  third parties the ability to understand the system of models, evaluate it, or  replicate it as necessary. It also lacked information that might constitute a  quality control plan or a model validation review report.
The lack of appropriate documentation of the system of models limits the  Federal Reserve’s ability to effectively identify and manage model risk  from the entire system of models used for the supervisory stress tests.  For example, without such documentation, Federal Reserve staff may  miss important connections between elements of component models,  which in turn may limit understanding of risks inherent in their modeling  choices. The absence of system-level documentation also impedes the  ability of independent parties, both internally and externally, to review,  understand, or evaluate the system of models. Additionally, it increases  risks associated with staff turnover (which could cause the Federal  Reserve to lose important knowledge about the design and functioning of  the system of models).
section_title: Sensitivity and Uncertainty Analyses Have Not Considered Effects on Post-Stress Capital Ratios
paragraphs: Based on our review of Federal Reserve documentation and interviews  with staff, the Federal Reserve has not conducted sensitivity and  uncertainty analyses of how its modeling affects the post-stress capital  ratios produced by the entire system of models. As previously described,  all models reflect some amount of uncertainty and are sensitive to the  assumptions and modeling decisions made during model development.  According to the Federal Reserve’s model risk-management guidance, an  integral element of model development is evaluating model features and  overall functioning to determine whether the model is performing as  intended, including by demonstrating that a model is accurate, robust,  and stable and by assessing potential limitations. The guidance states  that such an evaluation should include assessing the model’s behavior  over a range of input values and evaluating the impact of assumptions—a  type of assessment known as sensitivity analysis. It also notes the  importance of understanding the extent of model uncertainty and  inaccuracy—either quantitatively, such as with the confidence interval  around a statistical estimate, or qualitatively—and accounting for them  appropriately. This type of assessment is known as uncertainty analysis.  The assessments are related as sensitivity analysis tests the effects of  different sources of uncertainty on a model’s output and can help to  identify the greatest sources of uncertainty.
Model documentation we reviewed included some sensitivity and  uncertainty analyses at the component model level, but we did not find  that any such sensitivity or uncertainty analyses fully considered effects  on the extent of model risk associated with the final post-stress capital  ratios. For example, the analyses did not consider these effects for both  the numerator and denominator of the ratios. As described earlier, a post- stress capital ratio consists of a numerator that reflects the capital held by  the company and a denominator that measures the assets held by a  company (either on a total or risk-weighted basis), both projected for each  quarter of the stress scenario. When considering changes to component  model results (which the Federal Reserve uses as one of their primary  forms of sensitivity analysis), the Federal Reserve identifies the major  inputs that have changed for that model and analyzes the extent to which  changes in the model output can be attributed to each input.
One way that the Federal Reserve seeks to put the changes in  individual component models in context with each other and with  overall capital ratios is to divide the component model output (for  example, losses or revenues) by the risk-weighted assets of the  company prior to the assets being stressed in the tests. This approach  allows the Federal Reserve to understand the component models’  effects on the numerator of the post-stress capital ratio and to put  individual component model results on a common scale.
But, some component models also affect the denominator in the post- stress capital ratios. For example, the accrual loan loss models (see  fig. 8) that estimate losses on different portfolios of loans (such as  automobile or commercial loans) also provide an estimate used in  some risk-weighted asset calculations.
Thus, the Federal Reserve’s approach to sensitivity analysis does not  reflect a full consideration of the effects of a component model’s risk and  uncertainty on that of the post-stress capital ratios.
Model documentation standards from the Model Oversight Group indicate  that modeling teams should document the empirical performance of a  model to support its validity in projecting stress test losses, including  addressing—such as through sensitivity analysis—how a model’s output  responds to changes in key inputs and parameters. The oversight group  also has issued a guidance memorandum to modeling teams on the types  of sensitivity and uncertainty analysis they are encouraged to conduct  during model development. However, these guidance documents do not  address conducting sensitivity and uncertainty analyses that examine  how modeling decisions affect the overall stress test results. Federal  Reserve staff told us that performing sensitivity and uncertainty analyses  of the system of models was unnecessary because the system was  largely additive—a mathematical feature that ensures that tests on a  component model will fully capture its effects on the system of models— so that system-wide assessments would be redundant. However, as  discussed already, key interactions exist between component models  such that their joint outcome would not simply be a sum of the component  model outcomes. Federal Reserve staff also explained that performing  sensitivity and uncertainty analyses of the system of models—such as  calculating statistical confidence intervals for the post-stress capital  ratios—would be a complex and resource-intensive undertaking, which  may not provide information with a clear use.
Although the guidance does not address conducting sensitivity and  uncertainty analyses in relation to effects on stress test results, reviewers  from the Model Validation Unit who worked on the 2014 soundness  review of a stress test model used to project asset balances (balances  model) recommended that the modeling team assess the sensitivity of the  post-stress capital ratios to the balances model assumptions. The  balances model is the root model for the entire system of models—it is a  direct or indirect input for almost all other models—and therefore its  design is of particular importance to the accuracy, robustness, and  stability of the supervisory stress test process (see fig. 9). This model  makes assumptions related to market share and portfolio mix for firms  subject to the stress tests.
As the Model Validation Unit recommended in its review, one way to  assess the appropriateness of the assumptions made in the balances  modeling approach would be to test the effects of alternative assumptions  on the post-stress capital ratios and gauge the potential consequences of  any differences. Federal Reserve staff indicated that the supervisory  modeling team had not tested alternatives to these specific assumptions  because (1) the assumptions implement a policy decision by the Model  Oversight Group, and (2) the assumptions accomplished the policy  decision while remaining consistent with the oversight group principle of  simplicity and transparency. However, basing modeling assumptions on  policy goals does not preclude also assessing their effectiveness in  accomplishing the policy goals or the risk of unintended consequences  through testing the potential effects of alternative assumptions on the  stress test output. As of July 2016, the modeling team had not yet  addressed the recommendation of the validation unit.
Without assessing risks to the post-stress capital ratios posed by the  Federal Reserve’s approach to modeling, the Federal Reserve limits its  ability to understand, communicate, and manage the risks and reliability  of its supervisory stress test results. For example, the Federal Reserve’s  model risk-management guidance states that a company’s senior  management is responsible for regularly reporting to its board of directors  on significant model risks from individual component models and in the  aggregate. However, Federal Reserve staff are unable to communicate to  the Board of Governors the range and most sources of uncertainty  surrounding the post-stress capital ratio estimates produced by the  system of models because the Federal Reserve has not conducted the  analyses necessary to do so. Furthermore, sensitivity and uncertainty  analysis can result in changes to models and even small differences in  model estimates can be the difference between the Federal Reserve  objecting or not objecting to an institution’s capital plan.
section_title: Overall Risk Tolerances Not Articulated
paragraphs: The Federal Reserve has not articulated overall model risk tolerances— the amount of uncertainty or error margins that it would be willing to  accept around the post-stress capital ratios. The Federal Reserve’s  model risk-management guidance states that members of an institution’s  board of directors should ensure that the level of model risk is within their  tolerance. It also states that model risk-management policies approved  by the board or its delegates should promote the development of targets  for model accuracy and standards for acceptable levels of discrepancies.
However, neither the Board of Governors nor high-level management in  Banking Supervision and Regulation have identified model risk tolerances  for component model output or for the overall stress test results. Instead,  Federal Reserve staff said that Model Oversight Group reviews helped to  ensure a consistent approach to model risk. In addition to approving  decisions included in model documentation, the Model Oversight Group  has reviewed options that modeling teams have developed for resolving  identified problems with their models and have presented to the oversight  group. The presentation includes a discussion of the advantages and  disadvantages associated with each option. According to Federal  Reserve documentation, the oversight group will direct the modeling team  to pursue one of the options or continue developing alternatives. Federal  Reserve staff told us that decisions of the Model Oversight Group were  based on the principles they had developed and that they weighed a  modeling team’s options against how well they meet the principles. But it  was not always evident from the documentation what criteria the  oversight group used to make its determination about which option to  pursue. Even if one principle was cited to support a decision, it was not  clear if the option was consistent with other principles, making it difficult to  evaluate the consistency of Model Oversight Group decisions or their  application of predetermined risk tolerances.
In the absence of explicit direction about risk tolerances, supervisory  modeling teams may make decisions that have consequences for model  risk without evaluating the model risk against set criteria. For instance, it  also was not clear from the documentation we reviewed how much the  Model Oversight Group had learned about the options modeling teams  had considered and rejected before the presentations. In some cases we  reviewed, the modeling teams had fully developed the choices for  resolving model problems and were able to quantitatively compare  differences between the options—which represents a form of sensitivity  analysis and allows for an assessment of the model relative to an explicit  risk tolerance (although it did not appear from the documentation that risk  tolerances were applied in the decision process). In other cases, the  team’s work was at a more preliminary stage and did not include a  quantitative evaluation of the consequences of the options under  consideration.
Our review of model documentation also suggests that modeling teams  have made model risk tolerance decisions at the individual model level  with no documented reference to the impact of those decisions on other  models or consistency with other modeling decisions. Furthermore,  according to documents we reviewed and discussions with Federal  Reserve staff, the Federal Reserve has not made any efforts to determine  statistical or other thresholds at which each individual model will produce  results within tolerable uncertainty ranges in relation to the post-stress  capital ratio estimates. Instead, some modeling teams appeared to be  implicitly determining what constituted acceptably small variation between  their chosen model’s predictions and the historical data. For example, one  team assessed its model with statistical tests and deemed it reasonable,  providing supporting evidence in charts. There was no discussion of the  criteria the team used to determine their model’s soundness. Based on  the documentation we reviewed, this team did not appear to have made  any calculation of the practical magnitude of the consequences of these  statistical tests. The charts used to show the success of the model show  a 0.1 percentage point difference between actual and predicted values in  the data, in this case default rates of a loan portfolio. The supervisory  modeling team asserted that this was a small difference between  predicted and actual default rates. However, such a difference could have  material consequences for the post-stress capital ratios of companies, as  it represented 12 percent of the total predicted default rate of the  portfolio. Without an articulated risk tolerance, it is not clear whether this  is a large or small difference for the portfolio or the post-stress capital  ratios. As for all component models, this model required the Model  Oversight Group’s approval prior to going into production. However, the  oversight group’s review and approval is not a substitute for an identified  risk tolerance as required by the Federal Reserve’s standards.
In addition, the model documentation we reviewed indicated that some  supervisory modeling teams tested other individual models in addition to those they reported in the documentation. The undocumented models  generally failed to meet statistical or other tests for determining which  models would be included in the documentation. But this may lead  modeling teams to reject models with attributes that would be desirable at  the systems level and the decisions might not be transparent to the Model  Oversight Group due to lack of documentation.
According to Federal Reserve staff, the newly formed Supervisory Stress  Test Model Governance Committee has plans to expand communication  to the Board of Governors to provide Governors with more insight into  model development, model risk, and other outstanding concerns about  models. The expanded communication also may allow Governors to  communicate their model risk tolerances with the Federal Reserve staff  performing the supervisory stress tests and CCAR quantitative  assessment. However, Federal Reserve staff also told us that model risk  tolerances cannot be set prior to the completion of the models. But, this is  not consistent with the Federal Reserve’s model risk-management  guidance, which requires company management to set predetermined  thresholds of acceptability and for senior management to ensure that the  level of model risk is within their tolerance. In the same manner as for  other major areas of risk, tolerances can be articulated in a number of  ways including a combination of quantitative and qualitative approaches.  Without systematically identifying and communicating acceptable levels of  risk in its supervisory stress test models, the Federal Reserve may be  limited in its ability to effectively evaluate and manage its model risk.
section_title: Conclusions
paragraphs: The stress test programs implemented by the Federal Reserve during  and since the financial crisis of 2007–2009 have played a key role in  supervisory efforts to evaluate and maintain the stability of the U.S.  financial system. Overall, they represent important advances that  augment supervisory approaches to capital adequacy and planning that  were in place before the crisis. The Federal Reserve and other bank  regulators (i.e., FDIC and OCC) have issued similar stress test rules, but  OCC has made greater use of supervisory flexibility—granting extensions  to and exemptions from the requirements’ application—in implementing  them. This inconsistent approach to implementation could contribute to  competitive disadvantages between institutions and inconsistent oversight  of risk management by the regulators.
The Federal Reserve has integrated DFAST and CCAR into its  supervision of large banking organizations and made changes to the  programs in recent years at least partly in response to concerns raised by  the industry and market observers that, among other things, adjusted the  timing of the exercises, consolidated guidance on supervisory  expectations for capital planning, and modified certain technical aspects  of capital distribution restrictions and capital action assumptions. The  Federal Reserve has established an organizational structure for its CCAR  assessments that is guided by core principles and some best practices,  and it continues to annually refine and develop its stress test models.  However, limitations in analytical approaches and to disclosure present  challenges to risk assessment by the Federal Reserve and to  transparency. In some cases, the Federal Reserve has not always  followed its own guidance or principles.
Quantitative assessment. The Federal Reserve has based its  determinations on the results of both the supervisory and company- run stress tests. However, this creates tension between companies’  desire to avoid failing the CCAR quantitative assessment and the  robustness of their stress test decisions. By including company-run  tests in the CCAR quantitative assessment, the Federal Reserve  limits the risk-management and capital planning benefits for  participating companies—one of the Federal Reserve’s goals for  CCAR—without significantly increasing the effectiveness of the  quantitative assessment.
Qualitative assessment disclosure and communication. Although  it uses a decision-making framework to assess qualitative CCAR  submissions, the Federal Reserve has not publicly disclosed  information that would allow for a better understanding of its  assessment methodology or the reasons for objection determinations.  Transparency is a key feature of accountability and this limited  disclosure may hinder understanding of the CCAR program and limit  public and market confidence in the program and the extent to which  the Federal Reserve can be held accountable for its decisions. The  Federal Reserve also has not regularly updated guidance to firms  about supervisory expectations and peer practices related to the  qualitative assessment. Companies that must meet these  expectations annually may face challenges from the irregular timing of  communications, which could limit the Federal Reserve’s achievement  of its CCAR goals. In addition, the Federal Reserve has not  communicated time frames for responding to questions it receives  through the CCAR communications mailbox, which could hinder  companies’ management and planning of their CCAR submissions  and limit their ability to address supervisory concerns in a timely  fashion.
Scenario design. The Federal Reserve has conducted limited  analysis of some decisions that are important to designing stress test  scenarios. IMF principles for supervisory stress testing highlight the  risks of basing scenario design decisions solely on historical  experience, but the Federal Reserve’s decisions about the severity of  its scenarios have been driven by U.S. postwar historical experience.  Without a broader consideration, the Federal Reserve could miss  opportunities to assess and guard against relevant but unprecedented  risks to the banking system. In addition, the Federal Reserve has not  explicitly analyzed how to balance scenario severity choices’ influence  on banking system resiliency with potential economic effects. Without  more careful assessment of the trade-offs associated with scenario  severity, the Federal Reserve cannot be reasonably assured that the  scenario design process balances any improvements in the resiliency  of the banking system with any impact on the cost and availability of  credit.
The Federal Reserve also has not conducted analyses to determine if  its single severe supervisory scenario is sufficiently robust and reliable  to promote the resilience of the banking system against a range of  potential crises. Such analyses—including performing sensitivity  analysis involving multiple scenarios—could help the Federal Reserve  understand the range of outcomes that might result from different  scenarios and explore trade-offs associated with reliance on a single  severe supervisory scenario. Additionally, the Federal Reserve has  not assessed whether or how changes to the supervisory scenarios  could inadvertently amplify economic cycles (procyclicality)—which its  scenario design policy aims to avoid—until after it has finalized the  scenarios. Without additional analysis prior to completing and  publishing its scenarios, the Federal Reserve cannot be reasonably  assured that small adjustments to the scenario variables would  produce outcomes that neither amplify nor dampen economic cycles.
Model risk management. The Federal Reserve’s model risk- management efforts have not focused on the system of stress test  models and how component modeling choices affected overall stress  test results. In this sense, the Federal Reserve has limited its  perspective and it has not always followed its guidance for banking  institutions on model risk-management practices.
The Federal Reserve has not assessed its entire system of  models in relation to the model development principles that it has  applied to individual component models. By not applying those  principles to the system of models, the Federal Reserve has  limited its ability to manage the extent to which model risk is  introduced into the supervisory stress test models.
It has not developed appropriate documentation of the system of  models that would allow for effective management of the risks  posed by component model interactions. Without such  documentation, the Federal Reserve’s ability to effectively identify  and manage model risk from the entire system of models is  limited, and staff may miss important connections between  elements of component models, which in turn may limit  understanding of risks inherent in their modeling choices.
The Federal Reserve has not conducted sensitivity and  uncertainty analyses of how its modeling affects the post-stress  capital ratios. Without such assessments, the Federal Reserve  limits its ability to understand, communicate, and manage the risks  and reliability of its supervisory stress test results. Furthermore,  sensitivity and uncertainty analysis can result in changes to  models and even small differences in model estimates can be the  difference between the Federal Reserve approving or objecting to  an institution’s capital plan.
Staff have been unable to communicate information about the  range and sources of uncertainty surrounding the post-stress  capital ratio estimates to the Board because the Federal Reserve  has not conducted the necessary analyses. Unless staff  communicate such information, the Board may not be fully  informed of significant model risks from individual component  models and in the aggregate including when making decisions  based on stress test results.
Neither the Board of Governors nor senior staff have identified risk  tolerances for model output or overall stress test results. Without  systematically identifying and communicating acceptable levels of  risk in its supervisory stress test models, the Federal Reserve may  be limited in its ability to effectively evaluate and manage its  model risk.
Successfully managing model risk is a key objective because the Federal  Reserve uses the system’s overall stress test results with precision to  make CCAR determinations. A more holistic approach can help ensure  that it makes the determinations with a more complete understanding of  the stress test results’ uncertainty and sensitivity to component model  decisions and account for them appropriately.
section_title: Recommendations for Executive Action
paragraphs: We are making the following 15 recommendations:  To help improve the consistency of federal banking regulators’ stress test  requirements and help ensure that institutions overseen by different  regulators receive consistent regulatory treatment, the heads of the  Federal Reserve, FDIC, and OCC should harmonize their agencies’  approach to granting extensions and exemptions from stress test  requirements.
To help provide stronger incentives for companies to perform company- run stress tests in a manner consistent with Federal Reserve goals, the  Federal Reserve should remove company-run stress tests from the  CCAR quantitative assessment.
To increase transparency and improve CCAR effectiveness, the Federal  Reserve should take the following four actions:
Publicly disclose additional information that would allow for a better  understanding of the methodology for completing qualitative  assessments, such as the role of ratings and rankings and the extent  to which they affect final determination decisions.
For future determinations to object or conditionally not object to a  company’s capital plan on qualitative grounds, disclose additional  information about the reasons for the determinations.
Publicly disclose, on a periodic basis, information on capital planning  practices observed during CCAR qualitative assessments, including  practices the Federal Reserve considers stronger or leading practices.
Improve policies for official responses to CCAR companies by  establishing procedures for notifying companies about time frames  relating to Federal Reserve responses to company inquiries.
To strengthen the scenario design process, the Federal Reserve should  assess—and adjust as necessary—the overall level of severity of its  severely adverse scenario by taking the following two actions: establish a process to facilitate proactive consideration of levels of  severity that may fall outside U.S. postwar historical experience, and expand consideration of the trade-offs associated with different  degrees of severity.
To improve understanding of the range of potential crises against which  the banking system would be resilient and the outcomes that might result  from different scenarios, the Federal Reserve should assess whether a  single severe supervisory scenario is sufficient to inform CCAR decisions  and promote the resilience of the banking system. Such an assessment  could include conducting sensitivity analysis involving multiple severe  supervisory scenarios—potentially using CCAR data for a cycle that is  already complete, to avoid concerns about tailoring the scenario to  achieve a particular outcome.
To help ensure that Federal Reserve stress tests do not amplify future  economic cycles, the Federal Reserve should develop a process to test  its proposed severely adverse scenario for procyclicality annually before  finalizing and publicly releasing the supervisory scenarios.
Finally, to improve the Federal Reserve’s ability to manage model risk  and ensure that decisions based on supervisory stress test results are  informed by an understanding of model risk, the Federal Reserve should  take the following five actions:
Apply its model development principles to the combined system of  models used in the supervisory stress tests.
Create an appropriate set of system-level model documentation,  including an overview of how component models interact and key  assumptions made in the design of model interactions.
Design and implement a process to test and document the sensitivity  and uncertainty of the model system’s output—the post-stress capital  ratios used to make CCAR quantitative assessment determinations— including, at a minimum, the cumulative uncertainty surrounding the  capital ratios and their sensitivity to key model parameters,  specifications, and assumptions from across the system of models.
Design and implement a process to communicate information about  the range and sources of uncertainty surrounding the post-stress  capital ratio estimates to the Board during CCAR deliberations.
Design and implement a process for the Board and senior staff to  articulate tolerance levels for key risks identified through sensitivity  testing and for the degree of uncertainty in the projected capital ratios.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Federal Reserve, FDIC, and OCC  for review and comment.  The Federal Reserve, FDIC and OCC provided  written comments that we have reprinted in appendix III, IV, and V,  respectively. The Federal Reserve, FDIC and OCC also provided  technical comments that we have incorporated, as appropriate.
In their written comments the Federal Reserve, FDIC, and OCC generally  agreed with the recommendation that the heads of the Federal Reserve,  FDIC, and OCC should harmonize their agencies’ approach to granting  extensions and exemptions from stress test requirements. The FDIC  agreed that a consistent approach to extensions and exemptions was  important and noted its commitment to coordinating closely with the  Federal Reserve and OCC. The Federal Reserve, FDIC, and OCC noted  that although the agencies coordinate closely in administrating their  stress testing programs, going forward they each stated that they would  coordinate with the other agencies at least annually and more frequently,  if appropriate, to discuss any planned extensions and exemptions prior to  any action.
In its written comments, the Federal Reserve generally agreed with the  report’s other 14 recommendations and offered responses in the following  areas:
Regarding our recommendation to exclude company-run stress  tests from the CCAR quantitative assessment, the Federal  Reserve noted in its letter that the agency was already  considering a proposal that would set post-stress capital  requirements for covered institutions based solely on the  supervisory stress tests. It noted that this proposal was consistent  with our recommendation. While we have not yet had the  opportunity to assess this proposal in detail, excluding the  company-run tests from such a capital requirement could improve  incentives for the company-run stress tests.
Regarding our recommendation to strengthen the scenario design  process by considering levels of severity that fall outside U.S.  postwar history, the Federal Reserve noted in its letter that the  2012 and 2013 severely adverse scenarios featured  unemployment rates that were above what has been experienced  in postwar U.S. history. However, the level of the unemployment  rate for these years does not imply an established process  designed to facilitate a consistent consideration of severely  adverse scenarios outside of the postwar historical experience. In  response to the Federal Reserve’s written comments we modified  the language in our recommendation to clarify that we are  recommending an established process for a broader consideration  of severity given that the current scenario design policy and  process is focused on selecting economic conditions that reflect  the severity of postwar U.S. recessions. Without consistently and  proactively considering levels of severity outside postwar U.S.  historical experience, the Federal Reserve could miss  opportunities to assess and guard against relevant but  unprecedented risks to the banking system.
Regarding our recommendation to expand consideration of the  trade-offs associated with different degrees of severity, the  Federal Reserve noted in its letter that the scenario design  framework was not designed to generate the most severe  potential outcomes since that might impinge credit availability.  However, our recommendation does not call for the Federal  Reserve to generate scenarios that represent the most severe  potential outcomes. Our recommendation calls for the Federal  Reserve to assess whether more severe, or less severe,  scenarios might better balance changes in resiliency against the  need to extend credit. As we noted in our report, without a more  careful assessment of scenario severity, the Federal Reserve  cannot be reasonably assured that the scenario design process  balances any improvements in the resiliency of the banking  system with any impact on the cost and availability of credit.
Regarding our recommendation to assess the sufficiency of a  single severe scenario for the supervisory stress tests, the Federal  Reserve noted in its letter that expanding the number of scenarios  would be costly and burdensome. As we noted in the report, using  a single severe scenario could limit the resources required to  design and execute the stress tests. Although the Federal  Reserve states incorrectly in their letter that we describe these  costs as “substantial,” we did not assess these potential costs in  this report.  More importantly, our recommendation does not call  for the Federal Reserve to increase the number of severe  supervisory scenarios. Our recommendation calls for the Federal  Reserve to assess the sufficiency of a single severe supervisory  scenario. Absent such an assessment—which could be supported  by sensitivity analysis using more than one severe supervisory  scenario—CCAR decisions may not reflect the uncertainty in  stress test outcomes that might result from different scenarios.
Regarding our recommendation to test the severely adverse  scenario for procyclicality before finalization and public release,  the Federal Reserve noted in its letter that the scenario design  process had a feature designed to counteract procylicality. It also  noted that additional changes were under consideration to further  reduce procylicality (i.e., further limiting the increase in the  unemployment rate during a downturn).  However, as we noted in  the report, given the complexity of the system of models, without  conducting additional testing before releasing scenarios, the  Federal Reserve cannot be reasonably assured that small  adjustments to the unemployment rate would produce outcomes  that neither amplify nor dampen economic cycles.
Regarding the recommendations to increase transparency and  improve CCAR effectiveness, the Federal Reserve stated that  steps were taken to enhance transparency and highlighted  guidance released since 2011, including supervisory letters  released in 2015 and additional details in the CCAR 2016 results  disclosure. Importantly, the Federal Reserve stated that it will  continue to enhance transparency in the areas recommended in  our report. In addition, the Federal Reserve stated that it will  continue to enhance the process for responding to firms’ inquiries  while noting that complex questions may take longer to resolve.
Regarding our recommendation to improve documentation of the  system of models, the Federal Reserve asserts that it already  maintains comprehensive documentation of the development,  assessment, validation, and finalization of its system of models.  While the Federal Reserve does maintain extensive  documentation of each element of the system of models,  comprehensive documentation of the system of models as a  whole requires documentation of how component models interact  and key assumptions made in the design of model interactions,  which the Federal Reserve was not able to provide to us.  Without  this additional documentation, the Federal Reserve’s ability to  effectively identify and manage model risk from the entire system  of models is limited and may limit understanding of risks inherent  in its modeling choices.
Regarding our recommendation to test and document the  sensitivity and uncertainty of the model system’s output used to  make its quantitative determinations, the Federal Reserve notes  that it already assesses how model assumptions impact post- stress capital ratios. However, the Federal Reserve did not  provide us with documentation that demonstrated any  comprehensive assessments that tested the mathematical and  statistical implications of their system of models design. Lack of  such testing exposes the Federal Reserve to model risk and limits  its abilities to direct model development resources to the areas  that introduce the most uncertainty and risk to estimates of the  final post-stress capital ratios.
Regarding our recommendations to improve communication of the  range and sources of uncertainty surrounding the post-stress  capital ratio estimates to the Board during CCAR deliberations  and to articulate tolerance levels for key risks, the Federal  Reserve notes it established the Supervisory Stress Test Model  Governance Committee in 2015 and has future plans to advise the  Board on the state of model risk. The Committee was too new  during the bulk of our audit work to meaningfully assess its  implementation. However, we plan to continue to monitor the  Committee to determine whether their activities ultimately address  our recommendation.
We are sending copies of this report to the House Committee on  Financial Services, the Federal Reserve, FDIC, and OCC. In addition, the  report is available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-8678 or EvansL@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made major contributions to this report  are listed in appendix VI.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The Board of Governors of the Federal Reserve System (Federal  Reserve) conducts two stress test exercises: the Dodd-Frank Stress  Tests (DFAST) and the Comprehensive Capital Analysis and Review  (CCAR). This report (1) compares the DFAST and CCAR exercises and  discusses company and Federal Reserve views about the exercises’  costs and benefits; (2) examines the CCAR qualitative assessment,  including the extent of communication and disclosure; (3) examines how  the Federal Reserve designs the supervisory scenarios for the stress  tests; and (4) examines the Federal Reserve’s modeling process for the  stress tests.
To compare DFAST and CCAR, we reviewed Section 165(i) of the Dodd- Frank Act, the Federal Reserve’s final and amended capital plan and  stress test rules, and Federal Reserve policies and procedures about how  it has implemented and used DFAST and CCAR in its supervision of  banking institutions. We analyzed internal guidance documents and  instructions, methodology, and results publications related to DFAST and  CCAR; supervisory letters on stress testing and capital planning; public  statements by Federal Reserve officials; and other Federal Reserve  documentation about the programs. We interviewed staff from the offices  of the Federal Reserve that are responsible for DFAST and CCAR,  including the Federal Reserve’s Division of Banking Supervision and  Regulation, regarding the scope, goals, and utilization of each program.  We analyzed information and documentation on stress test extensions  and exemptions from the Federal Reserve, the Office of the Comptroller  of the Currency (OCC), and the Federal Deposit Insurance Corporation  (FDIC). We also interviewed staff from the Federal Reserve, OCC, and  FDIC about the use of extensions and exemptions.
To obtain views on the stress tests and their costs and benefits, we  judgmentally selected and interviewed 13 companies that participated in  CCAR in 2015 and 6 companies that were subject only to DFAST. To  select CCAR companies to interview, the team used information on  CCAR firms collected from the Federal Reserve and SNL Financial, a  private provider of data on the financial services industry. We used the 31  bank holding companies that participated in the 2015 CCAR cycle as our  selection pool and selected companies based on their size, industry type,  organization type, prior stress test participation, and history of CCAR  results. To identify and select companies that were subject to DFAST but  not CCAR, we used data from the Federal Reserve and its National  Information Center. To expand the coverage and information from each  interview, we selected bank holding companies subject only to DFAST  that also had a subsidiary depository institution subject to stress test  requirements (including firms subject to OCC or FDIC rules). We grouped  the depository institutions by charter type—(1) state-chartered banks that  were members of the Federal Reserve System, (2) state-chartered banks  that were not members of the Federal Reserve System, and (3)  nationally-chartered banks—and ordered them by total asset size. We  selected institutions with the largest amount of total assets and the  company with the smallest amount from each of the three groups that  also had a holding company subject to DFAST. If we were unable to  schedule interviews with selected companies we chose additional  companies based on the same selection criteria. We also reviewed  Federal Reserve statements on benefits and costs.
To characterize companies’ views throughout the report, we consistently  defined modifiers (e.g., “nearly all”) to quantify each group of  interviewees’ views as follows: “all” represents 100 percent of the group,  “nearly all” represents 80 percent to 99 percent of the group, “most”  represents 60 percent to 79 percent of the group, “several” represents 40  percent to 59 percent of the group, and “some” represents 20 percent to  39 percent of the group. While the percentage of the group of interviews  remains consistent, the number of interviews each modifier represents  differs based on the number of interviews in that grouping: 19 total CCAR  and DFAST firms, 13 CCAR companies, and 6 DFAST-only companies.  Table 8 provides the number of interviews in each modifier for each group  of interviews.
To examine the process used by the Federal Reserve to conduct the  CCAR qualitative assessment, we reviewed the Federal Reserve’s stress  test and capital plan rules and other publicly available documents  including annual stress test instructions and results, periodically released  guidance on supervisory expectations and other topics, and supervisory  letters regarding stress testing and bank supervision. We also examined  internal policies and procedures, training documents, and other program  documentation related to the CCAR qualitative assessments including  communication with companies and documentation of Board decision  making. The policy and procedure documents included CCAR program  manuals and project plans that described roles and responsibilities of  staff teams and oversight groups and identified the content and timing of  key tasks, among other things. Program documentation we analyzed  included evaluation memorandums from four assessment teams covering  8 companies from CCAR 2014 and 2015. We judgmentally selected  company-specific workpapers based on companies we interviewed and  the involvement of different staff teams from across the Federal Reserve  System. We also reviewed conclusion and recommendation  memorandums used in making object or non-objection determinations. To  examine communication with companies, we reviewed communication  procedures and company-specific feedback provided to companies  including questions and responses provided through the Federal  Reserve’s communication mailbox. We interviewed Federal Reserve staff  about how they conduct the qualitative assessment including their  policies, procedures, and decision-making process as well as their  communication with companies about the assessment and the Federal  Reserve’s supervisory expectations. We also interviewed officials from 13  CCAR companies about their experience with the qualitative assessment  and interaction with the Federal Reserve including the clarity of  supervisory expectations, program guidance, and feedback. We used  criteria from Standards for Internal Control in the Federal Government  and transparency principles, including directives issued by the Office of  Management and Budget, to evaluate the Federal Reserve’s qualitative  assessment process and communication with companies.
To examine how the Federal Reserve designs the supervisory scenarios  for the stress tests, we conducted interviews and reviewed public and  nonpublic documentation related to the scenario design process. We  interviewed Federal Reserve officials about the scenario design process,  including key considerations and rationales for scenario design policy  decisions. We interviewed officials from the International Monetary Fund  (IMF) and Bank for International Settlements regarding their own research  and experience conducting stress tests. We reviewed public Federal  Reserve documentation, including the Policy Statement which governs  the scenario design process, CCAR instructions, and the CCAR  assessment framework. We analyzed public data from the supervisory  quantitative scenarios from 2013 to 2016. We also reviewed nonpublic  Federal Reserve documents including internal presentations related to  proposed scenarios and CCAR results. To understand relevant standards  for complex analyses and stress testing, we reviewed IMF principles for  supervisory stress tests and Office of Management and Budget standards  for assessing the impact of regulations. Finally, we reviewed Basel  Committee on Banking Supervision analyses of the potential impact of  post-crisis reforms to strengthen bank capital and liquidity regulations and  IMF’s 2015 U.S. Financial Sector Assessment Program.
To examine the Federal Reserve’s supervisory stress test modeling  process, we collected and reviewed public and nonpublic Federal  Reserve documentation including DFAST- and CCAR-related  publications, internal guidance and procedures, policy statements, model  documentation, model validation reports, and internal presentations. For  model-specific documentation, we reviewed model documentation and  validation reports from the DFAST/CCAR 2015 stress test cycle (the most  recent available at the time of our examination) for a judgmentally  selected sample of component models. After reviewing publicly-available  model documentation and examples of nonpublic documentation provided  by the Federal Reserve, we requested and analyzed the documentation  and validation reports for four supervisory modeling teams, which we  selected based on our assessment of their likely importance to the  system of models or their potential for presenting analytical challenges.  We interviewed Federal Reserve staff from across the Federal Reserve  System, including staff involved with supervisory stress test model  development and validation, about the process for executing the  supervisory stress tests and the Federal Reserve’s model risk management practices. We assessed the Federal Reserve’s supervisory  stress test practices using the Federal Reserve’s guidance to bank  holding companies on their stress test model risk management activities.  To provide additional context for the Federal Reserve guidance, we  reviewed publications of the National Research Council, whose members  are drawn from the National Academy of Sciences, the National Academy  of Engineering, and the Institute of Medicine, on best practices in complex  modeling and model risk management.
We conducted this performance audit from December 2014 to November  2016 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Reviews by the Federal Reserve’s Model Validation Unit and the Office of Inspector General
paragraphs: Board of Governors of the Federal Reserve System (Federal  Reserve) Model Validation Unit review. The December 2014 review  findings included a shortcoming in policies and procedures, insufficient  model testing, insufficient planning and procedures to address the risks  posed by potential key personnel departures, and incomplete structures  and information flows to ensure proper oversight of model risk  management. The review resulted in six recommendations to address the  identified findings.
Office of Inspector General (OIG) report. The OIG report identified  continuing risks related to model validation and broader governance  practices in four areas.
Risks related to validation staffing and performance management  existed that may not be mitigated by the implementation of a new  staffing approach. These risks included insufficient performance  feedback to supplemental reviewers, dependence on key personnel,  and inadequate scrutiny of models.
Risks associated with model changes that occur late in the  supervisory stress testing cycle remained despite Federal Reserve  steps to address these risks.
The Federal Reserve did not maintain an accurate, complete, and  updated inventory of models as required by the Federal Reserve’s  model risk-management guidance.
In reviewing a sample of validation reports, limitations encountered by  reviewers during model validation were not always made clearly  identifiable for management in the validation reports submitted to  management.
To address these risks, the OIG report made eight recommendations to  the Division of Banking Supervision and Regulation, including to establish  processes for assessing the materiality of late-stage changes to models  that would clarify what changes required independent validation and  would leverage reviewer resources to validate such changes.
According to Federal Reserve staff and program documents, the Federal  Reserve has been implementing changes to address concerns raised by both reviews. For example, in response to the Model Validation Unit  review, the Federal Reserve officials said created the Supervisory Stress  Test Model Governance Committee was created to coordinate and  oversee its model risk-management efforts. Federal Reserve staff said  that the committee met for the first time in May 2015 and explained that  its agenda largely has been driven by responding to the findings of the  validation unit’s governance review, in particular around model risk. One  of the review’s findings was that incomplete governance structures and  information flows did not ensure proper oversight of model risk  management. The staff noted that the committee was formed to introduce  more structure and discipline to the model governance role, including by  clarifying reporting lines to the Director of the Federal Reserve’s Division  of Banking Supervision and Regulation, who oversees the Model  Oversight Group and the Model Validation Unit.
Federal Reserve staff said that the new committee provides a formal  venue for discussing differences of opinion and advising the director,  some of which was done informally in the past. The staff also noted that  they have been exploring opportunities to expand communication of  information about model risk with the Board of Governors, including  allowing Governors to communicate their preferences regarding modeling  decisions and levels of risk.
In its response to the OIG review, the Federal Reserve said that it had  already made improvements to address a number of the  recommendations and was taking actions in response to others.  According to OIG staff, all of the report’s recommendations remained  open as of July 2016.
section_title: Appendix III: Comments from the Board of Governors of the Federal Reserve System
section_title: Appendix IV: Comments from the Federal Deposit Insurance Corporation
section_title: Appendix V: Comments from the Office of the Comptroller of the Currency
section_title: Appendix VI: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
paragraphs: Lawrance L. Evans, Jr., (202) 512-8678, EvansL@gao.gov.
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Andrew Pauline (Assistant  Director), Kevin Averyt (Analyst-in-Charge), Nancy Barry, Abby Brown, A.  Nicole Clowers, Aaron Colsher, Michael Hoffman, Risto Laboski, Marc  Molino, Barbara Roesmann, and Jessica Sandler made key contributions  to this report. Other assistance was provided by Vida Awumey, Don  Brown, David Dornisch, Janet Eackloff, Nathan J Gottfried, Nicholas  John, Rob Letzler, Joseph O’Neill, Nadine Garrick Raidbard, Anne  Stevens, Karen Tremba, and Jason Wildhagen.