Federal Reserve: Additional Actions Could Help Ensure the Achievement of Stress Test Goals
Released: 2016-11-15T00:00:00

Letter

Background

Stress Test Types and Purposes

Stress testing is one of many risk-management tools used by both financial institutions and regulators. Complex financial institutions need management information systems, internal controls, and other processes that can help to identify, assess, and manage a range of risks across the organization that may arise from both internal and external sources, including rapid and unanticipated changes in financial markets.

Stress testing has been used throughout the financial industry for several decades. But as noted in a Federal Reserve Bank of New York staff report, stress testing before the recent financial crisis was seen as one of many risk-management tools and was not a major component of banking regulators’ supervisory programs. Since the recent financial crisis, the report explains that comprehensive firm-wide stress testing has become an integral and critical part of firms’ internal capital adequacy assessment processes and of banking regulators’ supervisory regimes. The expanded role of supervisory stress testing is discussed later in this report.

IMF has identified four major categories of stress testing, differentiated by purpose or goals: (1) internal risk management, used by firms to manage risks from their investment or asset portfolios and as an input for business planning; (2) crisis management, used by supervisors to assess whether institutions need additional capital during times of financial sector distress—such as with SCAP—and as an input for business restructuring plans; (3) microprudential (supervisory), used by supervisors to assess the health of an individual institution; and (4) macroprudential (surveillance), used by central banks and other authorities to analyze system-wide risks and vulnerability in addition to institution-specific risks. As discussed later in this report, the Federal Reserve’s CCAR and DFAST have elements of the internal risk management, microprudential, and macroprudential approaches.

Federal Banking Regulators

Federal banking regulators supervise the activities of banking institutions and require them to take corrective action when the institutions’ activities or overall performance present supervisory concerns or could result in financial losses to FDIC’s deposit insurance fund or violations of law or regulation. See table 1 for an overview of their functions.

Bank Capital

For banking institutions, capital exists to absorb unexpected losses and the amount of capital an institution holds is critical to its ability to continue operating by making loans to businesses and consumers. The Federal Reserve, FDIC, and OCC require institutions to maintain certain minimum levels of capital to promote stability across the banking industry and protect the nation’s financial system. These requirements identify various types of regulatory capital including common equity Tier 1 capital, additional Tier 1 capital, Tier 2 capital, and total capital. According to Federal Reserve staff, common equity tier 1 capital is considered the most significant capital that a banking institution can have to support its operations and absorb unexpected financial losses. It consists primarily of retained earnings (the profits a bank has earned but has not paid out to shareholders in the form of dividends or other distributions) and common stock, with deductions for items such as goodwill and deferred tax assets. Tier 2 capital contains supplementary capital elements such as subordinated debt, a portion of loan loss reserves, and certain other instruments. Total capital consists of the sum of Tier 1 and Tier 2 capital.

Regulators establish required capital levels in comparison with various measures of an institution’s assets and the minimum requirements are specified as a ratio (regulatory capital ratio). Regulators use different ratios to assess an institution’s capital adequacy. Among these are the Tier 1 risk-based capital ratio, which measures Tier 1 capital as a share of risk-weighted assets, and the Tier 1 leverage ratio, which measures Tier 1 capital as a share of average total consolidated assets. Other measures include the total risk-based capital ratio (total capital as a share of risk-weighted assets) and the common equity Tier 1 ratio (common equity tier 1 capital as a share of risk-weighted assets).

Federal Reserve’s Stress Test Programs Are Coordinated but Serve Different Purposes

The DFAST and CCAR programs vary in terms of the firms to which they apply and in their uses. DFAST applies to a broad range of banking institutions and consists of supervisory- and company-run stress tests to generate forward-looking information about institutions’ capital adequacy for the firms’ internal use and for public disclosure. The Federal Reserve uses CCAR (which builds on information from DFAST) to quantitatively and qualitatively evaluate the capital adequacy and capital planning processes of large bank holding companies. The Federal Reserve and other bank regulators have issued similar rules for DFAST company-run stress tests as required by the Dodd-Frank Act but have differed in their use of exemptions. Several of the companies subject to Federal Reserve stress tests that we interviewed identified a range of benefits from the tests and many companies described costs as well as factors contributing to those costs, although several companies did not track costs specifically related to the tests.

Under DFAST, the Federal Reserve and Subject Institutions Project Capital Levels under Common Economic Scenarios and Disclose Comparable Information

The Federal Reserve and a broad range of banking institutions use DFAST to (1) project how hypothetical adverse scenarios would affect an institution’s revenues and losses and ultimately its capital levels as measured by regulatory capital ratios, and (2) disclose comparable information on test results. The Federal Reserve’s primary goals for DFAST are the production of capital adequacy information for firms’ internal use and for public disclosure. The Federal Reserve aims to provide subject companies, the public, and supervisors with forward- looking information to help gauge the potential effect of stressful economic and financial conditions on the companies’ ability to absorb losses and continue operations. Federal Reserve staff we interviewed explained that the purpose of the public stress test disclosures was transparency and the promotion of market discipline by providing market participants with comparable information on the financial condition of banking institutions. The Federal Reserve also intends banking institutions to incorporate stress testing into their internal capital planning activities.

Subject Firms and Test Components

DFAST consists of supervisory- and company-run stress tests that are required by the Dodd-Frank Act and are based on a banking institution’s size and type (see table 2). Federal Reserve-supervised banks and holding companies with more than $10 billion in total consolidated assets must perform company-run tests. The Federal Reserve also conducts a supervisory stress test for bank holding companies with total consolidated assets of $50 billion or more. Banking institutions with more than $10 billion in total consolidated assets that are supervised by FDIC and OCC are subject to stress test rules issued by the agencies that also require the completion of annual company-run stress tests.

For the supervisory tests, the Federal Reserve uses data provided by the companies and a set of models developed or selected by the Federal Reserve. Companies use their own data and models to complete the company-run tests.

The supervisory stress tests performed by the Federal Reserve and the company-run stress tests conducted by DFAST firms share several key components necessary for producing post-stress capital ratios. These ratios, which are an important output of the stress tests, reflect projections of risk-weighted assets and balance sheet and income statement items under the stress scenarios and measure the amount of capital a banking institution has available to cover unexpected losses. For example, the Federal Reserve’s DFAST stress test rules generally require both types of tests to include, over a nine-quarter planning horizon, projections of revenues, regulatory capital ratios.

Other areas in which the tests share some common elements include data, assumptions, and scenarios. As stated previously, companies use their own data to complete company-run tests and report similar data to the Federal Reserve that it uses to perform the supervisory tests. The Federal Reserve’s stress test rules also prescribe a standard set of assumptions that are used in both types of tests. The assumptions involve capital actions, or decisions about transactions affecting capital levels, such as raising capital by issuing new capital instruments or returning capital to shareholders through dividend payments or share repurchases. These transactions can affect the outcome of regulatory capital ratios measured by the stress tests.

As required under the Dodd Frank Act, the Federal Reserve also annually defines three stress test scenarios—baseline, adverse, and severely adverse—that it uses for the supervisory stress test and requires DFAST firms to use in the annual company-run tests. The scenarios consist of hypothetical projections for 28 macroeconomic and financial variables. For instance, the variables include measures of the unemployment rate, gross domestic product, housing and equity prices, interest rates, and financial market volatility.

Baseline scenario. Generally reflects economic conditions expected by economic forecasters.

Adverse scenario. Features mild to moderate economic and financial stress driven by selected potential risk factors.

Severely adverse scenario. Features severe economic and financial stress, generally driven by a different set of risk factors than the adverse scenario.

The Federal Reserve’s rule requires certain elements to be included in public summaries of DFAST results. Pursuant to the Dodd-Frank Act, Federal Reserve rules for DFAST require subject banking institutions to submit their stress test results to the Federal Reserve and also to publicly disclose a summary of the results of the severely adverse scenario. The rules require the summary from the severely adverse scenario to include (1) a description of the types of risks and methodologies included in the stress test; (2) estimates of aggregate losses, pre-provision net revenue, provision for loan and lease losses, net income, and projected regulatory capital ratios; and (3) an explanation of the most significant causes for the changes in regulatory capital ratios. The Dodd-Frank Act also requires the Federal Reserve to disclose a summary of its supervisory stress test results. The Federal Reserve has done so in the aggregate and individually for each of the banking institutions subject to the supervisory tests and the disclosures have included similar types of information as required for DFAST institutions.

The use of standard approaches in the supervisory and company-run stress tests enhances the comparability and usefulness of public results disclosures. Specifically, the common capital action assumptions, stress scenarios, and nine-quarter planning horizon used in DFAST allow for more consistent capital adequacy assessments, while also allowing for a focus on the particular characteristics of different institutions. For example, as the Federal Reserve explained in its 2016 DFAST supervisory stress test results, differences in loan loss rates across institutions reflect differences in the risk characteristics of the portfolios held by each institution (both in relation to the type of lending of each portfolio and the loans within each portfolio). Stress test rules issued by FDIC and OCC for their supervised institutions also include the use of common scenarios and test horizons.

Uses

The Federal Reserve uses DFAST results to supplement its ongoing supervision and inform its CCAR evaluations. The Federal Reserve follows two different approaches that are based on an institution’s size and type.

The Federal Reserve conducts supervisory stress tests of the largest bank holding companies (those with at least $50 billion in total consolidated assets). The Federal Reserve then uses the supervisory and company-run stress test information as a basis for quantitative and qualitative CCAR evaluations. As we discuss in more detail later in the report, CCAR is a comprehensive assessment of a company’s capital adequacy and capital planning processes.

For all other DFAST institutions (holding companies and banks), the Federal Reserve does not conduct a supervisory test. Rather, it uses company-run stress tests to supplement its regular, ongoing supervision. For example, internal Federal Reserve guidance states that examiners are expected to assess the quality of a firm’s stress testing process and overall results as part of the broader assessment of a firm’s capital adequacy and risk-management process.

In addition, the Federal Reserve performs targeted DFAST examinations that consider how $10 to $50 billion institutions are completing DFAST stress tests and using the information they produce as part of their risk- management and capital planning processes. According to Federal Reserve documentation, in these examinations staff assess whether institutions that are not subject to CCAR have sound practices for meeting the Federal Reserve’s requirements and expectations for DFAST company-run stress tests. The examinations are structured around the requirements in the Federal Reserve’s DFAST company-run stress test rules, the standards in its final DFAST supervisory guidance for institutions with $10 billion–50 billion in assets, and the data reporting requirements in the reporting form for those companies. Staff from regional Federal Reserve Banks perform the targeted DFAST examinations. Federal Reserve staff we interviewed said that examiners performing DFAST examinations also conducted general supervisory examinations.

According to Federal Reserve staff and examiner guidance, DFAST does not represent a separate supervisory assessment, and the Federal Reserve does not make supervisory decisions based solely on a company’s stress test processes or results. Instead, for banking institutions that are not subject to CCAR, the Federal Reserve considers the results of the DFAST examinations and annual company-run stress tests as one of several factors influencing an institution’s supervisory examination rating. For example, the Federal Reserve’s internal guidance instructs examiners to consider the tests as one of many tools available to assist in the assessment of a company’s capital position and planning process and not to rely primarily upon a firm’s internal stress test results in assessing overall capital adequacy or risk management. Furthermore, Federal Reserve staff we interviewed said that DFAST was part of the overall supervisory framework for these firms and provided additional information for examiners to consider when assessing capital planning and other areas of supervisory focus. Federal Reserve guidance and staff also indicated that there were no DFAST-specific expectations for companies to meet minimum capital levels or any associated regulatory approvals (such as for proposed capital distributions).

CCAR Uses DFAST Stress Test Results and Plays a Larger and More Direct Role in Supervision

CCAR is a separate exercise in which the Federal Reserve uses information produced in DFAST as a key input to its supervisory evaluations of a subset of firms—large bank holding companies with total consolidated assets of $50 billion or more. The Federal Reserve’s goals for CCAR are to ensure that large bank holding companies have sufficient capital to withstand severely adverse economic and financial conditions and continue operations, and have strong processes for assessing their capital needs and managing their capital resources.

Subject Firms and Test Components

CCAR applies only to a subset of DFAST firms—the largest top-tier bank holding companies (with total consolidated assets of $50 billion or more) subject to the DFAST supervisory stress test (see table 3). It does not affect the other DFAST institutions—that is, certain banks and savings and loan holding companies with more than $10 billion in total consolidated assets or bank holding companies with total consolidated assets greater than $10 billion but less than $50 billion.

CCAR represents a comprehensive and independent supervisory evaluation and includes a quantitative assessment of a firm’s capital adequacy, and a qualitative assessment of its capital planning processes and capital policies. (In this section, we largely focus on the quantitative assessments; we discuss the qualitative assessment in more detail later in this report.)

Federal Reserve rules promulgated in conjunction with its stress test requirements call for CCAR institutions to submit annual capital plans to the Federal Reserve that include detailed descriptions of the company’s internal processes for assessing capital adequacy, its policies governing capital actions, and planned capital actions over the nine-quarter planning horizon.

Quantitative Assessment

In the quantitative assessment, the Federal Reserve evaluates whether a company would be able to make its planned capital distributions and meet minimum capital requirements throughout the stress period based on both supervisory and company-run stress test results. The Federal Reserve coordinates key aspects of the DFAST stress tests and the CCAR quantitative assessment such as the stress scenarios, planning horizon, and reporting requirements and time frames (see fig. 1). For example, companies submit information related to the annual DFAST stress tests to the Federal Reserve at the same time that they provide their capital plan information for CCAR. The inclusion of capital actions occurs after projecting revenues, losses, and net income, as one of the last steps in generating post-stress capital ratios.

For the quantitative assessment, the Federal Reserve uses essentially the same data, models, and projections from the DFAST supervisory stress test to calculate post-stress capital ratios for each CCAR firm. But one key distinction exists—for CCAR, the Federal Reserve uses a company’s proposed capital actions rather than the standard ones prescribed for DFAST. As with the supervisory tests, a company can use the same data, models, and projections from DFAST for the CCAR company-run stress tests, but with its planned rather than standard capital actions. As Federal Reserve staff explained, the line items relating to capital actions are the only difference between the stress tests used for DFAST and CCAR. According to these staff, standardized capital assumptions are used to project capital ratios in DFAST because the purpose of the supervisory stress test under DFAST is to estimate and disclose comparable capital adequacy information, while proposed actions are used for CCAR, which also evaluates companies’ planning processes.

In addition to the change in assumptions for capital actions, CCAR firms also conduct a stress test using at least one company-designed stress scenario that is specific to CCAR and not publicly disclosed. That is, companies still use the standard supervisory scenarios (baseline, adverse, and severely adverse) and develop another scenario that also would represent stressful conditions. The Federal Reserve requires CCAR firms to focus this additional scenario on the specific vulnerabilities of the company’s business activities and exposures. As such, company- run tests using the company-designed scenario can provide greater insight into firm-specific risks than company-run tests only using the standard supervisory scenarios. The Federal Reserve has said that it considers the results of the company-run tests—under both supervisory scenarios and the company designed scenario—in its quantitative assessment.

Inclusion of Company-Run Test Results in the CCAR Quantitative Assessment May Weaken Incentives for Severe Stress Tests

The Federal Reserve has stated that its goals for CCAR include ensuring that companies have sound capital planning and risk-management processes. Related to these goals, the Federal Reserve’s capital plan rule requires companies to use the results of company-run tests—under both supervisory scenarios and at least one stress scenario they design (company-designed scenario)—to conduct internal capital adequacy assessments that support their capital plans. Furthermore, CCAR requirements call for companies to use the company-designed scenario to stress the specific vulnerabilities of their risk profile and operations, including those related to the company’s capital adequacy and financial condition. The Federal Reserve uses the results of company-run tests in performing its CCAR quantitative assessments and has stated that a company will not receive an objection to its capital plan based on the assessment if it can meet minimum regulatory capital requirements under the company-run stress tests as well as the supervisory tests. However, this could weaken incentives for companies to create meaningful and severe stress tests that are useful for capital planning and risk management.

Based on data we analyzed for CCAR 2013 through CCAR 2015, post- stress capital ratios for the company-run tests (under both supervisory severely adverse scenario and the company-designed scenario) were higher than for the supervisory tests a substantial majority of the time (see fig. 2). That is, capital ratios declined less under the company-run tests than under the Federal Reserve’s, indicating that the companies’ results generally were further from breaching minimum capital requirements and thus less stressful.

Federal Reserve staff acknowledged that the inclusion of the company- run tests—specifically the test using the company-designed scenario—in the CCAR quantitative assessment may provide an incentive for companies to create less severe stress tests that would not generate losses large enough to breach minimum regulatory capital requirements. But, the staff did not believe these negative incentives warranted the elimination of the company-run stress test and added that the Federal Reserve could mitigate this risk through its evaluation of companies’ stress testing practices—including their scenarios and models—in the CCAR qualitative assessment. However, stress test modeling and scenario design involve considerable judgment and companies could make subtle changes that would not indicate manipulation or necessarily fail to meet Federal Reserve standards. Federal Reserve staff told us that the company-designed scenario provided an additional view on risk that increased the variety of stresses to which companies were subject under CCAR.

In addition, Federal Reserve guidance for the largest and most complex companies indicates that the firms should have more sophisticated models for their most material portfolios than for portfolios that are less significant. However, because the company-run tests typically produced higher post-stress capital ratios, they may not have meaningfully contributed to the CCAR quantitative assessment. Furthermore, in discussing why the Federal Reserve does not require disclosure of results based on company-designed scenarios with Federal Reserve staff, they explained that with a requirement to disclose such results, firms might focus on producing positive results rather than using the scenario to genuinely identify their most salient risks. Using the company-run test results (from supervisory and company-designed scenarios) in the quantitative assessment creates similar conflicting incentives for companies, which could limit the benefits of the tests and the achievement of Federal Reserve goals.

CCAR Qualitative Assessments

The Federal Reserve also uses DFAST-related information in its CCAR qualitative assessments, which we discuss in additional detail later in this report. For CCAR firms, the Federal Reserve’s qualitative assessment represents a dedicated and wide-ranging assessment of their capital planning processes. A substantial part of the assessment involves examining how companies perform stress tests—including those required under DFAST—and incorporate them in overall risk management and capital planning.

As part of this effort, the Federal Reserve assesses different aspects of the processes companies use to generate the DFAST company-run and other stress tests. Among other areas, this includes examining whether companies meet Federal Reserve expectations related to risk identification, scenario design, and loss and revenue estimation. For example, the Federal Reserve considers how a company’s stress testing practices capture the potential increase in losses or decrease in revenue that could result from the firm’s risks, exposures, and activities under stressful scenarios.

Semiannual Tests

Based on Federal Reserve Dodd-Frank Act requirements, institutions subject to CCAR must conduct company-run stress tests semiannually. The semiannual and annual stress tests are similar, but for the semiannual tests the Federal Reserve requires companies to develop and use at least three scenarios appropriate for their own risk profile and operations. The Federal Reserve has not used these tests as part of its annual CCAR quantitative assessment, and their role in the qualitative assessment has been limited. According to Federal Reserve staff, the Federal Reserve has used the semiannual tests significantly less than other DFAST stress test components and has not done the type of rigorous assessment of the semiannual stress test that it has done for the annual tests. However, Federal Reserve staff and internal guidance that we reviewed indicated that information from the semiannual tests could be used to help identify areas requiring additional focus in future CCAR cycles. For example, Federal Reserve staff said they performed limited reviews of companies’ semiannual stress test submissions—including information on stress scenarios and test results—to identify anomalies and other insights such as structural portfolio, modeling, or scenario changes.

Uses and Disclosure

CCAR plays a larger and more direct role in the supervision of subject institutions than DFAST. For example, in its publications of CCAR results the Federal Reserve has stated that it has made CCAR a cornerstone of its supervision of the largest and most complex financial institutions. The Federal Reserve uses the CCAR quantitative and qualitative assessments to determine whether to object or not object to an institution’s capital plan (including proposed capital actions such as dividend payments and share repurchases that affect the firm’s capital levels). The Federal Reserve can object to a company’s capital plan based on the quantitative or the qualitative assessment.

Federal Reserve staff also stated that the Federal Reserve has moved the focal point of its supervisory process for the largest firms toward the promotion of strong capital adequacy and liquidity planning, in addition to assessing the firms’ preparedness for recovery and resolution in bankruptcy. The staff said that the CCAR qualitative assessment is a primary contributor to overall supervision and a focal point for evaluating a company’s risk-management and internal controls. They explained that strong capital and liquidity planning requires companies to identify and measure risks, understand how risks change in adverse economic scenarios, and have robust internal controls and governance, all of which are elements of the qualitative assessment under CCAR.

The Federal Reserve also has been integrating the CCAR qualitative assessment into its regular, ongoing supervisory activities. According to the Federal Reserve’s instructions for the 2016 CCAR cycle, it is to conduct certain supervisory activities throughout the year that inform the annual CCAR qualitative assessment—which allows the Federal Reserve to consistently incorporate supervisory findings from all its examination work into the overall qualitative assessment. For example, Federal Reserve staff we interviewed said that if their supervision identified weaknesses in a company’s internal audit functions, these weaknesses would be relevant to the internal controls and governance assessments within CCAR. Federal Reserve staff explained that as part of the integration of the CCAR into year-round supervision, the Federal Reserve has established teams of subject-matter experts from across the Federal Reserve System to link their work with CCAR. For example, the Federal Reserve has already gathered subject-matter experts in modeling loss revenue to contribute to CCAR throughout the year.

The Federal Reserve publishes firm-specific CCAR results separately from its DFAST disclosures. Since it initiated CCAR in 2011, the Federal Reserve has increased disclosure of firm-specific CCAR results, including its supervisory assessments. After the initial CCAR cycle, the Federal Reserve published only an overview of its objectives and methodology and did not reveal whether it had objected or not objected to an institution’s capital plan. The Federal Reserve currently discloses firm- specific capital ratio projections (based on supervisory stress test results and firms’ proposed capital actions) and its decision to object or not object based on the quantitative or qualitative assessment. The Federal Reserve also discloses the reasons for any qualitative objections.

Regulators Developed Comparable Stress Test Rules, but OCC Has Made Greater Use of its Supervisory Flexibility

The Federal Reserve and other bank regulators issued similar company- run stress test rules but have used different degrees of supervisory flexibility to implement them, with OCC granting the most extensions or exemptions for firms. In addition to the Federal Reserve, the Dodd-Frank Act called for certain financial regulatory agencies, including OCC and FDIC, to issue rules requiring the financial companies they supervise with more than $10 billion in total consolidated assets to conduct stress tests. OCC and FDIC rules apply to certain banks and savings associations, some of which may have a holding company subject to the Federal Reserve’s stress test rules. The act required the agencies to coordinate their stress test rules so that they are consistent and comparable. In 2012, the Federal Reserve, OCC, and FDIC each adopted substantively similar rules implementing this requirement. For example, the agencies’ rules have similar reporting time frames, public disclosure requirements, and stress test methodologies and practices.

However, OCC has made greater use of supervisory flexibility in implementing the stress test requirements. Each agency’s rule allows for the acceleration or extension of time frame requirements, while OCC’s and FDIC’s rules also include provisions for them to modify any or all of the stress test requirements. Furthermore, OCC staff stated that the agency had interpreted the Dodd-Frank Act as not requiring all banking institutions within a bank holding company to conduct separate stress tests. But according to information from the Federal Reserve and FDIC, banking institutions covered by their rules are required to conduct stress tests regardless of their status within a holding company structure.

Each of the agencies extended the time frames for a limited number of firms. As shown in table 4, the Federal Reserve provided five firms with extensions of time—ranging from 3 to 12 months—to conduct and report the results of required company-run stress tests.

FDIC provided a 1-year extension to two firms for the first stress test cycle in 2013, as shown in table 5.

OCC has granted more stress test extensions than FDIC and the Federal Reserve and is the only agency to have approved exemptions from stress test requirements. OCC issued 14 1-year extensions—including one firm that received 3 consecutive extensions—and 2 shorter extensions for stress test cycles from 2013 to 2016 (see table 6). Three firms were granted stress test exemptions for one or more of the stress test cycles from 2013 to 2016 including a firm that had been exempted for three cycles. The firms OCC exempted were part of a larger banking organization in which both the parent holding company and affiliated bank were required to conduct stress tests.

The reasons cited for granting extensions and exemptions varied among the regulators. The time extensions issued by the Federal Reserve and FDIC generally occurred in the initial stress test cycles to allow additional time for firms to implement effective stress testing systems and compile necessary data. Most of the Federal Reserve’s extensions were technical in nature. One firm reduced its total consolidated assets below the $10 billion threshold and represented that it intended to maintain its assets below the threshold for the foreseeable future. Others involved mergers, including an institution that had made a business transaction based on the timing set forth in the Federal Reserve’s original stress test rules, which were subsequently revised resulting in the institution being subject to the requirements sooner than expected. Federal Reserve and FDIC staff stated that they based their determinations for granting extensions on applicable regulatory provisions and the relevant facts and circumstances of the particular case. OCC also issued extensions to provide institutions that had not previously been subject to regulatory stress test requirements with additional time to construct or enhance their stress testing frameworks. OCC staff noted that such institutions lacked the necessary infrastructure to complete a stress test submission at the time its final rule was implemented.

OCC granted extensions and exemptions from stress test requirements for different reasons. For example, according to OCC staff and documents we reviewed, OCC granted extensions and exemptions based on the extent of an institution’s activities, its relative size within a large bank holding company, and if the larger entities in the holding company structure—such as the lead bank and parent holding company—were subject to stress test requirements. The bases for the extensions and exemptions also included OCC’s conclusions about the satisfactory nature of an institution’s financial condition and capital levels and the nature of its business activities and strategies.

Our prior work on financial regulatory reform identified an important characteristic of consistent financial oversight—that similar institutions and risks, among other areas, should be subject to consistent regulation, oversight, and transparency—to help minimize negative competitive outcomes while harmonizing oversight. Without a consistent approach to the implementation of the stress test rules, regulators may not be regulating financial institutions that pose similar risks similarly, which could contribute to competitive disadvantages between institutions and inconsistent oversight of risk management. Additionally, financial institutions often have options for how to operate their business which affects who will be their regulator. For instance, banks can change their charters if such a change will allow them to have a regulator perceived as having less stringent regulations.

Firms Generally Indicated That Stress Tests Offered Important Benefits but Also Required Substantial Resources

Several companies subject to DFAST and CCAR that we interviewed identified a range of benefits but also described significant costs related to these exercises. We discussed the Federal Reserve’s stress test exercises with 19 bank holding companies, including 13 of the 31 companies that participated in CCAR in 2015. We also reviewed Federal Reserve statements and interviewed staff about costs and benefits.

Companies Views on Benefits

The company officials that we interviewed generally identified overall improvements in risk management and capital planning attributable to DFAST and CCAR, as the following examples illustrate.

Several firms said that their prior stress test efforts were fragmented with different business units across the firm assessing risks independently. The firms said that Federal Reserve stress test requirements have led to more comprehensive, enterprise-wide, and forward-looking capital adequacy assessments, including the identification and measurement of risks.

One firm said the consolidation of the firm’s stress testing and its integration with capital adequacy assessments had helped the firm to quantify its risk appetite, which was not accomplished prior to DFAST and CCAR.

Some firms also identified key benefits from improved data quality and capabilities, including enhanced data collection, analysis, and reporting.

Some firms said that the stress test exercises have led to a stronger focus on the governance of capital adequacy processes and an increased involvement of senior management and the board of directors in capital planning decisions.

Firms that distinguished between the benefits of different stress test components generally said that CCAR has been more beneficial than DFAST and that company-run tests have been more useful than the supervisory tests.

Furthermore, several companies explained that the risk management and capital planning improvements have provided additional benefits for the company’s business operations. Several firms’ officials said that the stress tests have helped improve their business decisions, including by taking a more strategic approach to capital, developing tools to analyze portfolio risks, and facilitating business planning and budgeting. For example, according to officials from one CCAR firm, the stress tests have led to better pricing decisions, strategic and investment focus, and optimization of investor returns. In addition, one smaller company that has been subject to DFAST but not CCAR said that it has used stress test information in its strategic decisions about different markets and geographical locations and incorporated information about risks from asset portfolios in its pricing decisions.

Finally, several companies also identified broader, system-wide benefits related to the stress tests. Some firms said that the stress tests have led to higher capital levels and improved risk management that have contributed to the stability of the financial system. Other firms noted that comparable stress test results provide an industry-wide view of capital adequacy and comparisons across companies, which can offer a broader view of relative risks than individual firm assessments.

Companies’ Views on Costs

While companies we interviewed generally recognized benefits, they also cited costs in complying with stress test requirements. Officials from many of the companies we interviewed indicated that DFAST and CCAR have resulted in significant costs, including for staff resources and other expenditures.

Most firms said that the stress test-related costs have increased from year to year or were expected to continue increasing, although some firms noted that costs have stabilized or declined. Several firms cited what they viewed as the Federal Reserve’s continually increasing supervisory expectations—in particular those related to CCAR qualitative requirements—as a main reason to expect continued growth in stress test-related costs.

At the same time, the firms stated that they have not collected information on the specific costs directly attributable to DFAST and CCAR.

Firms generally said that measuring stress test-specific costs is difficult because the tests involve many employees from around the company who have responsibilities beyond DFAST and CCAR.

About half of the companies we interviewed provided estimates of their stress test-related costs or staff resources, which varied widely. For example, for the six CCAR firms that provided cost estimates, recurring annual costs related to both DFAST and CCAR ranged from $4 million to $7 million to more than $90 million. Half of the estimates were for $15 million to $30 million of annual costs. Some CCAR firms provided estimates of the amount of staff resources rather than costs. These estimates varied from about 100 staff for one company to approximately 500 employees with part-time responsibility for the stress tests and an additional 2,000-plus employees spending part of their time supporting the stress tests for another company. In addition, more than a third of the companies said that they used consultants—often to a significant extent—to help complete the work required for DFAST and CCAR. For the non-CCAR firms that we interviewed, cost estimates ranged from around $250,000 to $2 million.

Many of the companies we interviewed identified particular factors behind what they viewed as the substantial costs required for the stress test exercises. These included requirements related to the CCAR qualitative assessment such as documentation of processes and controls and supervisory expectations for model development and validation. Several companies noted costs from integrating or upgrading stress test-related technology and risk-management systems and the vast amounts of detailed data needed to complete the stress tests. Many firms pointed to expenditures for consultants to assist with their stress testing efforts. Officials from one CCAR firm said that they hire consultants for many tasks including modeling, documentation, and technical writing; another firm cited the risk-identification process and model risk management as two main drivers of consulting expenditures. Some firms also stressed that competition among companies for qualified staff to perform stress testing had increased the cost of individuals with quantitative skills and modeling expertise. Two firms said that the supply of qualified employees has been low compared to demand related to the Federal Reserve’s stress test requirements.

In addition to direct costs, some firms indicated that they also faced costs related to holding excess capital to ensure they did not receive an objection from the CCAR quantitative assessment. In the CCAR quantitative assessment, the Federal Reserve can object to a company’s capital distribution plan if stress test results show the company’s post- stress capital ratios falling below required minimum levels. Some of the companies said that they have held more capital than they otherwise would to account for differences between their stress test results and the Federal Reserve’s supervisory test results. The firms stated that limited transparency about the Federal Reserve’s supervisory stress test models leads to uncertainty about exactly how much capital they need to hold to avoid an objection. The companies’ estimates of the amount of additional capital they have held to avoid a CCAR objection ranged from about $500 million for one firm to $15 billion for another firm.

Federal Reserve Views on Benefits and Costs

The Federal Reserve has stated that the stress tests have provided important benefits to the financial system and subject institutions. In issuing the CCAR results for 2016, the Federal Reserve stated that the increased capital levels of large bank holding companies since the financial crisis have at least in part been due to the stress test exercises. In addition to helping strengthen large firms’ capital positions, Federal Reserve officials have noted other substantial contributions that DFAST and CCAR have made to financial supervision in several key areas.

First, officials identified improved risk management, internal controls, and governance practices at institutions subject to DFAST and CCAR.

Second, officials noted that the stress test-related exercises have led to a more forward-looking and stress-based approach to assessing capital adequacy by the Federal Reserve and institutions, which represents an improvement over previous practices focused on traditional regulatory capital ratios (which only reflect past performance).

Third, officials observed that the horizontal nature of the stress tests— with a simultaneous review across multiple firms—has provided the Federal Reserve with a more consistent and industry-wide perspective on potential risks and vulnerabilities.

Fourth, some officials identified greater supervisory transparency associated with DFAST and CCAR, including disclosure of firm- specific information that could lead to greater market discipline and information on the Federal Reserve’s framework and methodology that could contribute to supervisory accountability.

According to Federal Reserve staff we interviewed, the recent financial crisis revealed that banking institutions—including the largest and most complex firms—had significant shortcomings and gaps in their risk- measurement and risk-management systems. These deficiencies included limitations in the collection and use of data for risk identification and management and in the ability to assess potential risks to the company during periods of stress, such as a lack of data on firm risks and exposures. The staff noted that these banking institutions needed to make investments to improve these fundamental risk-management capabilities because the largest banking institutions’ financial stability has implications for the financial system and economy. The staff indicated that costs to the firms will normalize at some point and the Federal Reserve did not expect that costs would continue to increase. They explained that initial costs to establish the necessary data, risk management, internal controls, governance, and stress testing capacity can be high but future costs will be less once firms establish the needed capabilities and processes, and required data are available. They noted that although some of the new costs come from the stress test exercises alone, even without the explicit DFAST and CCAR requirements, firms still would have to take these actions and incur costs because of broader supervisory expectations for enhancing risk management, internal controls, and governance processes, including around capital adequacy assessments.

CCAR Qualitative Assessments Include Multiple Levels of Review, but Communication of Methodology and Expectations Was Limited

The Federal Reserve has identified capital adequacy principles and established an organizational and oversight structure for assessing qualitative CCAR submissions. The assessment framework includes processes to help ensure consistency across evaluations. In the qualitative assessment, the Federal Reserve uses ratings and rankings to compare firms’ capital planning practices against supervisory expectations. However, it has not disclosed sufficient information that would allow for a clear understanding of its methodology. The Federal Reserve has provided companies with information on supervisory expectations and peer practices related to the qualitative assessment, but the infrequent timing of these communications and evolving peer practices can pose challenges to companies that must meet the expectations annually.

The Federal Reserve Identified Principles and Established an Organizational Structure for Qualitative Assessments

As discussed previously, CCAR qualitative assessments are comprehensive reviews of the capital planning processes and capital policies of large bank holding companies. The Federal Reserve has structured the evaluations for its qualitative assessments around seven principles of an effective capital adequacy process (see table 7), which it has identified in public guidance documents. The principles cover different assessment topics including risk management, stress testing practices, capital policies, internal controls, and overall governance of capital planning.

The seven principles each represent distinct aspects of a CCAR evaluation and, according to Federal Reserve staff, each principle could influence others (with a deficiency in one principle often highlighting a deficiency in another principle). For example, weaknesses across different principles can signal a weakness in effective governance.

Organizational Structure for Qualitative Assessments

The Federal Reserve has established a tiered organizational structure for its CCAR qualitative assessments, with roles and responsibilities assigned throughout the CCAR program. Federal internal control standards state the importance of establishing an organizational structure and clearly assigning responsibility for key roles. According to interviews with Federal Reserve staff and our review of internal agency documents, the Federal Reserve’s structure for completing CCAR qualitative assessments is headed by the Director of the Division of Banking Supervision and Regulation. The entities within the Federal Reserve that have roles in the process range from Reserve Bank examiners to the Board of Governors.

Evaluation teams. Teams of designated staff from across the Federal Reserve System initially review and evaluate companies’ capital plan submissions for the qualitative assessment. The two types of staff teams involved are (1) on-site examination teams (supervisory on-site teams), which consist of staff from the Reserve Bank that oversees the firm; and (2) subject-matter experts (horizontal evaluation teams) assigned to assess specific aspects of capital planning and stress testing for each CCAR firm.

CCAR Executive Committee. The Executive Committee manages the CCAR program and holds ultimate responsibility for the program’s design and execution. The Executive Committee is chaired by a senior officer from the Board of Governors staff and comprises senior staff from across the Federal Reserve System, including senior staff from the Large Institution Supervision Coordinating Committee (LISCC) Operating Committee, Large and Foreign Banking Organizations (LFBO) Management Group, and the Division of Financial Stability, which monitors financial markets and analyzes potential threats to financial stability. The Executive Committee reviews the evaluation teams’ assessments and provides final ratings for each principle at each company and consolidated rankings of all companies to the LISCC Operating Committee and LFBO Management Group. While the Executive Committee is ultimately responsible for the CCAR program, the Executive Committee delegates execution and administration to the CCAR Program Oversight Group. The CCAR Program Oversight Group works with evaluation teams during their assessments to maintain consistency before providing conclusions to the Executive Committee.

LISCC Operating Committee. The LISCC Operating Committee is responsible for setting priorities for and overseeing the execution of the LISCC supervisory program—for the largest and most systemically important financial institutions subject to Federal Reserve oversight. The Operating Committee is chaired by a senior officer from the Board of Governors Staff and includes senior officials from various divisions at the Board of Governors and Reserve Banks. The LISCC Operating Committee chair reports to the Director of Banking Supervision and Regulation. For the CCAR qualitative assessment, the Operating Committee provides final recommendations to the Director of the Division of Bank Supervision and Regulation to object or not object to the capital plans of companies in the LISCC portfolio.

LFBO Management Group. The Management Group oversees the supervision of large institutions ($50 billion or more in total assets)— including foreign banking organizations—not included in the LISCC portfolio. In CCAR, the LFBO Management Group reviews and provides feedback to supervisory on-site teams on their company- specific object or non-object recommendations for these firms. The LFBO Management Group does not provide final recommendations to the Director of the Division of Bank Supervision and Regulation. Instead, the Reserve Bank responsible for each non-LISCC firm determines final recommendations in consultation with Board staff.

Director of Banking Supervision and Regulation. Banking Supervision and Regulation oversees and develops regulations for Federal Reserve-supervised banking institutions. For CCAR qualitative assessments, the Director of Banking Supervision and Regulation makes the final recommendations to object or not object to the capital plans for each firm to the Board of Governors.

Board of Governors. The Board of Governors has ultimate decision- making authority for CCAR qualitative assessment determinations. Federal Reserve CCAR staff brief the Board of Governors on the final recommendations approved by the Director of Banking Supervision and Regulation. According to Federal Reserve staff, the Board reviews all assessments but to date has only voted on whether to implement recommendations to object or conditionally not object to a company’s capital plan.

Scope of Qualitative Assessments

Federal Reserve procedures call for staff to adjust the scope of the qualitative assessment based on a firm’s size, characteristics and the materiality of risks it poses to the financial system. According to these procedures, the Federal Reserve varies the scope of its reviews based on a company’s size and complexity, so that not all companies are assessed on every aspect of the qualitative assessment each year. However, for LISCC companies, the Federal Reserve reviews all key aspects of their capital planning and capital adequacy processes over the course of an annual CCAR cycle.

For non-LISCC companies the Federal Reserve procedures dictate a risk- focused approach to identify significant aspects of a company’s capital adequacy process that it will assess in that year’s CCAR cycle. According to Federal Reserve staff, this approach places priority on reviews of companies with larger risks and systemic importance by assigning additional resources based on company characteristics. These staff also said that risk-focused reviews help to efficiently allocate Federal Reserve staff resources and that the focus could change from year to year based on the Federal Reserve’s views on certain risk areas. For example, in 2016 the Federal Reserve horizontal evaluation teams reviewed only certain areas of their assigned principles for non-LISCC companies. Staff further explained that, when applying this approach, the Federal Reserve instructs its staff to avoid focusing on risks that may be immaterial.

Federal Reserve staff said that the risk-focused approach influences the staffing of evaluation teams for the qualitative assessment. For example, horizontal evaluation teams involved in the qualitative assessment include subject-matter experts from across the Federal Reserve System and the risk-focused approach helps ensure an appropriate number of subject- matter experts can be assigned to the teams.

Federal Reserve Has Established Processes to Help Ensure Consistency across Qualitative Assessments

According to our examination of Federal Reserve documentation, the process for qualitative assessments includes procedures, documentation, and training intended to help ensure consistency across the reviews as well as multiple levels of review and oversight. A number of teams are involved in CCAR evaluations (see fig. 3).

As previously discussed, the Federal Reserve assigns two types of teams to review qualitative submissions, supervisory on-site teams and horizontal evaluation teams. In CCAR, supervisory on-site teams are to assess company submissions based on evaluation principles 1, 4, 5, 6, and 7. Supervisory on-site teams are also responsible for documenting Federal Reserve staff recommendations as their assigned company’s submission progresses through the assessment process and making their own recommendations to the LISCC Operating Committee and the LFBO Management Group.

Cross-Firm Evaluations

To help ensure a comprehensive and consistent evaluation across companies, the Federal Reserve has divided specific responsibilities among horizontal evaluation teams. A designated team or a set of teams are to assess areas corresponding to a specific evaluation principle for all the CCAR submissions. While both firm-specific and horizontal evaluation teams are to perform principle-based assessments, in some cases the horizontal evaluation teams are to provide the assessment in relation to a specific principle and in other cases to supplement the work of on-site teams.

The horizontal evaluation teams are broken out into risk evaluation teams, capital and revenue assessment teams, and the capital adequacy process review team.

Risk evaluation teams evaluate a company’s ability to measure its risk exposures under stress and create estimates of potential losses from these risks. The teams assess and propose ratings for elements covered by principle 2, which encompasses six different areas of loss- estimation practices.

Capital and revenue assessment teams evaluate and propose ratings for elements covered by principle 3, which involve a company’s ability to effectively forecast available capital resources—including revenues and expenses—during stress periods.

The capital adequacy process review team supplements the work of the supervisory on-site teams. Specifically, the process review team provides a cross-firm assessment of supervisory on-site teams’ evaluations of certain areas of capital planning—such as risk management, capital policy, internal controls, and governance—while also supporting the on-site teams in making their assessments. According to CCAR program documents, the process review team performs an oversight function by providing objective assessments of some elements of companies’ capital plans and by evaluating the reasonableness, consistency, and completeness of the supervisory on-site teams’ assessments. The process review team produces its own summary assessments of principles 1, 4, 5, 6 and 7 to provide the Executive Committee and on-site teams with a horizontal perspective (for instance, the range of practice in key areas of focus across all CCAR companies).

Another team—the scenario evaluation team—assists supervisory on-site teams by assessing the completeness and severity of companies’ internally developed stress scenarios for all CCAR firms.

Internal CCAR program documents also establish expectations and mechanisms for teams to resolve any differences that arise during their evaluations through communication and collaboration. The program documents direct teams to send unresolved differences to the Executive Committee for further and final deliberation, if needed.

Training

According to CCAR procedures, the CCAR Executive Committee oversees centralized training for staff participating in the CCAR program, which helps support a consistent approach to evaluations. Federal Reserve staff said that all staff involved in developing and executing CCAR must take annual training and that staff involved in qualitative assessments participate in additional training throughout the course of each CCAR cycle.

Training materials have included overviews of the CCAR assessment framework, decision-making process, review processes, and documentation requirements. Horizontal evaluation team leads and subject-matter experts participate in additional technical training on topics such as new modeling techniques, modeling strengths and weaknesses, and leveraging information from past CCAR cycles, according to Federal Reserve staff. In addition, on-site teams and other relevant staff may be required to participate in training on principles relevant to their work as well as lessons learned from past CCAR cycles.

Documentation

Both types of evaluation teams are to record their findings and conclusions using template forms that also have instructions to further promote consistency across different teams and evaluations. The supervisory on-site teams and horizontal evaluation teams each are to produce memorandums detailing conclusions related to their assigned capital adequacy process principles. The conclusion memorandum contains a summary of a company’s practices related to the principle being evaluated and describes identified weaknesses and shortcomings. For each principle, the relevant teams are to record a proposed rating (on a scale of one to four), trend (such as stable, improving, or declining), and an overall assessment of the team’s conclusions. In addition, the teams’ conclusion memorandums are to highlight trends in a company’s practices compared to prior CCAR exercises and the observed range of current industry practice.

Supervisory on-site teams are also to produce recommendation memorandums for object and non-object decisions, which are to provide overall conclusions and support for a team’s recommendation. The memorandums are to describe potential new supervisory findings, including actions the company should take to remediate issues. They also are to provide a summary and status of outstanding CCAR-related supervisory issues (including matters requiring attention and matters requiring immediate attention).

Multiple Levels of Review

The review process is to start after the two groups of evaluation teams have completed their assessments of each CCAR company’s capital plan and capital adequacy processes and formally documented their conclusions. According to CCAR procedures, the teams are to submit their evaluations and conclusions to successive groups of senior management for additional review, including the CCAR Executive Committee (see fig. 4). As part of the assessment process, supervisory on-site teams are to propose object or not object recommendations based on the CCAR Executive Committee’s review and conclusions.

CCAR procedures call for Executive Committee staff to lead sessions with evaluation teams to review team findings and ensure evaluations are conducted in a consistent and comparative manner across all companies. The Executive Committee is to review findings from these sessions and develops overall assessments for each company. These overall assessments are to include findings from the sessions with evaluation teams and other CCAR information and may differ from evaluation teams’ original conclusions. In developing its overall assessments, the Executive Committee is to make adjustments to reflect different supervisory expectations for companies of various sizes and levels of complexity, according to the procedures and Federal Reserve staff.

The supervisory on-site teams are to develop internal recommendation memorandums based on the review sessions and Executive Committee assessments. CCAR procedures call for supervisory on-site teams to provide the recommendation memorandums to the LISCC Operating Committee or the LFBO Management Group and to the Reserve Bank responsible for the company’s supervision.

According to CCAR procedures, the LISCC Operating Committee (for firms in the LISCC portfolio), LFBO Management Group, and responsible Reserve Banks (for firms not in the LISCC portfolio) are to review the supervisory on-site teams’ recommendations, assessment information provided by the Executive Committee, and other information from throughout the assessment. The LISCC Operating Committee and responsible Reserve Banks then are to make final recommendations for the companies for which they are responsible and provide the recommendations to the Director of Banking Supervision and Regulation for approval.

Upon approval by the Director of Banking Supervision and Regulation, the Board of Governors is to be briefed on all CCAR recommendations. However, the Board has only voted on whether to approve objection or conditional non-objection recommendations, according to Federal Reserve staff and internal documents we reviewed.

Qualitative Assessments Use Ratings and Rankings to Reflect Evaluation of Firms’ Capital Planning Practices

In evaluating a company’s capital plan and completing the CCAR qualitative assessment, the Federal Reserve produces measurements— ratings and rankings—of the extent to which company practices meet supervisory expectations. As discussed previously, evaluation teams are to structure their assessments around the Federal Reserve’s seven principles of an effective capital adequacy process. During the assessments, teams are to evaluate companies’ current practices and assign each individual company a numerical rating for each principle and applicable subcomponents. The ratings are intended to measure the extent to which a company’s capital adequacy process meets supervisory expectations.

According to Federal Reserve program documentation, evaluation teams base their rating assessments on established supervisory guidance and supervisory expectations specific to capital planning. For example, when evaluating a company’s modeling practices, Federal Reserve teams may use prior supervisory guidance on model risk management, which is also incorporated into the 2015 supervisory and regulation letters. According to Federal Reserve staff, CCAR reviews have incorporated CCAR-specific expectations and also include other long-standing guidance on internal controls, risk management, and corporate governance, particularly where such guidance is applicable to practices that support capital planning. For example, these staff said that supervisory guidance and expectations for internal controls existed before the implementation of CCAR and that this type of guidance, which was relevant before the current stress testing regime, is now enhanced by the CCAR qualitative assessment.

Ratings

The Federal Reserve uses evaluation ratings to summarize findings related to the different review components (organized by principle and subcomponents) and to develop its overall qualitative assessment for each company. Federal Reserve program guidance also states that ratings are intended to help facilitate internal discussions around deficiencies in a company’s capital adequacy process and serve as the basis for making qualitative assessment determinations. Supervisory on- site teams develop overall ratings for their assigned capital adequacy process principles. In contrast, risk evaluation teams and capital and revenue assessment teams develop ratings for both subcomponents of their assigned principles and a consolidated rating for the overall principle.

The Federal Reserve’s rating system comprises four numerical scores:

1 - strong,

2 - satisfactory,

3 - fair, and

4 - unsatisfactory.

The Federal Reserve defines each score by the degree to which a company’s practices meet supervisory expectations. The top score reflects company practices that meet expectations and include sound, transparent, and repeatable processes. Intermediate scores represent practices that either generally meet or do not meet expectations. According to Federal Reserve staff, practices rated below satisfactory may not warrant an objection but would require remediation to avoid future objections. An unsatisfactory rating is used for practices that do not meet expectations, have critical deficiencies, and will require significant corrective action. In addition, Federal Reserve evaluation teams can use plus or minus modifiers to further differentiate the intermediate scores. Horizontal evaluation teams can develop further detailed guidance on rating modifiers but must adhere to general ratings guidelines. Federal Reserve program guidance instructs evaluation teams to consider the trend in a company’s practices relative to other CCAR companies and whether its practices were above, consistent with, or below peer practices. Federal Reserve internal guidance also instructs evaluation teams to consider progress towards remediating previously identified weaknesses when assigning ratings.

Rankings

The Federal Reserve also develops rankings to compare capital adequacy practices across CCAR companies and help ensure consistency across evaluations. According to CCAR procedures, rankings are developed at various levels of the qualitative assessment (capital adequacy planning subcomponents, principles, and overall).

The procedures call for horizontal evaluation teams to develop preliminary rankings directly from assigned ratings by grouping companies for each principle, and the Executive Committee assigns the firms to cohorts based on their ratings and rankings.

According to Federal Reserve staff, companies within cohorts are considered more similar in the overall quality of their practices than they are to companies in any other cohort. Federal Reserve staff also said that firms within a cohort are not ranked, but are generally listed in alphabetical order.

The procedures also state that the rankings’ relative comparisons allow the Federal Reserve to differentiate among companies that might have the same ratings but also exhibit differences in their processes that would allow for meaningful distinctions (that is, different rankings). Figure 5 provides a hypothetical example of the rating and ranking process. Federal Reserve staff explained that evaluation teams establish rankings by identifying relative strengths and weaknesses in each company’s processes.

CCAR procedures call for evaluation teams to provide proposed ratings and rankings to the Executive Committee based on their respective evaluations. The Executive Committee is to review and approve the proposed rankings (in the form of cohorts), including an overall ranking for each CCAR firm that is to include consideration of the company’s size, complexity, and systemic importance.

According to Federal Reserve staff, the most important role of the rankings and cohorts is to help ensure consistency in the Federal Reserve’s execution of the qualitative assessment. In addition, Federal Reserve staff stated that cohorts help ensure expectations are properly tailored for LISCC and non-LISCC firms. The staff noted that the comparative analysis and rankings help the Federal Reserve to identify what distinctions exist among individual CCAR firms. The staff explained that the Executive Committee reviews evaluation teams’ findings and supporting analyses to help determine which companies have stronger and weaker practices. Because of the small number of ratings categories, the staff said that a relative comparison allows Federal Reserve staff to distinguish which firms have the best practices among those that might have the same rating. The staff said that while rankings assist the Federal Reserve in ensuring consistency across CCAR assessments, objection determinations are made based on an evaluation of a company’s practices against supervisory expectations—and not in relation to how they compare to other CCAR firms. According to Federal Reserve staff, even the lowest-ranked companies may not necessarily have weaknesses in their capital planning processes that would warrant an objection. The evaluation teams’ ratings and rankings may change after deliberations between the evaluation teams and Executive Committee staff.

Differentiated Expectations

The Federal Reserve has applied different supervisory expectations to CCAR companies based on firm characteristics. Specifically, the Federal Reserve has identified expectations for capital planning and capital adequacy processes that reflect differences in a company’s size, scope of operations, activities, and systemic importance. For example, in December 2015 the Federal Reserve published supervisory letters explaining how its supervisory expectations differ for two groups of CCAR companies—the largest and most complex firms, and large and non- complex firms.

In these and other documents, the Federal Reserve has stated that it has heightened supervisory expectations for the largest and most complex companies in all areas of capital planning and that it expects such companies to have leading practices (in terms of sophistication, comprehensiveness, and robustness) for all of their portfolios and activities compared to other CCAR companies. It has declared that smaller and less complex companies will not be held to the same standard.

CCAR documentation we reviewed indicated that the Federal Reserve has used differentiated expectations for larger and more complex companies. For example, the Federal Reserve expects complex companies to have a formal risk-identification process with quarterly updates and use quantitative approaches for risk management. In contrast, the Federal Reserve expects non-complex companies to have a less formal risk-identification process and use either qualitative or quantitative approaches for risk management. Moreover, qualitative assessment results reflected better relative rankings for non-LISCC companies than LISCC companies, which the Federal Reserve attributed in part to its higher expectations for LISCC companies. Federal Reserve staff also explained that successful capital adequacy planning is more difficult for LISCC companies simply because of their large size and complexity, which together with the heightened supervisory expectations explains much of the differences in qualitative assessment results.

According to Federal Reserve staff, similar to the Dodd-Frank Act requirement for enhanced prudential standards, the Federal Reserve has higher expectations for the largest and most complex companies because problems at such firms are more likely to have negative consequences for the financial system and economy. It has also indicated that new CCAR companies may need time to build and implement internal systems necessary to meet CCAR requirements.

Other Factors Influencing Determinations

Multiple factors influence the Federal Reserve’s final qualitative assessment determinations. According to Federal Reserve staff, evaluation teams propose recommendations based on multiple factors related to (1) weaknesses identified in the qualitative assessment and (2) the severity of the weaknesses and the likelihood that a company can remediate weaknesses before the next CCAR review cycle. In particular, Federal Reserve procedures instruct staff to evaluate the severity, materiality, quantity, pervasiveness, and duration of identified process deficiencies when considering whether supervisory findings warrant an objection determination. In addition, objections may be due to deficiencies identified in multiple areas or based on a single fundamental weakness if that weakness is in an area critical to the company’s operations or if the severity of the deficiency places the reliability of the company’s overall capital plan into question.

Furthermore, program guidance indicates that, while all capital adequacy principles are important, some principles reflecting foundational practices and deficiencies in these areas may be more likely to result in an objection recommendation than others. Federal Reserve staff said that the Federal Reserve may decide to apply a conditional non-objection for various reasons, including that a company has significant deficiencies in certain areas of their capital planning processes, but would be able to address identified deficiencies relatively quickly, depending on the nature of the deficiency. If a company has a significant weakness in a critical area that is not addressed over time, Federal Reserve staff noted that the company may receive an objection even if it has tried to address the deficiency.

Federal Reserve Has Disclosed Limited Information about Its CCAR Qualitative Methodology and Has Not Provided Timely Guidance on Leading Practices

While the Federal Reserve has communicated certain information about CCAR qualitative assessments to participating companies and the public, it has not disclosed more detailed information that would allow for an understanding of the methodology for the assessments or updated guidance to firms about leading practices.

Limited Disclosures of Information about Methodology and Objections

While the Federal Reserve has communicated some CCAR-related information to the public and directly to CCAR companies, it has not provided the level of information necessary for a clear understanding of its qualitative assessment methodology, including its framework for evaluating the extent to which companies have met supervisory expectations and determinations, and for a clear understanding of the reasons for objection determinations. Federal Reserve communications about the qualitative assessments have included information about supervisory expectations and other topics. For example, the Federal Reserve has issued public documents that include an annual CCAR instructions and guidance document, which are published at the beginning of each CCAR cycle. In 2013, the Federal Reserve also issued a separate document describing its capital planning expectations and examples of the observed range of practices among CCAR companies. In December 2015, the Federal Reserve issued supervisory letters to consolidate previously-issued capital planning expectations and to clarify differences in expectations based on firm size and complexity. In addition, at the end of each CCAR cycle, the Federal Reserve has provided CCAR companies with confidential letters describing assessment findings and information on any supervisory matters requiring attention.

The Federal Reserve also has publicly released the identity of firms that have received an objection or conditional non-objection based on the qualitative assessment and a general description of the reasons for the objection or conditional non-objection in its annual CCAR results document. For example, the Federal Reserve’s published results have described the capital planning areas in which deficiencies were found, such as risk management and internal controls.

While these documents are helpful in providing some information about the CCAR assessment and its results, they do not provide information about the assessment framework, such as the role of various evaluation teams and descriptions of ratings and rankings and other associated processes. The Federal Reserve also has not publicly disclosed information about the nature of the deficiencies in capital planning areas leading to its objection determinations, such as why a firm’s risk management or internal controls were inadequate. For instance, the Federal Reserve’s 2016 CCAR results disclosure states that the reasons for qualitative objections for one firm were based on deficiencies in the risk management and control infrastructure. This includes risk- measurement processes, stress testing processes, and data infrastructure. It also stated that these deficiencies limited the reliability of the firm’s capital planning process and its ability to conduct a comprehensive capital adequacy assessment. The document did not discuss why, or in what ways, these areas did not meet Federal Reserve expectations. For example, it did not explain what elements or characteristics of the company’s risk measurement and stress testing processes and data infrastructure were not reasonable or appropriate.

According to an Office of Management and Budget (OMB) directive on open government, transparency promotes accountability by providing the public with information about government activities. While the Federal Reserve is not required to follow OMB’s guidance, this guidance identifies a set of actions for all agencies to take to increase transparency in government operations. Similarly, our prior work has recognized that transparency—balanced with the need to maintain sensitive regulatory information—is a key feature of accountability.

Federal Reserve staff told us that they have publicly described some aspects of the qualitative assessment framework, including factors that influence objection determinations, but have not published a description of the evaluation teams and ratings because they consider such aspects to be internal processes. According to these staff, the Federal Reserve may consider providing additional information about the qualitative assessment process in the future through public CCAR documentation or other communication with companies. However, the Federal Reserve has published information about its methodology for completing the supervisory stress test and CCAR quantitative assessment, which also involves internal supervisory processes, as illustrated in the following examples.

Although it has not disclosed specific details about its models, the Federal Reserve’s DFAST results document describes the supervisory stress test methodology used to produce the post-stress capital ratios that underlie the quantitative assessment. The disclosed information includes the analytical framework for the supervisory stress test, modeling approach, model methodology and validation, data inputs, and general descriptions of specific models used to project stressed capital ratios.

Publicly-available Federal Reserve supervision manuals disclose detailed information about the policies and procedures used in examinations of banking institutions as part of the Federal Reserve’s normal, ongoing supervision.

Without disclosing additional information that would allow for a better understanding of the Federal Reserve’s methodology for completing qualitative assessments and the reasons for objection determinations, financial markets and the public may have a limited understanding of this critical aspect of the CCAR program. The limited transparency could hinder public and market confidence in the Federal Reserve’s CCAR assessments and the extent to which the Federal Reserve can be held accountable for its decisions.

Federal Reserve Guidance about Practices Not Updated Since 2014 and Companies Cited Concerns about the Communication of Expectations

The Federal Reserve has not updated guidance on current and leading capital planning practices used by CCAR companies since 2014 and companies also cited concerns about how expectations were explained. The Federal Reserve has periodically issued guidance on current and leading capital planning practices used by CCAR companies. For example, it included the information in an appendix to the CCAR instructions for the 2015 cycle. The appendix described common themes in company practices that the Federal Reserve observed during the prior year’s review. In the appendix, the Federal Reserve stated that it included the information to build on expectations outlined in previous guidance and to provide additional clarification for specific areas in which companies continued to experience challenges. However, the Federal Reserve has not provided information on the range of current capital planning practices (including those it views as leading and lagging practices) since it issued the instructions document in October 2014.

Although the Federal Reserve’s recently published supervisory letters consolidated its capital planning expectations, they did not include information on observed industry practices or the Federal Reserve’s views on what constituted leading or lagging practices. In contrast, the expectations and range of practice document that the Federal Reserve issued in 2013 identified practices among CCAR companies that the Federal Reserve considered to be stronger (leading practices) as well as those that it deemed to be weaker for capital planning purposes. The 2013 document also clarified that practices identified as leading or industry best practices should not be considered a safe harbor and stated that the Federal Reserve anticipated that leading practices would continue to evolve as new data became available, techniques advanced, economic conditions shifted, and new risks emerged.

The Federal Reserve also has communicated its expectations for capital planning directly to companies through various channels. According to Federal Reserve staff, the Federal Reserve has communicated specific expectations to companies through its other supervisory activities and through discussions with the CCAR Executive Committee during CCAR reviews. For example, the Federal Reserve has communicated with companies throughout the year about their progress toward remediating any supervisory issues identified during the CCAR qualitative assessment. Staff also said that at the conclusion of a CCAR cycle, the Federal Reserve has directly discussed CCAR findings with companies and sent feedback letters describing the results of CCAR reviews and any related supervisory findings and matters requiring company attention. Federal Reserve staff further explained that these conversations have established what actions the company would take to address the contents of the feedback letter and that additional meetings have been held to obtain updates on progress toward remediating identified weaknesses and matters requiring attention. The Federal Reserve has stated that it generally expects identified weaknesses to be remediated before the next annual capital plan submission, where appropriate, but recognized that some efforts may require additional time.

However, many CCAR companies we interviewed expressed concerns about what they viewed as limited or unclear communication of capital planning expectations for the qualitative assessment by the Federal Reserve. While several companies indicated that the Federal Reserve’s communication and guidance had improved over time, most of the companies said that a lack of clarity about supervisory expectations posed challenges for them. Specifically,

Nearly all of the companies we spoke to said that at times feedback was inconsistent across Federal Reserve teams or lacked clarity.

One company said that ambiguity around Federal Reserve expectations for the qualitative assessment has made it difficult to implement necessary changes.

Most CCAR companies we spoke to raised concerns that Federal Reserve expectations have increased annually while guidance has sometimes been unclear, insufficient, or outdated. Several of these companies suggested that the qualitative assessment process could benefit from increased transparency and more granular and timely guidance.

Several companies we spoke to noted that Federal Reserve guidance on industry leading and lagging practices that had been provided—in particular, the 2013 document highlighting the Federal Reserve’s views on the range of company practices—has been helpful in understanding supervisory expectations.

Federal internal control standards state the importance of relevant, reliable, and timely communications, including with external stakeholders. Federal Reserve guidance also indicates that the Federal Reserve expects the largest and most complex CCAR firms to use leading capital planning practices—those that are the most sophisticated, comprehensive, and robust—and that these leading practices are expected to evolve over time.

According to Federal Reserve staff, the Federal Reserve has not decided whether it will issue additional guidance on company practices or supervisory expectations beyond the recently published supervisory letters. However, Federal Reserve staff stated that the Federal Reserve does not intend to update its 2013 guidance on the observed range of CCAR companies’ capital planning practices or publish another “common themes” appendix or similar guidance documents because its recently issued supervisory letters include consolidated guidance on supervisory expectations relating to firms’ capital planning processes. The documents also have technical appendixes containing specific expectations but do not include information on leading practices. The staff said that the Federal Reserve intends to use these documents as the primary mechanism for clarifying expectations. Federal Reserve staff also told us that communicating company-specific expectations occurs through direct communication with CCAR companies, including confidential feedback letters. Yet, without periodically updated guidance on observed capital planning practices and those that the Federal Reserve considers to be leading ones—especially as industry practices evolve—some CCAR companies may have difficulty determining what is necessary to meet Federal Reserve expectations, which could impede the achievement of CCAR goals.

Federal Reserve Has an Official CCAR Communications Channel but Has Not Specified Response Times for Questions from Companies

The Federal Reserve has designated an official communications channel for CCAR companies to ask questions related to CCAR, but has not communicated time frames for responding to questions. The Federal Reserve has designated a general e-mail address—referred to as the CCAR Communications Mailbox—to field all questions from CCAR companies and provide all responses and other official communications on behalf of the Federal Reserve. The Federal Reserve has instructed firms to submit CCAR-related questions to the mailbox and stated that only responses received through the mailbox will be considered official Federal Reserve responses, although meetings and other discussions with Federal Reserve staff may be arranged. According to Federal Reserve procedures, staff identify questions as being either broadly relevant to all CCAR companies or company-specific. The Federal Reserve distributes broadly relevant questions and responses to all CCAR firms through a frequently-asked-questions (FAQ) report, while the procedures call for the company-specific questions to receive a direct response.

The Federal Reserve has established an internal target for response times but it has not communicated this or other time frames to CCAR companies. Federal Reserve procedures for the CCAR mailbox identify an internal goal of providing companies with a response to submitted questions within 7 business days of receipt. Federal Reserve staff indicated that the time frame target is primarily intended to help process the questions internally and prepare responses. However, according to Federal Reserve procedures, communication with companies submitting questions consists only of acknowledging receipt of questions and providing a direct response after the completion of the internal review and response process.

However, most companies we interviewed identified limitations in receiving timely or helpful responses from the CCAR mailbox. For example, some companies said that the Federal Reserve’s mailbox responses tended to be general and standardized to apply to all companies rather than tailored to a company’s specific circumstances. In addition, two companies said that answers to some questions simply reiterated the stress test and capital plan rule, which was of limited usefulness. Several companies also explained that it could take multiple weeks or even months to receive responses to mailbox questions, which represented a considerable amount of time in the context of CCAR time frames. Officials from two companies noted that while 2 to 3 weeks may not seem like an excessive amount of time, the delays prevent companies from addressing the capital planning topics covered by the question as they await guidance from the Federal Reserve. The officials also stated that they understand that it takes time to research and review answers before they are sent to companies and that the Federal Reserve wants to provide considered and consistent responses. Several companies also identified improvements in communication with the Federal Reserve’s on- site teams, including their responsiveness to company questions.

Federal internal control standards state the importance of relevant, reliable, and timely communications including with external stakeholders. The Federal Reserve has stated that it designed the CCAR mailbox and FAQ process to help ensure that CCAR companies received timely and consistent responses to all submitted questions. Federal Reserve staff also explained that many questions require further research and internal deliberation before a response can be provided, which makes it difficult to commit to a specific response time frame. For example, according to Federal Reserve procedures, questions and responses that set new guidance or involve broad policy implications may require additional review including discussions with the CCAR oversight committees. Internal process documentation calls for Federal Reserve staff with subject-matter expertise in areas such as capital policy, balance sheet items, or model risk management to draft responses to questions while other subject-matter experts, a legal reviewer, and management review and approve the draft responses.

However, failure to establish and communicate time frames for responding to company inquiries may complicate companies’ management and planning of their CCAR submissions and hinder their ability to address supervisory concerns in a timely fashion. For example, due to CCAR deadlines, a company awaiting a response to a question the Federal Reserve has deemed to involve broad policy implications may have to proceed with developing its CCAR submission without receiving the guidance it needs from the Federal Reserve.

Supervisory Scenario Design Process Integrates Historical Data and Current Risks but Some Limitations Exist

Federal Reserve staff design supervisory scenarios for the supervisory stress tests in DFAST and CCAR by integrating data from historical recessions and the recent financial crisis with an assessment of current risks to financial stability. But limitations exist with some aspects of the scenario design, including consideration of trade-offs related to the choice of severity and assessment of the sufficiency of a single severe supervisory scenario.

Scenario Design Process Integrates Historical Data and Current Risks

Federal Reserve staff design supervisory scenarios for the supervisory stress tests in DFAST and CCAR by integrating data from historical recessions and the recent financial crisis with an assessment of current risks to financial stability. Federal Reserve staff design the scenarios according to Federal Reserve policy outlined in a November 2013 policy statement.

As previously discussed, the Federal Reserve annually creates multiple supervisory scenarios for DFAST and CCAR:

Baseline scenario. Generally reflects economic conditions expected by economic forecasters.

Adverse scenario. Features mild to moderate economic and financial stress driven by selected potential risk factors.

Severely adverse scenario. Features severe economic and financial stress, generally driven by a different set of risk factors than the adverse scenario.

Global market shock and counterparty default components (applicable to companies with large trading or custodial operations). These two components, applicable to a subset of companies, are designed to stress the trading and private equity (in the case of the global market shock), or counterparty positions (in the case of the counterparty default component) of bank holding companies with significant trading activities. These components are supplemental to both the adverse and severely adverse scenarios.

Overlapping teams of staff from across the Federal Reserve System simultaneously develop the macroeconomic scenarios (baseline, adverse, and severely adverse) and the market shock component. The macroeconomic scenarios feature stress events that evolve over 13 quarters while the global market shock and counterparty default components take place at a single point in time.

At the start of the process, the macroeconomic scenario design group meets to discuss salient risks that might be included in the scenarios, drawing on the Federal Reserve’s quarterly assessments of risks to financial stability and input from FDIC and OCC staff.

Macroeconomic modelers at the Federal Reserve translate these identified risks (such as a decline in housing prices) into projections for each of the 28 economic and financial variables included in the scenarios. The projections (over 13 quarters) represent the quantitative output of the scenario design process. The economic and financial variables include measures of the unemployment rate, gross domestic product, housing and equity prices, interest rates, and financial market volatility.

A separate scenario design group develops the global market shock component, which results in a set of instantaneous shocks to a broad range of financial market risk factors. These shocks involve large and sudden changes in asset prices, interest rates, and measurements of market risk. Price changes in the market shock scenario generally have been comparable to financial market developments in the second half of 2008 (the height of the financial crisis) and also featured larger declines when market valuations have been more elevated or to emphasize salient risks identified by the Federal Reserve.

Federal Reserve staff present the proposed scenarios to stress test oversight groups and division directors in the Federal Reserve and to officials from FDIC and OCC.

After considering feedback, the scenario design group provides options for final scenarios to the Board of Governors chair, vice chair, and other governors involved with bank supervision for their input and preferences.

Finally, the scenario design group completes the scenarios based on the principals’ choices, including adding a narrative description of the key factors driving the scenario, and releases the final scenarios on the Federal Reserve website.

Limitations Exist with Some Aspects of the Scenario Design Process

While the Federal Reserve has implemented a framework for designing supervisory scenarios, some aspects of the process have limitations, in particular regarding the choice of severity and the sensitivity of results to alternative severe scenarios.

Federal Reserve Has Not Considered Severity outside Postwar U.S. History

The Federal Reserve largely has relied on historical experience to establish the severity of the severely adverse scenario, operationalized primarily through the unemployment rate. More severe scenarios (such as those with higher unemployment rates) generally would have an adverse impact on loans and other assets, increasing losses, reducing income and profitability, and hence reducing the projected post-stress capital ratios for participating companies. IMF principles for supervisory stress testing suggest that supervisors should focus on tail risks—those risks associated with very unlikely but extreme events, including events that have not occurred in the past—and highlight the risks of basing scenario design decisions solely on historical experience.

The Federal Reserve’s decisions about the severity of its scenarios have been driven primarily by U.S. postwar historical experience. In designing stress test scenarios, the Federal Reserve primarily has used the change in and level of the unemployment rate to determine the severity of the scenario (with higher unemployment rates associated with more severe scenarios). According to Federal Reserve policy, the unemployment rate should rise by 3–5 percentage points and must reach a minimum of 10 percent in the severely adverse scenario. In practice, the Federal Reserve has increased the unemployment rate in the severely adverse scenario by 4 percentage points in each year from 2013 to 2015 and by 5 percentage points to reach the 10 percent minimum in the 2016 scenario. Federal Reserve staff stated that a 3–5 percentage point increase and a 10 percent unemployment rate minimum were consistent with postwar historical recessions and provided a reasonable basis for determining the overall severity of the severely adverse scenario. In discussing the 2015 severely adverse scenario, which had a peak unemployment rate of 10 percent, staff indicated that a higher or lower unemployment rate—for example, 9 percent or 12 percent—would be difficult to justify based on historical precedent. In particular, staff noted that a 12 percent unemployment rate has never been seen in postwar U.S. history. Also, it would have required an unprecedented 7 percentage point increase in the unemployment rate. Similarly, staff noted that a 9 percent unemployment rate would be relatively low compared to severe U.S. recessions. In other aspects of scenario design, in contrast, Federal Reserve policy is cognizant of the possibility that scenarios may produce risks that appear together in ways that could be outside of historical experience—and Federal Reserve Staff noted that supervisory scenarios have featured combinations of risks that have not occurred historically. Federal Reserve staff also noted that the cumulative severity associated with multiple, simultaneously stressed scenario variables could exceed historical postwar severity. However, the Federal Reserve’s scenario design policy and process are focused on selecting economic conditions that reflect the severity of postwar U.S. recessions. Without proactively considering levels of severity outside postwar U.S. historical experience, the Federal Reserve could miss opportunities to assess and guard against relevant but unprecedented risks to the banking system. For example, if scenario severity decisions had been made in the pre-crisis period based solely on historical conditions that had prevailed prior to 2006, any associated stress tests would have dramatically underestimated subsequent events.

Federal Reserve Has Not Assessed Certain Trade-offs Associated with Scenario Severity

According to Federal Reserve staff and our review of internal documents, the Federal Reserve has not explicitly analyzed how to balance the choice of the severity of the severely adverse scenario—and its influence on the resiliency of the banking system—with any impact on the cost and availability of credit. The overall severity of a stress scenario will affect how much capital that participating bank holding companies would need to hold to avoid an objection from the CCAR quantitative assessment and make planned capital distributions. Furthermore, a more severe scenario might induce companies to raise additional capital in the short term—and potentially pass costs on to borrowers—but increase the resiliency of the banking system over the long term.

OMB guidance states that regulatory analysis—a tool regulatory agencies use to anticipate and evaluate the likely consequences of rules—should be based on the best available scientific, technical, and economic information. While the Federal Reserve is not legally required to follow this guidance, it provides a strong set of analytical practices relevant to significant supervisory and regulatory exercises such as CCAR—and scenario design is a key element of CCAR. Research by the Basel Committee on Banking Supervision (Basel Committee)—an international standard-setting organization for bank regulation—on the potential economic impact of capital requirements provides an example of how the Federal Reserve could use available information to help assess the appropriate degree of severity in stress tests. In its assessments of post-crisis reforms to strengthen bank regulation, the Basel Committee assessed how changes to the level of required capital and liquidity would influence economic growth, the cost and availability of credit, and the likelihood and severity of future financial crises.

Federal Reserve staff noted that they were aware of a significant amount of academic literature on the relationship between bank capital and the economy, including the work of the Basel Committee. Moreover, Federal Reserve staff said that the scenarios were designed to match the severity of historical recessions, which assesses the resilience of the banking system and would allow companies to function through a severe recession. However, without more careful assessment of scenario severity, the Federal Reserve cannot be reasonably assured that the scenario design process balances any improvements in the resiliency of the banking system with any impact on the cost and availability of credit. These factors could be especially important when considering a level of severity without a postwar historical precedent—for example, by helping ensure that scenarios that might exceed postwar historical severity do not result in undesired increases in the cost and availability of credit.

Federal Reserve Has Not Assessed Sufficiency of a Single Severe Supervisory Scenario

The Federal Reserve has not conducted analysis to determine if a single severe supervisory scenario (that is, the severely adverse scenario) is sufficiently robust and reliable to promote the resilience of the banking system against a range of potential crises. The Federal Reserve’s policy statement on scenario design suggests that at times the stress tests may require additional supervisory scenarios to capture a large number of unrelated but significant risks. The CCAR quantitative assessment is based on more than one severe scenario, but the Federal Reserve designs only one of them. Participating institutions design an additional severe scenario, which is intended to reflect particular risks that might differ from the supervisory scenario in substantive ways. The company- designed scenario is implemented with a different capital action assumption, which limits its comparability to other CCAR stress tests. We discussed potential incentive problems associated with company-run CCAR stress tests earlier in the report.

There are advantages and disadvantages associated with reliance on a single severe supervisory scenario, as the Federal Reserve does with the supervisory stress tests. Advantages could include simplicity, transparency, and resource use—that is, using a single severe supervisory scenario simplifies communication and limits the resources required to design the scenarios and execute and analyze the supervisory stress tests. While it may be appropriate to use a single severe supervisory scenario, there are also potential disadvantages or risks associated with making the CCAR quantitative assessment based on a single severe supervisory scenario. For example, many different types of financial crises are possible, and the single selected scenario does not reflect a fuller range of possible outcomes. Similarly, IMF principles for supervisory stress tests note that future stress periods are uncertain and could be represented by a range of stress factors, each with a different likelihood of occurrence. Staff at IMF and Bank for International Settlements with whom we spoke also identified trade-offs associated with using one or multiple scenarios. For example, Bank for International Settlements staff noted that firms might hedge against the primary risks in a single scenario but not others that also might be relevant. Similarly, IMF staff said that ideally stress tests would use a large and diverse set of scenarios but also noted that this would increase cost and complexity.

Moreover, it is usually necessary to conduct sensitivity analysis to reveal whether, and to what extent, the results of an analysis are sensitive to plausible changes in the main assumptions. For the supervisory stress tests and CCAR quantitative assessment, the design and number of severely adverse scenarios represent key assumptions affecting results. However, the Federal Reserve has not conducted sensitivity analysis to determine whether its single severe scenario is sufficient to accomplish DFAST and CCAR goals. For example, it has not assessed how a range of severe scenarios may lead to different judgments about the overall resiliency of the banking system.

Because it has matched the severity of historical U.S. recessions, Federal Reserve officials asserted that the severely adverse scenario would protect against a range of potential crises. Federal Reserve officials also noted that they perform multiple stress tests using alternative scenarios outside of DFAST and CCAR, and also conduct a separate stress test of liquidity. Yet, without assessing the sufficiency of a single severe scenario in the context of DFAST and CCAR—such as performing sensitivity analysis involving multiple scenarios—the Federal Reserve is making CCAR decisions that may not reflect the range of potential crises against which the banking system would be resilient and the magnitude of the range of outcomes that might result from different scenarios. The Federal Reserve therefore may be limited in its ability to understand, communicate, and manage uncertainty associated with its use of the supervisory stress test results.

Federal Reserve Has Not Prospectively Assessed Whether Changes in Scenarios Might Inadvertently Amplify Economic Cycles

The Federal Reserve has not assessed whether or how the year-to-year changes it makes to the supervisory scenarios over an economic cycle could inadvertently amplify those cycles until after it has completed and published the scenarios. Because the supervisory scenarios used in the stress tests influence a company’s post-stress capital ratios, changes to the scenarios will affect how much capital participating companies need to hold to help ensure they do not receive a CCAR objection. Federal Reserve policy states that supervisory scenarios should feature stressful outcomes that do not induce greater procyclicality—that is, scenarios should not amplify cycles (swings in economic activity between expansion and contraction) in the financial system and economy. Procyclicality could, for example, result in firms needing to raise additional capital or reduce lending during a downturn.

After the disclosure of 2015 stress test results, Federal Reserve staff reported to the Board that the supervisory stress tests produced some procyclical results. Specifically, losses on portfolios of loans to consumers (such as credit cards or residential mortgages) in the test were procyclical partly because scenarios had caused the projected losses to shrink as actual economic conditions improved. Based on analysis conducted after the scenarios were finalized, Federal Reserve staff said that most of the decrease in projected losses (compared with the prior year’s stress test) resulted from improvements in bank balance sheets from the previous year, although changes to the scenario also contributed to lower projected losses. Because the scenario contributed to smaller losses as the economy improved, the scenario could produce larger losses as the economy deteriorates, lowering post-stress capital ratios and increasing the amount of capital required to avoid a CCAR objection. Moreover, because the analysis of the impact of annual scenario changes on losses occurs only after scenarios have been developed and made public, the Federal Reserve might learn of procyclical effects too late to take effective preventive steps—for example by adjusting relevant scenario variables before scenarios are made final. As a result, Federal Reserve stress tests could exacerbate future financial stress by increasing requirements on stress test participants while economic conditions are deteriorating.

Such an unintended impact of the supervisory scenario on losses could emerge because the complexity of the system of models (discussed in the following section) used by the Federal Reserve makes it difficult to anticipate the combined effects of changes to 28 scenario variables that influence the results of multiple supervisory stress test models. Federal Reserve staff stated that the Federal Reserve’s scenario design policy attempted to avert procyclicality by instituting the 10 percent unemployment rate minimum, to prevent the unemployment rate from falling too much in the scenario when the real economy improved and otherwise allowing the unemployment rate to increase from 3 to 5 percentage points—increasing the unemployment rate more in the scenario when actual unemployment is low, and raising it less when actual unemployment is high. However, the complexity of the system of models the Federal Reserve has used in the supervisory stress test implies that without additional, supporting analysis the Federal Reserve cannot be reasonably assured that small adjustments to the unemployment rate and other variables would produce outcomes that neither amplify nor dampen economic cycles.

Federal Reserve Management of Model Risk Has Not Focused on the System of Models

The Federal Reserve’s modeling process for the stress tests includes an oversight structure and internal reviews, but it has not focused its model risk management on the system of models that produce stress test results. To estimate the effect of stress test scenarios on companies’ ability to maintain capital, the Federal Reserve has developed individual component models that predict companies’ financial performance in the scenarios. The results of these component models are combined with assumed or planned capital actions of companies and form the system of models used by the Federal Reserve. The Federal Reserve has issued model risk-management guidance that defines model risk as the potential for adverse consequences from decisions based on incorrect or misused model outputs and states that it increases with factors such as greater model complexity and larger potential impact. However, the Federal Reserve has not focused its model risk-management efforts on the system of models, including not conducting sensitivity and uncertainty analyses of how its modeling choices affected the model risk associated with the overall stress test results (post-stress capital ratios).

Federal Reserve Has a Process for Development and Oversight of Models Used for Supervisory Stress Tests

The Federal Reserve has a development process for the models it uses to predict each institution’s post-stress capital ratios and has an oversight structure for the process.

Overview of Supervisory Stress Test Models

To estimate the effect of supervisory scenarios on companies’ regulatory capital ratios, the Federal Reserve has developed numerous empirical models that each predict a component of a company’s balance sheet, risk-weighted assets, or income statement (component models) for each of the 9 quarters of the stress test planning horizon. The Federal Reserve then combines the results of the component models with assumed or planned capital actions (for the companies) to produce the post-stress capital ratios. We refer to the combination of the component models that produces the post-stress capital ratios as the system of models.

The component models are either predictive or accounting models. The predictive models use historical data to estimate how economic stress events might affect an element of an institution’s financial performance, such as loan losses or revenues. Accounting models apply various accounting rules to an institution’s financial data or to outputs from the predictive models to construct aggregate accounting measures, such as allowances for loan and lease losses or pre-tax net income. Most component models use estimates produced by other component models as a source of data to make their projections.

Overview of Modeling Cycle

The Federal Reserve has implemented a 2-year development cycle for supervisory stress test models (see fig. 6). According to Federal Reserve staff and documentation, the overall goal of the development cycle is to continue refining and developing the models, while simultaneously producing reliable estimates for the annual DFAST and CCAR supervisory exercises. The Federal Reserve’s 2-year model development cycle (which is described in more detail in the following sections) involves the use of production and development models. Production models are used to produce annual estimates for the DFAST and CCAR exercises. Development models allow for more time to validate and evaluate major model changes before they are incorporated into the actual stress test exercises. Once a development model has been fully reviewed and approved, it replaces the corresponding production model.

The Federal Reserve annually refines and continues development of its models. Modeling approaches and variables can change over time. The Federal Reserve has stated that revisions to its models generally have reflected advances in modeling techniques, more detailed data, and longer histories of performance in different economic environments. In addition, changes in market or industry risk characteristics and regulatory or policy changes also may be a reason to make changes to existing models.

Responsibilities and Oversight by Process Phase

The Federal Reserve has implemented an organizational structure focused on model development, oversight, and review (see fig. 6). For example, according to Federal Reserve staff and program documents, the Model Oversight Group—a cross-functional group of senior Federal Reserve managers and stress test experts—coordinates model development policy and has overall responsibility for the model development process. According to Federal Reserve staff, the Model Oversight Group exercises close oversight of the planning and execution of the supervisory stress tests including changes to stress test models and the Director of Banking Supervision and Regulation has final authority to approve models prior to stress test production. The Model Coordination and Advisory Team assists the oversight group by serving as the first line of oversight over the modeling teams. According to its charter, the coordination team is tasked with initial reviews of model documentation, model changes, and model assessment presentations. Economists, senior quantitative analysts, and other technical specialists from across the Federal Reserve System staff the coordination team.

Development. The responsibility for executing model development lies with 11 supervisory modeling teams staffed with subject-matter experts. Each of the teams are responsible for developing models to predict elements of a company’s balance sheet or income statement, which are ultimately combined to predict post-stress capital ratios. For example, one modeling team is responsible for 22 separate models that predict components of net revenue before adjustments for loan loss provisions. Another is responsible for models that combine output from the loss, revenue, and other models to produce the regulatory capital ratios. The Model Oversight Group oversees the development process with assistance from the Model Coordination and Advisory Team. The Model Oversight Group also reviews and determines whether to approve development models to replace prior production models.

Preliminary assessment. According to Federal Reserve documentation, supervisory modeling teams then conduct a preliminary assessment of the model production process after the model development phase is complete. The main purpose of the assessment is to test that all models and processes perform as expected and remediate problems before the final production of the annual stress test results. The modeling teams use loan portfolio and other financial data submitted to the Federal Reserve by institutions subject to the supervisory stress test as inputs to their models. They apply the current portfolio data and scenarios to the production models—including the development models awaiting approval to become production models—to test the models and provide information to the Model Oversight Group about the effects on component model outcomes. The modeling teams analyze any differences from the previous year’s results, which involves estimating how much of the overall change in a component model’s result can be attributed to model changes, scenario changes, company portfolio changes, or other factors. Teams then present the results to the Model Oversight Group and identify any production or modeling problems, including any problems with the quality of company-provided data. The modeling teams revise models or data inputs as necessary in response to problems identified by the Model Oversight Group or the Model Coordination and Advisory Team during the preliminary assessment presentations and finalize their model documentation. Senior Federal Reserve staff approve or reject major modeling decisions made by the Model Oversight Group.

Validation. In the model validation phase, reviewers in the Model Validation Unit are to examine the models to identify potential problems, ranking them by level of concern and threat to model validity. According to Federal Reserve policy, staff undertaking validation activities are independent of the model development process for the models they review. Before 2016, validation reviewers volunteered to leave their primary duties in the Federal Reserve System to assist with the stress test model review process. For example, for the CCAR 2015 exercise, they spent approximately 8 weeks reviewing the models and returned for an additional 3 weeks to validate model changes made in response to the most severe findings. According to Federal Reserve staff and program documents, the Federal Reserve has been transitioning to a validation program that consists primarily of permanent, full-time staff.

For each modeling team, the Model Validation Unit employs various staff that are to review (1) model soundness and performance and (2) model change and implementation controls. Economists and other subject- matter experts from across the Federal Reserve are to evaluate the model design for conceptual soundness and performance (model soundness and performance review). Federal Reserve internal control experts are to evaluate the control processes surrounding model development and implementation (model change and implementation control review). Each group of reviewers in the validation unit is to write a report that summarizes any problems they identified. The reviewers are to rate problems according to their assessment of the problems’ severity, materiality, and risk posed to the reliability of the model.

Model finalization. To complete the models, the Federal Reserve has a policy that defines how the Model Oversight Group oversees the supervisory modeling teams’ implementation of responses to the findings of the Model Validation Unit. The oversight group is to review and approve model changes and the validation unit is to validate them. According to Federal Reserve staff, modeling teams generally will address at least those problems identified by validation unit reviewers that pose the highest risk to the validity of the model.

With the approval of the Director of the Division of Banking Supervision and Regulation, the oversight group may defer some changes—even those in response to problems rated in the most-risky category—either because (1) the changes require structural modifications to models that the group views as better implemented through the 2-year development cycle, (2) lack of available data, or (3) other priorities taking precedence. For example, in the 2015 stress test cycle, 24 high-risk findings of the validation unit were unaddressed or had associated model changes that had not yet been validated by the time the models went into production. Validation unit documentation also indicated 71 repeat findings (at a lower-risk level) that modeling teams had not addressed for at least a year. Even if a modeling team is able to address a finding, stress test model changes may or may not be validated by the Model Validation Unit during the same cycle in which modeling teams made the change. For example, in the past some model changes occurring at the end of the annual modeling process have not been reviewed by the Validation unit until the following cycle. According to Federal Reserve staff, the new, permanent staff approach for the Model Validation Unit is designed in part to increase the unit’s involvement in validating last-minute model changes. As described later and in appendix II, in a 2015 report the Federal Reserve Office of Inspector General (OIG) made a recommendation related to late-stage model changes, among other things.

Final review and approval and production of results. Supervisory modeling teams are then to conduct a second run of the production models to generate results for the final model assessment presentations. The second run is to use the final company data submissions as well as the final versions of the models and stress scenarios developed by the scenario design group. Each modeling team is to present its results to a group that consists of the Model Oversight Group, co-leaders of the Model Coordination and Advisory Team, and the deputy director of the Large Institution Supervision Coordinating Committee (LISCC) for final review and approval. The modeling teams may make certain model adjustments after the presentation in response to feedback. After addressing any concerns, the modeling teams are to calculate final results for all of the models. Federal Reserve staff then are to brief the Board of Governors on the results.

Disclosure of Model Methodology

The Federal Reserve has disclosed some information about the models underlying the supervisory stress test, including in an appendix to its annual publication of DFAST results. In these documents, the Federal Reserve has described in broad terms its analytical framework for the supervisory stress test as well as its modeling approach and some features of specific models. However, the Federal Reserve has not disclosed other information about the models it uses to execute the supervisory stress tests. For example, it has not disclosed a level of information about the models that would easily allow an external party to replicate the results of the supervisory stress tests.

Officials from several CCAR companies we interviewed said that limited transparency about the Federal Reserve’s models impaired their firm’s capital planning efforts. For example, the company officials explained that without more detailed information on the Federal Reserve’s model specifications, they were unable to understand the factors behind the supervisory stress test outcomes or reconcile them with the results of their own company-run tests. Several companies’ officials said that this prevented them from identifying the cause of poor stress test results and taking appropriate actions in response. These officials said that this limited transparency could result in companies holding additional capital as a precaution to better ensure that they do not receive an objection from the CCAR quantitative assessment.

However, the limited disclosure by the Federal Reserve reflects its concern about the potential for model information to influence the actions of covered companies in ways that undermine the purpose of the stress test exercises, among other potential adverse consequences. Federal Reserve staff said that more detailed disclosure of the underlying models would make it easier for companies to manage their capital and asset decisions in relation to the supervisory stress test (in other words, “game” the models) without necessarily limiting risk, thus resulting in the potential for a form of regulatory arbitrage (firms’ circumvention of regulation). In addition, Federal Reserve staff have noted that fuller disclosure could reduce the diversity of models used by companies—a problem termed model monoculture—with companies using models that imitated the Federal Reserve’s rather than developing internal models that best reflected their own risks. As Federal Reserve staff explained, companies need to develop models that they believe are best suited for their unique business activities and portfolios. Some company officials with whom we spoke acknowledged these trade-offs and said that they understood the Federal Reserve’s decision to limit disclosure of model details in light of such considerations. In our prior work on international standards for regulatory capital requirements—a regulatory setting analogous to the supervisory stress test—we also have noted that banks can arbitrage certain capital requirements by managing their portfolios specifically to reduce required capital levels without reducing risk.

Federal Reserve Reviews of Model Risk Management

While the current supervisory stress test modeling process has an oversight and review system in place as described above, both the Model Validation Unit and the Federal Reserve Office of Inspector General (OIG) have conducted reviews of the process and identified areas that would benefit from improvement. According to the Federal Reserve, it is committed to continuous assessment and enhancement of the supervisory models used in the stress testing program. As part of this commitment, the validation unit has completed multiple internal assessments, including an evaluation of the Federal Reserve’s governance of its model risk management activities for supervisory stress testing completed in December 2014. As described in the OIG report, the review by the Model Validation Unit determined that certain governance practices did not fully conform to the Federal Reserve’s supervisory guidance for banking institutions on model risk-management practices and exhibited fundamental weaknesses in key areas.

The OIG review was issued in October 2015 and examined the Federal Reserve’s model risk-management practices for supervisory stress tests, with a particular focus on the model validation process. The OIG report found that reviews by the Model Validation Unit to assess the Federal Reserve’s validation and governance activities had identified opportunities for improvement, but that additional actions could further enhance model risk-management practices. Both reviews used the Federal Reserve’s supervisory guidance for banking institutions on model risk-management practices as the primary criteria for evaluating the Federal Reserve’s own processes. See appendix II for more information on these reviews.

Federal Reserve Does Not Address Cumulative Risk and Uncertainty from the System of Models That Produce Stress Test Results

The Federal Reserve has not focused its risk-management efforts (including those relating to model development guidance, documentation, sensitivity and uncertainty analyses, and risk tolerances) on the system of models that produce the stress test results—the post-stress capital ratios. As mentioned previously, the Federal Reserve’s model risk-management guidance defines model risk as the potential for adverse consequences from decisions based on incorrect or misused model outputs, which increases with factors such as greater model complexity and larger potential impact. The guidance states that organizations should manage model risk both from individual models and in the aggregate and establishes a definition of models that encompasses both component models and a system of models. The guidance also notes that aggregate model risk is affected by interaction and dependencies among models; reliance on common assumptions, data, or methodologies; and any other factors that could adversely affect several models and their outputs at the same time. However, the Federal Reserve’s organizational structure for the stress tests does not include a formal process through which model development or risk management at the aggregate—or system-of-models—level is implemented.

The connections and relationships between the individual component models combine to create a system of models that produces the post- stress capital ratios. Figure 7 provides a high-level overview of the interactions among the component models used in the supervisory stress tests. The Federal Reserve uses component models to project a company’s balance sheet assets—the “Asset Balances Models” and “Trading Assets Models” illustrated in the figure—and risk-weighted assets. It then calculates changes in the company’s net income using separate component models to project different parts of an institution’s revenue, expenses, and losses, as well as changes in its loan loss allowance. Next, other component models project changes in equity and regulatory capital by combining projected net income and capital actions. The final step estimates post-stress capital ratios by joining equity and regulatory capital projections with total assets and risk-weighted assets projections.

Based on our review of Federal Reserve documentation and interviews with staff, the Federal Reserve has not assessed its entire system of models in relation to the principles that it has applied to individual component models. The Model Oversight Group has developed a set of modeling principles to assist in managing the design of the Federal Reserve’s supervisory stress tests, including managing its risks. According to its procedures, the oversight group intends for these principles to guide design and other decision-making in the model development process.

The level at which the Federal Reserve applies the model development principles is important because it is the combined system of models rather than any individual model that generates the relevant stress test results—the post-stress capital ratios. As with all models, the Federal Reserve’s models used to produce supervisory stress test results reflect some amount of uncertainty and are sensitive to the assumptions and modeling decisions made when developing each model. For example, model developers must make assumptions about how model inputs interact, which inputs are relevant, and how or if historical data are relevant to the outcome that the model seeks to predict. Each of those decisions can affect a component model’s results. As such, each component model contributes elements of uncertainty to the overall result of the system of models—the final post-stress capital ratios in the case of the supervisory stress tests.

The extent and nature of any interaction among component models in the supervisory stress tests also will introduce risk to the post-stress capital ratio estimates. In addition, the overall effects on post-stress capital ratios of choices about how component models interact may be unclear. For example, poor decisions about component model interactions could result in post-stress capital ratios that are insufficiently responsive to economic changes included in a stress scenario or that respond to economic stress very differently from the way they would in reality.

In the Federal Reserve model documentation we reviewed, supervisory modeling teams applied model design principles at the component model level. For example, the modeling teams justified various component modeling choices using the principles. When selecting from two versions of a component model, both of which appeared to perform well, a team cited the principle of simplicity in selecting the version with fewer variables.

Although Model Oversight Group reviews and oversight are applied across the system of models, the documentation we reviewed did not discuss key aspects of the interactions between the component models. For example, it did not consider how the component models are combined into the system of models and how and to what extent those choices may introduce statistical uncertainty to the post-stress capital ratios. Federal Reserve staff indicated that model development decisions are closely overseen and approved by the oversight group. The staff explained that the close oversight provided by the group provides an adequate assessment of the effect of component model design decisions on the post-stress capital ratios produced by the system of models. They noted that the application of the principles at the component level combined with the role of the oversight group means that the principles are applied at the system-of-models level. As an example of how that oversight operates, the Federal Reserve staff provided documentation of what they described as the consideration of a cross-model issue. However, the documentation did not discuss the potential effects of design decisions at the component level on post-stress capital ratios, including any effects from the interaction of component models. In addition, the Federal Reserve documentation we reviewed about model development and implementation and oversight by the Model Oversight Group did not support that the Federal Reserve has conducted systems- level analyses of the effect of modeling decisions on the post-stress capital ratios.

By largely focusing the modeling principles on the component models and not applying those principles to the system of models, the Federal Reserve has limited its ability to manage the extent to which model risk is introduced into the supervisory stress test models.

Lack of Appropriate Documentation for System of Models

The Federal Reserve has not developed appropriate documentation of the system of models that would allow for effective management of the risks posed by component model interactions. Appropriate model documentation is necessary for assessing, managing, and communicating model risk, and the Federal Reserve’s supervisory guidance identifies appropriate documentation as one of the elements of a sound model risk-management process.

According to Federal Reserve procedures, each supervisory modeling team is responsible for maintaining its own model documentation and each review team in the Model Validation Unit is in charge of documenting its findings related to model limitations and other areas. The procedures call for documentation to follow guidance from the Model Oversight Group and the Model Validation Unit in both format and content. The procedures and guidance indicate that appropriate documentation for component models include descriptions of model design, data, and methodology; a quality control plan; and review reports.

The model description documentation is expected to give third parties the ability to understand the model, evaluate it, monitor its development, and replicate it as necessary.

The quality control plan provides documentation of modeling team processes to help ensure that the models have been implemented as intended by the design specification and to mitigate the potential for model error or misapplication. The plan specifies roles and procedures for checking or processing model data, documenting and approving changes, testing the model, and other control activities.

The review reports record the independent assessment of modeling teams’ models. The reports document the areas the Model Validation Unit reviewed, evaluation of the models, issues uncovered and an assessment of the risk they pose to the reliability and performance of the model, and an assessment of the modeling team responses to previous year’s findings.

The Federal Reserve procedures and guidance as well as best practices suggest that similar documentation would be appropriate for the system of models.

However, the Federal Reserve does not have a similar set of documentation for the system of models. Instead, Federal Reserve staff stated that they have used a single document to serve this purpose. The document has recorded Model Oversight Group decisions including some that apply to multiple models—which are controlled by more than one modeling team—over the course of a stress test cycle. Yet, the Model Oversight Group decisions document that we reviewed did not include descriptions of model design, data, and methodology sufficient to give third parties the ability to understand the system of models, evaluate it, or replicate it as necessary. It also lacked information that might constitute a quality control plan or a model validation review report.

The lack of appropriate documentation of the system of models limits the Federal Reserve’s ability to effectively identify and manage model risk from the entire system of models used for the supervisory stress tests. For example, without such documentation, Federal Reserve staff may miss important connections between elements of component models, which in turn may limit understanding of risks inherent in their modeling choices. The absence of system-level documentation also impedes the ability of independent parties, both internally and externally, to review, understand, or evaluate the system of models. Additionally, it increases risks associated with staff turnover (which could cause the Federal Reserve to lose important knowledge about the design and functioning of the system of models).

Sensitivity and Uncertainty Analyses Have Not Considered Effects on Post-Stress Capital Ratios

Based on our review of Federal Reserve documentation and interviews with staff, the Federal Reserve has not conducted sensitivity and uncertainty analyses of how its modeling affects the post-stress capital ratios produced by the entire system of models. As previously described, all models reflect some amount of uncertainty and are sensitive to the assumptions and modeling decisions made during model development. According to the Federal Reserve’s model risk-management guidance, an integral element of model development is evaluating model features and overall functioning to determine whether the model is performing as intended, including by demonstrating that a model is accurate, robust, and stable and by assessing potential limitations. The guidance states that such an evaluation should include assessing the model’s behavior over a range of input values and evaluating the impact of assumptions—a type of assessment known as sensitivity analysis. It also notes the importance of understanding the extent of model uncertainty and inaccuracy—either quantitatively, such as with the confidence interval around a statistical estimate, or qualitatively—and accounting for them appropriately. This type of assessment is known as uncertainty analysis. The assessments are related as sensitivity analysis tests the effects of different sources of uncertainty on a model’s output and can help to identify the greatest sources of uncertainty.

Model documentation we reviewed included some sensitivity and uncertainty analyses at the component model level, but we did not find that any such sensitivity or uncertainty analyses fully considered effects on the extent of model risk associated with the final post-stress capital ratios. For example, the analyses did not consider these effects for both the numerator and denominator of the ratios. As described earlier, a post- stress capital ratio consists of a numerator that reflects the capital held by the company and a denominator that measures the assets held by a company (either on a total or risk-weighted basis), both projected for each quarter of the stress scenario. When considering changes to component model results (which the Federal Reserve uses as one of their primary forms of sensitivity analysis), the Federal Reserve identifies the major inputs that have changed for that model and analyzes the extent to which changes in the model output can be attributed to each input.

One way that the Federal Reserve seeks to put the changes in individual component models in context with each other and with overall capital ratios is to divide the component model output (for example, losses or revenues) by the risk-weighted assets of the company prior to the assets being stressed in the tests. This approach allows the Federal Reserve to understand the component models’ effects on the numerator of the post-stress capital ratio and to put individual component model results on a common scale.

But, some component models also affect the denominator in the post- stress capital ratios. For example, the accrual loan loss models (see fig. 8) that estimate losses on different portfolios of loans (such as automobile or commercial loans) also provide an estimate used in some risk-weighted asset calculations.

Thus, the Federal Reserve’s approach to sensitivity analysis does not reflect a full consideration of the effects of a component model’s risk and uncertainty on that of the post-stress capital ratios.

Model documentation standards from the Model Oversight Group indicate that modeling teams should document the empirical performance of a model to support its validity in projecting stress test losses, including addressing—such as through sensitivity analysis—how a model’s output responds to changes in key inputs and parameters. The oversight group also has issued a guidance memorandum to modeling teams on the types of sensitivity and uncertainty analysis they are encouraged to conduct during model development. However, these guidance documents do not address conducting sensitivity and uncertainty analyses that examine how modeling decisions affect the overall stress test results. Federal Reserve staff told us that performing sensitivity and uncertainty analyses of the system of models was unnecessary because the system was largely additive—a mathematical feature that ensures that tests on a component model will fully capture its effects on the system of models— so that system-wide assessments would be redundant. However, as discussed already, key interactions exist between component models such that their joint outcome would not simply be a sum of the component model outcomes. Federal Reserve staff also explained that performing sensitivity and uncertainty analyses of the system of models—such as calculating statistical confidence intervals for the post-stress capital ratios—would be a complex and resource-intensive undertaking, which may not provide information with a clear use.

Although the guidance does not address conducting sensitivity and uncertainty analyses in relation to effects on stress test results, reviewers from the Model Validation Unit who worked on the 2014 soundness review of a stress test model used to project asset balances (balances model) recommended that the modeling team assess the sensitivity of the post-stress capital ratios to the balances model assumptions. The balances model is the root model for the entire system of models—it is a direct or indirect input for almost all other models—and therefore its design is of particular importance to the accuracy, robustness, and stability of the supervisory stress test process (see fig. 9). This model makes assumptions related to market share and portfolio mix for firms subject to the stress tests.

As the Model Validation Unit recommended in its review, one way to assess the appropriateness of the assumptions made in the balances modeling approach would be to test the effects of alternative assumptions on the post-stress capital ratios and gauge the potential consequences of any differences. Federal Reserve staff indicated that the supervisory modeling team had not tested alternatives to these specific assumptions because (1) the assumptions implement a policy decision by the Model Oversight Group, and (2) the assumptions accomplished the policy decision while remaining consistent with the oversight group principle of simplicity and transparency. However, basing modeling assumptions on policy goals does not preclude also assessing their effectiveness in accomplishing the policy goals or the risk of unintended consequences through testing the potential effects of alternative assumptions on the stress test output. As of July 2016, the modeling team had not yet addressed the recommendation of the validation unit.

Without assessing risks to the post-stress capital ratios posed by the Federal Reserve’s approach to modeling, the Federal Reserve limits its ability to understand, communicate, and manage the risks and reliability of its supervisory stress test results. For example, the Federal Reserve’s model risk-management guidance states that a company’s senior management is responsible for regularly reporting to its board of directors on significant model risks from individual component models and in the aggregate. However, Federal Reserve staff are unable to communicate to the Board of Governors the range and most sources of uncertainty surrounding the post-stress capital ratio estimates produced by the system of models because the Federal Reserve has not conducted the analyses necessary to do so. Furthermore, sensitivity and uncertainty analysis can result in changes to models and even small differences in model estimates can be the difference between the Federal Reserve objecting or not objecting to an institution’s capital plan.

Overall Risk Tolerances Not Articulated

The Federal Reserve has not articulated overall model risk tolerances— the amount of uncertainty or error margins that it would be willing to accept around the post-stress capital ratios. The Federal Reserve’s model risk-management guidance states that members of an institution’s board of directors should ensure that the level of model risk is within their tolerance. It also states that model risk-management policies approved by the board or its delegates should promote the development of targets for model accuracy and standards for acceptable levels of discrepancies.

However, neither the Board of Governors nor high-level management in Banking Supervision and Regulation have identified model risk tolerances for component model output or for the overall stress test results. Instead, Federal Reserve staff said that Model Oversight Group reviews helped to ensure a consistent approach to model risk. In addition to approving decisions included in model documentation, the Model Oversight Group has reviewed options that modeling teams have developed for resolving identified problems with their models and have presented to the oversight group. The presentation includes a discussion of the advantages and disadvantages associated with each option. According to Federal Reserve documentation, the oversight group will direct the modeling team to pursue one of the options or continue developing alternatives. Federal Reserve staff told us that decisions of the Model Oversight Group were based on the principles they had developed and that they weighed a modeling team’s options against how well they meet the principles. But it was not always evident from the documentation what criteria the oversight group used to make its determination about which option to pursue. Even if one principle was cited to support a decision, it was not clear if the option was consistent with other principles, making it difficult to evaluate the consistency of Model Oversight Group decisions or their application of predetermined risk tolerances.

In the absence of explicit direction about risk tolerances, supervisory modeling teams may make decisions that have consequences for model risk without evaluating the model risk against set criteria. For instance, it also was not clear from the documentation we reviewed how much the Model Oversight Group had learned about the options modeling teams had considered and rejected before the presentations. In some cases we reviewed, the modeling teams had fully developed the choices for resolving model problems and were able to quantitatively compare differences between the options—which represents a form of sensitivity analysis and allows for an assessment of the model relative to an explicit risk tolerance (although it did not appear from the documentation that risk tolerances were applied in the decision process). In other cases, the team’s work was at a more preliminary stage and did not include a quantitative evaluation of the consequences of the options under consideration.

Our review of model documentation also suggests that modeling teams have made model risk tolerance decisions at the individual model level with no documented reference to the impact of those decisions on other models or consistency with other modeling decisions. Furthermore, according to documents we reviewed and discussions with Federal Reserve staff, the Federal Reserve has not made any efforts to determine statistical or other thresholds at which each individual model will produce results within tolerable uncertainty ranges in relation to the post-stress capital ratio estimates. Instead, some modeling teams appeared to be implicitly determining what constituted acceptably small variation between their chosen model’s predictions and the historical data. For example, one team assessed its model with statistical tests and deemed it reasonable, providing supporting evidence in charts. There was no discussion of the criteria the team used to determine their model’s soundness. Based on the documentation we reviewed, this team did not appear to have made any calculation of the practical magnitude of the consequences of these statistical tests. The charts used to show the success of the model show a 0.1 percentage point difference between actual and predicted values in the data, in this case default rates of a loan portfolio. The supervisory modeling team asserted that this was a small difference between predicted and actual default rates. However, such a difference could have material consequences for the post-stress capital ratios of companies, as it represented 12 percent of the total predicted default rate of the portfolio. Without an articulated risk tolerance, it is not clear whether this is a large or small difference for the portfolio or the post-stress capital ratios. As for all component models, this model required the Model Oversight Group’s approval prior to going into production. However, the oversight group’s review and approval is not a substitute for an identified risk tolerance as required by the Federal Reserve’s standards.

In addition, the model documentation we reviewed indicated that some supervisory modeling teams tested other individual models in addition to those they reported in the documentation. The undocumented models generally failed to meet statistical or other tests for determining which models would be included in the documentation. But this may lead modeling teams to reject models with attributes that would be desirable at the systems level and the decisions might not be transparent to the Model Oversight Group due to lack of documentation.

According to Federal Reserve staff, the newly formed Supervisory Stress Test Model Governance Committee has plans to expand communication to the Board of Governors to provide Governors with more insight into model development, model risk, and other outstanding concerns about models. The expanded communication also may allow Governors to communicate their model risk tolerances with the Federal Reserve staff performing the supervisory stress tests and CCAR quantitative assessment. However, Federal Reserve staff also told us that model risk tolerances cannot be set prior to the completion of the models. But, this is not consistent with the Federal Reserve’s model risk-management guidance, which requires company management to set predetermined thresholds of acceptability and for senior management to ensure that the level of model risk is within their tolerance. In the same manner as for other major areas of risk, tolerances can be articulated in a number of ways including a combination of quantitative and qualitative approaches. Without systematically identifying and communicating acceptable levels of risk in its supervisory stress test models, the Federal Reserve may be limited in its ability to effectively evaluate and manage its model risk.

Conclusions

The stress test programs implemented by the Federal Reserve during and since the financial crisis of 2007–2009 have played a key role in supervisory efforts to evaluate and maintain the stability of the U.S. financial system. Overall, they represent important advances that augment supervisory approaches to capital adequacy and planning that were in place before the crisis. The Federal Reserve and other bank regulators (i.e., FDIC and OCC) have issued similar stress test rules, but OCC has made greater use of supervisory flexibility—granting extensions to and exemptions from the requirements’ application—in implementing them. This inconsistent approach to implementation could contribute to competitive disadvantages between institutions and inconsistent oversight of risk management by the regulators.

The Federal Reserve has integrated DFAST and CCAR into its supervision of large banking organizations and made changes to the programs in recent years at least partly in response to concerns raised by the industry and market observers that, among other things, adjusted the timing of the exercises, consolidated guidance on supervisory expectations for capital planning, and modified certain technical aspects of capital distribution restrictions and capital action assumptions. The Federal Reserve has established an organizational structure for its CCAR assessments that is guided by core principles and some best practices, and it continues to annually refine and develop its stress test models. However, limitations in analytical approaches and to disclosure present challenges to risk assessment by the Federal Reserve and to transparency. In some cases, the Federal Reserve has not always followed its own guidance or principles.

Quantitative assessment. The Federal Reserve has based its determinations on the results of both the supervisory and company- run stress tests. However, this creates tension between companies’ desire to avoid failing the CCAR quantitative assessment and the robustness of their stress test decisions. By including company-run tests in the CCAR quantitative assessment, the Federal Reserve limits the risk-management and capital planning benefits for participating companies—one of the Federal Reserve’s goals for CCAR—without significantly increasing the effectiveness of the quantitative assessment.

Qualitative assessment disclosure and communication. Although it uses a decision-making framework to assess qualitative CCAR submissions, the Federal Reserve has not publicly disclosed information that would allow for a better understanding of its assessment methodology or the reasons for objection determinations. Transparency is a key feature of accountability and this limited disclosure may hinder understanding of the CCAR program and limit public and market confidence in the program and the extent to which the Federal Reserve can be held accountable for its decisions. The Federal Reserve also has not regularly updated guidance to firms about supervisory expectations and peer practices related to the qualitative assessment. Companies that must meet these expectations annually may face challenges from the irregular timing of communications, which could limit the Federal Reserve’s achievement of its CCAR goals. In addition, the Federal Reserve has not communicated time frames for responding to questions it receives through the CCAR communications mailbox, which could hinder companies’ management and planning of their CCAR submissions and limit their ability to address supervisory concerns in a timely fashion.

Scenario design. The Federal Reserve has conducted limited analysis of some decisions that are important to designing stress test scenarios. IMF principles for supervisory stress testing highlight the risks of basing scenario design decisions solely on historical experience, but the Federal Reserve’s decisions about the severity of its scenarios have been driven by U.S. postwar historical experience. Without a broader consideration, the Federal Reserve could miss opportunities to assess and guard against relevant but unprecedented risks to the banking system. In addition, the Federal Reserve has not explicitly analyzed how to balance scenario severity choices’ influence on banking system resiliency with potential economic effects. Without more careful assessment of the trade-offs associated with scenario severity, the Federal Reserve cannot be reasonably assured that the scenario design process balances any improvements in the resiliency of the banking system with any impact on the cost and availability of credit.

The Federal Reserve also has not conducted analyses to determine if its single severe supervisory scenario is sufficiently robust and reliable to promote the resilience of the banking system against a range of potential crises. Such analyses—including performing sensitivity analysis involving multiple scenarios—could help the Federal Reserve understand the range of outcomes that might result from different scenarios and explore trade-offs associated with reliance on a single severe supervisory scenario. Additionally, the Federal Reserve has not assessed whether or how changes to the supervisory scenarios could inadvertently amplify economic cycles (procyclicality)—which its scenario design policy aims to avoid—until after it has finalized the scenarios. Without additional analysis prior to completing and publishing its scenarios, the Federal Reserve cannot be reasonably assured that small adjustments to the scenario variables would produce outcomes that neither amplify nor dampen economic cycles.

Model risk management. The Federal Reserve’s model risk- management efforts have not focused on the system of stress test models and how component modeling choices affected overall stress test results. In this sense, the Federal Reserve has limited its perspective and it has not always followed its guidance for banking institutions on model risk-management practices.

The Federal Reserve has not assessed its entire system of models in relation to the model development principles that it has applied to individual component models. By not applying those principles to the system of models, the Federal Reserve has limited its ability to manage the extent to which model risk is introduced into the supervisory stress test models.

It has not developed appropriate documentation of the system of models that would allow for effective management of the risks posed by component model interactions. Without such documentation, the Federal Reserve’s ability to effectively identify and manage model risk from the entire system of models is limited, and staff may miss important connections between elements of component models, which in turn may limit understanding of risks inherent in their modeling choices.

The Federal Reserve has not conducted sensitivity and uncertainty analyses of how its modeling affects the post-stress capital ratios. Without such assessments, the Federal Reserve limits its ability to understand, communicate, and manage the risks and reliability of its supervisory stress test results. Furthermore, sensitivity and uncertainty analysis can result in changes to models and even small differences in model estimates can be the difference between the Federal Reserve approving or objecting to an institution’s capital plan.

Staff have been unable to communicate information about the range and sources of uncertainty surrounding the post-stress capital ratio estimates to the Board because the Federal Reserve has not conducted the necessary analyses. Unless staff communicate such information, the Board may not be fully informed of significant model risks from individual component models and in the aggregate including when making decisions based on stress test results.

Neither the Board of Governors nor senior staff have identified risk tolerances for model output or overall stress test results. Without systematically identifying and communicating acceptable levels of risk in its supervisory stress test models, the Federal Reserve may be limited in its ability to effectively evaluate and manage its model risk.

Successfully managing model risk is a key objective because the Federal Reserve uses the system’s overall stress test results with precision to make CCAR determinations. A more holistic approach can help ensure that it makes the determinations with a more complete understanding of the stress test results’ uncertainty and sensitivity to component model decisions and account for them appropriately.

Recommendations for Executive Action

We are making the following 15 recommendations: To help improve the consistency of federal banking regulators’ stress test requirements and help ensure that institutions overseen by different regulators receive consistent regulatory treatment, the heads of the Federal Reserve, FDIC, and OCC should harmonize their agencies’ approach to granting extensions and exemptions from stress test requirements.

To help provide stronger incentives for companies to perform company- run stress tests in a manner consistent with Federal Reserve goals, the Federal Reserve should remove company-run stress tests from the CCAR quantitative assessment.

To increase transparency and improve CCAR effectiveness, the Federal Reserve should take the following four actions:

Publicly disclose additional information that would allow for a better understanding of the methodology for completing qualitative assessments, such as the role of ratings and rankings and the extent to which they affect final determination decisions.

For future determinations to object or conditionally not object to a company’s capital plan on qualitative grounds, disclose additional information about the reasons for the determinations.

Publicly disclose, on a periodic basis, information on capital planning practices observed during CCAR qualitative assessments, including practices the Federal Reserve considers stronger or leading practices.

Improve policies for official responses to CCAR companies by establishing procedures for notifying companies about time frames relating to Federal Reserve responses to company inquiries.

To strengthen the scenario design process, the Federal Reserve should assess—and adjust as necessary—the overall level of severity of its severely adverse scenario by taking the following two actions: establish a process to facilitate proactive consideration of levels of severity that may fall outside U.S. postwar historical experience, and expand consideration of the trade-offs associated with different degrees of severity.

To improve understanding of the range of potential crises against which the banking system would be resilient and the outcomes that might result from different scenarios, the Federal Reserve should assess whether a single severe supervisory scenario is sufficient to inform CCAR decisions and promote the resilience of the banking system. Such an assessment could include conducting sensitivity analysis involving multiple severe supervisory scenarios—potentially using CCAR data for a cycle that is already complete, to avoid concerns about tailoring the scenario to achieve a particular outcome.

To help ensure that Federal Reserve stress tests do not amplify future economic cycles, the Federal Reserve should develop a process to test its proposed severely adverse scenario for procyclicality annually before finalizing and publicly releasing the supervisory scenarios.

Finally, to improve the Federal Reserve’s ability to manage model risk and ensure that decisions based on supervisory stress test results are informed by an understanding of model risk, the Federal Reserve should take the following five actions:

Apply its model development principles to the combined system of models used in the supervisory stress tests.

Create an appropriate set of system-level model documentation, including an overview of how component models interact and key assumptions made in the design of model interactions.

Design and implement a process to test and document the sensitivity and uncertainty of the model system’s output—the post-stress capital ratios used to make CCAR quantitative assessment determinations— including, at a minimum, the cumulative uncertainty surrounding the capital ratios and their sensitivity to key model parameters, specifications, and assumptions from across the system of models.

Design and implement a process to communicate information about the range and sources of uncertainty surrounding the post-stress capital ratio estimates to the Board during CCAR deliberations.

Design and implement a process for the Board and senior staff to articulate tolerance levels for key risks identified through sensitivity testing and for the degree of uncertainty in the projected capital ratios.

Agency Comments and Our Evaluation

We provided a draft of this report to the Federal Reserve, FDIC, and OCC for review and comment. The Federal Reserve, FDIC and OCC provided written comments that we have reprinted in appendix III, IV, and V, respectively. The Federal Reserve, FDIC and OCC also provided technical comments that we have incorporated, as appropriate.

In their written comments the Federal Reserve, FDIC, and OCC generally agreed with the recommendation that the heads of the Federal Reserve, FDIC, and OCC should harmonize their agencies’ approach to granting extensions and exemptions from stress test requirements. The FDIC agreed that a consistent approach to extensions and exemptions was important and noted its commitment to coordinating closely with the Federal Reserve and OCC. The Federal Reserve, FDIC, and OCC noted that although the agencies coordinate closely in administrating their stress testing programs, going forward they each stated that they would coordinate with the other agencies at least annually and more frequently, if appropriate, to discuss any planned extensions and exemptions prior to any action.

In its written comments, the Federal Reserve generally agreed with the report’s other 14 recommendations and offered responses in the following areas:

Regarding our recommendation to exclude company-run stress tests from the CCAR quantitative assessment, the Federal Reserve noted in its letter that the agency was already considering a proposal that would set post-stress capital requirements for covered institutions based solely on the supervisory stress tests. It noted that this proposal was consistent with our recommendation. While we have not yet had the opportunity to assess this proposal in detail, excluding the company-run tests from such a capital requirement could improve incentives for the company-run stress tests.

Regarding our recommendation to strengthen the scenario design process by considering levels of severity that fall outside U.S. postwar history, the Federal Reserve noted in its letter that the 2012 and 2013 severely adverse scenarios featured unemployment rates that were above what has been experienced in postwar U.S. history. However, the level of the unemployment rate for these years does not imply an established process designed to facilitate a consistent consideration of severely adverse scenarios outside of the postwar historical experience. In response to the Federal Reserve’s written comments we modified the language in our recommendation to clarify that we are recommending an established process for a broader consideration of severity given that the current scenario design policy and process is focused on selecting economic conditions that reflect the severity of postwar U.S. recessions. Without consistently and proactively considering levels of severity outside postwar U.S. historical experience, the Federal Reserve could miss opportunities to assess and guard against relevant but unprecedented risks to the banking system.

Regarding our recommendation to expand consideration of the trade-offs associated with different degrees of severity, the Federal Reserve noted in its letter that the scenario design framework was not designed to generate the most severe potential outcomes since that might impinge credit availability. However, our recommendation does not call for the Federal Reserve to generate scenarios that represent the most severe potential outcomes. Our recommendation calls for the Federal Reserve to assess whether more severe, or less severe, scenarios might better balance changes in resiliency against the need to extend credit. As we noted in our report, without a more careful assessment of scenario severity, the Federal Reserve cannot be reasonably assured that the scenario design process balances any improvements in the resiliency of the banking system with any impact on the cost and availability of credit.

Regarding our recommendation to assess the sufficiency of a single severe scenario for the supervisory stress tests, the Federal Reserve noted in its letter that expanding the number of scenarios would be costly and burdensome. As we noted in the report, using a single severe scenario could limit the resources required to design and execute the stress tests. Although the Federal Reserve states incorrectly in their letter that we describe these costs as “substantial,” we did not assess these potential costs in this report. More importantly, our recommendation does not call for the Federal Reserve to increase the number of severe supervisory scenarios. Our recommendation calls for the Federal Reserve to assess the sufficiency of a single severe supervisory scenario. Absent such an assessment—which could be supported by sensitivity analysis using more than one severe supervisory scenario—CCAR decisions may not reflect the uncertainty in stress test outcomes that might result from different scenarios.

Regarding our recommendation to test the severely adverse scenario for procyclicality before finalization and public release, the Federal Reserve noted in its letter that the scenario design process had a feature designed to counteract procylicality. It also noted that additional changes were under consideration to further reduce procylicality (i.e., further limiting the increase in the unemployment rate during a downturn). However, as we noted in the report, given the complexity of the system of models, without conducting additional testing before releasing scenarios, the Federal Reserve cannot be reasonably assured that small adjustments to the unemployment rate would produce outcomes that neither amplify nor dampen economic cycles.

Regarding the recommendations to increase transparency and improve CCAR effectiveness, the Federal Reserve stated that steps were taken to enhance transparency and highlighted guidance released since 2011, including supervisory letters released in 2015 and additional details in the CCAR 2016 results disclosure. Importantly, the Federal Reserve stated that it will continue to enhance transparency in the areas recommended in our report. In addition, the Federal Reserve stated that it will continue to enhance the process for responding to firms’ inquiries while noting that complex questions may take longer to resolve.

Regarding our recommendation to improve documentation of the system of models, the Federal Reserve asserts that it already maintains comprehensive documentation of the development, assessment, validation, and finalization of its system of models. While the Federal Reserve does maintain extensive documentation of each element of the system of models, comprehensive documentation of the system of models as a whole requires documentation of how component models interact and key assumptions made in the design of model interactions, which the Federal Reserve was not able to provide to us. Without this additional documentation, the Federal Reserve’s ability to effectively identify and manage model risk from the entire system of models is limited and may limit understanding of risks inherent in its modeling choices.

Regarding our recommendation to test and document the sensitivity and uncertainty of the model system’s output used to make its quantitative determinations, the Federal Reserve notes that it already assesses how model assumptions impact post- stress capital ratios. However, the Federal Reserve did not provide us with documentation that demonstrated any comprehensive assessments that tested the mathematical and statistical implications of their system of models design. Lack of such testing exposes the Federal Reserve to model risk and limits its abilities to direct model development resources to the areas that introduce the most uncertainty and risk to estimates of the final post-stress capital ratios.

Regarding our recommendations to improve communication of the range and sources of uncertainty surrounding the post-stress capital ratio estimates to the Board during CCAR deliberations and to articulate tolerance levels for key risks, the Federal Reserve notes it established the Supervisory Stress Test Model Governance Committee in 2015 and has future plans to advise the Board on the state of model risk. The Committee was too new during the bulk of our audit work to meaningfully assess its implementation. However, we plan to continue to monitor the Committee to determine whether their activities ultimately address our recommendation.

We are sending copies of this report to the House Committee on Financial Services, the Federal Reserve, FDIC, and OCC. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-8678 or EvansL@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VI.

Appendix I: Objectives, Scope, and Methodology

The Board of Governors of the Federal Reserve System (Federal Reserve) conducts two stress test exercises: the Dodd-Frank Stress Tests (DFAST) and the Comprehensive Capital Analysis and Review (CCAR). This report (1) compares the DFAST and CCAR exercises and discusses company and Federal Reserve views about the exercises’ costs and benefits; (2) examines the CCAR qualitative assessment, including the extent of communication and disclosure; (3) examines how the Federal Reserve designs the supervisory scenarios for the stress tests; and (4) examines the Federal Reserve’s modeling process for the stress tests.

To compare DFAST and CCAR, we reviewed Section 165(i) of the Dodd- Frank Act, the Federal Reserve’s final and amended capital plan and stress test rules, and Federal Reserve policies and procedures about how it has implemented and used DFAST and CCAR in its supervision of banking institutions. We analyzed internal guidance documents and instructions, methodology, and results publications related to DFAST and CCAR; supervisory letters on stress testing and capital planning; public statements by Federal Reserve officials; and other Federal Reserve documentation about the programs. We interviewed staff from the offices of the Federal Reserve that are responsible for DFAST and CCAR, including the Federal Reserve’s Division of Banking Supervision and Regulation, regarding the scope, goals, and utilization of each program. We analyzed information and documentation on stress test extensions and exemptions from the Federal Reserve, the Office of the Comptroller of the Currency (OCC), and the Federal Deposit Insurance Corporation (FDIC). We also interviewed staff from the Federal Reserve, OCC, and FDIC about the use of extensions and exemptions.

To obtain views on the stress tests and their costs and benefits, we judgmentally selected and interviewed 13 companies that participated in CCAR in 2015 and 6 companies that were subject only to DFAST. To select CCAR companies to interview, the team used information on CCAR firms collected from the Federal Reserve and SNL Financial, a private provider of data on the financial services industry. We used the 31 bank holding companies that participated in the 2015 CCAR cycle as our selection pool and selected companies based on their size, industry type, organization type, prior stress test participation, and history of CCAR results. To identify and select companies that were subject to DFAST but not CCAR, we used data from the Federal Reserve and its National Information Center. To expand the coverage and information from each interview, we selected bank holding companies subject only to DFAST that also had a subsidiary depository institution subject to stress test requirements (including firms subject to OCC or FDIC rules). We grouped the depository institutions by charter type—(1) state-chartered banks that were members of the Federal Reserve System, (2) state-chartered banks that were not members of the Federal Reserve System, and (3) nationally-chartered banks—and ordered them by total asset size. We selected institutions with the largest amount of total assets and the company with the smallest amount from each of the three groups that also had a holding company subject to DFAST. If we were unable to schedule interviews with selected companies we chose additional companies based on the same selection criteria. We also reviewed Federal Reserve statements on benefits and costs.

To characterize companies’ views throughout the report, we consistently defined modifiers (e.g., “nearly all”) to quantify each group of interviewees’ views as follows: “all” represents 100 percent of the group, “nearly all” represents 80 percent to 99 percent of the group, “most” represents 60 percent to 79 percent of the group, “several” represents 40 percent to 59 percent of the group, and “some” represents 20 percent to 39 percent of the group. While the percentage of the group of interviews remains consistent, the number of interviews each modifier represents differs based on the number of interviews in that grouping: 19 total CCAR and DFAST firms, 13 CCAR companies, and 6 DFAST-only companies. Table 8 provides the number of interviews in each modifier for each group of interviews.

To examine the process used by the Federal Reserve to conduct the CCAR qualitative assessment, we reviewed the Federal Reserve’s stress test and capital plan rules and other publicly available documents including annual stress test instructions and results, periodically released guidance on supervisory expectations and other topics, and supervisory letters regarding stress testing and bank supervision. We also examined internal policies and procedures, training documents, and other program documentation related to the CCAR qualitative assessments including communication with companies and documentation of Board decision making. The policy and procedure documents included CCAR program manuals and project plans that described roles and responsibilities of staff teams and oversight groups and identified the content and timing of key tasks, among other things. Program documentation we analyzed included evaluation memorandums from four assessment teams covering 8 companies from CCAR 2014 and 2015. We judgmentally selected company-specific workpapers based on companies we interviewed and the involvement of different staff teams from across the Federal Reserve System. We also reviewed conclusion and recommendation memorandums used in making object or non-objection determinations. To examine communication with companies, we reviewed communication procedures and company-specific feedback provided to companies including questions and responses provided through the Federal Reserve’s communication mailbox. We interviewed Federal Reserve staff about how they conduct the qualitative assessment including their policies, procedures, and decision-making process as well as their communication with companies about the assessment and the Federal Reserve’s supervisory expectations. We also interviewed officials from 13 CCAR companies about their experience with the qualitative assessment and interaction with the Federal Reserve including the clarity of supervisory expectations, program guidance, and feedback. We used criteria from Standards for Internal Control in the Federal Government and transparency principles, including directives issued by the Office of Management and Budget, to evaluate the Federal Reserve’s qualitative assessment process and communication with companies.

To examine how the Federal Reserve designs the supervisory scenarios for the stress tests, we conducted interviews and reviewed public and nonpublic documentation related to the scenario design process. We interviewed Federal Reserve officials about the scenario design process, including key considerations and rationales for scenario design policy decisions. We interviewed officials from the International Monetary Fund (IMF) and Bank for International Settlements regarding their own research and experience conducting stress tests. We reviewed public Federal Reserve documentation, including the Policy Statement which governs the scenario design process, CCAR instructions, and the CCAR assessment framework. We analyzed public data from the supervisory quantitative scenarios from 2013 to 2016. We also reviewed nonpublic Federal Reserve documents including internal presentations related to proposed scenarios and CCAR results. To understand relevant standards for complex analyses and stress testing, we reviewed IMF principles for supervisory stress tests and Office of Management and Budget standards for assessing the impact of regulations. Finally, we reviewed Basel Committee on Banking Supervision analyses of the potential impact of post-crisis reforms to strengthen bank capital and liquidity regulations and IMF’s 2015 U.S. Financial Sector Assessment Program.

To examine the Federal Reserve’s supervisory stress test modeling process, we collected and reviewed public and nonpublic Federal Reserve documentation including DFAST- and CCAR-related publications, internal guidance and procedures, policy statements, model documentation, model validation reports, and internal presentations. For model-specific documentation, we reviewed model documentation and validation reports from the DFAST/CCAR 2015 stress test cycle (the most recent available at the time of our examination) for a judgmentally selected sample of component models. After reviewing publicly-available model documentation and examples of nonpublic documentation provided by the Federal Reserve, we requested and analyzed the documentation and validation reports for four supervisory modeling teams, which we selected based on our assessment of their likely importance to the system of models or their potential for presenting analytical challenges. We interviewed Federal Reserve staff from across the Federal Reserve System, including staff involved with supervisory stress test model development and validation, about the process for executing the supervisory stress tests and the Federal Reserve’s model risk management practices. We assessed the Federal Reserve’s supervisory stress test practices using the Federal Reserve’s guidance to bank holding companies on their stress test model risk management activities. To provide additional context for the Federal Reserve guidance, we reviewed publications of the National Research Council, whose members are drawn from the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine, on best practices in complex modeling and model risk management.

We conducted this performance audit from December 2014 to November 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Reviews by the Federal Reserve’s Model Validation Unit and the Office of Inspector General

Board of Governors of the Federal Reserve System (Federal Reserve) Model Validation Unit review. The December 2014 review findings included a shortcoming in policies and procedures, insufficient model testing, insufficient planning and procedures to address the risks posed by potential key personnel departures, and incomplete structures and information flows to ensure proper oversight of model risk management. The review resulted in six recommendations to address the identified findings.

Office of Inspector General (OIG) report. The OIG report identified continuing risks related to model validation and broader governance practices in four areas.

Risks related to validation staffing and performance management existed that may not be mitigated by the implementation of a new staffing approach. These risks included insufficient performance feedback to supplemental reviewers, dependence on key personnel, and inadequate scrutiny of models.

Risks associated with model changes that occur late in the supervisory stress testing cycle remained despite Federal Reserve steps to address these risks.

The Federal Reserve did not maintain an accurate, complete, and updated inventory of models as required by the Federal Reserve’s model risk-management guidance.

In reviewing a sample of validation reports, limitations encountered by reviewers during model validation were not always made clearly identifiable for management in the validation reports submitted to management.

To address these risks, the OIG report made eight recommendations to the Division of Banking Supervision and Regulation, including to establish processes for assessing the materiality of late-stage changes to models that would clarify what changes required independent validation and would leverage reviewer resources to validate such changes.

According to Federal Reserve staff and program documents, the Federal Reserve has been implementing changes to address concerns raised by both reviews. For example, in response to the Model Validation Unit review, the Federal Reserve officials said created the Supervisory Stress Test Model Governance Committee was created to coordinate and oversee its model risk-management efforts. Federal Reserve staff said that the committee met for the first time in May 2015 and explained that its agenda largely has been driven by responding to the findings of the validation unit’s governance review, in particular around model risk. One of the review’s findings was that incomplete governance structures and information flows did not ensure proper oversight of model risk management. The staff noted that the committee was formed to introduce more structure and discipline to the model governance role, including by clarifying reporting lines to the Director of the Federal Reserve’s Division of Banking Supervision and Regulation, who oversees the Model Oversight Group and the Model Validation Unit.

Federal Reserve staff said that the new committee provides a formal venue for discussing differences of opinion and advising the director, some of which was done informally in the past. The staff also noted that they have been exploring opportunities to expand communication of information about model risk with the Board of Governors, including allowing Governors to communicate their preferences regarding modeling decisions and levels of risk.

In its response to the OIG review, the Federal Reserve said that it had already made improvements to address a number of the recommendations and was taking actions in response to others. According to OIG staff, all of the report’s recommendations remained open as of July 2016.

Appendix III: Comments from the Board of Governors of the Federal Reserve System

Appendix IV: Comments from the Federal Deposit Insurance Corporation

Appendix V: Comments from the Office of the Comptroller of the Currency

Appendix VI: GAO Contact and Staff Acknowledgments

GAO Contact

Lawrance L. Evans, Jr., (202) 512-8678, EvansL@gao.gov.

Staff Acknowledgments

In addition to the contact named above, Andrew Pauline (Assistant Director), Kevin Averyt (Analyst-in-Charge), Nancy Barry, Abby Brown, A. Nicole Clowers, Aaron Colsher, Michael Hoffman, Risto Laboski, Marc Molino, Barbara Roesmann, and Jessica Sandler made key contributions to this report. Other assistance was provided by Vida Awumey, Don Brown, David Dornisch, Janet Eackloff, Nathan J Gottfried, Nicholas John, Rob Letzler, Joseph O’Neill, Nadine Garrick Raidbard, Anne Stevens, Karen Tremba, and Jason Wildhagen.