Federal Student Loans: Education Needs to Improve Its Income-Driven Repayment Plan Budget Estimates
Released: 2016-11-30T00:00:00

Letter

Background

Direct Loan Repayment Plans

Education’s Direct Loan program provides financing to students and their parents to help students obtain postsecondary education. This program is currently the largest federal direct loan program with $912 billion in outstanding loans as of June 2016. Under this program, Education issues several types of student loans described in the following sidebar.

William D. Ford Federal Direct Loan Types Subsidized Stafford Loans: Available only to undergraduate students with financial need (generally the difference between their cost of attendance and a measure of their ability to pay, known as expected family contribution). The interest rate as of July 1, 2016 is 3.76 percent. Borrowers are not responsible for paying interest on these loans while in school and during certain periods of deferment. Unsubsidized Stafford Loans: Available both to undergraduate and graduate school students irrespective of financial need. Interest rates as of July 1, 2016 are 3.76 percent for undergraduates and 5.31 percent for graduate school borrowers. Borrowers must pay all interest on these loans. PLUS Loans: Available to graduate student borrowers and parents of dependent undergraduates. The interest rate as of July 1, 2016, is 6.31 percent. Borrowers must pay all interest on these loans. Consolidation Loans: Available to student and parent borrowers wanting to combine multiple federal student loans (including those listed above) into one loan. Repayment periods are extended up to 30 years, thereby lowering monthly payments. Interest rates are equal to the weighted average of the underlying loans.

Education offers a variety of repayment plans for Direct Loan borrowers: Standard, Graduated, Extended, and Income-Driven. Income-Driven Repayment (IDR) is an umbrella term that describes a number of repayment plans available to Direct Loan borrowers who meet specific eligibility requirements, as seen in figure 1. Unlike the Standard, Graduated, and Extended repayment plans, IDR plans offer loan forgiveness at the end of the repayment term. Additionally, their repayment terms are longer than under the Standard and Graduated plans, which are set at 10 years for non-consolidated loans.

Borrowers in IDR plans generally have lower monthly payments compared to the Standard 10-year repayment plan. They may also pay less in the long term than they would under the Standard 10-year repayment plan due to the opportunity for eventual loan forgiveness. However, some borrowers may pay more. Borrowers in IDR plans can ultimately pay more in interest on their loans than they would under the Standard 10-year repayment plan due to longer repayment periods. Some borrowers will also fully repay their loans before their IDR plan repayment term ends and, therefore, not receive forgiveness. Additionally, under current tax law any amount forgiven under these plans is subject to federal income tax.

In addition to making monthly payments more manageable (and eventually reducing the total amount owed for some borrowers receiving forgiveness), IDR plans may also reduce the risk of default. Borrowers who default on student loans face serious consequences, including damaged credit ratings and difficulty obtaining affordable credit in the future. In 2015, we reported that borrowers in two IDR plans had much lower default rates than borrowers in the Standard repayment plan. Specifically, among borrowers who entered repayment from fiscal year 2010 through fiscal year 2014, less than 1 percent of borrowers in the Income-Based Repayment and Pay As You Earn had defaulted on their loan, compared to 14 percent in the Standard repayment plan.

To participate in an IDR plan, borrowers must provide documentation of their adjusted gross income (which we generally refer to as income in this report) to their loan servicer and certify their family size for an eligibility determination. Borrowers must recertify this information annually, which is used to update the borrower’s monthly payment amount. A borrower who fails to provide updated income information can remain in an IDR plan in order to qualify for future loan forgiveness, but their monthly payments will no longer be based on their income. Rather, payments will generally revert to the amount that would be owed under the Standard 10-year repayment plan until the borrower submits the required information.

Borrowers who work in public service may lower their long-term loan costs by participating in the Public Service Loan Forgiveness (PSLF) program while repaying their loans through an IDR plan. Beginning in October 2017, borrowers eligible for PSLF can have their remaining Direct Loan balances forgiven after at least 10 years of payments in eligible repayment plans, generally an IDR plan or the Standard 10-year repayment plan. As we recently reported, PSLF may provide substantial savings over the life of the loan for qualifying borrowers in IDR plans compared to what they would pay without the PSLF benefit. In contrast, borrowers in the Standard 10-year repayment plan would pay their loans in full by the time they were eligible for forgiveness under PSLF. (See figure 2.)

Participation in IDR plans has grown over time, as seen in figure 3. According to currently available quarterly data released by Education, the percent of outstanding Direct Loan dollars being repaid through IDR plans doubled from June 2013 to June 2016 to 40 percent. The percent of borrowers participating in IDR plans more than doubled over the same time period to 24 percent. However, as we previously reported, some borrowers who could benefit from IDR plans may still not be aware of them.

Proposals to Modify IDR Plans

As the variety of IDR options available to borrowers has expanded in recent years, there have been numerous reform proposals with a variety of goals ranging from simplifying IDR plans and better targeting their benefits to changing the tax treatment of IDR plan loan forgiveness. For instance, recent President’s budgets have proposed limiting the available IDR plan options for new borrowers to one revised IDR plan designed to better target benefits to the highest-need borrowers. A proposal has been introduced in the current Congress that would similarly make only one IDR plan available to new borrowers and target more generous benefits to those with lower incomes. Additional legislative proposals would automatically enroll all borrowers in a version of income-driven repayment and withhold payments from borrowers’ paychecks. Other proposed legislation would allow for automatic annual recertification of borrowers’ incomes and automatically place certain delinquent borrowers in an IDR plan. Another proposal would expand IDR plan eligibility to parents with Parent PLUS loans for dependent students. Legislation has also been introduced that would exempt student loan forgiveness under certain IDR plans from being taxed as income.

Subsidy Cost Estimates

As required by the Federal Credit Reform Act of 1990, Education estimates the long-term costs, known as subsidy costs, of the Direct Loan program annually for inclusion in the President’s budget. For Direct Loans, subsidy costs represent the estimated cost to the government of extending credit over the life of the loan, excluding administrative costs. (In this report, we generally refer to subsidy costs as “costs.”) Subsidy cost estimates are calculated based on the net present value of lifetime estimated cash flows to and from the government associated with these loans. For Direct Loans, cash flows from the government include loan disbursements to borrowers, while cash flows to the government include repayments of loan principal, interest and fee payments, and recoveries on defaulted loans. A positive subsidy cost estimate indicates that the government anticipates a net cost, while a negative subsidy cost estimate indicates that the government anticipates generating net subsidy income, not counting administrative costs.

Education also annually reestimates the cost of loans made in each fiscal year, known as a loan cohort. Reestimates take into account actual loan performance as well as changes in assumptions about future performance, such as how many borrowers will default or how many will participate in different repayment plans. Reestimates may result in increases or decreases in subsidy cost estimates. No loan cohorts have been fully repaid, and estimates for all cohorts continue to be updated annually in the President’s budget.

To estimate subsidy costs, Education has developed a student loan cash flow model (the student loan model) that incorporates a variety of assumptions about the future. These assumptions concern various aspects of loan performance, such as how many borrowers will prepay their loans, how many borrowers will default, and how successful default collection activities will be. Education uses a supplementary model to assist with the task of estimating repayment patterns for loans in IDR plans. (See appendix II for a description of how this supplementary model for estimating IDR plan repayment patterns works.) In the spring of 2015, Education initiated a redesign of its overall student loan model with technical support from Treasury and guidance from OMB in what is anticipated to be a multi-year project.

Education’s Budget Estimates of IDR Plan Costs Are Growing, but Actual Costs Will Not Be Known for Many Years

Education Estimates That Loans in IDR Plans Will Have Substantial Costs to the Government

Through our analysis of data underlying the President’s fiscal year 2017 budget, we found that Education estimates that Direct Loans in IDR plans will cost the government about $74 billion over their repayment term. More specifically, Education estimates that about $355 billion in loans will enter an IDR plan, and $281 billion will ultimately be paid by borrowers.

As a result, Education expects a 21 percent subsidy rate, or an average cost to the government of $21 per every $100 in loans disbursed. See figure 4.

All of the Direct Loan types eligible to participate in IDR plans contribute to the $74 billion Education estimates the government will incur in subsidy costs. Of these loan types, Consolidation loans are estimated to be the most costly, as seen in figure 5. Consolidation loans, which combine multiple existing federal student loans into one loan, are larger on average than other types of Direct Loans, and may have higher balances forgiven at the end of their repayment term. Further, Education officials said that some borrowers in IDR plans with Consolidation loans have higher default risks than other borrowers, which leads to higher expected subsidy rates for these loans. Education estimates lower subsidy costs for Subsidized and Unsubsidized Stafford and PLUS loans for graduate student borrowers (known as Grad PLUS loans) than Consolidation loans.

As figure 6 shows, Education estimates higher subsidy costs for loans participating in IDR plans from more recent loan cohorts compared to loans from older cohorts.

Figure 7 shows that these higher estimated costs track closely with the higher loan volume (or total loan dollars) estimated to enter IDR plans for more recent loan cohorts. Education officials confirmed that this higher estimated loan volume is likely related to three key factors: more generous IDR plans available for loans issued since fiscal year increased efforts to make borrowers aware of IDR plans, and increased overall volume of Direct Loans issued as a result of increased college attendance following the 2008 economic downturn and the end of the Federal Family Education Loan program (which guaranteed federal student loans issued by private lenders) in 2010.

While borrowers in IDR plans in more recent loan cohorts have access to more generous benefits (which could lead to higher government costs), these loan cohorts do not have higher estimated subsidy rates than earlier loan cohorts, as seen in figure 8. Direct Loan subsidy rates fluctuate according to changes in a variety of factors, and are particularly sensitive to changes in government borrowing costs and borrower interest rates. As we previously reported, government borrowing costs fell sharply in 2009, due to historically low interest rates of Treasury securities. This phenomenon contributed to lower overall estimated subsidy rates for Direct Loans issued following the 2008 loan cohort.

Education’s Current IDR Subsidy Cost Estimates Are Higher than Previously Expected, but Actual Costs Will Not Be Known for Many Years

Education has raised its estimates of IDR plan costs in recent years through its annual process of revising past budget estimates to account for actual loan performance and updated assumptions about future loan performance. In figure 9, we compare Education’s original IDR plan subsidy cost estimates for loans issued in recent cohorts to its current subsidy cost estimates prepared for the President’s fiscal year 2017 budget. Our results show that current estimated IDR plan costs are more than double what was originally expected for these cohorts. For instance, Education originally estimated in the President’s fiscal year 2012 budget that IDR plan costs for the 2012 cohort would be $1.2 billion. As of the fiscal year 2017 budget, Education’s estimate had grown to $3 billion. (We also compared Education’s fiscal year 2016 IDR plan budget estimates to its fiscal year 2017 budget estimates to illustrate how Education’s cost estimates changed over one budget cycle, and present the results of that analysis in appendix IV.)

As seen in figure 10, subsidy rates have remained relatively stable from original to current estimates, while the volume of loans expected to be repaid in IDR plans has increased dramatically. Because Education expects loans in IDR pans to have positive subsidy rates (or to have costs to the government), this growth in estimated loan volume has been accompanied by increasing estimates of IDR plan costs.

According to our data analysis and interviews with Education officials, Education may have originally underestimated the volume of loans that would enter IDR plans from these cohorts for several reasons: 1. Education did not include Grad PLUS loans in its IDR plan subsidy estimates until the fiscal year 2015 budget, even though Grad PLUS loans have been eligible for IDR plans since they were first issued in 2006. Education officials said that they had to make a model adjustment in order to include Grad PLUS loans in IDR estimates. Prior to this adjustment they assumed all Grad PLUS loans would be repaid in other repayment plans. 2. Policy changes made IDR plans more generous and available to more borrowers after Education originally estimated costs for some cohorts. For example, the Pay As You Earn repayment plan was implemented in fiscal year 2013 and retroactively made more generous benefits available to certain borrowers with loans issued as early as the 2008 cohort. 3. While some eligible borrowers still may not be aware of IDR plans, participation rates are growing, and officials responsible for budget estimates may not have adequately anticipated participation growth.

While we previously reported that there are substantial challenges associated with estimating Direct Loan subsidy costs, these challenges are increased for Direct Loans in IDR plans due to their complex features and other uncertainties. It is difficult for Education to estimate which borrowers have incomes low enough to benefit from or be eligible for IDR plans because Education does not collect income information for all Direct Loan borrowers. Additionally, IDR plan participation rates are difficult to predict. While participation has been growing rapidly in recent years, it is unclear at what rate it will continue to grow. It is also challenging to predict how the incomes of borrowers already participating in IDR plans will change over time and how much loan principal will ultimately be forgiven. Further complicating Education’s task is the fact that the large majority of loans expected to be repaid in IDR plans are from recent cohorts, and many borrowers in these cohorts have not yet started repaying their loans. As a result, there is limited actual repayment data available to inform Education’s estimates. Further, no borrower has received loan forgiveness under IDR plans.

Volatility in subsidy cost estimates is generally expected to be greatest early in the life of a loan cohort, and to decrease over time as more actual repayment data are incorporated into estimates. When we compared original, third-year, and currently estimated IDR plan subsidy cost estimates for several recent cohorts, we found that third-year estimates were generally closer to current estimated costs than the original, as figure 11 illustrates. However, estimates will continue to change over time, and actual subsidy costs of a loan cohort will not be known until all loans in the cohort have been repaid, which may take 40 years.

Current Estimated IDR Plan Costs Are Offset by Greater Estimated Income from Loans in Other Repayment Plans

While loans in IDR plans are expected to have long-term costs to the government, loans in other repayment plans (Standard, Graduated, and Extended) are expected to generate greater subsidy income, as seen in figure 12. Figure 12 also illustrates that Education currently expects income to be higher for more recent cohorts than older cohorts. However, as mentioned previously, subsidy cost estimates change over time, and the actual costs or income attributable to any Direct Loan cohort will not be known until all loans in the cohort are repaid.

Subsidy income estimates for loans participating in non-IDR plans vary by loan type and repayment plan. Unsubsidized Stafford and PLUS loans participating in the Standard 10-year repayment plan are estimated to result in the greatest subsidy income to the government. This could be due in part to the higher interest rates charged to borrowers with Unsubsidized Stafford and PLUS loans compared to Subsidized Stafford loans, as well as a higher volume of loans participating in Standard repayment compared to other repayment plan options. See figure 13.

Further, as with loans in IDR plans, Education’s estimates of subsidy income from loans in non-IDR plans have changed over time and will continue to fluctuate as they are updated with actual repayment data and revised assumptions about future cash flows. We found that estimated income associated with loans participating in non-IDR plans increased (about $19 billion more) for some cohorts and decreased (about $36 billion less) for other cohorts when we compared Education’s original and current estimates for those cohorts (2009-2015).

While Education currently estimates that loans in IDR plans will have costs to the government, these plans are designed to provide relief to struggling borrowers, which could indicate that government subsidies may be expected. By tying monthly payments to borrowers’ incomes, IDR plans help make potentially onerous student debt payments more affordable for many individuals. Because these borrowers’ repayment amounts may be lower than they otherwise would be, borrowers in IDR plans may have more success in making their loan payments than borrowers in other plans. As we previously reported, substantially lower percentages of participants in the Income-Based Repayment and Pay As You Earn repayment plans had defaulted on their loans compared to those in the Standard 10-year repayment plan, and the great majority of borrowers in these IDR plans were in active repayment status (e.g., not in delinquency, default, or forbearance). Further, because IDR plans attract borrowers experiencing difficulty repaying their loans in other plans, increased IDR participation from these borrowers may lead to lower subsidy rates for non-IDR plans.

IDR Plan Budget Estimates May Be Unreliable Due to Limitations in Education’s Approach and Inadequate Quality Control Practices

Education’s approach to estimating IDR plan costs has numerous weaknesses that may result in unreliable budget estimates. Poor quality control practices, such as inadequate model testing, contributed to issues we identified. Further, because Education publishes only limited information about its estimates, it may be difficult for policymakers to assess expected plan costs and consider the potential for alternative outcomes.

Education’s Approach to Estimating IDR Plan Costs May Produce Unreliable Budget Estimates

Due to several methodological limitations, Education’s approach to estimating IDR plan costs may result in unreliable budget estimates. First, Education did not adequately assess the reliability of the data it uses to forecast borrower incomes over time, or assess the level of error these data or its forecasting methods introduced into its IDR plan budget estimates. Second, it did not consider how inflation would affect borrowers’ incomes over time. Third, Education unrealistically assumes that no borrower will fail to recertify their income, which is required of borrowers annually to maintain lower income-driven payment amounts. Fourth, Education does not account for future growth in IDR plan participation rates. Fifth, Education does not produce separate cost estimates for each of the five IDR plans currently available to borrowers. Finally, Education’s cost estimates for Subsidized Stafford, Unsubsidized Stafford, and Grad PLUS loans in IDR plans do not account for likely differences in how they will perform over time.

Borrower Income Data and Forecasting Methods

Education’s IDR plan cost estimates are vulnerable to unidentified error because Education has not adequately assessed the reliability of the estimated borrower income data and methods it uses to forecast borrower incomes many years into the future—information that is vital to determining how much borrowers will owe and repay on their loans over time. Education conducted only limited, informal testing to assess the data’s reliability, in part because the agency had short timeframes in which to develop its approach to estimating IDR plan costs, according to officials we interviewed. Education did not measure the amount of error these data introduced into IDR plan cost estimates to determine whether it was acceptable, or if alternative data were needed.

Through our data reliability testing, we identified patterns in the estimated historical income data suggesting reliability problems that could make them unacceptable for Education’s purposes. An analysis by Treasury (the agency that created the estimated historical income data) indicates that the data fluctuate on average by 44 percent more per year than the actual income data upon which they were based. In figure 14, we illustrate this fluctuation for five randomly selected borrowers from the estimated dataset over the first 10 years of their repayment period. (See appendix III for more information on how these data were estimated and our evaluation of them.)

Education uses individuals’ estimated historical incomes, such as those illustrated in figure 14, to make numerous sequential calculations that determine how much each borrower will owe and pay in each year of the borrower’s repayment period. While the estimated historical income data appeared more reasonable in the aggregate, Education officials confirmed that any unusual fluctuations in them at the individual borrower level could affect the quality of IDR plan budget estimates.

In addition to being vulnerable to error associated with the estimated historical income data they use, Education’s IDR plan budget estimates may further be affected by error associated with the agency’s method for forecasting borrowers’ incomes for up to 30 years into the future. The accuracy of any forecast—separately from the reliability of the data used for forecasting—depends on how well the data and forecasting methods can estimate future incomes. However, Education did not assess the amount of error in its forecasts of borrower incomes. Until Education assesses its forecasting methodology, its IDR plan cost estimates may be vulnerable to unidentified error.

Both federal guidance for estimating subsidy costs and Education’s own information quality standards emphasize the importance of ensuring that estimates are based on reliable data. Education’s information quality standards and generally accepted statistical practices also recommend measuring error to assess its impact on estimates. Education officials agreed with the concerns we raised regarding their borrower income data and said they are open to improving data quality as necessary to help ensure reliable IDR plan budget estimates. Quality data and methods are essential to Education’s estimation approach, and both should be assessed to determine whether they produce reasonable results. (See appendix III for more information on error associated with Education’s data and methods.)

Effect of Inflation

In addition to insufficiently assessing the reliability of its income data and forecasting methods, Education has not adjusted its income forecasts for inflation, causing IDR plan budget estimates to appear higher than they otherwise would be. Adjusting for inflation would increase borrowers’ future incomes and payment amounts, because loan payments are based on borrowers’ incomes. Increasing payment amounts would, in turn, decrease costs to the government. When asked, Education officials said they did not adjust income forecasts for inflation because they did not identify patterns in the estimated historical income data suggesting that incomes would be affected by inflation. Whether or not these patterns were evident when reviewing the data, there was inflation over the almost 20-year period covered by the historical dataset and there is likely to be inflation in the future. Federal guidance for estimating subsidy costs stresses the importance of taking economic effects into account when estimating loan performance. For IDR plan costs, this would include the extent to which inflation affects borrower incomes and payment amounts.

By choosing not to adjust income forecasts to capture inflation’s future effects, Education over-estimated IDR plan costs. When we used Education’s data and computer programs to adjust borrowers’ future incomes for inflation, as well as the federal poverty guidelines used to calculate their discretionary incomes, we found that IDR plan budget estimates declined by over $17 billion, when compared to Education’s current IDR plan budget estimates. (See figure 15.) In light of the substantial effects of inflation on borrower incomes and loan repayment amounts, inflation adjustment is essential to developing reliable IDR plan budget estimates. Until Education adjusts for inflation, its budget estimates will continue to inaccurately represent potential IDR plan costs.

Additionally, Education assumes that all borrowers in IDR plans will recertify their incomes every year as required, which is likely to be inaccurate and could lead Education to overstate IDR plan costs. In fact, we recently reported that over half of borrowers in an Education sample failed to do so. When borrowers fail to recertify their income, Education generally increases their payments to what they would owe under the Standard 10-year repayment plan until they submit their required recertification. For some borrowers who fail to recertify their income, payments could increase by hundreds of dollars a month. While some borrowers may subsequently recertify within a few months, others may never recertify. Because Education does not take these occurrences into account, it underestimates what borrowers will pay when their certification lapses.

Education officials told us they did not include certification lapses in their approach to estimating IDR plan costs because they lacked recertification data linked to individuals. They also believed that certification lapses would not have a large impact on their estimates. Initially, officials said the agency is taking steps to reduce the number of borrowers failing to recertify. However, officials later acknowledged that these efforts are in the early stages of implementation, and there have been some setbacks. Until efforts to improve recertification rates are put in place, certification lapses will likely continue. Further, without data indicating that certification lapses do not have a large impact on borrower payment amounts, Education may overstate IDR plan costs. Federal guidance for estimating subsidy costs states that the information used in the estimation process should reflect actual repayment patterns for loans whose costs are being estimated, which would include instances when a borrower’s payment amount changes due to program rules. Obtaining data on borrowers’ actual repayment patterns after they fail to recertify their income could help Education determine whether its current approach appropriately accounts for the impact of recertification failure on IDR plan costs, and determine whether changes are needed.

Future IDR Plan Participation

Education likely underestimates IDR plan participation because it assumes all borrowers will remain in their currently selected repayment plan for their entire repayment period. This assumption conflicts with the fact that borrowers can switch into or out of IDR plans at any time, and IDR plan participation has grown in recent years. Participation is also likely to continue growing. Education agreed with our recent recommendation that the agency increase its efforts to make all borrowers aware of IDR plans. Further, as previously mentioned, the Administration recently announced a goal to enroll 2 million additional borrowers in IDR plans.

As a result of Education’s likely underestimation of IDR plan participation, its IDR plan budget estimates may be biased downward, or appear lower than they otherwise should be. We found that Education’s IDR plan budget estimates for loans issued in recent cohorts have more than doubled over what was originally expected ($53 billion vs. $25 billion), primarily because of higher than expected participation in IDR plans. Federal guidance for estimating subsidy costs of federal loan programs states that it is preferable to use methods to estimate costs that are more sophisticated than relying solely on historical data, such as borrowers’ past plan selection.

While Education’s current student loan model was not designed to project future changes in plan participation, officials told us that despite the challenge of predicting future borrower behavior they are working with Treasury to develop a more sophisticated model, and have begun incorporating this enhancement into a test version of this new model. Additional work remains to ensure that the new model reasonably reflects trends in IDR plan participation—particularly borrowers switching into IDR plans from other repayment plans. For instance, IDR plans have not yet been added to the new model, which currently includes only the Standard and Extended repayment plans. Education’s model redesign is anticipated to be a multi-year project, and until the model has been completed and tested to ensure reasonable results, Education’s IDR plan budget estimates are vulnerable to underestimated IDR plan participation and costs.

Cost Estimates for Different IDR Plans

Additionally, Education does not produce separate cost estimates for each of the five IDR plans currently available, even though these plans provide different benefits to borrowers and will likely have different costs to the government. For instance, the Income-Contingent Repayment plan has less generous provisions for borrowers than the Pay As You Earn plan, and as a result will likely have lower costs to the government. However, Education does not estimate these plans’ costs separately. According to Education officials, the student loan model, which it uses to generate official estimates of total Direct Loan costs, was created when only one IDR plan was available and cannot produce separate estimates for each IDR plan. While the supplementary model Education uses to estimate IDR plan repayment patterns could track repayment streams separately for each plan, its outputs must conform to the structure of the larger student loan model.

Federal guidance for estimating subsidy costs for federal loan programs specifies that agencies should assess the impact of changes in laws or regulations (such as the introduction of new repayment plans) on the reliability of estimates and should ensure that an agency’s methodology reflects these changes. While Education officials expressed concern about the complexity of estimating separate costs for each IDR plan, OMB staff told us that Education should add this capability as part of Education’s efforts to develop a more sophisticated model. Incorporating the ability to track costs of each IDR plan separately would help ensure that estimates more accurately reflect the current loan environment and provide valuable information to policymakers interested in streamlining student loan repayment options moving forward.

Repayment Patterns for Different Loan Types

Lastly, Education combines repayment patterns for several types of loans eligible for IDR plans, obscuring likely differences in their performance over time. As a result, its budget estimates for Subsidized Stafford, Unsubsidized Stafford, and Grad PLUS loans in IDR plans are based on identical repayment patterns, although these types of loans have numerous distinct features. For instance, the current interest rate on a Grad PLUS loan is almost double that of a Subsidized Stafford loan, leading borrowers with Grad PLUS loans to owe much more in interest on those loans over time. Conversely, borrowers with Subsidized Stafford loans will pay down principal on their loans more quickly over time because less of their payment goes toward interest. However, Education’s cost estimates do not reflect higher expected interest payments on Grad PLUS loans in IDR plans or faster principal repayment on Subsidized Stafford loans in IDR plans, because they are based on aggregate repayment patterns that include both types of loans. Education officials told us that, as a result of this practice, all differences in published subsidy rates for these loan types are wholly attributable to fees charged to borrowers at the time the loans are issued and how much interest accrues during the relatively short period that borrowers are still in school.

Because Education’s estimates do not reflect differences in performance over the decades that loans in IDR plans may be in repayment, users of the budget are missing key information that could help them assess how IDR plan costs vary by loan type. As an example, some experts have raised concerns that Grad PLUS loans could have relatively high forgiveness amounts because they are larger on average than Stafford loans and may have a large amount of outstanding loan principal at the end of their repayment term. Due to limitations in Education’s current approach, users of the budget cannot determine the extent to which this concern affects subsidy rates for Grad PLUS loans in IDR plans.

According to Education officials, they could have separately estimated repayment patterns for each loan type, but did not believe that it was important to do so for several reasons. First, officials stated that they focused their efforts on estimating separate repayment patterns for Consolidation loans because they make up the majority of loans in IDR plans. However, nearly half of IDR plan loan volume—or $164 billion—is made up of Subsidized Stafford, Unsubsidized Stafford, and Grad PLUS loans, and it is important to estimate their repayment patterns accurately as well. Second, officials stated that they did not believe it was necessary to maintain separate repayment patterns for each type of loan because borrowers often have a mix of loans and repay them simultaneously. While this is true, policymakers have an interest in budget information that accurately reflects expected costs for each type of loan eligible for IDR plans. Further, federal guidance for estimating subsidy costs of federal loan programs states that loan characteristics—such as loan types—are critical for identifying factors that predict subsidy costs and should be preserved. Until Education separately tracks repayment patterns for each type of loan in IDR plans, its cost estimates will continue not to take into account important differences in loan characteristics, calling into question the reliability of the cost estimates, and policymakers will be unable to assess the relative costs of different types of loans.

Education’s Quality Control Practices Are Inadequate to Help Ensure That Its IDR Plan Budget Estimates Are Reliable

Inadequate quality control practices contribute to concerns we identified regarding Education’s approach to estimating IDR plan costs. First, management has not ensured that the agency’s supplementary model for estimating IDR plan repayment patterns is properly documented. Second, management has not reviewed or approved that model. Third, management has not ensured that the model has been sufficiently tested for reliability.

Documentation

Education has not ensured that its supplementary model for estimating IDR plan repayment patterns is properly documented. While a broad narrative summary of the model is available, agency officials confirmed that other technical documentation recommended in federal guidance for estimating subsidy costs does not exist. For instance, Education does not have a flow chart or other similar documentation specifying how elements of the estimation process—which is implemented by nearly 50 computer programs—are sequenced and interact with each other. Additionally, the numerous mathematical formulas embedded in these programs are not separately documented, and there is no data dictionary to decode the variable names and values. Standards for internal control in the federal government state that documentation is a necessary part of an effective internal control system. Federal guidance for estimating subsidy costs states that model documentation should be thorough enough that a knowledgeable independent person could follow the estimation process and replicate its results with little to no assistance. Such documentation is not available for Education’s supplementary model for estimating IDR plan repayment patterns.

We recently recommended that Education improve documentation of its overall process for estimating costs of Direct Loans. Education agreed with this recommendation, and officials stated that they were in the process of improving their documentation practices.

Review and Approval

Further, we found that Education’s managers did not review and approve the supplementary model for estimating IDR plan repayment patterns, as recommended in federal guidance for estimating subsidy costs, after it was developed by staff. Additionally, as a good practice, we have found that agencies often hire an independent firm to ensure that model calculations are accurate and consistent with documentation. However, Education officials confirmed that their supplementary model for estimating IDR plan repayment patterns has not been reviewed by an independent firm.

Some of the concerns we identified in the previous section of our report regarding Education’s estimation approach could have been identified and resolved through an internal management review or independent external review. For instance, we found that the decision not to adjust borrower income forecasts for inflation causes IDR plan budget estimates to be $17 billion higher than they otherwise would be. We also found that PSLF loan forgiveness was programmed to begin a year after the benefit will actually become available to eligible borrowers. When we revised these programs to allow loans to be forgiven a year earlier, estimated IDR plan costs rose by $70 million. Agency staff told us this decision was made because borrowers were not likely to make the 120 consecutive on- time payments necessary to qualify for immediate forgiveness. However, Education already makes assumptions about when borrowers will not make scheduled loan payments. An internal management review or independent external review may have pointed toward another solution—such as adjusting how often borrowers are assumed to have periods of non-payment—rather than simply delaying the PSLF start date.

We recently recommended that Education create a documented process for management review and approval of its student loan model. Education agreed with this recommendation, and officials told us they also hoped to have their revised student loan model reviewed by an outside party in the future.

Model Testing

Although Education currently expects loans in IDR plans to be the most costly component of the Direct Loan portfolio, management has not ensured that its supplementary model for estimating IDR plan repayment patterns has been thoroughly tested. Such testing can help identify weaknesses so that they can be addressed, and help ensure that estimates are reasonable. As we previously mentioned, Education had not conducted the necessary testing to thoroughly assess the reliability of its borrower income data or measured error associated with its income forecasting methodology. Without such testing, Education officials do not know whether their data and methods produce reasonable results, or if alternatives are needed.

Further, Education conducted sensitivity analysis on only one key assumption—borrower incomes—at the request of OMB. Federal guidance for estimating subsidy costs states that agencies should conduct sensitivity analysis—which involves adjusting an assumption up or down by a fixed proportion—or other testing to identify which assumptions have the largest influence on cost estimates. This information helps management anticipate the cost implications of alternative scenarios and focus oversight resources on key assumptions to help ensure that they are reliable and reasonable. However, Education officials told us they only conducted sensitivity analysis when asked by others, preferring instead to focus their resources on developing a single set of assumptions they believed were best. Developing a sound set of assumptions is, of course, important. Sensitivity analysis supports, rather than detracts from, this effort.

For instance, little is known about how many borrowers are eligible for or will participate in PSLF when it becomes available in October 2017. Despite this uncertainty and concerns among some experts and policymakers that PSLF could be costly to the government, Education has not conducted sensitivity analysis on its PSLF participation assumption.

In order to illustrate the importance of conducting sensitivity analysis on major assumptions, we first revised Education’s computer programs to increase the percentage of borrowers expected to participate in PSLF by 5 and 10 percentage points. As illustrated in figure 16, costs rose by $4.4 and $9 billion, respectively.

We then decreased the percentage of borrowers participating in PSLF by 5 and 10 percentage points. As seen in figure 17, costs fell by similar amounts.

Our results illustrate the potential for PSLF costs to be different than what Education currently expects, and why it is important for Education to monitor this assumption and adjust it as necessary to ensure that it is reasonable. Without conducting similar sensitivity analysis on other major assumptions, monitoring those assumptions carefully, and adjusting them as necessary to ensure that they are reliable, Education’s budget estimates are vulnerable to bias that could result in costs being over- or understated by billions of dollars.

Published Information May Not Be Sufficient for Policymakers to Accurately Assess Expected IDR Plan Costs

In addition to identifying limitations in Education’s approach to estimating IDR plan costs and its quality control practices, we also found that Education has not published sufficient information about its estimates for policymakers to readily assess expected IDR plan costs. The kinds of information that Education has not published—and that could be useful to policymakers—include (1) total expected costs, (2) trend in estimates, (3) sensitivity analysis results, (4) limitations in estimates, and (5) estimated forgiveness amounts. Education officials noted that the department takes its responsiveness to policymakers and the general public seriously, and that the agency has responded to information requests about IDR plan cost estimates by congressional staff. However, congressional interest in IDR plans is high, and currently available information may be insufficient for policymakers to accurately assess likely plan costs, and consider the potential for alternative outcomes.

For instance, as a part of the President’s budget, Education publishes IDR plan loan volume and subsidy rate estimates for loans issued in the current and two most recent cohorts. This information can be used to calculate expected IDR plan costs for this limited group of loans. However, it is not possible to use this information to determine total expected costs for all loans in IDR plans. Additionally, Education has disclosed in reports accompanying the President’s budget that IDR plans are major contributors to upward revisions in estimated Direct Loan costs as a whole—it has not reported the amount by which IDR plan costs have risen or clearly described the reasons why.

Using unpublished data from Education, we found that total current expected IDR plan costs are about $74 billion, or $21 for every $100 issued. We also found that expected IDR plan costs have doubled from $25 to $53 billion for loans issued from fiscal years 2009 through 2016— primarily due to the growing volume of loans expected to be repaid in IDR plans. Publishing more comprehensive information like this could help policymakers better understand currently expected costs and monitor trends in the Direct Loan portfolio.

Additionally, by publishing sensitivity analysis results and limitations in estimates, Education could help policymakers understand what is known about possible IDR plan costs, and what is still unknown. Our own sensitivity analysis illustrates that IDR plan costs could be billions of dollars more or less than currently estimated if PSLF participation is higher or lower than expected. Given growth in IDR plan cost estimates over time due to the rising volume of loans expected to be repaid in these plans, it would also be useful to disclose that current estimates assume that no borrowers will switch from other repayment plans into IDR plans in the future.

Lastly, sharing the amount of principal Education expects to forgive on loans in IDR plans could help policymakers better understand a key plan feature that contributes to their expected costs. Education officials raised concerns that publishing forgiveness amounts could be misleading, because it is possible for the government still to generate income on loans with principal forgiven, particularly if borrower interest payments exceed forgiveness amounts. While this is true, loan amounts forgiven do represent foregone cash flows to the government. Further, legislation has been introduced in Congress to make forgiveness under certain IDR plans tax-free. Sharing information about expected forgiveness amounts could help policymakers better understand the scope of currently expected loan forgiveness and the potential tax implications of excluding forgiveness from taxable income.

We calculated currently expected IDR plan forgiveness amounts using cash flow estimates provided by Education. For our analysis, we calculated the amount of loan principal Education expects borrowers in IDR plans to repay, and the amount it expects borrowers not to repay due to forgiveness and other reasons. Our results are in figure 18.

When discussing expanded information sharing, Education officials and OMB staff agreed that there could be value in reporting additional information about IDR plan cost estimates. An Education official raised concerns about the agency’s ability to publish additional cost information, because OMB determines what is presented in the President’s budget. OMB staff agreed that such information would be too detailed for the President’s budget, but suggested that Education could provide more detailed IDR plan cost information through separate reports. Education’s strategic plan emphasizes the importance of information transparency as a tool to encourage data-driven decision-making and improve the U.S. educational system. Standards for internal control in the federal government also note that management should share quality information externally. By more thoroughly disclosing IDR plan cost information— such as total estimated costs, sensitivity analysis results, key limitations in estimates, and expected forgiveness amounts—Education could help policymakers better assess the cost implications of current IDR plan provisions and consider whether reforms are needed.

Conclusions

Policymakers need reliable budget estimates to help align federal expenditures with policy priorities. In an environment of scarce resources, quality budget information becomes all the more important, as policymakers face difficult funding decisions. While IDR plans are a promising tool to help alleviate the burden of student loan debt and reduce borrowers’ risk of default, they may be costly for the federal government. Some uncertainty is unavoidable when anticipating long- term loan costs, but we found numerous shortcomings in Education’s estimation approach and quality control practices that call into question the reliability of its budget estimates and affect the quality of information Congress has to make informed budget decisions.

Because Education administers the federal government’s largest direct loan program, it is especially important that the agency corrects its methodological weaknesses associated with estimating IDR plan costs. More specifically, until Education assesses and improves the quality of data and methods it uses to forecast borrowers’ future incomes and accounts for inflation in its estimates, its IDR plan budget estimates may be unreliable. Further, until Education obtains data needed to estimate the impact of income recertification lapses on borrower payment amounts, it will not know whether borrower payments are currently underestimated and whether adjustments are needed to avoid overstating IDR plan costs. In addition, until Education’s planned revisions to its student loan model have been completed and tested to ensure reasonableness, the agency’s IDR plan budget estimates will not reasonably reflect participation trends in IDR plans, particularly the extent to which borrowers in other repayment plans may switch into them. In the interim, Education may continue to understate IDR plan costs by billions of dollars, as past trends in estimates indicate. Without separately tracking how available IDR plans and the types of loans eligible for them perform relative to each other over time, Education’s estimates will lack the detail needed to inform policymakers’ ongoing efforts to streamline plans and better target costs.

In addition to correcting its methodological weaknesses, Education could enhance the reliability of its budget estimates by implementing more robust quality control practices. Implementing our previous recommendation to more thoroughly document and review its approach could help Education’s management identify and resolve weaknesses. More robust model testing, including more extensive sensitivity analysis, could also help Education’s management identify and mitigate problems that may reduce the reliability of its budget estimates.

Moreover, as Education works to improve the quality of its IDR plan budget estimates, it could also help policymakers better understand the scope of currently expected costs and the potential for alternative outcomes by publishing more detailed information about its estimates, such as total estimated costs, the results of sensitivity analysis, key limitations, and expected forgiveness amounts. This information could help better support efforts to assess the cost-effectiveness of IDR plans and design any needed reforms.

Recommendations for Executive Action

We recommend that the Secretary of Education take the following six actions: 1. Assess and improve, as necessary, the quality of data and methods used to forecast borrower incomes, and revise the forecasting method to account for inflation in estimates. 2. Obtain data needed to assess the impact of income recertification lapses on borrower payment amounts, and adjust estimated borrower repayment patterns as necessary. 3. Complete efforts to incorporate repayment plan switching into the agency’s redesigned student loan model, and conduct testing to help ensure that the model produces estimates that reasonably reflect trends in Income-Driven Repayment plan participation. 4. As a part of the agency’s ongoing student loan model redesign efforts, add the capability to produce separate cost estimates for each Income-Driven Repayment plan and more accurately reflect likely repayment patterns for each type of loan eligible for these plans. 5. More thoroughly test the agency’s approach to estimating Income- Driven Repayment plan costs, including by conducting more comprehensive sensitivity analysis on key assumptions and adjusting those assumptions (such as the agency’s Public Service Loan Forgiveness participation assumption) to ensure reasonableness. 6. Publish more detailed Income Driven Repayment plan cost information— beyond what is regularly provided through the President’s budget—including items such as total estimated costs, sensitivity analysis results, key limitations, and expected forgiveness amounts.

Agency Comments and Our Evaluation

We provided a draft of our report to the U.S. Department of Education (Education) for its review and comment. We provided relevant excerpts from our report to the U.S. Department of the Treasury and incorporated its technical comments as appropriate. We provided a draft of our report to the Office of Management and Budget for technical review, and did not receive technical comments in response.

Education generally agreed with our recommendations, stating that in light of growing IDR plan participation, the agency has focused efforts on improving IDR plan budget estimates. Additionally, Education said that estimating the federal student loan costs is a task it takes very seriously, and that the agency is constantly seeking to enhance and refine its models.

First, Education agreed to assess and improve its borrower income forecasts, and listed additional factors it wished to consider when determining how to incorporate inflation into its forecasts.

Second, Education agreed to attempt to obtain data to assess the impact of income recertification lapses on borrower payment amounts. Education reiterated its belief that such lapses may only have a small impact on plan costs, but did not provide data to support that view. We clarified the language in our recommendation to indicate that model adjustments should only be undertaken as needed, based on the outcome of Education’s review of relevant data.

Third, Education also agreed to incorporate repayment plan switching into its redesigned student loan model, and reiterated that efforts to incorporate this capability had begun despite challenges inherent in predicting borrower behavior.

Fourth, Education agreed to add the capability to produce separate cost estimates for each IDR plan and each eligible loan type into its redesigned student loan model. Given the concern Education raised in its letter that revising its current approach to improve loan-type estimates may not be a good use of resources, we revised our recommendation to clarify that this improvement could be undertaken as a part of the agency’s longer-term efforts to redesign its student loan model.

Fifth, Education agreed to test its approach to estimating IDR plan costs more thoroughly, including through more comprehensive sensitivity analysis. Education further explained its rationale for delaying the Public Service Loan Forgiveness (PSLF) start date in its cost model, citing preliminary evidence suggesting that few borrowers will make the 120 consecutive on-time payments necessary to receive forgiveness in the program’s first year. Education also raised concerns that using the correct start date (which we found caused estimated costs to rise by $70 million) would overstate costs. We noted Education’s rationale and concerns in our report, and responded that another solution—such as adjusting how often borrowers are assumed not to make scheduled loan payments— may be more appropriate than simply delaying the PSLF start date.

Education agreed with our sixth recommendation to publish more detailed IDR plan cost information and stated that it plans to present sensitivity analysis results and key limitations in upcoming financial reports.

Education’s comments are reproduced in appendix V.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 15 days from the report date.

At that time, we will send copies to interested congressional committees and to the U.S. Departments of Education and the Treasury and the Office of Management and Budget. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (617) 788-0534 or emreyarrasm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI.

Appendix I: Objectives, Scope, and Methodology

This appendix discusses in detail our methodology for addressing (1) the U.S. Department of Education’s (Education’s) current Income-Driven (IDR) Repayment plan budget estimates and how they have changed over time and (2) the extent to which Education’s approach to estimating IDR plan costs and its quality control practices help ensure reliable budget estimates.

To address these objectives, we reviewed relevant federal laws, regulations, and guidance on the William D. Ford Federal Direct Loan (Direct Loan) Program and IDR plans. We reviewed documentation and interviewed officials from Education about the agency’s approach to estimating costs and its quality control practices. We also interviewed officials from the Congressional Budget Office and the U.S. Department of the Treasury (Treasury) and staff at the Office of Management and Budget (OMB), as well as higher education policy experts, to discuss issues related to federal budgeting practices and estimated IDR plan costs. To answer our first objective, we analyzed and reported on data underlying Education’s annual budget estimates for the Direct Loan program. To answer our second objective, we evaluated Education’s estimation approach and conducted sensitivity analysis to determine the impact of alternative assumptions on Education’s cost estimates. We also calculated the proportion of loan dollars Education expects to forgive under IDR plans using estimated cash flow data provided by Education. These analyses are described in more detail below. To assess the reliability of Education’s budget estimates, we interviewed agency officials, reviewed related documentation, and conducted extensive electronic testing. We believe the data are reliable to report in objective one as a representation of the funding Education reports is necessary to operate the Direct Loan program, and in objective two, to illustrate the sensitivity of Education’s budget estimates to different assumptions about future loan repayment activity and to illustrate currently expected forgiveness amounts.

Objective One: Review of Current and Past IDR Plan Budget Estimates

To analyze Education’s current IDR plan budget estimates and how they have changed over time, we reviewed Education’s annual submissions to the President’s budget for fiscal years 2011 through 2017, which include estimated IDR plan loan volume and subsidy rates for Direct Loans to be issued in the year the of the budget and the two preceding fiscal years. For example, the budget submission for fiscal year 2011 included estimated IDR plan costs for loans in fiscal years 2009, 2010, and 2011. We used these budgets to identify the original IDR plan cost estimates for the 2009 through 2016 cohorts. Education did not publish subsidy cost estimates by repayment plan prior to the 2011 budget and could not easily provide the information necessary to determine original IDR plan cost estimates for previous cohorts.

We also reviewed supplemental unpublished data from Education to illustrate current IDR plan subsidy cost estimates for loans issued in fiscal years 1995 through 2017 using assumptions underlying their estimates for the President’s fiscal year 2016 and 2017 budget. We used these supplemental data, along with published data for the 2017 cohort, to calculate current total reestimated subsidy costs and subsidy income for each repayment plan, loan cohort, and loan type. We also compare original published IDR plan subsidy estimates for the 2009-2016 cohorts to the current reestimated IDR plan subsidy costs for those cohorts. We limited our comparison to these cohorts because Education did not publish subsidy cost estimates by repayment plan in earlier budgets and does not maintain information that would be needed to identify past estimates. (In appendix IV, we also compare Education’s IDR plan subsidy cost estimates for the fiscal year 2017 budget with those prepared for the fiscal year 2016 budget to illustrate how estimates changed from one budget to the next.) We compared the supplemental unpublished data to published data from the fiscal year 2016 and 2017 credit supplements to the President’s budget and interviewed Education to clarify reasons for minor discrepancies.

Objective 2: Review of Education’s Approach to Estimating IDR Plan Costs and Sensitivity Analysis of Education’s IDR Repayment Model

Evaluation of Education’s Estimation Approach

To understand and evaluate Education’s approach to estimating the cost of loans in IDR plans, we first reviewed available documentation from Education on the supplementary model Education created to estimate repayment patterns of loans in IDR plans (referred to as the IDR plan repayment model in this appendix). We also reviewed documentation on Education’s student loan model, which uses information from the IDR repayment model and other assumptions to calculate total subsidy costs. (See appendix II for detailed information on how Education estimates IDR costs using these models.) This documentation provided limited details regarding the steps of Education’s IDR repayment model or how assumptions were operationalized and programmed in the model.

Given the limited documentation available regarding Education’s IDR plan repayment model, we reviewed the computer programs and datasets used in the model. Education provided us with SAS program files and data input files used in the model. The data input files contained the sample of Direct Loan borrowers Education used in its analysis as well as estimated historical incomes of those borrowers provided by Treasury. (See appendix III for more information on these historical income estimates.) The SAS program files implementing the model forecasted those borrowers’ future incomes and scheduled IDR plan payment amounts, as well as forecasted events that would lead to non-payment, such as default, death or disability, prepayment of loans through consolidation, or forgiveness of loans through the Public Service Loan Forgiveness (PSLF) program. To get further clarification on the documentation, data, and computer programs provided, we interviewed Education officials who created and manage the IDR plan repayment model and the overall student loan model, which is used to calculate subsidy costs for all Direct Loans.

We assessed the IDR plan repayment model’s major assumptions for reasonableness and evaluated them against federal guidance for estimating subsidy costs developed by the Federal Accounting Standards Advisory Board. We evaluated methods used in the model, particularly Education’s approach to forecasting borrower incomes, against this guidance and accepted practices in statistics and the social sciences. We also assessed whether the model appropriately replicated IDR plan program rules. Finally, we conducted an in-depth review of the Treasury- created estimated historical income data used in Education’s approach. We assessed the reasonableness of the data by conducting electronic testing and producing summary statistics, which we asked Treasury to compare to the actual taxpayer data upon which its estimates were based. We reviewed related documentation from Treasury about the estimation process, and interviewed Treasury officials to clarify factual details and obtain their views on the process. (See appendix III for more information on our review of these data and Education’s subsequent forecasting approach.)

IDR Plan Repayment Model Sensitivity Analysis

Based on our detailed review of the assumptions, methods, and data used in the IDR plan repayment model, we identified two separate areas for testing the sensitivity of Education’s IDR plan cost estimates to changes in assumptions. First, we tested the effects of inflation on income projections and poverty guidelines, both of which are used to estimate borrower payment amounts. We adjusted borrower incomes and poverty guidelines for inflation due to the exclusion of inflation from Education’s current model and the results of a prior Education analysis showing that cost estimates were sensitive to changes in borrower incomes. Second, we tested Education’s assumption about PSLF participation and the year borrowers would be first eligible for forgiveness under the program. We focused on PSLF participation because actual participation is not yet known for this program and Education assumed that any borrower they estimated to be eligible for PSLF would choose to participate. We carried out each sensitivity analysis by rewriting relevant portions of the existing SAS computer programs that Education developed to implement the IDR plan repayment model.

To conduct these analyses, we first produced baseline cash flow estimates using the existing programs we received from Education. We sought to produce baseline estimates that were identical to those from Education’s existing model. The baseline replication ensured that the new model assumptions, rather than different versions of programs or input data, were solely responsible for any changes in the estimates. The replication process included selecting random samples of the data files and using the SAS Compare procedure to detect any differences in observations and variables. We interviewed Education officials to confirm the sequence and versions of programs and to establish our final baseline file.

After producing baseline estimates, we wrote two new sets of SAS program files to implement each new assumption and produce new cash flow output for each analysis. The final output data, which we sent to Education to produce subsidy rates, consisted of cash flows summed across all borrowers in repayment within each fiscal year and within loan population type (non-consolidated loans, loans consolidated from default, and loans consolidated not from default). We provided this output to Education officials, who uploaded and ran the new estimates through the larger student loan model. Education officials provided revised subsidy rates for each loan type and origination cohort, reflecting the new IDR plan cash flows under our alternative assumptions. For each sensitivity analysis, we compared the baseline and revised IDR cash flows and subsidy cost estimates and calculated the percent change.

We tested Education’s assumptions regarding borrower participation and the first year that borrowers are eligible for PSLF. Education estimates borrower eligibility for PSLF using survey data that may not be representative of borrowers in newer IDR plans. In addition, Education assumes that 100 percent of borrowers who are estimated to be eligible for PSLF will choose to participate after making 120 payments in a qualifying repayment plan. Lastly, Education assumes that no borrower will become eligible to benefit from PSLF until a year after the program is scheduled to begin. To assess the impact of altering these three assumptions, we increased and decreased the estimated percentage of borrowers eligible and participating in PSLF by 10 and 5 percentage points, and moved up the PSLF start date by a full year.

Adjusting Projected Incomes and Poverty Guidelines for Inflation We tested the extent to which cost estimates were sensitive to adjusting incomes and poverty guidelines for inflation for future years after 2013. As described in appendix II, Education forecasts borrowers’ incomes by substituting the historical incomes of borrowers with similar characteristics, but does it not adjust these projected incomes for inflation. Education also uses 2013 poverty guideline data for future years, with no inflation adjustment. To implement this adjustment, we obtained inflation factors from OMB for all future repayment years, and inflated Education’s forecasted borrower incomes and poverty guidelines into the appropriate year’s dollar units. Specifically, we applied adjustment factors to the 2013 dollar amounts to inflate them to each future year’s dollar units. We then applied the existing repayment model using the inflated incomes and poverty guidelines as input, without altering any additional model assumptions or calculations.

Calculating IDR Plan Forgiveness Amounts

To calculate expected forgiveness amounts for loans entering repayment in fiscal years 1995 through 2017, we analyzed cash flow data from Education, which provided detailed information on the amount of loan principal expected to be paid and not repaid. First, we determined the overall amount of loan principal in IDR plans estimated not to be repaid for any reason, as Education recommended. We did this by subtracting the amount of principal expected to be repaid from the total volume of loans disbursed to borrowers. The remaining amount represented loan principal estimated not to be repaid. We then subtracted the amount of loan principal estimated to be discharged due to a borrower’s death or disability. We attributed the remaining balance of unpaid principal to loan forgiveness under IDR plans and PSLF. Because Education expects to recover all defaulted loan principal through the collections process, loan defaults did not contribute to total non-payment of loan principal.

Evaluation of Education’s Quality Control and Information Sharing Practices

We assessed Education’s quality control practices by reviewing relevant documentation and interviewing officials in the office responsible for developing and managing the estimates. We evaluated Education’s practices against federal guidance related to estimating subsidy costs and standards for internal control in the federal government. We also assessed Education’s information sharing against standards for internal control in the federal government, and Education’s strategic plan.

We conducted this performance audit from March 2015 to November 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Description of Education’s Approach to Estimating Costs of Income- Driven Repayment Plans

Education’s Approach to Estimating Costs of Loans in Income-Driven Repayment Plans

Based on our review of Education’s computer programs, model documentation, and interviews with agency officials, we confirmed that Education estimates subsidy costs for loans in Income-Driven Repayment (IDR) plans in the following way. First, Education estimates how many loan dollars will enter IDR plans from each loan cohort. Second, Education estimates repayment patterns for those loans over time. It performs this first task within its larger student loan model that calculates cash flows for cohorts of loans and incorporates various assumptions about the future. Education addresses the second task inside a supplementary microsimulation model for estimating IDR plan repayment patterns—referred to as the IDR plan repayment model in this appendix— that was designed to predict the repayment behavior of individual borrowers from a sample of borrowers with loans in IDR plans. Through interviews, Education officials stated that they combine the resulting pieces of information in their larger student loan model to generate subsidy cost estimates.

Estimating How Many Loan Dollars Will Enter IDR Plans

Education estimates the percentage of loans in each cohort that will enter each repayment plan—Standard, Extended, Graduated, and IDR—inside its student loan model. According to Education’s model documentation and follow-up information from agency officials, Education based its cost estimates reported in the President’s fiscal year 2017 budget on a random sample of loans drawn from its National Student Loan Data System in January 2015. For loans issued after September 2014, Education applied repayment plan participation rates from a past cohort. For Consolidation loans, Education used 2014 cohort data because borrowers generally begin repaying those loans immediately. For non- consolidated loans, which generally do not enter repayment for several years while borrowers are in school, Education used participation rates from the 2011 cohort. IDR plan participation rates for the 2015 through 2017 cohorts were adjusted upward in comparison to the 2014 cohort to account for expanded eligibility for two newer IDR plans (Pay As You Earn and Revised Pay As You Earn). For the fiscal year 2017 budget, this upward adjustment ranged from 1.4 to 6.2 percent, depending on the cohort and type of loan. Education officials stated that they then apply the percentage of loans assumed to enter IDR plans to the total dollar value of loans originated (or loan volume) in each loan cohort.

Estimating Repayment Patterns for Loans in IDR Plans

Education uses a separate IDR plan repayment model to forecast cash flows, which we refer to as repayment patterns, of loans in IDR plans based on a sample of borrowers with loans in repayment as of September 2013. This random sample of borrowers was also drawn from the National Student Loan Database and reflected all loan activity through the end of fiscal year 2013, including but not limited to, the amount borrowed, loan type, and repayment plan. From this sample, Education selected all borrowers who had already begun repaying their loans under an IDR plan by September 2013. For the purpose of modeling future loan cohorts, Education assumes all borrowers entering repayment after September 2013 will have the same characteristics as borrowers who entered repayment in 2013.

Education then estimates how much each of these borrowers will owe on their loans over a 31-year span, based on the borrower’s estimated adjusted gross income (income) and family size for each year in the repayment period and the rules of the IDR plan selected. Education used a two-step process to forecast borrower income and family size: 1. To forecast borrowers’ future incomes, Education first worked with Treasury to estimate past incomes, filing status, and family sizes of the sample of borrowers who had entered repayment by the end of fiscal year 2013. Treasury developed these estimates because Education does not collect tax data on all borrowers. Treasury collects this information for all U.S. tax filers, but did not share actual data from these borrowers’ tax filings due to privacy restrictions. Instead, it created a tax file that contained substituted, or “imputed,” information based on borrower characteristics including age, gender, loan balance, dependency status and family income. Treasury first estimated if a borrower would file taxes in a given year. For each borrower estimated to file taxes, it then imputed estimated nominal incomes and number of tax exemptions (approximating family size) for each of the borrower’s repayment years that occurred in tax years 1996 through 2013. For example, borrowers who entered repayment in 1996 would have 18 years of imputed incomes while borrowers entering repayment in 2000 would have 14 years. (See Evaluating Income Data Used in Education’s Approach in appendix III for more information on Treasury’s methodology to estimate borrower incomes and our assessment of error associated with its approach). 2. To forecast future incomes of each borrower in its sample from 2014 through the end of the borrower’s repayment period (up to 31 years in the future), Education first converted the estimated historical income data from Treasury from calendar years into “repayment years.” A borrower who began repaying his loan in calendar year 2000 would have estimated historical income data covering repayment years 1 through 14 (formerly calendar years 2000 through 2013). To forecast that borrower’s future income in repayment year 15, Education first matched the borrower with a set of borrowers with similar characteristics. Education then randomly selected a borrower from this matched set of borrowers and substituted the nominal historical income observation from the same repayment year. It repeated this step for each subsequent year of the borrower’s maximum repayment period, choosing a different borrower’s nominal historical income observation in each year. Because Education matched the borrowers in their file with Treasury’s based on their repayment year (as opposed to calendar year), the nominal historical income values used in the forecasts could come from various non-sequential calendar years. (See Evaluating Income Data Used in Education’s Approach in appendix III for more information on Education’s methodology and the error associated with its approach).

Once Education forecasted incomes and family size for each borrower in the sample’s entire repayment period, Education then applied the rules of the borrower’s selected IDR plan to calculate the amount the borrower would owe over time. For instance, a borrower in the New Income-Based Repayment plan would pay 10 percent of her discretionary income for 20 years, whereas a borrower in the Income-Contingent Repayment plan would pay 20 percent of her discretionary income for 25 years. For borrowers in the file who had not yet selected an IDR plan as September 2013 (i.e., those estimated to enter repayment in 2014 or later), Education selects a plan for them based on assumptions about borrower behavior. For borrowers in the Income-Contingent Repayment plan, the IDR plan repayment model annually reevaluates whether they will switch into the Income-Based Repayment plan, and does so if the borrower’s monthly payment amount would be lowered by at least $50 by switching into the Income-Based Repayment plan. The IDR plan repayment model switches a borrower into the Revised Pay As You Earn Plan if that borrower is not eligible for the Pay As You Earn or New Income-Based Repayment plans and if the borrower saw his or her payment fall by $50 compared to what would be owed under the Income-Based Repayment plan. Borrowers were assumed to choose Revised Pay As You Earn if the present value of the payments with Revised Pay As You Earn were lower than the present value of payments without Revised Pay As You Earn using a 30 percent discount rate. Education used a high discount rate because borrowers would likely place much less weight on the higher payments that would be likely to occur toward the end of the repayment period. Borrowers already in IDR repayment were assumed to choose Revised Pay As You Earn in the first year their payments fell by $50 a month or more. Borrowers who had not yet chosen an IDR plan were assumed to choose Revised Pay As You Earn if their payments would be lower in the first year. Borrowers stay in an IDR plan for their entire repayment period, even if their income rises beyond the point at which they would qualify if they were applying in that year, in order to calculate possible loan forgiveness amounts.

The IDR plan repayment model also includes predictions about when borrowers will delay repaying their loans (through deferment and forbearance); when they will fail to repay their loans (due to default, death, and disability); when they will prepay their loans through consolidation; and when their loan balances will be forgiven due to participation in the Public Service Loan Forgiveness program or at the end of their IDR plan’s full repayment term.

The IDR plan repayment model’s final output consists of cash flows received (broken out by principal and interest) and foregone (such as through default, death, and disability) for each of the 31 years of repayment. These cash flows are summed across all borrowers who enter repayment in the same year for three different groups: (1) borrowers with Subsidized Stafford, Unsubsidized Stafford, and Graduate PLUS loans, (2) borrowers who defaulted and then consolidated their loans, and (3) borrowers who consolidated their loans without defaulting.

Exporting Cash Flows to Student Loan Model

Cash flows from the IDR plan repayment model are then exported to the larger student loan model. According to Education officials, the student loan model allocates these cash flows, which are organized by the year in which loans enter repayment, back to the appropriate loan origination cohorts using an assumption about the rate at which loans originated in a given year will enter repayment. Education assumes that all loans being repaid in IDR plans in a particular loan origination cohort will have the same cash flow patterns as loans in the sample used in the IDR plan repayment model.

Calculating Subsidy Costs

The student loan model discounts estimated cash flows to present value using the Office of Management and Budget’s credit subsidy calculator tool to determine the subsidy cost. The subsidy rate is determined by taking the ratio of the subsidy cost to the volume of loan obligations estimated to be made in that year.

Appendix III: Evaluation of Income Data Used in Education’s Approach

Income Data Used in Education’s Approach

The U.S. Department of Education’s (Education’s) supplementary model for estimating Income-Driven Repayment (IDR) plan repayment patterns—referred to as the IDR plan repayment model—blends statistical analysis and assumptions about the future behavior of borrowers.

General Use of Historical Income Estimates

Education uses data on borrowers’ historical incomes to estimate their incomes in future repayment years. According to Education staff, the previous version of the agency’s IDR plan repayment model used data from the Current Population Survey, a general population sample survey administered by the U.S. Census Bureau, which the agency matched to a sample of borrower data. Education staff told us they searched for a different source of income data beginning in 2013, due to the relatively short 2-year Current Population Survey panel length. The short panel required Education to combine incomes from different individuals and Current Population Survey samples to project over enough repayment years.

For Education’s current IDR plan repayment model, developed in 2015, agency staff sought income and other data from federal income tax returns, as collected by the U.S. Department of the Treasury (Treasury). Taxpayer data offered the potential for more accurate data than matched Current Population Survey data, which only covered a sample of borrowers for relatively short 2-year period. (According to Education officials, the Current Population Survey does not contain data on student borrowing so the prior model had to assume that borrowers and non- borrowers had the same income patterns.) Despite the expected benefits of using actual taxpayer data, Treasury staff indicated that rules concerning taxpayer privacy prevented them from providing data on actual borrower incomes directly.

Education staff said they initially hoped to receive data from Treasury that matched borrowers’ actual incomes as closely as possible, perhaps with a random distortion to protect taxpayer privacy. Staff mentioned Education’s National Postsecondary Student Aid Study restricted-use file as an example of a similar dataset. However, Treasury’s chosen approach involved imputing borrower income categories. Education staff then requested that Treasury convert these categorical values into dollar- scaled incomes for use in Education’s IDR plan repayment model.

Methodology for Assembling Historical Income Estimates

Based on our review of documentation from Treasury and Education, to assemble the data prior to imputation, Treasury matched a sample of borrower data that Education drew from the National Student Loan Data System containing loan information from September 2013 to their tax return data for filing years 1996 through 2013. Treasury assumed that borrowers did not file returns if they did not have matching tax return data for a given year. Key tax variables matched to the file for loan modeling purposes included adjusted gross income, number of exemptions, and filing status, among others. The final matched dataset included observations for approximately 1.3 million borrowers in each of 18 tax years.

After matching the files, Treasury used a data mining algorithm, known as “graphical models,” to create an imputed version of the matched data. According to Education staff, they asked Treasury to provide an imputed dataset that resembled the actual data as closely as possible, for all of the tax variables joined to their borrower records. Education staff said they expected incomes to be accurate within categories but having random distortion to preserve taxpayer privacy. Treasury staff told us that they lacked the time and prior experience with Education’s data to have a pre-existing model to meet these specifications. Instead, Treasury used graphical methods to automatically identify a model that best fit the joint distribution of the data across several variables and allowed for the simulation of new imputed data. Treasury staff said that this approach was simpler than what they might have done given more time, but it is unclear whether greater complexity in the model would have yielded better results.

Based on our review of documentation and interviews with Treasury staff, Treasury’s exact method of imputation had several steps. First, Treasury rescaled all variables from their natural scales into discrete categories, which primarily affected borrower incomes, which are naturally measured in continuous nominal dollars. Using categories of incomes rather than the continuous dollar scale allowed Treasury to develop a model using graphical methods that required less computing power. Second, Treasury used graphical methods to identify the relationships (or dependencies) among the borrower and tax variables in the matched data, in the form of multivariate crosstabulations. The model first estimated the probability that a borrower would file a tax return in a given year, and then modeled the joint distribution of the data, given that the first-stage model estimated that a borrower would file a return. After estimating these crosstabulations, Treasury created a single imputed dataset by drawing random variates from the fitted joint distribution of the data, in order to replace records in the actual dataset.

When imputing incomes, Treasury staff told us they took an additional step to transform the imputed income variables from a categorical to continuous scale by drawing random dollar values from probability distributions. For borrowers with incomes imputed in the lowest and highest categories, Treasury simulated continuous incomes by drawing from normal or log-normal distributions with moments set to their sample values. For borrowers in all other categories, Treasury drew independent random variates from uniform distributions with support on the range of each imputed category.

According to Treasury staff, they constrained the imputation model to replicate some of the longitudinal structure in the tax data over time within borrowers. The model imputed a borrower’s income in the current year based on the borrower’s one-year lagged income, which generally ensures that the imputed data recreates the correlation between incomes in adjacent years (i.e., the first-order autocorrelation). In addition, Treasury staff said that the imputation variables were stratified by year, in order to allow the conditional distributions to vary over time.

Error in Historical Income Estimates

Any imputed data will have imputation error, the amount of which depends on the predictive power of the model or method used to create them. According to statistical theory, imputation models can produce imputed datasets that are systematically biased, in the sense that the imputed distribution of the data does not resemble the actual distribution across many imputations. Imputation can also produce imputed datasets that are unbiased but have a high degree of imputation error. (More specifically, the variance of a model’s posterior predictive distribution can be large.) In these cases, the imputed distribution of the data will resemble the actual distribution across many imputations, but the imputed distribution in any one sample of imputed data, often one random simulation, may be quite different. Measuring error can involve calculating confidence intervals. A larger confidence interval relative to the estimate would suggest imputed data that are more prone to error. A user of imputed data typically would consider the size of a confidence interval as one criterion when assessing whether imputed data are sufficiently precise for a specific application, along with the imputation model’s potential bias and the user’s tolerance for error.

These two types of imputation error can affect the analysis of imputed data. Ordinary methods of statistical analysis generally assume that variables (like borrower incomes) are measured without error—an assumption that is clearly not valid when analyzing imputed data. Analyses that do not account for imputation error can produce estimates that are biased or more or less precise than ordinary statistical theory would imply, depending on the nature of the analysis.

To address these features of imputed data, generally accepted statistical practices suggest a number of methods for the analysis of imputed data. One common method uses “multiple imputation” to impute the data several times, producing “implicate datasets.” Implicates are randomly generated copies of the imputed data, produced by the same imputation model. The imputed data will vary randomly across implicates, depending on the nature and precision of the imputation method, because most imputation models have a partially random or probabilistic structure. By assessing the degree to which analytical results vary across implicates, analysts can incorporate the error of imputation when estimating the error of their estimates more generally. Imputation error “propagates” into other measures of precision, such as sampling error.

As an alternative or complement to assessing imputation error directly, the statistical literature recommends that analysts use imputed data as preliminary information, prior to replicating their analyses using actual data. This approach applies to situations in which an analyst may not access the underlying data, but can provide computer code or algorithms to another analyst who may access the actual data and replicate the work.

Education staff told us they did not request information that would have allowed them to assess how imputation error would have affected its cost estimates, nor did they provide their computer code to a Treasury analyst who had access to the data to replicate their work. Education staff conducted an informal assessment of the quality of the borrower income data by reviewing correlations between incomes and key factors like education and borrowing levels. They did not use more formal methods to assess and address imputation error in their estimates, such as those discussed above. In addition, Education staff did not thoroughly document and evaluate the imputation methods that Treasury used, nor did they request evidence of an adequate model fit. Instead, according to Education staff, Treasury provided Education with limited documentation which included a broad overview of the imputation approach. Treasury developed detailed documentation of the imputation model and error at our request, several months after Education had accepted the final imputed data. Treasury staff reported having computing and other resource constraints that affected their choice of models and methods. These constraints would have affected Treasury’s capacity to run their model multiple times and produce multiple implicate datasets.

Comparison of Estimated and Actual Income Data

Through our review of summary documentation and limited descriptive and graphical analysis from Treasury, we found indications of imputation error that Education may not find acceptable for its purposes. This error relates to the imputation of incomes in the lowest and highest categories as well as the longitudinal structure of borrower incomes over time. This error warrants further evaluation by Education, given that the agency sought income data that would resemble borrowers’ actual, historical incomes as closely as possible, including accurate longitudinal profiles of incomes. Accurate longitudinal profiles of income are important, because Education’s IDR plan repayment model includes a number of calculations at the borrower level, such as specific payment amounts and borrowers’ eligibility for specific IDR plans, which use the sequence of data within borrowers over time as inputs and are re-calculated each year over the repayment period.

Treasury staff told us they did not seek to impute incomes on the dollar scale. Rather, staff imputed income categories, and then evaluated model fit using the categorical distributions of the imputed and actual income data. After developing and validating the categorical imputation model, Treasury provided a simple transformation of the income categories into dollar values, using the secondary imputation methods we describe above. Treasury staff described this aspect of the imputation as a practical solution to meet needs that Education clarified after Treasury had developed the imputation method and its goals. Because dollar- scaled incomes were not originally specified, Treasury staff told us they did not assess the fit of the imputed dollar-scaled incomes to the joint distribution of the data.

Error in Dollar-Scaled Incomes in Highest and Lowest Categories

Treasury’s comparison of the imputed and actual income data indicates the imputed categorical data generally resembled the actual data, but its secondary step to produce dollar scaled data introduced additional error, particularly for observations in the highest and lowest income categories. Treasury provided us with tabulations and plots of the imputed and actual data, along with predictive p-values. The frequency statistics showed that the marginal and joint distributions of key variables were generally similar in the imputed and actual data for the categorical data, but that the secondary imputation of dollar scale incomes produced additional error for borrowers in the lowest and highest income categories. As the Treasury-provided summary statistics in table 1 show, the mean imputed income was about 2.1 times smaller than the mean actual income among borrowers who earned $12,000 or less. The imputed mean was about 1.9 times larger than the actual mean for borrowers in the highest income category. Treasury officials agreed that imputation error may be greater in the lowest and highest categories, but speculated that the error may not be practically consequential for the calculation of income-based loan payments. However, because these data form the foundation of numerous individual-level sequential calculations that determine what borrowers are estimated to repay to the government, error associated with the data should be measured and its effect on budget estimates should be assessed.

By design, Treasury’s imputation model ensured that the correlation between incomes in adjacent years was similar in the imputed and actual data. Despite this important constraint, the model did not seek to accurately impute complete, realistic profiles of dollar-scaled incomes over time within the same borrower for all observed years. Consequently, the imputed profiles of incomes over time within borrowers were not designed to ensure that they resemble those observed in the actual data at the individual level. Treasury staff confirmed this feature of the imputed data.

Our limited exploratory analysis of the imputed dollar-scaled income data revealed patterns consistent with these features of the imputation. Incomes were less strongly correlated between adjacent years in the imputed data than in the actual data, based on statistics that Treasury staff provided. Specifically, let Var(AGIt | ADIt-1, AGIt-2, … , AGIt-k) denote the marginal variance of incomes at time t, conditional on k previous values. In the imputed data, the limited evidence available to us suggests that the estimated conditional variance in the imputed data may exceed the actual variance in the population of student loan borrowers. The Pearson correlation between incomes in the current and previous year, truncated to the interval of positive incomes less than $1 million, was 0.84 in the actual data and 0.58 in the imputed data. In other words, one measure of the year-to-year instability of incomes was about 44 percent larger than in the actual data. In addition, the absolute value of incomes changed between adjacent years by 52 percent in the actual data but 75 percent in the imputed data. Consistent with these aggregate statistics, figure 19 shows how imputed incomes vary by large amounts from year to year within the same 10 randomly selected borrowers presented previously in the report, this time with their tax filing status indicated. Figure 20 illustrates how imputed incomes vary by large amounts from year to year within the same 60 randomly selected borrowers.

The secondary imputation of incomes in dollars may explain the patterns above. Simulating dollar-scaled incomes from a set of uniform, normal, or log-normal distributions would have added some amount of approximation error, potentially inflating the conditional variance above in the imputed data. The degree of error would depend on how strongly the actual income distribution within each category diverged from the assumed distribution (e.g., its nonlinearity when simulated as uniform). Consistent with this explanation, Treasury staff reported nearly identical Pearson correlations between current and 1-year lagged categorical incomes in the imputed and actual data, at 0.69 and 0.67, respectively. The approximation error for continuous incomes may have compounded across years when Treasury staff independently simulated incomes for the same borrowers in adjacent years, without constraining the imputed distribution to preserve the potential dependency of incomes across second-order lags and higher.

Treasury staff emphasized that adjusted gross income can be more volatile over time than other measures of income. Adjusted gross income includes gross wages, business income, and asset income, among other sources, as well as certain deductions and credits. According to Treasury staff, adjusted gross income can vary more substantially over time than other sources of income, such as wages, and that such variation is common among upper-income filers. However, we found that the absolute value of imputed adjusted gross incomes varied between adjacent years by 15 to 77 percent for the middle 50 percent of the sample of borrowers with adjusted gross incomes above 0 and less than $400,000. The widespread nature of the volatility conflicts with an explanation that emphasizes volatile sources of income, deductions, and credits among borrowers with high incomes.

We did not receive sufficient information to fully evaluate the nature and extent of imputation error in the Treasury data, and how it would affect Education’s IDR plan cost estimates. For instance, we did not receive Treasury’s computer code or the actual tax data. Instead, Treasury staff described the analysis in interviews and written briefing slides, as well as a 7-page summary of the analysis that they previously provided to Education. The correlations and proportional change statistics above are limited in their ability to fully describe complete profiles of incomes at the individual borrower level and their dependence over time, because they describe linear associations only between data from adjacent years. Additional analysis, with full access to the imputation model code and tax data and a thorough assessment of the longitudinal structure of incomes within borrowers over time, is necessary to confirm the imputation error suggested by the limited evidence we obtained from Treasury.

Forecasting Future Borrower Incomes and Characteristics

The IDR plan repayment model uses the imputed data on borrower incomes and other characteristics to forecast these data for future repayment years that have not yet occurred. The model uses a different method of imputation, known as the “hot-deck,” to make these forecasts. Below, we describe this method in detail and evaluate it against generally accepted statistical practices.

Education’s Process of Forecasting Borrower Incomes and Other Characteristics

According to our review of the imputed data that Education received from Treasury, the data could span a variable portion of each borrower’s repayment history. Education received imputed data for tax years 1996 through 2013. For a borrower who entered repayment in 1996, this period would span the entire historical repayment period through 2013, but it would not cover future years when the borrower may still be in repayment. Conversely, for a borrower who entered repayment in 1986 and repaid all debt in 2000, the data would span the last four historical years but not the first 10. Many other types of overlap are possible.

The repayment model uses these historical data and hot-deck methods to impute or forecast data for repayment years that have not yet occurred. The hot-deck is a general purpose method of imputation, which statistical organizations commonly use to impute missing survey data. For a set of records needing imputation, hot-deck methods use a set of covariates to identify one or more records in the data with observed values on all variables, which is similar to the record needing imputation. The method then substitutes the observed values for the values needing imputation, often using random selection among the donor records when multiple donor records are available.

Once the repayment model forecasts data for unobserved repayment years, it treats them as known, observed data. The repayment model uses the forecasted data as input for the second stage of modeling, which applies various assumptions about how borrowers will repay their loans over time. The second stage modeling incorporates neither the error associated with Treasury’s imputation of the matched tax and loan data nor the error associated with Education’s hot-deck forecasting.

Education’s Forecasting Process and Prediction Error

As mentioned previously, any method of imputing or forecasting unknown observations will have error associated with its predictions. Although the nature of this error depends on the method and data, generally accepted statistical practices typically recommend quantifying the error and incorporating it into subsequent analysis of the predictions. Education’s method of forecasting borrower incomes does not quantify the error associated with the method or incorporate it into subsequent analyses.

Education’s analysis seeks to forecast the values of several variables, most notably income, for a set of borrowers over up to 31 future repayment years. One could view this as either a longitudinal econometric modeling problem or a general purpose imputation of missing data. Using either approach, accepted statistical practice involves quantifying and propagating the error that is inherently associated with prediction.

An econometric approach would use an explicit statistical model for how the forecasted variable depends on several other variables (or covariates). Additional assumptions would describe how the imputed variable varies over time, either through covariates (such as time indicators) or assumptions about the variable’s random fluctuation around its long-term mean (such as an error term with an autoregressive order 1 structure). These model assumptions provide explicit formulas to predict future values of the variable and to quantify the likely error of prediction. The latter formulas for prediction error (or the posterior predictive distribution) can allow analysts to propagate the error of prediction into subsequent analyses of those predictions.

An alternative approach would use more advanced methods of imputing missing data, such as multiple, maximum likelihood, or expectation- maximization imputation. These methods assume an explicit probability model for the joint distribution of the data, with parameters that can be estimated from the data. Analysts can use various methods for drawing from the fitted data distribution, in order to generate multiple implicate datasets, as discussed above. This allows for analysts to quantify and propagate error across subsequent analyses of the imputed data.

Education’s application of the hot-deck method does not follow these general statistical principles. The method imputes the future values of all unknown data, using donor cases that are similar on a set of covariates, such as gender and highest educational program level. After making predictions, the method does not quantify the prediction error of the estimates, using in-sample statistics such as mean-squared error, misclassification rates, deviance statistics, predictive p-values, or the estimated variance of the posterior predictive distribution.

Once the repayment model forecasts the income data for future years, it assumes that the estimates have zero prediction error associated with them, or equivalently, that the error does not affect the repayment model’s loan cost estimates. Since any applied method of forecasting or imputation produces error, and Education’s IDR plan cost estimates are highly sensitive to changes in borrower income forecasts, it is important for Education to measure this error and determine its ultimate impact on IDR plan cost estimates.

Moreover, the IDR repayment model uses source data that have their own unquantified imputation error from Treasury’s imputation. These two sources of error—Treasury’s imputation and Education’s forecast—may interact and combine in ways that further increase the bias and imprecision of Education’s loan cost estimates. The presence of multiple forms of error, at different stages of analysis, emphasizes the importance of propagating all sources of error through the entire analysis, or else eliminating imputation error from the imputed data by using actual observations. At a minimum, Education should acknowledge the presence of imputation error and identify how it might affect estimates from the repayment model. Such acknowledgements would provide more transparent information to users of its estimates, compared to point estimates that do not disclose the limitations of the source data.

Statistical organizations accept the need for users of imputed data, such as Education, to quantify and assess the effects of imputation error, despite their release of public data that have been imputed. The U.S. Census Bureau warns users that methods of estimating sampling error will underestimate total error when data have been imputed. In recent years, the Census has generated imputed data for several surveys, the Survey of Income and Program Participation and Longitudinal Business Database, but has warned that analyzing imputed data without necessary corrections may understate the variance of estimates. This guidance to data users is consistent with the criteria discussed above, which recommend quantifying and propagating imputation error, despite statistical agencies’ widespread use of imputed data in public data products.

Appendix IV: Supplemental Direct Loan Subsidy Cost Data

The following tables include a summary of available loan cohort data underlying the U.S. Department of Education’s (Education’s) submissions to the President’s fiscal year 2016 and fiscal year 2017 budgets. These tables are provided to illustrate how Education’s estimates of IDR plan costs shifted over the past two President’s budgets. Some of the differences are attributable to the change in Education’s methodology for estimating IDR plan costs, which was implemented for the President’s fiscal year 2017 budget and is described in this report. Other differences are due to the policy assumptions in place when the budgets were developed. Specifically, for the fiscal year 2016 budget, Education used provisional policies for its newest IDR plan that were under negotiation. Estimates prepared for both the President’s fiscal year 2016 and fiscal year 2017 budgets included legislative proposals affecting new borrowers. Finally, the fiscal year 2017 budget estimates include increased costs associated with the addition of the 2017 loan cohort, as well as the updated current reestimated costs of older cohorts.

Appendix V: Comments from the U.S. Department of Education

Appendix VI: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Kris Nguyen (Assistant Director), Ellen Phelps Ranen (Analyst-in-Charge), Rachel Beers, James Bennett, John Karikari, Karissa Robie, Amber Sinclair, and Jeff Tessin made key contributions to this report. Additional assistance was provided by Deborah Bland, Jessica Botsford, Russ Burnett, Marcia Carlsen, David Chrisinger, Cole Haase, Carol Henn, Susan J. Irving, Marissa Jones, Sheila McCoy, Erin McLaughlin, Jeffrey G. Miller, Andrew Nelson, Jason Palmer, Jessica Rider, Amrita Sen, Brian Schwartz, Michelle St. Pierre, Adam Wendel, Charlie Willson, and Rebecca Woiwode.