Federal Student Loans: Education Needs to Improve Its Income-Driven Repayment Plan Budget Estimates
Published: 2016-11-15T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: As of June 2016, 24 percent of Direct Loan borrowers repaying their loans (or 5.3 million borrowers) were doing so in IDR plans, compared to 10 percent in June 2013. Education expects these plans to have costs to the government. GAO was asked to review Education's IDR plan budget estimates and estimation methodology.
This report examines: (1) current IDR plan budget estimates and how those estimates have changed over time, and (2) the extent to which Education's approach to estimating costs and quality control practices help ensure reliable estimates. GAO analyzed published and unpublished budget data covering Direct Loans made from fiscal years 1995 through 2015 and estimated to be made in 2016 and 2017; analyzed and tested Education's computer code used to estimate IDR plan costs; reviewed documentation related to Education's estimation approach; and interviewed officials at Education and other federal agencies.
section_title: What GAO Found
paragraphs: For the fiscal year 2017 budget, the U.S. Department of Education (Education) estimates that all federally issued Direct Loans in Income-Driven Repayment (IDR) plans will have government costs of $74 billion, higher than previous budget estimates. IDR plans are designed to help ease student debt burden by setting loan payments as a percentage of borrower income, extending repayment periods from the standard 10 years to up to 25 years, and forgiving remaining balances at the end of that period. While actual costs cannot be known until borrowers repay their loans, GAO found that current IDR plan budget estimates are more than double what was originally expected for loans made in fiscal years 2009 through 2016 (the only years for which original estimates are available). This growth is largely due to the rising volume of loans in IDR plans.
Education's approach to estimating IDR plan costs and quality control practices do not ensure reliable budget estimates. Weaknesses in this approach may cause costs to be over- or understated by billions of dollars. For instance:
Education assumes that borrowers' incomes will not grow with inflation even though federal guidelines for estimating loan costs state that estimates should account for relevant economic factors. GAO tested this assumption by incorporating inflation into income forecasts, and found that estimated costs fell by over $17 billion.
Education also assumes no borrowers will switch into or out of IDR plans in the future despite participation growth that has led budget estimates to more than double from $25 to $53 billion for loans made in recent fiscal years. Predicting plan switching would be advisable per federal guidance on estimating loan costs. Education has begun developing a revised model with this capability, but this model is not complete and it is not yet clear when or how well it will reflect IDR plan participation trends.
Insufficient quality controls contributed to issues GAO identified. For instance:
Education tested only one assumption for reasonableness, and did so at the request of others, although such testing is recommended in federal guidance on estimating loan costs. Without further model testing, Education's estimates may be based on unreasonable assumptions.
Due to growing IDR plan popularity, improving Education's estimation approach is especially important. Until that happens, IDR plan budget estimates will remain in question, and Congress's ability to make informed decisions may be affected.
section_title: What GAO Recommends
paragraphs: GAO is making six recommendations to Education to improve the quality of its IDR plan budget estimates. These include adjusting borrower income forecasts for inflation, completing planned model revisions and ensuring that they generate reasonable predictions of participation trends, and testing key assumptions. Education generally agreed with GAO's recommendations and noted actions it would take to address them.

section_title: Letter
paragraphs: Currently, over 30 million student loan borrowers hold more than $900  billion in William D. Ford Federal Direct Loans (Direct Loans), and there  are indications that some face difficulties repaying their loans. For  instance, almost 20 percent of Direct Loan borrowers were delinquent on  their loan payments at the end of 2015, and more than a million  borrowers defaulted on their loans over the 2015 fiscal year. Since 2009,  the U.S. Department of Education (Education) has created several new  Income-Driven Repayment (IDR) plans that borrowers can select to  reduce challenges they face in repaying their loans. These plans primarily  base payment amounts on a borrower’s income and extend repayment  periods from the standard 10 years to up to 25 years with any remaining  balance forgiven at the end of that period.
Participation in IDR plans is growing, as is their estimated cost to the  government. By June 2016, 24 percent of borrowers repaying their loans  (or 5.3 million borrowers) were doing so in IDR plans, and it is likely that  more will join in the future. We reported in 2015 that there were gaps in  awareness about IDR plans. In April 2016, the Administration announced  a new goal to add 2 million new borrowers to IDR plans over the next  year, through efforts including targeted outreach. Education has also  increased its estimates of IDR plan costs. For instance, in its fiscal year  2016 budget justification, Education revised its prior-year estimates of  Direct Loan costs upward by $12.3 billion, and cited growing IDR plan  enrollment as the primary cause.
Given questions among policymakers and experts about IDR plan costs  to the federal government, you asked us to examine Education’s IDR plan  budget estimates and evaluate its approach to estimating costs. In this  report, we examine:  1.  What are Education’s current IDR plan budget estimates and how  have they changed over time?  2.  To what extent do Education’s approach to estimating IDR plan costs  and quality control practices help ensure reliable budget estimates?
We used a variety of methods to answer these questions. We reviewed  relevant federal laws, regulations, and guidance regarding subsidy cost  estimates and IDR plans. We also interviewed officials from the  Congressional Budget Office and the U.S. Department of the Treasury  (Treasury) and staff from the Office of Management and Budget (OMB),  the office that oversees the formulation of the President’s budget, as well  as higher education policy experts to discuss issues related to federal  budgeting practices and estimated IDR plan costs. Additionally, we  reviewed documentation and interviewed officials from Education about  the agency’s approach to estimating costs and its quality control  practices.
We analyzed data underlying Education’s annual budget estimates for the  Direct Loan program. To address our first objective, we analyzed data  from two sources: (1) Education’s annual submissions to the President’s  budget for fiscal years 2011 through 2017 and (2) supplemental data  provided by Education that rely on the data and assumptions underlying  its fiscal year 2016 and 2017 budget estimates. To address our second  objective, we reviewed the computer programs and data Education uses  to estimate repayment patterns for loans in IDR plans. We then used  these programs to generate our own estimates of how changing selected  assumptions would affect cost estimates. We also analyzed estimated  cash flow data for loans in IDR plans to estimate the proportion of loan  dollars Education expects to be forgiven through these plans. To assess the reliability of Education’s budget estimates, we interviewed agency  officials, reviewed related documentation, and conducted extensive  electronic testing. We believe the data are reliable to report on the  funding Education reports is necessary to operate the Direct Loan  program, and to illustrate the sensitivity of Education’s budget estimates  to different assumptions about future loan repayment activity. We  evaluated Education’s approach to estimating IDR plan costs and quality  control practices using guidance for estimating subsidy costs issued by  the Federal Accounting Standards Advisory Board, accepted statistical  practices, and Education’s information quality guidelines, among other  sources. We assessed Education’s information sharing practices against  Education’s strategic plan and standards for internal control in the federal  government. See appendix I for more information on our methodology  and appendix II for more information on Education’s approach.
We conducted this performance audit from March 2015 to November  2016 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: Direct Loan Repayment Plans
paragraphs: Education’s Direct Loan program provides financing to students and their  parents to help students obtain postsecondary education. This program is  currently the largest federal direct loan program with $912 billion in  outstanding loans as of June 2016. Under this program, Education  issues several types of student loans described in the following sidebar.
William D. Ford Federal Direct Loan Types  Subsidized Stafford Loans: Available only to  undergraduate students with financial need  (generally the difference between their cost of  attendance and a measure of their ability to  pay, known as expected family contribution).  The interest rate as of July 1, 2016 is 3.76  percent. Borrowers are not responsible for  paying interest on these loans while in school  and during certain periods of deferment.   Unsubsidized Stafford Loans: Available  both to undergraduate and graduate school  students irrespective of financial need.  Interest rates as of July 1, 2016 are 3.76  percent for undergraduates and 5.31 percent  for graduate school borrowers. Borrowers  must pay all interest on these loans.  PLUS Loans: Available to graduate student  borrowers and parents of dependent  undergraduates. The interest rate as of July 1,  2016, is 6.31 percent. Borrowers must pay all  interest on these loans.  Consolidation Loans: Available to student  and parent borrowers wanting to combine  multiple federal student loans (including those  listed above) into one loan. Repayment  periods are extended up to 30 years, thereby  lowering monthly payments. Interest rates are  equal to the weighted average of the  underlying loans.
Education offers a variety of repayment plans for Direct Loan borrowers:  Standard, Graduated, Extended, and Income-Driven. Income-Driven  Repayment (IDR) is an umbrella term that describes a number of  repayment plans available to Direct Loan borrowers who meet specific  eligibility requirements, as seen in figure 1. Unlike the Standard,  Graduated, and Extended repayment plans, IDR plans offer loan  forgiveness at the end of the repayment term. Additionally, their  repayment terms are longer than under the Standard and Graduated  plans, which are set at 10 years for non-consolidated loans.
Borrowers in IDR plans generally have lower monthly payments  compared to the Standard 10-year repayment plan. They may also pay  less in the long term than they would under the Standard 10-year  repayment plan due to the opportunity for eventual loan forgiveness.  However, some borrowers may pay more. Borrowers in IDR plans can  ultimately pay more in interest on their loans than they would under the  Standard 10-year repayment plan due to longer repayment periods.  Some borrowers will also fully repay their loans before their IDR plan  repayment term ends and, therefore, not receive forgiveness.  Additionally, under current tax law any amount forgiven under these plans  is subject to federal income tax.
In addition to making monthly payments more manageable (and  eventually reducing the total amount owed for some borrowers receiving  forgiveness), IDR plans may also reduce the risk of default. Borrowers  who default on student loans face serious consequences, including  damaged credit ratings and difficulty obtaining affordable credit in the  future. In 2015, we reported that borrowers in two IDR plans had much  lower default rates than borrowers in the Standard repayment plan.  Specifically, among borrowers who entered repayment from fiscal year  2010 through fiscal year 2014, less than 1 percent of borrowers in the  Income-Based Repayment and Pay As You Earn had defaulted on their  loan, compared to 14 percent in the Standard repayment plan.
To participate in an IDR plan, borrowers must provide documentation of  their adjusted gross income (which we generally refer to as income in this  report) to their loan servicer and certify their family size for an eligibility  determination. Borrowers must recertify this information annually, which  is used to update the borrower’s monthly payment amount. A borrower  who fails to provide updated income information can remain in an IDR  plan in order to qualify for future loan forgiveness, but their monthly  payments will no longer be based on their income. Rather, payments will  generally revert to the amount that would be owed under the Standard  10-year repayment plan until the borrower submits the required  information.
Borrowers who work in public service may lower their long-term loan  costs by participating in the Public Service Loan Forgiveness (PSLF)  program while repaying their loans through an IDR plan. Beginning in  October 2017, borrowers eligible for PSLF can have their remaining  Direct Loan balances forgiven after at least 10 years of payments in  eligible repayment plans, generally an IDR plan or the Standard 10-year  repayment plan. As we recently reported, PSLF may provide substantial  savings over the life of the loan for qualifying borrowers in IDR plans  compared to what they would pay without the PSLF benefit. In contrast,  borrowers in the Standard 10-year repayment plan would pay their loans  in full by the time they were eligible for forgiveness under PSLF. (See  figure 2.)
Participation in IDR plans has grown over time, as seen in figure 3.  According to currently available quarterly data released by Education, the  percent of outstanding Direct Loan dollars being repaid through IDR plans  doubled from June 2013 to June 2016 to 40 percent. The percent of  borrowers participating in IDR plans more than doubled over the same  time period to 24 percent. However, as we previously reported, some  borrowers who could benefit from IDR plans may still not be aware of  them.
section_title: Proposals to Modify IDR Plans
paragraphs: As the variety of IDR options available to borrowers has expanded in  recent years, there have been numerous reform proposals with a variety  of goals ranging from simplifying IDR plans and better targeting their  benefits to changing the tax treatment of IDR plan loan forgiveness. For  instance, recent President’s budgets have proposed limiting the available  IDR plan options for new borrowers to one revised IDR plan designed to  better target benefits to the highest-need borrowers. A proposal has  been introduced in the current Congress that would similarly make only  one IDR plan available to new borrowers and target more generous  benefits to those with lower incomes. Additional legislative proposals  would automatically enroll all borrowers in a version of income-driven  repayment and withhold payments from borrowers’ paychecks. Other  proposed legislation would allow for automatic annual recertification of  borrowers’ incomes and automatically place certain delinquent borrowers  in an IDR plan. Another proposal would expand IDR plan eligibility to  parents with Parent PLUS loans for dependent students. Legislation has  also been introduced that would exempt student loan forgiveness under  certain IDR plans from being taxed as income.
section_title: Subsidy Cost Estimates
paragraphs: As required by the Federal Credit Reform Act of 1990, Education  estimates the long-term costs, known as subsidy costs, of the Direct Loan  program annually for inclusion in the President’s budget. For Direct  Loans, subsidy costs represent the estimated cost to the government of  extending credit over the life of the loan, excluding administrative costs.  (In this report, we generally refer to subsidy costs as “costs.”) Subsidy  cost estimates are calculated based on the net present value of lifetime  estimated cash flows to and from the government associated with these  loans. For Direct Loans, cash flows from the government include loan  disbursements to borrowers, while cash flows to the government include  repayments of loan principal, interest and fee payments, and recoveries  on defaulted loans. A positive subsidy cost estimate indicates that the  government anticipates a net cost, while a negative subsidy cost estimate  indicates that the government anticipates generating net subsidy income,  not counting administrative costs.
Education also annually reestimates the cost of loans made in each fiscal  year, known as a loan cohort. Reestimates take into account actual loan  performance as well as changes in assumptions about future  performance, such as how many borrowers will default or how many will  participate in different repayment plans. Reestimates may result in  increases or decreases in subsidy cost estimates. No loan cohorts have  been fully repaid, and estimates for all cohorts continue to be updated  annually in the President’s budget.
To estimate subsidy costs, Education has developed a student loan cash  flow model (the student loan model) that incorporates a variety of  assumptions about the future. These assumptions concern various  aspects of loan performance, such as how many borrowers will prepay  their loans, how many borrowers will default, and how successful default  collection activities will be. Education uses a supplementary model to  assist with the task of estimating repayment patterns for loans in IDR  plans. (See appendix II for a description of how this supplementary model  for estimating IDR plan repayment patterns works.) In the spring of 2015,  Education initiated a redesign of its overall student loan model with  technical support from Treasury and guidance from OMB in what is  anticipated to be a multi-year project.
section_title: Education’s Budget Estimates of IDR Plan Costs Are Growing, but Actual Costs Will Not Be Known for Many Years
subsections: section_title: Education Estimates That Loans in IDR Plans Will Have Substantial Costs to the Government
paragraphs: Through our analysis of data underlying the President’s fiscal year 2017  budget, we found that Education estimates that Direct Loans in IDR plans  will cost the government about $74 billion over their repayment term.  More specifically, Education estimates that about $355 billion in loans will  enter an IDR plan, and $281 billion will ultimately be paid by borrowers.
As a result, Education expects a 21 percent subsidy rate, or an average  cost to the government of $21 per every $100 in loans disbursed. See  figure 4.
All of the Direct Loan types eligible to participate in IDR plans contribute  to the $74 billion Education estimates the government will incur in subsidy  costs. Of these loan types, Consolidation loans are estimated to be the  most costly, as seen in figure 5. Consolidation loans, which combine  multiple existing federal student loans into one loan, are larger on  average than other types of Direct Loans, and may have higher balances  forgiven at the end of their repayment term. Further, Education officials  said that some borrowers in IDR plans with Consolidation loans have  higher default risks than other borrowers, which leads to higher expected  subsidy rates for these loans. Education estimates lower subsidy costs  for Subsidized and Unsubsidized Stafford and PLUS loans for graduate  student borrowers (known as Grad PLUS loans) than Consolidation  loans.
As figure 6 shows, Education estimates higher subsidy costs for loans  participating in IDR plans from more recent loan cohorts compared to  loans from older cohorts.
Figure 7 shows that these higher estimated costs track closely with the  higher loan volume (or total loan dollars) estimated to enter IDR plans for  more recent loan cohorts. Education officials confirmed that this higher  estimated loan volume is likely related to three key factors: more generous IDR plans available for loans issued since fiscal year  increased efforts to make borrowers aware of IDR plans, and  increased overall volume of Direct Loans issued as a result of  increased college attendance following the 2008 economic downturn  and the end of the Federal Family Education Loan program (which  guaranteed federal student loans issued by private lenders) in 2010.
While borrowers in IDR plans in more recent loan cohorts have access to  more generous benefits (which could lead to higher government costs),  these loan cohorts do not have higher estimated subsidy rates than  earlier loan cohorts, as seen in figure 8. Direct Loan subsidy rates  fluctuate according to changes in a variety of factors, and are particularly  sensitive to changes in government borrowing costs and borrower interest  rates. As we previously reported, government borrowing costs fell  sharply in 2009, due to historically low interest rates of Treasury  securities. This phenomenon contributed to lower overall estimated  subsidy rates for Direct Loans issued following the 2008 loan cohort.
section_title: Education’s Current IDR Subsidy Cost Estimates Are Higher than Previously Expected, but Actual Costs Will Not Be Known for Many Years
paragraphs: Education has raised its estimates of IDR plan costs in recent years  through its annual process of revising past budget estimates to account  for actual loan performance and updated assumptions about future loan  performance. In figure 9, we compare Education’s original IDR plan  subsidy cost estimates for loans issued in recent cohorts to its current  subsidy cost estimates prepared for the President’s fiscal year 2017  budget. Our results show that current estimated IDR plan costs are  more than double what was originally expected for these cohorts. For  instance, Education originally estimated in the President’s fiscal year  2012 budget that IDR plan costs for the 2012 cohort would be $1.2 billion.  As of the fiscal year 2017 budget, Education’s estimate had grown to $3  billion. (We also compared Education’s fiscal year 2016 IDR plan  budget estimates to its fiscal year 2017 budget estimates to illustrate how  Education’s cost estimates changed over one budget cycle, and present  the results of that analysis in appendix IV.)
As seen in figure 10, subsidy rates have remained relatively stable from  original to current estimates, while the volume of loans expected to be  repaid in IDR plans has increased dramatically. Because Education  expects loans in IDR pans to have positive subsidy rates (or to have costs  to the government), this growth in estimated loan volume has been  accompanied by increasing estimates of IDR plan costs.
According to our data analysis and interviews with Education officials,  Education may have originally underestimated the volume of loans that  would enter IDR plans from these cohorts for several reasons:  1.  Education did not include Grad PLUS loans in its IDR plan subsidy  estimates until the fiscal year 2015 budget, even though Grad PLUS  loans have been eligible for IDR plans since they were first issued in  2006. Education officials said that they had to make a model  adjustment in order to include Grad PLUS loans in IDR estimates.  Prior to this adjustment they assumed all Grad PLUS loans would be  repaid in other repayment plans.  2.  Policy changes made IDR plans more generous and available to more  borrowers after Education originally estimated costs for some cohorts.  For example, the Pay As You Earn repayment plan was implemented  in fiscal year 2013 and retroactively made more generous benefits  available to certain borrowers with loans issued as early as the 2008  cohort.  3.  While some eligible borrowers still may not be aware of IDR plans,  participation rates are growing, and officials responsible for budget  estimates may not have adequately anticipated participation growth.
While we previously reported that there are substantial challenges  associated with estimating Direct Loan subsidy costs, these challenges  are increased for Direct Loans in IDR plans due to their complex features  and other uncertainties. It is difficult for Education to estimate which  borrowers have incomes low enough to benefit from or be eligible for IDR  plans because Education does not collect income information for all  Direct Loan borrowers.  Additionally, IDR plan participation rates are  difficult to predict. While participation has been growing rapidly in recent  years, it is unclear at what rate it will continue to grow. It is also  challenging to predict how the incomes of borrowers already participating  in IDR plans will change over time and how much loan principal will  ultimately be forgiven. Further complicating Education’s task is the fact  that the large majority of loans expected to be repaid in IDR plans are  from recent cohorts, and many borrowers in these cohorts have not yet  started repaying their loans. As a result, there is limited actual repayment data available to inform Education’s estimates. Further, no borrower has  received loan forgiveness under IDR plans.
Volatility in subsidy cost estimates is generally expected to be greatest  early in the life of a loan cohort, and to decrease over time as more actual  repayment data are incorporated into estimates. When we compared  original, third-year, and currently estimated IDR plan subsidy cost  estimates for several recent cohorts, we found that third-year estimates  were generally closer to current estimated costs than the original, as  figure 11 illustrates. However, estimates will continue to change over  time, and actual subsidy costs of a loan cohort will not be known until all  loans in the cohort have been repaid, which may take 40 years.
section_title: Current Estimated IDR Plan Costs Are Offset by Greater Estimated Income from Loans in Other Repayment Plans
paragraphs: While loans in IDR plans are expected to have long-term costs to the  government, loans in other repayment plans (Standard, Graduated, and  Extended) are expected to generate greater subsidy income, as seen in  figure 12. Figure 12 also illustrates that Education currently expects  income to be higher for more recent cohorts than older cohorts.  However, as mentioned previously, subsidy cost estimates change over  time, and the actual costs or income attributable to any Direct Loan cohort  will not be known until all loans in the cohort are repaid.
Subsidy income estimates for loans participating in non-IDR plans vary by  loan type and repayment plan. Unsubsidized Stafford and PLUS loans  participating in the Standard 10-year repayment plan are estimated to  result in the greatest subsidy income to the government. This could be  due in part to the higher interest rates charged to borrowers with  Unsubsidized Stafford and PLUS loans compared to Subsidized Stafford  loans, as well as a higher volume of loans participating in Standard  repayment compared to other repayment plan options. See figure 13.
Further, as with loans in IDR plans, Education’s estimates of subsidy  income from loans in non-IDR plans have changed over time and will  continue to fluctuate as they are updated with actual repayment data and  revised assumptions about future cash flows. We found that estimated  income associated with loans participating in non-IDR plans increased  (about $19 billion more) for some cohorts and decreased (about $36  billion less) for other cohorts when we compared Education’s original and  current estimates for those cohorts (2009-2015).
While Education currently estimates that loans in IDR plans will have  costs to the government, these plans are designed to provide relief to  struggling borrowers, which could indicate that government subsidies may  be expected. By tying monthly payments to borrowers’ incomes, IDR  plans help make potentially onerous student debt payments more  affordable for many individuals. Because these borrowers’ repayment  amounts may be lower than they otherwise would be, borrowers in IDR  plans may have more success in making their loan payments than  borrowers in other plans. As we previously reported, substantially lower  percentages of participants in the Income-Based Repayment and Pay As  You Earn repayment plans had defaulted on their loans compared to  those in the Standard 10-year repayment plan, and the great majority of  borrowers in these IDR plans were in active repayment status (e.g., not in  delinquency, default, or forbearance). Further, because IDR plans  attract borrowers experiencing difficulty repaying their loans in other  plans, increased IDR participation from these borrowers may lead to  lower subsidy rates for non-IDR plans.
section_title: IDR Plan Budget Estimates May Be Unreliable Due to Limitations in Education’s Approach and Inadequate Quality Control Practices
paragraphs: Education’s approach to estimating IDR plan costs has numerous  weaknesses that may result in unreliable budget estimates. Poor quality  control practices, such as inadequate model testing, contributed to issues  we identified. Further, because Education publishes only limited  information about its estimates, it may be difficult for policymakers to  assess expected plan costs and consider the potential for alternative  outcomes.
subsections: section_title: Education’s Approach to Estimating IDR Plan Costs May Produce Unreliable Budget Estimates
paragraphs: Due to several methodological limitations, Education’s approach to  estimating IDR plan costs may result in unreliable budget estimates. First,  Education did not adequately assess the reliability of the data it uses to  forecast borrower incomes over time, or assess the level of error these  data or its forecasting methods introduced into its IDR plan budget  estimates. Second, it did not consider how inflation would affect  borrowers’ incomes over time. Third, Education unrealistically assumes  that no borrower will fail to recertify their income, which is required of  borrowers annually to maintain lower income-driven payment amounts.  Fourth, Education does not account for future growth in IDR plan  participation rates. Fifth, Education does not produce separate cost  estimates for each of the five IDR plans currently available to borrowers.  Finally, Education’s cost estimates for Subsidized Stafford, Unsubsidized  Stafford, and Grad PLUS loans in IDR plans do not account for likely  differences in how they will perform over time.
subsections: section_title: Borrower Income Data and Forecasting Methods
paragraphs: Education’s IDR plan cost estimates are vulnerable to unidentified error  because Education has not adequately assessed the reliability of the  estimated borrower income data and methods it uses to forecast borrower  incomes many years into the future—information that is vital to  determining how much borrowers will owe and repay on their loans over  time. Education conducted only limited, informal testing to assess the  data’s reliability, in part because the agency had short timeframes in  which to develop its approach to estimating IDR plan costs, according to  officials we interviewed. Education did not measure the amount of error  these data introduced into IDR plan cost estimates to determine whether  it was acceptable, or if alternative data were needed.
Through our data reliability testing, we identified patterns in the estimated  historical income data suggesting reliability problems that could make  them unacceptable for Education’s purposes. An analysis by Treasury  (the agency that created the estimated historical income data) indicates  that the data fluctuate on average by 44 percent more per year than the  actual income data upon which they were based. In figure 14, we  illustrate this fluctuation for five randomly selected borrowers from the  estimated dataset over the first 10 years of their repayment period. (See  appendix III for more information on how these data were estimated and  our evaluation of them.)
Education uses individuals’ estimated historical incomes, such as those  illustrated in figure 14, to make numerous sequential calculations that  determine how much each borrower will owe and pay in each year of the  borrower’s repayment period. While the estimated historical income data  appeared more reasonable in the aggregate, Education officials  confirmed that any unusual fluctuations in them at the individual borrower  level could affect the quality of IDR plan budget estimates.
In addition to being vulnerable to error associated with the estimated  historical income data they use, Education’s IDR plan budget estimates  may further be affected by error associated with the agency’s method for  forecasting borrowers’ incomes for up to 30 years into the future. The  accuracy of any forecast—separately from the reliability of the data used  for forecasting—depends on how well the data and forecasting methods  can estimate future incomes. However, Education did not assess the  amount of error in its forecasts of borrower incomes. Until Education  assesses its forecasting methodology, its IDR plan cost estimates may be  vulnerable to unidentified error.
Both federal guidance for estimating subsidy costs and Education’s own  information quality standards emphasize the importance of ensuring that  estimates are based on reliable data. Education’s information quality  standards and generally accepted statistical practices also recommend  measuring error to assess its impact on estimates. Education officials  agreed with the concerns we raised regarding their borrower income data  and said they are open to improving data quality as necessary to help  ensure reliable IDR plan budget estimates. Quality data and methods are  essential to Education’s estimation approach, and both should be  assessed to determine whether they produce reasonable results. (See  appendix III for more information on error associated with Education’s  data and methods.)
section_title: Effect of Inflation
paragraphs: In addition to insufficiently assessing the reliability of its income data and  forecasting methods, Education has not adjusted its income forecasts for  inflation, causing IDR plan budget estimates to appear higher than they  otherwise would be. Adjusting for inflation would increase borrowers’  future incomes and payment amounts, because loan payments are based  on borrowers’ incomes. Increasing payment amounts would, in turn,  decrease costs to the government. When asked, Education officials said  they did not adjust income forecasts for inflation because they did not  identify patterns in the estimated historical income data suggesting that  incomes would be affected by inflation. Whether or not these patterns  were evident when reviewing the data, there was inflation over the almost  20-year period covered by the historical dataset and there is likely to be  inflation in the future. Federal guidance for estimating subsidy costs  stresses the importance of taking economic effects into account when  estimating loan performance. For IDR plan costs, this would include the  extent to which inflation affects borrower incomes and payment amounts.
By choosing not to adjust income forecasts to capture inflation’s future  effects, Education over-estimated IDR plan costs. When we used  Education’s data and computer programs to adjust borrowers’ future  incomes for inflation, as well as the federal poverty guidelines used to  calculate their discretionary incomes, we found that IDR plan budget  estimates declined by over $17 billion, when compared to Education’s  current IDR plan budget estimates. (See figure 15.) In light of the  substantial effects of inflation on borrower incomes and loan repayment  amounts, inflation adjustment is essential to developing reliable IDR plan  budget estimates. Until Education adjusts for inflation, its budget  estimates will continue to inaccurately represent potential IDR plan costs.
Additionally, Education assumes that all borrowers in IDR plans will  recertify their incomes every year as required, which is likely to be  inaccurate and could lead Education to overstate IDR plan costs. In fact,  we recently reported that over half of borrowers in an Education sample  failed to do so. When borrowers fail to recertify their income, Education  generally increases their payments to what they would owe under the  Standard 10-year repayment plan until they submit their required  recertification. For some borrowers who fail to recertify their income,  payments could increase by hundreds of dollars a month. While some  borrowers may subsequently recertify within a few months, others may  never recertify. Because Education does not take these occurrences into  account, it underestimates what borrowers will pay when their certification  lapses.
Education officials told us they did not include certification lapses in their  approach to estimating IDR plan costs because they lacked recertification  data linked to individuals. They also believed that certification lapses  would not have a large impact on their estimates. Initially, officials said  the agency is taking steps to reduce the number of borrowers failing to  recertify. However, officials later acknowledged that these efforts are in  the early stages of implementation, and there have been some  setbacks. Until efforts to improve recertification rates are put in place,  certification lapses will likely continue. Further, without data indicating that  certification lapses do not have a large impact on borrower payment  amounts, Education may overstate IDR plan costs. Federal guidance for  estimating subsidy costs states that the information used in the estimation  process should reflect actual repayment patterns for loans whose costs  are being estimated, which would include instances when a borrower’s  payment amount changes due to program rules. Obtaining data on  borrowers’ actual repayment patterns after they fail to recertify their  income could help Education determine whether its current approach  appropriately accounts for the impact of recertification failure on IDR plan  costs, and determine whether changes are needed.
section_title: Future IDR Plan Participation
paragraphs: Education likely underestimates IDR plan participation because it  assumes all borrowers will remain in their currently selected repayment  plan for their entire repayment period. This assumption conflicts with the  fact that borrowers can switch into or out of IDR plans at any time, and  IDR plan participation has grown in recent years. Participation is also  likely to continue growing. Education agreed with our recent  recommendation that the agency increase its efforts to make all  borrowers aware of IDR plans. Further, as previously mentioned, the  Administration recently announced a goal to enroll 2 million additional  borrowers in IDR plans.
As a result of Education’s likely underestimation of IDR plan participation,  its IDR plan budget estimates may be biased downward, or appear lower  than they otherwise should be. We found that Education’s IDR plan  budget estimates for loans issued in recent cohorts have more than  doubled over what was originally expected ($53 billion vs. $25 billion),  primarily because of higher than expected participation in IDR plans.  Federal guidance for estimating subsidy costs of federal loan programs  states that it is preferable to use methods to estimate costs that are more  sophisticated than relying solely on historical data, such as borrowers’  past plan selection.
While Education’s current student loan model was not designed to project  future changes in plan participation, officials told us that despite the  challenge of predicting future borrower behavior they are working with  Treasury to develop a more sophisticated model, and have begun  incorporating this enhancement into a test version of this new model.  Additional work remains to ensure that the new model reasonably reflects  trends in IDR plan participation—particularly borrowers switching into IDR  plans from other repayment plans. For instance, IDR plans have not yet  been added to the new model, which currently includes only the Standard  and Extended repayment plans. Education’s model redesign is  anticipated to be a multi-year project, and until the model has been  completed and tested to ensure reasonable results, Education’s IDR plan  budget estimates are vulnerable to underestimated IDR plan participation  and costs.
section_title: Cost Estimates for Different IDR Plans
paragraphs: Additionally, Education does not produce separate cost estimates for  each of the five IDR plans currently available, even though these plans  provide different benefits to borrowers and will likely have different costs  to the government. For instance, the Income-Contingent Repayment plan  has less generous provisions for borrowers than the Pay As You Earn  plan, and as a result will likely have lower costs to the government.  However, Education does not estimate these plans’ costs separately.  According to Education officials, the student loan model, which it uses to  generate official estimates of total Direct Loan costs, was created when  only one IDR plan was available and cannot produce separate estimates  for each IDR plan. While the supplementary model Education uses to  estimate IDR plan repayment patterns could track repayment streams  separately for each plan, its outputs must conform to the structure of the  larger student loan model.
Federal guidance for estimating subsidy costs for federal loan programs  specifies that agencies should assess the impact of changes in laws or  regulations (such as the introduction of new repayment plans) on the  reliability of estimates and should ensure that an agency’s methodology  reflects these changes. While Education officials expressed concern  about the complexity of estimating separate costs for each IDR plan,  OMB staff told us that Education should add this capability as part of  Education’s efforts to develop a more sophisticated model. Incorporating  the ability to track costs of each IDR plan separately would help ensure  that estimates more accurately reflect the current loan environment and  provide valuable information to policymakers interested in streamlining  student loan repayment options moving forward.
section_title: Repayment Patterns for Different Loan Types
paragraphs: Lastly, Education combines repayment patterns for several types of loans  eligible for IDR plans, obscuring likely differences in their performance  over time. As a result, its budget estimates for Subsidized Stafford,  Unsubsidized Stafford, and Grad PLUS loans in IDR plans are based on  identical repayment patterns, although these types of loans have  numerous distinct features. For instance, the current interest rate on a  Grad PLUS loan is almost double that of a Subsidized Stafford loan,  leading borrowers with Grad PLUS loans to owe much more in interest on  those loans over time. Conversely, borrowers with Subsidized Stafford  loans will pay down principal on their loans more quickly over time  because less of their payment goes toward interest. However,  Education’s cost estimates do not reflect higher expected interest  payments on Grad PLUS loans in IDR plans or faster principal repayment  on Subsidized Stafford loans in IDR plans, because they are based on  aggregate repayment patterns that include both types of loans. Education  officials told us that, as a result of this practice, all differences in  published subsidy rates for these loan types are wholly attributable to  fees charged to borrowers at the time the loans are issued and how much  interest accrues during the relatively short period that borrowers are still  in school.
Because Education’s estimates do not reflect differences in performance  over the decades that loans in IDR plans may be in repayment, users of  the budget are missing key information that could help them assess how  IDR plan costs vary by loan type. As an example, some experts have  raised concerns that Grad PLUS loans could have relatively high  forgiveness amounts because they are larger on average than Stafford  loans and may have a large amount of outstanding loan principal at the  end of their repayment term. Due to limitations in Education’s current  approach, users of the budget cannot determine the extent to which this  concern affects subsidy rates for Grad PLUS loans in IDR plans.
According to Education officials, they could have separately estimated  repayment patterns for each loan type, but did not believe that it was  important to do so for several reasons. First, officials stated that they  focused their efforts on estimating separate repayment patterns for  Consolidation loans because they make up the majority of loans in IDR  plans. However, nearly half of IDR plan loan volume—or $164 billion—is  made up of Subsidized Stafford, Unsubsidized Stafford, and Grad PLUS  loans, and it is important to estimate their repayment patterns accurately  as well. Second, officials stated that they did not believe it was necessary  to maintain separate repayment patterns for each type of loan because  borrowers often have a mix of loans and repay them simultaneously.  While this is true, policymakers have an interest in budget information that  accurately reflects expected costs for each type of loan eligible for IDR  plans. Further, federal guidance for estimating subsidy costs of federal  loan programs states that loan characteristics—such as loan types—are  critical for identifying factors that predict subsidy costs and should be  preserved. Until Education separately tracks repayment patterns for  each type of loan in IDR plans, its cost estimates will continue not to take  into account important differences in loan characteristics, calling into  question the reliability of the cost estimates, and policymakers will be  unable to assess the relative costs of different types of loans.
section_title: Education’s Quality Control Practices Are Inadequate to Help Ensure That Its IDR Plan Budget Estimates Are Reliable
paragraphs: Inadequate quality control practices contribute to concerns we identified  regarding Education’s approach to estimating IDR plan costs. First,  management has not ensured that the agency’s supplementary model for  estimating IDR plan repayment patterns is properly documented. Second,  management has not reviewed or approved that model. Third,  management has not ensured that the model has been sufficiently tested  for reliability.
subsections: section_title: Documentation
paragraphs: Education has not ensured that its supplementary model for estimating  IDR plan repayment patterns is properly documented. While a broad  narrative summary of the model is available, agency officials confirmed  that other technical documentation recommended in federal guidance for  estimating subsidy costs does not exist. For instance, Education does  not have a flow chart or other similar documentation specifying how  elements of the estimation process—which is implemented by nearly 50  computer programs—are sequenced and interact with each other.  Additionally, the numerous mathematical formulas embedded in these  programs are not separately documented, and there is no data dictionary  to decode the variable names and values. Standards for internal control in  the federal government state that documentation is a necessary part of an  effective internal control system. Federal guidance for estimating  subsidy costs states that model documentation should be thorough  enough that a knowledgeable independent person could follow the  estimation process and replicate its results with little to no assistance.  Such documentation is not available for Education’s supplementary  model for estimating IDR plan repayment patterns.
We recently recommended that Education improve documentation of its  overall process for estimating costs of Direct Loans. Education agreed  with this recommendation, and officials stated that they were in the  process of improving their documentation practices.
section_title: Review and Approval
paragraphs: Further, we found that Education’s managers did not review and approve  the supplementary model for estimating IDR plan repayment patterns, as  recommended in federal guidance for estimating subsidy costs, after it  was developed by staff. Additionally, as a good practice, we have found  that agencies often hire an independent firm to ensure that model  calculations are accurate and consistent with documentation. However,  Education officials confirmed that their supplementary model for  estimating IDR plan repayment patterns has not been reviewed by an  independent firm.
Some of the concerns we identified in the previous section of our report  regarding Education’s estimation approach could have been identified  and resolved through an internal management review or independent  external review. For instance, we found that the decision not to adjust  borrower income forecasts for inflation causes IDR plan budget estimates  to be $17 billion higher than they otherwise would be. We also found that  PSLF loan forgiveness was programmed to begin a year after the benefit  will actually become available to eligible borrowers. When we revised  these programs to allow loans to be forgiven a year earlier, estimated IDR  plan costs rose by $70 million. Agency staff told us this decision was  made because borrowers were not likely to make the 120 consecutive on- time payments necessary to qualify for immediate forgiveness.  However, Education already makes assumptions about when borrowers  will not make scheduled loan payments. An internal management  review or independent external review may have pointed toward another  solution—such as adjusting how often borrowers are assumed to have  periods of non-payment—rather than simply delaying the PSLF start  date.
We recently recommended that Education create a documented process  for management review and approval of its student loan model.  Education agreed with this recommendation, and officials told us they  also hoped to have their revised student loan model reviewed by an  outside party in the future.
section_title: Model Testing
paragraphs: Although Education currently expects loans in IDR plans to be the most  costly component of the Direct Loan portfolio, management has not  ensured that its supplementary model for estimating IDR plan repayment  patterns has been thoroughly tested. Such testing can help identify  weaknesses so that they can be addressed, and help ensure that  estimates are reasonable. As we previously mentioned, Education had  not conducted the necessary testing to thoroughly assess the reliability of  its borrower income data or measured error associated with its income  forecasting methodology. Without such testing, Education officials do not  know whether their data and methods produce reasonable results, or if  alternatives are needed.
Further, Education conducted sensitivity analysis on only one key  assumption—borrower incomes—at the request of OMB. Federal  guidance for estimating subsidy costs states that agencies should  conduct sensitivity analysis—which involves adjusting an assumption up  or down by a fixed proportion—or other testing to identify which  assumptions have the largest influence on cost estimates. This  information helps management anticipate the cost implications of  alternative scenarios and focus oversight resources on key assumptions  to help ensure that they are reliable and reasonable. However, Education  officials told us they only conducted sensitivity analysis when asked by  others, preferring instead to focus their resources on developing a single  set of assumptions they believed were best. Developing a sound set of  assumptions is, of course, important. Sensitivity analysis supports, rather  than detracts from, this effort.
For instance, little is known about how many borrowers are eligible for or  will participate in PSLF when it becomes available in October 2017.  Despite this uncertainty and concerns among some experts and  policymakers that PSLF could be costly to the government, Education has  not conducted sensitivity analysis on its PSLF participation assumption.
In order to illustrate the importance of conducting sensitivity analysis on  major assumptions, we first revised Education’s computer programs to  increase the percentage of borrowers expected to participate in PSLF by  5 and 10 percentage points. As illustrated in figure 16, costs rose by $4.4  and $9 billion, respectively.
We then decreased the percentage of borrowers participating in PSLF by  5 and 10 percentage points. As seen in figure 17, costs fell by similar  amounts.
Our results illustrate the potential for PSLF costs to be different than what  Education currently expects, and why it is important for Education to  monitor this assumption and adjust it as necessary to ensure that it is  reasonable. Without conducting similar sensitivity analysis on other major  assumptions, monitoring those assumptions carefully, and adjusting them  as necessary to ensure that they are reliable, Education’s budget  estimates are vulnerable to bias that could result in costs being over- or  understated by billions of dollars.
section_title: Published Information May Not Be Sufficient for Policymakers to Accurately Assess Expected IDR Plan Costs
paragraphs: In addition to identifying limitations in Education’s approach to estimating  IDR plan costs and its quality control practices, we also found that  Education has not published sufficient information about its estimates for  policymakers to readily assess expected IDR plan costs. The kinds of  information that Education has not published—and that could be useful to  policymakers—include (1) total expected costs, (2) trend in estimates, (3)  sensitivity analysis results, (4) limitations in estimates, and (5) estimated  forgiveness amounts. Education officials noted that the department takes  its responsiveness to policymakers and the general public seriously, and  that the agency has responded to information requests about IDR plan  cost estimates by congressional staff. However, congressional interest in  IDR plans is high, and currently available information may be insufficient  for policymakers to accurately assess likely plan costs, and consider the  potential for alternative outcomes.
For instance, as a part of the President’s budget, Education publishes  IDR plan loan volume and subsidy rate estimates for loans issued in the  current and two most recent cohorts. This information can be used to  calculate expected IDR plan costs for this limited group of loans.  However, it is not possible to use this information to determine total  expected costs for all loans in IDR plans. Additionally, Education has  disclosed in reports accompanying the President’s budget that IDR plans  are major contributors to upward revisions in estimated Direct Loan costs  as a whole—it has not reported the amount by which IDR plan costs have  risen or clearly described the reasons why.
Using unpublished data from Education, we found that total current  expected IDR plan costs are about $74 billion, or $21 for every $100  issued. We also found that expected IDR plan costs have doubled from  $25 to $53 billion for loans issued from fiscal years 2009 through 2016— primarily due to the growing volume of loans expected to be repaid in IDR  plans. Publishing more comprehensive information like this could help  policymakers better understand currently expected costs and monitor  trends in the Direct Loan portfolio.
Additionally, by publishing sensitivity analysis results and limitations in  estimates, Education could help policymakers understand what is known  about possible IDR plan costs, and what is still unknown. Our own  sensitivity analysis illustrates that IDR plan costs could be billions of  dollars more or less than currently estimated if PSLF participation is  higher or lower than expected. Given growth in IDR plan cost estimates  over time due to the rising volume of loans expected to be repaid in these  plans, it would also be useful to disclose that current estimates assume  that no borrowers will switch from other repayment plans into IDR plans in  the future.
Lastly, sharing the amount of principal Education expects to forgive on  loans in IDR plans could help policymakers better understand a key plan  feature that contributes to their expected costs. Education officials raised  concerns that publishing forgiveness amounts could be misleading,  because it is possible for the government still to generate income on  loans with principal forgiven, particularly if borrower interest payments  exceed forgiveness amounts. While this is true, loan amounts forgiven do  represent foregone cash flows to the government. Further, legislation has  been introduced in Congress to make forgiveness under certain IDR  plans tax-free. Sharing information about expected forgiveness amounts  could help policymakers better understand the scope of currently  expected loan forgiveness and the potential tax implications of excluding  forgiveness from taxable income.
We calculated currently expected IDR plan forgiveness amounts using  cash flow estimates provided by Education. For our analysis, we  calculated the amount of loan principal Education expects borrowers in  IDR plans to repay, and the amount it expects borrowers not to repay due  to forgiveness and other reasons. Our results are in figure 18.
When discussing expanded information sharing, Education officials and  OMB staff agreed that there could be value in reporting additional  information about IDR plan cost estimates. An Education official raised  concerns about the agency’s ability to publish additional cost information,  because OMB determines what is presented in the President’s budget.  OMB staff agreed that such information would be too detailed for the  President’s budget, but suggested that Education could provide more  detailed IDR plan cost information through separate reports. Education’s  strategic plan emphasizes the importance of information transparency as  a tool to encourage data-driven decision-making and improve the U.S.  educational system. Standards for internal control in the federal  government also note that management should share quality information  externally. By more thoroughly disclosing IDR plan cost information— such as total estimated costs, sensitivity analysis results, key limitations  in estimates, and expected forgiveness amounts—Education could help  policymakers better assess the cost implications of current IDR plan  provisions and consider whether reforms are needed.
section_title: Conclusions
paragraphs: Policymakers need reliable budget estimates to help align federal  expenditures with policy priorities. In an environment of scarce resources,  quality budget information becomes all the more important, as  policymakers face difficult funding decisions. While IDR plans are a  promising tool to help alleviate the burden of student loan debt and  reduce borrowers’ risk of default, they may be costly for the federal  government. Some uncertainty is unavoidable when anticipating long- term loan costs, but we found numerous shortcomings in Education’s  estimation approach and quality control practices that call into question  the reliability of its budget estimates and affect the quality of information  Congress has to make informed budget decisions.
Because Education administers the federal government’s largest direct  loan program, it is especially important that the agency corrects its  methodological weaknesses associated with estimating IDR plan costs.  More specifically, until Education assesses and improves the quality of  data and methods it uses to forecast borrowers’ future incomes and  accounts for inflation in its estimates, its IDR plan budget estimates may  be unreliable. Further, until Education obtains data needed to estimate the impact of income recertification lapses on borrower payment  amounts, it will not know whether borrower payments are currently  underestimated and whether adjustments are needed to avoid overstating  IDR plan costs. In addition, until Education’s planned revisions to its  student loan model have been completed and tested to ensure  reasonableness, the agency’s IDR plan budget estimates will not  reasonably reflect participation trends in IDR plans, particularly the extent  to which borrowers in other repayment plans may switch into them. In the  interim, Education may continue to understate IDR plan costs by billions  of dollars, as past trends in estimates indicate. Without separately  tracking how available IDR plans and the types of loans eligible for them  perform relative to each other over time, Education’s estimates will lack  the detail needed to inform policymakers’ ongoing efforts to streamline  plans and better target costs.
In addition to correcting its methodological weaknesses, Education could  enhance the reliability of its budget estimates by implementing more  robust quality control practices. Implementing our previous  recommendation to more thoroughly document and review its approach  could help Education’s management identify and resolve weaknesses.  More robust model testing, including more extensive sensitivity analysis,  could also help Education’s management identify and mitigate problems  that may reduce the reliability of its budget estimates.
Moreover, as Education works to improve the quality of its IDR plan  budget estimates, it could also help policymakers better understand the  scope of currently expected costs and the potential for alternative  outcomes by publishing more detailed information about its estimates,  such as total estimated costs, the results of sensitivity analysis, key  limitations, and expected forgiveness amounts. This information could  help better support efforts to assess the cost-effectiveness of IDR plans  and design any needed reforms.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Secretary of Education take the following six  actions:  1.  Assess and improve, as necessary, the quality of data and methods  used to forecast borrower incomes, and revise the forecasting method  to account for inflation in estimates.  2.  Obtain data needed to assess the impact of income recertification  lapses on borrower payment amounts, and adjust estimated borrower  repayment patterns as necessary.  3.  Complete efforts to incorporate repayment plan switching into the  agency’s redesigned student loan model, and conduct testing to help  ensure that the model produces estimates that reasonably reflect  trends in Income-Driven Repayment plan participation.  4.  As a part of the agency’s ongoing student loan model redesign efforts,  add the capability to produce separate cost estimates for each  Income-Driven Repayment plan and more accurately reflect likely  repayment patterns for each type of loan eligible for these plans.  5.  More thoroughly test the agency’s approach to estimating Income- Driven Repayment plan costs, including by conducting more  comprehensive sensitivity analysis on key assumptions and adjusting  those assumptions (such as the agency’s Public Service Loan  Forgiveness participation assumption) to ensure reasonableness.  6.  Publish more detailed Income Driven Repayment plan cost  information— beyond what is regularly provided through the  President’s budget—including items such as total estimated costs,  sensitivity analysis results, key limitations, and expected forgiveness  amounts.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of our report to the U.S. Department of Education  (Education) for its review and comment. We provided relevant excerpts  from our report to the U.S. Department of the Treasury and incorporated  its technical comments as appropriate. We provided a draft of our report  to the Office of Management and Budget for technical review, and did not  receive technical comments in response.
Education generally agreed with our recommendations, stating that in  light of growing IDR plan participation, the agency has focused efforts on  improving IDR plan budget estimates. Additionally, Education said that  estimating the federal student loan costs is a task it takes very seriously,  and that the agency is constantly seeking to enhance and refine its  models.
First, Education agreed to assess and improve its borrower income  forecasts, and listed additional factors it wished to consider when  determining how to incorporate inflation into its forecasts.
Second, Education agreed to attempt to obtain data to assess the impact  of income recertification lapses on borrower payment amounts. Education  reiterated its belief that such lapses may only have a small impact on plan  costs, but did not provide data to support that view. We clarified the  language in our recommendation to indicate that model adjustments  should only be undertaken as needed, based on the outcome of  Education’s review of relevant data.
Third, Education also agreed to incorporate repayment plan switching into  its redesigned student loan model, and reiterated that efforts to  incorporate this capability had begun despite challenges inherent in  predicting borrower behavior.
Fourth, Education agreed to add the capability to produce separate cost  estimates for each IDR plan and each eligible loan type into its  redesigned student loan model. Given the concern Education raised in its  letter that revising its current approach to improve loan-type estimates  may not be a good use of resources, we revised our recommendation to  clarify that this improvement could be undertaken as a part of the  agency’s longer-term efforts to redesign its student loan model.
Fifth, Education agreed to test its approach to estimating IDR plan costs  more thoroughly, including through more comprehensive sensitivity  analysis. Education further explained its rationale for delaying the Public  Service Loan Forgiveness (PSLF) start date in its cost model, citing  preliminary evidence suggesting that few borrowers will make the 120  consecutive on-time payments necessary to receive forgiveness in the  program’s first year. Education also raised concerns that using the correct  start date (which we found caused estimated costs to rise by $70 million)  would overstate costs. We noted Education’s rationale and concerns in  our report, and responded that another solution—such as adjusting how  often borrowers are assumed not to make scheduled loan payments— may be more appropriate than simply delaying the PSLF start date.
Education agreed with our sixth recommendation to publish more detailed  IDR plan cost information and stated that it plans to present sensitivity  analysis results and key limitations in upcoming financial reports.
Education’s comments are reproduced in appendix V.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 15 days from the  report date.
At that time, we will send copies to interested congressional committees  and to the U.S. Departments of Education and the Treasury and the  Office of Management and Budget. In addition, the report will be available  at no charge on the GAO website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (617) 788-0534 or emreyarrasm@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix VI.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This appendix discusses in detail our methodology for addressing (1) the  U.S. Department of Education’s (Education’s) current Income-Driven  (IDR) Repayment plan budget estimates and how they have changed  over time and (2) the extent to which Education’s approach to estimating  IDR plan costs and its quality control practices help ensure reliable  budget estimates.
To address these objectives, we reviewed relevant federal laws,  regulations, and guidance on the William D. Ford Federal Direct Loan  (Direct Loan) Program and IDR plans. We reviewed documentation and  interviewed officials from Education about the agency’s approach to  estimating costs and its quality control practices. We also interviewed  officials from the Congressional Budget Office and the U.S. Department  of the Treasury (Treasury) and staff at the Office of Management and  Budget (OMB), as well as higher education policy experts, to discuss  issues related to federal budgeting practices and estimated IDR plan  costs. To answer our first objective, we analyzed and reported on data  underlying Education’s annual budget estimates for the Direct Loan  program. To answer our second objective, we evaluated Education’s  estimation approach and conducted sensitivity analysis to determine the  impact of alternative assumptions on Education’s cost estimates. We also  calculated the proportion of loan dollars Education expects to forgive  under IDR plans using estimated cash flow data provided by Education.  These analyses are described in more detail below. To assess the  reliability of Education’s budget estimates, we interviewed agency  officials, reviewed related documentation, and conducted extensive  electronic testing. We believe the data are reliable to report in objective  one as a representation of the funding Education reports is necessary to  operate the Direct Loan program, and in objective two, to illustrate the  sensitivity of Education’s budget estimates to different assumptions about  future loan repayment activity and to illustrate currently expected  forgiveness amounts.
subsections: section_title: Objective One: Review of Current and Past IDR Plan Budget Estimates
paragraphs: To analyze Education’s current IDR plan budget estimates and how they  have changed over time, we reviewed Education’s annual submissions to  the President’s budget for fiscal years 2011 through 2017, which include  estimated IDR plan loan volume and subsidy rates for Direct Loans to be  issued in the year the of the budget and the two preceding fiscal years.  For example, the budget submission for fiscal year 2011 included  estimated IDR plan costs for loans in fiscal years 2009, 2010, and 2011.  We used these budgets to identify the original IDR plan cost estimates for  the 2009 through 2016 cohorts. Education did not publish subsidy cost  estimates by repayment plan prior to the 2011 budget and could not  easily provide the information necessary to determine original IDR plan  cost estimates for previous cohorts.
We also reviewed supplemental unpublished data from Education to  illustrate current IDR plan subsidy cost estimates for loans issued in fiscal  years 1995 through 2017 using assumptions underlying their estimates  for the President’s fiscal year 2016 and 2017 budget. We used these  supplemental data, along with published data for the 2017 cohort, to  calculate current total reestimated subsidy costs and subsidy income for  each repayment plan, loan cohort, and loan type. We also compare  original published IDR plan subsidy estimates for the 2009-2016 cohorts  to the current reestimated IDR plan subsidy costs for those cohorts. We  limited our comparison to these cohorts because Education did not  publish subsidy cost estimates by repayment plan in earlier budgets and  does not maintain information that would be needed to identify past  estimates. (In appendix IV, we also compare Education’s IDR plan  subsidy cost estimates for the fiscal year 2017 budget with those  prepared for the fiscal year 2016 budget to illustrate how estimates  changed from one budget to the next.) We compared the supplemental  unpublished data to published data from the fiscal year 2016 and 2017  credit supplements to the President’s budget and interviewed Education  to clarify reasons for minor discrepancies.
section_title: Objective 2: Review of Education’s Approach to Estimating IDR Plan Costs and Sensitivity Analysis of Education’s IDR Repayment Model
subsections: section_title: Evaluation of Education’s Estimation Approach
paragraphs: To understand and evaluate Education’s approach to estimating the cost  of loans in IDR plans, we first reviewed available documentation from  Education on the supplementary model Education created to estimate  repayment patterns of loans in IDR plans (referred to as the IDR plan  repayment model in this appendix). We also reviewed documentation on  Education’s student loan model, which uses information from the IDR  repayment model and other assumptions to calculate total subsidy costs.  (See appendix II for detailed information on how Education estimates IDR  costs using these models.) This documentation provided limited details  regarding the steps of Education’s IDR repayment model or how  assumptions were operationalized and programmed in the model.
Given the limited documentation available regarding Education’s IDR plan  repayment model, we reviewed the computer programs and datasets  used in the model. Education provided us with SAS program files and  data input files used in the model. The data input files contained the  sample of Direct Loan borrowers Education used in its analysis as well as  estimated historical incomes of those borrowers provided by Treasury.  (See appendix III for more information on these historical income  estimates.) The SAS program files implementing the model forecasted  those borrowers’ future incomes and scheduled IDR plan payment  amounts, as well as forecasted events that would lead to non-payment,  such as default, death or disability, prepayment of loans through  consolidation, or forgiveness of loans through the Public Service Loan  Forgiveness (PSLF) program. To get further clarification on the  documentation, data, and computer programs provided, we interviewed  Education officials who created and manage the IDR plan repayment  model and the overall student loan model, which is used to calculate  subsidy costs for all Direct Loans.
We assessed the IDR plan repayment model’s major assumptions for  reasonableness and evaluated them against federal guidance for  estimating subsidy costs developed by the Federal Accounting Standards  Advisory Board. We evaluated methods used in the model, particularly  Education’s approach to forecasting borrower incomes, against this  guidance and accepted practices in statistics and the social sciences. We  also assessed whether the model appropriately replicated IDR plan  program rules. Finally, we conducted an in-depth review of the Treasury- created estimated historical income data used in Education’s approach.  We assessed the reasonableness of the data by conducting electronic  testing and producing summary statistics, which we asked Treasury to  compare to the actual taxpayer data upon which its estimates were  based. We reviewed related documentation from Treasury about the  estimation process, and interviewed Treasury officials to clarify factual  details and obtain their views on the process. (See appendix III for more  information on our review of these data and Education’s subsequent  forecasting approach.)
section_title: IDR Plan Repayment Model Sensitivity Analysis
paragraphs: Based on our detailed review of the assumptions, methods, and data  used in the IDR plan repayment model, we identified two separate areas  for testing the sensitivity of Education’s IDR plan cost estimates to  changes in assumptions. First, we tested the effects of inflation on income  projections and poverty guidelines, both of which are used to estimate  borrower payment amounts. We adjusted borrower incomes and poverty  guidelines for inflation due to the exclusion of inflation from Education’s  current model and the results of a prior Education analysis showing that  cost estimates were sensitive to changes in borrower incomes. Second,  we tested Education’s assumption about PSLF participation and the year  borrowers would be first eligible for forgiveness under the program. We  focused on PSLF participation because actual participation is not yet  known for this program and Education assumed that any borrower they  estimated to be eligible for PSLF would choose to participate. We carried  out each sensitivity analysis by rewriting relevant portions of the existing  SAS computer programs that Education developed to implement the IDR  plan repayment model.
To conduct these analyses, we first produced baseline cash flow  estimates using the existing programs we received from Education. We  sought to produce baseline estimates that were identical to those from  Education’s existing model. The baseline replication ensured that the new  model assumptions, rather than different versions of programs or input  data, were solely responsible for any changes in the estimates. The  replication process included selecting random samples of the data files  and using the SAS Compare procedure to detect any differences in  observations and variables. We interviewed Education officials to confirm  the sequence and versions of programs and to establish our final baseline  file.
After producing baseline estimates, we wrote two new sets of SAS  program files to implement each new assumption and produce new cash  flow output for each analysis. The final output data, which we sent to  Education to produce subsidy rates, consisted of cash flows summed  across all borrowers in repayment within each fiscal year and within loan  population type (non-consolidated loans, loans consolidated from default, and loans consolidated not from default). We provided this output to  Education officials, who uploaded and ran the new estimates through the  larger student loan model. Education officials provided revised subsidy  rates for each loan type and origination cohort, reflecting the new IDR  plan cash flows under our alternative assumptions. For each sensitivity  analysis, we compared the baseline and revised IDR cash flows and  subsidy cost estimates and calculated the percent change.
We tested Education’s assumptions regarding borrower participation and  the first year that borrowers are eligible for PSLF. Education estimates  borrower eligibility for PSLF using survey data that may not be  representative of borrowers in newer IDR plans. In addition, Education  assumes that 100 percent of borrowers who are estimated to be eligible  for PSLF will choose to participate after making 120 payments in a  qualifying repayment plan. Lastly, Education assumes that no borrower  will become eligible to benefit from PSLF until a year after the program is  scheduled to begin. To assess the impact of altering these three  assumptions, we increased and decreased the estimated percentage of  borrowers eligible and participating in PSLF by 10 and 5 percentage  points, and moved up the PSLF start date by a full year.
Adjusting Projected Incomes and Poverty Guidelines for Inflation  We tested the extent to which cost estimates were sensitive to adjusting  incomes and poverty guidelines for inflation for future years after 2013. As  described in appendix II, Education forecasts borrowers’ incomes by  substituting the historical incomes of borrowers with similar  characteristics, but does it not adjust these projected incomes for  inflation. Education also uses 2013 poverty guideline data for future  years, with no inflation adjustment. To implement this adjustment, we  obtained inflation factors from OMB for all future repayment years, and  inflated Education’s forecasted borrower incomes and poverty guidelines  into the appropriate year’s dollar units. Specifically, we applied  adjustment factors to the 2013 dollar amounts to inflate them to each  future year’s dollar units. We then applied the existing repayment model  using the inflated incomes and poverty guidelines as input, without  altering any additional model assumptions or calculations.
section_title: Calculating IDR Plan Forgiveness Amounts
paragraphs: To calculate expected forgiveness amounts for loans entering repayment  in fiscal years 1995 through 2017, we analyzed cash flow data from  Education, which provided detailed information on the amount of loan  principal expected to be paid and not repaid. First, we determined the  overall amount of loan principal in IDR plans estimated not to be repaid  for any reason, as Education recommended. We did this by subtracting  the amount of principal expected to be repaid from the total volume of  loans disbursed to borrowers. The remaining amount represented loan  principal estimated not to be repaid. We then subtracted the amount of  loan principal estimated to be discharged due to a borrower’s death or  disability. We attributed the remaining balance of unpaid principal to loan  forgiveness under IDR plans and PSLF. Because Education expects to  recover all defaulted loan principal through the collections process, loan  defaults did not contribute to total non-payment of loan principal.
section_title: Evaluation of Education’s Quality Control and Information Sharing Practices
paragraphs: We assessed Education’s quality control practices by reviewing relevant  documentation and interviewing officials in the office responsible for  developing and managing the estimates. We evaluated Education’s  practices against federal guidance related to estimating subsidy costs and  standards for internal control in the federal government. We also  assessed Education’s information sharing against standards for internal  control in the federal government, and Education’s strategic plan.
We conducted this performance audit from March 2015 to November  2016 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Description of Education’s Approach to Estimating Costs of Income- Driven Repayment Plans
subsections: section_title: Education’s Approach to Estimating Costs of Loans in Income-Driven Repayment Plans
paragraphs: Based on our review of Education’s computer programs, model  documentation, and interviews with agency officials, we confirmed that  Education estimates subsidy costs for loans in Income-Driven Repayment  (IDR) plans in the following way. First, Education estimates how many  loan dollars will enter IDR plans from each loan cohort. Second,  Education estimates repayment patterns for those loans over time. It  performs this first task within its larger student loan model that calculates  cash flows for cohorts of loans and incorporates various assumptions  about the future. Education addresses the second task inside a  supplementary microsimulation model for estimating IDR plan repayment  patterns—referred to as the IDR plan repayment model in this appendix— that was designed to predict the repayment behavior of individual  borrowers from a sample of borrowers with loans in IDR plans. Through  interviews, Education officials stated that they combine the resulting  pieces of information in their larger student loan model to generate  subsidy cost estimates.
subsections: section_title: Estimating How Many Loan Dollars Will Enter IDR Plans
paragraphs: Education estimates the percentage of loans in each cohort that will enter  each repayment plan—Standard, Extended, Graduated, and IDR—inside  its student loan model. According to Education’s model documentation  and follow-up information from agency officials, Education based its cost  estimates reported in the President’s fiscal year 2017 budget on a  random sample of loans drawn from its National Student Loan Data  System in January 2015. For loans issued after September 2014,  Education applied repayment plan participation rates from a past cohort.  For Consolidation loans, Education used 2014 cohort data because  borrowers generally begin repaying those loans immediately. For non- consolidated loans, which generally do not enter repayment for several  years while borrowers are in school, Education used participation rates  from the 2011 cohort. IDR plan participation rates for the 2015 through  2017 cohorts were adjusted upward in comparison to the 2014 cohort to  account for expanded eligibility for two newer IDR plans (Pay As You  Earn and Revised Pay As You Earn). For the fiscal year 2017 budget, this  upward adjustment ranged from 1.4 to 6.2 percent, depending on the  cohort and type of loan. Education officials stated that they then apply the  percentage of loans assumed to enter IDR plans to the total dollar value  of loans originated (or loan volume) in each loan cohort.
section_title: Estimating Repayment Patterns for Loans in IDR Plans
paragraphs: Education uses a separate IDR plan repayment model to forecast cash  flows, which we refer to as repayment patterns, of loans in IDR plans  based on a sample of borrowers with loans in repayment as of September  2013. This random sample of borrowers was also drawn from the  National Student Loan Database and reflected all loan activity through the  end of fiscal year 2013, including but not limited to, the amount borrowed,  loan type, and repayment plan. From this sample, Education selected all  borrowers who had already begun repaying their loans under an IDR plan  by September 2013. For the purpose of modeling future loan cohorts,  Education assumes all borrowers entering repayment after September  2013 will have the same characteristics as borrowers who entered  repayment in 2013.
Education then estimates how much each of these borrowers will owe on  their loans over a 31-year span, based on the borrower’s estimated  adjusted gross income (income) and family size for each year in the  repayment period and the rules of the IDR plan selected. Education used  a two-step process to forecast borrower income and family size:  1.  To forecast borrowers’ future incomes, Education first worked with  Treasury to estimate past incomes, filing status, and family sizes of  the sample of borrowers who had entered repayment by the end of  fiscal year 2013. Treasury developed these estimates because  Education does not collect tax data on all borrowers. Treasury collects  this information for all U.S. tax filers, but did not share actual data  from these borrowers’ tax filings due to privacy restrictions. Instead, it  created a tax file that contained substituted, or “imputed,” information  based on borrower characteristics including age, gender, loan  balance, dependency status and family income. Treasury first  estimated if a borrower would file taxes in a given year. For each  borrower estimated to file taxes, it then imputed estimated nominal  incomes and number of tax exemptions (approximating family size) for  each of the borrower’s repayment years that occurred in tax years  1996 through 2013. For example, borrowers who entered repayment  in 1996 would have 18 years of imputed incomes while borrowers  entering repayment in 2000 would have 14 years. (See Evaluating  Income Data Used in Education’s Approach in appendix III for more  information on Treasury’s methodology to estimate borrower incomes  and our assessment of error associated with its approach).  2.  To forecast future incomes of each borrower in its sample from 2014  through the end of the borrower’s repayment period (up to 31 years in  the future), Education first converted the estimated historical income  data from Treasury from calendar years into “repayment years.” A  borrower who began repaying his loan in calendar year 2000 would  have estimated historical income data covering repayment years 1  through 14 (formerly calendar years 2000 through 2013). To forecast  that borrower’s future income in repayment year 15, Education first  matched the borrower with a set of borrowers with similar  characteristics. Education then randomly selected a borrower from  this matched set of borrowers and substituted the nominal historical  income observation from the same repayment year. It repeated this  step for each subsequent year of the borrower’s maximum repayment  period, choosing a different borrower’s nominal historical income  observation in each year. Because Education matched the borrowers  in their file with Treasury’s based on their repayment year (as  opposed to calendar year), the nominal historical income values used  in the forecasts could come from various non-sequential calendar  years. (See Evaluating Income Data Used in Education’s Approach in  appendix III for more information on Education’s methodology and the  error associated with its approach).
Once Education forecasted incomes and family size for each borrower in  the sample’s entire repayment period, Education then applied the rules of  the borrower’s selected IDR plan to calculate the amount the borrower  would owe over time. For instance, a borrower in the New Income-Based  Repayment plan would pay 10 percent of her discretionary income for 20  years, whereas a borrower in the Income-Contingent Repayment plan  would pay 20 percent of her discretionary income for 25 years. For  borrowers in the file who had not yet selected an IDR plan as September  2013 (i.e., those estimated to enter repayment in 2014 or later),  Education selects a plan for them based on assumptions about borrower  behavior. For borrowers in the Income-Contingent Repayment plan, the  IDR plan repayment model annually reevaluates whether they will switch  into the Income-Based Repayment plan, and does so if the borrower’s  monthly payment amount would be lowered by at least $50 by switching  into the Income-Based Repayment plan. The IDR plan repayment model  switches a borrower into the Revised Pay As You Earn Plan if that  borrower is not eligible for the Pay As You Earn or New Income-Based  Repayment plans and if the borrower saw his or her payment fall by $50  compared to what would be owed under the Income-Based Repayment  plan. Borrowers were assumed to choose Revised Pay As You Earn if the  present value of the payments with Revised Pay As You Earn were lower  than the present value of payments without Revised Pay As You Earn  using a 30 percent discount rate. Education used a high discount rate  because borrowers would likely place much less weight on the higher  payments that would be likely to occur toward the end of the repayment  period. Borrowers already in IDR repayment were assumed to choose  Revised Pay As You Earn in the first year their payments fell by $50 a  month or more. Borrowers who had not yet chosen an IDR plan were  assumed to choose Revised Pay As You Earn if their payments would be  lower in the first year. Borrowers stay in an IDR plan for their entire  repayment period, even if their income rises beyond the point at which  they would qualify if they were applying in that year, in order to calculate  possible loan forgiveness amounts.
The IDR plan repayment model also includes predictions about when  borrowers will delay repaying their loans (through deferment and  forbearance); when they will fail to repay their loans (due to default,  death, and disability); when they will prepay their loans through  consolidation; and when their loan balances will be forgiven due to  participation in the Public Service Loan Forgiveness program or at the  end of their IDR plan’s full repayment term.
The IDR plan repayment model’s final output consists of cash flows  received (broken out by principal and interest) and foregone (such as  through default, death, and disability) for each of the 31 years of  repayment. These cash flows are summed across all borrowers who  enter repayment in the same year for three different groups: (1) borrowers  with Subsidized Stafford, Unsubsidized Stafford, and Graduate PLUS  loans, (2) borrowers who defaulted and then consolidated their loans, and  (3) borrowers who consolidated their loans without defaulting.
section_title: Exporting Cash Flows to Student Loan Model
paragraphs: Cash flows from the IDR plan repayment model are then exported to the  larger student loan model. According to Education officials, the student  loan model allocates these cash flows, which are organized by the year in  which loans enter repayment, back to the appropriate loan origination  cohorts using an assumption about the rate at which loans originated in a  given year will enter repayment. Education assumes that all loans being  repaid in IDR plans in a particular loan origination cohort will have the  same cash flow patterns as loans in the sample used in the IDR plan  repayment model.
section_title: Calculating Subsidy Costs
paragraphs: The student loan model discounts estimated cash flows to present value  using the Office of Management and Budget’s credit subsidy calculator  tool to determine the subsidy cost. The subsidy rate is determined by  taking the ratio of the subsidy cost to the volume of loan obligations  estimated to be made in that year.
section_title: Appendix III: Evaluation of Income Data Used in Education’s Approach
subsections: section_title: Income Data Used in Education’s Approach
paragraphs: The U.S. Department of Education’s (Education’s) supplementary model  for estimating Income-Driven Repayment (IDR) plan repayment  patterns—referred to as the IDR plan repayment model—blends  statistical analysis and assumptions about the future behavior of  borrowers.
subsections: section_title: General Use of Historical Income Estimates
paragraphs: Education uses data on borrowers’ historical incomes to estimate their  incomes in future repayment years. According to Education staff, the  previous version of the agency’s IDR plan repayment model used data  from the Current Population Survey, a general population sample survey  administered by the U.S. Census Bureau, which the agency matched to a  sample of borrower data. Education staff told us they searched for a  different source of income data beginning in 2013, due to the relatively  short 2-year Current Population Survey panel length. The short panel  required Education to combine incomes from different individuals and  Current Population Survey samples to project over enough repayment  years.
For Education’s current IDR plan repayment model, developed in 2015,  agency staff sought income and other data from federal income tax  returns, as collected by the U.S. Department of the Treasury (Treasury).  Taxpayer data offered the potential for more accurate data than matched  Current Population Survey data, which only covered a sample of  borrowers for relatively short 2-year period. (According to Education  officials, the Current Population Survey does not contain data on student  borrowing so the prior model had to assume that borrowers and non- borrowers had the same income patterns.) Despite the expected benefits  of using actual taxpayer data, Treasury staff indicated that rules  concerning taxpayer privacy prevented them from providing data on  actual borrower incomes directly.
Education staff said they initially hoped to receive data from Treasury that  matched borrowers’ actual incomes as closely as possible, perhaps with  a random distortion to protect taxpayer privacy. Staff mentioned  Education’s National Postsecondary Student Aid Study restricted-use file  as an example of a similar dataset. However, Treasury’s chosen approach involved imputing borrower income categories. Education staff  then requested that Treasury convert these categorical values into dollar- scaled incomes for use in Education’s IDR plan repayment model.
section_title: Methodology for Assembling Historical Income Estimates
paragraphs: Based on our review of documentation from Treasury and Education, to  assemble the data prior to imputation, Treasury matched a sample of  borrower data that Education drew from the National Student Loan Data  System containing loan information from September 2013 to their tax  return data for filing years 1996 through 2013. Treasury assumed that  borrowers did not file returns if they did not have matching tax return data  for a given year. Key tax variables matched to the file for loan modeling  purposes included adjusted gross income, number of exemptions, and  filing status, among others. The final matched dataset included  observations for approximately 1.3 million borrowers in each of 18 tax  years.
After matching the files, Treasury used a data mining algorithm, known as  “graphical models,” to create an imputed version of the matched data.  According to Education staff, they asked Treasury to provide an imputed  dataset that resembled the actual data as closely as possible, for all of  the tax variables joined to their borrower records. Education staff said  they expected incomes to be accurate within categories but having  random distortion to preserve taxpayer privacy. Treasury staff told us that  they lacked the time and prior experience with Education’s data to have a  pre-existing model to meet these specifications. Instead, Treasury used  graphical methods to automatically identify a model that best fit the joint  distribution of the data across several variables and allowed for the  simulation of new imputed data. Treasury staff said that this approach  was simpler than what they might have done given more time, but it is  unclear whether greater complexity in the model would have yielded  better results.
Based on our review of documentation and interviews with Treasury staff,  Treasury’s exact method of imputation had several steps. First, Treasury  rescaled all variables from their natural scales into discrete categories,  which primarily affected borrower incomes, which are naturally measured  in continuous nominal dollars. Using categories of incomes rather than  the continuous dollar scale allowed Treasury to develop a model using  graphical methods that required less computing power. Second, Treasury  used graphical methods to identify the relationships (or dependencies)  among the borrower and tax variables in the matched data, in the form of  multivariate crosstabulations. The model first estimated the probability  that a borrower would file a tax return in a given year, and then modeled  the joint distribution of the data, given that the first-stage model estimated  that a borrower would file a return. After estimating these  crosstabulations, Treasury created a single imputed dataset by drawing  random variates from the fitted joint distribution of the data, in order to  replace records in the actual dataset.
When imputing incomes, Treasury staff told us they took an additional  step to transform the imputed income variables from a categorical to  continuous scale by drawing random dollar values from probability  distributions. For borrowers with incomes imputed in the lowest and  highest categories, Treasury simulated continuous incomes by drawing  from normal or log-normal distributions with moments set to their sample  values. For borrowers in all other categories, Treasury drew independent  random variates from uniform distributions with support on the range of  each imputed category.
According to Treasury staff, they constrained the imputation model to  replicate some of the longitudinal structure in the tax data over time within  borrowers. The model imputed a borrower’s income in the current year  based on the borrower’s one-year lagged income, which generally  ensures that the imputed data recreates the correlation between incomes  in adjacent years (i.e., the first-order autocorrelation). In addition,  Treasury staff said that the imputation variables were stratified by year, in  order to allow the conditional distributions to vary over time.
section_title: Error in Historical Income Estimates
paragraphs: Any imputed data will have imputation error, the amount of which  depends on the predictive power of the model or method used to create  them. According to statistical theory, imputation models can produce  imputed datasets that are systematically biased, in the sense that the  imputed distribution of the data does not resemble the actual distribution  across many imputations. Imputation can also produce imputed datasets  that are unbiased but have a high degree of imputation error. (More  specifically, the variance of a model’s posterior predictive distribution can  be large.) In these cases, the imputed distribution of the data will  resemble the actual distribution across many imputations, but the imputed  distribution in any one sample of imputed data, often one random  simulation, may be quite different. Measuring error can involve  calculating confidence intervals. A larger confidence interval relative to  the estimate would suggest imputed data that are more prone to error. A  user of imputed data typically would consider the size of a confidence  interval as one criterion when assessing whether imputed data are  sufficiently precise for a specific application, along with the imputation  model’s potential bias and the user’s tolerance for error.
These two types of imputation error can affect the analysis of imputed  data. Ordinary methods of statistical analysis generally assume that  variables (like borrower incomes) are measured without error—an  assumption that is clearly not valid when analyzing imputed data.  Analyses that do not account for imputation error can produce estimates  that are biased or more or less precise than ordinary statistical theory  would imply, depending on the nature of the analysis.
To address these features of imputed data, generally accepted statistical  practices suggest a number of methods for the analysis of imputed data.  One common method uses “multiple imputation” to impute the data  several times, producing “implicate datasets.” Implicates are randomly  generated copies of the imputed data, produced by the same imputation  model. The imputed data will vary randomly across implicates,  depending on the nature and precision of the imputation method, because  most imputation models have a partially random or probabilistic structure.  By assessing the degree to which analytical results vary across  implicates, analysts can incorporate the error of imputation when  estimating the error of their estimates more generally. Imputation error  “propagates” into other measures of precision, such as sampling error.
As an alternative or complement to assessing imputation error directly,  the statistical literature recommends that analysts use imputed data as  preliminary information, prior to replicating their analyses using actual  data. This approach applies to situations in which an analyst may not  access the underlying data, but can provide computer code or algorithms  to another analyst who may access the actual data and replicate the  work.
Education staff told us they did not request information that would have  allowed them to assess how imputation error would have affected its cost  estimates, nor did they provide their computer code to a Treasury analyst  who had access to the data to replicate their work. Education staff  conducted an informal assessment of the quality of the borrower income  data by reviewing correlations between incomes and key factors like  education and borrowing levels. They did not use more formal methods to  assess and address imputation error in their estimates, such as those  discussed above. In addition, Education staff did not thoroughly document  and evaluate the imputation methods that Treasury used, nor did they  request evidence of an adequate model fit. Instead, according to  Education staff, Treasury provided Education with limited documentation  which included a broad overview of the imputation approach. Treasury  developed detailed documentation of the imputation model and error at  our request, several months after Education had accepted the final  imputed data. Treasury staff reported having computing and other  resource constraints that affected their choice of models and methods.  These constraints would have affected Treasury’s capacity to run their  model multiple times and produce multiple implicate datasets.
section_title: Comparison of Estimated and Actual Income Data
paragraphs: Through our review of summary documentation and limited descriptive  and graphical analysis from Treasury, we found indications of imputation  error that Education may not find acceptable for its purposes. This error  relates to the imputation of incomes in the lowest and highest categories  as well as the longitudinal structure of borrower incomes over time. This  error warrants further evaluation by Education, given that the agency  sought income data that would resemble borrowers’ actual, historical  incomes as closely as possible, including accurate longitudinal profiles of  incomes. Accurate longitudinal profiles of income are important, because  Education’s IDR plan repayment model includes a number of calculations  at the borrower level, such as specific payment amounts and borrowers’  eligibility for specific IDR plans, which use the sequence of data within  borrowers over time as inputs and are re-calculated each year over the  repayment period.
Treasury staff told us they did not seek to impute incomes on the dollar  scale. Rather, staff imputed income categories, and then evaluated model  fit using the categorical distributions of the imputed and actual income  data. After developing and validating the categorical imputation model,  Treasury provided a simple transformation of the income categories into  dollar values, using the secondary imputation methods we describe  above. Treasury staff described this aspect of the imputation as a  practical solution to meet needs that Education clarified after Treasury  had developed the imputation method and its goals. Because dollar- scaled incomes were not originally specified, Treasury staff told us they  did not assess the fit of the imputed dollar-scaled incomes to the joint  distribution of the data.
subsections: section_title: Error in Dollar-Scaled Incomes in Highest and Lowest Categories
paragraphs: Treasury’s comparison of the imputed and actual income data indicates  the imputed categorical data generally resembled the actual data, but its  secondary step to produce dollar scaled data introduced additional error,  particularly for observations in the highest and lowest income categories.  Treasury provided us with tabulations and plots of the imputed and actual  data, along with predictive p-values. The frequency statistics showed  that the marginal and joint distributions of key variables were generally  similar in the imputed and actual data for the categorical data, but that the  secondary imputation of dollar scale incomes produced additional error  for borrowers in the lowest and highest income categories. As the  Treasury-provided summary statistics in table 1 show, the mean imputed  income was about 2.1 times smaller than the mean actual income among  borrowers who earned $12,000 or less. The imputed mean was about 1.9  times larger than the actual mean for borrowers in the highest income  category. Treasury officials agreed that imputation error may be greater in  the lowest and highest categories, but speculated that the error may not  be practically consequential for the calculation of income-based loan  payments. However, because these data form the foundation of  numerous individual-level sequential calculations that determine what  borrowers are estimated to repay to the government, error associated  with the data should be measured and its effect on budget estimates  should be assessed.
By design, Treasury’s imputation model ensured that the correlation  between incomes in adjacent years was similar in the imputed and actual  data. Despite this important constraint, the model did not seek to  accurately impute complete, realistic profiles of dollar-scaled incomes  over time within the same borrower for all observed years. Consequently,  the imputed profiles of incomes over time within borrowers were not  designed to ensure that they resemble those observed in the actual data  at the individual level. Treasury staff confirmed this feature of the imputed  data.
Our limited exploratory analysis of the imputed dollar-scaled income data  revealed patterns consistent with these features of the imputation.  Incomes were less strongly correlated between adjacent years in the  imputed data than in the actual data, based on statistics that Treasury  staff provided. Specifically, let Var(AGIt | ADIt-1, AGIt-2, … , AGIt-k)  denote the marginal variance of incomes at time t, conditional on k  previous values. In the imputed data, the limited evidence available to us  suggests that the estimated conditional variance in the imputed data may  exceed the actual variance in the population of student loan borrowers.  The Pearson correlation between incomes in the current and previous  year, truncated to the interval of positive incomes less than $1 million,  was 0.84 in the actual data and 0.58 in the imputed data. In other words,  one measure of the year-to-year instability of incomes was about 44  percent larger than in the actual data. In addition, the absolute value of  incomes changed between adjacent years by 52 percent in the actual  data but 75 percent in the imputed data. Consistent with these aggregate  statistics, figure 19 shows how imputed incomes vary by large amounts  from year to year within the same 10 randomly selected borrowers  presented previously in the report, this time with their tax filing status  indicated. Figure 20 illustrates how imputed incomes vary by large  amounts from year to year within the same 60 randomly selected  borrowers.
The secondary imputation of incomes in dollars may explain the patterns  above. Simulating dollar-scaled incomes from a set of uniform, normal, or  log-normal distributions would have added some amount of  approximation error, potentially inflating the conditional variance above in  the imputed data. The degree of error would depend on how strongly the  actual income distribution within each category diverged from the  assumed distribution (e.g., its nonlinearity when simulated as uniform).  Consistent with this explanation, Treasury staff reported nearly identical  Pearson correlations between current and 1-year lagged categorical  incomes in the imputed and actual data, at 0.69 and 0.67, respectively.  The approximation error for continuous incomes may have compounded  across years when Treasury staff independently simulated incomes for  the same borrowers in adjacent years, without constraining the imputed  distribution to preserve the potential dependency of incomes across  second-order lags and higher.
Treasury staff emphasized that adjusted gross income can be more  volatile over time than other measures of income. Adjusted gross income  includes gross wages, business income, and asset income, among other  sources, as well as certain deductions and credits. According to Treasury  staff, adjusted gross income can vary more substantially over time than  other sources of income, such as wages, and that such variation is  common among upper-income filers. However, we found that the  absolute value of imputed adjusted gross incomes varied between  adjacent years by 15 to 77 percent for the middle 50 percent of the  sample of borrowers with adjusted gross incomes above 0 and less than  $400,000. The widespread nature of the volatility conflicts with an  explanation that emphasizes volatile sources of income, deductions, and  credits among borrowers with high incomes.
We did not receive sufficient information to fully evaluate the nature and  extent of imputation error in the Treasury data, and how it would affect  Education’s IDR plan cost estimates. For instance, we did not receive  Treasury’s computer code or the actual tax data. Instead, Treasury staff  described the analysis in interviews and written briefing slides, as well as  a 7-page summary of the analysis that they previously provided to  Education. The correlations and proportional change statistics above are  limited in their ability to fully describe complete profiles of incomes at the  individual borrower level and their dependence over time, because they  describe linear associations only between data from adjacent years.  Additional analysis, with full access to the imputation model code and tax  data and a thorough assessment of the longitudinal structure of incomes  within borrowers over time, is necessary to confirm the imputation error  suggested by the limited evidence we obtained from Treasury.
section_title: Forecasting Future Borrower Incomes and Characteristics
paragraphs: The IDR plan repayment model uses the imputed data on borrower  incomes and other characteristics to forecast these data for future  repayment years that have not yet occurred. The model uses a different  method of imputation, known as the “hot-deck,” to make these forecasts.  Below, we describe this method in detail and evaluate it against generally  accepted statistical practices.
subsections: section_title: Education’s Process of Forecasting Borrower Incomes and Other Characteristics
paragraphs: According to our review of the imputed data that Education received from  Treasury, the data could span a variable portion of each borrower’s  repayment history. Education received imputed data for tax years 1996  through 2013. For a borrower who entered repayment in 1996, this period  would span the entire historical repayment period through 2013, but it  would not cover future years when the borrower may still be in  repayment. Conversely, for a borrower who entered repayment in 1986  and repaid all debt in 2000, the data would span the last four historical  years but not the first 10. Many other types of overlap are possible.
The repayment model uses these historical data and hot-deck methods to  impute or forecast data for repayment years that have not yet occurred.  The hot-deck is a general purpose method of imputation, which statistical  organizations commonly use to impute missing survey data. For a set of  records needing imputation, hot-deck methods use a set of covariates to  identify one or more records in the data with observed values on all  variables, which is similar to the record needing imputation. The method  then substitutes the observed values for the values needing imputation,  often using random selection among the donor records when multiple  donor records are available.
Once the repayment model forecasts data for unobserved repayment  years, it treats them as known, observed data. The repayment model  uses the forecasted data as input for the second stage of modeling, which  applies various assumptions about how borrowers will repay their loans  over time. The second stage modeling incorporates neither the error  associated with Treasury’s imputation of the matched tax and loan data  nor the error associated with Education’s hot-deck forecasting.
section_title: Education’s Forecasting Process and Prediction Error
paragraphs: As mentioned previously, any method of imputing or forecasting unknown  observations will have error associated with its predictions. Although the  nature of this error depends on the method and data, generally accepted  statistical practices typically recommend quantifying the error and  incorporating it into subsequent analysis of the predictions. Education’s  method of forecasting borrower incomes does not quantify the error  associated with the method or incorporate it into subsequent analyses.
Education’s analysis seeks to forecast the values of several variables,  most notably income, for a set of borrowers over up to 31 future  repayment years. One could view this as either a longitudinal econometric  modeling problem or a general purpose imputation of missing data. Using  either approach, accepted statistical practice involves quantifying and  propagating the error that is inherently associated with prediction.
An econometric approach would use an explicit statistical model for how  the forecasted variable depends on several other variables (or  covariates). Additional assumptions would describe how the imputed  variable varies over time, either through covariates (such as time  indicators) or assumptions about the variable’s random fluctuation around  its long-term mean (such as an error term with an autoregressive order 1  structure). These model assumptions provide explicit formulas to predict  future values of the variable and to quantify the likely error of prediction.  The latter formulas for prediction error (or the posterior predictive  distribution) can allow analysts to propagate the error of prediction into  subsequent analyses of those predictions.
An alternative approach would use more advanced methods of imputing  missing data, such as multiple, maximum likelihood, or expectation- maximization imputation. These methods assume an explicit probability  model for the joint distribution of the data, with parameters that can be  estimated from the data. Analysts can use various methods for drawing  from the fitted data distribution, in order to generate multiple implicate  datasets, as discussed above. This allows for analysts to quantify and  propagate error across subsequent analyses of the imputed data.
Education’s application of the hot-deck method does not follow these  general statistical principles. The method imputes the future values of all unknown data, using donor cases that are similar on a set of covariates,  such as gender and highest educational program level. After making  predictions, the method does not quantify the prediction error of the  estimates, using in-sample statistics such as mean-squared error,  misclassification rates, deviance statistics, predictive p-values, or the  estimated variance of the posterior predictive distribution.
Once the repayment model forecasts the income data for future years, it  assumes that the estimates have zero prediction error associated with  them, or equivalently, that the error does not affect the repayment  model’s loan cost estimates. Since any applied method of forecasting or  imputation produces error, and Education’s IDR plan cost estimates are  highly sensitive to changes in borrower income forecasts, it is important  for Education to measure this error and determine its ultimate impact on  IDR plan cost estimates.
Moreover, the IDR repayment model uses source data that have their  own unquantified imputation error from Treasury’s imputation. These two  sources of error—Treasury’s imputation and Education’s forecast—may  interact and combine in ways that further increase the bias and  imprecision of Education’s loan cost estimates. The presence of multiple  forms of error, at different stages of analysis, emphasizes the importance  of propagating all sources of error through the entire analysis, or else  eliminating imputation error from the imputed data by using actual  observations. At a minimum, Education should acknowledge the  presence of imputation error and identify how it might affect estimates  from the repayment model. Such acknowledgements would provide more  transparent information to users of its estimates, compared to point  estimates that do not disclose the limitations of the source data.
Statistical organizations accept the need for users of imputed data, such  as Education, to quantify and assess the effects of imputation error,  despite their release of public data that have been imputed. The U.S.  Census Bureau warns users that methods of estimating sampling error  will underestimate total error when data have been imputed. In recent  years, the Census has generated imputed data for several surveys, the  Survey of Income and Program Participation and Longitudinal Business  Database, but has warned that analyzing imputed data without necessary corrections may understate the variance of estimates. This guidance to  data users is consistent with the criteria discussed above, which  recommend quantifying and propagating imputation error, despite  statistical agencies’ widespread use of imputed data in public data  products.
section_title: Appendix IV: Supplemental Direct Loan Subsidy Cost Data
paragraphs: The following tables include a summary of available loan cohort data  underlying the U.S. Department of Education’s (Education’s) submissions  to the President’s fiscal year 2016 and fiscal year 2017 budgets. These  tables are provided to illustrate how Education’s estimates of IDR plan  costs shifted over the past two President’s budgets. Some of the  differences are attributable to the change in Education’s methodology for  estimating IDR plan costs, which was implemented for the President’s  fiscal year 2017 budget and is described in this report. Other differences  are due to the policy assumptions in place when the budgets were  developed. Specifically, for the fiscal year 2016 budget, Education used  provisional policies for its newest IDR plan that were under negotiation.  Estimates prepared for both the President’s fiscal year 2016 and fiscal  year 2017 budgets included legislative proposals affecting new  borrowers. Finally, the fiscal year 2017 budget estimates include  increased costs associated with the addition of the 2017 loan cohort, as  well as the updated current reestimated costs of older cohorts.
section_title: Appendix V: Comments from the U.S. Department of Education
section_title: Appendix VI: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Kris Nguyen (Assistant Director),  Ellen Phelps Ranen (Analyst-in-Charge), Rachel Beers, James Bennett,  John Karikari, Karissa Robie, Amber Sinclair, and Jeff Tessin made key  contributions to this report. Additional assistance was provided by  Deborah Bland, Jessica Botsford, Russ Burnett, Marcia Carlsen, David  Chrisinger, Cole Haase, Carol Henn, Susan J. Irving, Marissa Jones,  Sheila McCoy, Erin McLaughlin, Jeffrey G. Miller, Andrew Nelson, Jason  Palmer, Jessica Rider, Amrita Sen, Brian Schwartz, Michelle St. Pierre,  Adam Wendel, Charlie Willson, and Rebecca Woiwode.