Asylum: Variation Exists in Outcomes of Applications Across Immigration Courts and Judges
Released: 2016-11-14T00:00:00

Letter

Background

DOJ’s Process for Adjudicating Asylum Applications

Immigration court hearings are adversarial proceedings in which an individual appears before an immigration judge for adjudication of charges of removability from the United States, and may raise defense to removal by seeking asylum and other forms of relief or protection from removal. Immigration judges are to assess whether an applicant has credibly demonstrated that he or she is eligible to be granted asylum, and whether a grant of asylum is warranted as a matter of discretion. An applicant is eligible for asylum if he or she (1) applies from within the United States; (2) suffered past persecution, or has a well-founded fear of future persecution, based on race, religion, nationality, membership in a particular social group, or political opinion; and (3) is not statutorily barred from applying for or being granted asylum. When making a credibility determination, the immigration judge is to consider the totality of the circumstances and all relevant factors. EOIR follows the same procedures for defensive asylum applications and affirmative asylum referrals from USCIS. For affirmative asylum referrals, the immigration judge reviews the case de novo, meaning that the judge evaluates the applicant’s affirmative asylum application anew and is not bound by the USCIS asylum officer’s previous determination. As shown in Figure 1, in conducting removal proceedings and adjudicating cases before them, judges in immigration courts conduct an initial master calendar hearing to, among other things, ensure the applicant understands the immigration court proceedings and schedule the merits hearing, during which the judge hears claims for immigration relief, such as asylum.

An individual in removal proceedings in immigration court may request multiple types of relief before an immigration judge, and, if deemed removable, the judge makes a decision as to whether the individual satisfies the applicable eligibility requirements of any requested relief. During the merits hearing, judges hear testimony from the respondent and any other witnesses, oversee cross-examinations by DHS’s U.S. Immigration and Customs Enforcement (ICE) trial attorneys, and review relevant evidence, including evidence related to claims for asylum or other relief. Additionally, the immigration judge may question the applicant or other witnesses. Judges render oral or written decisions at the end of immigration court proceedings. As a general matter, immigration judges have discretion in rendering decisions under 8 C.F.R. § 1003.10(b), which states that “n deciding the individual case before them, and subject to the applicable governing standards, immigration judges shall exercise their independent judgment and discretion and may take any action consistent with their authorities under the Act and regulations that is appropriate and necessary for the disposition of such cases.” In addition, asylum is a discretionary form of relief under U.S. immigration law, in contrast to a mandatory type of immigration relief or protection, such as withholding of removal. With respect to asylum, section 208(a) of the Immigration and Nationality Act (INA) provides that the Attorney General or the Secretary of Homeland Security may grant asylum to an applicant who meets the definition of a refugee under INA § 101(a)(42)(A).

Potential Outcomes of Asylum Applications in Immigration Court

There are five possible outcomes of an asylum application—grant, denial, withdrawal, abandonment, or other. If a judge grants asylum, the asylee is eligible to apply for adjustment to lawful permanent resident status after one year. Individuals who have been granted asylum are considered qualified individuals for the purpose of eligibility for federal, and state or local, public benefits. Asylees are authorized for employment in the United States as a result of their asylum status. The other four possible outcomes of an asylum application—denial, withdrawal, abandonment, and other—do not result in an asylum grant, and therefore do not convey such benefits. According to EOIR headquarters officials, the outcome of an individual’s asylum application is not necessarily determinative of the outcome of the individual’s overall removal case. For example, an individual whose asylum application ends in an outcome of denial, withdrawal, or other may still be granted other forms of relief or protection from removal, such as withholding or cancellation of removal. Or, the individual may be found removable and not eligible for any form of relief or protection from removal. In this latter case, the individual would be subject to removal pursuant to the judge’s order of removal once it has become administratively final. According to EOIR headquarters officials, an outcome of other for an asylum application is recorded if the judge did not need to reach a decision on the merits of the asylum application. For example, ICE’s Office of the Principal Legal Advisor may exercise prosecutorial discretion in seeking, and the immigration judge may grant, administrative closure of removal proceedings for an individual whose case is not an enforcement priority for DHS.

EOIR’s Legal Access Programs

EOIR’s OLAP oversees various programs aimed at increasing access to legal services and information for individuals appearing before immigration court, as well as improving the effectiveness of EOIR’s adjudication process. According to EOIR, since April 2000, OLAP has worked to improve access to legal information and counseling and increase the level of representation for individuals appearing before the immigration courts. This has been carried out primarily through initiatives that facilitate access to information and help create new incentives for attorneys, non-profit organizations and their representatives, and law students to accept pro bono cases. In particular, OLAP established LOP and LOPC to provide legal information to targeted populations regarding immigration court processes and possible forms of relief, including asylum. LOP, which EOIR established in 2003, provides legal information to individuals in detention who are awaiting their removal proceedings regarding their rights and the immigration process to assist them in making better informed decisions earlier in immigration court proceedings. LOPC, which EOIR established in 2010, provides legal orientation presentations to custodians of unaccompanied alien children (UAC) released from custody from the Office of Refugee Resettlement, to ensure the child’s appearance at all immigration court hearings. To provide LOP and LOPC services, OLAP contracts with the Vera Institute of Justice (Vera), which subcontracts with non-profit organizations across the country. OLAP has also established other legal access services, including self-help materials and other programs intended to increase pro bono representation and serve vulnerable populations.

EOIR Data Indicate that Outcomes of Completed Asylum Applications Varied Over Time and Across Immigration Courts and Judges

Asylum Application Outcomes

A total of 740,922 individuals with asylum applications that were completed between fiscal years 1995 and 2014 met the criteria to be included in our sample. Of those, 595,795 (or 80 percent) were included in our analysis of asylum application completions and asylum grant rates. Since 2008, when we last reported on asylum outcomes, EOIR’s immigration judges granted asylum to between 4,508 and 6,090 individuals in our sample each year.

The number of individuals granted affirmative asylum each year has ranged from 3,060 to 4,166. The number of individuals denied affirmative asylum decreased each fiscal year from fiscal year 2008 to fiscal year 2014. Figure 2 illustrates outcomes of completed affirmative asylum applications during this period.

Since our 2008 report, the number of defensive asylum applications that were granted by immigration judges has ranged from 1,343 to 1,924 per year (fiscal years 2008 through 2014). As illustrated in Figure 3, the number of defensive asylum applications that were denied increased each fiscal year from 2011 to 2014. The number of defensive asylum applications with outcomes of “other” increased sharply from 828 in fiscal year 2009 to 3,960 in fiscal year 2014. As previously discussed, an outcome of “other” for an asylum application is recorded if the immigration judge did not need to reach a decision on the merits of the application. EOIR officials stated that the impact of DHS’s 2011 memorandum on prosecutorial discretion for immigration enforcement is a key reason for this increase—when DHS exercises prosecutorial discretion during removal proceedings, judges are able to administratively close removal cases without reaching a decision on the merits of the case, including associated asylum applications.

Asylum Grant Rates

For fiscal years 1995 through 2014, EOIR data indicate that affirmative and defensive asylum grant rates varied over time and across immigration courts, applicants’ country of nationality, and individual immigration judges within courts. In 2008, we reported the asylum grant rate as the number of asylum applications granted divided by the total number of all granted and denied applications. In other words, the grant rate depended on the number of completed asylum applications that ended in a decision by the immigration judge of either grant or denial. In this report, to provide data on all of the outcomes of completed asylum applications during our period of analysis, we included the five possible outcomes in calculating the asylum grant rate. These include applications that were granted and denied, as well as those that had an outcome of withdrawn, abandoned, or other. Thus, for the purposes of this report, we define the asylum “grant rate” as the number of granted asylum applications divided by the total number of completed applications (completions include applications that are granted, denied, withdrawn, abandoned, or end in an outcome of “other”) during the period of analysis. An expanded definition allows us to describe the outcomes of completed asylum applications, including outcomes other than granted or denied, and trends in those outcomes. Using this definition, annual grant rates for affirmative asylum applications ranged from 21 to 44 percent from fiscal year 2008 through fiscal year 2014. In the same time period, grant rates for defensive asylum applications ranged from 15 to 26 percent. Figure 4 illustrates asylum grant rates for fiscal years 1995 through 2014.

In 2008, we reported that affirmative asylum grant rates varied by applicants’ country of nationality. Country conditions vary from one country to another, including countries’ political climates and human rights records. Thus, differences in the extent to which applicants from various countries are granted or denied asylum in the United States is not unexpected. For the period since our 2008 report, from May 2007 through fiscal year 2014, affirmative asylum grant rates continued to vary by applicant nationality, as shown in Figure 5. For example, the grant rate for affirmative applicants exceeded 50 percent for asylum applicants from some countries, including Sudan, Cameroon, Burma (Myanmar), Sri Lanka, Yugoslavia, China, Ethiopia, Egypt, and Eritrea. For other countries, the affirmative asylum grant rate was less than 5 percent, including El Salvador, Cuba, Mexico, Ecuador, Nicaragua, Guatemala, Honduras, and Philippines.

In 2008, we reported that defensive asylum grant rates also varied by applicants’ country of nationality. From May 2007 through fiscal year 2014, defensive asylum grant rates continued to vary by country of nationality. For example, the grant rate for defensive applicants exceeded 50 percent for asylum applicants from some countries, including Cameroon, Iraq, China, Burma (Myanmar), Ethiopia, and Eritrea (see fig. 6). For other countries, the defensive asylum grant rate was less than 5 percent, including Laos, Philippines, and Mexico.

In 2008, we reported that affirmative and defensive asylum grant rates varied depending on the immigration court that heard the case. We found similar levels of variation from May 2007 through fiscal year 2014. For example, the grant rate for affirmative applicants in the New York immigration court was 66 percent, while the grant rate in the Omaha and Atlanta immigration courts was less than 5 percent (see fig. 7).

From May 2007 through fiscal year 2014, defensive asylum grant rates also continued to vary depending on the immigration court that heard the case. For example, the grant rate in the New York immigration court was 52 percent, while the grant rates in the Omaha, Atlanta, and Bloomington, Minnesota, courts were less than 5 percent (see fig. 8).

From May 2007 through fiscal year 2014, within each immigration court, the asylum grant rate for individual immigration judges varied. Affirmative asylum grant rates for individual judges ranged from 0 to 83 percent. Among the four courts with more than 15 judges in our sample, the difference between the lowest and highest grant rates in each court, or the “range,” extended from 17 percentage points in Miami to 60 percentage points in New York, as illustrated in Table 1.

Figure 9 illustrates the range of affirmative asylum grant rates for judges in their primary immigration courts from May 2007 through fiscal year 2014.

From May 2007 through fiscal year 2014, defensive asylum grant rates across individual judges ranged from 0 to 80 percent. Among the four courts with more than 15 judges in our sample, the difference between the lowest and highest grant rates in each court, or the “range,” extended from 16 percentage points in Miami to 70 percentage points in New York, as illustrated in Table 2.

Figure 10 illustrates the range of defensive asylum grant rates for judges in their primary immigration courts from May 2007 through fiscal year 2014.

Grant Rates for Asylum Applications Can be Affected by Case Characteristics and Outcome Variance Persists when Controlling for Certain Factors

Potential Explanations Offered for Variation in Outcomes of Asylum Applications

Immigration judges have discretion in rendering decisions and asylum is a discretionary form of relief under U.S. immigration law. Immigration judges we spoke with in six immigration courts stated that the judicial discretion provided for in U.S. immigration law is one reason that decisions on asylum applications, even among judges in the same court, could vary. For example, while a judge adjudicating an asylum claim is permitted by law to require that corroborating evidence be provided in support of otherwise credible, persuasive, and factually specific testimony, such testimony could also be legally sufficient for another judge to determine that the applicant has met his or her burden of proof without corroboration. In addition, researchers have offered potential explanations for variation in asylum application outcomes. For example, a study published in 2015 by researchers at the University of Texas at Dallas found that variation in asylum application outcomes is not unexpected given the institutional conditions under which judges operate, including the particular requirements of the law, the lack of corroborating evidence that is common in asylum cases, and the difﬁculty of assessing credibility.

EOIR Data Show Variation in Outcomes of Asylum Applications Across Courts and Judges Persists when Holding Certain Case and Judge Characteristics Constant

On the basis of our analysis of EOIR data from fiscal years 1995 through 2014, we found that asylum grant rates varied across courts and judges, holding constant various case and judge characteristics. While we were able to analyze various characteristics of cases, judges, and courts, the design of EOIR’s case management system did not allow us to measure and control for a number of factors—such as the nature or key characteristics of the claim of persecution or the gender of the applicant— that may be relevant to variability in asylum grant rates. EOIR’s case management system was designed to manage and track workloads across immigration courts rather than to collect all data on the details of individual proceedings. In addition, each asylum application presents unique facts and circumstances that judges must consider in rendering their decisions. As a result, our analysis could not hold constant the underlying facts and merits of individual asylum applications because EOIR’s case management system does not collect that information. Further, according to EOIR officials, immigration judges are not required to document each factor in the case management system they consider in their overall adjudication of an asylum claim. Some of these factors may be relevant to variation in asylum application outcomes, such as whether there was an adverse credibility determination. In addition, there may be other factors that could affect variation in asylum grant rates including whether a particular judge was assigned to a juvenile or other docket that does not randomly assign cases to judges. Nonetheless, the data available allowed us to control for certain factors of each asylum application, which enabled us to compare asylum outcomes across immigration courts and judges.

Factors Associated with Grant Rates

We used multivariate statistical methods to estimate an expected asylum grant rate for a representative applicant, holding constant various characteristics of individual cases, judges, and courts. We estimated rates separately for affirmative and defensive asylum applications and for each of four time periods across fiscal years 1995 through 2014. Analyzing asylum grant rates for a representative applicant, holding constant factors such as whether the applicant was represented by counsel, applicant country of nationality, and judge gender, allowed us to compare asylum grant rates across courts and judges (see Appendix III for more details about our analysis).

For the period since our 2008 report (May 2007 through fiscal year 2014), we estimate that, for each of the factors noted below, a representative affirmative asylum applicant would be granted asylum at a rate higher (or lower) than an applicant with differing case factors, and the difference would be statistically significant. The remaining factors we held constant—judge gender, most circuit courts, and judge and court asylum caseloads—had no statistically distinguishable associations with affirmative asylum grant rates for cases heard in immigration court. For example:

Representation. Applicants who were represented by legal counsel were granted asylum at a rate 3.1 times higher than applicants who were not represented.

Dependents. Applicants with at least one dependent were granted asylum at a rate 1.7 times higher than applicants without dependents.

Date of affirmative asylum application. Applicants who applied within one year of entering the United States were granted asylum at a rate 2.4 times higher than applicants who applied later.

Presidential administration under which judges were appointed.

Judges who were appointed by the Attorney General during the administrations of Presidents George H.W. Bush, Reagan, Carter, Nixon, or Johnson were generally more likely to grant asylum than those appointed during the administrations of Presidents Clinton, George W. Bush, and Obama, holding constant years of experience as a judge. Judges appointed during the administrations of Presidents Clinton, George W. Bush, and Obama granted asylum at rates that were statistically indistinguishable from each other.

Judge years of experience. Judges with more experience were less likely to grant asylum, holding constant the presidential administration of appointment. Judges with an additional 7 years of experience were 28 percent less likely to grant asylum.

Many of the factors we analyzed had similar associations for both affirmative and defensive asylum grant rates. From May 2007 through fiscal year 2014, we estimate that, for each of the factors noted below, a representative defensive asylum applicant would be granted asylum at a rate higher (or lower) than an applicant with differing case factors, and the difference would be statistically significant. Judge experience and the presidential administration under which judges were appointed had no statistically distinguishable associations with defensive asylum grant rates. For example:

Representation. Applicants who were represented by legal counsel were granted asylum at a rate 1.8 times higher than applicants who were not represented.

Dependents. Applicants with at least one dependent were granted asylum at a rate 1.7 times higher than applicants without dependents.

Date of defensive asylum application. Applicants who applied within one year of entering the United States were granted asylum at a rate 5.0 times higher than applicants who applied later. This association is substantially larger than the association for affirmative applications made during the same time, and the difference is statistically distinguishable from zero.

Judge gender. Female judges granted asylum for defensive applications at a rate 1.4 times higher than male judges. There was no statistically meaningful association between judge gender and asylum grant rates among affirmative cases.

Asylum Grant Rates Varied Significantly Across Immigration Courts for a Representative Applicant After Controlling for Certain Factors

From May 2007 through fiscal year 2014, we estimated that the affirmative asylum grant rate would vary by 29 percentage points if different immigration courts heard the case of a representative applicant with the same average characteristics we measured. As shown in table 3, the grant rate for a representative applicant in one court would be 19 percent, whereas it would be 48 percent in another court for the same representative applicant. The size of this range varied across the four time periods we analyzed, from a low of 10 percentage points in the first period to a high of 37 percentage points in the second period.

For defensive asylum completions in the most recent period, we estimated that the defensive asylum grant rate would vary by 38 percentage points if different immigration courts heard the case of a representative applicant with the same average characteristics we measured. As shown in table 3, the grant rate for a representative applicant in one court would be 18 percent, whereas it would be 56 percent in another court for the same representative applicant. The size of this range varied across the four time periods we analyzed, from a low of 11 percentage points in the first period to a high of 38 percentage points in the third period.

Asylum Grant Rates Varied Significantly Across Immigration Judges for a Representative Applicant After Controlling for Certain Factors

From May 2007 through fiscal year 2014, we estimated that the affirmative asylum grant rate would vary by 47 percentage points if different immigration judges heard the case of a representative applicant with the same average characteristics we measured. As shown in table 4, the grant rate for a representative applicant by one judge would be 13 percent, whereas it would be 60 percent before another judge for the same representative applicant. The size of this range varied across the four time periods we analyzed, from a low of 29 percentage points in the first period to a high of 47 percentage points in the fourth period.

For completed defensive asylum applications in the most recent period, we estimated that the defensive asylum grant would vary by 57 percentage points if different immigration judges heard the case of a representative applicant with the same average characteristics we measured. As shown in table 4, the grant rate for a representative applicant by one judge would be 11 percent, whereas it would be 68 percent before another judge for the same representative applicant. The size of this range varied across the four time periods we analyzed, from a low of 22 percentage points in the first period to a high of 57 percentage points in the fourth period.

EOIR Has Programs in Place to Facilitate Access to Legal Resources for Asylum Applicants, but Could Improve Efforts to Measure Program Results

EOIR Facilitates Legal Resources for Asylum Applicants and Monitors Results

Through LOP and LOPC, EOIR has provided legal orientations to approximately 450,000 individuals. EOIR established LOP in 2003 to provide legal orientations to persons in detention prior to their first hearing before an immigration judge. In 2010, EOIR established LOPC to provide legal orientation presentations to custodians of UAC released from custody from the Office of Refugee Resettlement to help ensure the child’s appearance at all immigration proceedings. LOP and LOPC provide legal information through group and individual orientations, as well as self-help workshops and pro bono legal referrals to those in removal proceedings in immigration court. According to DOJ and EOIR, LOP and LOPC are key efforts to, among other things, increase the efficiency of immigration court proceedings and improve access to basic legal services for individuals in such proceedings. For example, DOJ’s Congressional Budget Submission for Fiscal Year 2016 reported that LOP is designed to assist detained individuals in making better informed decisions earlier in their immigration court proceedings, thereby improving access to basic legal services, especially for low income individuals, while increasing the efficiency of the court hearing and detention process. Regarding LOPC, in February 2016, EOIR’s Director testified that LOPC “helps the immigration court process function more efficiently and effectively by providing valuable information to the custodians of children who arrive in the United States without a parent or guardian.”

Since 2005, EOIR has contracted with the Vera Institute of Justice (Vera), which subcontracts with non-profit organizations, to administer LOP and LOPC. As of July 2016, EOIR administered LOP at 36 detention facilities, 2 non-detained locations and LOPC in 15 cities across the United States. According to EOIR’s September 2011 blanket purchase agreement with Vera and associated statements of work, LOP and LOPC at each site, are to provide, among other things, individual and group legal orientations and the dissemination of written legal orientation materials to individuals in immigration removal proceedings. LOP and LOPC sites provide self- help workshops, pro bono referral services and individual and group orientations. In particular, these orientations are to review the range of rights available to such individuals and the forms of relief from removal that may or may not be available to them. Table 5 provides information on the goals and objectives for LOP and LOPC in the 2011 blanket purchase agreement.

Consistent with the 2011 blanket purchase agreement, Vera submits quarterly reports to EOIR with data on, for example, the number of unique individuals who attended legal orientations and the number of referrals to pro bono legal services during the prior quarter. According to Vera’s quarterly report summarizing LOP data through fiscal year 2015, between about 35,000 and about 65,000 individuals attended LOP legal orientations per year from fiscal years 2008 through 2015 (see fig. 11). According to EOIR headquarters officials, the decrease in the number of individuals attending LOP orientations since fiscal year 2011 mirrors the decrease in the number of individuals detained and placed in removal proceedings during this time period.

Regarding LOPC, according to Vera’s quarterly report summarizing LOPC data through fiscal year 2015, legal orientations to custodians increased from about 1,500 in fiscal year 2011, when the program started, to almost 14,000 in fiscal year 2015 (see fig. 12).

EOIR headquarters officials stated that they review information that Vera provides through the quarterly reports to check compliance with contract requirements and to monitor the programs’ progress against stated objectives. In addition, according to EOIR officials, Vera and EOIR officials conduct site visits of LOP and LOPC sites to meet with local legal service organizations to observe presentations and workshops and discuss program performance. Vera also conducts monthly meetings with site staff. EOIR officials responsible for overseeing LOP and LOPC also noted that they hold regular discussions with Vera staff and use the information collected by Vera to monitor progress. In particular, EOIR officials stated that they monitor Vera’s data to identify increases or decreases in the number of individuals served in each LOP and LOPC location to determine site staffing levels or changes to existing sites.

EOIR Could Improve Efforts to Measure LOP and LOPC Effectiveness

EOIR monitors LOP and LOPC through a variety of mechanisms, but has not established a system of performance measures, including establishing a baseline, to regularly evaluate the effectiveness of LOP and LOPC to determine whether these programs are having a measurable impact in meeting program objectives. EOIR officials stated that they have monitored LOP and LOPC performance through site visits and monthly conference calls with Vera, as well as Vera’s quarterly reports, annual reports, and budget analyses. Regarding LOP, EOIR officials stated that they have used many of the findings and recommendations from a one-time evaluation of the program in 2008 as a guide for monitoring LOP performance. The evaluation, conducted by Vera at the request of EOIR, found that LOP participants moved faster through immigration courts, received fewer in absentia removal orders and that LOP could effectively prepare individuals to represent themselves in immigration courts, among other things. In the 2008 LOP evaluation, Vera also concluded that immigration judges stated that LOP increased immigration court efficiency and detention facility staff stated that LOP improved detention conditions. EOIR’s 2011 blanket purchase agreement related to LOP and LOPC states that EOIR can request that the Contractor (i.e., Vera) produce various deliverables as needed and/or as requested by EOIR, including an evaluation of LOP program outcomes and a “performance outcome measurement program” to accurately and continuously measure and obtain quantitative and qualitative data on LOP’s stated objectives. Although the 2008 evaluation provided EOIR officials with valuable insight into the effectiveness of the LOP program, EOIR officials stated that they have not requested that Vera conduct a similar comprehensive outcome performance measurement program consistent with the 2011 blanket purchase agreement. Further, as of July 2016, EOIR officials stated that they have not required Vera to develop a formal performance outcome measurement program, or other formal performance measures, for LOP as allowed for in the blanket purchase agreement because of the variation in how LOP providers operate programs at each site.

Regarding LOPC, the 2011 blanket purchase agreement states that, as specified in an executed blanket purchase agreement task order, Vera is to develop a system for measuring the performance of the program against its stated objectives by collecting program data on the number of UAC custodians served and the number of UACs represented by counsel, among other things. In addition, EOIR officials stated that they requested that Vera evaluate LOPC’s performance in 2014, consistent with the 2011 agreement. In August 2016, EOIR provided us a copy of the final evaluation report. Among other things, the report states that in absentia rates were lower for UAC whose custodians attended an LOPC orientation. In addition, custodians who Vera surveyed after attending an LOPC orientation reported learning about children’s legal rights and options for relief, as well as the importance of appearing in immigration court. In the report, Vera recommended that EOIR and Vera, among other things, consistently identify UAC in EOIR’s case management system and follow up with custodians to encourage attendance at orientations. EOIR officials stated in July 2016 that they do not plan to request another evaluation of LOPC, but will use these evaluation findings and recommendations to make any necessary program improvements.

Consistent with requirements outlined in GPRAMA, performance measurement is the ongoing monitoring and reporting of program accomplishments, particularly towards pre-established goals and agencies are to establish performance measures to assess progress towards goals. While GPRAMA is applicable to the department or agency level, performance goals and measures are important management tools to all levels of an agency, including the program, project or activity level. Agencies can use performance measurement to make various types of management decisions to improve programs and results, such as developing strategies and allocating resources, and identify problems and take corrective action. Because of its ongoing nature, performance measurement can serve as an early warning system to management and as a vehicle for improving accountability to the public.

EOIR headquarters officials stated that they have not established performance measures for LOP and LOPC because of the variation in how providers operate these programs at detention facilities and non- detained immigration courts, which results in the inability to have a “one- size fits all” approach. Rather, EOIR headquarters officials explained that program managers at headquarters regularly assess LOP and LOPC sites on an individual basis to (1) identify the number of individuals served by each full time staff position with the local subcontracted organization and (2) determine whether demand for services has increased or decreased. Based on this information, headquarters officials make necessary adjustments, such as closing a site with decreased demand for services and opening a new site with new demand. EOIR officials stated that this process provides flexibility to make changes at LOP and LOPC sites as needed in response to differing conditions at each location. However, EOIR has not established baselines for these data it collects from Vera and therefore does not have adequate performance measurements against which to compare the data to determine whether the programs are achieving stated goals. Solely tracking increases or decreases in program data, such as changes in the number of individuals served in detention facilities, does not allow EOIR to fully evaluate its LOP and LOPC programs as such changes in the data may not be an indicator of program success or increased efficacy. Developing and implementing performance measures, including establishing a baseline, to independently and periodically determine whether LOP and LOPC are having a measurable impact would better position EOIR to make any adjustments necessary to improve the programs’ performance.

Conclusions

EOIR’s primary mission is to adjudicate immigration cases in a careful and timely manner while ensuring the standards of due process and fair treatment for all parties involved. EOIR’s LOP and LOPC aim to achieve the dual goals of increasing the efficiency of immigration court proceeding, while also increasing individuals’ ability to make timely decisions about their immigration cases through receipt of early and accurate legal information. Our multivariate analysis of EOIR data on completed asylum applications from May 2007 through fiscal year 2014 indicates that whether an applicant has legal representation, in particular, can have a significant effect on the outcome of the application. Since 2008, EOIR has facilitated access to legal resources to thousands of individuals through LOP and LOPC legal orientations. While EOIR monitors LOP and LOPC through a variety of mechanisms, establishing a system of performance measures, including establishing a baseline, could better position EOIR to determine whether these programs are having a measurable impact in meeting program objectives and goals.

Recommendation for Executive Action

To better assess whether the LOP and LOPC are having a measurable impact in meeting their program objectives, the Director of EOIR should develop and implement a system of performance measures, including establishing a baseline, to regularly evaluate the effectiveness of LOP and LOPC and assess whether the programs are achieving their stated goals.

Agency Comments and Our Evaluation

We provided a draft of this report to DOJ and DHS for their review and comment. DOJ did not provide official written comments to include in this report. However, in an e-mail received on October 24, 2016, an Associate General Counsel for EOIR stated that EOIR concurred with our recommendation. DHS did not provide written comments on our draft report. DOJ and DHS provided technical comments, which we incorporated as appropriate.

Should you or your staff have questions about this report, please contact me at (202) 512-8777 or gamblerr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff that made key contributions to this report are listed in appendix IV.

Appendix I: Executive Office for Immigration Review (EOIR) Training for Immigration Judges

EOIR provides training to new immigration judges and annual training to all incumbent judges to assist them in rendering decisions on asylum applications. Since 2008, EOIR has expanded its training program for new immigration judges (see table 6).

In addition to training requirements for new immigration judges, EOIR holds annual conferences for all immigration judges at EOIR headquarters. EOIR held an annual conference in person for incumbent immigration judges in fiscal years 2008, 2009, 2010 and 2015. According to EOIR officials, EOIR did not host the annual conference in fiscal years 2011, 2012, and 2013 due to resource constraints. In fiscal year 2014, EOIR planned to hold the conference, but canceled it due to the increase in workload that resulted from the large numbers of unaccompanied children and families who were apprehended at the southwest border and placed in removal proceedings in the summer of 2014, according to EOIR officials. According to the EOIR headquarters official responsible for training, EOIR decides whether to have the conference each year based on available funding. During years when the conference was not offered in person, EOIR video recorded conference sessions on discs and mailed them to immigration judges to watch.

Appendix II: Objectives, Scope, and Methodology

Our objectives were to (1) describe the extent of variation in the outcomes of completed asylum applications over time and across immigration courts and judges; (2) discuss the factors associated with variability in the outcomes of completed asylum applications; and (3) examine the extent to which the Executive Office for Immigration Review (EOIR) has taken action to facilitate access to legal resources, including representation, for asylum applicants.

To address all three objectives, we reviewed EOIR documents describing asylum applications statistics; policies, procedures related to immigration court proceedings; manuals and documents describing EOIR’s case management system; guidance and training provided to judges for adjudicating asylum applications; and documentation on EOIR’s legal access programs. We interviewed EOIR headquarters officials responsible for overseeing immigration court proceedings; EOIR’s case management system; immigration judge training and guidance and legal access programs. We also visited 10 immigration courts in Tacoma, Washington; Seattle, Washington; New York, New York; Newark, New Jersey; Elizabeth, New Jersey; Los Angeles, California; Adelanto, California; Miami, Florida; Krome, Florida; and, Arlington, Virginia. At these locations, we interviewed supervisory immigration judges, immigration judges, court administrators and observed removal proceedings for individuals applying for asylum. We also interviewed supervisory immigration judges, immigration judges and court administrators by telephone at two additional immigration courts in El Paso, Texas and Atlanta, Georgia. Because immigration judges have large caseloads and constrained schedules, we interviewed judges who were available to speak with us during our scheduled interviews. We interviewed the court administrator and supervisory immigration judge for each immigration court we visited or contacted. During the immigration court interviews, we used semi-structured interview questions to ask about (1) potential factors associated with changes in asylum applications outcomes over time, (2) training and guidance provided to immigration judges for adjudicating asylum applications, and (3) EOIR immigration court efforts to facilitate access to legal resources for asylum seekers. We selected these sites based on a variety of factors, including courts that adjudicate a large number of asylum cases, detained and non-detained dockets, a range of grant rates, and circuit court jurisdictions. We also interviewed staff from U.S. Immigration and Customs Enforcement (ICE), Office of the Principal Legal Advisor, Offices of the Chief Counsel and asylum pro bono/advocacy organizations proximate to each immigration court interviewed to gain their perspectives on (1) potential factors associated with changes in asylum applications outcomes over time and (2) immigration court efforts to facilitate access to legal resources for asylum seekers. The results from our site visits cannot be generalized more broadly to all immigration courts or immigration judges. However, they provided important context and insights into EOIR’s efforts to assist immigration judges in adjudicating asylum applications, perspectives on training and guidance provided to immigration judges and EOIR efforts to facilitate access to legal resources.

Extent of Variation in Completed Asylum Applications

To describe the extent of variation in the outcomes of completed asylum applications, we analyzed data from EOIR about completions of asylum applications from fiscal year 1995 through fiscal year 2014, the most current data available at the time of our analysis. We obtained EOIR records of all immigration court removal proceedings that occurred during the period covered by our analysis. We selected those records where the immigration judge made the decision on the asylum application and eliminated decisions that were made after appeals. We selected only records for “lead” applicants, using the applicant’s alien number—a unique registration number that the Department of Homeland Security assigns to foreign nationals in immigration court proceedings—as the unit of analysis. We eliminated duplicate decisions for a spouse and/or dependent children because they derive from the decision on the lead applicant. As in our 2008 report, we selected the immigration courts and countries of nationality that contributed a minimum of 800 affirmative and 800 defensive asylum decisions on asylum applications from fiscal year 1995 through fiscal year 2014; our analysis in this report included applicants from 41 countries and 28 immigration courts. We selected the 800 minimum in each category to help ensure a sufficient number of completed asylum applications for our analysis. The results of our analysis cannot be generalized to asylum seekers from other countries or to other immigration courts. We used EOIR’s case management system to identify immigration court proceedings where an immigration judge had made a decision on an applicant’s asylum application. We assessed the reliability of the data used in our analyses through electronic testing, analyzing related database documentation, and working with agency officials to reconcile discrepancies between the data and documentation that we received. We determined that the EOIR data on completions of asylum applications were sufficiently reliable for the purposes of describing the extent of variation in the outcomes of completed asylum applications. We reported separately on affirmative and defensive completed asylum applications to control for characteristics shared by cases in each of those groups that could affect outcomes, such as whether the asylum application had already been reviewed by a U.S. Citizenship and Immigration Services (USCIS) asylum officer.

We also obtained biographical information from EOIR on those immigration judges who had served during the time period of our analysis. We merged these biographical data with the EOIR immigration court proceedings data. In doing so, we produced a dataset for the analysis that combined proceedings records with information on the characteristics of the applicants, the immigration judges, the immigration courts, and the completion or decision on the applicants’ asylum applications. We selected immigration judges with completions on at least 50 affirmative and 50 defensive asylum applications from fiscal year 1995 through fiscal year 2014. We selected the 50 minimum in each category to help ensure having judges who completed a sufficient number of asylum applications for our analysis. In analyzing differences in asylum decision across immigration judges, we excluded those immigration judges who heard fewer than 50 affirmative cases in our analyses of affirmative asylum decisions and fewer than 50 defensive cases in our analyses of defensive asylum decision. We also excluded cases heard by immigration judges other than in their primary court in order to simplify the presentation and avoid reaching inappropriate conclusions that can occur when calculations are based on small numbers of cases. We assessed the reliability of the data used in our analyses through electronic testing and working with agency officials to reconcile discrepancies between the data and documentation that we received. We determined that the EOIR immigration judge biographical data were sufficiently reliable for the purposes of producing a dataset that included characteristics of immigration judges in order to describe the extent of variation in the outcomes of completed asylum applications.

In 2008, we reported the asylum “grant rate” as the number of asylum applications granted divided by the total number of all granted and denied applications. In other words, the grant rate depended on the number of completed asylum applications that ended in a decision by the immigration judge of either grant or denial. In this report, to provide data on all of the outcomes of completed asylum applications during our period of analysis, we included the five possible outcomes in calculating the asylum grant rate. These include applications that were granted and denied, as well as those that had an outcome of withdrawn, abandoned, or other. Thus, for the purposes of this report, we define the asylum grant rate as the number of granted asylum applications divided by the total number of completed applications (completions include applications that are granted, denied, withdrawn, abandoned, or end in an outcome of “other”) during the period of analysis. An expanded definition allows us to describe the outcomes of completed asylum applications, including outcomes other than granted or denied, and trends in those outcomes. Although we analyzed data from fiscal year 1995 through fiscal year 2014, our reporting is focused on the time period from May 2007 through fiscal year 2014 because this is the most recent period and includes asylum completions since we last reported on asylum outcomes in 2008.

Factors Associated with Variability in Completed Asylum Applications

To discuss the factors associated with variability in the outcomes completed asylum applications, we used multivariate statistical modeling that held constant various other factors that could be associated with asylum grant rates. Our analysis used multivariate statistical methods to attribute the unique contribution of each factor to variation in asylum grant rates. Our analysis updated our previous work on this issue, which took a similar but not identical approach. Previously, we used multiple logistic regression to estimate the associations between various case, judge, and court factors and the probability that an applicant would receive asylum. Our current analysis generally takes the same approach to update our prior findings with more recent data on asylum applications completed from May 2007 through fiscal year 2014. However, we revised aspects of our data collection, preparation, and analysis to adjust for changes to the available administrative data and to allow for more precise reporting on the results. For a more detailed discussion of our multivariate analyses, see Appendix III.

As in our 2008 report, data limitations prevented us from fully isolating variation due to the unique judge or court. EOIR’s case management system collects information to meet EOIR’s administrative needs, such as the contact information of the applicant, the identity of the judge, and the languages spoken by the applicant. We used data from this system for statistical analysis because they are the primary data available on the population of interest. However, EOIR’s system does not collect data on all the details of individual proceedings that would be relevant to fully isolate variation across judges and courts. In particular, we cannot hold constant all relevant facts and circumstances of each case because EOIR’s case management system does not collect that information. Such facts and circumstances could be legally relevant and affect an applicant’s chance of receiving asylum. As a result, our estimates of the unique variation due to judges and courts may reflect circumstances of the cases that we cannot measure. Nonetheless, the data available allowed us to control for certain factors of each asylum application, enabling us to compare outcomes across immigration courts and judges.

The data available allowed us to hold constant certain characteristics of each case, such as applicant language, applicant country of nationality, judge experience, and court district. Accounting for these characteristics improves upon a simple comparison of asylum grant rates across judges and courts, without any adjustment. Table 7 below lists the factors used in our analysis and the source of the data. These characteristics may be correlated with the underlying facts and circumstances of each case, and therefore may indirectly explain factors we cannot measure. Data limitations prevented us from controlling for factors other than those listed in table 7 that could have contributed to variability in asylum application outcomes.

EOIR Actions to Facilitate Access to Legal Resources

To examine the extent to which EOIR has taken action to facilitate access to legal resources, we reviewed the Legal Orientation Program (LOP) and the Legal Orientation Program for Custodians of Unaccompanied Alien Children (LOPC) program documents and data, including the 2011 blanket purchase agreement and statements of work covering fiscal years 2013 through 2016; quarterly reports, annual reports; site visit reports, and evaluation reports; and, data reported by the Vera Institute of Justice (Vera) on the number of unique individuals served by LOP (fiscal years 2008 through 2015) and LOPC (fiscal years 2011 through 2015). We analyzed these time periods because our 2008 report reviewed EOIR data through 2007 (for LOP) and LOPC was established in 2010. We also reviewed documentation on EOIR’s other legal access services, including self-help materials and programs intended to increase pro bono representation and serve vulnerable populations, the BIA Pro Bono Project, the Model Hearing Program, National Qualified Representative Program and the justice AmeriCorps Legal Service for Unaccompanied Children program. To obtain additional context on the LOP and LOPC, we reviewed DOJ’s Fiscal Year 2016 Congressional Budget Submission and testimony statements about LOPC provided by EOIR’s director. We also interviewed headquarters officials responsible for overseeing the LOP and LOPC, as well as officials from Vera, who administer LOP and LOPC through a contract with EOIR. We analyzed the documentation on LOP and LOPC and information provided by EOIR officials on these programs in light of principles outlined in the Government Performance and Results Act (GPRA) of 1993, as updated by the GPRA Modernization Act (GPRAMA) of 2010, to assess EOIR’s efforts to measure LOP and LOPC progress and results against performance measures. We focused our review on the LOP and LOPC because these are among EOIR’s longest-standing legal access programs, they have received more funding than EOIR’s Office of Legal Access Programs’ (OLAP) other programs, and they have served tens of thousands of individuals since 2003 at immigration courts throughout the United States.

We conducted this performance audit from January 2015 to October 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix III: Multivariate Statistical Analysis of Outcomes of Completed Asylum Applications, Fiscal Years 1995 through 2014

To analyze how asylum grant rates varied across immigration judges and courts, we used multivariate statistical modeling that held constant various factors that could be associated with asylum grant rates. Below, we describe the goals of this analysis and the design and modeling methods we used. We describe the scope of our analysis and the data we analyzed in appendix II.

Goals of the Analysis

The primary goal of our analysis was to isolate variation in asylum grant rates due to the unique immigration judge or court assigned to each case. Variation in asylum grant rates could arise from the facts and circumstances of each case, and researchers have found that variation is not unexpected given the institutional conditions under which judges operate, including the particular requirements of the law, the standard under which a judge may request corroborating evidence, and the difﬁculty of assessing credibility. We sought to hold constant relevant variables at the case, judge, and court levels, to estimate the amount of residual variation that remained across immigration judges and courts.

Variation in asylum grant rates across immigration judges and courts could reflect various other factors that are statistically correlated with them. For example, if immigration courts tended to receive applications from asylum-seekers of the same country of nationality, asylum grant rates could potentially vary systematically based on nationality rather than on the unique nature of the judge or court. Similarly, the quality and accessibility of legal representation across the United States could potentially affect the outcomes of asylum applications. Since immigration judges’ decisions on asylum applications may incorporate many factors, such as credibility and the facts underlying the claims, simple univariate comparisons of asylum grant rates across judges and courts cannot distinguish among these various potential sources of variation.

Design and Modeling Methods

The data available from the Executive Office for Immigration Review’s (EOIR) case management system have a multilevel structure. Multiple applications for asylum are clustered within the same judges, courts, languages, and nationalities. These groups can be clustered among themselves, with or without nesting. Our data processing, discussed above, assigned one court to each judge, so that each judge was nested within the court in which they adjudicated the most cases. The remaining groups—courts, languages, and countries of nationality—were not necessarily nested. Each language could have been associated with multiple nationalities and vice versa. Each court could have been associated with multiple languages and countries of nationality; however, in practice, courts could have heard cases from a predominate combination of languages and nationalities, such as Chinese nationals who speak Mandarin and apply for asylum in New York.

The scope of our analysis included all completed asylum applications that met the screening criteria discussed in appendix II and that ended in outcomes of grant, deny, withdrawal, abandonment, or “other.” However, we dichotomized the outcome of interest to reflect whether the applicant was granted asylum or the application ended with any of the other outcomes. Expanding the scope of our sample to include all completed asylum applications rather than only applications that were granted or denied allowed us to analyze the outcomes of all completed asylum applications. Dichotomizing the outcome produced asylum grant rates as a proportion of applications originally filed that meet our screening criteria, which prevented bias from potentially changing withdrawal practices over time.

The data and outcome of interest suggested that a hierarchical or mixed logistic regression model would adequately reflect the data generation process. Accordingly, we developed a mixed model that represented the grouping variables—judge, court, country of nationality, and language, respectively—with random intercepts. Random effects allowed asylum grant rates to be correlated within and across groups, which parameterized our hypothesis that rates may systematically vary across groups, all else being equal on the factors we analyzed. Random effects made the model parsimonious because we could use the group variance parameters to concisely describe variation across groups with many levels. In contrast, modeling group variation with fixed effects would have required estimating several hundred explicit parameters, one for each group level, which would have consumed degrees of freedom and complicated model interpretation.

Substantively, random effects accurately represented the random assignment of judges to cases, which EOIR generally uses as an administrative policy. Random assignment of countries of nationality, languages, and courts has less of a direct substantive interpretation, but would be consistent with an underlying data generation process where the sample of asylum cases heard in any particular time period would be drawn from an underlying population distribution of immigration courts and applicant nationalities and languages.

We held constant case, judge, and court characteristics using covariates with fixed parameters. The smaller number of parameters associated with these covariates made a fixed effects approach easier to apply and interpret. We assumed that the covariate effects did not vary across groups, so that only the model’s intercept varies randomly. We had no prior expectation that specific covariate effects should have varied across groups. Moreover, increasing the number of random effects would have increased the complexity of the model and could have made it hard to estimate computationally.

We modeled the data separately by asylum type (affirmative and defensive) and time period. This allowed all model parameters to vary between asylum types and over time. The alternative approach, pooling the data, potentially would have obscured important differences in the affirmative and defensive asylum processes and important changes in the processes over the 20 years that our data spanned. Specifically, we modeled Yi, a dichotomous variable indicating whether applicant i in the affirmative or defensive processes was granted asylum. Each applicant was associated with a judge j, court c, nation n, and language l. Consistent with a hierarchical logistic model, we assumed that x denoted a vector of covariates with fixed coefficients β. α (.) denoted the group random effects, each of which was normally distributed and centered on zero. α was the population average intercept, with intercepts for each group given by α + α (.). The variances of the random effects around the population average were denoted by σ (.).

We used most of the covariates from our 2008 report on this issue. However, we excluded several covariates that were unassociated with outcome in that report and that were labor-intensive to measure. For example, measuring a judge’s prior work experience for our prior report involved biographical research on each judge. Specifically, the covariate vector x included the following: whether the applicant had one or more dependents (indicator); whether the applicant sought asylum within one year of entry into the United States (indicator); whether an applicant was represented by counsel recognized to practice in immigration court (indicator); whether the applicant was ever detained (indicator for being currently detained or having been detained and released, compared to never having been detained; defensive cases only);

Presidential administration under which judges were appointed (vector of indicators, with the subsample mode set as the omitted level); judge years of experience (linear); judge gender (indicator); judge asylum caseload since October 1, 1994, the earliest date in our data, and in the 90 days prior to the completion of the asylum application, respectively (linear);

Circuit court (vector of indicators, with the subsample mode set as the omitted level); and

Court asylum caseload since October 1, 1994, the earliest date in our data, and in the 90 days prior to the decision, respectively (linear).

For each subsample defined by asylum type (affirmative or defensive) and period, we rescaled the continuous covariates such that the vector of sample means equaled 0 and the vector of variances equaled 1. We rescaled each categorical covariate, including the groups used to define random effects, such that the first level identified the mode and served as the omitted level in a vector of indicator variables in xi for each variable. After rescaling, α estimates the overall asylum grant rate for an applicant having modal or mean values on x and α (.) . σ(.)

GAO, U.S. Asylum System: Significant Variation Existed in Asylum Outcomes Across Immigration Courts and Judges, GAO-08-940 (Washington, D.C: Sept. 25, 2008). of asylum grant rates across judges, courts, nations, and languages around the population average for an applicant at the mode or mean of xi.

The center of the data at α does not necessarily correspond to a realistic application for asylum. For example, all judges do not practice at the modal court and all languages may not be spoken in the modal nation. Nevertheless, rescaling facilitates estimation and interpretation of the model. All inference can be done on α and α (.) directly, with no transformation required. This allowed us to easily describe variation in asylum grant rates for a hypothetical representative case in the distribution of each covariate and group.

The key parameters of interest are the variances of the group random , and implicitly, the fitted group-level distribution of asylum effects, σ(.) grant rates for a modal or mean applicant: To describe variation across groups on the probability scale, we estimated the quantiles of this group distribution on the logit scale and then transformed them using the inverse logistic function.

We estimated the model using maximum approximated likelihood methods, as implemented in the lme4 package in R. We estimated the confidence intervals of fixed parameters using Wald approximations and the fitted covariance matrix. We estimated the confidence intervals of the group random effect variances using profiled likelihood methods, as implemented in lme4. We transformed the estimated confidence intervals from the logit scale to the probability scale using the inverse logistic calculation described above.

Results

Tables 8 through 14 report estimates of the fixed parameters on the exponentiated logit scale and estimates of the random effect variances on the logit scale, separately for affirmative and defensive applications. The exponentiated scale allows us to interpret the results as odds-ratios or semi-elasticities for categorical or continuous variables, respectively. We omit estimates for the fixed effect parameters on the logit scale to prevent the calculation of exact grant probabilities. Please see the body of our report for further interpretation of the results.

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Acknowledgments

In addition to the contact named above, Kathryn Bernet, Assistant Director; Joel Aldape, David Alexander, Emily Christoff, Dominick Dale, Jack Granberg, Grant Mallie, Taylor Matheson, Jan Montgomery, Jon Najmi, Mary Pitts, and Jeff Tessin made significant contributions to this report.