Federal Judgeships: The General Accuracy of District and Appellate Judgeship Case-Related Workload Measures
Published: 2013-09-10T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The demands on judges' time are largely a function of both the number and complexity of the cases on their dockets. To measure the case-related workload of district court judges, the Judicial Conference has adopted weighted case filings. The purpose of the district court case weights was to create a measure of the average judge time that a specific number and mix of cases filed in a district court would require. Importantly, the weights were designed to be descriptive, not prescriptive--that is, the weights were designed to develop a measure of the national average amount of time that judges actually spent on specific cases, not to develop a measure of how much time judges should spend on various types of cases. Moreover, the weights were designed to measure only case-related workload. Judges have noncase-related duties and responsibilities, such as administrative tasks, that are not reflected in the case weights.
With few exceptions, such as cases that are remanded to a district court from the court of appeals, each civil or criminal case filed in a district court is assigned a case weight. For example, in the 2004 case weights--which are still in use--drug possession cases are weighted at 0.86, while civil copyright and trademark cases are weighted at 2.12. The total annual weighted filings for a district are determined by summing the case weight associated with all the cases filed in the district during the year. A weighted case filing per authorized judgeship is the total annual weighted filings divided by the total number of authorized judgeships. The Judicial Conference uses weighted filings of 430 or more per authorized judgeship as an indication that a district may need additional judgeships. Thus, for example, a district with 460 weighted filings per authorized judgeship, including newly requested judgeships, could be considered for an additional judgeship. However, the Judicial Conference does not consider a district for additional judgeships, regardless of its weighted case filings, if the district does not request any additional judgeships.
section_title: What GAO Found
paragraphs: Based on GAO's 2003 report, it was found that the district court case weights approved in 1993 to be a reasonably accurate measure of the average time demands a specific number and mix of cases filed in a district court could be expected to place on the district judges in that court. The methodology used to develop the weights used a valid sampling procedure, developed weights based on actual case-related time recorded by judges from case filings to disposition, and included a measure (standard errors) of the statistical confidence in the final weight for each weighted case type. Without such a measure, it is not possible to objectively assess the accuracy of the final case weights.
At the time of GAO's 2003 report, the Subcommittee on Judicial Statistics of the Judicial Conference's Judicial Resources Committee had approved the research design for revising the 1993 case weights, with a goal of having new weights submitted to the Judicial Resources Committee for review in the summer of 2004. The design for the new case weights relied on three sources of data for specific types of cases: (1) data from automated databases identifying the docketed events associated with the cases; (2) data from automated sources on the time associated with courtroom events for cases, such as trials or hearings; and (3) consensus of estimated time data from structured, guided discussion among experienced judges on the time associated with noncourtroom events for cases, such as reading briefs or writing opinions.
In addition, GAO found that the principal quantitative measure the Judicial Conference used to assess the need for additional courts of appeals judgeships was adjusted case filings. The measure is based on data available from standard statistical reports for the courts of appeals. The adjusted filings workload measure is not based on any empirical data regarding the time that different types of cases required of appellate judges.
The Judicial Conference's policy is that courts of appeals with adjusted case filings of 500 or more per 3-judge panel may be considered for 1 or more additional judgeships. Courts of appeals generally decide cases using constantly rotating 3-judge panels. Thus, if a court had 12 authorized judgeships, those judges could be assigned to four panels of 3 judges each. In assessing judgeship needs for the courts of appeals, the conference may also consider factors other than adjusted filings, such as the geography of the circuit or the median time from case filings to disposition.

section_title: Letter
paragraphs: I appreciate the opportunity to submit this statement on our work on case- related workload measures for district court and courts of appeals judges.  My statement today is based on work completed and reported in 2003  and discussed in testimony on June 17, 2008, and September 30, 2009.  My statement is focused exclusively on these workload measures; we  have not assessed or taken a position on the Judicial Conference’s  pending request for additional judgeships.
Biennially, the Judicial Conference, the federal judiciary’s principal policy- making body, assesses the judiciary’s needs for additional judgeships. If  the conference determines that additional judgeships are needed, it  transmits a request to Congress identifying the number, type (such as  courts of appeals or district courts), and location of the judgeships it is  requesting.
In assessing the need for additional judgeships, the Judicial Conference  considers a variety of information, including responses to its biennial  survey of individual courts, temporary increases or decreases in case  filings, and other factors specific to an individual court. However, the  Judicial Conference’s analysis begins with the quantitative case-related  workload measures it has adopted for the district courts and courts of  appeals—weighted case filings and adjusted case filings, respectively.  These two measures recognize, to different degrees, that the time  demands on judges are largely a function of both the number and  complexity of the cases on their dockets. Some types of cases may  demand relatively little time and others may require many hours of work.  Generally, each case filed in a district court is assigned a weight  representing the average amount of judge time the case is expected to  require. The weights are relative to one another: the higher the case  weight, the greater the time the case would be expected to require. For  example, on average, a case with a relative weight of 2.0 would be  expected to require twice as much judge time as a case with a weight of  1.0. In the courts of appeals, all case filings are weighted equally at 1.0,  except for pro se case filings—those in which one or both parties are not  represented by an attorney—which are discounted.
Using these measures, individual courts whose past case-related  workload meets the threshold established by the Judicial Conference may  be considered for additional judgeships. These thresholds are 430  weighted case filings per authorized judgeship for district courts and 500  adjusted case filings per three-judge panel of authorized judgeships for  courts of appeals (courts of appeals judges generally hear cases in  rotating panels of three judges each). Authorized judgeships are the total  number of judgeships authorized by statute for each district court or court  of appeals.
The Judicial Conference relies on these quantitative workload measures  to be reasonably accurate measures of judges’ case-related workload.  Whether these measures are reasonably accurate rests in turn on the  soundness of the methodology used to develop them. This statement  summarizes information and recommendations from our 2003 report and  related updates to the methodology including (1) whether the judiciary’s  quantitative case-related workload measures were reasonably accurate  measures of district judges’ and courts of appeals judges’ case-related  workload, and (2) actions the judiciary has taken to implement  recommendations from our 2003 report.
Our 2003 report and 2008 and 2009 testimonies were based on the  results of our review of documentation provided by the Federal Judicial  Center (FJC) and the Administrative Office of the U.S. Courts (AOUSC)  on the history and development of the case-related workload measures  and interviews with officials in each organization. The scope of our work  did not include how the Judicial Conference used these case-related  workload measures to develop any specific request for additional district  or courts of appeals judgeships. In addition, in August 2013, we met with  FJC and AOUSC officials to discuss any updates that have been made to  the methodologies for developing case-related workload measures. While  we obtained some updated information for the purposes of this statement,  we did not evaluate these methodologies or the judiciary’s efforts in this  area. This work was conducted in accordance with generally accepted  government auditing standards. Those standards require that we plan  and perform the audit to obtain sufficient, appropriate evidence to provide  a reasonable basis for our findings and conclusions based on our audit  objectives. We believe that the evidence obtained provides a reasonable  basis for our findings and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: The demands on judges’ time are largely a function of both the number  and complexity of the cases on their dockets. To measure the case- related workload of district court judges, the Judicial Conference has  adopted weighted case filings. The purpose of the district court case  weights was to create a measure of the average judge time that a specific  number and mix of cases filed in a district court would require.  Importantly, the weights were designed to be descriptive, not  prescriptive—that is, the weights were designed to develop a measure of  the national average amount of time that judges actually spent on specific  cases, not to develop a measure of how much time judges should spend  on various types of cases. Moreover, the weights were designed to  measure only case-related workload. Judges have noncase-related duties  and responsibilities, such as administrative tasks, that are not reflected in  the case weights.
With few exceptions, such as cases that are remanded to a district court  from the court of appeals, each civil or criminal case filed in a district court  is assigned a case weight. For example, in the 2004 case weights—which  are still in use—drug possession cases are weighted at 0.86, while civil  copyright and trademark cases are weighted at 2.12. The total annual  weighted filings for a district are determined by summing the case weight  associated with all the cases filed in the district during the year. A  weighted case filing per authorized judgeship is the total annual weighted  filings divided by the total number of authorized judgeships. The Judicial  Conference uses weighted filings of 430 or more per authorized judgeship  as an indication that a district may need additional judgeships. Thus, for  example, a district with 460 weighted filings per authorized judgeship,  including newly requested judgeships, could be considered for an  additional judgeship. However, the Judicial Conference does not consider  a district for additional judgeships, regardless of its weighted case filings,  if the district does not request any additional judgeships.
section_title: Accuracy of Judiciary’s Workload Measures Could Not Be Determined
subsections: section_title: 1993 District Case Weights Reasonably Accurate, but Accuracy of 2004 Case Weights Could Not Be Statistically Determined
paragraphs: In our 2003 report, we found the district court case weights approved in  1993 to be a reasonably accurate measure of the average time demands  a specific number and mix of cases filed in a district court could be  expected to place on the district judges in that court. The methodology  used to develop the weights used a valid sampling procedure, developed  weights based on actual case-related time recorded by judges from case  filings to disposition, and included a measure (standard errors) of the  statistical confidence in the final weight for each weighted case type.  Without such a measure, it is not possible to objectively assess the  accuracy of the final case weights.
At the time of our 2003 report, the Subcommittee on Judicial Statistics of  the Judicial Conference’s Judicial Resources Committee had approved  the research design for revising the 1993 case weights, with a goal of  having new weights submitted to the Judicial Resources Committee for  review in the summer of 2004. The design for the new case weights relied  on three sources of data for specific types of cases: (1) data from  automated databases identifying the docketed events associated with the  cases; (2) data from automated sources on the time associated with  courtroom events for cases, such as trials or hearings; and (3) consensus  of estimated time data from structured, guided discussion among  experienced judges on the time associated with noncourtroom events for  cases, such as reading briefs or writing opinions.
As we reported in 2009, according to FJC, the subcommittee wanted a  study that could produce case weights in a relatively short period of time  without imposing a substantial record-keeping burden on district judges.  FJC staff provided the subcommittee with information about various  approaches to case weighting, and the subcommittee chose an event- based method—that is, a method that used data on the number and types  of events, such as trials and other evidentiary hearings, in a case. The  design did not involve the type of time study that was used to develop the  1993 case weights. Although the proposed methodology appeared to  offer the benefit of reduced judicial burden (no time study data collection),  potential cost savings, and reduced calendar time to develop the new  weights, we had two areas of concern—the challenge of obtaining  reliable, comparable data from two different data systems for the analysis  and the limited collection of actual data on the time judges spend on  cases.
First, the design assumed that judicial time spent on a given case could  be accurately estimated by viewing the case as a set of individual tasks or  events in the case. Information about event frequencies and, where  available, time spent on the events would be extracted from existing  administrative databases and reports and used to develop estimates of  the judge time spent on different types of cases. For event data, the  research design proposed using data from two databases (one of which  was new in 2003 and had not been implemented in all district courts) that  would have to be integrated to obtain and analyze the event data. FJC  proposed creating a technical advisory group to address this issue. In  August 2013, FJC officials told us that the process of integrating the two  data systems, though labor-intensive, was successful and resulted in  accurate data. However, we have not reviewed the integration process for  the two data systems, so we cannot determine the effectiveness of this  process or whether accurate data resulted.
Second, we reported that the research design did not require judges to  record time spent on individual cases, as was done for the 1993 case  weights. Actual time data would be limited to that available from existing  databases and reports on the time associated with certain courtroom  events and proceedings for different types of cases. However, a majority  of district judges’ time is spent on case-related work outside the  courtroom. The time required for noncourtroom events—and some  courtroom events that did not have actual time data available—would be  derived from structured, guided discussion of groups of 8 to 13  experienced district court judges in each of the 12 regional circuits (about  100 judges in all). The judges would develop estimates of the time  required for different events in different types of cases within each circuit  using FJC-developed “default values” as the reference point for  developing their estimates. These default values would be based in part  on the existing case weights and, in part, on other types of analyses.  Following the meetings of the judges in each circuit, a national group of  24 judges (2 from each circuit) would consider the data from the 12 circuit  groups and develop the new weights.
The accuracy of judges’ time estimates is dependent upon the experience  and knowledge of the participating judges and the accuracy and reliability  of the judges’ recall about the average time required for different events in  different types of cases—about 150 if all the case types in the 1993 case  weights were used. In 2003, we found that these consensus data could  not have been used to calculate statistical measures of the accuracy of  the resulting case weights. Thus, we concluded that the planned  methodology did not make it possible to objectively, statistically assess  how accurate the new case weights are—weights whose accuracy the  Judicial Conference relies upon in assessing judgeship needs.
In August 2013, AOUSC officials stated that, since 2005, for purposes of  determining the need for an additional authorized judgeship, a district’s  weighted case filings per authorized judgeship is calculated by including  the potential additional judgeship. For example, if a district had total  weighted filings of 4,600 and 9 authorized judgeships, and it planned to  request 1 additional judgeship, its weighted filings per authorized  judgeship, for purposes of the judgeship request process, would be 460.  Without including the potential additional judgeship in the calculation, the  weighted case filings would be about 511. AOUSC officials stated in  August 2013 that the judiciary adopted the proposed methodology in  2004 and does not have plans to update the 2004 district court case  weights.
section_title: Accuracy of Courts of Appeals Case-Related Workload Measure Could Not Be Assessed
paragraphs: In 2003, we found that the principal quantitative measure the Judicial  Conference used to assess the need for additional courts of appeals  judgeships was adjusted case filings. The measure is based on data  available from standard statistical reports for the courts of appeals. The  adjusted filings workload measure is not based on any empirical data  regarding the time that different types of cases required of appellate  judges.
The Judicial Conference’s policy is that courts of appeals with adjusted  case filings of 500 or more per 3-judge panel may be considered for 1 or  more additional judgeships. Courts of appeals generally decide cases  using constantly rotating 3-judge panels. Thus, if a court had 12  authorized judgeships, those judges could be assigned to four panels of 3  judges each. In assessing judgeship needs for the courts of appeals, the  conference may also consider factors other than adjusted filings, such as  the geography of the circuit or the median time from case filings to  disposition.
Essentially, the adjusted case filings workload measure counts all case  filings equally, with two exceptions. First, cases refiled and approved for  reinstatement are excluded from total case filings. Second, pro se cases  are weighted at 0.33, or one-third as much as other cases, which are  weighted at 1.0. For example, a court with 600 total pro se case filings in  a year would be credited with 198 adjusted pro se case filings (600 x  0.33). Thus, a court of appeals with 1,600 filings (excluding  reinstatements)—600 pro se cases and 1,000 non-pro se cases—would  be credited with 1,198 adjusted case filings (198 discounted pro se cases  plus 1,000 non-pro se cases). If this court had 6 judges (allowing two  panels of 3 judges each), it would have 599 adjusted case filings per 3- judge panel, and, thus, under Judicial Conference policy, could be  considered for an additional judgeship.
The current court of appeals workload measure, which, AOUSC officials  stated, was adopted in 1996, represents an effort to improve the previous  measure. In our 1993 report on judgeship needs assessment, we found  that the restraint of individual courts of appeals, not the workload  standards, seemed to have determined the actual number of appellate  judgeships the Judicial Conference requested. At the time the current  measure was developed and approved, using the new benchmark of 500  adjusted case filings resulted in judgeship numbers that closely  approximated the judgeship needs of the majority of the courts of  appeals, as the judges of each court perceived them. The current courts  of appeals case-related workload measure principally reflects a policy  decision using historical data on filings and terminations. It is not based  on empirical data regarding the judge time that different types of cases  may require. On the basis of the documentation we reviewed for our 2003  report, we determined that there was no empirical basis for assessing the  potential accuracy of adjusted case filings as a measure of case-related  judge workload.
section_title: Judiciary Has Taken No Action to Address Our 2003 Recommendations
paragraphs: In our 2003 report, we recommended that the Judicial Conference of the  United States update the district court case weights using a methodology that  supports an objective, statistically reliable means of calculating the  accuracy of the resulting weights, and develop a methodology for measuring the case-related workload of  courts of appeals judges that supports an objective, statistically  reliable means of calculating the accuracy of the resulting workload  measures and that addresses the special case characteristics of the  Court of Appeals for the D.C. Circuit.
Neither of these recommendations has been implemented, and in August  2013, AOUSC officials stated that the judiciary does not have plans to  update the 2004 district court case weights or the 1996 court of appeals  adjusted filings weights. With regard to our 2003 recommendation for  updating the district court case weights, we reported that FJC agreed that  the method used to develop the new case weights would not permit the  calculation of standard errors, but that other methods could be used to  assess the integrity of the resulting case weight system. In response, we  noted that the Delphi technique to be used for developing out-of-court  time estimates was most appropriate when more precise analytical  techniques were not feasible and the issue could benefit from subjective  judgments on a collective basis. More precise techniques were available  for developing the new case weights and were to be used for developing  new bankruptcy court case weights.
In our 2003 report, we also concluded that the methodology the Judicial  Conference decided to begin in June 2002 for the revision of the  bankruptcy case weights offered an approach that could be usefully  adopted for the revision of the district court case weights. The  bankruptcy court methodology used a two-phased approach. First, new  case weights were to be developed based on time data recorded by  bankruptcy judges for a period of weeks—a methodology very similar to  that used to develop the bankruptcy case weights that existed in 2003 at  the time of our report. The accuracy of the new case weights could be  assessed using standard errors. The second part represents  experimental research to determine if it is possible to make future  revisions of the weights without conducting a time study. The data from  the time study could be used to validate the feasibility of this approach. If  the research determined that this was possible, the case weights could be  updated more frequently with less cost than required by a time study. We  concluded in 2003 that that approach could provide (1) more accurate  weighted case filings than the design developed and used for the  development of the 2004 district court case weights, and (2) a sounder  method of developing and testing the accuracy of case weights that were  developed without a time study. However, we have not reviewed the  effectiveness of this methodology or confirmed whether the judiciary  implemented this approach.
With regard to our recommendation on improving the case-related  workload measure for the courts of appeals, the Chair of the Committee  on Judicial Resources commented that the workload of the courts of  appeals entails important factors that have defied measurement, including  significant differences in case-processing techniques. We recognized that  there were significant methodological challenges in developing a more  precise workload measure for the courts of appeals. However, we stated  that using the data available, neither we nor the Judicial Conference  could have assessed the accuracy of adjusted case filings as a measure  of the case-related workload of courts of appeals judges.
The Ranking Member of the Subcommittee on Bankruptcy and the Courts  has requested that we conduct a full review of the case-related workload  measures for district court and courts of appeals judges, including a  follow-up on our 2003 recommendations. Such a review will allow us to  evaluate the judiciary’s methodology and efforts over the last 10 years.
Mr. Chairman, this concludes my statement for the record.
section_title: Contact and Staff Acknowledgments
paragraphs: For further information about this statement, please contact David C.  Maurer, Director, Homeland Security and Justice Issues, on (202) 512- 9627 or maurerd@gao.gov. In addition to the contact named above, the  following individuals also made major contributions to this testimony:  Chris Currie, Acting Director; David P. Alexander, Assistant Director;  Brendan Kretzschmar; Jean M. Orland; Rebecca Kuhlmann Taylor; and  Janet G. Temko.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.