Discussion-piece #6

Predicted and Actual Ridership of Proposed New Starts Projects

Federal Transit Administration

June 6, 2006

1.  [Purposes of ridership reviews.]{.underline} FTA periodically
    compares the actual ridership against the ridership predictions for
    major transit projects using Federal "New Starts" funds. The
    analysis has three purposes: (1) to provide an up-to-date picture on
    the reliability of ridership forecasts as the basis for
    decisionmaking on proposed New Starts projects; (2) to identify any
    needed improvements in the technical methods used to make the
    forecasts; and (3) to identify any appropriate modifications to the
    way that FTA uses New Starts forecasts in project evaluation.

2.  [Pickrell report.]{.underline} FTA published the initial review in
    1990 in the report *Urban Rail Transit Projects: Forecast Versus
    Actual Ridership and Cost* (commonly referred to as the Pickrell
    report after its primary author). That review considered ten
    projects and found that only one project generated actual ridership
    that was more than 50 percent of the predicted ridership
    (specifically, 72 percent) Actual ridership for the other nine
    projects was less than 50 percent of their forecasts.

3.  [2003 report.]{.underline} FTA prepared (but has not yet released)
    the 2003 report *Predicted and Actual Impacts of New Starts
    Projects: Capital Cost, Operating Cost and Ridership Data*
    (hereafter termed the Phase-1 report) to consider the 19 New Starts
    projects (both rail and bus guideways) that opened for revenue
    service since the 1990 report. The post-1990 projects showed
    improvements in the quality of forecasts. Four of the 19 projects
    generated ridership that was between 70 and 80 percent of their
    forecasts. Another three projects generated ridership between 80 and
    100 percent of their forecasts. And three projects had actual
    ridership that exceeded their forecasts by modest amounts. Table 1
    summarizes the 19 projects, their ridership forecasts, and their
    actual (or extrapolated) ridership in the forecast year.

4.  [Pickrell update.]{.underline} The 2003 report also included an
    updated (year 2000) look at the ten projects reviewed by Pickrell.
    Two of those ten projects had year-2000 ridership close to forecast
    levels; two others showed growth since the 1990 report but were
    still far below forecast levels; three projects had little change in
    ridership; and three experienced declines in ridership since 1990.

5.  [Phase-1 conclusions.]{.underline} The 2003 report suggested several
    possible reasons for the improved quality of transit forecasts
    post-Pickrell, including greater forecasting experience, more
    formalized forecasting procedures and guidelines, increased scrutiny
    of forecasts and the planning process by government agencies and the
    public, improved forecasting technical methods, and improved
    computing technology. The report also observed forecasts for people
    movers, busways, and starter rail lines tended to be least reliable
    while forecasts for system expansions (additional lines in new
    corridors or extensions of existing lines in the same corridor) were
    relatively more reliable.

6.  [Phase-2.]{.underline} In 2006, further FTA-sponsored analysis of
    completed projects concluded in the draft report *Predicted and
    Actual Ridership of New Starts Projects: Detailed Analysis* (not yet
    released; hereafter the Phase-2 report) undertook detailed reviews
    of the ridership forecasts for seven of the nineteen Phase-1
    projects (as identified in Table 1). This work faced a substantial
    hurdle in the general unavailability of detailed information on the
    forecasts themselves. The forecasts were prepared 10 to 20 years ago
    and supporting documents and data sets (zone-level demographics,
    trip tables, zone definitions, and coded transit and highway
    networks) were simply not available. The case studies included two
    "successful" forecasts that were within ±20 percent of actual
    ridership and five "less successful" forecasts that were more than
    twice the actual ridership.

7.  [Successful forecasts.]{.underline} The two projects with successful
    forecasts -- San Diego El Cajon and Portland Westside -- were
    expansions of existing light rail systems. While it was extremely
    difficult in a retrospective analysis to confirm the level of
    quality control and reasonableness checks during the forecasting
    process, a review of both the calibration and validation tests and
    the results, as well as transit paths and skims, suggests that these
    procedures have been more rigorously followed in areas with
    successful forecasts. To some extent, the success of the two
    forecasts was the product of offsetting errors. While both forecasts
    were within ±20 percent of actual project-specific ridership, both
    missed actual levels of systemwide ridership more than ±20 percent
    and relied upon corridor-level demographic forecasts that also
    varied from actual outcomes by more than +/- 20 percent.

8.  [Less-successful forecasts]{.underline}. The five less-successful
    forecasts appear to have been subject to multiple types of errors of
    varying magnitude. Sources of error included erroneous model inputs,
    problematic model properties, and mistakes in model application --
    and all forecasts were subject to more than one of these errors.

-   [Input errors.]{.underline} The most frequent error involved the
    magnitude and location of future population and employment growth, a
    problem in all seven of the case studies, contributing both to the
    less successful forecasts and the offsetting errors that may have
    masked other problems in the successful forecasts. Because transit
    relies heavily on walking for access/egress, errors in demographic
    forecasts at the regional and/or corridor levels are compounded by
    incorrect allocations to zones within walking distances of
    fixed-guideway stations. Other sources of input error include the
    representation of future-year transportation networks (both highway
    and transit), inadequate detail in the zone system used to represent
    the region, as well as prices for transit fares, gasoline, and
    parking. At least one (and usually more) of these input errors
    specifically contributed to the forecasting error in each of the
    "less successful" case studies.

-   [Model-property errors.]{.underline} A common problem in the
    less-successful forecasts was the overestimation of future highway
    congestion. This problem may be the result of problematic
    demographic forecasts filtering through the model chain. However,
    overestimation of highway congestion appeared to occur even where
    regional trip tables generally replicated actual travel patterns
    indicated by census journey-to-work information and household
    surveys. In such cases the culprit is the model set itself, likely
    problems time-of-day distributions and/or network assignment.

-   [Model-application errors.]{.underline} Haste in the completion of
    forecasts to support funding application or environmental documents
    appears to have led to improper representation of changes in project
    scope or transit service plans in the travel forecasts. Other
    changes in scope and service plans have occurred after the forecasts
    were completed, without a corresponding update in the forecasts. In
    at least one case the model was validated to an outdated set of
    observed data before being used for the transit forecasts.

9.  [Absence of detailed records.]{.underline} While some insights were
    available from the seven case studies, by far the most significant
    outcome of the Phase-2 effort was the clear finding that useful
    comparisons of forecasts with actual outcome are not possible with
    the largely non-existent records of the forecasts. This outcome has
    significant implications for the usefulness of the Before-and-After
    studies that are now a required element of New Starts projects that
    receive Full Funding Grant Agreements and suggests the need to
    formalize the preservation of forecasts so that meaningful reviews
    of their accuracy are possible.

**Table 1: Predicted and Actual Ridership for Phase-1 Projects -
Forecast Year Comparison**

+-------------------+------+-------+-------+----------+------+-------+
| **Project**       | **   | **For |       | **Actual | *    |       |
|                   | Fore | ecast |       | (pr      | *Rat |       |
|                   | cast | Avg   |       | ojected) | io - |       |
|                   | Ye   | We    |       | B        | Fore |       |
|                   | ar** | ekday |       | oardings | cast |       |
|                   |      | B     |       | in       | yr   |       |
|                   |      | oardi |       | Forecast | a    |       |
|                   |      | ngs** |       | Year**   | ctua |       |
|                   |      |       |       |          | l/Fo |       |
|                   |      |       |       |          | reca |       |
|                   |      |       |       |          | st** |       |
+-------------------+------+-------+-------+----------+------+-------+
|                   |      | *     | **F   |          | **Ac | **A   |
|                   |      | *AA/D | EIS** |          | tual | ctual |
|                   |      | EIS** |       |          | vs.  | vs.   |
|                   |      |       |       |          | A    | F     |
|                   |      |       |       |          | A/DE | EIS** |
|                   |      |       |       |          | IS** |       |
+-------------------+------+-------+-------+----------+------+-------+
| Jacksonville ASE  | 1995 | 4     | 4     | 2,       | 6%   | 6%    |
|                   |      | 2,472 | 2,472 | 627^(1)^ |      |       |
+-------------------+------+-------+-------+----------+------+-------+
| Miami             | 2000 | 2     | 2     | 4,209    | 21%  | 21%   |
| Omni/Brickell     |      | 0,404 | 0,404 |          |      |       |
+-------------------+------+-------+-------+----------+------+-------+
| Houston SW        | 2005 | 2     | 2     | 9,066    | 33%  | 33%   |
| Transitway \*     |      | 7,280 | 7,280 |          |      |       |
+-------------------+------+-------+-------+----------+------+-------+
| Atlanta North     | 2005 | 5     | 5     | 21,595   | 38%  | 38%   |
| Line \*           |      | 7,120 | 7,120 |          |      |       |
+-------------------+------+-------+-------+----------+------+-------+
| LA Red Line \*    | 2000 | 29    | 29    | 128,     | 44%  | 43%   |
|                   |      | 5,721 | 7,733 | 659^(1)^ |      |       |
+-------------------+------+-------+-------+----------+------+-------+
| Pittsburgh West   | 2005 | 2     | 2     | 10,      | 44%  | 44%   |
| B\'Way            |      | 3,369 | 3,369 | 200^(3)^ |      |       |
+-------------------+------+-------+-------+----------+------+-------+
| Chicago Orange    | 2000 | 11    | 11    | 54,042   | 46%  | 46%   |
| Line \*           |      | 8,760 | 8,760 |          |      |       |
+-------------------+------+-------+-------+----------+------+-------+
| San Jose          | 1990 | 4     | 4     | 19,      | 48%  | 48%   |
| Guadalupe         |      | 1,200 | 1,200 | 738^(2)^ |      |       |
+-------------------+------+-------+-------+----------+------+-------+
| San Jose Tasman   | 2005 | 1     | 1     | 9,110    | 61%  | 66%   |
| West \*           |      | 4,875 | 3,845 |          |      |       |
+-------------------+------+-------+-------+----------+------+-------+
| Baltimore LRT     | 2005 | 1     | 1     | 8,207    | 70%  | 67%   |
| Ext.              |      | 1,804 | 2,230 |          |      |       |
+-------------------+------+-------+-------+----------+------+-------+
| Baltimore Johns   | 2005 | 1     | 1     | 10,049   | 74%  | 74%   |
| Hopkins           |      | 3,600 | 3,600 |          |      |       |
+-------------------+------+-------+-------+----------+------+-------+
| Portland          | 1    | 6     | 4     | 49,999   | 83%  | 101%  |
| W                 | 995/ | 0,314 | 9,448 |          |      |       |
| estside-Hillsboro | 2005 |       |       |          |      |       |
| \*                |      |       |       |          |      |       |
+-------------------+------+-------+-------+----------+------+-------+
| Dallas South Oak  | 2005 | 3     | 3     | 29,307   | 86%  | 86%   |
| Cliff             |      | 4,170 | 4,170 |          |      |       |
+-------------------+------+-------+-------+----------+------+-------+
| BART Colma        | 2000 | 1     | 1     | 13,482   | 89%  | 89%   |
|                   |      | 5,200 | 5,200 |          |      |       |
+-------------------+------+-------+-------+----------+------+-------+
| Salt Lake South   | 2010 | 2     | 2     | 25,201   | 95%  | 110%  |
| LRT               |      | 6,500 | 3,000 |          |      |       |
+-------------------+------+-------+-------+----------+------+-------+
| St. Louis Initial | 1995 | 4     | 3     | 43,      | 105% | 118%  |
| System            |      | 1,800 | 7,100 | 711^(4)^ |      |       |
+-------------------+------+-------+-------+----------+------+-------+
| San Diego El      | 2000 | 2     | 2     | 23,478   | 109% | 109%  |
| Cajon \*          |      | 1,600 | 1,600 |          |      |       |
+-------------------+------+-------+-------+----------+------+-------+
| Denver SW LRT     | 2015 | 2     | 2     | 23,      | 109% | 109%  |
|                   |      | 2,000 | 2,000 | 988^(5)^ |      |       |
+-------------------+------+-------+-------+----------+------+-------+
| St. Louis St.     | 2010 | 1     | 2     | 16,965   | 142% | 84%   |
| Clair Ext.        |      | 1,960 | 0,274 |          |      |       |
+-------------------+------+-------+-------+----------+------+-------+
| Denver I-25 HOV   | 2000 | not   | not   | 8,853    | NA   | NA    |
|                   |      | s     | s     |          |      |       |
|                   |      | tated | tated |          |      |       |
+-------------------+------+-------+-------+----------+------+-------+
| Seattle Bus       | 1990 | not   | not   | 44,400   | ##   | NA    |
| Tunnel            |      | s     | s     |          | #### |       |
|                   |      | tated | tated |          | # NA |       |
+-------------------+------+-------+-------+----------+------+-------+

^(1)^ Actual boardings in forecast year given for 2001 since this is the
first full year of operation.

^(2)^ Actual boardings in forecast year given for 1992 since this is the
first full year after opening

^(3)^ Actual boardings are assumed to increase 1,200 daily riders over
2002 as an additional park and ride lot is completed.

^(4)^ Actual boardings given for 1999 since Airport station did not open
until 1998. Forecast year boardings reached by applying the average
annual growth in transit boardings achieved by the project sponsor
between 1990 and 2002.

^(5)^ Denver has experienced relatively fast ridership growth over the
past decade. Since the forecast year remains far in the future,
continued growth at recent trends appears overly ambitious. FTA assumed
that the Denver project will achieve a growth rate 2/3rds of the growth
rate observed between 1990 and 2002. Even at this lower assumed growth
rate, this project is very likely to exceed its AA/DEIS forecasts by a
significant margin.

\* Selected for detailed analysis in the Phase-2 study.

**Table 2. Predicted and Actual Ridership for Phase II Case Studies:
Summary of Findings by Project**

+-----------+----------------------------------------------------------+
| **Cit     | **Summary of Findings**                                  |
| y/Project |                                                          |
| Name**    |                                                          |
+-----------+----------------------------------------------------------+
| Atlanta   | -   2005 observed boardings only 40% of forecast         |
|           |     boardings                                            |
| MARTA     |                                                          |
| North     | -   Observed rail system ridership less than forecast    |
| Line      |                                                          |
| Extension | -   Observed overall transit ridership close to forecast |
|           |     but widely fluctuates year-to-year                   |
|           |                                                          |
|           | -   Forecasting error caused by failure to achieve       |
|           |     predicted employment levels in station areas in      |
|           |     primary travel market, underestimation of regional   |
|           |     employment, fluctuations in overall system           |
|           |     ridership, inaccurate transit coding conventions in  |
|           |     the model, poor trip distribution model,             |
|           |     over-reliance on mode choice adjustment factors, and |
|           |     validation to outdated observed data set.            |
+-----------+----------------------------------------------------------+
| Chicago   | -   2000 observed project boardings only 46% of forecast |
|           |     boardings                                            |
| CTA       |                                                          |
| Orange    | -   Observed system-wide rail boardings close to         |
| Line      |     forecast                                             |
|           |                                                          |
|           | -   Observed transit system boardings close to forecast  |
|           |                                                          |
|           | -   Forecasting error caused by failure to account for   |
|           |     demographic changes in study area / corridor, and    |
|           |     poor model structure, especially for trip            |
|           |     distribution and mode choice                         |
+-----------+----------------------------------------------------------+
| Houston   | -   2005 projected (from 2002 observed) boardings only   |
|           |     33% of forecast boardings                            |
| METRO     |                                                          |
| Southwest | -   Observed transit system ridership less than forecast |
| T         |                                                          |
| ransitway | -   Forecasting error caused by failure to achieve       |
|           |     predicted population and employment levels in the    |
|           |     study corridor and region, failure to achieve        |
|           |     predicted land uses in station areas, overestimation |
|           |     of future highway congestion, poor transit coding    |
|           |     and zone system, and changes to project following    |
|           |     completion of forecasts                              |
+-----------+----------------------------------------------------------+
| Los       | -   2001 (1^st^ year of full line operation) observed    |
| Angeles   |     boardings 43% of (2000) forecast boardings           |
|           |                                                          |
| MTA Red   | -   Observed transit system boardings 72% of forecast    |
| Line      |     boardings                                            |
|           |                                                          |
|           | -   Forecasting error caused by poor model inputs for    |
|           |     transit fares, gasoline costs, fuel economy, poor    |
|           |     transit-access coding, failure to achieve employment |
|           |     forecasts, failure to fully restructure background   |
|           |     bus network to eliminate direct competition with     |
|           |     line and provide feeder service, service changes due |
|           |     to conversion from trunk line to trunk/branch        |
|           |     operations, relocation of line to less attractive    |
|           |     transit corridor, and length of time needed to       |
|           |     construct and operate full line                      |
+-----------+----------------------------------------------------------+
| Portland  | -   2002 observed boardings 8% over 2005 predicted       |
|           |     boardings                                            |
| Tri-Met   |                                                          |
| Westside/ | -   2001 observed LRT system boardings 3% over 2005      |
|           |     predicted boardings                                  |
| Hillsboro |                                                          |
| LRT       | -   Forecasting success caused by realistic and          |
|           |     quality-controlled transit service inputs, previous  |
|           |     experience operating LRT, higher than forecast       |
|           |     population/employment growth                         |
|           |                                                          |
|           | -   Approximately 10% to 15% of the success may be       |
|           |     attributed to underestimation of growth              |
|           |                                                          |
|           | -   Good model features, such as extra trip purposes,    |
|           |     cars per worker variable, use of choice models for   |
|           |     demographic inputs, inclusion of non-mechanized      |
|           |     trips in mode choice, good model accounting of       |
|           |     transit accessibility and use of mode-of-access      |
|           |     model in mode choice may have contributed to         |
|           |     forecasting success                                  |
|           |                                                          |
|           | -   Errors in population and employment forecasts may    |
|           |     have helped ridership forecast for project but are   |
|           |     indicative of larger errors in the demographic and   |
|           |     employment model (offsetting errors)                 |
+-----------+----------------------------------------------------------+
| San Diego | -   2000 observed boardings 9% over 2000 predicted       |
|           |     boardings                                            |
| MTDB El   |                                                          |
| Cajon LRT | -   2000 observed LRT system boardings 57% over 2000     |
|           |     predicted boardings                                  |
|           |                                                          |
|           | -   2000 observed transit system boardings 2% over 2000  |
|           |     predicted boardings                                  |
|           |                                                          |
|           | -   Forecasting success caused by realistic model inputs |
|           |     and quality control, good model features, and        |
|           |     greater than expected population and employment      |
|           |     growth in the corridor                               |
|           |                                                          |
|           | -   Approximately 15% to 20% of the success may be       |
|           |     attributed to underestimation of growth              |
|           |                                                          |
|           | -   Errors in population and employment forecasts may    |
|           |     have helped ridership forecast for project but are   |
|           |     indicative of larger errors in the demographic and   |
|           |     employment model (offsetting errors)                 |
|           |                                                          |
|           | -   Large forecasting error for LRT system overall       |
|           |     suggests problems with mode choice model             |
+-----------+----------------------------------------------------------+
| San Jose  | -   2005 observed boardings only 25% of 2005 predicted   |
|           |     boardings                                            |
| VTA       |                                                          |
| Tasman    | -   Forecasting error caused by severe economic          |
| West LRT  |     contraction in corridor and surrounding region,      |
|           |     overestimation of highway congestion, poor TAZ       |
|           |     system, unrefined trip distribution model, poor      |
|           |     network inputs, and poor transit assignment          |
+-----------+----------------------------------------------------------+