U.S. Postal Service: Actions Needed to Make Delivery Performance Information More Complete, Useful, and Transparent
Released: 2015-10-05T00:00:00

Letter

Background

Following enactment of PAEA in 2006, USPS updated its delivery service standards for market-dominant products, which define the number of days USPS is to take to deliver the mail in a timely manner. USPS’s delivery service standards are set forth in federal regulations and differ depending on the type of mail, the time of day and location at which USPS receives the mail, and the mail’s final destination. For example, USPS standards for delivery of 2-day single-piece First-Class Mail require the mail to be received by a specified cutoff time on the day it is accepted, which varies depending on geographic location and where the mail is deposited (e.g., in a collection box, at a post office, or at a mail processing facility). This mail must then be delivered on the second regular delivery day (Monday to Saturday) to be considered “on time.”

USPS measures delivery performance against its delivery service standards. For a given piece of mail, USPS first measures the transit time—that is, the number of days it takes from the point that the mail is accepted into USPS’s system until its delivery to a home or business. Then USPS compares this delivery time against delivery service standards to determine whether the mail was delivered on time. See figure 1 for USPS’s delivery performance of single-piece First-Class Mail from fiscal years 2011 to 2015. The second quarter of fiscal year 2015 experienced a significant decline in on-time delivery performance that USPS attributes to operational changes enacted in January 2015 coupled with adverse winter weather. However, performance improved in the next quarter.

Since 2012, USPS has instituted several initiatives aimed at reducing expenses in its mail delivery and processing operations and networks as part of broader efforts to address its fiscal challenges and move toward financial viability. These initiatives included changing mail delivery service standards for some types of mail and then consolidating 141 mail processing facilities in 2012 and 2013. As we reported in September 2014, USPS changes to its delivery service standards increased the number of days for some mail to be delivered and still be considered on time. Further, effective January 5, 2015, USPS changed the delivery service standard for single-piece First-Class Mail sent to a nearby destination from 1 to 2 days. Table 2 presents these changes for market- dominant mail, which consists of First-Class Mail (e.g., correspondence, bills, payments, and statements); Standard Mail (mainly advertising); Periodicals (mainly magazines and local newspapers); and Package Services (mainly Media/Library Mail and Bound Printed Matter).

To understand how changes in service standards affected expected transit times for First-Class Mail, we asked USPS to estimate the volumes of First-Class Mail subject to 1-day, 2-day, and 3-5-day delivery service standards for fiscal years 2011 through the second quarter of fiscal year 2015—the first quarter after USPS made its most recent changes to delivery service standards. USPS estimated that the percentage of First- Class Mail volume subject to a 1-day standard decreased from 38 percent in fiscal year 2011 to 13 percent in the second quarter of fiscal year 2015 (see fig. 2). When on-time delivery of First-Class Mail is redefined from a 1-day standard to a 2-day standard, USPS can take longer to deliver the mail for it to be considered “on time.”

Based on delivery service standards, USPS sets annual performance targets for the percentage of mail that is to be delivered on time, and PRC annually assesses and reports USPS’s performance towards meeting these targets. PRC has responsibility for assuring delivery performance data are complete and accurate, and must approve any internal delivery performance measurement systems (i.e., systems administered by USPS as opposed to an external contractor).

USPS is subject to legal requirements to create delivery service standards for market-dominant products, measure delivery performance, and report the results. Likewise, PRC is subject to legal requirements to specify how USPS should measure and report delivery performance, as well as requirements for using these data to provide oversight over USPS delivery performance. Key requirements for USPS and PRC are summarized in table 3.

USPS uses two primary methods for measuring delivery performance:

Tracking Barcoded Mail: Since 2011, USPS has measured and reported delivery performance for most types of market-dominant mail by using the time it is accepted at postal facilities to “start the clock” and scans of barcoded mail pieces to “stop the clock” by external, third-party reporters who receive the mail. This mail transit time is compared against delivery service standards to determine whether mail is delivered within the standard and thus considered on time. Most barcoded mail is tracked through USPS’s Full-Service Intelligent Mail program, which requires participating mailers that send bulk mail (i.e., mail entered in bulk quantities such as bills, advertisements, and magazines) to apply unique Intelligent Mail barcodes to mail pieces and provide USPS with electronic documentation for each mailing. Under this program, USPS uses a census approach that aims to measure all qualifying mail pieces in its mail processing network rather than a sampling approach. USPS commented that a census- type approach enables it to use the information to better manage day- to-day conditions throughout its network and that such visibility would not be available through sampling.

Sending Test Mail Pieces: Since 1990, USPS has measured and reported on-time delivery performance for single-piece First-Class Mail through the External First-Class Mail measurement system (EXFC). Under this sampling system, an external contractor arranges for anonymous droppers to send test mail pieces from street collection boxes and private office-building lobby chutes to external, third-party reporters at residences and businesses. As will be discussed in more detail below, in January 2015, USPS proposed replacing its EXFC measurement system for single-piece First-Class Mail in favor of a system based on tracking barcoded mail.

Delivery Performance Information Has Expanded but Remains Incomplete, and PRC Has Not Fully Assessed Causes of Incomplete Data or the Effectiveness of USPS Actions

Measurement of Delivery Performance Has Expanded but Remains Incomplete

USPS’s measurement of on-time delivery performance has expanded greatly over the past 9 years, but remains incomplete because only about 55 percent of market-dominant mail volume is currently included in measurement. The remaining 45 percent is not included in measurement for two main reasons: (1) lack of trackable barcodes or (2) lack of needed information. USPS told us that it wants to include virtually all market-dominant mail in delivery performance measurement. To assess completeness, we determined measurement coverage—the percentage of mail included in measurement—as well as the various causes for why mail is not included in measurement and their possible effect on measured results. To the extent that mail is not included in measurement, performance data are not complete and may not be representative. There is not a minimum threshold of mail that is to be included in measurement for it to be considered representative. In general, the risk that measurement is not representative is greater if mail not included in measurement may be systematically different than mail included in measurement. In particular, if the unmeasured mail has different characteristics than the measured mail, and those characteristics are associated with the likelihood of on-time delivery, then the risk of a non-representative measurement is greater.

As of the second quarter of fiscal year 2015, USPS measured on-time delivery performance for about half (55 percent) of market-dominant mail volume—up from only one-sixth of volume (16 percent) in fiscal year 2006 (see fig. 3). This increase in measurement coverage represents noteworthy progress by USPS and the mailing industry, which have devoted management commitment and significant resources to implement and participate in measurement systems for bulk mail that comprise most mail volume. Notably, USPS implemented measurement systems for bulk First-Class Mail, Standard Mail, and Periodicals. The number of pieces of market-dominant bulk mail included in delivery performance measurement has increased greatly in recent years—from 96 million pieces in the first quarter of fiscal year 2010 to 14.9 billion pieces in the second quarter of fiscal year 2015. Thus, the percentage of market-dominant bulk mail included in measurement increased from less than one percent in the first quarter of fiscal year 2010 to 48 percent in the second quarter of fiscal year 2015. Meanwhile, USPS provides performance measurement that covers virtually all single-piece First- Class Mail through mailing test pieces as part of its long-standing EXFC system. However, single-piece First-Class Mail comprises a small and declining percentage of market-dominant mail volume—down from 20 percent in the first quarter of fiscal year 2010 to 14 percent in the second quarter of fiscal year 2015.

USPS’s measurement coverage has varied by class of mail (see fig. 4). The percentage of mail included in measurement is greatest for First- Class Mail (both bulk and single-piece mail), followed by Standard Mail, Periodicals, and market-dominant Package Services (mainly Media Mail/Library Mail and Bound Printed Matter) and has improved over time for each class. Most progress began in fiscal year 2011, when mailer participation increased significantly in Full-Service Intelligent Mail—a program that enables business mailers to track the progress of their barcoded mail through USPS’s mail processing system. Meanwhile, USPS’s EXFC measurement system continues to send test mail pieces to measure delivery performance of single-piece First-Class Mail.

USPS’s measurement of on-time delivery performance for market- dominant mail remains incomplete because, as noted above, only about half of bulk mail volume was included in measurement as of the second quarter of fiscal year 2015. USPS tracks bulk mail using barcodes and electronic information about the mailing. The main causes for incomplete measurement of bulk mail can be broadly grouped into two different reasons: (1) mailers not applying a unique Intelligent Mail barcode to each mail piece to enable tracking (trackable barcodes)needed information.

In addition, some mailers may not apply trackable barcodes due to the type of mail they are entering, such as certain types of locally-entered and delivered mail that are not eligible for barcode-based postage discounts. at a USPS facility. About 4.2 billion pieces of mail (about 45 percent of all exclusions) were excluded in the second quarter of fiscal year 2015 due to “no start-the-clock.” 2. No mail piece barcode scan was recorded by USPS’s automation equipment: This prevents USPS from being able to track the barcoded mail. About 1.5 billion pieces of mail (16 percent of all exclusions) were excluded in the second quarter of fiscal year 2015 due to a lack of a barcode scan recorded by USPS automated equipment. 3. Inaccuracies in mail preparation: These include deficiencies in preparing bundles of mail or the quality of barcodes on mail pieces. About 1.2 billion pieces of mail (13 percent of all exclusions) were excluded in this quarter due to inaccuracies in mail preparation.

The percentage of market-dominant mail volume that was eligible for measurement but excluded for various reasons increased from 21 to 26 percent over the period from the fourth quarter of fiscal year 2013 to the second quarter of fiscal year 2015. Thus, exclusion of eligible mail from measurement has become a more important cause for why measurement data are not complete. USPS officials told us that the increase in excluded mail volume can be attributed to the overall increase of mail volume eligible for inclusion in measurement.

USPS told us about several actions it has taken to reduce exclusions, including steps to improve the scanning of barcoded mail. USPS also reported that it is collaborating with the mailing industry and working with individual mailers to improve the quality of mail preparation and compliance with requirements that can reduce the volume of mail excluded from measurement. As previously discussed, delivery performance may be different for mail included in measurement compared to mail that is not included in measurement. USPS told us that it has not studied whether on-time delivery performance varies for mail sent by mailers that do not participate in its measurement programs. Thus, the effect of non-participation on delivery performance measurement is unknown. However, available information indicates that non-participation can affect results for some Standard Mail products— particularly if product-specific results are not weighted to reflect key characteristics of the mail. Large volume mailers, which are most likely to apply barcodes and thus have on-time delivery performance measured, reportedly use additional mailing practices to facilitate the timely delivery of their mail, such as entering large volumes of advertising mail close to its final destination. Destination-entered Standard Mail is more likely to be included in measurement than other “end-to-end” Standard Mail and is more likely to be delivered on time than end-to-end Standard Mail. For example, in its latest Annual Compliance Determination Report, PRC noted that 86 percent of Standard Mail Flats (a Standard Mail product generally consisting of large flat-sized advertising such as catalogs) were delivered on time in fiscal year 2014 when they were destination entered (entered at a postal facility generally closer to the final destination of the mail), but only 50 to 66 percent were delivered on time when they were not destination entered. USPS told us that the mail volume and preparation of destination-entry Standard Mail enable it to be more likely to be included in measurement as well as be delivered on time. Destination-entry mail also has sufficient volume and preparation to enable it to bypass various postal network processing and transportation (e.g., locally-entered and delivered so it is handled by only one processing facility), which according to USPS, is the reason this mail is more likely to be delivered on time.

USPS Has Proposed Actions to Implement New Measurement Capabilities

In January 2015, USPS proposed replacing its EXFC measurement system for single-piece First-Class Mail in favor of a system based on tracking barcoded single-piece mail. USPS stated its proposed system is intended to incorporate a larger and more representative population of mail pieces in measurement than EXFC does today. PRC established a public inquiry docket (a type of proceeding) to review USPS’s proposal,which is still ongoing as of September 21, 2015. In this proceeding, USPS said it expected the data from its proposed system to provide PRC with the ability to perform its responsibilities with a high degree of confidence and to reasonably inform the public regarding the quality of service provided for market-dominant products.

However, some parties that commented on USPS’s proposal questioned whether the proposed system would produce representative results. For example, one concern was that the proposed system would measure mail deposited in blue collection boxes and at postal retail counters, but not measure the 38 percent of single-piece First-Class Mail that carriers pick up from customer mailboxes. Some parties commented that mail picked up by carriers may arrive at mail processing facilities a day later than mail deposited into collection boxes, such as situations where carriers return too late to the office for the mail to be transferred to transportation that day to a local mail processing facility. Others expressed concern that only barcoded single-piece mail would be eligible for measurement, which would exclude stamped mail without a barcode (such as personal correspondence and greeting cards).

In USPS’s written reply to comments made on its proposal, USPS responded that critics overstated the difference between handling of mail left for carrier pickup and collection mail, and that the barcoded mail measured by its proposed system would provide a reasonable indicator of performance for all mail collected and transported to the processing facility. Among other comments, USPS responded that mail processing for single-piece First-Class Mail is conducted over a range of hours each day, offering a substantial window of opportunity to accommodate mail arriving later than normal, including mail that missed the last scheduled dispatch from facilities where carriers bring stamped mail for forwarding to processing facilities. USPS further responded that it is not yet feasible for it to measure delivery performance for single-piece First-Class Mail left for carrier pickup, but added that it was revising its proposal to measure stamped and metered mail left at postal retail lobby chutes.

In its June 2015 interim order, PRC commented that because USPS’s proposal is still in development, PRC lacked sufficient information to make decisions concerning whether or not the proposed systems will be suitable for reporting service performance to PRC. PRC added that given that EXFC appears to have been producing reliable results for a considerable number of years, PRC cannot approve a new system to replace EXFC until the new system is similarly operational and verifiable. PRC directed USPS to plan to run EXFC and the proposed new system in parallel for a sufficient time to ensure it is operational and verifiable. PRC explained that test results demonstrating that the EXFC and new system generate objective and reliable measurements for all affected products over a period of four consecutive quarters would appear to be an acceptable demonstration.

On August 25, 2015, USPS filed its statistical design plan for its proposed new system with the PRC, which explained the sampling methodology and the methodology for calculating results and their margins of error. However, USPS had not yet made public some other major aspects of its proposed new system—such as quality control procedures and internal controls including methods to address errors in collecting data. Since USPS has relied on the EXFC system since 1990 to measure delivery performance for single-piece First-Class Mail, a thorough review of detailed information on its proposed system will be important not only to PRC, but to stakeholders including Congress and the mailing industry. In this regard, PRC’s proceeding to evaluate this new system continues and the time frame for completion remains open-ended since there is no statutory deadline and PRC has not established a deadline.

PRC Has Not Fully Assessed the Causes of USPS’s Incomplete Measurement Data and USPS’s Actions to Improve Data Quality

Although PRC reports have provided data on the amount of mail included in measurement of delivery performance, these reports have not fully assessed why these measurements were incomplete or whether USPS actions will achieve complete performance data. In addition, USPS officials told us that they have not established a time frame for achieving complete measurement. PRC uses these performance data to annually assess USPS’s delivery performance against targets that USPS has established for on-time delivery. Thus, delivery performance data that are complete and representative are essential for PRC to correctly determine whether USPS has met its delivery performance targets. Complete information is vital for effective management, oversight, and accountability purposes.

Further, representatives of mailing industry groups and some mailers told us and commented in PRC proceedings that PRC should become more involved in issues regarding the quality of measurement data for on-time delivery performance, including issues regarding the exclusion of mail from measurement. These representatives provided us with a variety of suggestions in this regard, such as performing more in-depth and frequent oversight to ensure USPS measurement is complete. One said that USPS is still struggling to scan barcoded mail despite joint USPS- mailer efforts over the past decade. Another representative said that although PRC has oversight of USPS service performance, measurement, and reporting, there is little consequence to USPS as a result of not meeting its targets for on-time delivery or for deficiencies in its measurement and reporting practices. A third said that PRC should hold USPS accountable for improving measurement data by requiring a business plan where USPS would lay out the steps it needs to take and time frames for implementing its initiatives.

PRC’s annual compliance reports have discussed how much mail volume for each type of mail is included in measurement and when USPS did not report performance results due to a lack of measurable data. However, PRC has not fully pursued the main causes for incomplete data (i.e., lack of trackable barcodes and lack of information causing data to be excluded from measurement). PRC reports have expressed concern with low levels of participation for certain types of mail, stating that low levels cause unreliable measurement. However, these PRC reports have not fully assessed the effectiveness of USPS actions taken or planned and associated timeframes with respect to the main causes for incomplete data. PRC could pursue the causes for incomplete data within its annual compliance reviews or it may initiate a separate proceeding. Furthermore, as previously discussed, by law, PRC may initiate a proceeding to improve the quality, accuracy, or completeness of data that USPS annually provides to PRC for its annual compliance determination whenever it appears the data have become significantly inaccurate or can be significantly improved. PRC officials told us that they have not been asked by any stakeholder to initiate such a proceeding. Nor has PRC exercised its option to initiate a proceeding on its own authority to address issues that impact the completeness of performance data.

PRC and USPS officials told us that they are both opposed to having PRC initiate a proceeding focused on issues for improving the completeness of delivery performance measurement for two key reasons. 1. PRC officials believe that USPS’s delivery performance measures are generally sufficiently accurate, reliable, and representative for PRC to meet its legal responsibilities for assessing USPS’s compliance with service performance standards at the national level. Further, PRC officials told us that they believe non-measured mail has about the same on-time performance results as measured mail. However, PRC and USPS officials told us that neither have compared the performance of mail included and not included in measurement to determine if any differences exist. As previously discussed, available information indicates that non-participation in measurement can affect reported results for on-time delivery performance. Large volume mailers, who are most likely to have their mail barcoded and thus have on-time delivery performance measured, reportedly use additional mailing practices to facilitate timely delivery, such as entering large volumes of advertising mail close to its final destination. Destination-entered advertising mail is more likely to be included in measurement and is more likely to be delivered on time. 2. USPS officials stated that a new proceeding to consider data quality and completeness issues is not necessary because the current proceeding before the PRC (the performance measurement to replace EXFC) provides a public forum for consideration of the quality of service performance data, as well as mail excluded from measurement. However, according to publicly available documents in the current proceeding, PRC has not explored issues of delivery performance measurement data for bulk mail that are excluded from USPS’s current measurement systems, the multiple causes for these exclusions, and USPS actions under way and planned to address the causes. The proceeding also has not thoroughly explored mailers’ concerns regarding data exclusions, such as exclusion rules and mailer views regarding time frames for making progress on reducing exclusions.

A PRC proceeding that focuses solely on issues of data quality and completeness—particularly the problem of data exclusions—may facilitate these issues receiving the fullest attention and making more rapid progress by USPS and the mailing industry toward achieving more complete measurement. As previously noted, while USPS has made progress toward achieving completeness since 2006—as illustrated by figure 3 earlier in this report—45 percent of market-dominant mail is still not measured. Performance information is sufficiently complete when it has the coverage to enable representative measurement of the percentage of mail delivered on time. While there is not a minimum threshold of mail that is to be included in measurement for it to be representative, the risk that measurement is not representative increases as more mail is not included in measurement because on-time delivery performance may be different for mail that is included in measurement from mail that is not included. Therefore, having a proceeding solely focusing on data quality and completeness could give USPS and postal stakeholders such as PRC, Congress, business mailers, and the general public the opportunity to conduct an in-depth evaluation of the quality of delivery performance data, identify practical opportunities to improve data quality, and establish actions and time frames for making progress. Having such a proceeding also could help PRC develop a better understanding of issues regarding the quality of delivery performance data and thereby be in a better position to conduct ongoing oversight of data quality and its annual compliance determination.

Reported Delivery Performance Information Is Not Sufficiently Useful or Transparent for Oversight of Delivery Performance in all Areas

USPS and PRC reports on delivery performance are not as useful as they could be for effective oversight. USPS and PRC annual compliance reports provide delivery performance analysis, as legally required. This information is reported at the national level. This analysis, however, does not facilitate an understanding of results and trends below the national level, such as for USPS’s 67 districts, to identify variations and areas where improvements in performance may be needed. USPS and PRC annual and quarterly reports on delivery performance information are not as useful for other oversight purposes or management and congressional decision making. For example, these reports do not include sufficient analysis to hold USPS accountable for meeting its statutory mission to provide prompt, reliable, and efficient services in all areas of the nation and regular postal services to rural areas. Further, delivery performance information is not sufficiently transparent as it is not readily available on respective USPS and PRC websites. Thus, it is difficult for effective oversight and for stakeholders to understand trends and develop analysis of USPS performance information.

Delivery Performance Information Is Not Reported in a Useful or Transparent Manner to Hold USPS Accountable for Meeting Its Mission

We have reported that ensuring information is useful to assist management and congressional decision making is key to the principles embodied in GPRA and the GPRA Modernization Act of 2010 framework for meeting fiscal, management, and performance challenges. USPS and PRC reports, however, provided little analysis to facilitate an understanding of results and trends below the national level. USPS and PRC websites do provide annual and quarterly delivery performance results on the national level and for each of USPS’s 7 areas and 67 districts. In addition, PRC provided annual delivery performance trend data at the national level in its annual compliance determinations covering fiscal years 2013 and 2014.however, are not sufficiently useful for determining variations in delivery performance across the nation or determining whether performance has improved in areas where performance has not met service standards or targets. National averages aggregate the mail delivery performance of different parts of the country into an average for the entire nation. Thus, on-time delivery performance in one section of the country may be masked by on-time delivery performance in another section of the country. A national average alone does not enable stakeholders to understand if certain areas of the country are experiencing poor delivery performance.

Trend data solely at the national level, To better understand the range and variations in delivery performance across the nation, we analyzed trends in quarterly delivery performance at the district level. Our analysis showed how national data can mask wide variations in performance by various districts over time. For example, we analyzed quarterly performance for single-piece First-Class Mail with a 3-to-5-day delivery service standard for each of the 64 postal districts in the contiguous 48 states and the District of Columbia for the second quarter of fiscal years 2013 to 2015. For the second quarter of fiscal year 2015, none of the districts met that quarter’s performance target of 95 percent of mail delivered on time. Performance for that quarter ranged from 44 percent to 80 percent (see fig. 5). However, when analyzing the second quarter of the previous 2 fiscal years, of the 10 districts with the lowest scores in the second quarter of fiscal year 2015, 9 were below the national average in fiscal year 2014, and all 10 were lower than the national average in fiscal year 2013, but to a much lesser degree. Of the 10 districts with the highest scores in the second quarter of fiscal year 2015, 8 were above the national average in the second quarter of fiscal years 2014 and 2015.

In addition, USPS’s reporting of delivery performance information is not sufficiently transparent. To be considered transparent, the criteria we identified suggest that delivery performance information is to be reported in a manner that is easily accessible and readily available. USPS, however, posts only its most recent quarterly report of area and district- level data on its public website.request numerous files from USPS to compile data necessary for understanding performance trends, such as whether on-time delivery is As a result, stakeholders would have to improving or getting worse. USPS told us that its reporting of delivery service information meets statutory requirements, and that it is not required to maintain quarterly trend data for delivery performance on its website. However, USPS can elect to maintain quarterly trend data on its website. A large mailer association we spoke with stated that USPS should be so transparent that everyone understands general performance and any factors contributing to good and poor performance.

Similar to USPS, PRC’s reporting of delivery performance information is not readily available to stakeholders. While PRC also posts delivery performance information provided by USPS on its public website, stakeholders would have to find numerous files in multiple locations on its website to compile data necessary for understanding performance trends, such as whether on-time delivery is improving or getting worse. In addition, PRC’s reports are not easily accessible. PRC has reported its annual assessment of USPS’s delivery performance in fiscal year 2014 in two reports that are filed on its website at different times and at different links, while USPS’s quarterly data are posted at another link on PRC’s website. The lack of easily accessible and readily available performance information on USPS’s and PRC’s part impedes the ability of Congress, mailers, and customers to review and hold USPS accountable for its performance and to use the information to develop realistic expectations for when their mail will be delivered.

Rural Delivery Performance Information Is Not Required to Be Reported

USPS and PRC are not required to report—and do not report—delivery information for rural and non-rural areas, thus limiting effective oversight in these areas. USPS and PRC officials told us that they do not provide information or analysis to assess delivery performance specifically for rural areas because they are not legally required to do so. Without data on rural delivery performance, Congress cannot determine the extent delivery performance is timely in rural versus non-rural areas, and neither USPS nor PRC can prove or disprove any perceptions that rural areas may be affected differently than non-rural areas.

Several Members of Congress and others have raised questions about whether delivery performance in rural areas has been negatively affected by changes USPS has implemented since fiscal year 2012 to reduce its expenses. For example, according to the National Newspaper Association (NNA), community newspapers have been negatively affected since USPS consolidated some postal facilities. Further, problems have emerged when newspapers, often in rural areas, had to be delivered outside of the local area and experienced a decline in service. NNA has requested that PRC gather information about the data that could be produced about rural mail to identify the sources of delivery problems, such as manual processing, increased travel distances, or inefficient processing plants. NNA has argued that “the possibility that what ails NNA newspapers also ails rural mail in general is more than a random guess.” In May 2015, two Members concerned about the lack of digital tracking in rural areas requested a PRC study on the feasibility of reporting on rural mail delivery performance. Other congressional requests for rural delivery performance information are also pending. For example, in a recent Senate report, the Senate Appropriations Committee directed USPS to take steps related to reporting delivery performance in rural areas. In July 2015, the Senate report accompanying the Senate Financial Services and General Government Appropriations Bill, 2016 directed USPS and PRC to report mail delivery performance to specifically include mail delivery from rural towns to other rural towns; from rural towns to urban areas; and from urban areas to rural towns. The Committee requested the methodology used to develop this information within 60 days of enactment of the Act with a subsequent report due by March 1, 2016.

USPS has not reported data on on-time delivery performance based on a rural or non-rural distinction. USPS officials told us that no overall assessment of rural delivery service, separate and apart from urban/suburban delivery, has been undertaken since PAEA required delivery performance measurement, reporting, and assessment. USPS officials added that its delivery performance data provide a basis for internal diagnosis and assessment of operations and service, and satisfy USPS’s reporting obligations to PRC. USPS officials noted that its reports are generated at the national, area, and district level for these purposes, but are not routinely further disaggregated on the basis of whether particular districts or ZIP Codes are rural, suburban or urban in nature. On-time delivery performance information at the district level cannot inform stakeholders on delivery performance in rural areas since each of USPS’s 64 districts in the continental United States contains at least one core area with a population over 10,000 and thus is not entirely rural.

USPS officials told us that USPS’s service performance measurement systems do not differentiate between rural and urban locations and that it may be cost prohibitive to attempt to measure performance of mail pieces in rural areas using an external data system.

However, in response to the recent congressional request for PRC to report on rural mail delivery performance, USPS told us that it has begun collaborating with the technical staff at PRC to determine how measurement may account for rural origin and destination points and that its new, proposed internal service performance measurement plan might provide greater insight on service performance measurement specific to rural areas, assuming USPS and PRC can arrive at a reasonable definition of “rural” origin and destination points. At this time, however, USPS officials added that USPS was at an exploratory stage of the analysis and were not able to offer definitive conclusions on the feasibility of adding this feature to USPS’s measurement plans.

PRC officials told us that they are currently working with USPS to determine how they will respond to the congressional request for rural delivery performance information. PRC officials also told us that PRC has limited its previous assessments regarding whether USPS met its delivery service standards for market-dominant types of mail to national results and has not conducted any rural-level analysis. PRC officials told us that PRC does not play a direct role (e.g., either annually or quarterly) in monitoring or reporting on USPS’s universal delivery service obligation (aside from annually estimating the cost of universal postal service), noting that PRC is not legally required to do so, nor has PRC been directed by Congress to play this role. PRC officials added that PRC has not considered requiring USPS to report quarterly and annual information on delivery speed and reliability in urban versus rural areas, because 1) PRC has not been specifically mandated by statute to require USPS to provide delivery service performance information separately for rural and urban areas and 2) in PRC’s previous assessments, Congress has not provided specific direction requiring USPS to implement such measurement and reporting.

As noted previously, USPS officials told us that the costs of additional requirements for USPS to collect and report urban and rural delivery performance information through existing measurement systems would likely greatly outweigh the benefits. However, USPS and PRC were not able to provide specific cost estimates related to having USPS measure and report on delivery performance in rural and urban areas. We asked USPS for this information, but it did not provide such cost information, with USPS officials explaining that there is no clear definition or defined approach to measure what should be considered rural. USPS officials also told us that the cost would depend on the specificity of the data, such as whether there would be national-level results for urban and rural areas or detailed geographic breakdowns. We also asked PRC about the costs of providing delivery performance information in rural and urban areas. PRC responded that it has the authority to specify requirements for USPS’s delivery performance measurement, but that when considering reporting requirements for USPS, it is to give consideration to unnecessary or unwarranted administrative effort and expense by USPS. On this matter, PRC officials said that they do not know what the costs might be for USPS to collect data on delivery performance in rural and urban areas. Neither of the congressional directives mentioned above regarding studying delivery performance in rural areas directly address the costs associated with requiring rural delivery performance information. Without cost estimates, Congress may not have all the information it needs to understand the full implications of requiring data on delivery services in rural and non-rural areas.

Conclusions

Quality delivery performance information is needed for USPS and postal stakeholders such as PRC, Congress, business mailers, and the general public to develop useful analysis that can help oversee or assess the balance between USPS’s cost-cutting to address its poor financial situation while maintaining affordable postal rates and providing timely, universal delivery service. Thus, it is important for both USPS and PRC to report delivery performance information in a sufficiently complete, transparent, and useful manner.

Although USPS has made progress since PAEA was enacted in 2006, its delivery performance information is not complete, and it is unclear when USPS will achieve its goal of measuring on-time delivery for nearly all market-dominant mail volume. USPS measured on-time delivery for only 55 percent of market-dominant mail volume in the second quarter of fiscal year 2015. As a result, data may not be representative because performance may be different for mail not included in measurement. Although PRC’s reports provide data on the amount of mail included in measurement, they have neither fully assessed the reasons why these measurements are incomplete, nor specified what actions USPS needs to take and the related time frames needed to achieve complete performance measurement. PRC may initiate proceedings to improve the completeness and quality of delivery performance data, but it has not exercised this option. Although USPS and PRC are opposed to such a proceeding, we believe that a PRC proceeding that focuses on issues of data completeness—particularly the problem of excluding mail due to a lack of information—could facilitate more rapid progress by USPS and the mailing industry toward complete measurement.

USPS and PRC annual and quarterly reports on delivery performance information are not as useful for oversight purposes beyond the annual compliance assessments because they do not include sufficient analysis that would facilitate holding USPS accountable for meeting its statutory mission to provide prompt, reliable, and efficient services in all areas of the nation, including rural areas. For example, neither USPS nor PRC reports trend data below the national level for all of USPS’s 67 districts to indicate whether performance is improving or getting worse in different parts of the nation. Further, delivery performance information is not sufficiently transparent as it is not readily available or easily accessible on either USPS’s or PRC’s website. Also, postal stakeholders—such as PRC, Congress, business mailers, and the general public—cannot determine whether delivery performance is a problem in rural areas because USPS and PRC are not required to report delivery performance information separately for rural versus non-rural areas. USPS believes that such an analysis would be costly, even though it does not know how much it would actually cost. Such cost information would be useful for Congress to have in order to assess whether developing this information would be appropriate. In addition, USPS and PRC are in the process of responding to a recent congressional request to determine the feasibility of reporting on rural mail delivery performance, which could facilitate determining the associated costs.

Matter for Congressional Consideration

To assist in determining whether to require USPS and PRC to report on delivery performance for rural and non-rural areas, Congress should direct USPS to provide cost estimates related to providing this information.

Recommendations for Executive Action

To improve the completeness of USPS delivery performance information, we recommend that the Acting Chairman of PRC and the other PRC Commissioners exercise PRC’s statutory authority to hold a public proceeding involving USPS, the mailing industry, and interested parties to address how USPS can improve the completeness of USPS’s delivery performance information.

To improve the usefulness and transparency of USPS’s and PRC’s reporting of delivery performance information, we recommend that:

The Postmaster General provide additional and readily available delivery performance information, such as trend data for on-time delivery performance for all 67 postal districts.

The Acting Chairman of PRC and the other PRC Commissioners provide readily available data and additional analysis of USPS’s delivery performance information so that stakeholders can better understand trends and variations in mail delivery performance.

Agency Comments and Our Evaluation

We provided a draft of this report to USPS and PRC for review and comment. USPS and PRC provided written responses, which are reproduced, respectively, in appendixes II and III of this report. PRC and USPS agreed with the recommendations addressed to them. Specifically, PRC agreed to hold a proceeding to address how USPS can improve the completeness of USPS’s delivery performance—after, as we reported, initially indicating it was opposed to such a proceeding. Although not the addressee of this recommendation, USPS disagreed with it stating that its measurement systems conform to the Office of Management and Budget’s standards and guidelines for statistical surveys, and that it employs a contractor with long-standing expertise in developing statistically valid and reliable systems. However, we found that key data quality issues involve the lack of completeness of census-type measurement and the associated risk of non-sampling error—issues that are separate from matters of statistical design. Further, USPS said its continuing collaboration with the mailing industry is more likely to stimulate industry cooperation and buy-in rather than lengthy, time- consuming proceedings before the PRC. We continue to believe that a new PRC proceeding on data quality would add value to its continuing collaboration with USPS and the mailing industry. Our report notes that representatives of mailing industry groups and some mailers told us, and commented in PRC proceedings, that PRC should become more involved in issues of the quality of measurement data for on-time delivery performance, including issues regarding the exclusion of mail from measurement.

Both USPS and PRC agreed with our recommendations to improve the usefulness and transparency of delivery performance information that they report. USPS acknowledged that the delivery service performance data it reports on its website lacks the granularity of the reports it publicly files with PRC, and does not serve the purposes of in-depth congressional oversight. USPS added that although it is not clear that typical household mailers would use or find value in that level of data granularity, it will pursue establishing a distinct portal for public access to delivery service performance reports, in an effort to be more transparent. Likewise, PRC agreed that information it receives and produces regarding performance measures for USPS can be better organized, and it has updated its website in response to our recommendation. We are encouraged by USPS’s and PRC’s willingness to adopt these the Congress’s recommendations since, as we have recently reported,and the public’s confidence in the quality of performance information that federal agencies are using to assess and achieve results requires that information be publicly reported in a clear and readily accessible way.

Although both agencies agreed with the recommendations addressed to them, they disagreed with certain findings, conclusions, and the supporting analytical basis used in this report. Key among the disagreements were our treatment of the completeness of USPS’s data, the appropriateness of the criteria we used for our assessment of PRC’s oversight and analysis, and the usefulness of USPS’s and PRC’s reporting on delivery performance.

Regarding our treatment of data completeness, USPS said it understood in the abstract the basis for our finding that delivery performance may differ for mail included in measurement than mail that is not measured. However, it disagreed that this is the case in practice. Specifically, USPS stated that mail pieces that are included in performance measurement should have virtually the same performance results as mail not included in measurement. Additionally, PRC stated that data reliability has markedly improved as a result of PRC’s directives to USPS regarding measurement systems. As discussed in our report, although the completeness of measurement has improved over the past 9 years, 45 percent of market-dominant mail is still not included in measurement. Also, we identified a number of reasons to be concerned that delivery performance may be different for mail that is included in measurement than mail not included in measurement. For example, large-volume mailers, who are most likely to apply barcodes and thus have on-time delivery performance measured, use additional mailing practices to facilitate the timely delivery of their mail, such as entering large volumes of advertising mail close to its final destination. In addition, destination- entry mail has sufficient volume and preparation to enable it to bypass various postal network processing and transportation (e.g., locally entered and delivered so it is handled by only one processing facility)—a strategy that, according to USPS, is the reason this mail is more likely to be delivered on time. As we reported, USPS has set a goal of including virtually all market-dominant mail in measurement, using a census-type approach, and continues to strive for including more mail volume in measurement. Further, USPS raised concern that increasing the proportion of mail included in measurement data may come at potential significant costs, which we believe is a topic that could be further explored in the recommended proceeding.

USPS disagreed with the specific example given in our report that destination-entered Standard Mail is more likely to be included in measurement than other “end-to-end” Standard Mail and is more likely to be delivered on time. This example suggests that results for Standard Mail as a whole are higher than they would be if all Standard Mail were included in measurement. USPS said that when compiling the national on-time delivery percentage for all Standard Mail, it weights results for measured mail pieces by shape and entry type so they are compiled in proportion to their prevalence in the entire population of Standard Mail. However, it does not appear that USPS applies weighting procedures when compiling results for individual Standard Mail products—such as Standard Mail Flats and Standard Mail Letters that have significant proportions of both destination-entry and end-to-end mail. Therefore, we have clarified our report to state that available information indicates that non-participation can affect results for some Standard Mail products, particularly if product-specific results are not weighted to reflect key characteristics of the mail.

In addition, USPS and PRC made various comments on the importance of the statistical properties of USPS systems that measure on-time delivery performance. USPS said it employs a firm with long-standing expertise in developing measurement systems that are statistically valid and reliable, and provided a letter from this contractor that the measurement systems are designed in a manner to be statistically valid and representative. PRC said it reviews USPS data using statistical principles that determine whether service performance data are sufficient and the results are meaningful. Specifically, PRC said sampling fractions, confidence intervals, and margins of error are the primary factors it uses to determine whether data are accurate and reliable. We agree statistical considerations should inform the assessment of data collected through sampling. However, we also note that such statistical principles are not relevant to evaluate the quality of incomplete data collected using census- type measurement, which is the case for most types of market-dominant mail. An error created when non-measured mail has different on-time delivery performance than measured mail is a “non-sampling error”—as opposed to a “sampling error” that is associated with measurement based on a random sample. Non-sampling errors can affect results, regardless of how valid the statistical design of USPS’s measurement systems may be. As previously discussed, results based on incomplete data can be affected when the measurement process disproportionately includes mail that is more likely to be delivered on time.

PRC also was critical of our focus on data completeness, stating that it is not a meaningful statistical measure and that PRC has not concluded that the percentage of mail in measurement should be the primary determinant of accurate, reliable, or representative measurement data. USPS stated that our report does not specify what an appropriate level of measurement may be.

However, PRC has not defined what an appropriate percentage of mail in measurement would be for measuring on-time delivery based on census- type measurement, while USPS has set a goal of including virtually all mail volume in measurement. For most types of market-dominant mail, measuring on-time delivery performance involves census-type measurement as well as measurement based on sampling. For the mail that is included in the measurement based on a census-type approach, to assess non-sampling error would require determining whether the mail not included in measurement systematically differed from the mail included in the measurement, particularly regarding characteristics associated with on-time delivery.

Regarding the appropriateness of the criteria we used for our assessment of PRC’s oversight and analysis, PRC stated that our assessment was based on GAO-created criteria rather than the statutory requirements in PAEA. We agree that our assessment was not intended to determine PRC’s compliance with its statutory requirements. Rather, our review used criteria that are appropriate for assessing an organization’s practices for reporting delivery performance information that would be useful for management and congressional decision-making. As noted in our report, the criteria we used to assess USPS’s and PRC’s measurement and reporting of delivery performance information are based on current laws—including PAEA—and regulations, as well as previously identified practices used by high-performing agencies, and prior GAO reports. Specifically, our criteria are a result of reviewing delivery performance measurement and reporting provisions applicable to USPS and PRC in PAEA, and PRC regulations, which we summarized in table 3. In addition, we believe that certain government principles can help inform congressional and executive branch decision-making to address challenges. For example, USPS should disclose more information about the accuracy and validity of its performance data and actions to address limitations to the data. Our prior work has found that without useful performance information, it is difficult to monitor agencies’ progress toward critical goals.

PRC also disagreed with a statement in our draft report that its reports have not assessed why USPS’s delivery performance measurements were incomplete nor specified what actions USPS needs to take to achieve complete performance-measurement data. PRC said it has assessed the primary reasons measured mail may be inaccurate, unreliable, or not representative of nationwide performance, including data not in Full-Service Intelligent Mail, uncategorized mail, invalid data, and low district-level volumes. PRC said its reports have regularly directed USPS to improve data reliability and accuracy by increasing participation in Full-Service Intelligent Mail, increasing measured volumes for mail product categories in certain districts, and increasing the number of districts providing results.

We agree that PRC reports have addressed some issues related to the quality of delivery performance data, such as providing data on the amount of mail included in delivery performance measurement and expressing concern with low levels of participation for certain types of mail. However, as we discuss in our report, PRC has not fully assessed why these measurements were incomplete, whether USPS actions will achieve complete performance data, why lack of participation remains a significant issue, and whether there are practical opportunities to make progress. Further, recent PRC reports have not assessed what have become the primary causes for excluding mail pieces from measurement, including no “start-the-clock” information, no mail piece barcode scan recorded by USPS automation equipment, and inaccuracies in mail preparation. Thus, we continue to believe that PRC has opportunities to improve its oversight and encourage PRC and all stakeholders to explore these causes in its forthcoming proceeding.

Regarding the usefulness of reporting, PRC and USPS disagreed with our characterization that USPS’s and PRC’s reports are not sufficiently useful for effective oversight. PRC objected to the implication that it is not fully successful in meeting its oversight responsibilities, and added that it has provided strong oversight in achieving the transparency and accountability required by Congress and that its reports are useful. While we recognize that PRC is not statutorily required to assess USPS’s performance in providing mail to all parts of the country—including rural areas—USPS is still responsible for adhering to these requirements and no other oversight agency exists to hold USPS accountable to these requirements. Given the broad scope of recent changes in postal operations, network consolidations, and service standard changes, Members of Congress and other postal stakeholders have raised concerns about the impact of these changes on delivery performance. Thus, effective oversight is even more critical to ensure that any delivery performance problems are promptly identified and addressed.

USPS and PRC also provided technical comments that we incorporated as appropriate.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Postmaster General, the Acting Chairman of the Postal Regulatory Commission (PRC), the other PRC Commissioners, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff has any questions about this report, please contact me at (202) 512-2834 or rectanusl@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff making key contributions to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

This report assesses (1) the U.S. Postal Service’s (USPS) measurement of mail delivery performance and the Postal Regulatory Commission’s (PRC) oversight of this measurement and (2) USPS’s and PRC’s reporting of this information.

To conduct this work, we assessed whether USPS’s measurement of its delivery performance is complete and whether USPS’s and PRC’s reporting on this performance is useful and transparent. To make our assessments, we compared USPS’s and PRC’s measurement and reporting efforts to specific elements associated with these criteria. We originally developed these criteria for a 2006 report that assessed USPS’s delivery service standards, measures, and reporting. In developing those criteria, we identified applicable laws related to USPS’s mission, ratemaking, and reporting, and practices used by high-performing organizations related to delivery service standards, measurement, and reporting, including practices identified through our past work. For this review, as table 4 below illustrates, we adapted and updated each criteria identified in the 2006 report. We reviewed current laws, previously identified practices used by high-performing agencies, and prior GAO reports to identify specific, observable elements associated with each criteria, in order to make a more direct assessment on the extent delivery performance information is complete, useful, and transparent. For example, we reviewed provisions in the Postal Accountability and Enhancement Act (PAEA) and implementing PRC regulationsestablished the legal framework for measurement of mail delivery that performance, PRC’s oversight of this measurement, and reporting of this information. To identify practices for reporting delivery performance information that would be useful for management and congressional decision making, we reviewed the Government Performance and Results Act of 1993 (GPRA), and the GPRA Modernization Act of 2010 framework for meeting fiscal, management, and performance challenges, practices used by high-performing agencies, and prior GAO reports.

To assess delivery performance measurement, we reviewed documentation of mail delivery performance, the measurement systems used to develop this information, and limitations of these systems. We also reviewed USPS’s annual reports to Congress and PRC, PRC’s annual compliance determinations, Mailers’ Technical Advisory Committeemeasurement systems and the data USPS collects. In addition, we reviewed relevant documentation regarding USPS’s proposal to replace its External First-Class Mail measurement system (EXFC), including USPS’s proposal, stakeholder comments on the proposal, and USPS’s reply responses to stakeholder comments. Between December 2014 and June 2015, we received written responses and data from USPS and PRC related to mail delivery performance measurement and associated limitations and interviewed USPS and PRC officials. USPS’s responses contained data on the amount of mail ineligible for delivery performance measurement and excluded from delivery performance measurement in fiscal years 2010 through the second quarter of fiscal year 2015. We assessed the reliability of USPS’s data through a review of related documents, such as written responses from USPS. We found these data sufficiently reliable for providing a general description related to the completeness of delivery performance information. To assess PRC’s oversight of delivery performance information, we reviewed PRC’s annual compliance determinations and other reports, obtained written responses from PRC and USPS, and interviewed PRC and USPS officials. We also interviewed representatives of mailing industry groups and business mailers with expertise on delivery performance measurement and postal issues to discuss the completeness of delivery performance information reported by USPS and PRC’s assessment of this information. We used our professional judgment to select these representatives; thus, the responses we received from them are not generalizable to the entire mailing industry. We also reviewed laws, regulations, and PRC orders and determinations to identify any guidance or requirements for USPS and PRC related to the quality of delivery performance information. presentations, and other documentation on USPS’s current To assess reported delivery performance information, we reviewed the mail delivery performance information reported in USPS annual reports to Congress, PRC annual compliance determinations and other reports, and on the USPS and PRC websites. We assessed the usefulness of the reported information to provide oversight over how effectively USPS fulfills its statutory mission to provide prompt, reliable, and efficient services to all areas of the country (universal delivery service), and a maximum degree of effective service in rural areas. We also interviewed representatives of mailing industry groups and business mailers with expertise on delivery performance measurement and postal issues to discuss the usefulness of delivery performance information reported by USPS and PRC. We used our professional judgment to select these representatives; thus the responses we received from them are not generalizable to the entire mailing industry. To determine the extent that the delivery performance information is transparent, we reviewed delivery performance information USPS and PRC disclose on their websites to assess the extent to which it is easily accessible and readily available. We also reviewed laws and statutory regulations to identify any requirements related to reporting delivery performance information in a transparent manner.

We conducted this performance audit from October 2014 to September 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the U.S. Postal Service

Appendix III: Comments from the Postal Regulatory Commission

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the individual named above, key contributors to this report were Teresa Anderson (Assistant Director); Samer Abbas; Kenneth John; Thanh Lu; Malika Rice; Amy Rosewarne; Kelly Rubin; and Crystal Wesco.