U.S. Postal Service: Addressing Policy Gaps Could Improve Pilot Design and Evaluation for Postal Innovations
Released: 2019-03-14T00:00:00

Letter

Background

USPS’s Financial Position and Strategic Goals

USPS’s mission is to provide universal postal service while operating as a self-financing entity, but USPS’s current financial position is not sustainable. To achieve its mission, USPS must cover its expenses through revenues generated from the sale of its products and services. However, USPS’s total operating expenses have exceeded total operating revenue each year since fiscal year 2007, including a $2.6 billion loss from operations in fiscal year 2017 alone (see fig. 1). Moreover, we have reported that USPS’s overall financial condition is deteriorating. For example, in August 2018 we reported that USPS had about $149 billion in unfunded liabilities and debt at the end of fiscal year 2017. As a result, USPS’s financial condition remains on our list of high- risk areas needing attention by Congress and the executive branch.

According to USPS financial documents, its ability to sell innovative products and services will be a key factor in improving its financial condition. Thus, USPS established a strategic goal to “innovate faster to deliver value” to its customers, by making investments in innovations that respond to rapidly evolving customer needs. A key element of this effort is to accelerate testing of innovative products and services to better serve these needs, according to USPS.

While USPS is allowed to develop certain new postal products and services, there are statutory restrictions that currently limit the range of innovations USPS can offer. For example, under current statute, USPS is not permitted to ship alcoholic beverages. Similarly, although USPS is explicitly authorized to provide services to federal executive agencies (e.g., passport services), such authorization does not include services to state, local, and tribal governments. Legislation has been introduced in previous sessions of Congress that would permit USPS to deliver alcoholic beverages and allow USPS to provide property and services to state, local, and tribal governments under certain conditions. According to USPS officials, USPS supports these legislative proposals, which could enhance its ability to offer innovative products and services.

USPS Postal Innovation Pilot Policies and Responsibilities

According to USPS officials, two USPS handbooks include policies applicable to piloting key innovations. Specifically, the first handbook includes requirements, procedures, and responsibilities for all types of investment programs and projects undertaken by USPS, regardless of size, cost, or complexity. This handbook, for example, requires the identification and documentation of lessons learned for all investments and projects. The second handbook includes requirements and procedures specifically for major operating expense investments. This handbook, among other things, establishes requirements and procedures meant to ensure that new and enhanced products and services consistently meet customer needs, generate new revenue, and strengthen USPS as a business. This handbook further states that USPS has a responsibility to subject new initiatives to rigorous financial analysis, testing, and measurement, to determine whether these initiatives will make a positive financial contribution to the organization and ensure that USPS’s leadership has appropriate information for effective decision- making.

USPS’s Office of Product Innovation generally has lead responsibility for piloting innovations. According to USPS’s policies, a project manager is responsible for establishing and coordinating a cross-functional team to design and evaluate the pilot. This team typically includes officials from a variety of USPS departments, such as finance, general counsel, information technology, marketing, and operations. The project manager, with support from the cross-functional team, is responsible for preparing a proposal for the pilot that includes key information, such as the pilot’s objectives and performance measures, and overseeing pilot implementation and communication with key stakeholders.

In some cases, PRC has a role in overseeing postal innovation pilots. For example, USPS must notify PRC before it pilots any postal product innovation for which it will impose a price (i.e., a pilot that generates revenue for USPS) and must subsequently report quarterly revenue, volume, and cost data. PRC also ensures that certain safeguards are maintained during the pilot, such as limitations on the pilot’s duration and revenue. However, according to PRC officials, the commission has limited involvement in other areas of USPS’s efforts to develop innovations.

USPS Piloted Key Innovations Are Intended Primarily to Generate Revenue or Improve Customers’ Experience

USPS piloted 24 key innovations from fiscal years 2013 through 2017. For example, USPS piloted an Identity Verification Service that allows users to verify their identity either remotely (i.e., online) or in person at a postal facility. Similarly, USPS piloted an innovation to allow mailers to print shipping labels, track packages, and schedule package pick-ups by accessing USPS data. The primary goal of the majority of these key innovations (16 of 24) was to generate revenue, while the primary goal for the remaining innovations was generally to improve customers’ experience using USPS products or services (see fig. 2). Appendix I includes a complete list of key innovations USPS piloted from fiscal years 2013 through 2017.

The following discussion provides additional information about the 4 key innovations we selected as illustrative examples of USPS’s efforts to pilot innovative products and services.

Same-Day Delivery: From December 2012 to December 2015, USPS piloted same-day delivery for consumer e-commerce purchases (see fig. 3). According to the pilot proposal, this innovation was intended to generate revenue for USPS by allowing it to leverage its existing delivery infrastructure to capture part of the growing e-commerce market. To determine the potential scalability of same-day delivery, USPS first tested its operational feasibility and potential demand in several major metropolitan areas, including San Francisco, New York, and Phoenix. During the pilot, USPS delivered photos, chocolates, water, electronics, and other goods from 38 participating mailers to consumers in these areas. At the pilot’s conclusion, USPS decided to continue offering same-day delivery to interested participating mailers under Priority Mail contracts.

Grocery Delivery: From November 2014 to October 2017, USPS piloted a grocery delivery product in nine selected metropolitan areas. According to USPS, the innovation was intended to generate additional revenue by taking advantage of the growing market for grocery delivery. To test the innovation’s operational feasibility, USPS required the pilot’s sole participating mailer to bring totes containing groceries and other prepackaged goods ordered by customers directly to post offices (see fig. 4). USPS was then responsible for sorting the totes and delivering them to customers. According to its proposal for this pilot, USPS expected grocery delivery to provide a substantial revenue generation opportunity. At the pilot’s conclusion, like same- day delivery, USPS decided to continue offering grocery delivery with the participating mailer under a Parcel Select contract.

Informed Delivery: From spring 2014 through July 2016, USPS piloted a notification service called Informed Delivery in Northern Virginia and New York. According to USPS, this innovation is intended to bridge the gap between the physical and digital worlds by, for example, emailing customers with a scanned image of the exterior address side of letter-sized mail they should receive later that day (see fig. 5). Informed Delivery can also allow mailers to conduct marketing campaigns by integrating other elements—such as hyperlinks to mailers’ websites—into the email and other notifications that customers receive. In its proposal to pilot Informed Delivery, USPS stated the pilot was intended to help USPS understand the service’s business opportunity and increase the certainty of its potential benefits, which included retaining mail volume and generating new revenue from large advertisers. In addition, the pilot aimed to generate “statistically valid data” on how subscribers respond to marketing campaigns that mailers conduct. According to USPS, more than 70,000 customers were actively using the service at the pilot’s conclusion. In July 2016, USPS decided to end the pilot and launch the service nationally. According to USPS, about 13 million customers were subscribed to the service as of October 2018. USPS aims to have 40 million customers subscribed to the service by 2020.

Keyless Parcel Lockers: Since October 2013, USPS has piloted keyless parcel lockers that allow customers to independently pick up packages in 98 selected post offices. According to USPS, among other things, this innovation is intended to reduce the number of missed package deliveries to customers’ post office boxes and thereby reduce USPS’s delivery costs (see fig. 6). The purpose of the pilot is to assess the performance and use of the lockers and to assess their performance. In October 2013, USPS began pre-testing the technical performance of 10 prototype keyless parcel locker units at post offices in New York City and Northern Virginia. Following this pre-test, in February 2015, USPS approved the installation of 50 made-to-order locker units in selected post offices across the country. Finally, in May 2016 USPS expanded the pilot to include an additional 50 units, including 2 units that a senior USPS official told us were not yet installed. As of November 2018, the pilot is still ongoing.

USPS’s Policies for Piloting Key Innovations Reflect Some but Not All Leading Practices for Pilot Design and Evaluation

USPS’s policies applicable to piloting key innovations fully reflect two of the five leading practices for pilot design and evaluation that we identified in prior GAO work and relevant standards for internal control (see table 1). These policies do not, however, fully reflect the other three leading practices due to policy gaps. Further, we found that USPS had not consistently followed its policies to document lessons learned at the conclusion of each pilot, as discussed below.

Senior USPS officials acknowledged that gaps exist in its policies for pilot design and evaluation because they were not developed by USPS to fully reflect all leading practices. These policy gaps limit the extent to which USPS can ensure that it is making good resource allocation decisions based on pilot experiences. Below we further discuss the extent to which USPS’s policies reflect the five leading pilot practices we identified as well as how USPS applied these leading practices among the four piloted innovations that we reviewed.

Establish appropriate and measurable objectives linked with identified performance measures: We found USPS’s policies do not fully reflect this leading practice. While USPS policies require that project managers establish pilot objectives and performance measures, they do not require that each objective be linked with identified performance measures. As a result, some pilots may have objectives without an associated performance measure.

For example, although USPS established both objectives and performance measures for each of the four innovations we selected for review, it did not consistently link each established objective to performance measures. USPS’s proposal to pilot same-day delivery, for example, had objectives of generating new revenue and improving customers’ experience. However, while the proposal included performance measures associated with generating new revenue—i.e., package volume, gross revenue, and net revenue—it did not identify and link any performance measures with its objective of improving customer experience. Similarly, USPS’s proposal to pilot keyless parcel lockers included improving customers’ experience as one of its objectives. However, while the proposal included a variety of performance measures—reduction in the number of missed deliveries to post office boxes, locker rate utilization, and on-time locker installation—it did not identify and link any performance measures with its improving customers’ experience objective. Absent such measures, USPS may not know whether customers have experienced an improvement using keyless parcel lockers compared to using manual, keyed parcel lockers. Linking all objectives to performance measures could help ensure that USPS has the performance information to assess the extent to which a pilot has achieved all of its objectives.

USPS officials told us that it can be difficult to measure performance for some objectives related to customer experience. While measuring customers’ experience can be challenging, it is important to understand the extent to which a pilot has achieved all of its objectives. Further, USPS has demonstrated that it can measure improvement in customers’ experience. For example, during its Informed Delivery pilot, USPS conducted a consumer survey with approximately 5,500 Informed Delivery subscribers to collect data on consumer adoption and satisfaction. In the survey, USPS found that over 80 percent were satisfied or very satisfied with the service. According to USPS officials, this data helped USPS to measure the pilot’s success in meeting its objective of improving customers’ experience.

Articulate a methodology for evaluating pilot performance: We found that USPS’s policies fully reflect this leading practice because the policies require officials to develop and communicate a methodology for evaluating pilot performance. Articulating such a methodology helps managers to identify the types and sources of performance information necessary to evaluate the pilot. USPS’s policies require project managers to work with the pilot’s cross-functional team to develop and reach consensus on the methodology. These policies also require the project manager and cross-functional team to identify data needs, data sources, and how the data will be evaluated.

For the four innovations we reviewed, we found that USPS articulated a methodology for evaluating the pilot’s performance. For example, for the Informed Delivery pilot, USPS identified its customer registration system as the method for tracking progress toward performance measures related to the number of Informed Delivery subscribers. Similarly, in its proposal to pilot keyless parcel lockers, USPS identified its central parcel locker monitoring system as a method of tracking progress toward performance measures related to utilization of keyless parcel lockers.

Evaluate pilot performance and identify and document lessons learned: We found that USPS’s policies fully reflect this leading practice, but USPS did not consistently follow its policy that requires documenting lessons learned. Specifically, the policies require project managers to evaluate performance and document lessons learned at the conclusion of each pilot. Doing so can enable USPS to identify information needed to make conclusions about the pilot’s scalability and ensures that such information will be accessible to inform future related efforts. However, among the key innovations we selected for review USPS had not consistently documented lessons learned (see table 2).

USPS officials told us that they discussed lessons learned during ongoing monitoring of pilot performance for these innovations, but had only documented lessons learned for its Informed Delivery pilot. Specifically, USPS identified lessons learned in its July 2016 proposal to launch its Informed Delivery service nationally. In this proposal, we found that USPS identified some lessons learned about the pilot related to user satisfaction and adoption rates. USPS officials told us that this information helped to inform USPS’s decision to launch the service nationally. However, USPS officials acknowledged that USPS did not document lessons learned for the other two concluded pilots that we selected for review (same-day delivery and grocery delivery).

Senior USPS officials told us that USPS had not consistently documented lessons learned at the conclusion of pilots across the 24 key innovations because it had not developed tools, such as a template, or training that could help ensure such consistency. Without consistently documenting lessons learned for all of its pilots, USPS risks losing information garnered during pilot implementation that could be relevant to future innovation efforts. Doing so can be particularly important because, according to a senior USPS official, officials responsible for pilot projects sometimes retire or leave USPS for employment elsewhere, creating a gap in knowledge of pilot experiences. Standards for internal control underscore the importance of maintaining documentation in order to retain organizational knowledge and mitigate the risk of having knowledge limited to a few personnel.

Draw and document conclusions about scalability based on pilot results: USPS’s policies do not fully reflect this leading practice. These policies require that project managers draw conclusions based on the results and lessons learned from the pilot. According to USPS officials, conclusions may include determining scalability—i.e., whether, how, and when to integrate pilot activities into overall efforts. However, USPS’s policies do not specifically require that officials document these conclusions. Documenting conclusions about scalability based on pilot results helps to ensure retention of organizational knowledge related to the pilot that may inform future decisions.

Among the three innovations that we selected for review for which the pilots had concluded (i.e., same-day delivery, grocery delivery, and Informed Delivery), USPS officials told us that senior leadership discussed the results and lessons of the pilots and made determinations regarding whether, how, and when to launch them more broadly, but that they did not document these decisions or the rationale for them. By not documenting conclusions, USPS risks losing information that could affect the success of future related efforts and that could inform future USPS leadership of the rationale for maintaining investments in activities upon which pilots were based.

Documenting conclusions for innovation pilots can be especially important in cases in which USPS decides to continue or expand pilot activities even when the pilots do not meet all of their intended objectives. For example, USPS’s same-day delivery and grocery delivery pilots had revenue objectives, along with associated performance measures; however, neither pilot achieved these objectives. For the same-day delivery pilot, costs exceeded revenue in 12 of the 13 fiscal year quarters in which the pilot was conducted, according to data USPS reported to PRC. Likewise, USPS data indicate it did not reach its annual revenue target for its grocery delivery pilot. Similarly, USPS’s pilot of Informed Delivery was intended to generate “statistically valid data” on how consumers respond to mailer marketing campaigns. However, according to a senior USPS official, the pilot did not generate the data as intended, because no such campaigns were conducted during the pilot.

As discussed earlier in this report, USPS did not discontinue any of these three selected innovations when their pilots concluded. Although USPS may have had good reasons to continue with, or more broadly launch, these innovations despite the pilots not meeting all of their objectives, the lack of documentation regarding its reasoning and decisions limits information relevant to whether USPS is making judicious use of limited resources.

Ensure appropriate two-way communication at all stages of the pilot with key internal and external stakeholders in order to understand and address their views: USPS’s policies do not fully reflect this leading practice.

USPS’s policies require the involvement of key internal stakeholders in pilots. Specifically, USPS’s policies require the involvement of cross- functional teams—which include legal, finance, and other departments— and varying levels of review during the design and implementation of pilot proposals. However, USPS’s policies do not address communication with key external stakeholders. According to USPS officials, some pilot projects may be confidential or have limited or no direct effect on external stakeholders and, thus, communication with external stakeholders may not be appropriate. While external stakeholder communication may not be appropriate with some pilots, such communication, as appropriate, can help to ensure that issues critical to the success of a pilot activity are identified and addressed.

Among the innovations that we selected for review, USPS officials explained various steps taken to involve internal stakeholders in the design and evaluation of the pilots, such as the involvement of cross functional teams to develop pilot proposals. Further, while USPS’s policies do not address external stakeholder communication, we found that USPS employed strategies for some of the innovations we selected for review to communicate with some external stakeholders—i.e., industry associations and mailers. For example, a representative of a postal association told us that USPS shared information and sought input about its Informed Delivery pilot during a quarterly meeting with industry groups. Similarly, a mailer we interviewed told us that USPS had shared information and sought input on the Informed Delivery pilot through direct outreach with the mailer.

However, USPS did not consistently employ strategies to communicate with some key external stakeholders among the innovations that we selected for review. Specifically, USPS did not design or implement strategies to obtain feedback from consumers on its pilots for same-day delivery, grocery delivery, or keyless parcel lockers, despite the fact that each of these innovations directly affected consumers. In contrast, as previously discussed, for its Informed Delivery pilot, USPS planned and conducted a survey to obtain consumer feedback, the results of which helped USPS project managers support the proposal to expand the service nationally. Absent communication with all key stakeholders, USPS risks not having a complete understanding of perspectives that could inform the viability of its innovations.

Conclusions

In recent years, USPS has sought to compete in a challenging business environment by piloting innovations intended primarily to generate revenue and enhance customers’ experience. The policies that USPS uses for piloting key innovations fully reflect some leading practices for pilot design and evaluation, such as articulating a methodology for evaluating pilot performance. However, addressing gaps between USPS’s policies and leading practices related to linking objectives and performance measures, documenting conclusions, and communicating with key external stakeholders would enable USPS leadership to better assess the outcomes of its pilots, understand the rationale for conclusions about scalability based on pilot results, and gauge customers’ reactions to innovative products and services. Moreover, developing tools or training to ensure that USPS consistently implements its policy of documenting lessons learned from pilots would provide USPS with key information to inform future related efforts.

Recommendations for Executive Action

We are making the following two recommendations to USPS: The Postmaster General should direct the Vice President of Product Innovation to develop policies that fully reflect leading practices for pilot design and evaluation in areas such as linking objectives and performance measures; documenting conclusions about scalability based on pilot results; and communicating with key external stakeholders, as appropriate. (Recommendation 1)

The Postmaster General should direct the Vice President of Product Innovation to develop tools, such as a template, or training to help ensure USPS consistently documents lessons learned at the conclusion of pilots, as required by USPS policies. (Recommendation 2)

Agency Comments and Our Evaluation

We provided a draft of this product to USPS and PRC for comment. USPS provided a written response, which is reproduced in appendix II of this report. In its response, USPS did not state whether it agreed with our recommendations, but described actions that it plans to take related to each. These actions, if fully implemented, would meet the intent of our recommendations. For example, USPS stated that it would develop policies specifically for pilot design, and would reflect leading practices for pilot design and evaluation based upon best practice research. USPS also noted that it would develop training to ensure consistent documentation of lessons learned from its pilots. USPS added that this planned training would cover best practices for pilot tests.

Regarding our first recommendation USPS said that pilots are only one step in a larger process for developing innovations. We agree with this and noted in our report that piloting is one key element of USPS’s efforts to innovate. Nonetheless, given USPS’s financial position, effectively piloting innovations is a critical step to ensure that USPS invests its limited resources on innovations that are most likely to improve its long- term viability. USPS also stated that flexibility is important in innovation pilots, particularly as it pertains to linking pilot objectives with performance measures. We continue to believe that linking objectives with performance measures is key to effectively evaluating pilots. In so doing, however, there is flexibility to adjust pilot objectives and performance measures as new information is gleaned during the pilot. Finally, with regard to communication with external stakeholders during pilots, USPS said that it communicates consistently with external stakeholders regarding pilots at Mailers’ Technical Advisory Committee meetings (MTAC). In our report, we noted that USPS employed strategies to communicate with some external stakeholders—i.e., industry associations and mailers. We continue to believe, however, in the importance of communication with all key external stakeholders, which may include stakeholders, such as consumers, that do not participate in MTAC meetings.

USPS and PRC also provided technical comments, which we incorporated as appropriate.

We are sending copies of this report to the appropriate congressional committee, the Postmaster General, Chairman of PRC, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-2834 or RectanusL@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III.

Appendix I: List of U.S. Postal Service Key Piloted Innovations, Fiscal Years 2013-2017

The U.S. Postal Service (USPS) piloted 24 key innovations from fiscal years 2013 through 2017 (see table 3).

Appendix II: Comments from the U.S. Postal Service

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the individual named above, Derrick Collins (Assistant Director); William Colwell and James Leonard (analysts in charge); Barbara El Osta; Geoffrey Hamilton; Gina Hoover; Anthony Jackson; and Laurel Voloder made key contributions to this report.