MEP Advisory Board Performance & 

Research Development Working Group: 

Performance Framework 

Final Report

March 29, 2019

100 BUREAU DRIVE M/S 4800 GAITHERSBURG, MD 20899

www.nist.gov/mep | PHONE: (301) 975-5020
2

Introduction and Background

The NIST MEP Advisory Board created a Performance and Research Development Working 

Group1in January 2018. The Working Group was charged with focusing on the issue of 

performance measurement and management, evaluation, and research to support the MEP 

National NetworkTM. The Working Group focused on providing input and guidance on the 

performance evaluation and Program performance measurement processes of the MEP National 

NetworkTM. This report is intended to lead to improved Center evaluation processes and 

approaches, the promotion of National Network learning, and improving the portfolio of NIST 

MEP data and analysis services for Centers. Through this effort, the Working Group is providing 

feedback and suggestions on supporting and enhancing NIST MEP’s performance and evaluation 

systems including research to leverage and enrich NIST MEP’s performance and evaluation 

management system. 

The Working Group met several times over the course of its work. This document 

summarizes the findings, observations, and recommendations of the Working Group to the MEP 

Advisory Board. These findings are based on the Working Group’s review of the information and 

data NIST MEP collects, its performance measurement and evaluation portfolio, and the new 

Panel Review process. The working group also examined Network performance results, Center 

and Network performance trends, other external reviews of the MEP performance measurement, 

 

1The Working Group was led by Leslie Taito and its members included: Jose Anaya, Carolyn 

Cason, Joe Eddy, Kathay Rennels, and Jim Wright. Bernadine Hawes and Jeff Wilcox were ExOfficio members of the working group. The Working Group was supported by Kenneth Voytek 

and Chancy Lyford. 
3

management approach2, and potential new approaches to performance improvement. 

Observations

The Working Group is impressed with the strong and robust foundation the NIST MEP 

Program has established to collect and use information to assess Network and Center 

performance. The effort and resources NIST MEP have devoted to performance measurement 

and evaluation is impressive. This has been a consistent and important focus for the program 

since it was created more than 30 years ago. A listing of some but not all studies is included 

Appendix A to this memo. The focus on performance has guided the program since its inception 

and demonstrates it has made a positive impact on the Program and Centers. Much of the 

information and studies conducted by NIST MEP over the years has served as the foundation to 

show the Program makes a meaningful difference not only for the clients it serves but has 

broader economic benefits that extend beyond positive client outcomes. 

The Working Group finds one of the distinctive strengths of the NIST MEP Program 

through the years has been its consistent focus on measuring Program performance and impacts, 

demonstrating accountability to stakeholders and leading to improved Program and Center 

performance. The Program relies on a range of tools and approaches measuring Program 

 

2 These reports include NIST MEP (1999) A Report to Congress - The NIST Manufacturing 

Extension Partnership: A Network for Success, National Academy of Public Administration 

(2004) The Manufacturing Extension Partnership Program: Alternative Business Models, Stone 

& Associates and the Center for Regional Economic Competitiveness (2010), Re-examining the 

Manufacturing Extension Partnership Business Model, and the National Academies of Science 

(2014) 21st Century Manufacturing: The Role of the Manufacturing Extension Partnership 

Program.
4

performance including administrative data on clients served, what services are provided, and the 

results of these services including the following: 

➢ The NIST MEP client impact survey collects information on the impact of the program 

regarding sales, jobs, investments, and cost savings, which are keystones of the Program. 

This information is valuable to understand how the services delivered improve client 

performance and what services and methods should be used more extensively by the 

Program.

➢ The Working Group observed the current IMPACT metrics report consists of important 

quantitative measures of performance and a provides a consistent approach to examining 

Center performance. The IMPACT metrics cover many of the key aspects of performance 

including impact and market penetration and provide for Center to Center comparisons as 

well as trend information. The Working Group suggests that additional tools and 

information would provide MEP Centers with additional opportunities for benchmarking 

and performance improvement. The Working Group explored how performance across 

the MEP National Network had improved over time after the new IMPACT metrics were 

implemented.

➢ The new Panel Review process and report that focuses on performance trends of a 

Center. The Working Group found that the revised Panel Review process, put in the 

statute in the American Competitiveness and Innovation Act of 2017, is an important 

change as it places the primary focus of the Review on impacts and results and those 

factors that may be associated with improved results and impacts. This evolution of the 

Panel Review process and its focus on performance and results should continue. and 

elements such as Annual Center Reviews should be aligned to focus on the same issues. 

➢ The Program has also periodically undertaken formal impact evaluations that involve 

comparing the performance of MEP clients to non-clients and has used the results to 

inform program strategies and objectives. Continued comparative analysis would be 

informative and doing such studies should continue. 

The mix of methods used by the program to assess performance has been necessary and 

valuable to the Program and Centers based on the Working Group’s review. By focusing on 

economic impact, the Program and the National Network are focused on ensuring that its 

activities are improving the competitiveness and productivity of U.S. small- and medium-sized 
5

manufacturing firms and are also having broader economic impacts.3 However, the Working 

Group observed that additional tools and information would provide MEP Centers with 

additional opportunities for benchmarking and performance improvement. Areas of focus for the 

Working Group’s analysis included:

❖ Understanding the Influence of Metrics on Center Performance: The Working Group 

explored how performance across the MEP National NetworkTM had improved over time 

when the new IMPACT metrics were implemented. The IMPACT metrics report and the 

revised Panel Review process reflects the in-depth and on-going revisions of the 

reporting and performance measurement system. 

❖ Identifying the Factors Most Important to Predicting Center Success: The focus of the 

Panel Reviews on Center Performance and trends in performance over time and the 

factors that correlate with or high or low performance is encouraging. The Working 

Group encourages NIST MEP Panel Review’s to evolve and sharpen its focus on the 

factors most relevant and important to performance. 

The Working Group defines performance management as including the activities, decisions 

and processes, and strategies that affect performance and incorporating this perspective into the 

Panel Review process is important. This may include more systematic research addressed below. 

The performance measurement and management approach of NIST MEP serves as a model for 

technology based economic development programs and is firmly established within the MEP 

National NetworkTM and it is widely accepted and expected by NIST MEP stakeholders. This is 

evidenced by the studies cited earlier that reviewed the performance and evaluation approach 

used by NIST MEP and other external reviews including studies appearing in peer reviewed 

journals. 

 

3 See, for instance, Robey, J. et al. (2018). The National-Level Economic Impact of the 

Manufacturing Extension Partnership (MEP): Estimates for Fiscal Year 2017. W.E. Upjohn 

Institute for Employment for Employment Research. 
6

Continuous Improvement in Performance Management

Understanding Center performance and the factors distinguishing high- and low-performing 

Centers is fundamental to Center and Program management and will require discovering how 

organizational practices, policies, and other factors come together to determine performance. It is 

not enough to diagnose or examine organizational performance, but the Program also needs to 

examine why an organization performs as it does. If the Program can come to understand these 

underpinnings of performance, NIST MEP can begin to develop prescriptions for improving 

performance. The Working Group believes this could be addressed by examining the effect of 

organizational structure, strategies, resources, service models, external environment factors, and 

internal factors including management practices and behaviors on Center performance.

Leveraging information to enhance learning and linking the information and data to users in 

innovative ways will be critical to the long-term health and vitality of the Program. NIST MEP

should also continually seek new means of facilitating MEP National NetworkTM and Center 

learning to improve performance. This will involve more than just collecting information-- the

information collected by NIST MEP needs to be leveraged by Centers and linked to other 

initiatives. The Working Group believes that this information represents a competitive advantage 

the Program can leverage in new and different ways. The working encourages the Program to 

focus some of its resources to collect, analyze, and report data and information relevant to NIST 

MEP management, Centers, stakeholders and other partners. This will continue to grow as the 

Program evolves in response to changes in the manufacturing landscape and resulting alterations 

in client needs.

Over time, this may involve new approaches and methods for evaluating and measuring the 

impact of the MEP National NetworkTM and Center performance. The Working Group 
7

recommends that the Program also continue to evolve its performance measurement approach in 

the following ways: 

• The Working Group strongly encourages NIST MEP to continue to work in concert 

with the Centers and other stakeholders to refine its performance measurement 

approach. 

• The Working Group recommends that NIST MEP implement and introduce revisions 

in a thoughtful and measured manner as appropriate. Measures matter. 

• Finally, the Program should ensure that the performance measures in place do not 

have unintended consequences and that the burden on the Centers and clients is 

minimized to the extent practical. 

Embracing performance management best-practices requires that both NIST MEP and Center 

management use performance data and information to help them make decisions with the aim of 

continually improving services to manufacturers and Center performance. This may mean 

training and assisting Centers to be more sophisticated consumers of data and to use and 

incorporate data and analysis into their decision-making process. Also, the Working Group 

believes a renewed focus by NIST MEP on data as a service may be a starting place. This may 

include building out the range of data services and reports that Centers can access and use. 

Recommendations for Research and Development

The Working Group identified a set of research topics and ideas for the MEP Advisory 

Board to consider going forward. The Working Group grouped these topics into three areas. 

The first is focused on supporting NIST MEP and the National Network to improve 

performance. This research may help guide the transformation of the National Network and 

leverage partnerships. The second set of topics is focused on the broader manufacturing 

landscape to enhance the National Network as a champion for manufacturing and a thought 

leader as well as empowering manufacturers. The third set of topics is focused on ensuring that 
8

NIST MEP supports and builds out the infrastructure to support research and evaluation efforts. 

We recognize that each of these is a significant undertaking, but we view these as setting out a 

longer-term research program strategy for the program. 

The Performance and Research Development Working Group provides the following 

recommendations to the MEP Advisory Board.

Research on NIST MEP

• Undertake research on the factors related to differences in center performance. The 

Panel Review is currently based on a set of hypotheses about factors that relate to high 

or low Center performance, but we have not determined which factors do account for 

high or low Center performance or how they interrelate. This research would give NIST 

MEP, panelists, and Center management a tool to benchmark best in class along with 

identifying the key drivers of success (i.e., staffing talent/experience, financial impact of 

Federal and state funding, funding by stakeholders and/or interested parties and partners)

that can be utilized in future planning and training. 

• A more nuanced view of Center performance including not just a focus on minimum 

but to distinguish levels of performance. This research will provide a more detailed 

analysis and a picture of Center performance across the National Network and permit 

centers and managers to distinguish themselves among Centers in terms of high 

performance. This will also provide Centers and stakeholders with the ability to evaluate 

and analyze performance based on criteria that is not merely pass/fail but generates 

guidance and information to determining strategic areas for improvement, development 

and growth. 

• Expand the use of case studies and other approaches to examine pilot programs and 

new services. NIST MEP is rolling out new and innovative experimental services, and 

when the special funding ceases, the new service or program does not become 

sustainable and ceases as well. Having the opportunity to systematically analyze how 

these pilots and new services were rolled out to discover what lessons learned that can be

applied to other initiatives is important. This may help NIST MEP and program 

managers measure whether the pilot programs can be replicated, how the programs 

evolved, and the factors that enabled sustainability, and ultimately success. 

• Research on the factors that relate to the level and likelihood of MEP clients reporting 

impacts. Centers and staff have often asked what types of projects lead to better 

outcomes and what types of outcomes can be expected from different services. This will 

provide a deeper level of understanding to NIST MEP and allow Center management to 
9

have predictive elements when reviewing current services and creating new program 

offerings. This will provide Centers with better intelligence on those services that make 

the most difference to clients and what they can expect. We also encourage the Board to 

suggest NIST MEP continue to focus on long-term longitudinal formal impact 

evaluations to provide additional information on program outcomes. 

Research on the Manufacturing Landscape

• Research on underserved segments of the manufacturing marketplace. The MEP 

National Network needs to work with more manufacturers, including those underserved 

in terms of location (such as economically-disadvantaged areas), size (micro 

manufacturers), start-ups, and veteran, minority, and woman-owned businesses. This 

research should include understanding types of services needed by underserved 

manufacturer populations (i.e., strategy, markets, technology, business continuity) and 

how the National Network can fashion programs and services to better reach out to and 

serve these segments. This would expand market penetration. 

• Mapping and defining the Manufacturing Ecosystem/Industrial Commons.

Identification of MEP Center ecosystems will highlight successful models (Universitybased, stand-alone) and the integration and linkages to economic development 

organizations and programs of each helping to drive that success. Focus should also be on 

the relationships with advanced manufacturing groups such as universities and federal 

labs, Manufacturing USA Institutes and industry-specific trade associations to deliver 

high value services and nurture a system that develops a strong ecosystem to support 

manufacturing. The ecosystem includes both the hard and soft infrastructure that supports 

manufacturing and is a way to coalesce a region’s institutions and actors around their 

shared goals such as strengthening manufacturing by increase research and development, 

skills and workforce development, and building out and strengthening connections 

among suppliers and other support services.

• More research into the broader economic and business environment faced by 

manufacturers. This research can include looking at trends and changes in 

manufacturing in the US and across states and regions. Understanding the manufacturing 

environment may also include focusing on other emerging trends regarding new business 

models, technologies, and what they mean for SMMs and MEP Centers. This could be 

supplemented with information gathered in MEP surveys and potentially creating a panel 

of manufacturers to use for special surveys, etc. This may also help to identify emerging 

issues that may lead to targeted approaches for system services along with options for 

new programs and service delivery models. 

• Supply Chain Assistance. Supply chain assistance has been an episodic focus of MEP. 

NIST MEP and the National Network should convene research and other analysis to 

better understand how to identify supply chains beyond Tier 1 and 2 in different industry 
10

verticals to better make the case to stakeholders and OEMs that SMMs are critical links 

in supply chains for U.S. based OEMs. Better optics, definition, and identification of 

particular supply chains would enable national and global supply chains participants to 

have the ability to identify and participate in the long-term sustainability of supply chain 

services and partners. This would improve the targeting and delivery of services to clients 

to optimize and improve supply chain performance. This could be another way to expand 

MEP market penetration leading to more clients and impacts. In addition, NIST MEP 

might consider what is needed to develop a business model that could create a sustainable 

supply chain practice across the National Network and how a supply chain practice could 

be deployed and sustained. 

Supporting Research and Evaluation 

• Broaden the bench of researchers focusing on manufacturing (particularly SMMs) 

and MEP Program. MEP should broaden and encourage a new generation of researchers 

and scholars to focus on issues relevant to MEP. This will aid NIST MEP and National 

Network a true champion for manufacturing. This could include collaborating with 

graduate students and post docs for example. NIST MEP should seek to make its data 

and access to the National Network an incentive to researchers and scholars for their 

research projects or projects mutually defined. NIST MEP should also investigate 

leveraging think tanks and industry associations to broaden the messaging, scope, and 

research into manufacturing and MEP. Also, the Working groups encourages NIST MEP 

to make the resources and investments available to support a robust research and 

evaluation agenda. 

Conclusion

The Working Group applauds NIST MEP on its commitment to evaluation and research over 

the Program’s existence. The performance measurement and evaluation system, including the 

Panel Reviews and its focus on performance, represents a model for government and business 

assistance programs. The NIST MEP and the National Network has leveraged the data and its 

approach to demonstrate and document that it delivers measurable results. A commitment to a 

robust research agenda both on NIST MEP, manufacturing extension, and manufacturing 

generally is a useful exercise to think through and deploy. While we have made 

recommendations for the Program to consider, we also recognize that the Program should be 
11

agile and careful in setting out its research and evaluation agenda. The Program should plan its 

agenda to ensure it can respond to new and unforeseen issues that may emerge. 
12

Appendix A: Selected Studies on MEP

Brandt, P., Shrank, A, & Whitford, J. (2018). Brokerage and Boots on the Ground: Complements 

or Substitutes in the Manufacturing Extension Partnerships?, Economic Development Quarterly. 

32(4), pp 288-299.

Bartik, Timothy J. (2018). What Works to Help Manufacturing-Intensive Local Economies? 

Upjohn Institute Technical Report 18-035. 

Robey, J. et al. (2018). The National-Level Economic Impact of the Manufacturing Extension 

Partnership (MEP): Estimates for Fiscal Year 2017. W.E. Upjohn Institute for Employment for 

Employment Research. 

Lipscomb, C.A. et al. (2018). Evaluating the Impact of Manufacturing Extension Services on 

Establishment Performance. Economic Development Quarterly, 32(1), pp. 29-43.

Oldsman, E. (2017). Upskilling SMEs: How governments fund training and consulting. 

Comparing experiences from Asia, Europe and North America. International Labor Organization. 

Whitford, J. & Brandt, P. (2017). Fixing network failures? The contested case of the American 

Manufacturing Extension Partnership. Socio Economic Review, 15(2), pp. 331-357. 

Robey, J. et al. (2017). The National-Level Economic Impact of the Manufacturing Extension 

Partnership (MEP) W.E. Upjohn Institute for Employment for Employment Research. 

Sargent, John F. (2016). The Manufacturing Extension Partnership Program. Washington, D.C.: 

Congressional Research Service. 

Lipscomb, C.A. et al. (2015). Evaluating the Long-Term Effect of NIST MEP Services on 

Establishment Performance. Washington, D.C Census Bureau CES 15-09. 

ftp://ftp2.census.gov/ces/wp/2015/CES-WP-15-09.pdf 

Government Accountability Office. (2014). Manufacturing Extension Partnership: Most Federal 

Spending Directly Supports Work with Manufacturers, but Distribution Could Be Improved. 

Washington, DC: GAO.

Shapira, P.S. & Wessner, C.W. (Eds.). (2013) 21

st

Century Manufacturing: The role of the 

Manufacturing Extension Partnership Program. Washington, D.C.: National Academies Press. 

Government Accountability Office. (2013). Global Manufacturing: Foreign Government 

Programs Differ in Some Key Respects From Those in the United States. Washington, DC: 

GAO.
13

Frazier, R.M. (2012). The Imperatives of Successful Policy Implementation: A Case Study of the 

Hollings MEP Program’s Implementation in Arkansas. International Journal of Learning & 

Development. 2(4). 

Ordowich, C. et al. (2012). Evaluating the impact of MEP services on establishment 

performance: A preliminary empirical investigation. Washington, D.C. Census Bureau. CES12-

15. http://www2.census.gov/ces/wp/2012/CES-WP-12-15.pdf

Government Accountability Office. (2011). Factors for Evaluating the Cost Share of 

Manufacturing Extension Partnership Program to Assist Small and Medium-Sized 

Manufacturers. Washington, DC: GAO.

Ezell, S.J. & Atkinson, R.D. (2011). International Benchmarking of Countries’ Policies and 

Programs Supporting SME Manufacturers. Washington, D.C.: The International Technology and 

Innovation Foundation. 

Ezell, S.J. & Atkinson, R.D. (2011). The Case for a National Manufacturing Strategy. 

Washington, D.C.: The International Technology and Innovation Foundation. 

Stone & Associates & Center for Regional Economic Competitiveness. (2010). Re-examining the 

Manufacturing Extension Partnership Business Model: Alternatives for Increasing the 

Program’s Impact on U.S. Manufacturing Sector Performance.

Helper, S. & Wial, H. (2010). Strengthening American Manufacturing: A New Federal Approach.

Washington, D.C: Brookings Institution. 

National Academy of Public Administration. (2004). The Manufacturing Extension Partnership 

Program: Alternative Business Models. Washington, DC: NAPA. 

Davila, N. (2004). Evaluating Manufacturing Extension: A Multidimensional Approach. 

Economic Development Quarterly, 18(3), 286-302.

Voytek, K.P., Lellock, K.L., & Schmit, M.A. (2004). Developing performance metrics for 

science and technology programs: The case of the manufacturing extension partnership program. 

Economic Development Quarterly, 18(2), 174-185.

National Academy of Public Administration. (2003). Re-examining the Core Premise of the 

MEP Program. Washington, DC: NAPA.

National Institute of Standards and Technology, Manufacturing Extension Partnership. (1999). A 

Report to Congress - The NIST Manufacturing Extension Partnership: A Network for Success. 

Gaithersburg, MD: Author. 

Jarmin, R.S. (1999). Evaluating the impact of manufacturing extension on productivity growth. 

Journal of Policy Analysis and Management, 18(1), 99-119.
14

Feller, I. & Nelson. J.P. (1999). “The Microeconomics of Manufacturing Modernization 

Programs.” Research Policy 28(4): 807–818.

Glasmeier, A. K., Feller, I., Fuellhart, K.G, & Mark, M.M. (1998). “The Relevance of Firm 

Learning to the Design of Manufacturing Modernization Programs.” Economic Development 

Quarterly 12 (2), 107–127.

Feller, I. (1997). “Manufacturing Technology Centers as Components of Regional Technology 

Infrastructures.” Regional Science and Urban Economics 27(2), 181–197.

Feller, I., Glasmeier, A., & Mark, M. (1996). Issues and perspectives on evaluating 

manufacturing modernization programs. Research Policy, (25(3), 309-319.

National Academy of Sciences. (1993). Learning to Change: Opportunities to Improve the 

Performance of Small Manufacturers. Washington, D.C.: National Academies Press.