Minutes 

Meeting of the Judges Panel of the 

Malcolm Baldrige National Quality Award 

National Institute of Standards and Technology 

Wednesday, June 15, 2022 • 9:00 AM 

Attendees 

Judges: Cary Hill, Keith Everett, Kevin Johnson, Lynda Johnson, Amy Katschman, Sophia McIntyre, 

Jennifer Niswonger, Amy Pugh, Bruce Requa, Meridith Wentz, Gary Wilson, Allyson Young 

NIST: Jamie Ambrosi, Dawn Bailey, Rebecca Bayless, Jacqueline Deschamps, Robert Fangmeyer, Barbara 

Fischer, Robert Hunt, Elif Karakas, Darren Lowe, Michelle Pena, Robyn Verner, Kelly Welsh 

Meeting start: 9:00 am 

WELCOME AND MEETING OVERVIEW {Robert Fangmeyer, Cary Hill} 

Robert Fangmeyer and Cary Hill welcomed everyone to the meeting, reviewed the intentions for today's 

meeting, and encouraged all attendees to jump into the conversation and share their questions, 

intellect, knowledge, and creativity. 

APPROVAL OF THE MINUTES OF THE NOVEMBER 2021 JUDGES PANEL MEETING (Cary Hill) 

Unanimous approval. 

2022 PROCESS CHANGES (Robert Hunt} 

External Program Review and Award Process Pause 

Fangmeyer explained that the Department of Commerce (DOC) has been very interested in the Baldrige 

Program and expressed an interest in reinvigorating the program and the award. Therefore, DOC has 

called for an independent, external review for recommendations to strengthen the reach and impact of 

the program. 

Fangmeyer noted that external reviewers would engage with the Baldrige community and would need 

to get the work done quickly if we are to have an award process in 2023. 

Fangmeyer said Baldrige, NIST, and DOC will review the recommendations, and determinations will be 

made about reasonable changes and how to source and implement them. 

Some Judges expressed concern about the timeline; others said it is an exciting time to re-energize the 

program, "a positive transformation opportunity." 

Overseers Award Process Focus Group 

Fangmeyer reminded the Judges of the purpose of the Overseers' Award Process Focus Group: to 

identify the barriers that prevent more organizations from participating in the Baldrige Award process 

and how we can remedy those. Pattie Skriba, former Chair of the Judges Panel, was invited to join the 

focus group as an expert facilitator. 

Fangmeyer noted that among the barriers are some that are at the core of Baldrige, including our very 

broad, and somewhat vague definition of "excellence" and what's truly necessary for an organization to 

be considered a role model. Additionally, the focus group suggested that we start by looking at an 
organization's results and then assess the processes used to achieve those results. This would be a true 

paradigm shift in the evaluation process. 

Fangmeyer noted that these and other factors will be addressed in more detail during the joint meeting 

of the Judges and Overseers the following day. 

Break: 10:40 am 

Return: 10:56 am 

2022 PROCESS CHANGES (Robert Hunt) 

Robert Hunt provided an overview of the 2022 evaluation process and reviewed the typical judging 

process to inform the Judges' thinking when hearing about potential process improvements. 

2022 Process Changes 

Based on years of feedback and an overseers' charge to make improvements, the following changes 

were made: 

• A Virtual Evaluation phase replaces Independent and Consensus Reviews to accommodate 

virtual applicant interviews. 

• Judges will receive team scores from Virtual Evaluation only. Examiners will no longer score 

during an Independent Review stage; they need information from virtual interviews first. Then 

they score as a team. 

• Hybrid site visits and efficiency improvements: Examiner team members conduct virtual 

interviews to clarify what's in the scorebook they received during the Virtual Evaluation. What 

they heard on virtual interviews can be verified on-site. It is anticipated that site visits would 

take less time than the current process. 

• It is anticipated that these changes will reduce cycle time and provide more accurate feedback 

reports. 

• Because feedback reports will be simpler to produce, understand, and finalize, this will save staff 

time, as well. 

To support those changes, Hunt described the training for master examiners, team leaders, returning 

examiners, and new examiners, which included coaching, a case study, and online elearning modules. 

2022 Award Applications 

Applications received in 2022: Health care 11, Service 1, Nonprofit 5 = 17. 

With the award pause, the 2022 applicants were invited to receive a "feedback-only" review. Ten 

organizations asked to stay in the process. 

Application history by sector was explained, including changes driven by the loss of federal funding in 

2012, particularly award eligibility changes to require applicants to receive the top-level recognition 

through an Alliance for Performance Excellence member program before applying for the Baldrige 

Award, and a doubling of fees to help recover costs. In 2020, there were fewer applicants due to COVID19 impacting all sectors; in 2021, application numbers have started to rebound. 

Hunt shared the history of examiner applications, which have been trending down since the early part of 

the 2000s, in part due to fewer award applications and changes associated with the loss of funding (e.g., 

no longer reimbursing examiners for travel costs associated with attending training in Gaithersburg). 

Hunt noted that the typical developmental path for examiners is to become senior examiners and, 

ultimately, master examiners. 
H_unt also noted that due to the limited number of examiner coaches, only 37 new examiners were 

selected in 2022. Some new examiners are fast tracked; for example, physicians, nurses, C-level folks to 

get them into the system. Criteria for selecting returning examiners include performance, availability, 

years of service, board balance, history of dropping, leadership and mentoring ability, Criteria and 

process knowledge, and multiple applicants from same organization. 

For 2022, 240 examiners were selected--about 64% of applications received. The low selection rate was 

due to the relatively low number of award applicants (17). 

To recruit examiners, the program reaches out to the Alliance members, uses social media, and reaches 

out to associations and other sector organizations. 

Jamie Ambrosi noted that there is a dual challenge of retention and recruitment. We need to get 

examiners networked and connected, as well as consider their workload and work environment-baby 

boomers to millennials and gen-z, who expect a mobile-friendly, virtual system, which we do not yet 

possess. 

Several Judges noted the need to increase funding to support such initiatives and defray the costs 

associated with participating in the award process, as well as concerns around an extended pause in the 

award process. 

JUDGES' ORIENTATION (Robert Hunt, Cary Hill) 

The Judging Process 

Key Roles & Responsibilities 

• Work as a team 

• Select applicants to advance to Site Visit Review; recommend award recipients (but not in 2022) 

• Recommend process changes to the Board of Overseers 

• Provide input into the development of the Criteria 

• Serve as ambassadors 

Hunt reviewed what would typically happen at the August and November meetings. 

In August, Judges look at detailed scoring profiles (without knowing organization names) to determine 

which applicants should move on to site visit. After site visit determinations are made, Baldrige staff 

members identify known conflicts and ask the Judges to self-identify any additional actual or potential 

perceived conflicts with every organization chosen for site visit. 

For the November meeting, where the Judges recommend award recipients, Hunt reviewed the 

timelines, key process steps, outcomes, and the roles of individual, lead, and backup Judges. Following 

the review, experienced Judges shared their lessons learned and best practices related to preparation 

for the November meeting, including the following: 

• Judging is not the same as examining. Your goal is to ensure you understand what the examiner 

team found 

• Participation of NIST staff is crucial to Judges' success 

• Day one of November meeting starts slow, as Judges find their rhythm 

• New Judges will not be first to present at the November meeting 
• As new Judges may have questions, other Judges are very willing to help and support. Feel free 

to reach out. Everyone is there to help each other. 

• Having an assigned mentor was very helpful 

• Lead and backup roles will be one experienced, one new. 

Break: 11:56 am 

Return: 12:47 pm 

Regarding the pending external review, Cary Hill asked the Judges, "If Baldrige could be anything out of 

its 'reimagining,' what would you like it to be?" 

• Baldrige becomes the number-one model for organizations to improve their businesses to 

become top performers, in all sizes, types, configurations 

• Accessible to all, especially smaller organizations and minority-owned businesses that have 

harder challenges with sustaining success 

• About defining what excellence really means. Every good system always needs a gut check. 

Clearly define what excellence looks like. 

• How do you systematize the reinvention of Baldrige on a periodic basis? 

• Funded to its full capability to achieve its goal 

• Known and used as a path for performance excellence, not necessarily an award 

• More well-known and used by more organizations, used in universities as part of business 

school curriculums 

• Taught in every business school and taught to engage students at a younger age 

• Love to see it impact education in a positive way, spread to more education organizations, 

especially K-12, impact our future leaders 

• Framework to help organization understand how quality tools fit; it's the framework for 

performance excellence 

• Like to think we can maintain Baldrige Program and Criteria as a best practice tool to help 

organizations to improve and recognize that improvement. Would not like to see the program 

go the direction of simply an organization that gives an award. Keep focus on helping 

organizations improve 

JUDGES' SURVEY OF 2021 APPLICANTS (Dawn Bailey) 

Background 

• A Judges Survey is sent to each award applicant about three weeks after feedback reports are 

sent out; responses can be anonymous 

• Survey asks about likelihood to recommend and satisfaction with Criteria, Baldrige Award 

process, feedback reports 

• Net Promoter Score approach used. In an effort to address the issues such as low response rates 

and not actionable feedback, we went from a traditional satisfaction survey to the current Net 

Promoter Score approach in 2010. 

• 2021: Second year of all-virtual site visits and delayed award announcement are believed to 

have impacted evaluations 

In 2021, 14 applications were received and evaluated during the first two phases of the Baldrige Award 

process. Seven organizations were selected to receive site visits. 
A little more than half-eight-responded to the survey. Two email reminders were sent; however, the 

award announcement was not until March 4 (about three months after feedback reports received), 

which we believe impacted how many organizations responded to the survey. At least one organization 

said that it would wait for the announcement to respond; we did not send reminders again after March 

4. 

2021 Data Summary 

• Likely to Recommend the Criteria: 8 scores at 10 = NPS 100 

• Criteria Relevance to my Organization: 8 scores at 10 = NPS 100 

• Likelihood to Reapply, Excludes Baldrige Award Recipients: 5 scores at 10, 1 at 9, 2 skipped so 

neutral= NPS 60 

• Participation in Baldrige Award process: 5 scores at 10, 2 at 9, 1 at 1 = 70-10= NPS 60 

Intent to Reapply 

In 2021, there were some negative comments regarding the delay in the award announcement and that 

this made reapplying a difficult decision of senior leadership. 

Key Findings 

• Delays in award announcement impacted satisfaction 

• All applicants highly satisfied with framework, but some concerns remain with feedback reports 

• Survey feedback can inform 2023-2024 framework revisions (e.g., help applicants/examiners 

better understand expectations for innovation, diversity) 

• Much of this feedback informs and is addressed by the 2022 award process redesign 

The Judges noted that feedback report satisfaction trended lower and was mostly neutral. This could be 

an anomaly because award announcement was delayed and COVID caused virtual site visits. It's a little 

disturbing that everything has moved to neutral. 

There may not have been enough root cause analysis to determine why scores are trending down. 

One Judge suggested that we have to do a better job of letting organizations define diversity-equityinclusion in their own ways; we need to acknowledge them for what they're doing. 

There was a suggestion that feedback report satisfaction be shared with team leaders and an 

expectation of applicants to provide feedback before scoring. Certainly, their scoring in the award 

process will impact their level of satisfaction. 

PREPARATION FOR BOARD OF OVERSEERS MEETING {Robert Fangmeyer, Cary Hill) 

Fangmeyer noted that this joint meeting is an opportunity for the Judges to share with the overseers 

their satisfaction with the evaluation and judging process. 

The Judges reaffirmed their 2021 process. Judges select site-visited applicants based on blinded data put 

forth by the examiner team. They do not know which is a repeat applicant. Often the Judges have gone 

back and reviewed their process in real-time to ensure there is no bias. It was noted that there have 

been years when a prior recipient applied and has not gotten the award. The Judges are confident they 

follow the same process with every applicant and are objective in their determinations. The judging 

process is reviewed at the end of the deliberation for each applicant. Judges feel there has not been any 

bias relative to prior recipients. It was noted that with a cap of 18 awards, nothing prevents Judges from 
giving all site-visited applicants an award (as long as there are fewer than 18 site visits, which has always 

been the case). 

DISCUSSION (Cary Hill) 

Concerning the potential changes driven by the external review, it was noted that it is crucial that the 

judging process not lose its objectivity. It was also noted that we may need to clarify what criteria make 

an organization a national role model, which should inform the evaluation and judging processes. 

Another judge recommended that a shorter application that can be done online would be easier to 

produce and attract more applicants. 

Hill reiterated that this is an exciting time for Baldrige, and he's grateful for the gifts and expertise that 

everyone is sharing. Fangmeyer thanked everyone and confirmed the time and location for the joint 

meeting the next day. 

Adjourn: 3:04 p.m. 

Remaining 2022 Meeting Dates 

June 16 (with the Board of Overseers; in-person) 

December 6 (chair only; virtual) 

Ihere?;~my knowledge, the foregoing minutes are accurate and complete. 

Cary Hill 

Chair Candidate, Judges Panel