Federal Register / Vol. 87, No. 46 / Wednesday, March 9, 2022 / Notices 13295 

and supplements thereto on the grounds 

that new information, evaluated 

together with the evidence available 

when the application was approved, 

showed there is a lack of substantial 

evidence that the drug is effective under 

the conditions of use prescribed, 

recommended, or suggested in the 

labeling. The Agency again invited 

Glenwood, and any other interested 

person(s) who would be adversely 

affected by the withdrawal of approval 

of NDA 007663, to submit: (1) On or 

before September 19, 1977, a written 

notice of appearance and request for 

hearing and (2) on or before October 17, 

1977, the data, information, and 

analyses relied upon to justify a hearing. 

On September 12, 1977, Glenwood 

filed a written notice of appearance and 

requested a hearing, and on October 17, 

1977, Glenwood submitted data in 

support of its hearing request. Along 

with these submissions, Glenwood 

requested that the Agency delay action 

on the hearing request until the firm had 

conducted another placebo-controlled 

study. Subsequently, Glenwood 

initiated a clinical trial at the Downstate 

Medical Center of the State University 

of New York and supplemented its 

hearing request with additional data, 

including a progress report on the 

clinical trial of POTABA conducted at 

the Downstate Medical Center. 

Following a meeting between 

Glenwood and FDA on November 18, 

1985, Glenwood sponsored another 

controlled clinical trial, and the final 

study report was submitted on February 

4, 1993. 

By letter dated October 21, 2010, FDA 

asked Glenwood whether it wanted to 

pursue its pending hearing request 

regarding POTABA. By letter dated 

November 11, 2010, Glenwood affirmed 

its hearing request. 

By letter dated June 8, 2020, FDA 

again asked Glenwood whether it 

wanted to pursue its pending hearing 

request regarding POTABA. By letter 

dated July 2, 2020, Cheplapharm 

Arzneimittel GmbH, successor-ininterest to Glenwood LLC, stated that it 

did not wish to pursue the hearing 

request for POTABA. 

III. Conclusions and Order 

There are no outstanding hearing 

requests regarding potassium 

aminobenzoate oral preparations under 

Docket No. FDA–1977–N–0015, DESI 

7663. Therefore, as proposed in the 

NOOH, FDA withdraws approval of 

NDA 007663 under section 505(e) of the 

FD&C Act. 

Shipment in interstate commerce of 

any drug product identified in this 

docket under DESI 7663, or any IRS 

product, that is not the subject of an 

approved NDA or abbreviated new drug 

application is unlawful as of the 

effective date of this notice (see DATES). 

Any person who wishes to determine 

whether this notice covers a specific 

product should write to Astrid LopezGoldberg at the Center for Drug 

Evaluation and Research (see FOR

FURTHER INFORMATION CONTACT). Firms 

should be aware that, after the 

applicable date of this notice (see 

DATES), FDA intends to take 

enforcement action without further 

notice against any firm that 

manufactures or ships in interstate 

commerce any unapproved product 

covered by this notice. 

IV. Discontinued Products 

Firms must notify the Agency of 

certain product discontinuations in 

writing under section 506C(a) of the 

FD&C Act (21 U.S.C. 356c). See http:// 

www.fda.gov/Drugs/DrugSafety/ 

DrugShortages/ucm142398.htm. Some 

firms may have previously discontinued 

manufacturing or distributing products 

covered by this notice without 

discontinuing the listing as required 

under section 510(j) of the FD&C Act (21 

U.S.C. 360(j)). Other firms may 

discontinue manufacturing or 

distributing listed products in response 

to this notice. All firms are required to 

electronically update the listing of their 

products under 510(j) of the FD&C Act 

to reflect discontinuation of unapproved 

products covered by this notice (21 CFR 

207.57(b)). Questions on electronic drug 

listing updates should be sent to 

eDRLS@fda.hhs.gov. In addition to the 

required update, firms can also notify 

the Agency of product discontinuation 

by sending a letter, signed by the firm’s 

chief executive officer and fully 

identifying the discontinued product(s), 

including the product National Drug 

Code number(s), and stating that the 

manufacturing and/or distribution of the 

product(s) have been discontinued. The 

letter should be sent electronically to 

Astrid Lopez-Goldberg (see FOR FURTHER

INFORMATION CONTACT). FDA plans to 

rely on its existing records, including its 

drug listing records, the results of any 

future inspections, or other available 

information, when it identifies violative 

products for enforcement action. 

Dated: March 3, 2022. 

Lauren K. Roth, 

Associate Commissioner for Policy. 

[FR Doc. 2022–04971 Filed 3–8–22; 8:45 am] 

BILLING CODE 4164–01–P 

DEPARTMENT OF HEALTH AND 

HUMAN SERVICES 

Food and Drug Administration 

[Docket No. FDA–2022–N–0075] 

Food and Drug Administration Quality 

Metrics Reporting Program; 

Establishment of a Public Docket; 

Request for Comments 

AGENCY: Food and Drug Administration, 

HHS. 

ACTION: Notice; establishment of a 

public docket; request for comments. 

SUMMARY: The Food and Drug 

Administration (FDA or Agency) is 

announcing the establishment of a 

docket to solicit comments on changes 

to FDA’s previously proposed quality 

metrics reporting program (QM 

Reporting Program). This notice 

describes considerations for refining the 

QM Reporting Program based on lessons 

learned from two pilot programs with 

industry that were announced in the 

Federal Register in June 2018, a Site 

Visit Program and a Quality Metrics 

Feedback Program, as well as 

stakeholder feedback on FDA’s 2016 

revised draft guidance for industry 

entitled ‘‘Submission of Quality Metrics 

Data.’’ FDA is interested in responses to 

the questions listed in section III of this 

document, in addition to any general 

comments on the proposed direction for 

the program. This notice is not intended 

to communicate our regulatory 

expectations for reporting quality 

metrics data to FDA but is instead 

intended to seek input from industry to 

inform the future regulatory approach. 

DATES: Submit either electronic or 

written comments by June 7, 2022. 

ADDRESSES: You may submit comments 

as follows. Please note that late, 

untimely filed comments will not be 

considered. Electronic comments must 

be submitted on or before June 7, 2022. 

The https://www.regulations.gov 

electronic filing system will accept 

comments until 11:59 p.m. Eastern Time 

at the end of June 7, 2022. Comments 

received by mail/hand delivery/courier 

(for written/paper submissions) will be 

considered timely if they are 

postmarked or the delivery service 

acceptance receipt is on or before that 

date. 

Electronic Submissions 

Submit electronic comments in the 

following way: 

• Federal eRulemaking Portal: ≤ 

https://www.regulations.gov. Follow the 

instructions for submitting comments. 

Comments submitted electronically, 

including attachments, to https:// 


jspears on DSK121TN23PROD with NOTICES1
13296 Federal Register / Vol. 87, No. 46 / Wednesday, March 9, 2022 / Notices 

www.regulations.gov will be posted to 

the docket unchanged. Because your 

comment will be made public, you are 

solely responsible for ensuring that your 

comment does not include any 

confidential information that you or a 

third party may not wish to be posted, 

such as medical information, your or 

anyone else’s Social Security number, or 

confidential business information, such 

as a manufacturing process. Please note 

that if you include your name, contact 

information, or other information that 

identifies you in the body of your 

comments, that information will be 

posted on https://www.regulations.gov. 

• If you want to submit a comment 

with confidential information that you 

do not wish to be made available to the 

public, submit the comment as a 

written/paper submission and in the 

manner detailed (see ‘‘Written/Paper 

Submissions’’ and ‘‘Instructions’’). 

Written/Paper Submissions 

Submit written/paper submissions as 

follows: 

• Mail/Hand delivery/Courier (for 

written/paper submissions): Dockets 

Management Staff (HFA–305), Food and 

Drug Administration, 5630 Fishers 

Lane, Rm. 1061, Rockville, MD 20852. 

• For written/paper comments 

submitted to the Dockets Management 

Staff, FDA will post your comment, as 

well as any attachments, except for 

information submitted, marked and 

identified, as confidential, if submitted 

as detailed in ‘‘Instructions.’’ 

Instructions: All submissions received 

must include the Docket No. FDA– 

2022–N–0075 for ‘‘FDA Quality Metrics 

Reporting Program; Establishment of a 

Public Docket; Request for Comments.’’ 

Received comments, those filed in a 

timely manner (see ADDRESSES), will be 

placed in the docket and, except for 

those submitted as ‘‘Confidential 

Submissions,’’ publicly viewable at 

https://www.regulations.gov or at the 

Dockets Management Staff between 9 

a.m. and 4 p.m., Monday through 

Friday, 240–402–7500. 

• Confidential Submissions—To 

submit a comment with confidential 

information that you do not wish to be 

made publicly available, submit your 

comments only as a written/paper 

submission. You should submit two 

copies total. One copy will include the 

information you claim to be confidential 

with a heading or cover note that states 

‘‘THIS DOCUMENT CONTAINS 

CONFIDENTIAL INFORMATION.’’ The 

Agency will review this copy, including 

the claimed confidential information, in 

its consideration of comments. The 

second copy, which will have the 

claimed confidential information 

redacted/blacked out, will be available 

for public viewing and posted on 

https://www.regulations.gov. Submit 

both copies to the Dockets Management 

Staff. If you do not wish your name and 

contact information to be made publicly 

available, you can provide this 

information on the cover sheet and not 

in the body of your comments and you 

must identify this information as 

‘‘confidential.’’ Any information marked 

as ‘‘confidential’’ will not be disclosed 

except in accordance with 21 CFR 10.20 

and other applicable disclosure law. For 

more information about FDA’s posting 

of comments to public dockets, see 80 

FR 56469, September 18, 2015, or access 

the information at: https://

www.govinfo.gov/content/pkg/FR-2015- 

09-18/pdf/2015-23389.pdf. 

Docket: For access to the docket to 

read background documents or the 

electronic and written/paper comments 

received, go to https://

www.regulations.gov and insert the 

docket number, found in brackets in the 

heading of this document, into the 

‘‘Search’’ box and follow the prompts 

and/or go to the Dockets Management 

Staff, 5630 Fishers Lane, Rm. 1061, 

Rockville, MD 20852, 240–402–7500. 

FOR FURTHER INFORMATION CONTACT: Jean 

Chung, Center for Drug Evaluation and 

Research, Food and Drug 

Administration, 10903 New Hampshire 

Ave., Bldg. 75, Rm. 6655, Silver Spring, 

MD 20993, 301–796–1874, jean.chung@

fda.hhs.gov. 

SUPPLEMENTARY INFORMATION: 

I. Background 

A. Quality Metrics 

For pharmaceutical manufacturing, 

quality metrics are objective means of 

measuring, evaluating, and monitoring 

the product and process life cycle to 

proactively identify and mitigate quality 

risks; thereby managing operations at 

higher levels of safety, efficacy, 

delivery, and performance. Quality 

metrics are used throughout the drug 

and biological product industry to 

monitor quality control systems and 

processes and drive continuous 

improvement efforts in manufacturing. 

Quality metrics are important because 

failure to update and innovate 

manufacturing practices and lack of 

operational reliability (i.e., state of 

control) can lead to quality problems 

that have a negative impact on public 

health. 

The minimum standard for ensuring 

that a manufacturer’s products are safe 

and effective is compliance with current 

good manufacturing practice (CGMP) 

requirements as outlined in current 

regulations and as recommended in 

current policies (21 CFR parts 210 and 

211 for drug products and the 

International Conference on 

Harmonisation guidance for industry 

entitled ‘‘Q7 Good Manufacturing 

Practice Guidance for Active 

Pharmaceutical Ingredients’’ (September 

2016); available at: https://www.fda.gov/ 

regulatory-information/search-fdaguidance-documents/q7-goodmanufacturing-practice-guidanceactive-pharmaceutical-ingredientsguidance-industry). However, 

compliance with CGMP does not 

necessarily indicate whether a 

manufacturer is investing in 

improvements and striving for 

sustainable compliance, which is the 

state of having consistent control over 

manufacturing performance and quality. 

Sustainable CGMP compliance is 

difficult to achieve without a focus on 

continual improvement. 

An effective Pharmaceutical Quality 

System (PQS) ensures both sustainable 

CGMP compliance and supply chain 

robustness. Quality metrics data can 

contribute to a manufacturer’s ability to 

develop an effective PQS because 

metrics provide insight into 

manufacturing performance and enable 

the identification of opportunities for 

updates and innovation to 

manufacturing practices. Quality 

metrics also play an important role in 

supplier oversight and can be used to 

inform the oversight of outsourced 

activities and material suppliers as well 

as appropriate monitoring activities to 

minimize supply chain disruptions. 

Quality metrics data provided by 

establishments can also be useful to 

FDA. These data can assist the Agency 

in developing compliance and 

inspection policies and practices to 

improve the Agency’s ability to predict, 

and therefore possibly mitigate, future 

drug shortages, and to encourage the 

pharmaceutical industry to implement 

innovative quality management systems 

for pharmaceutical manufacturing. For 

example, quality metrics data can be 

applied to FDA’s risk-based inspection 

scheduling, reducing the frequency and/ 

or length of routine surveillance 

inspections for establishments with 

metrics data that suggest sustainable 

compliance. Additionally, the 

submission of quality metrics data can 

provide ongoing insight into an 

establishment’s operations between 

inspections. 

As part of FDA’s shift towards a riskbased approach to regulation, the 

Agency proposed to develop and 

implement a QM Reporting Program to 

support its quality surveillance 

activities, as described in section I.B of 

this notice. Under this program, FDA 


jspears on DSK121TN23PROD with NOTICES1
Federal Register / Vol. 87, No. 46 / Wednesday, March 9, 2022 / Notices 13297 

intends to analyze the quality metrics 

data submitted by establishments to: (1) 

Obtain a more quantitative and objective 

measure of manufacturing quality and 

reliability at an establishment; (2) 

integrate the metrics and resulting 

analysis into FDA’s comprehensive 

quality surveillance program; and (3) 

apply the results of the analysis to assist 

in identifying products at risk for 

quality problems (e.g., quality-related 

shortages and recalls). 

B. FDA Guidance for Industry on the 

Submission of Quality Metrics Data 

In July 2015, FDA issued the draft 

guidance entitled ‘‘Request for Quality 

Metrics’’ (80 FR 44973), which 

described a potential mandatory 

program for product-based reporting of 

quality metrics. Under this proposed 

program, manufacturers would have 

submitted four primary metrics (lot 

acceptance rate (LAR), product quality 

complaint rate (PQCR), invalidated/ 

overturned out-of-specification rate 

(IOOSR), and annual product review 

(APR) or product quality review on-time 

rate) and three optional metrics (senior 

management engagement, corrective and 

preventative action (CAPA) 

effectiveness, and process capability/ 

performance). Stakeholder comments on 

the guidance included concerns 

regarding the burden associated with 

collecting, formatting, and submitting 

data at a product level across multiple 

establishments; technical comments on 

the proposed metrics and definitions; 

and legal concerns regarding the 

proposed mandatory program. 

Stakeholder commenters also suggested 

a phased-in approach to allow learning 

by both industry and FDA. 

In response to this feedback, FDA 

published a revised draft guidance in 

November 2016 entitled ‘‘Submission of 

Quality Metrics Data’’ (81 FR 85226). 

The 2016 guidance described an initial 

voluntary phase of the QM Reporting 

Program, with participants reporting 

data either by product or establishment, 

through an FDA submission portal. FDA 

removed one of the four metrics from 

the 2015 draft guidance and requested 

submission of the remaining three key 

metrics: (1) LAR to measure 

manufacturing process performance; (2) 

IOOSR to measure laboratory 

robustness; and (3) PQCR to measure 

patient or customer feedback and 

proposed incentives for participation. 

This guidance also described how FDA 

intended to utilize the submitted data. 

Stakeholder comments on the guidance 

indicated that the FDA-standardized 

definitions remained a challenge and 

incentives to participate in a voluntary 

program needed to be strengthened (e.g., 

direct collaboration with FDA to 

develop the program was an example of 

a strong incentive). Commenters 

requested a better understanding of the 

value and utility of the data to be 

submitted to FDA and how FDA would 

measure success of the program. 

Commenters also expressed a preference 

for a pilot program to gather industry 

input before implementing a 

widespread QM Reporting Program. 

C. Lessons Learned From FDA’s Quality 

Metrics Pilot Programs 

In Federal Register notices issued on 

June 29, 2018, FDA announced the 

availability of two pilot programs, a 

Quality Metrics Site Visit Program (83 

FR 30751) and a Quality Metrics 

Feedback Program (83 FR 30748) for any 

establishment that has a quality metrics 

program developed and implemented by 

the quality unit and used to support 

product and process quality 

improvement. 

The Quality Metrics Site Visit 

Program offered experiential learning 

for FDA staff and provided participating 

establishments an opportunity to 

explain the advantages and challenges 

associated with implementing and 

managing a Quality Metrics program. 

For example, participants provided 

feedback in the form of case studies to 

demonstrate the differences between the 

metric definitions proposed in the FDA 

draft guidances and definitions 

commonly used by industry for the 

same metrics. They proposed changes to 

the definitions, justifying why those 

changes (if any) would be needed. FDA 

toured the operations of 14 

establishments worldwide and engaged 

with establishments on topics such as: 

How quality metrics data are collected, 

analyzed, communicated (e.g., 

dashboards, business intelligence 

platforms), and reported throughout the 

organization in a structured and 

centralized manner; how management 

utilizes quality metrics data to monitor 

the performance of their supply 

network; how management leverages 

metrics to promote data-driven 

decisions; how an establishment 

implements and monitors continuous 

improvements based on metrics; how 

various quality metrics are defined; how 

actions were taken from observations 

resulting from quality metrics data 

reviews; and how efforts to proactively 

mitigate and prevent shortages are 

coordinated. 

In the Quality Metrics Feedback 

Program, participating establishments 

presented their quality metrics programs 

to FDA staff. The presentations were 

followed by discussions and knowledge 

sharing that focused on analytical 

strategies, exploratory data analyses, 

data preparation and structure, and 

visualizations for communication, as 

well as demonstrations on how FDA 

plans to analyze the data using 

advanced analytical techniques (e.g., 

data/text mining, interactive 

visualizations), sophisticated statistical 

methods (e.g., control charts, time series 

analysis), and machine learning (e.g., 

predictive analytics, natural language 

processing). In these discussions, FDA 

also obtained feedback on industry’s 

anticipated challenges in applying the 

approach described in FDA’s revised 

draft guidance. Participants had the 

opportunity to submit their quality 

metrics data through an FDA 

submission portal and provide feedback 

on their user experience. The industry 

participants represented different 

sectors of the pharmaceutical industry 

including innovator drug products, 

generic drug products, nonprescription 

(also known as over-the-counter (OTC)) 

drug products, and biological products. 

The dedicated meetings with industry 

during the two pilot programs that 

focused on data analytics resulted in the 

following key lessons learned for FDA, 

which will inform the direction of the 

QM Reporting Program: 

1. Different industry sectors prefer 

different metrics due to their individual 

operations and business dynamics 

needs. Therefore, it is necessary to 

implement a program with sufficient 

flexibility when choosing metrics. 

Identifying critical practice areas (e.g., 

manufacturing process performance) 

and allowing establishments to select 

appropriate metrics from several options 

is a more feasible approach. 

2. Any metric chosen to be reported 

should be meaningful to the practice 

area being measured, and the data 

collected on that metric should be able 

to influence decision making about 

process improvements and capital 

investments. 

3. In some instances, a combination of 

metrics rather than a single metric is 

preferred to assess a particular practice 

area. 

4. The majority of participants prefer 

to report data at an establishment level 

and have the capability to segment by 

product, but some participants prefer 

product-level reporting due to their 

business structure (e.g., a vertically 

integrated company). 

5. Calculating LAR and PQCR based 

on the definitions in the 2016 revised 

draft guidance can result in 

mathematical discrepancies such as 

rates over 100 percent or invalid 

calculations (i.e., dividing by zero)). 

These discrepancies are caused by 

inherent variabilities from real-time 


jspears on DSK121TN23PROD with NOTICES1
13298 Federal Register / Vol. 87, No. 46 / Wednesday, March 9, 2022 / Notices 

operations (e.g., lots may not be 

dispositioned in the same quarter in 

which they were started) or how 

denominators are defined for a specified 

period of time. 

6. While LAR and IOOSR are quality 

metrics that are routinely monitored by 

establishments, they are not discerning 

metrics due to limited variability over 

time or limited scope and can result in 

false positives by highlighting 

nonexistent performance issues. Other 

metrics should be identified as 

surrogates for manufacturing process 

performance and laboratory robustness. 

Examples include, but are not limited 

to, right-first-time rate, process 

capability, and adherence to lead time. 

7. The effectiveness of the quality 

system is a critical component of a QM 

Reporting Program as evidenced by 

numerous establishments collecting 

data around their PQS. Examples 

include metrics related to the 

effectiveness of CAPA programs, repeat 

deviations, maintenance programs, and 

timeliness. 

8. Metrics related to quality culture 

are important indicators of performance 

and reliability, but unlike other quality 

metrics, it is difficult to capture quality 

culture at an establishment based on 

numerical metrics alone. Both 

numerical key performance indicators 

(KPIs) (e.g., APR timeliness and near 

misses) and qualitative summaries (e.g., 

descriptions of management 

commitment or quality planning) can be 

used to further understand quality 

culture. 

9. FDA’s analysis of the data 

submitted during the Quality Metrics 

Feedback Program indicates that the use 

of statistical quality control applications 

(e.g., statistical process control and 

process capability) and machine 

learning/natural language processing are 

appropriate and meaningful analytical 

strategies to assess quality metrics data 

submitted by establishments. 

II. Proposed Direction for an FDA QM 

Reporting Program 

FDA has applied the lessons learned 

from the pilot programs and other 

stakeholder feedback toward refining 

the QM Reporting Program that was 

presented in the 2016 revised draft 

guidance. In this section, we summarize 

a potential direction for the program, 

and in section III we request input on 

specific aspects of this approach. 

FDA believes that a change in the 

entities responsible for collecting and 

submitting quality metrics data is not 

needed. ‘‘Covered establishments,’’ as 

defined in the 2016 revised draft 

guidance, are establishments engaged in 

the manufacture, preparation, 

propagation, compounding or 

processing of a ‘‘covered drug product’’ 

(products subject to an approved 

application under section 505 of the 

Federal Food, Drug, and Cosmetic Act 

(FD&C Act) (21 U.S.C. 355) or section 

351 of the Public Health Service Act; 

legally marketed pursuant to section 

505G of the FD&C Act (21 U.S.C. 355h) 

(nonprescription drugs marketed 

without an approved drug application); 

or marketed as unapproved finished 

drug products) or an active 

pharmaceutical ingredient used in the 

manufacture of a covered drug product. 

‘‘Covered establishments’’ include 

contract laboratories, contract 

sterilizers, and contract packagers. 

FDA is considering changes to other 

aspects of the QM Reporting Program. 

Stakeholders have indicated that 

different industry sectors may prefer 

different quality metrics. To provide 

flexibility to manufacturers, FDA would 

focus less on standardization of quality 

metrics and definitions. Instead, FDA 

would identify practice areas that are 

critical to ensure sustainable product 

quality and availability and would 

permit manufacturers to select a 

metric(s) from each practice area that 

are meaningful and enable 

establishments to identify continual 

improvement opportunities. The metric 

definitions would not specify how 

establishments calculate particular 

metrics. Rather, the reporting 

establishment would select the most 

appropriate metric(s) from each practice 

area and inform FDA how it was 

calculated. Through the collective 

feedback gathered from pilot 

participants, FDA has identified the 

following four general practice areas as 

appropriate at this time for the QM 

Reporting Program: (1) Manufacturing 

Process Performance, (2) PQS 

Effectiveness, (3) Laboratory 

Performance, (4) Supply Chain 

Robustness. Examples of quality metrics 

associated with each practice include 

the following: 

1. Manufacturing Process Performance 

• Process Capability/Performance 

Indices (Cpk/Ppk): A measure that 

compares the output of a process to the 

specification limits and can be 

calculated as a proportion (e.g., total 

number of attributes with Ppk greater 

than 1.33 divided by total number of 

attributes where Ppk is used). It is 

important to consider standard 

deviation measurements using a 

reasonable sample size. 

• LAR: A measure of the proportion 

of lots that were accepted in a given 

time period. Examples of inputs that can 

be used to calculate LAR include lots 

completed, lots dispositioned, lots 

attempted, lots rejected, lots released, 

lots approved, abandoned lots, and 

parallel/backup lots. 

• Right-First-Time Rate: A measure of 

the proportion of lots manufactured 

without the occurrence of a nonconformance. Examples of inputs that 

can be used to calculate a right-firsttime rate include number of deviations, 

lots dispositioned, lots attempted, 

number of nonconformances, and lots 

approved in the first pass. 

• Lot Release Cycle Time: A measure 

of the amount of time it takes for the lot 

disposition process. Lot release cycle 

time can be calculated with an 

appropriate unit of measurement such 

as number of hours or days. 

2. PQS Effectiveness 

• CAPA Effectiveness: A measure of 

the proportion of CAPA plan 

implemented and deemed effective (i.e., 

effectiveness verifications closed as 

effective). Examples of inputs that can 

be used to calculate CAPA effectiveness 

include number of CAPAs initiated, 

CAPAs closed on time, CAPAs closed as 

‘‘effective,’’ overdue CAPAs, and CAPAs 

resulting in retraining. 

• Repeat Deviation Rate: A measure 

of the proportion of recurring deviation 

measures. Examples of inputs that can 

be used to calculate repeat deviation 

rate include total number of deviations 

and number of deviations with the same 

assignable root cause. 

• Change Control Effectiveness: A 

measure of timeliness and effectiveness 

of implemented changes to GMP 

facilities, systems, equipment, or 

processes. Examples of inputs that can 

be used to calculate this metric include 

on-time closure of the change, total 

number of late effectiveness checks, 

total number of changes initiated, 

number of changes that are initiated 

reactively versus proactively, and total 

number of changes deemed effective. 

• Overall Equipment Effectiveness: A 

measure of operating productivity, 

utilizing planned production time. 

Overall equipment effectiveness can be 

calculated using inputs related to 

availability (e.g., planned production 

time, operating time), performance (e.g., 

production capacity), and quality (e.g., 

production output that does not result 

in acceptable product). 

• Unplanned Maintenance: A 

measure of the proportion of 

maintenance time that was not planned 

or scheduled. Examples of inputs that 

can be used to calculate this metric 

include total maintenance hours and 

planned maintenance hours. 


jspears on DSK121TN23PROD with NOTICES1
Federal Register / Vol. 87, No. 46 / Wednesday, March 9, 2022 / Notices 13299 

3. Laboratory Performance 

• Adherence to Lead Time: A 

measure of the proportion of tests in the 

laboratory that are completed on time 

according to schedule requirements. 

Adherence to lead time can be 

calculated, for example, by tracking 

initiation and testing turnover time in 

release and stability tests (i.e., the 

number of days between the start date 

and completion date for quality control 

(QC)); tracking data review and 

documentation; tracking final result 

reporting prior to batch disposition; or 

comparing QC testing completion date 

against the target date. 

• Right-First-Time Rate: A measure of 

the proportion of tests conducted 

without the occurrence of a deviation. 

Right-first-time rate as a metric for 

laboratory performance can be 

calculated, for example, by tracking the 

invalid assay rate, the number of assays 

invalidated due to human errors, or 

CGMP documentation errors during 

review. 

• IOOSR: A measure that indicates a 

laboratory’s ability to accurately 

perform tests. Examples of inputs that 

can be used to calculate this metric 

include total number of tests conducted 

and total number of out-of-specification 

results invalidated due to an aberration 

of the measurement process. 

• Calibration Timeliness: A measure 

of a laboratory’s adherence to 

inspecting, calibrating, and testing 

equipment for its intended purposes as 

planned. This metric can be measured 

by tracking calibration criteria and 

schedules. 

4. Supply Chain Robustness 

• On-Time In-Full (OTIF): A measure 

of the extent to which shipments are 

delivered to their destination containing 

the correct quantity and according to the 

schedule specified in the order. This 

metric can be calculated using inputs 

such as the number of orders shipped, 

number of past due orders, or number 

of orders shipped within tolerance. 

• Fill Rate: A measure that quantifies 

orders shipped as a percentage of the 

total demand for a given period. 

Examples of inputs that can be used to 

calculate this metric include total 

number of orders shipped, the number 

of orders placed, and the number of 

orders received. 

• Disposition On-Time: A measure of 

the proportion of lots in which the 

disposition was carried out on time. 

Examples of inputs that can be used to 

calculate this metric include the total 

number of lots dispositioned and the 

total number of lots dispositioned on 

time. 

• Days of Inventory On-Hand: A 

measure of how a company utilizes the 

average inventory available. It is the 

number of days that inventory remains 

in stock. 

Given that the majority of participants 

in the pilot programs prefer to report 

data at an establishment level, FDA is 

considering an approach for aggregating 

and reporting quality metrics data at the 

establishment level, with the option to 

segment by manufacturing train, 

product type, or product level (e.g., 

application number or product family). 

Once the data are submitted, FDA 

intends to analyze the information with 

statistical and machine learning 

methods to provide useful insights for 

inspection resource allocation. 

Examples include examination of 

product trends and clusters; exploratory 

and time-series analyses for signal 

identification, thereby monitoring the 

health of the establishment over time; 

and utilizing quality metrics data as an 

input into machine learning models to 

assist in determining an establishment’s 

overall PQS effectiveness. 

III. Request for Comments 

We are seeking comment on the 

following aspects of FDA’s proposed 

direction for its QM Reporting Program. 

We note that the questions posed in this 

section are not meant to be exhaustive. 

We are also interested in any other 

pertinent information that stakeholders 

and any other interested parties would 

like to provide on FDA’s QM Reporting 

Program. FDA encourages stakeholders 

to provide the rationale for their 

comments, including available 

examples and supporting information. 

A. Reporting Levels 

1. Do you agree that reporting should 

be aggregated at an establishment level? 

2. Would reporting at an 

establishment level facilitate submission 

of quality metrics data by contract 

manufacturing organizations? 

3. If you normally assess metrics by 

product family at an establishment, 

what are useful definitions of ‘‘product 

family’’ from your industry sector? 

B. Practice Areas and Quality Metrics 

1. If you think the general practice 

areas listed in section II of this notice 

would not meet the objectives of FDA 

QM Reporting Program, what other 

practice areas should FDA consider? 

2. If FDA were to consider Quality 

Culture as one of the general practice 

areas, what are the critical components 

of a robust quality culture and can any 

of these components be measured 

quantitatively? If so, how do you 

recommend quality culture information 

be captured as a quantitative metric 

(e.g., near misses, APR on-time, binary 

response to Quality Culture survey, or 

other numerical metrics/KPIs)? 

3. Do you think that any of the 

examples of quality metrics proposed by 

FDA would not be an appropriate 

measure for the designated practice 

area? 

4. What other metrics should FDA 

consider for a designated practice area? 

5. FDA is interested in an 

establishment’s experience with 

implementing process capability and 

performance metrics. For example, how 

would you report Cpk and/or Ppk to 

FDA as part of the QM Reporting 

Program (e.g., reporting Cpk and/or Ppk 

for certain products, aggregated at the 

establishment level)? 

6. A metric may need to be changed 

or adjusted by an establishment to better 

monitor PQS effectiveness, inform 

appropriate business strategy, or capture 

insightful trends, thereby driving 

continual improvement behaviors. What 

criteria should be applied to justify 

changing or modifying a quality metric 

(by either the establishment or by FDA)? 

How frequently would you expect 

changes or modifications to be needed? 

7. When would you rely on multiple 

metrics versus a single metric as an 

indicator when assessing a particular 

practice area (e.g., two metrics are 

considered in combination because one 

metric influences the other)? What 

combination of metrics have been 

meaningful and useful? 

C. Other Considerations 

1. Are there considerations unique to 

specific product categories (e.g., generic 

drug products, OTC drug products, or 

biological products) that should be 

addressed in the QM Reporting 

Program? 

2. What would be the optimal 

reporting frequency for quality metrics 

data submissions (e.g., monthly, 

quarterly, or yearly, and segmented by 

quarter or month)? 

3. In instances where a manufacturer 

is not able to extract domestic data and 

its submission to FDA contains both 

U.S. and foreign data, how can these 

data be submitted to FDA in a manner 

that would still be informative? 

4. Are there any other aspects of 

FDA’s proposed direction for the 

program that FDA should address in 

future policy documents? 

Dated: February 28, 2022. 

Lauren K. Roth, 

Associate Commissioner for Policy. 

[FR Doc. 2022–04972 Filed 3–8–22; 8:45 am] 

BILLING CODE 4164–01–P 


jspears on DSK121TN23PROD with NOTICES1