Document Headings

                    Document headings vary by document type but may contain                    the following:                  
                -                       the agency or agencies that issued and signed a document                    
                -                       the number of the CFR title and the number of each part                      the document amends, proposes to amend, or is directly                      related to                    
                -                       the agency docket number / agency internal file number                    
                -                       the RIN which identifies each regulatory action listed in                      the Unified Agenda of Federal Regulatory and Deregulatory                      Actions                    

                    See the                                          Document Drafting Handbook                                        for more details.                  

###### Department of Health and Human Services

###### Food and Drug Administration

          - [Docket No. FDA-2022-N-0075]

# AGENCY:

Food and Drug Administration, HHS.

# ACTION:

Notice; establishment of a public docket; request for comments.

# SUMMARY:

                        The Food and Drug Administration (FDA or Agency) is announcing the establishment of a docket to solicit comments on changes to FDA's previously proposed quality metrics reporting program (QM Reporting Program). This notice describes considerations for refining the QM Reporting Program based on lessons learned from two pilot programs with industry that were announced in the                         **Federal Register**                         in June 2018, a Site Visit Program and a Quality Metrics Feedback Program, as well as stakeholder feedback on FDA's 2016 revised draft guidance for industry entitled “Submission of Quality Metrics Data.” FDA is interested in responses to the questions listed in section III of this document, in addition to any general comments on the proposed direction for the program. This notice is not intended to communicate our regulatory expectations for reporting quality metrics data to FDA but is instead intended to seek input from industry to inform the future regulatory approach.                    

# DATES:

Submit either electronic or written comments by June 7, 2022.

# ADDRESSES:

                        You may submit comments as follows. Please note that late, untimely filed comments will not be considered. Electronic comments must be submitted on or before June 7, 2022. The                         *https://www.regulations.gov*                         electronic filing system will accept comments until 11:59 p.m. Eastern Time at the end of June 7, 2022. Comments received by mail/hand delivery/courier (for written/paper submissions) will be considered timely if they are postmarked or the delivery service acceptance receipt is on or before that date.                    

### Electronic Submissions

Submit electronic comments in the following way:
      -  Federal eRulemaking Portal: >https://www.regulations.gov.                      Follow the instructions for submitting comments. Comments submitted electronically, including attachments, to                      https://      ( print page 13296)    www.regulations.gov                      will be posted to the docket unchanged. Because your comment will be made public, you are solely responsible for ensuring that your comment does not include any confidential information that you or a third party may not wish to be posted, such as medical information, your or anyone else's Social Security number, or confidential business information, such as a manufacturing process. Please note that if you include your name, contact information, or other information that identifies you in the body of your comments, that information will be posted on                      https://www.regulations.gov. 
      - If you want to submit a comment with confidential information that you do not wish to be made available to the public, submit the comment as a written/paper submission and in the manner detailed (see “Written/Paper Submissions” and “Instructions”).

### Written/Paper Submissions

Submit written/paper submissions as follows:
      -  Mail/Hand delivery/Courier (for written/paper submissions):                      Dockets Management Staff (HFA-305), Food and Drug Administration, 5630 Fishers Lane, Rm. 1061, Rockville, MD 20852.                
      - For written/paper comments submitted to the Dockets Management Staff, FDA will post your comment, as well as any attachments, except for information submitted, marked and identified, as confidential, if submitted as detailed in “Instructions.”

*Instructions:*                     All submissions received must include the Docket No. FDA-2022-N-0075 for “FDA Quality Metrics Reporting Program; Establishment of a Public Docket; Request for Comments.” Received comments, those filed in a timely manner (see                     **ADDRESSES**                    ), will be placed in the docket and, except for those submitted as “Confidential Submissions,” publicly viewable at                     *https://www.regulations.gov*                     or at the Dockets Management Staff between 9 a.m. and 4 p.m., Monday through Friday, 240-402-7500.                
      - Confidential Submissions—To submit a comment with confidential information that you do not wish to be made publicly available, submit your comments only as a written/paper submission. You should submit two copies total. One copy will include the information you claim to be confidential with a heading or cover note that states “THIS DOCUMENT CONTAINS CONFIDENTIAL INFORMATION.” The Agency will review this copy, including the claimed confidential information, in its consideration of comments. The second copy, which will have the claimed confidential information redacted/blacked out, will be available for public viewing and posted on https://www.regulations.gov.                      Submit both copies to the Dockets Management Staff. If you do not wish your name and contact information to be made publicly available, you can provide this information on the cover sheet and not in the body of your comments and you must identify this information as “confidential.” Any information marked as “confidential” will not be disclosed except in accordance with  21 CFR 10.20  and other applicable disclosure law. For more information about FDA's posting of comments to public dockets, see  80 FR 56469 , September 18, 2015, or access the information at:                      https://www.govinfo.gov/​content/​pkg/​FR-2015-09-18/​pdf/​2015-23389.pdf. 

*Docket:*                     For access to the docket to read background documents or the electronic and written/paper comments received, go to                     *https://www.regulations.gov*                     and insert the docket number, found in brackets in the heading of this document, into the “Search” box and follow the prompts and/or go to the Dockets Management Staff, 5630 Fishers Lane, Rm. 1061, Rockville, MD 20852, 240-402-7500.                

# FOR FURTHER INFORMATION CONTACT:

                        Jean Chung, Center for Drug Evaluation and Research, Food and Drug Administration, 10903 New Hampshire Ave., Bldg. 75, Rm. 6655, Silver Spring, MD 20993, 301-796-1874,                         *jean.chung@fda.hhs.gov.*

# SUPPLEMENTARY INFORMATION:

## I. Background

### A. Quality Metrics

                    For pharmaceutical manufacturing, quality metrics are objective means of measuring, evaluating, and monitoring the product and process life cycle to proactively identify and mitigate quality risks; thereby managing operations at higher levels of safety, efficacy, delivery, and performance. Quality metrics are used throughout the drug and biological product industry to monitor quality control systems and processes and drive continuous improvement efforts in manufacturing. Quality metrics are important because failure to update and innovate manufacturing practices and lack of operational reliability (                    *i.e.,*                     state of control) can lead to quality problems that have a negative impact on public health.                

                    The minimum standard for ensuring that a manufacturer's products are safe and effective is compliance with current good manufacturing practice (CGMP) requirements as outlined in current regulations and as recommended in current policies (21 CFR parts 210 and 211 for drug products and the International Conference on Harmonisation guidance for industry entitled “Q7 Good Manufacturing Practice Guidance for Active Pharmaceutical Ingredients” (September 2016); available at:                     *https://www.fda.gov/​regulatory-information/​search-fda-guidance-documents/​q7-good-manufacturing-practice-guidance-active-pharmaceutical-ingredients-guidance-industry*). However, compliance with CGMP does not necessarily indicate whether a manufacturer is investing in improvements and striving for sustainable compliance, which is the state of having consistent control over manufacturing performance and quality. Sustainable CGMP compliance is difficult to achieve without a focus on continual improvement.                

An effective Pharmaceutical Quality System (PQS) ensures both sustainable CGMP compliance and supply chain robustness. Quality metrics data can contribute to a manufacturer's ability to develop an effective PQS because metrics provide insight into manufacturing performance and enable the identification of opportunities for updates and innovation to manufacturing practices. Quality metrics also play an important role in supplier oversight and can be used to inform the oversight of outsourced activities and material suppliers as well as appropriate monitoring activities to minimize supply chain disruptions.

Quality metrics data provided by establishments can also be useful to FDA. These data can assist the Agency in developing compliance and inspection policies and practices to improve the Agency's ability to predict, and therefore possibly mitigate, future drug shortages, and to encourage the pharmaceutical industry to implement innovative quality management systems for pharmaceutical manufacturing. For example, quality metrics data can be applied to FDA's risk-based inspection scheduling, reducing the frequency and/or length of routine surveillance inspections for establishments with metrics data that suggest sustainable compliance. Additionally, the submission of quality metrics data can provide ongoing insight into an establishment's operations between inspections.

                    As part of FDA's shift towards a risk-based approach to regulation, the Agency proposed to develop and implement a QM Reporting Program to support its quality surveillance activities, as described in section I.B of this notice. Under this program, FDA                           ( print page 13297)                        intends to analyze the quality metrics data submitted by establishments to: (1) Obtain a more quantitative and objective measure of manufacturing quality and reliability at an establishment; (2) integrate the metrics and resulting analysis into FDA's comprehensive quality surveillance program; and (3) apply the results of the analysis to assist in identifying products at risk for quality problems (                    *e.g.,*                     quality-related shortages and recalls).                

### B. FDA Guidance for Industry on the Submission of Quality Metrics Data

In July 2015, FDA issued the draft guidance entitled “Request for Quality Metrics” (80 FR 44973), which described a potential mandatory program for product-based reporting of quality metrics. Under this proposed program, manufacturers would have submitted four primary metrics (lot acceptance rate (LAR), product quality complaint rate (PQCR), invalidated/overturned out-of-specification rate (IOOSR), and annual product review (APR) or product quality review on-time rate) and three optional metrics (senior management engagement, corrective and preventative action (CAPA) effectiveness, and process capability/performance). Stakeholder comments on the guidance included concerns regarding the burden associated with collecting, formatting, and submitting data at a product level across multiple establishments; technical comments on the proposed metrics and definitions; and legal concerns regarding the proposed mandatory program. Stakeholder commenters also suggested a phased-in approach to allow learning by both industry and FDA.

                    In response to this feedback, FDA published a revised draft guidance in November 2016 entitled “Submission of Quality Metrics Data” (81 FR 85226). The 2016 guidance described an initial voluntary phase of the QM Reporting Program, with participants reporting data either by product or establishment, through an FDA submission portal. FDA removed one of the four metrics from the 2015 draft guidance and requested submission of the remaining three key metrics: (1) LAR to measure manufacturing process performance; (2) IOOSR to measure laboratory robustness; and (3) PQCR to measure patient or customer feedback and proposed incentives for participation. This guidance also described how FDA intended to utilize the submitted data. Stakeholder comments on the guidance indicated that the FDA-standardized definitions remained a challenge and incentives to participate in a voluntary program needed to be strengthened (                    *e.g.,*                     direct collaboration with FDA to develop the program was an example of a strong incentive). Commenters requested a better understanding of the value and utility of the data to be submitted to FDA and how FDA would measure success of the program. Commenters also expressed a preference for a pilot program to gather industry input before implementing a widespread QM Reporting Program.                

### C. Lessons Learned From FDA's Quality Metrics Pilot Programs

                    In                     **Federal Register**                     notices issued on June 29, 2018, FDA announced the availability of two pilot programs, a Quality Metrics Site Visit Program (83 FR 30751) and a Quality Metrics Feedback Program (83 FR 30748) for any establishment that has a quality metrics program developed and implemented by the quality unit and used to support product and process quality improvement.                

                    The Quality Metrics Site Visit Program offered experiential learning for FDA staff and provided participating establishments an opportunity to explain the advantages and challenges associated with implementing and managing a Quality Metrics program. For example, participants provided feedback in the form of case studies to demonstrate the differences between the metric definitions proposed in the FDA draft guidances and definitions commonly used by industry for the same metrics. They proposed changes to the definitions, justifying why those changes (if any) would be needed. FDA toured the operations of 14 establishments worldwide and engaged with establishments on topics such as: How quality metrics data are collected, analyzed, communicated (                    *e.g.,*                     dashboards, business intelligence platforms), and reported throughout the organization in a structured and centralized manner; how management utilizes quality metrics data to monitor the performance of their supply network; how management leverages metrics to promote data-driven decisions; how an establishment implements and monitors continuous improvements based on metrics; how various quality metrics are defined; how actions were taken from observations resulting from quality metrics data reviews; and how efforts to proactively mitigate and prevent shortages are coordinated.                

                    In the Quality Metrics Feedback Program, participating establishments presented their quality metrics programs to FDA staff. The presentations were followed by discussions and knowledge sharing that focused on analytical strategies, exploratory data analyses, data preparation and structure, and visualizations for communication, as well as demonstrations on how FDA plans to analyze the data using advanced analytical techniques (                    *e.g.,*                     data/text mining, interactive visualizations), sophisticated statistical methods (                    *e.g.,*                     control charts, time series analysis), and machine learning (                    *e.g.,*                     predictive analytics, natural language processing). In these discussions, FDA also obtained feedback on industry's anticipated challenges in applying the approach described in FDA's revised draft guidance. Participants had the opportunity to submit their quality metrics data through an FDA submission portal and provide feedback on their user experience. The industry participants represented different sectors of the pharmaceutical industry including innovator drug products, generic drug products, nonprescription (also known as over-the-counter (OTC)) drug products, and biological products.                

The dedicated meetings with industry during the two pilot programs that focused on data analytics resulted in the following key lessons learned for FDA, which will inform the direction of the QM Reporting Program:

                    1. Different industry sectors prefer different metrics due to their individual operations and business dynamics needs. Therefore, it is necessary to implement a program with sufficient flexibility when choosing metrics. Identifying critical practice areas (                    *e.g.,*                     manufacturing process performance) and allowing establishments to select appropriate metrics from several options is a more feasible approach.                

2. Any metric chosen to be reported should be meaningful to the practice area being measured, and the data collected on that metric should be able to influence decision making about process improvements and capital investments.

3. In some instances, a combination of metrics rather than a single metric is preferred to assess a particular practice area.

                    4. The majority of participants prefer to report data at an establishment level and have the capability to segment by product, but some participants prefer product-level reporting due to their business structure (                    *e.g.,*                     a vertically integrated company).                

                    5. Calculating LAR and PQCR based on the definitions in the 2016 revised draft guidance can result in mathematical discrepancies such as rates over 100 percent or invalid calculations (                    *i.e.,*                     dividing by zero)). These discrepancies are caused by inherent variabilities from real-time                           ( print page 13298)                        operations (                    *e.g.,*                     lots may not be dispositioned in the same quarter in which they were started) or how denominators are defined for a specified period of time.                

6. While LAR and IOOSR are quality metrics that are routinely monitored by establishments, they are not discerning metrics due to limited variability over time or limited scope and can result in false positives by highlighting nonexistent performance issues. Other metrics should be identified as surrogates for manufacturing process performance and laboratory robustness. Examples include, but are not limited to, right-first-time rate, process capability, and adherence to lead time.

7. The effectiveness of the quality system is a critical component of a QM Reporting Program as evidenced by numerous establishments collecting data around their PQS. Examples include metrics related to the effectiveness of CAPA programs, repeat deviations, maintenance programs, and timeliness.

                    8. Metrics related to quality culture are important indicators of performance and reliability, but unlike other quality metrics, it is difficult to capture quality culture at an establishment based on numerical metrics alone. Both numerical key performance indicators (KPIs) (                    *e.g.,*                     APR timeliness and near misses) and qualitative summaries (                    *e.g.,*                     descriptions of management commitment or quality planning) can be used to further understand quality culture.                

                    9. FDA's analysis of the data submitted during the Quality Metrics Feedback Program indicates that the use of statistical quality control applications (                    *e.g.,*                     statistical process control and process capability) and machine learning/natural language processing are appropriate and meaningful analytical strategies to assess quality metrics data submitted by establishments.                

## II. Proposed Direction for an FDA QM Reporting Program

FDA has applied the lessons learned from the pilot programs and other stakeholder feedback toward refining the QM Reporting Program that was presented in the 2016 revised draft guidance. In this section, we summarize a potential direction for the program, and in section III we request input on specific aspects of this approach.

FDA believes that a change in the entities responsible for collecting and submitting quality metrics data is not needed. “Covered establishments,” as defined in the 2016 revised draft guidance, are establishments engaged in the manufacture, preparation, propagation, compounding or processing of a “covered drug product” (products subject to an approved application under section 505 of the Federal Food, Drug, and Cosmetic Act (FD&C Act) (21 U.S.C. 355) or section 351 of the Public Health Service Act; legally marketed pursuant to section 505G of the FD&C Act (21 U.S.C. 355h) (nonprescription drugs marketed without an approved drug application); or marketed as unapproved finished drug products) or an active pharmaceutical ingredient used in the manufacture of a covered drug product. “Covered establishments” include contract laboratories, contract sterilizers, and contract packagers.

FDA is considering changes to other aspects of the QM Reporting Program. Stakeholders have indicated that different industry sectors may prefer different quality metrics. To provide flexibility to manufacturers, FDA would focus less on standardization of quality metrics and definitions. Instead, FDA would identify practice areas that are critical to ensure sustainable product quality and availability and would permit manufacturers to select a metric(s) from each practice area that are meaningful and enable establishments to identify continual improvement opportunities. The metric definitions would not specify how establishments calculate particular metrics. Rather, the reporting establishment would select the most appropriate metric(s) from each practice area and inform FDA how it was calculated. Through the collective feedback gathered from pilot participants, FDA has identified the following four general practice areas as appropriate at this time for the QM Reporting Program: (1) Manufacturing Process Performance, (2) PQS Effectiveness, (3) Laboratory Performance, (4) Supply Chain Robustness. Examples of quality metrics associated with each practice include the following:

### 1. Manufacturing Process Performance

    -  Process Capability/Performance Indices (Cpk/Ppk):                      A measure that compares the output of a process to the specification limits and can be calculated as a proportion (                     e.g.,                      total number of attributes with Ppk greater than 1.33 divided by total number of attributes where Ppk is used). It is important to consider standard deviation measurements using a reasonable sample size.                
    -  LAR:                      A measure of the proportion of lots that were accepted in a given time period. Examples of inputs that can be used to calculate LAR include lots completed, lots dispositioned, lots attempted, lots rejected, lots released, lots approved, abandoned lots, and parallel/backup lots.                
    -  Right-First-Time Rate:                      A measure of the proportion of lots manufactured without the occurrence of a non-conformance. Examples of inputs that can be used to calculate a right-first-time rate include number of deviations, lots dispositioned, lots attempted, number of nonconformances, and lots approved in the first pass.                
    -  Lot Release Cycle Time:                      A measure of the amount of time it takes for the lot disposition process. Lot release cycle time can be calculated with an appropriate unit of measurement such as number of hours or days.                

### 2. PQS Effectiveness

    -  CAPA Effectiveness:                      A measure of the proportion of CAPA plan implemented and deemed effective (                     i.e.,                      effectiveness verifications closed as effective). Examples of inputs that can be used to calculate CAPA effectiveness include number of CAPAs initiated, CAPAs closed on time, CAPAs closed as “effective,” overdue CAPAs, and CAPAs resulting in retraining.                
    -  Repeat Deviation Rate:                      A measure of the proportion of recurring deviation measures. Examples of inputs that can be used to calculate repeat deviation rate include total number of deviations and number of deviations with the same assignable root cause.                
    -  Change Control Effectiveness:                      A measure of timeliness and effectiveness of implemented changes to GMP facilities, systems, equipment, or processes. Examples of inputs that can be used to calculate this metric include on-time closure of the change, total number of late effectiveness checks, total number of changes initiated, number of changes that are initiated reactively versus proactively, and total number of changes deemed effective.                
    -  Overall Equipment Effectiveness:                      A measure of operating productivity, utilizing planned production time. Overall equipment effectiveness can be calculated using inputs related to availability (                     e.g.,                      planned production time, operating time), performance (                     e.g.,                      production capacity), and quality (                     e.g.,                      production output that does not result in acceptable product).                
    -  Unplanned Maintenance:                      A measure of the proportion of maintenance time that was not planned or scheduled. Examples of inputs that can be used to calculate this metric include total maintenance hours and planned maintenance hours.                           ( print page 13299)      

### 3. Laboratory Performance

    -  Adherence to Lead Time:                      A measure of the proportion of tests in the laboratory that are completed on time according to schedule requirements. Adherence to lead time can be calculated, for example, by tracking initiation and testing turnover time in release and stability tests (                     i.e.,                      the number of days between the start date and completion date for quality control (QC)); tracking data review and documentation; tracking final result reporting prior to batch disposition; or comparing QC testing completion date against the target date.                
    -  Right-First-Time Rate:                      A measure of the proportion of tests conducted without the occurrence of a deviation. Right-first-time rate as a metric for laboratory performance can be calculated, for example, by tracking the invalid assay rate, the number of assays invalidated due to human errors, or CGMP documentation errors during review.                
    -  IOOSR:                      A measure that indicates a laboratory's ability to accurately perform tests. Examples of inputs that can be used to calculate this metric include total number of tests conducted and total number of out-of-specification results invalidated due to an aberration of the measurement process.                
    -  Calibration Timeliness:                      A measure of a laboratory's adherence to inspecting, calibrating, and testing equipment for its intended purposes as planned. This metric can be measured by tracking calibration criteria and schedules.                

### 4. Supply Chain Robustness

    -  On-Time In-Full (OTIF):                      A measure of the extent to which shipments are delivered to their destination containing the correct quantity and according to the schedule specified in the order. This metric can be calculated using inputs such as the number of orders shipped, number of past due orders, or number of orders shipped within tolerance.                
    -  Fill Rate:                      A measure that quantifies orders shipped as a percentage of the total demand for a given period. Examples of inputs that can be used to calculate this metric include total number of orders shipped, the number of orders placed, and the number of orders received.                
    -  Disposition On-Time:                      A measure of the proportion of lots in which the disposition was carried out on time. Examples of inputs that can be used to calculate this metric include the total number of lots dispositioned and the total number of lots dispositioned on time.                
    -  Days of Inventory On-Hand:                      A measure of how a company utilizes the average inventory available. It is the number of days that inventory remains in stock.                

                    Given that the majority of participants in the pilot programs prefer to report data at an establishment level, FDA is considering an approach for aggregating and reporting quality metrics data at the establishment level, with the option to segment by manufacturing train, product type, or product level (                    *e.g.,*                     application number or product family).                

Once the data are submitted, FDA intends to analyze the information with statistical and machine learning methods to provide useful insights for inspection resource allocation. Examples include examination of product trends and clusters; exploratory and time-series analyses for signal identification, thereby monitoring the health of the establishment over time; and utilizing quality metrics data as an input into machine learning models to assist in determining an establishment's overall PQS effectiveness.

## III. Request for Comments

We are seeking comment on the following aspects of FDA's proposed direction for its QM Reporting Program. We note that the questions posed in this section are not meant to be exhaustive. We are also interested in any other pertinent information that stakeholders and any other interested parties would like to provide on FDA's QM Reporting Program. FDA encourages stakeholders to provide the rationale for their comments, including available examples and supporting information.

### A. Reporting Levels

1. Do you agree that reporting should be aggregated at an establishment level?

2. Would reporting at an establishment level facilitate submission of quality metrics data by contract manufacturing organizations?

3. If you normally assess metrics by product family at an establishment, what are useful definitions of “product family” from your industry sector?

### B. Practice Areas and Quality Metrics

1. If you think the general practice areas listed in section II of this notice would not meet the objectives of FDA QM Reporting Program, what other practice areas should FDA consider?

                    2. If FDA were to consider Quality Culture as one of the general practice areas, what are the critical components of a robust quality culture and can any of these components be measured quantitatively? If so, how do you recommend quality culture information be captured as a quantitative metric (                    *e.g.,*                     near misses, APR on-time, binary response to Quality Culture survey, or other numerical metrics/KPIs)?                

3. Do you think that any of the examples of quality metrics proposed by FDA would not be an appropriate measure for the designated practice area?

4. What other metrics should FDA consider for a designated practice area?

                    5. FDA is interested in an establishment's experience with implementing process capability and performance metrics. For example, how would you report Cpk and/or Ppk to FDA as part of the QM Reporting Program (                    *e.g.,*                     reporting Cpk and/or Ppk for certain products, aggregated at the establishment level)?                

6. A metric may need to be changed or adjusted by an establishment to better monitor PQS effectiveness, inform appropriate business strategy, or capture insightful trends, thereby driving continual improvement behaviors. What criteria should be applied to justify changing or modifying a quality metric (by either the establishment or by FDA)? How frequently would you expect changes or modifications to be needed?

                    7. When would you rely on multiple metrics versus a single metric as an indicator when assessing a particular practice area (                    *e.g.,*                     two metrics are considered in combination because one metric influences the other)? What combination of metrics have been meaningful and useful?                

### C. Other Considerations

                    1. Are there considerations unique to specific product categories (                    *e.g.,*                     generic drug products, OTC drug products, or biological products) that should be addressed in the QM Reporting Program?                

                    2. What would be the optimal reporting frequency for quality metrics data submissions (                    *e.g.,*                     monthly, quarterly, or yearly, and segmented by quarter or month)?                

3. In instances where a manufacturer is not able to extract domestic data and its submission to FDA contains both U.S. and foreign data, how can these data be submitted to FDA in a manner that would still be informative?

4. Are there any other aspects of FDA's proposed direction for the program that FDA should address in future policy documents?

Dated: February 28, 2022.

Lauren K. Roth,

Associate Commissioner for Policy.

[FR Doc. 2022-04972 Filed 3-8-22; 8:45 am]

BILLING CODE 4164-01-P