Guidance Document

 WHO/PQT: vector control

24 July 2017

Prequalification Timeline Key Performance Indicators (KPIs)

1. Introduction
The purpose of this document is to provide background information relating to prequalification (PQ) timeline
indicators. It also covers harmonization of PQ timeline calculation, and the method for computing WHO time,and defines the new set of key performance indicators (KPI) relating to PQ timeline.
This document should be read in conjunction with the summary table of PREQUALIFICATION TIMELINEKPIs presented separately, which contains details of targets.
See table in annex for the list of all indicators, including targets.

Key words
Application
An application expresses the intent of a manufacturer to participate in a given WHO process; it includes thesubmission of a dossier (see definition below).
Assessment
An assessment is a WHO process that consists of the evaluation of an application and concludes with a“pass/fail” decision.
CohortIn this context of defining PQ timeline indicators, the word cohort is used to define a group of products at agiven point in time, but does not necessarily imply that follow up over time is carried out.
Dossier
A dossier is a written document including all necessary evidence for WHO to make an informed decision. It isused in a broader sense than an actual regulatory dossier would be.
ProductA product is the subject of the assessment. In this context of defining PQ timeline indicators, the following
products are considered:







1

API: active pharmaceutical ingredient
FPP: finished pharmaceutical product
FVP: finished vaccine product
IMD: immunization device
IVD: in vitro diagnostic
MCD: male circumcision device
VCAI: vector control active ingredient
VCP: vector control product.

1

APIMF: active pharmaceutical ingredient master file, the API equivalent to a dossier, used in API and FPP
prequalification.
Prequalification Timeline Key Performance Indicators (KPIs)

1

Guidance Document

 WHO/PQT: vector control

24 July 2017

2. Timeline Calculation
In the past, the process for calculating PQ timelines and WHO PQ time was different for IVDs, medicines andvaccines.
The harmonization of the PQ timeline calculation is based on the principle that time is allocated to either WHOor the manufacturer. Mutual stop-clock time is applicable only in exceptional situations, and if both WHO andthe manufacturer agree on the stop-clock conditions.
The new algorithm proposed for determining WHO and manufacturer time is the following:

2

Rule 1 – From acceptance for assessment to dossier final decision , the split between WHO time andmanufacturer time applies only to the dossier review component.
Rule 2 – If at the time of dossier final decision the inspection or, when applicable, the laboratorycomponent, is still being performed, then after this point the split between WHO time andmanufacturer time is counted only with respect to the longest-running component, i.e. the component
that determines conclusion of the prequalification process, and therefore of the prequalification
timeline.

This algorithm is based on the assumption that the dossier review is the principal component of theassessment. If another component cannot be finalized within the overall dossier review time, it is considered
to be delaying the prequalification decision, and therefore should also be included in the timeline calculation.
The spilt between WHO and manufacturer time for the inspection and the laboratory components is also
calculated starting from acceptance of the dossier for assessment, but is taken into account only for the
calculation of the overall WHO PQ time, after the dossier component has been completed. In principle, all of
the time occurring between acceptance for assessment and the actual inspection, or the laboratory evaluation,is counted as WHO time. However, if the manufacturer delays the inspection or the laboratory evaluation, thetime between the initial date planned by WHO and the date proposed by the manufacturer is subtracted from
WHO time related to the specific component.
This timeline calculation algorithm will be embedded in the new IT system. Potential implementation in thecurrent systems should be evaluated, but is likely to be challenging.
See diagrams 1 & 2 in appendix for detailed figures explaining how timelines will be computed.

3. New Set of Performance Indicators
New performance indicators related to PQ timeline have been developed in addition to the current
measurements: total PQ time split between WHO time and manufacturer time. The new set of PQ
performance indicators includes both existing and new indicators. The purpose of these new indicators is toenable measurement of the progress of applications through the PQ pipeline more closely, rather than only
measuring the timeline at the end of the PQ assessment. It also aims at monitoring the performance of thedifferent components of PQ assessment. The components of PQ assessment are defined as: dossier review;inspection of the manufacturing site to verify good manufacturing practice or quality management systemcompliance — except for IMD — and performance evaluation (only for IVD) or laboratory testing (only forFVP).

2Dossier final decision is when all rounds of complete dossier review (i.e. for all sections of the dossier, if sections arereviewed separately) have been finalized.
Prequalification Timeline Key Performance Indicators (KPIs)

2

Guidance Document

 WHO/PQT: vector control

24 July 2017

A subset of the new set of indicators recently developed is proposed as key performance indicators (KPIs),
for the purpose of public reporting on PQ performance. These performance indicators are application-based
performance indicators, which measure the trends in overall application progress through the PQ pipeline.
The concept of first action is also introduced, with different types of first action, defined as:

3

dossier first action : communicating the result of complete (i.e. all sections of the dossier, if sections
are reviewed separately) initial dossier review to the manufacturer,
4
inspection first action : communicating the result of inspection (inspection report or desk review
outcome) to the manufacturer,
5
laboratory first action : communicating the result of laboratory testing/performance evaluation to themanufacturer, if applicable.

The performance indicators become the time to first action, defined as the time from acceptance forassessment to each of the above first actions, calculated independently for each component. The aim is tomeasure WHO responsiveness, i.e. the time after which the manufacturer can expect first complete feedback,covering each assessment component.
In addition, the following first action will also be used:screening first action: requesting further information (if necessary), or communicating theacceptance/rejection of application for assessment decision (if no further information is requested).

This leads to the performance indicator time to screening first action, defined as the time from application
submission to screening first action.
See diagrams 3 & 4 in appendix for detailed figures explaining how first actions will be computed.
Performance indicators are applicable to the following product types:diagnostics (Dx):medicines (Mx):in-vitro diagnostics (IVDs), male-circumcision devices (MCDs)
finished pharmaceutical products (FPPs), active pharmaceutical ingredients(APIs)
vaccines (Vx):finished vaccine products (FVPs), immunization device (IMDs)
vector control (VCx): vector control products (VCPs), vector control active ingredients (VCAIs).

Currently, prequalification of MCDs and IMDs is not subject to performance measurement related to PQ
timeline. However they will be covered by the new IT system; performance measurement will start once thesystem has gone live. Most of the new indicators presented below will not be measurable under the current
data management systems.
The targets for indicators have been set for 2018 and apply to all product types, unless specified
otherwise. The targets may be revised one year after implementation, to include new targets from 2019onwards, taking into account preliminary results.

3

Dossier first action is applicable to the review of full and abridged dossiers.
Inspection first action is only measured for initial inspections, i.e. inspections linked to a new PQ application.
5
Laboratory first action is only applicable to FVP and IVD.
4

Prequalification Timeline Key Performance Indicators (KPIs)

3

Guidance Document

 WHO/PQT: vector control

24 July 2017

Application‐based performance indicators
This category consists of two types of indicator:

those currently used to measure performance upon completion of prequalification assessment (forproducts that have attained prequalification), presented below under the annual PQ cohort headings,
those for measuring performance of applications as they progress through the PQ pipeline, presented
below under the submission cohort, assessment cohort and change cohort headings.

 Annual PQ cohort (products prequalified in a calendar year)
6

Each of the indicators — presented in tables below — is calculated separately by product type and split
between products assessed through the full assessment procedure and products assessed through theabridged assessment procedure.
Products prequalified (i.e. the annual PQ cohort)100

Number of products prequalified

Time to prequalification (from acceptance for assessment to prequalification)
110

Median WHO PQ time111

Median manufacturer PQ time112

Median total PQ time

KPI 1

7% of products prequalified at or below target WHO PQ time Submission cohort (PQ applications submitted for PQ assessment in a calendar year)
6

Each of the indicators — presented in tables below — is calculated separately by product type and split
between products assessed through the full assessment procedure and products assessed through theabridged assessment procedure.
PQ applications submitted for PQ assessment (i.e. the submission cohort)200

Number of PQ applications submitted

Time to screening first action
210
KPI 2

Number of screening first actions% of screening first actions taken at or below target time

6Product types: API; FPP; FVP; IMD; IVD; MCD; VCAI; VCP.
Total PQ time is split between WHO PQ time and manufacturer PQ time; a mutual stop clock is applicable only in
exceptional situations, if both WHO and the manufacturer agree on the stop-clock conditions.
7

Prequalification Timeline Key Performance Indicators (KPIs)

4

Guidance Document

 WHO/PQT: vector control

24 July 2017

 Assessment cohort (PQ applications accepted for PQ assessment in a calendar year)
6

Each of the indicators — presented in tables below — is calculated separately by product type and split
between products assessed through the full assessment procedure and products assessed through theabridged assessment procedure; the cohort is followed for 3 years.
PQ applications accepted for PQ assessment (i.e. the assessment cohort)300

Time to dossier first action
310
KPI 3.1



Number of dossier first actions% of dossier first actions taken at or below target time

Time to inspection first action
320
KPI 3.2



Number of PQ applications accepted

Number of inspection first actions% of inspection first actions taken at or below target time

Time to laboratory first action
330
KPI 3.3

Number of laboratory first actions% of laboratory first actions taken at or below target time

 Change cohort (post-PQ change applications accepted for change assessment in a calendar year)
6

Each of the indicators — presented in tables below — is calculated separately by product type and split
8
according to the different types of post-PQ changes .
Post-PQ change applications accepted for change assessment (i.e. the change cohort)400Number of post-PQ change applications assessed

Time to post-PQ change first action
410
KPI 4

Number of post-PQ changes first actions% of post-PQ change first actions taken at or below target time

8

APIMF: major/minor amendments, immediate/annual notifications; FPP: major/minor variations, immediate/annualnotifications; FVP: variations with prior-approval, immediate/annual notifications; IVD: reportable/non-reportable changes.
Prequalification Timeline Key Performance Indicators (KPIs)

5

Guidance Document

 WHO/PQT: vector control

24 July 2017

4. Appendix

Prequalification Timeline Key Performance Indicators (KPIs)

6

Guidance Document

 WHO/PQT: vector control

Prequalification Timeline Key Performance Indicators (KPIs)

24 July 2017

7

Guidance Document

 WHO/PQT: vector control

Prequalification Timeline Key Performance Indicators (KPIs)

24 July 20178

Guidance Document

 WHO/PQT: vector control

Prequalification Timeline Key Performance Indicators (KPIs)

24 July 20179