EUROPEAN COMMISSIONJOINT RESEARCH CENTRE
Institute for Health and Consumer Protection
European Centre for the Validation of Alternative Methods (ECVAM)ECVAMSCIENTIFIC
ADVISORYCOMMITTEE(ESAC)ESAC Working Group Peer Review Consensus Reportona Givaudan-coordinated study transferability and reliability of the
KeratinoSens assay for skin sensitisation testing.

Title page information
File name
Abbreviated title of ESAC
requestRelating to ESAC REQUEST Nr.
Request discussed through

TEMPLATE_ESAC-WG_REPORT-v3.doc
ECVAM study concerning the transferability and reliability of the
KeratinoSens assay
ESAC request ER2011-04
The ESAC WG conducted the peer review from December 2011 to
September 2012. Two face-to-face meetings were organized(December 2011, and February 2012), followed by two telephone
conferences (February and April 2012) and finalized by written
procedure in November 2012.

TABLE OF CONTENTS
ESAC WORKING GROUP ......................................................................................................................................... 4
NOTE ON THIS REPORTING TEMPLATE .................................................................................................................. 5
ABBREVIATIONS USED IN THE DOCUMENT ........................................................................................................... 5
EXECUTIVE SUMMARY ........................................................................................................................................... 6
1. DATA COLLECTION – INFORMATION / DATA SOURCES USED ........................................................................... 9
1.1 EXISTING DATA USED AS REFERENCE DATA ............................................................................................................. 9
1.2 EXISTING DATA USED AS TESTING DATA ................................................................................................................10
1.3 SEARCH STRATEGY ............................................................................................................................................11
1.4 SELECTION CRITERIA APPLIED TO THE AVAILABLE INFORMATION ................................................................................11
2. STUDY OBJECTIVE AND DESIGN ........................................................................................................................11
2.1 CLARITY OF THE DEFINITION OF THE STUDY OBJECTIVE .............................................................................................11
2.2 ANALYSIS OF THE SCIENTIFIC RATIONALE PROVIDED ................................................................................................12
2.3 ANALYSIS OF THE REGULATORY RATIONALE PROVIDED ............................................................................................15
2.4 APPROPRIATENESS OF THE STUDY DESIGN.............................................................................................................16
2.5 APPROPRIATENESS OF THE STATISTICAL EVALUATION ..............................................................................................17
3. TEST DEFINITION (MODULE 1) ..........................................................................................................................24
3.1 QUALITY AND COMPLETENESS OF THE OVERALL TEST DEFINITION ..............................................................................24
3.2 QUALITY OF THE BACKGROUND PROVIDED CONCERNING THE PURPOSE OF THE TEST METHOD ........................................25
3.3 QUALITY OF THE DOCUMENTATION AND COMPLETENESS OF SOPS AND PREDICTION MODELS ........................................25
4. DATA QUALITY ...................................................................................................................................................26
4.1 OVERALL QUALITY OF THE EVALUATED DATA .........................................................................................................26
4.2 SUFFICIENCY OF THE EVALUATED DATA IN VIEW OF THE STUDY OBJECTIVE ...................................................................26
4.3 QUALITY OF THE REFERENCE DATA FOR EVALUATING RELIABILITY AND RELEVANCE ........................................................27
5. TEST MATERIALS ................................................................................................................................................27
5.1 SUFFICIENCY OF THE NUMBER OF EVALUATED TEST ITEMS IN VIEW OF THE STUDY OBJECTIVE ..........................................28
5.2 REPRESENTATIVENESS OF THE TEST ITEMS WITH RESPECT TO APPLICABILITY.................................................................28
6. WITHIN-LABORATORY REPRODUCIBILITY (MODULE 2) ...................................................................................33
6.1 ASSESSMENT OF REPEATABILITY AND REPRODUCIBILITY IN THE SAME LABORATORY ......................................................33
6.2 CONCLUSION ON WITHIN-LABORATORY REPRODUCIBILITY AS ASSESSED BY THE STUDY ..................................................34
7. TRANSFERABILITY (MODULE 3) .........................................................................................................................35
7.1 QUALITY OF DESIGN AND ANALYSIS OF THE TRANSFER PHASE ...................................................................................35
7.2 CONCLUSION ON TRANSFERABILITY TO A SECOND LABORATORY AS ASSESSED BY THE STUDY...........................................35
8. BETWEEN-LABORATORY REPRODUCIBILITY (MODULE 4) ................................................................................37
8.1 ASSESSMENT OF REPRODUCIBILITY IN DIFFERENT LABORATORIES...............................................................................37
8.2 CONCLUSION ON REPRODUCIBILITY AS ASSESSED BY THE STUDY ................................................................................38
9. PREDICTIVE CAPACITY (MODULE 5) ..................................................................................................................38
9.1 ADEQUACY OF THE ASSESSMENT OF THE PREDICTIVE CAPACITY IN VIEW OF THE PURPOSE ..............................................39
9.2 OVERALL RELEVANCE (BIOLOGICAL RELEVANCE AND ACCURACY) OF THE TEST METHOD IN VIEW OF THE PURPOSE ..............39
10. APPLICABILITY DOMAIN (MODULE 6) .............................................................................................................41
10.1 APPROPRIATENESS OF STUDY DESIGN TO CONCLUDE ON APPLICABILITY DOMAIN, LIMITATIONS AND EXCLUSIONS ............41
ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 2 of 77

10.2 QUALITY OF THE DESCRIPTION OF APPLICABILITY DOMAIN, LIMITATIONS, EXCLUSIONS ................................................41
11. PERFORMANCE STANDARDS (MODULE 7) .....................................................................................................43
11.1 ADEQUACY OF THE PROPOSED ESSENTIAL TEST METHOD COMPONENTS ..................................................................43
11.2 ADEQUACY OF THE REFERENCE CHEMICALS.........................................................................................................43
12. READINESS FOR STANDARDISED USE .............................................................................................................4312.1 ASSESSMENT OF THE READINESS FOR REGULATORY PURPOSES ................................................................................43
12.2. ASSESSMENT OF THE READINESS FOR OTHER USES ...............................................................................................44
12.3 CRITICAL ASPECTS IMPACTING ON STANDARDISED USE ..........................................................................................44
12.4 GAP ANALYSIS ...............................................................................................................................................45
13. OTHER CONSIDERATIONS................................................................................................................................45
14. CONCLUSIONS ON THE STUDY ........................................................................................................................46
14.1 SUMMARY OF THE RESULTS AND CONCLUSIONS OF THE STUDY................................................................................46
14.1.1 Test items ..........................................................................................................................................46
14.1.2 Summary of study results .................................................................................................................46
14.2 EXTENT TO WHICH STUDY CONCLUSIONS ARE JUSTIFIED BY THE STUDY RESULTS ALONE................................................47
14.3 EXTENT TO WHICH CONCLUSIONS ARE PLAUSIBLE IN THE CONTEXT OF EXISTING INFORMATION .....................................47
15. RECOMMENDATIONS ......................................................................................................................................49
15.1 GENERAL RECOMMENDATIONS .........................................................................................................................49
15.2 SPECIFIC RECOMMENDATIONS (E.G. CONCERNING IMPROVEMENT OF SOPS) ............................................................49
16. REFERENCES .....................................................................................................................................................50
17. ANNEXES ..........................................................................................................................................................52

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 3 of 77

ESAC Working Group
This report was prepared by the "ESAC Working Group (ESAC WG) for Sensitization/KeratinoSens,
charged with conducting a detailed scientific peer review of the ECVAM study concerning thetransferability and reliability of the KeratinoSens assay.
The ESAC WG had been set up by the ESAC during its meeting on March 2011 (ESAC 34). Basis for thescientific review was the ECVAM request to ESAC concerning a scientific review (ESAC request
ER2011-04).
This report was endorsed by the ESAC WG on 22.10.2012 and represents the consensus view of the
ESAC WG.
This ESAC WG peer review consensus report was endorsed by the ESAC on 17.12.2012.

The ESAC WG had the following members:









Dr. Erwin Roggen (ESAC member, Chair of ESAC WG and rapporteur)
Prof. Walter Pfaller (ESAC member, ESAC Vice Chair)Prof. Wallace Hayes (ESAC member)Dr. Maja Alecsic (external expert)
Dr. Emanuela Corsini (external expert)
Dr. David Lovell (external expert)
Dr. Michael Woolhiser (external expert)
Prof. Yong Heo (external expert)

ESAC Coordination / Scientific Secretariat:

Dr. Claudius Griesinger (ESAC Coordination)
Dr. Alexandre Angers (specific support)

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 4 of 77

NOTE ON THIS REPORTING TEMPLATE
The template follows the ECVAM modular approach and allows at the same time for the descriptionof the analysis and conclusions concerning more specific questions. The template was approved by
the ESAC through written procedure on 29 October 2010.
The template can be used for various types of validation studies (e.g. prospective full studies,retrospective studies, performance-based studies and prevalidation studies).
Depending on the study type and the objective of the study, not all sections may be applicable.
However, for reasons of consistency and to clearly identify which information requirements have notbeen sufficiently addressed by a specific study, this template is uniformly used for the evaluation ofvalidation studies.
Explanatory notes to the paragraph titles (in green) have been added on 17 November 2010.
These notes provide guidance on the type of information / analysis expected under eachsection. Depending on the purpose and scope of the study to be reviewed, some of the
aspects mentioned in the explanatory notes may not be applicable or only be applicable tosome extent. Moreover, the explanatory notes are not intended to represent an exhaustive
list of possible issues to be addressed under the respective heading, but are thought toprovide some guidance with respect to the considerations typically expected.

For all of the template’s numbered sections the summary view of ESAC WG is given in bold
followed by more detailed comments ("general observations" and "specific observations").

ABBREVIATIONS USED IN THE DOCUMENT

BLR

Between-laboratory reproducibility



ECVAM

European Centre for the Validation of Alternative Methods



ESAC

ECVAM Scientific Advisory Committee



ESAC WG

ESAC Working Group



GCCP

Good Cell Culture Practice

GLP

Good Laboratory PracticePC

Positive Control



SOP

Standard Operating Procedure (used here as equivalent to 'protocol')

VC

Vehicle Control



VMT

Validation Management Team

WLR

Within-laboratory reproducibility

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 5 of 77

Executive summary
Following a request from ECVAM to ESAC for peer review of and scientific advice on an ECVAMcoordinated prevalidation study concerning the KeratinoSens assay, an ESAC Working Group (ESAC
WG) was set up by ESAC. The ESAC WG was charged with conducting a detailed scientific peer reviewof the ECVAM study concerning the transferability and reliability of the KeratinoSens assay.
The ESAC WG had been set up by the ESAC during its meeting on March 2011 (ESAC 34). Basis for thescientific review was the ECVAM request to ESAC concerning a scientific review (ESAC request
ER2011-04).
The date for the opinion was set to be 4-5 October 2011 (ESAC 35). However, ambiguities andinconsistencies in the report required clarification by the test submitter. Two requests regarding
clarification were sent from the WG (via the ECVAM Coordinator) to Givaudan: 16.12.2011 and08.02.2012. These extra steps resulted in substantial delays.
The ESAC WG conducted the peer review from December 2011 to September 2012. Two face-to-facemeetings were organized (December 2011, and February 2012), followed by two telephone
conferences (February and April 2012) and finalized by written procedure in November 2012.
------------------------------------------------------------------------------------------------------The WG was presented with a wealth of information about the test chemicals, and the assessment of
WLR, transferability, BLR and predictive capacity of the test. Also the applicability domain of the test
was addressed in detail.
It was obvious from the submitted material that this study had not been under EURL-ECVAM
supervision. The data and the flow of events would have been more transparent if the report had
followed the lay-out of this ESAC report. It would have been very helpful if the test submitters had
formulated their own conclusions/opinions when referring to any of the numerous attachments that
had followed the report. By referral to the attachments, the WG had to work out for itself what wasmeant and how the data had been interpreted by the submitter.
The WG identified a number of ambiguities and inconsistencies, without explanations being provided,
which added hurtles to the evaluation of the report.
1.

Ambiguities:

2.

It was not clear why the statistical approach applied was chosen for the evaluation of the test
results.
The test design was not clear.
Inconsistencies:

Data analysis apparently moved from a test result oriented (e.g. Imax, EC1.5) to a prediction(S/NS) oriented approach.
Test acceptance criteria changed over time without explanation as to why this occurred.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 6 of 77

Acceptance criteria were not consistently applied.
Chemicals that were used for test development and refinement were inappropriately
included in the assessment of the BLR and the predicitive capacity.

The WG addressed these issues by requesting additional information and re-analysis of the data fromthe test submitter (See Annexes).
------------------------------------------------------------------------------------------------------The information provided did not provide clarification about the statistical approach applied in the
study. The WG decide not to go into further discussion, and to focus on the outcome of theprediction model (S/NS).
The test design was sufficiently clarified, and the data were re-analysed on the basis of the various
identified test acceptance criteria. This allowed the WG to assess properly the reproducibility,transferability and predictive capacity.
The WG attempted to recalculate the predictive capacity of the KeratinoSens based upon thechemicals that had not been included in test development and refinement. Since the number of wellcharacterized non-sensitizers (i.e. chemicals with negative LLNA outcome) among the eligible
chemicals was considered too low, therefore the WG requested data on more negative compounds.
------------------------------------------------------------------------------------------------------On the basis of this requested additional information the WG came to the following conclusions:
Test chemicals:
The 113 selected chemicals represented a sufficient number of materials, reasonable structural
diversity and a variety of sensitising potency classes. Pre- and pro-haptens were included. Therefore,
the selection of chemicals was considered sufficient to gain information on the applicability domain
and the limitations of the test method.
The small number of non-sensitizers (N=4) in the list of chemicals (N=46) considered eligible forassessing the predictive capacity of the test was supplemented with 80 chemicals with negative
LLNA.
WLR (14 chemicals):
The WG considers the concordance reported acceptable and in agreement with target value (85%)
for WLR performance standards as published in international accepted guidelines (Performancestandards of TG439 in vitro skin irritation).
Transferability (7 chemicals):
The conclusion on transferability was justified on the basis of concordant predictions (S/NS) between
the lead laboratory and the naive laboratories. The WG endorses the conclusion that the test method
can be transferred to naive laboratories that are experienced with cell culture techniques.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 7 of 77

BLR (21 chemicals):
The S/NS prediction gave comparible results for the majority of chemicals (85.7 – 90.5%) amonglaboratories, taking into consideration the explanations given for the outliers.
Predictive capacity:
The conclusions regarding the predictivity are sound given the overall value of 76.6%. The key point
here is that the ‘weight of evidence’ data were considered for comparison as opposed to a single
assay outcome.
Compiling all the reliable data provided by the test submitter (N = 213), the KeratinoSens revealed an
acceptable sensitivity, specificity and accuracy of 79.3%, 84.5% and 81.7%, respectively.
Negative results cannot, however, exclude the sensitization potential as weak and low moderatesensitizers are likely to be missed.
Applicability domain:
The applicability domain is less clearly defined with this data set and it is prudent to assess thisfurther by testing additional sets of chemicals that are not obviously part of the applicability domain.
It is clear that specific amine reactivity and requirement for some form of activation are not the only
issues that may need to be addressed.
------------------------------------------------------------------------------------------------------The WG made the following recommendations:
The test method can be used for S/NS identification of chemicals. Therefore, the test was considered
ready for the next steps in the ECVAM process. A Validation study should however include more welldefined non-sensitizing compounds. Furthermore, a consistent use of acceptance criteria (Annex 4)
should be assured.
Since the test revealed issues around weak and low to moderate sensitizers, negative results cannot
rule out a sensitization potential. This problem should be clearly flagged and/or addressed to besolved.
At the SOP level, the test submitters were recommended to modify the 96-well plate design, whichcurrently is prone to bias.
Integration of this assay with other predictive tests as they emerge needs to be based on a better
defined applicability domain.
Eventual combination of the KeratinoSens assay with a reactivity based approach needs to include
unambiguous identification of reactivity and any specificity associated with it.
Training should be considered for laboratories with no experience with this test.
ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 8 of 77

1. Data collection – information / data sources used

1.1 Existing data used as reference data

General comments:
The chemicals were selected from the ECVAM, ICCVAM and Sens-it-iv databases (DB). Three different
sets of chemicals were distinguished (Table 1).
From the submitted information it was not very clear to the WG how the reference data were used
to identify for the purpose of this study credible reference sensitizers (S) and non-sensitizers (NS).
Specific comments:The ‘Silver List’ (N=67, Attachment 11) was derived from the ICCVAM, ECVAM, Sens-it-iv lists.
It was unclear for the WG how the data were integrated with result the classification of thecompounds as sensitizers and non-sensitizers.
Additional input requested: By telephone the test submitter (Andres Natsch) explained thatclassification of the selected compounds was based upon concordant results for LLNA andGuinea pig (GP) results.

Additional chemicals sent by ECVAM (N=8) originated from the ECVAM Coordinative
Prevalidation Study. The ‘weight-of-evidence’ (WoE) approach used for selecting thesechemicals needed further explanation.
Additional input requested: By telephone the test submitter (Andres Natsch) explained thatclassification was based upon concordant LLNA and GP data, with Ni-Chloride and Xylene asadditional compounds.



The chemicals in the Extended list (N=46, Attachment 12c), were selected from the ICCVAM
Validation paper and database The WoE approach used for selecting these chemicals needs
further explanation.
Additional input requested: By telephone the test submitter (Andres Natsch) explained thatclassification was based upon concordant LLNA and GP results, or LLNA and humanmaximization test.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 9 of 77

Table 1:
Chemical sets

Derivation

SourceSilver List (N = 67)

Concordant results for LLNA and GP
data

ICCVAM, ECVAM, Sens-it-iv

ECVAM list (N = 8)

Concordant results for LLNA, GP, + Ni
chloride and Xylene

Published data (ECVAM, ICCVAM DB)

Extended list (N = 46)

Concordant results for LLNA and GP
data,
OR
LLNA
and
human
maximization test

ICCVAM validation paper, DB

1.2 Existing data used as testing data

General comments:
As with the reference data, the information submitted did not provide a clarification of how theexisting data were used to identify sensitizers (S) and non-sensitizers (NS).
Specific comments:
Additional input requested: See ‘Specific Comments’ and Table 1 in section 1.1
The Silver List was used for test development, refinement and evaluation. Data obtained from this list
were also included in the assessment of the predictive capacity of the test. A bias may have been
introduced in the assessment of the predictive capacity of the test, because the test was optimized
to detect these chemicals.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 10 of 77

1.3 Search strategy

General comments:
The test submitters extracted their reference and test data from established highly credible sources(ECVAM, ICCVAM and Sens-it-iv databases).

1.4 Selection criteria applied to the available information

General comments:
The VMG relied fully on the quality of the data provided in the various existing databases.

2. Study objective and design

2.1 Clarity of the definition of the study objective

General comments:
The objective of this study is defined comprehensibly in two specific test reports (attachments. 4cand 12c) and as attachment 17e: "a screening method to test for the potential of chemicals to be skinsensitizing--The main focus is the testing of chemicals evaluated under REACH and under the
Cosmetic Directive."
The study has been designed to generate information on the test methods’ transferability andreproducibility to allow recommendations to be made on these two aspects in view of the future useof this test method in an integrated approach for the full replacement of the currently used
regulatory animal tests. In addition the data generated in this study will inform possible futureevaluations of the test methods’ predictive capacity.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 11 of 77

Specific comment:
It would have been helpful if the objectives had been specifically formulated in the body of thesubmission.

2.2 Analysis of the scientific rationale provided
General comments:
The KeratinoSens assay addresses indirectly steps 2 (haptenization) and step 3 (epidermal
inflammation) in the mechanism of skin sensitization induction (Adler et al., 2011). The relevance of
Nrf2-Keap1 pathway to skin sensitisation is explained by the direct reactivity of sensitising materials
to key cysteine residues of Keap1 (Nrf2 repressor protein).
The available evidence for this links to processes of sensitization via sensitizer-induced up-regulationof genes with ARE in their promoter sequence. The question of direct reactivity with Keap1 cysteine
residues remains unanswered.
Not all skin sensitisers appear to up-regulate the Nrf2 pathway. This can be explained by either aneed for metabolic transformation or an exclusive reactivity of some test chemicals with lysine.
It may be possible that additional or alternative pathways can be activated via modification ofcysteine residues on Keap1 protein.
Specific comments:
The Nrf2 regulatory pathway, comprising of the repressor protein Keap1 (kelch-like ECH-associatedprotein 1), the transcription factor Nrf2 and the antioxidant response element (ARE), has been shown
to play a critical role in protecting a variety of tissues from a wide array of toxic insults. It is emerging
as one of the key toxicity pathways induced by skin sensitisers. The transcription factor Nrf2
regulates the battery of genes that are induced from the ARE in response to an electrophilic insult tocoordinate cytoprotective response. The Nrf2-ARE system regulates expression of numerous
cytoprotective enzymes and under non-stimulated conditions Keap1 negatively regulates nuclear
translocation of Nrf2 and facilitates degradation of Nrf2 via proteasome. Upon exposure toelectrophilic chemical Nrf2 is liberated from Keap1-dependent degradation and accumulates in the
nucleus (Maruyama and Itoh, 2005).
Figure 1: Mode of action
It has been shown that Keap1 interacts with some electrophilic
chemicals, leading to activation of ARE-dependent genes,
however the precise nature of this interaction is not clear. It is
postulated that skin sensitising electrophiles covalently modify
the key cysteine residues on Keap1 which sets off the describedsequence of events. Natsch and Emter (2008) investigated AREdependent gene induction by skin sensitising chemicals andreported that majority of these chemicals do induce genes

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 12 of 77

regulated by this element (Fig. 1).
In itself, this interaction does not constitute functional in vivo relevance. Natsch et al (2010) reviewed
recent literature in the field allowing insights into in vivo relevance and identifying some missing links
for further research. The integrated hypothesis they put forward suggests that there are two
separate signalling cascades for cysteine and lysine reactive chemicals, ultimately leading to Th1 andTh2 responses respectively, with modification of the key cysteine residues on Keap1 being the firststep in the Th1 pathway. Several lines of evidence exist that indicate that Nrf2-Keap1-ARE regulatory
pathway is activated by cysteine-reactive skin sensitizers:









The sensor protein Keap1 contains highly reactive cysteine residues and covalentmodification of these leads to dislocation from Nrf2, which then accumulates and activates
genes with ARE promoter sequence
Natsch and Emter (2008) used Hepa1C1C7 murine hepatoma cells and measured induction of
ARE-regulated quinone reductase in response to 96 skin sensitising chemicals. In parallel, anumber of chemicals were tested in the reporter cell line AREc32 (containing an 8-fold repeatof the ARE sequence upstream of the luciferase gene. 14/15 of strong/extreme sensitisersand 30/34 moderate sensitisers induced ARE-dependent luciferase activity and in many cases
this was paralleled by an induction of quinine reductase activity in murine hepatoma cell line.
Subsequently, Emter et al (2010) have used a human relevant cell line with luciferase
reporter gene under control of a single copy of the ARE element of the human AKR1C2 genestably inserted into HaCaT keratinocytes. The different steps in the signalling cascade (Keap1
binding, nuclear accumulation of Nrf2 and binding to the consensus ARE sequence) have notbeen investigated separately. Other investigators showed Nrf2 increase in dendritic cells (DC)upon treatment with skin sensitizers (Ade et al 2009 and Megherbi et al 2009). Similarly,
Python et al (2009) showed up-regulation of two key Nrf2-regulated genes, CES1 and NQO1
in cinnamic aldehyde treated DCs and DC cell line MUTZ3, indicating an up-regulation if theNrf2 pathway in both cell types. However, not all skin sensitizers appear to up-regulate theNrf2 pathway and part of the explanation for some of these materials is either a need formetabolic transformation (generally poor in cell lines) or exclusive lysine reactivity of some of
the test chemicals.
Evidence for the in vivo relevance of the Nrf2 pathway induction in skin sensitisation comesfrom studies on the effects of DNCB and oxazolone on the Nrf2 -/- knockout mice (ref). These
animals show a reduced reaction to sensitizers, with Th1 cytokine IFNγ response completely
abolished, and Th2 cytokine IL-4 unaffected. These data indicate that the induction of Nrf2
pathway is essential for sensitisation to occur.
Further in vivo studies reveal up-regulation of IFNγ and IFNγ-regulated genes by sensitizers(DNCB, oxazolone and toluene-2,4-diisocyanate) but not irritants (croton oil and nonanoicacid) (Ku et al., 2008). All three sensitizers have intrinsic reactivity to both cysteine andlysine. Boverhof et al (2009) tested lysine reactive chemicals and did not observed upregulation of the IFNγ gene.
A large body of evidence exists for differential cytokine induction in the lymph nodes byeither respiratory or skin sensitizers (anhydrides and isocyanates Vs DNCB), which led to aconclusion that respiratory senitizers preferentially induce Th2 cytokines and skin sensitizers
induce Th1 cytokines. Natsch et al suggest that this difference is due to selective reactivity tolysine of respiratory sensitisers, compared to preferential cysteine reactivity or mixed
reactivity to cysteine and lysine for skin sensitizers. The authors argue that solely lysine
reactive sensitizers are rare amongst sensitizers, however peptide reactivity assays show that
there are a number of solely lysine reactive chemicals which are known sensitizers.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 13 of 77

Furthermore, human proteins are very rich in amine-type nucleophiles and cysteine residues
are relatively rare and most often unavailable in the secreted proteins as they form
disulphide bridges.
The relevance of this pathway to skin sensitisation is explained by the direct reactivity of sensitisingmaterials to key cysteine residues of Keap1, however there is no direct evidence of this, rather anindirect evidence via up-regulation of genes with ARE in their promoter sequence. This event is thenexploited in the KeratinoSens assay, but the question of direct reactivity with Keap1 cysteine residues
remains unanswered in this case.
According to Dinkova-Kostova et al (2005), the chemicals that induce phase 2 enzymes via Nrf2
pathway are structurally dissimilar and from a variety of structural classes but all appear to be able toreact with sulfhydryl groups via oxido-reduction, alkylation or disulphide interchange. Natsch et alstate that this interaction is covalent in nature (i.e. solely alkylation). The fact that some solely lysine
reactive chemicals appear to induce Nrf2 pathway in KeratinoSens assay is probably due to the abilityof these chemicals to oxidise cysteine residues on Keap1 rather than covalently modify them. Indeed,reactivity studies show that chemicals (in particular sensitising aldehydes) conjugate to lysine
residues but can oxidise cysteine residues without actually generating adducts. It is therefore logicalthat this pathway may also be activated by simple (chemically driven) oxidation of key cysteine
residues on Keap1. For example, benzaldehyde or phenylacetaldehyde are not directly reactive withcysteine, but both strongly oxidise thiols, both react via Schiff base formation to amine based
nucleophiles, yet both are positive in KeratinoSens assay.
Specificity of the interaction with key cyssteine residues has been studied extensively with a variety
of inducers of this pathway (reviewed by Dinkova-Kostova et al., 2005). The consensus of suchstudies suggests that only certain set of key cysteine residues (four key residues are often mentionedout of the 27 residues in human Keap1) are likely to lead to the Nrf2 pathway induction, and thatsome reactive chemicals might not induce the Nrf2 if they modify cysteine residues that are not fromthis key set. Others argue that there are more than the four exclusive Cys residues which mediate
electrophile/oxidative stress (Holland et al 2008 and others).
It is likely that this system is much more finely tuned in vivo, where the concentrations of Keap1 anda reactive chemical are much lower, and the modification sites vary between the chemicals. It is also
likely that different reactions will modify different cysteine residues and all of this flexibility in thesystem allows for the variety of situations which can trigger the phase 2 response in cells.
The interaction of Keap1 cysteine residues with GSH/GSSG has also been a subject of study (Holland
et al 2008). This mechanism of Nrf2 pathway induction involves glutathionylation of cysteine residues
on Keap1 (as a result of oxidative stress and disturbance of the cellular GSH/GSSG balance) inducing
formation of both type 1 and type 2 disulphides. However, only about half of Keap1 cysteine residues
are subject to this type of interaction. There is a suggestion, but no direct evidence of this happening
in vivo, although the above study worked with physiologically relevant GSH/GSSG ratios. It is possiblethat this pathway may also be activated by simple (chemically driven) oxidation of key cysteine
residues on Keap1.
Kobayashi et al (2009) show some evidence that suggests that targeting particular set of cysteine
residues will dictate what type of response (distinct biological effect) will be resulting from achemical treatment (‘cysteine code’). There is also evidence that Keap1 inhibits the NF-κB signalling
pathway via induction (through a direct interaction) of IKKβ degradation (Lee et al., 2009). Similar to
ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 14 of 77

the Nrf2 pathway discussed above, it appears that Keap1 specifically interacts with IKKβ, a kinase
which regulates the ubiquitination of IkB protein (inhibitor of kB). Upon stimuli (which canpresumably include modification of Keap1 protein), IKKβ is released from its complex with Keap1 andis able to catalyse the phosphorylation of 3 serine residues on IkB, thus tagging this protein for
ubiquitination and proteosomal degradation. This releases NF-kB from its complex with IkB andallows its nuclear translocation where it binds the response elements (RE) on the relevant parts ofDNA, resulting in the upregulation of proteins and change in cell function. Thus it is possible that
additional or alternative pathways can be activated via modification of Cys residues on Keap1 protein.
On this background, it can be concluded that the applicability domain for the KeratinoSens may not
be as clear as suggested by the test submitters. Further research should be undertaken to acquire abetter understanding of the mechanisms driving the Keap1-Nrf2 pathway.

2.3 Analysis of the regulatory rationale provided
General comments:
The regulatory objective of the study is clearly stated and comprehensibly defined: a screening
method to test for the potential of chemicals to be skin sensitizing.
The main focus is the testing of chemicals evaluated under REACH and under the Cosmetic Directive.
The test is proposed as a stand-alone method for classification and labelling (Regulation No
1272/2008 on classification, labelling and packaging of substances and mixtures) or as part of an
integrated testing strategy and combined with read-across for a basic risk assessment and safety
prediction.
Specific comments:
The integrated testing strategy for skin sensitisation has not yet been identified or proposed in any
detail. The test submitters suggest that in combination with other assays and read-across a basic risk
assessment may be achieved. It is difficult to envisage the exact role KeratinoSens assay would playin as yet undefined strategy.
It was stated that for classification and labelling the assay could be used as a stand-alone method,
however, there are clear limitations with the applicability domain and there may be a requirement
for additional reactivity testing:limitations with respect to scoring moderate and weak sensitizers (See Module 5).
potential limitations with the applicability domain (Module 6) imposing additional
reactivity testing for some chemicals which would fail to be identified as sensitizers
in KeratinoSens assay.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 15 of 77

2.4 Appropriateness of the study design

General comments:
An overall study design was not provided, but Modules 1-6 were carefully described. The number oftest items ranged from 7 chemicals to a larger set of 67 chemicals (44 sensitizers and 23 nonsensitizers).
The selection of the test items (with a good spread between potency categories) as well as thenumber of test items (both sensitizers and non-sensitizers) is appropriate for the purpose of thestudy. The number of laboratories involved in the ring trial was sufficient.
Some of the inter-laboratory studies were run with coded samples. None of the studies were under
GLP. The overall technical aspects of the various studies were conducted in a quality fashion.
Test acceptance criteria were established, but not subsequently adhered to in the final studies.
Specific comments:
The Silver List of chemicals (Natsch and Emter, 2008) was used for test development, refinement andevaluation. Results obtained from this list were also included in the assessment of the predictive
capacity of the test. By including test chemicals previously used for training the test, a bias might
have been introduced in the assessment of the predictive capacity of the test.
The test chemicals cover:a) a molecular weight range from 30 to 388 daltons,b) a cLogP range from -4.8 to 5.2,c) a range of skin sensitizer classes (from no sensitizer tor mild to extreme sensitisers),d) all key reference lists of chemicals which have been published on skin sensitization.
The test method (SOPs) was transferred to 5 laboratories and a pre-validation study conducted which
comprised 6 of the 7 modules required by ECVAM for test validation.
There was no retesting for unqualified tests, nor was there a strategy described to do so.
It would have been useful to report reactivity data (actual or predicted) of the chosen chemical
dataset.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 16 of 77

2.5 Appropriateness of the statistical evaluation

General comments:
Analysis of the appropriateness of the statistical evaluation indentified a number of issues needing
clarification (See ‘Specific comments’, pp 17-19).
Additional input requested: The WG requested additional information by letter (16.12.2011 and08.02.2012) to clarify issues related to:

Test designo It was unclear how the KeratinoSens assay was carried out to derive a final
prediction (S/NS)
Acceptance criteriao Why acceptance criteria were not consistently applied, and varied between
different sections of the document;o Concerns that these inconsistencies affected WLR and BLR calculations
Statistical methods
o Apparent inconsistency between the statistical analyses reported(attachments 7a-10e, and Natsch et al. (2011)).

The additional information received back was analysed by the WG, and the following conclusions
were drawn:

Test designo It was agreed that the schematic outline, updated by Givaudan by adding a
MTT parallel plate to assess cytotoxicity (condition 4 of the Test AcceptanceCriteria), helped in understanding how a KeratinoSens test is carried out toachieve a final prediction (Fig. 2).
Acceptance criteriao The WG established that the criteria sets found and described by the ESAC
WG corresponded to the criteria sets communicated by Givaudan in the
resubmission ("clarification") as shown in Table 2. The ESAC WG agreed thatthe question of which criteria had been used to analyse in particular the ringtrial data was now sufficiently clear. Moreover, it was now clear whichcriteria Givaudan recommends for future use of the assay (Annex 3, Section
3.2).
Statistical methods
o The WG did not consider that the additional information sent by Givaudan
regarding the statistical analyses provided the clarification they wererequesting. In any case, the WG decided to focus on the analysis of thewithin laboratory reproducibility (WLR) in terms of concordance of thepredictions obtained within the same laboratory. This means that the only
study that can provide this information is in attachment 4c, the evaluation of
14 chemicals at Givaudan, performed in three complete experiments.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 17 of 77

oConcerning the occurrence and impact of the invalid runs, the WG requested
Givaudan to provide a final analysis of the results in the submission, taking
into account three different scenarios. On the basis of this new information,
the WG came to the following conclusions:
 WLR
 The ESAC WG agreed that the resubmitted re-analysis data
were satisfactory with regard to answering the question towhat extent the non-qualified test results might haveinfluenced the WLR analysis. The impact was felt to be
negligible because in case 2 (the most stringent criteria) only
3 individual laboratory predictions had not qualified. The
data were found sufficient to judge WLR.
 BLR
 Agreement was reached regarding the deletion of the fourthcolumn in the Word document as well as the corresponding
columns in the excel files. These columns were intended toallow analysis of concordance of predictions on the basis offour laboratories only instead of the five laboratories that
had participated in the ring trial. The rationale for thiscolumn was that with an increasing number of labs it may be
more likely to get non-concordant results. Givaudan, whenplanning the ring trial had been unaware that it is common
practice in the context of validation to use three laboratories
within a ring trial for assessing transferability and betweenlab reproducibility (BLR). However, as this analysis had notbeen properly conducted in the resubmitted data package,
the ESAC WG felt that this approach should not be followedup when finalising the ESAC review.
 No agreement was reached with respect to the question
whether or not the data on BLR (when taking non-qualified
laboratory predictions into account: case 2 and 3) were
sufficient to judge reproducibility between laboratories. The
reason for this uncertainty lies with one of the principal
flaws of the planning and execution of the study, i.e. that the
Test Acceptance Criteria (TACs) provided to the participating
laboratories during the ring trial (a) had not been applied
when analysing the data (a variation of the initial TACs had
been employed) as they were found too stringent, (b) thatno provisions for re-testing had been made in case test
results would not meet the TACs and (c) that, as aconsequence, the final data matrix contained an appreciable
number of non-qualified test results that were included in
the analysis.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 18 of 77

Figure 2: Test design.

a) Replicate
level

Replicates 3x(=in triplicate)

Parallel
MTT
plate

Averaging ofparameter
Preliminary
predictionb) Run level(repetition level)(here “repetition”)
acc. to averaging(SOP)”Repetition1” =’rep’c) Final test outcome=prediction using PM

Replicates 3x(=in triplicate)

Parallel
MTT
plate

Preliminary
prediction(repetition level)
acc. to averaging(SOP)”Repetition2” =’rep’

Replicates 3x(=in triplicate)

Parallel
MTT
plate

Preliminary
prediction(repetition level)
acc. to averaging(SOP)”Repetition3” =’rep’

Test Outcome (TOC)
using prediction
model: final decision
based on mode ofrepetitions

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 19 of 77

Table 2: Acceptance criteria

Criteria Set found by WG and communicated in the
document "Final Clarifications…" to Givaudan by the
ESAC Sec. (sent to Givaudan 8.2.2012)
Criteria Set 1 (page 7 of submission).

Criteria Set 2 (Att. 7a SOP of ring trial):
contained in the ring trial SOP but neither used byparticipating labs nor by lead lab for final analysis. This set
should be ignored for the purposed of the review.
Criteria Set 3 (page 9 of submission)

Criteria Set as described in the resubmission from
Givaudan (received by ECVAM 14.3.2012)
Criteria Set 1 (page 7 of submission)
Additional clarification (in blue) reg. condition 1: CA
positive in the range 4-64uM.
NOTE ESAC SEC: These criteria (and not Criteria 2
communicated in SOP of ring trial) were used to analyse
the ring trial data. However, some ring trial data wereinvalid when applying these criteria. As no rules forretesting in case of unfulfilled TACs had been stipulated
beforehand, this led to some non-qualified test results
being included in the analysis contained in the full
submission to ECVAM. Having realised that non-qualified
test results had been included in the analysis on
reproducibility and predictive capacity, the ESAC WG
requested (on 8/2/2012) a reanalysis of the data using (a)concordance of predictions as a measure for reliabilityand (b) performing this analysis applying or not-applying
the specified test acceptance criteria.

Criteria Set 2
Additional clarification (in blue) reg. condition 1: CA
positive in the range 4-64uM
NOTE ESAC SEC: These are the test acceptance criteria as
currently used by Givaudan and as recommended for
future use.

Criteria Set 4 (page 17 of Invittox protocol):This set is identical to set 3, apart from a minor typo (7.5
instead of 7.0 uM in condition 2). This set should be
ignored for the purposes of the review.

For more detailed information, please consult Annexes 1-4.
Specific comments:
The methodology for assessing the within and between laboratory variability from a quantitative
perspective is difficult to follow. There are two separate sections dealing with apparently different
statistical analyses of the same data. Attachments 10.a. to 10.f. There is also a statistical analysis ofdata in Attachment 4c plus graphs in 4c.
It seems that ECVAM’s internal statistician had expressed difficulties in understanding the analyses
provided using the geometric means for assessing within and between laboratory variability using
geometric means and requested a review of the analysis by the test developer. One issue was about
the appropriateness of the use of the geometric standard deviation for the assessment. It seems thatthe developer responded by providing another analysis of the data set using a different methodology.
There is very little discussion of the comparison between the two methods but the developer
suggests that the two analyses provide compatible conclusions (and seems to confirm this in a
separate note sent after a teleconference with the panel.)
ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 20 of 77

Each experiment consists of three repeats of three replicates. The description of the hierarchicaldesign and interpretation of sources of variability in the attachment 10.f is difficult to follow or to see
how it is used.
It’s not completely clear if the n’s for means and, particularly, SDs are based upon 3 or 9 measures.
They are probably based upon 3 values. This remains to be clarified.
The discussion about the estimates of variability measured by SDs especially by the use of geometricmeans and log2 transformations is difficult to follow. It is not clear completely clear what specificpoints are being made or how the values are interpreted.
It is not completely clear why the analysis of variability is based upon data transformed to the log of
2. The choice of a log transformation to the base 2 seems to be to try to relate the variability seen to
the log2 dose spacing on the plates. It is not completely obvious that this transformation can bejustified on statistical grounds. (This was, presumably, one of the reasons the ECVAM statistician
wanted an ‘independent’ review of the data.).
The relationship described between the geometric standard deviation and the data expressed as log
to base2 and to a 95% confidence interval needs to be clarified and expressed more clearly.
Is there evidence that the log transformation is an appropriate one? Is there a possibility that thetransformation is too ‘fierce’? The untransformed data may result in higher variability being
associated with higher means (i.e. means and SDs positively correlated). The log transformation may
make the lower means appear to be more variable (i.e. means and SDs negatively correlated). The
figure at the end of attachment 3 shows the untransformed data are approximately normal but witha long right-hand tail. The transformed data, although symmetrical, had ‘fat’ tails with an appreciable
number of very low and very high values.
Some of the summary statistics have been checked by hand by the panel and the relationshipbetween the test results and the summary statistics described in the text have been confirmed.
However, the description of the precise methods used to produce the table in attachment 4b are
currently very difficult to follow and it is hard to relate the tables of the test results with the table ofthe summary statistics.
When the term % variance is used it presumably relates to the coefficient of variance (CV) measured
as (SD/Mean) x 100.
It was not completely clear if CVs based upon logarithmically transformed data were produced. It is
possibly that this was not done: but the text is unclear. This approach is, though, not usuallyconsidered an appropriate way of handling data. Problems arise if the log transformation results in azero or negative value. (For log10 then log(X) =0 when X =1 and is negative when X is between 0 and
1 so with transformed data the CV can apparently be negative).
Attachment 10f
Attachment 10f discusses the statistical analysis of the within and between laboratory variability.
There is mention of analysis of variance. It is not clear if this related directly to an ANOVA carried out
on the data or a more general discussion on how to handle variability.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 21 of 77

It appears to be based upon some sort of ANOVA methodology. It is not clear how the data wereanalysed. Were transformed data used? (In a separate note the reviewer says no.) Variancecomponents are specifically mentioned which seems to imply that some sort of ANOVA is used. The
specific ANOVA and software should be provided.
It is not clear what the term in the report ‘parameter’ refers to.
It is not clear how the variances excluding a laboratory are compared with the variances including alllaboratories. What statistical test is used to show the derived ratio is statistically significant(P<0.0001) which leads to Laboratory 1 being termed the ‘worst performing laboratory’.
The case is made that laboratory 1 had a poorer quantitative (but not qualitative) performance (i.e.
more variable than the other laboratories) due to less opportunity to run the method.
How do the two tables in 10f relate to one another? What are the numbers in these tables (there are
no legends)?
Statistical analysis of single plate data
The statistical methods that are used to identify statistically significant differences are not clearly
defined. It appears that t-tests have been carried out for different concentrations against thenegative control wells.
The decision criteria for a positive result (a significant difference in a t-test AND a fold change) isbased upon relative (fold) difference rather than an absolute (because of variability in the Imax
value) difference. Could this be an issue? The 1.5-fold concentration is derived by ‘linear
interpolation of the value above and below the threshold’.
Diagnostic statistics
Cooper statistics (standard statistics used in diagnostic tests) are presented. In practice, these convey
limited information given the relatively small n values. Good statistical practice would have thempresented with confidence intervals.
Experimental design
The 96 plate design is susceptible to biases in the allocation of the test articles. For instance, there
may also be edge effects or other localized effects on the plates. The current design may introduce
bias and increase variability.
It is suggested to look at Nature Biotechnology paper on plate design (Nathalie Malo, James A Hanley,
Sonia Cerquozzi, Jerry Pelletier & Robert Nadon (2006) Statistical practice in high-throughput
screening data analysis. Nature Biotechnology 24, 167 – 175.
Handling of censored data
The method for handling censored data (e.g. >2000) seems to be inconsistent. In some cases themean is calculated including the censored data, in some cases excluding it. Including or excluding

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 22 of 77

censored data will affect the size of the standard deviation and consequently the CV. The choice ofinclusion or exclusion seems to be subjective.
Negative control data
1.8 Variability of 20% presumably relates to a CV of 20%.
Other points
The data set for 2-EHA is interesting in terms of reproducibility, effect of cytotoxicity. Rep2 forexperiment 1 just reaches a fold increase of 1.5. Visually, it is the most different pattern of the 9
replicate experiments. Four out of 9 experiments seem to be reported as negative because ofcytotoxicity.
The reviewing statistician had not been involved in study design and original study evaluation, but
was from Givaudan the developer of the test method and is therefore not independent.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 23 of 77

3. Test definition (Module 1)

3.1 Quality and completeness of the overall test definition

General comments:
Overall , the test definition is complete and clearly outlined: the scientific background on which the test procedure is based; the intended purpose of the test as well as the need for the suggested test;
 the technical details of test procedure:o description of the test system;o parameters and endpoints measured;o quality criteria;o definition of positive and negative controls as well as benchmarks;o acceptance criteria applied to the results.

Specific comments:
An important limitation in the test definition is that during the prevalidation process acceptancecriteria drifted, resulting in 3 sets of data (See section 2.5, p14).
In parallel cytotoxicity was assessed and expressed as an IC 50 value (MTT assay), but this was not
specifically mentioned (See section 2.5, p14).
As the transfected cell line expresses a certain luciferase luminescence the fold increase over thisbackground was assessed. This is determined for the full dose response curve ranging from 0,98 µM
to 2000µM. From this curve an EC1.5 value was determined (gene induction reaching 50% over the
background).
Induction relative to DMSO background in each well was measured. The experiment was accepted
when the coefficient of variation for the DMSO background calculated from 3 triplicate plates in eachtest (18 x 96 well plates) for repeated measurements was below 20%.
Cinnamic aldehyde also must be positive in each accepted test.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 24 of 77

3.2 Quality of the background provided concerning the purpose of the test method

General comments:
The provided background information is adequate, clearly formulated and justified by the dossier(see Module 1: Test definition) (See also sections 2.2 and 2.3 of this report).

3.3 Quality of the documentation and completeness of SOPs and prediction models

General comments:
As submitted, the documentation (including the SOP) contained a number of ambiguities withrespect to how the KeratinoSens test was to be performed and how the final prediction was derived.
The WG requested more information from Givaudan for clarification of the observed issues (For a list,
see section 2.5, p14). On the basis of the new information and after update of the submitted material
by Givaudan, the WG concluded that the schematic outline (Fig. 2) was clear, and that the it wastransparent which acceptance criteria Givaudan recommends for future use of the assay.
Specific comments:
The raw data entering the processing with the Excel template should have been added for all
measurements not just for 2 examples, so that a complete reconstruction of data provided was
possible.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 25 of 77

4. Data quality

4.1 Overall quality of the evaluated data

General comments:
The data quality was deemed adequate for the purpose of this study. However, a number oflimitations should be listed:The participating laboratories were not given a list with requirements defining theacceptability of the data produced .
There was no quality check of the incoming data, except in the transfer phase, and the checkdiffered from those in the SOP.
Test data for which the test acceptance criteria were not met were included. According to
the submitter this was for the purpose of assessing the rigidity of the test acceptance criteria.
For the purpose of demonstrating reproducibility these data should have been excluded
and/or repeated.

4.2 Sufficiency of the evaluated data in view of the study objective

General comments:
The quality of the data provided was deemed sufficient to judge WLR. The WG did not reach an
agreement regarding the question of whether or not the data on BLR (when taking non-qualified
laboratory predictions into account: case 2 and 3) were sufficient to judge reproducibility between
laboratories. The reason for this lack of agreement is explained in section 2.5 (p. 15).

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 26 of 77

4.3 Quality of the reference data for evaluating reliability and relevance 1

General comments:
The quality of the reference data is considered to be sound (see also section 1). The reference data
selected only allowed assessment of sensitizers versus non-sensitizers (S/NS) not an assessment ofpotency (quantitative responses). The latter is based upon the observations that the test underscoresmoderate and weak sensitizers (See Module 5).

1

OECD guidance document Nr. 34 on validation defines relevance as follows: "Description of relationship of thetest to the effect of interest and whether it is meaningful and useful for a particular purpose. It is the extent towhich the test correctly measures or predicts the biological effect of interest. Relevance incorporates
consideration of accuracy (concordance) of a test method."
ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 27 of 77

5. Test materials
5.1 Sufficiency of the number of evaluated test items in view of the study objective
General comments:
Overall, 114 chemicals were tested. These represented a good number of materials, reasonablestructural diversity and a variety of sensitising potency classes.
The number of test items was considered adequate to draw conclusions about the transferability(N=7) and reproducibility (N=21) of the test.
The small number of non-sensitizers (N=4) in the extended list of chemicals (N=48) for assessing thepredictive capacity of the test was considered too low. The 67 chemicals used for development,
refinement and evaluation of the test were not taken into consideration in assessing the predictive
capacity by the WG.

5.2 Representativeness of the test items with respect to applicability
General comments:
The chemicals tested span a range of molecular weights (30-388 Da), and cLogP (-4.8-5.2), cover thefull range of skin sensitizer potency (weak-extreme) and included a wide array of structural classes.
Pre- and pro-haptens were included. Therefore, the selection of chemicals was sufficient to gain
information on the applicability domain and limitation of the test method.
The results indicate that the KeratinoSens assay frequently underscores moderate as well as weaksensitizers, i.e. there is a high false negative rate (Tables 3-5).
Specific comments:
It would have been very useful to have added reactivity data to this data set for all the chemicals(including the origin of the data). This is mainly to substantiate claims about the applicability domain
being specific to cysteine reactivity of the sensitzsers. For example, there are materials in this data
set which are positive in the KeratinoSens assay are lysine but not cysteine reactive(phenylacetaldehyde, dihydroeugenol, hexylcinnamic aldehyde, hydroxycitronellal, imidazolidinylurea, benzaldehyde). Furthermore, eugenol and phenyl benzoate are cysteine reactive but negativein the KeratinoSens assay.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 28 of 77

Table 3:
Summary results of KERATINOSENS Extended List by chemicals of known skin sensitization potency
SENSITIZATION CLASSa

NUMBER OF CHEMICALS TESTED

Luciferase EC1.5

No induction% false negative

Extreme/strong

27

23

4

14.4

Moderate

30

21

9

30.0

Weak

26

20

6

23.1

Non classified

3

2

1% false positive

Non sensitizers

28

Total sensitizers

83 (+ 3 non classified)Total non sensitizers

28

Total compounds

114

5

23

21.7

testeda

Sensitization class is based on LLNA EC3 values.

Table 4:
Summary results of KERATINOSENS Silver list by chemicals of known skin sensitization potency

SENSITIZATIONNUMBER OF CHEMICALS TESTED

Luciferase EC1.5

No induction% false negative

Extreme/strong

12

11

1

8.3

Moderate

16

13

3

18.8

Weak

16

14

2

12.5

CLASSa% false positive
Non sensitizers

23

Total sensitizers

44

Total non sensitizers

23

Total compounds

67

4

19

17.4

testeda

Sensitization class is based on LLNA EC3 values.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 29 of 77

Table 5.
Summary results of KERATINOSENS Additional chemicals by chemicals of known skin sensitization
potencySENSITIZATION CLASSa

NUMBER OF CHEMICALS TESTED

Luciferase EC1.5

No induction% false negative

Extreme/strong

15

13

2

20.0

Moderate

13

7

6

46.2

Weak

12

8

4

33.3

Non classified

3

2

1% false positive

Non sensitizers

5

Total sensitizers

40 (+ 3 non classified)Total non sensitizers

4

Total compounds tested

48

1

4

20

a

Sensitization class is based on LLNA EC3 values.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 30 of 77

Figure 3: Representation of potency classes in the various studies.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 31 of 77

In the following page a graphic representation of the three tables is presented (Fig, 3). For sensitizers
the red bar represents chemicals misclassified. The percent of chemicals misclassified (false negative
for sensitizers, false positive for non sensitizers) is reported on the top of each bar.
Regarding the correlation between in vitro EC1.5 and in vivo LLNA EC3, the following values arereported (Fig. 4):

Figure 4: Correlation between in vitro EC1.5 and in vivo LLNA EC3.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 32 of 77

6. Within-laboratory reproducibility (Module 2)

6.1 Assessment of repeatability and reproducibility in the same laboratory

General comments:
The WLR assessment was presented by the submitters using two studies and types of analyses:
1) Analysis of variability of the measured parameters (e.g. EC1.5, IC50, Imax) (N=28)
The within-laboratory reproducibility (WLR) was initially assessed by running a ring-trial using 28
chemicals selected from the same list (Silver list, N=67) that had provided the chemicals for
optimising the test (e.g. the prediction model) by the lead laboratory (Casati et al.).
With regard to this initial assessment, the WG believes that it was the intention of the test
submitters to analyse WLR on the basis of the test values including EC15, IC50 and Imax (see section
2.5). As chemicals were tested only once (=1 experiment) in triplicates, rather than in threeconsecutive experiments, and considerable variation between the triplicate values within anexperiment were observed, the test values could not, and were not, used to assess test
reproducibility.
The correlation between in vitro EC1.5 and in vivo LLNA EC3 was o.43 (CI 0.20-0.60) was reported(Fig. 4).
The WG compared the predictions obtained by the lead laboratory during this ring trial withhistorical data on these 28 chemicals generated by the lead laboratory (when testing 67 chemicals of
the silver list (attachment 2)). When comparing these sets of predictions the apparent concordancein predictivity for the 28 chemicals was 96%. It should be noted though that such comparative
analysis of data is not good practice for the purpose of determining WLR within the context of a(pre)validation study.
2) Analysis of concordance of prediction (N=14)
An additional series of 14 chemicals was run in the lead laboratory, including eight coded chemicals
supplied by ECVAM (double-blind) and six additional chemicals selected by Givaudan. These six
chemicals had not been previously tested and were blinded only to the operator. The WG considers
this study as the primary source for assessing reproducibility of the test method.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 33 of 77

When considering all available data, concordant predictions were obtained for 12 out of 14
substances over 3 consecutive experiments (85.7% concordance). Non concordant predictions wereobtained for R(+)Limonene and 2-Ethylhexyl acrylate.
The defined acceptance criteria were not met for three substances due to the positive control(cinnamic aldehyde) being slightly out of the specified range. These substances were Beryllium
sulphate, R(+)Limonene and 4-amino benzoic acid.
Additional input requested: To understand better the impact of unqualified data on the prediction interms of S/NS, the test submitter was requested (in writing) to re-assess the reproducibility of thetest by redoing the calculations only on the basis of test values that qualified according to either setof criteria (Annex 4, p60). The updated calculations were submitted (Annex 4, p61 (C. 1)).

6.2 Conclusion on within-laboratory reproducibility as assessed by the study

General comments:
Concordant results were obtained for 12/14 chemicals (85.7%) when all available data were included.
The WG endorsed the conclusion of the VMG that the test is reproducible within laboratories. WG
considered this concordance in agreement with target values (85%) for WLR performance standards
as published in international accepted guidelines (e.g. Performance standards of TG439 in vitro skinirritation).
The ESAC WG agreed that the re-analysis that was submitted upon request (see section 6.1)
satisfactory answered the question to which extent non-qualified test results might have influenced
the WLR analysis. The impact on WLR was considered negligible as even under the most stringent
criteria (set 2 in Annex 4, p60) only 3 individual laboratory predictions had not qualified.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 34 of 77

7. Transferability (Module 3)

7.1 Quality of design and analysis of the transfer phase

General comments:
In general, the transferability assessment of the test was well-designed, e.g. preparation of an SOP,
use of the same batches of seven chemicals, a rationale for chemical selection, and assurance criteriaof data generated from external laboratories.
Even though potential problems affecting test performance were anticipated, no face-to-facetraining was performed as the test submitter did not deem this necessary.
Specific comments:
The SOP (Attachment 7a, 2009-17-07 version) used for transferability is a different version, eventhough very similar to the INVITTOX protocol KeratinoSens (last up-dated at 2010-20-08; Attachment
1). There is no description of the differences included in the report, nor is it explained why a newversion was preferred.
Data quality (Attachment 8a & 8b) from external laboratories was assessed by the lead laboratory.
Assessment was on the basis of the ‘DMSO wells’ variability, dose-response curve reproducibility
within the laboratory, and significant induction by the positive control cinnamic aldehyde. Even
though the criteria for acceptance of ‘external’ data were explained, no detailed assessment
procedure (e.g. statistical analyses) was given.
Additional input requested: To understand better the impact of unqualified data on the prediction interms of S/NS, the test submitter was requested (in writing) to re-assess the reproducibility of thetest by redoing the calculations on the basis of test values that qualified according to either set ofcriteria (Annex 4, p60). The updated calculations were submitted (Annex 4, p62 (C. 3)).

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 35 of 77

7.2 Conclusion on transferability to a second laboratory as assessed by the study

General comments:
The conclusion on transferability was justified on the basis of concordant predictions (S/NS) between
the lead laboratory and the naive laboratories. The WG endorses the conclusion that the test method
can be transferred to naive laboratories that are experienced with cell culture techniques.
Specific comments:
As the SOP (INVITTOX protocol KeratinoSens: Attachment 1) was suggested to be sufficiently detailed
to perform the test, no face-to-face training was organized before transfering the test to naivelaboratories. However, the WG raised concerns about the reliability of luciferase measurements fortransferability. Differences in the brand of luminometer or substrate were demonstrated by the test
submitters not to affect the liability of the luminescence measurement. Based on this fact, it seems
obvious to the WG that the variation observed in luminescence measurements between laboratories
is due to lack of experience, stressing the necessity of operating a number of training experiments inthe naïve laboratory before the test method can be used to identify skin sensitizers.
Variability among the laboratories was observed in the dose-response curve or EC1.5 (Attachment 8a& 8b) for cinnamic aldehyde and ethylene glycol dimethacrylate. No further explanation was
however given whether these variabilities originated from the chemicals’ own physico-chemical
characteristics or from luminescence measurement issues.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 36 of 77

8. Between-laboratory reproducibility (Module 4)

8.1 Assessment of reproducibility in different laboratories

General comments:
As for the WLR, the BLR was assessed on the basis of concordance between results, accuracy anddata dispersion by comparing geometric standard deviation of both EC1.5 and IC50 values. In total 21
chemicals were tested.
The BRL was assessed by a ring-trial involving 4 naive laboratories and the lead laboratory.
Additional input requested: To understand better the impact of unqualified data on the prediction interms of S/NS, the test submitter was requested (in writing) to re-assess the reproducibility of thetest by redoing the calculations only on the basis of test values that qualified according to either setof criteria (Annex 4, p60). The updated calculations were submitted (Annex 4, p61 (C. 2)).
Specific comments:
It was not clear to the WG how/where the coding was performed and how ‘blinding’ was assured.
If the question is one of criteria reproducibility, then this is not very good when looking at data fromlaboratories 2, 3 and 4. If the question focuses on concordance, reproducibility is better.
 Case 1 (all data included): the concordance is very good (85.7 %).
 Case 2 (only qualified predictions based on Criteria set 1): The reproducibility is not
acceptable due to poor performance of laboratories 2, 3 and also 4. Considering only thesubstances for which at least 3 laboratories produce acceptable data (N = 17) the
concordance is 88%.
 Case 3 (only qualified predictions based on Criteria set 2): The concordance is acceptable
for laboratories 1, 4 and 5 (> 85%) (N = 21). Too many unqualified runs are observed withlaboratories 2 and 3.
The above indicates that perhaps a formal training phase would have been useful after all (See also
section 7.2).
Givaudan, when planning the ring trial had been unaware that it is common practice in the context ofvalidation to use three laboratories within a ring trial for assessing transferability and between-lab
reproducibility (BLR). While it was acknowledged by the WG that with an increasing number oflaboratories it may be more likely to get non-concordant results, the WG agreed not to consider the

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 37 of 77

analysis by the test submitter of the concordance of predictions on the basis of four laboratories only
instead of the five laboratories that had participated in the ring trial.

8.2 Conclusion on reproducibility as assessed by the study

General comments:
The S/NS prediction gave concordant results for the majority of chemicals (85.7 – 90.5%), taking into
consideration the explanations give for the outliers, also between laboratories. (See section 6.1).
The test acceptance criteria provided to the participating laboratories during the ring trial had notbeen applied when analysing the data. The reason for this inconsistency was that the criteria were
found to be too stringent. In contrast with WLR and transferability assessment, these non-qualified
data had an effect on the concordance of predictions (Annex 4, p62 (C. 2)).
No provisions were made for re-testing in the case of nonqualified predictions.
Specific comments:
It was noted by the WG that an attempt was made to assess the BRL using 5 laboratories. It is
acknowledged that ECVAM requires only ‘at least’ 3 laboratories participating in a prevalidation
study. It is also acknowledged that the test developer involved 5 laboratories thereby raising the bar
significantly. However, it is not appropriate to choose the best 3 of 5 where the data is best. For the
purposes of reproducibility assessment one may consider all 5 labs or choose 3 laboratories upfront
to compare.
Eleven out of 15 rated positive in all 5 laboratories, while 4 out of 6 non sensitizers were correctlyclassified. Contradictory results were obtained for Eugenol (S in 2 laboratories, and NS in 3
laboratories). The explanation given for the Eugenol reactivity should be reassessed.
The irritants diethylphthalate and SDS were positive in one laboratory. Phenylbenzoate was a clear
false negative.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 38 of 77

9. Predictive capacity (Module 5)

9.1 Adequacy of the assessment of the predictive capacity in view of the purpose

General comments:
The WG was impressed by the wealth of information provided by the test submitter on the 113
chemicals assessed in this study. Based upon the 113 chemicals included in the study, the predicitive
capacity of the KeratinoSens assay was 78%.
The 114 chemicals include the 67 chemicals of the Silver list. Including chemicals that were used fordevelopment, refinement and evaluation of a test system might induce a bias in the assessment of
the predictive capacity and was therefore considered by the WG as a limitation.
Considering only the new chemicals (43 sensitizers and 3 non-sensitizers), the calculations showedthat the predictive capacity (69%) was considerably lower than the 78% presented by the submitter.
It was noted that the number of new qualified non-sensitizers used in this study was consideredinsufficient (N = 3).
Additional input requested: The submitters were requested to submit additional data on chemicals
with negative LLNA reference data. Such data were provided for an additional 80 chemicals.
Compiling all the data provided by the submitters (N = 220), the KeratinoSens assay revealed asensitivity, specificity and accuracy of 79.3%, 79.8% and 79.5%. Omitting the reactive peptide
alkylating chemicals, for which the LLNA data were not trusted and no human data were available,
the remaining 213 chemicals resulted in a sensitivity, specificity and accuracy of 79.3%, 84.5% and81.7% (Annex 4, p64 (C8)).
The WG observed a poor performance of the test with weak sensitizers. Based on the predictions
using the 113 chemicals, 41% of the weak and 86% of the very weak sensitizers were missed (Table
6). Furthermore, the frequency false negative results were found to increase with decreasing potencyof the test chemical. This limitation is not indicated clearly in the submission.
Table 6 - Distribution of the predictive capacity over the potency classes
SENSITIZATION CLASS
ExtremeStrongModerate
Weak
Very weak/none

N° of chemicals
6
12
39
27
7

Luciferase induction(positive)
6
10
34
16
1

No luciferase induction(negative)
0
2
5
11
6

% false negative withineach class
0
17
13
41
86

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 39 of 77

Specific comments:
The test submitters assessed the predicitive capacity of the test in three steps. Based on thecalculations the predictive capacity ranged between 96.7 and 78%. However, for each step carefulconsiderations have to be made.


Step 1:
Predicitive capacity based on the screening in the lead laboratory of 67 chemicals (Silver list)
was 85.1%, but chemicals were used for development, refinement and evaluation of the test.

Step 2:
Predictive capacity based on the screening of the 28 chemicals of the Ring study was 85.496.7%, but the 28 chemicals were a subset of the Silver list chemicals.



Step 3:
Predictive capacity based on the entire list of 114 chemicals was 78%, but only 46 newchemicals were included.

It is interesting to note that similar figures can be obtained using the AREc32 cell line (a stable humanbreast carcinoma cell line transfected with an ARE element) (Natsch and Emter, Tox Sc 102, 2008).

9.2 Overall relevance (biological relevance and accuracy) of the test method in view of the
purposeGeneral comments:
While the presented calculation of the predicitive capacity of the KeratinoSens was criticized, the WG
is confident that the test can identify strong and moderate sensitizers. However, the WG is lessconfident that this test can identify weak sensitizers and possibly also moderate sensitizers at thelower end of the scale.
The accuracy, sensitvity and specificity appears acceptable. If we consider all the chemicals tested,
the overall accuracy is 79.5%, with a sensitivity of 79.3% and a specificity of 79.8, which can beconsidered acceptable.
However, the criticism and the necessity to underlie limitations in term of identification of weak andmoderate sensitizers still remain.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 40 of 77

10. Applicability domain (Module 6)

10.1 Appropriateness of study design to conclude on applicability domain, limitations andexclusions

General comments:
The applicability domain was described in the section 1.6 of KeratinoSens report. The authors stateda variety of chemical classes which were expected to be successfully tested in the KeratinoSens
assay. The exclusions were mainly related to issues of solubility or stability in vehicle (e.g.
interactions with the vehicle, such as hydrolysis).
The WG discussed this issue (See section 2.2) and came to the conclusion that there is indirect
evidence that the applicability domain of the test may extent to chemicals that not (only) react withthe cysteine residues of Keap1. Alternative mechanisms may lead to Nrf2 activiation.
The study design allowed testing of some of the limitations of the applicability domain.

10.2 Quality of the description of applicability domain, limitations, exclusions

General comments:
The anticipated applicability domain of the KeratinoSens assay is described in the section 1.6.3.
(KeratinoSens report). The clearly explained limitations are for chemicals with extremes of cLogP
which would not be testable in this assay due to poor solubility. Another clear limitation adequately
explained is related to the stability of the chemical in the solvent system (DMSO, water), resulting inhydrolysis or interaction with DMSO, thus resulting in a change of the test material throughpreparation. These limitations are common to many in vitro/in chemico test methods.
The applicability domain which is less well described concerns chemical reactivity. The consensus
opinion on reactivity is unclear. Reactivity is described in section 1.1.2 as covalent modification of the
key cysteine residues on Keap1, resulting in activation of the Nrf2 signalling pathway,. From theliterature it is clear however, that this pathway can be activated by other means – such as oxidationof the key Cys residues on Keap1 as well as other type of modification of Keap1 Cysteines (e.g.
glutathionylation, see Holland et al 2008). This is potentially the reason why some of the exclusively
Lys reactive residues can be positive in KeratinoSens assay: whilst such chemicals generate adductswith Lys and not Cys residue, they can potentially oxidise those cysteines generating the same effect.
The argument about exclusivity of Lys reactivity of some chemicals is not always applicable, as some
ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 41 of 77

of the materials quoted as lysine reactive are also cysteine reactive, and therefore should be positivein this assay (such as phenyl benzoate, see Aleksic et al 2009). There are also exclusively Lys reactive
chemicals which are positive in this assay.
The suggested inclusion of the peptide reactivity assay with KeratinoSens in an integrated testing
strategy is a good one, providing that the peptide reactivity does make a distinction between lysine
and cysteine reactivity. The assay suggested in the attachment 12c of the TST does not provide this
distinction.
The added value of the cell based assay over a reactivity assays would be a possibility of somemetabolic activation being available to capture the true prohaptens, which would not be reactive in
direct peptide reactivity assays. The activation mechanisms of this particular cell line appear to bepoor.
In conclusion, the stated applicability domain is likely to be broader than that stated in the TST.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 42 of 77

11. Performance standards (Module 7)

11.1 Adequacy of the proposed Essential Test Method Components
Not relevant

11.2 Adequacy of the Reference Chemicals
Not relevant

12. Readiness for standardised use

12.1 Assessment of the readiness for regulatory purposes

General comments:
The WG considered the test method sufficiently mature for classification and labelling of chemicals(relevant to Regulation EC N° 1271/2008).
Negative results however have to be considered with care as weak sensitizers (and possibly also
moderate sensitizers at the lower end of the scale) will be probably missed (see section 9).
Unless this issue gets resolved, the KeratinoSens assay has to be seen as one brick in an intergrated
testing strategy of weight-of-evidence approach. The consideration of the chemistry /reactivity must
be included either by combination with a peptide reactivity test or predictive chemistry assessment.
This reactivity assessment should include consideration concerning activating mechanism(s).
Specific comments:
The performance of the Keratinosens assay extended list is acceptable (Specificity = 82.1%; Sensitivity= 76.7%), if a specificity = 80% and sensitivity = 70% is the accepted criteria.
Furthermore, the possibility to use the EC1.5 values for sensitization potency classification is unlikely
from the data presented.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 43 of 77

It is likely that the combined analysis of this and/or other biomarkers rather than the analysis of asingle biomarker will give even more satisfactory results. It is anticipated that the combination ofdifferent in vitro assays will increase the accuracy, i.e., in the KeratinoSens assay, the inclusion of thepeptide reactivity data will increase accuracy from 85.1% to 89.6% (Emter et al., 2010). The h-CLAT
has an accuracy of 75.9% with CD86 alone, while 93.1% in combination with CD54 (Sakaguchi et al.,2009).
It is, however, important to note that the combination of the KeratinoSens with the DPRA increases
sensitivity (correct identification of positive compounds) to 84.9% but its specificity will decrease to78.6%. The rate of false positive will increase.

12.2. Assessment of the readiness for other uses
General comments:
The KeratinoSens assay was considered useful for screening purposes, to identify molecular initiatorsand to gain mechanistic information on the role of e.g. oxidative stress in sensitization.

12.3 Critical aspects impacting on standardised use

General comments:
Since the applicability domain of the test is still not fully defined, and probably is not limited to direct
reactivity of a chemical with key cysteine residues in Keap1, it can be anticipated that chemicals are
excluded from testing for the wrong reason.
The dependence of the predictive capacity of the test on decreasing potency has to be consideredwith care (See section 9).
The WG was concerned about an intellectual property (IP) issue, related to the use of the luciferase
gene from Promega. A license from Promega is required for commercial uses. This may haveeconomic consequences (attachment 1, p22).
The use of the Promega luciferase gene in the KeratinoSens cell line is still linked to the assay
substrate from Promega. A question is whether it is mandatory to use Promega’s substrate, orwhether one purchase a similar substrate from other companies?
The 96 plate design as submitted was susceptible to biases in the allocation of the test articles. Theremay be edge effects resulting in bias and increased variability.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 44 of 77

12.4 Gap analysis

General comments:
Weak and low-moderate sensitizers, as well as pro-haptens performed poorly.
When considering cytotoxicity, more emphasis could have been given to the GSH status of the cells
and their GSH regenerating capacity. This system may have an impact on the inherent chemical
reactivity whether directly conjugating to GSH or oxidising it.
The data do not support the expectation that this test can be used as a stand-alone test (preliminary,
waiting for PC and reproducibility assessment).
The correlation between in vivo and in vitro data is weak because there was a relative high variability
among the in vitro scores of chemicals belonging to the same potency class (Natsch et al., 2009).

13. Other considerations
The report mentioned the high power of the study design. Some indication of the size of effects thatcan be reasonably detected would be useful.
The chemicals selected from the silver list (N=28) for reproducibility assessment were also used fortest development, refinement and evaluation. This may induce a bias because the test may havebeen optimized for these chemicals.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 45 of 77

14. Conclusions on the study

14.1 Summary of the results and conclusions of the study

14.1.1 Test items
Overall, 114 chemicals representing structural diversity and a variety of sensitising potency classes
were tested. The chemicals tested span over a range of molecular weights (30-388 Da), of cLogP (4.8-5.2), cover the full range of skin sensitizer potency (weak-extreme) and widely differing structuralclasses were tested. Pre- and pro-haptens were included as well.

14.1.2 Summary of study results
WLR (14 chemicals):
Concordant results were obtained for 12/14 chemicals (85.7%) when including the available data.
Transferability (7 chemicals):
The conclusion on transferability was that the test method can be transferred to naive laboratories
that are experienced with cell culture techniques.
BLR (21 chemicals):
Among the 21 chemicals tested, 11 out of 15 were called positive in all 5 laboratories, while 4 out of
6 non sensitizers were correctly classified. Some irritants were called positive in one laboratory.
The VMG considered the test reproducible in the laboratories involved in the study.
Predictive capacity:
The test submitters assessed the predicitive capacity of the test in three steps. Based on thecalculations the predictive capacity ranged between 78% and 96.7. However, for each step carefulconsiderations have to be made.


Step 1:
Predicitive capacity based on the screening in the lead laboratory of 67 chemicals (Silver list)
was 85.1%, but these chemicals were used for the development, refinement and evaluationof the test.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 46 of 77



Step 2:
Predictive capacity based on the screening of the 28 chemicals in the Ring study was 85.496.7%, but these 28 chemicals were a subset of the Silver list chemicals.



Step 3:
Predictive capacity based on the entire list of 114 chemicals was 78%, but only 46 newchemicals were included.

14.2 Extent to which study conclusions are justified by the study results alone
Test chemicals:
A total of 114 chemicals was tested which represented a good number of materials, reasonablestructural diversity and a variety of sensitising potency classes. The chemicals tested extended over arange of molecular weights (30-388 Da), of cLogP (-4.8-5.2), cover the full range of skin sensitizer
potency (weak to extreme) and differing structural classes. Pre- and pro-haptens were also included.
Therefore, the selection of chemicals was considered sufficient to gain information on theapplicability domain and limitation of the test method.
The number of test items was considered sufficient to draw conclusions about the transferability(N=7) and reproducibility (N=21) of the test.
The small number of non-sensitizers (N=4) in the extended list of chemicals (N=48) for assessing thepredictive capacity of the test was considered too low. The 67 chemicals used for development,
refinement and evaluation of the test were not taken into consideration for assessing the predictive
capacity by the WG.
WLR (14 chemicals):
When considering all available data including tests that did not meet the specified test acceptancecriteria (unqualified tests), concordant predictions were obtained for 12 out of 14 substances over 3
consecutive experiments (85.7% concordance). For three substances the defined acceptance criteria
were not met. When excluding these non-qualified results from the data matrix, concordant
predictions were obtained for 10 out of 14 chemicals resulting in a concordance of 71.0%.
Including all available data concordant results were obtained for 12/14 chemicals (85.7%). The WG
considers this concordance acceptable and in agreement with target values (85%) for WLR
performance standards as published in international accepted guidelines (e.g. Performancestandards in TG439 in vitro skin irritation).
Transferability (7 chemicals):
In general, the transferability assessment of the test was well-designed, e.g. preparation of an SOP,
use of the same batches of chemicals, rationale for chemical selection, and assurance criteria of data
generated from external laboratories.
ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 47 of 77

Even though potential problems affecting test performance were anticipated, no face-to-facetraining was performed as the test submitter did not deem this necessary.
The conclusion on transferability was justified on the basis of concordant predictions (S/NS) between
the lead laboratory and the naive laboratories. The WG endorses the conclusion that the test method
can be transferred to naive laboratories that are experienced with cell culture techniques.
BLR (21 chemicals):
Among the 21 chemicals tested, 11 of 15 rated positive in all the 5 laboratories, while 4 of 6 nonsensitizers were correctly classified. Some irritants were classed as positive in one laboratory.
The WG discussed in detail the question of whether or not the data on BLR were sufficient to judge
reproducibility between laboratories. As explained in section 2.5, the reason for this discussion was
the observation that the TACs provided to the participating laboratories during the ring trial (a) had
not been applied when analysing the data as they were found too stringent, (b) that no provisions forre-testing had been made and (c) that the final data matrix contained an appreciable number of nonqualified test results that were included in the analysis.
Predictive capacity:
The conclusions regarding the predictivity are sound given the overall value of 76.6%, the key here isthat weight of evidence data were used for comparison rather than to a single assay outcome (suchas LLNA EC3% values).
Compiling all the data (including additional chemicals with negative LLNA) provided by the submitters(N = 220), the KeratinoSens revealed a sensitivity, specificity and accuracy of 79.3%, 79.8% and79.5%. Omission of the reactive, peptide alkylating chemicals, for which the LLNA data were notreliable and where not supported by human data, the remaining 213 chemicals resulted in asensitivity, specificity and accuracy of 79.3%, 84.5% and 81.7% (Annex 4, p64 (C8)).
Negative results cannot, however, exclude the sensitization potential as weak and low moderatesensitizers are likely to be missed.
Applicability domain is less clearly defined with this data set and it is prudent to assess this further by
testing the additional set of chemicals with possible issues. It is clear that specific amine reactivity
and requirement for some form of activation are not the only issues that may need to be addressed.

14.3 Extent to which conclusions are plausible in the context of existing information
A large body of impressive experimental work was carried out using a robust test method. Based on
the information supplied, the conclusions are plausible.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 48 of 77

15. Recommendations
15.1 General recommendations
The test method can be used for S/NS identification of chemicals. Therefore, the test was considered
ready for the next steps in the ECVAM process. A Validation study should however include more welldefined non-sensitizing compounds. Furthermore, a consistent use of proposed acceptance criteria isa necessary prerequisite.
Negative results cannot rule out a sensitization potential because the test revealed issues aroundweak and low moderate sensitizers. This problem needs to be clearly flagged.
On the basis of the data provided, the developer should be advised to evaluate critically the values
obtained in laboratories 2 and 3, and justify any new criteria to be applied during validation of thetest with the clear proviso that any further modification after this to the protocol would raise serious
concerns about the protocol's 'general applicability' and the assay's transferability and reproducibility.
Integration of this assay with other predictive tests when they emerge needs to be based on a better
defined applicability domain.
Combination of KeratinoSens with a reactivity based approach needs to include unambiguous
identification of reactivity and any specificity associated with it. Furthermore, complex reactivity
pathways need to be considered and studied in the context of this assay because some in chemico
derived reactivity data may be an artefact of the test system rather than a true reflection of actual
reactivity of the said chemicals in vivo. With specific reactivity in mind, consideration needs to begiven to other effects which are reactivity-driven in a cell, such as the depletion of GSH via direct
conjugation and chemically-driven GSH oxidation
Training should be considered for laboratories that have no experience with the KeratinoSens test.

15.2 Specific recommendations (e.g. concerning improvement of SOPs)
The 96 plate design is susceptible to biases because of the allocation of the test articles. The test
submitters were recommended to look at Nature Biotechnology paper on plate design (Nathalie
Malo, James A Hanley, Sonia Cerquozzi, Jerry Pelletier & Robert Nadon (2006) Statistical practice inhigh-throughput screening data analysis. Nature Biotechnology 24, 167 – 175.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 49 of 77

16. References
Ade N., Leon F., Pallardy M., Peiffer JL, Kerdine-Romer S., Tissier M.H., Bonnet P.A., Fabre I. andOurlin J.C. (2009) HMOX1 and NQO1 genes are upregulated in response to contact sensitisers in
dendritic cells and THP-1 cell line: role of the Keap1/Nrf2 pathway. Toxicol Sci. 107(2): 451-460
Adler S, Baketter D, Creton S, et al. (2009) Alternative (non-animal) methods for cosmetic testing:current status and future prospects – 2010. Arch Toxicol DOI 10.1007/s00204-011-0693-2.
Aleksic M., Thain E., Roger D., Saib O., Davies M., Li J., Aptula A., Zazzeroni R. (2009) Reactivity
Profiling: Covalent Modification of Single Nucleophile Peptides for Skin Sensitisation Risk Assessment.
Toxicol Sci. 108(2), 401–411
Barrett JC and Ts'o PO (1978) Relationship between somatic mutation and neoplastic transformation.
Proc. Natl. Acad. Sci. USA, 75: 3297-301.
Boverhof D.R. et al (2009) Evaluation of a toxicogenomic approach to the local lymph node assay(LLNA). Toxicol. Sci. 107(2): 427-39
Casati, S.,Aeby, P.,Kimber, I.,Maxwell, G.,Ovigne, J. M.,Roggen, E.,Rovida, C.,Tosti, L.and Basketter, D.,(2009) Selection of chemicals for the development and evaluation of in vitro methods for skinsensitisation testing. Altern Lab Anim 37, 305-12.
Combes R, Balls M, Curren R, Fischbach M, Fusenig N, Kirkland D, Lasne A, Landolph J, LeBoeuf R,
Marquardt H, McCormick J, Mueller L, Rivedal E, Sabbioni E, Tanaka N, Vasseur P and Yamasaki H(1999) Cell transformation assay as predictors of human carcinogenicity. Alter. Lab. Anim., 27: 745-67.
Dinkova-Kostova A.T., Holtzclaw W.D. and Kensler T.W. (2005) The role of Keap1 in cellular protective
responses. Chem. Res. Toxicol. 18: 1779-1791
Emter R., Ellis G. and Natsch A. (2010) Performance of the novel keratrinocyte-based reporter cellline to screen skin sensitisers in vitro. Toxicol. Appl. Pharmacol. 245, 281-290
Holland R., Hawkins A.E., Eggler A.L., Mesecar A.D., Fabris D., Fishbein J.C. (2008) Prospective Type 1
and Type 2 Disulfides of Keap1 Protein. Chem. Res. Toxicol. 2008, 21, 2051–2060
Kim, H. J.,Barajas, B.,Wang, M.and Nel, A. E., (2008) Nrf2 activation by sulforaphane restores the agerelated decrease of T(H)1 immunity: role of dendritic cells. J Allergy Clin Immunol 121, 1255-1261 e7.
Kobayashi M., Li L., Iwamoto N., nakajima-Takagi Y., Kaneko H., Nakayama Y., Eguchi M., Wada Y.,
Kumagai Y., Yamamoto M. (2009) The antioxidant defense system Keap1-Nrf2 comprises a multiple
sensing mechanism for responding to a wide range of chemical compounds. Mol Cell Biol 29(2): 493502

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 50 of 77

Ku, H.O. et al (2008) Analysis of differential gene expression in auricular lymph nodes draining skin
exposed to sensitisers and irritants. Toxicol Lett 177(1):1-9
Lee DF, Kuo HP, Liu M, Chou CK, Xia W, Du Y, Shen J, Chen CT, Huo L, Hsu MC, Li CW, Ding Q, Liao TL,
Lai CC, Lin AC, Chang YH, Tsai SF, Li LY, Hung MC. (2009) KEAP1 E3 ligase-mediated downregulation of
NF-kappaB signaling by targeting IKKbeta. Mol Cell. 2009 Oct 9;36(1):131-40.
Maruyama A. and Itoh K. (2005) The role of Nrf2 in the protection against inflammation and innate
immunity. Hirosaki Med. J. 59: S167-171
Megherbi R., Kiorpelidou E., Foster B., Rowe C., Naisbitt D.J., Goldring C.E. and Park B.K. (2009) Roleof protein haptenation intriggering maturation events in the dendritic cell surrogate cell line THP-1.
Toxicol Appl Pharmacol 238(2): 120-32
Natsch A. (2010) the Nrf2-Keap1-ARE toxicity pathway as a cellular sensor for skin sensitisers –functional relevance and a hypothesis on innate reactions to skin sensitisers. Toxicol. Sci. 113 (2):284-92Natsch A. and Emter R. (2008) Skin sensitisers induce antioxidant response element dependent
genes: application to the in vitro testing of the sensitisation potential of the chemicals. Toxicol. Sci.
102 (1): 110-9.
NTP, NTP website at http://www.ntp-server.niehs.nih.gov.
OECD (2005) Guidance document on the validation and international acceptance of new or updated
test methods for hazard assessment. OECD Series on testing and assessment Nr. 34.
Python F., Goebel C. and Aeby P. (2009) Comparative DNA microarray analysis of human monocyte
derived dendritic cells and MUTZ-3 cells exposed to the moderate skin sensitiser cinnamaldehyde.
Toxicol Appl Pharmacol 239 (3): 273-83

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 51 of 77

17. Annexes
Annex 1Document date:
Author:23.1.2012
Andreas Natsch, Givaudan in vitro toxicology laboratory

KeratinoSens test submission
Response to questions raised by the peer-review panel and forwarded to Givaudan by the ESAC
Coordinator on 16.12.2011:
The following request was obtained:
We would kindly like to request the clarifications listed below:
1) a copy of the full statistical report, including the printouts of the specific analyses
carried out, which was used to produce the Givaudan Statistical Report (attachment
10f). The availability of this full analysis may help with the interpretation of theanalyses of within-and between-laboratory reproducibility. The WG is currently ofthe opinion that the results of this second analysis may not be completely consistent
with those carried out in attachment 7a-10e and those reported in the paper by
Natsch et al (2011) and would therefore like to comprehend the statistical analyses
employed for the purpose of the submission package.
2) clarification as to whether also non qualified tests (=those not meeting theperformance criteria) have been used possibly also in the context ofa) the analysis of the study "further evaluation of the intralaboratory reproducibilityof the KeratinoSens assay to detect skin sensitizers" (attachment 4c).
b) the analysis of the study "further evaluation of the predictivity of the
KeratinoSens assay to detect skin sensitizers" (attachment 12c).
3) submission of amended tables that identify the occurrence of non-qualified tests.
This definitely concerns table 2 of the SOP (ring trial), but possibly also otherstudies/attachments (see point 2).
4) submission of amended analyses of reproducibility and (preliminary) predictive
capacity on the basis of qualified test results only.

Point 1)
The initial statistical analysis, which was also presented in the paper on the ring study, was madedirectly by the Givaudan in vitro toxicology laboratory. This is entirely based on descriptive statistics,
as we felt this is sufficient to display the data and to describe the within and between laboratory
variation of the data (Original attachments 10d and 10e).
Upon review of the data submission by the ECVAM statistician a more detailed statistical evaluation
was requested. The data package was therefore submitted to Dr. Paul Talsma, the Givaudan
statistician who was involved neither in the study design nor in the original study evaluation. He
ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 52 of 77

proposed the additional evaluation based on a statistical analysis of the impact individual labs had on
the overall data variance for the quantitative parameters. This independent approach wassummarized in Attachment 10f and represents an alternative analysis of the quantitative parameters(EC1.5, IC50, Imax) and was thus included in the final submission.
As some inconsistencies were detected by the peer-review panel we have re-checked all the data forthis latter analysis, and indeed found one mistake in the data-transfer to the statistical software forthe parameter IC50, for which, for the lead lab two repetitions and the average were used instead of
the three repetitions (shift in data source by one column). This mistake was corrected and theanalysis re-run. This is updated in the revised file
Attachment10f_Statistical analysis BLR_revised.doc.
The impact of this mistake is small and only affects the IC50 parameter and not the EC1.5 and I max.
The conclusions were not changed as can be seen in this amended document.
As requested we also include the printout of the statistical analysis software of this revised analysis.
This can be found in the file
Attachment10f_Statistical analysis BLR_SAS output.doc.

Point 2) to 4):
General consideration:
Certainly this additional requested analysis and clarifications on the effect of the runs not qualified
by the quantitative criteria for cinnamic aldehyde are very important, in order to clearly evaluate
how sensitive the assay and the conclusions are in regard to these quantitative criteria for the
positive control.
However, we may have complicated things by defining these quantitative criteria for cinnamic
aldehyde before knowing how important they really are. It may have been a better approach to only
define the criterium: ‘Cinnamic aldehyde must be positive in each valid run at the low subtoxic
concentration selected in the SOP’, and then do a post-hoc analysis how the quantitative values ofthe positive control may affect the results.
It is important to keep in mind, that the first criterium, namely that the positive control is positive at
the low, non-cytotoxic range selected for this control (statistically significant induction above the 1.5fold threshold between 4 and 64 µM) was fulfilled in each run used in the analysis in all studies. Thusin all the trials from the ring study laboratories and in all the trials summarized in attachment 4c,cinnamic aldehyde gave a positive result. In addition, as can be seen in attachment 17a, in 188
consecutive runs in the Lead lab, cinnamic aldehyde was always positive at the selected dose-range.
We consider this an important indication of the stability of the assay over time and the robustness of
the assay in general.
Point 2 a) (Attachment 4c):
18 runs were performed for attachment 4c. As indicated in the report, in two runs the criteria were
not fulfilled, but they were just at the borderline:- In Experiment 2, run 2, plate 2, the automatically calculated EC1.5 was at 30.08 instead of a
maximum of 30- In Experiment 3, run 3, plate 1, the induction at 64 µM was at 1.993 instead of 2.0.
Since these two runs were only in the second digit different from the target, they were not repeatedand used for the analysis. Also the dose-response depicted below would clearly indicate these werevalid runs. Nevertheless, we have re-analyzed the data now without these data and the updated test
report is attached as the file:
ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 53 of 77

Attachment 4b_second intralab study_wo_borderline-runs.pdf
In this revised report all calculations were made without these two borderline runs.
Omitting these data had no impact on the analysis of the intralaboratory variability of the EC1.5 andIC50 values. As an example, the geometric standard deviations of the EC 1.5 values with and without
these two runs are shown in Table 1a and Table 1b below.
Omitting these two runs did affect the predictions for three chemicals in Experiment three, as forthese three chemicals we then only have two repetitions and a conclusion cannot be made regarding
their positive or negative rating in this third repetition. These chemicals include Limonene andBeryllium sulfate, which had already been discussed in the previous version of the report asborderline chemicals.

3.00

fold induction

fold induction

3.00

1.50

induction CA Exp 3,
rep3, plate 1
Grand average
0.00
0.00

20.00

40.00

60.00

80.00

1.50

induction CA Exp 2,
rep2, plate 2
Grand average
0.00
0.00

20.00

40.00

60.00

80.00

conc. micromolar

conc. micromolar

Figure 1. The positive control in the two runs, for which the quantitative criteria for cinnamic
aldehyde were not met, compared to the grand average of the positive control in the second WLR
study.

Table 1 a. Geometric standard deviation of the EC1.5 values for the positive chemicals including thetwo runs with borderline results for cinnamic aldehyde
Experiment 1

Experiment 2

Experiment 3

Betweenexperiments

GMP

1.65

1.57

1.13

1.66

GFN

1.25

1.08

1.07

1.13

GCS

1.08

1.02

1.16

1.05

GBT

1.21

1.41

1.45

1.56

DNBS

1.19

1.12

1.19

1.13

2-EHA

1.41

1.33

1.01

1.13

2-PPA

1.15

1.26

1.12

1.09

4-AmC

1.22

1.30

1.24

1.07

1.17

1.23

Average
1.27
1.26
For each Experiment the geometric standard deviation of the three repetitions was calculated
2
Indicates the geometric standard deviation over the geometric means of each experiment
1

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 54 of 77

Table 1 b. Geometric standard deviation of the EC1.5 values for the positive chemicals, excluding
the two runs with borderline results for cinnamic aldehyde
Experiment 1

Experiment 2

Experiment 3

Betweenexperiments

GMP

1.65 1

1.57

1.11

1.67 2)

GFN

1.25

1.08

1.09

1.13

GCS

1.08

1.02

1.20

1.08

1.21

1.41

1.65

1.58

DNBS

1.19

1.08

1.19

1.16

2-EHA

1.41

1.42

1.01

1.15

2-PPA

1.15

1.25

1.12

1.14

4-AmC

1.22

1.43
1.28

1.24
1.20

1.05
1.25

Average

1.27

GBT

Point 2b): (Attachment 12c)
Attachment 12 contains data from the original screening on the ‘silver list’ – these chemicals were
tested with the previous positive control tert-butyl-hydroquinone.
Note: Among these 67 chemicals, 28 chemicals were repeated in the ring study with the positive
control cinnamic aldehyde, and the criteria were all fulfilled in all runs in the lead lab in the ring study.
Only for one chemical (Hexylcinnamic aldehyde) a discordant result between these two studies
performed with the different positive controls was recorded (see Attachment 4a). Since this chemical
is borderline also in the BLR analysis, we consider that this change in the positive control had noeffect on overall performance.
The additional chemicals were tested with the current positive control and these data were now reevaluated to analyze the occurrence and impact of runs not meeting the quantitative criteria forcinnamic aldehyde. Runs which clearly did not meet the criteria had been excluded from analysis abinitio (such runs are marked yellow in attachment 17a). However some runs were used for the
analysis in Attachment 12 c, for which criteria were slightly out of the range. This is summarized in
the Table 2 below. In most cases one criterium was fulfilled. In particular, some results with an EC1.5
between 30 and 35 µM were used.
One critical run which had been included is highlighted in bold, in this case both criteria were not met,
(fold induction 10.9, EC1.5 at 6.6 µM): However, the two chemicals in Attachment 12c with data fromthis run (3-aminophenol and 3-Dimethyl-amino-1-propylamine) were repeated in 4 or 6 runs, andthus reporting of this run did not affect the conclusion for these two chemicals in attachment 12c.
If we do analysis without the runs not meeting the quantitative criteria (see Table 2, bold column“reps. Positive w/o non-qual. Runs”), the rating does not change for all but two chemicals:- Amylcinnamic aldehyde, similarly to hexyl cinnamic aldehyde, is borderline, sometimes
positive and negative in some repetitions. The result with the excluded run changes from 3 of
6 positive to 2 of 5 positive, and we would be even less confident of a positive rating- 2-hydroxypropyl methacrylate (see below)
Besides these two borderline chemicals we have a group of chemicals for which we are left with one
repetition, if we exclude the runs not fulfilling the criteria. To be sure whether we had a correct
rating in absence of these runs, we now repeated these chemicals in additional runs.
ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 55 of 77

Table 3 shows the quantitative data and rating of these chemicals in the original submission
compared with a new analysis including only runs with cinnamic aldehyde within the target range.
For all these chemicals we reach the same yes/no rating and similar quantitative data with these newruns fulfilling the quantitative criteria, with the exception of 2-hydroxypropyl-methacrylate. Thischemical had before an average EC1.5 at 1025µM just above the threshold, and it falls below thethreshold of 1000 µM with the new data. It would therefore be rated positive with the new data, but
in the rather weak range.
Note: This is the only chemical which was included as non-sensitizer in Attachment 12c due tonegative rating in both the LLNA and guinea pig tests, but which still gives covalent adduct formation
with peptides (our unpublished results in Attachment 12c) and (Gerberick et al., 2007; Aleksic et al.,2009). Interestingly 2-hydroxypropyl-methacrylate is known to cause relatively frequent positives
human patch test reactions in occupation settings and due to artificial nails (Kanerva et al., 1997;
Lazarov, 2007) and thus a weak human sensitization potential certainly exists.
Conclusion Point 2b) / Attachment 12c:
This analysis contained some runs not meeting the quantitative criteria for cinnamic aldehyde. Whilein some cases the conclusion was based on many repetitions and excluding these runs does notaffect the overall conclusions drawn, there were some cases where exclusion of these runs would
leave us with insufficient data. In these cases repetitions were run, and the same conclusions werereached as before. The only case where the final conclusions from above analysis is different is
amylcinnamic aldehyde, which is similarly borderline as hexylcinnamic aldehyde, and 2hydroxypropyl-methacrylate which just becomes positive in the revised analysis.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 56 of 77

EUROPEAN COMMISSIONJOINT RESEARCH CENTRE
Institute for Health and Consumer Protection
European Centre for the Validation of Alternative Methods (ECVAM)

Table 2. Chemicals which had been tested with cinnamic aldehyde as positive control included in Attachment 12c

Name

CAS-Number

LLNA
EC3

Imax

EC 1.5

IC50

Pos /
Neg

reps.
Positive

reps.
Positive
w/o nonqual.
Runs

Methylisoeugenol

93-16-3

Pos. 3)

1.4

>2000

815.1

0

0 of 40 of 4

All within range

4-Methylcatechol452-86-8

Pos. 3)

8.1

19.2

71.7

1

2 of 2

1 of 1

1 run criteria 64µM at 1.7-fold

1-Bromododecane
Diphenylmethane-4,4'diisocyanate

143-15-7

Pos. 3)

2.2

44.0

98.0

1

5 of 6

5 of 5

1 run criteria 64µM at 1.7-fold

101-68-8

Pos. 3)

2.4

121.8

>2000

51323-71-8

Pos. 3)

2.03

12.14

19.3

2 of 23 of 4 atcytotox2 of 23 of 4 atcytotox

All within range

Dodecyl methanesulfonate

1
atcytotox4-Nitrobenzyl chloride

100-14-1

Pos. 3)

93.4

4.0

27.6

1

2 of 22 of 2All within range

Chlorpromazine hydrochloride

69-09-0

Pos. 3)

1.1

>2000

10.1

7787-56-6

Pos. 3)

5.7

15.4

51.3

0 of 99 of 9 atcytotox0 of 8
8 of 8 atcytotox8 of 9 within range (attachment 4c)

Beryllium sulfate

0
atcytotox

Methyl methacrylate

80-62-6

90

1.7

424.4

>2000

1

2 of 32 of 3

All within range

d-Limonene

5989-27-5

52.7

1.2

>2000

82.3

0

0 of 20 of 2All within range

Penicillin G

61-33-630

10.7

1308.6>2000

00 of 40 of 4

All within range

Methylhexanedione

13706-86-0

25.8

23.4

49.8

1431.91

2 of 22 of 2All within range

Cyclamen aldehyde

103-95-7

22.3

3.1

111.9

190.8

1

3 of 43 of 4

All within range

Geraniol

106-24-1

21.74

2.0

209.8

722.0

1

2 of 22 of 2All within range

Butyl acrylate141-32-2

~20

6.9

37.7

200.9

1

2 of 22 of 2All within range

Estragole

140-67-0

20.2

1.3

>2000

419.0

0

0 of 20 of 2All within range

Lilial80-54-6

18.7

1.1

>2000

94.5

0

0 of 20 of 2All within range

Amylcinnamic aldehyde

122-40-7

11.5

1.56

14.4

46.8

1

3 of 62 of 5

1 run criteria EC1.5 at 32 µM

Bromohexane

111-25-1

10

2.0

128.1

391.9

1

2 of 22 of 2All within range

Methylanisylidene acetone104-27-8

9.3

835.8

14.8

159.3

1

2 of 2

1 of 1

1 run criteria EC1.5 at 34 µM

Phenylpropionaldehyde

93-53-8

6.3

9.1

64.8

195.1

1

2 of 22 of 2All within range

Creosol

93-51-6

5.8

1.0

>2000

>2000

0

0 of 30 of 3

All within range

3,4-dihydrocoumarin

119-84-6

5.6

1.0

>2000

>2000

0

0 of 20 of 2All within range

Farnesol4602-84-0

5.5

1.6

13.0

23.3

1

2 of 22 of 2All within range

KeratinoSens result

Positive control cinnamic aldehyde

All within range

8 of 9 within range (attachment 4c)

trans-2-Hexenal

6728-26-3

5.5

85.4

83.4

802.8

1

4 of 44 of 4

All within range

Propylidene phthalide

17369-59-4

3.7

1.1

>2000

717.4

0

0 of 20 of 23-Aminophenol

591-27-5

3.2

1.4

>2000

>2000

0

1 of 60 of 4

All within range
1 run both criteria not fulfilled, 1
run EC1.5 at 31.6

3-Dimethyl-amino-1-propylamine 109-55-7

2.2

30.2

85.8

1337.91

4 of 43 of 3

1 run both criteria not fulfilled

Diethylmaleate

141-05-9

2.1

60.7

9.4

361.1

1

4 of 44 of 4

All within range

Methylisothiazolinone

2682-20-4

1.9

22.6

11.8

139.0

1

4 of 44 of 4

All within range

Trimellitic anhydride552-30-7

1.42

1.1

>2000

>2000

0

0 of 20 of 1

1 run criteria EC1.5 at 32 µM

1,3-phenylenediamine

108-45-2

0.49

2.5

82.5

>2000

1

2 of 22 of 2All within range

N,N-dimethyl-4-nitrosoaniline

138-89-6

0.48

8.2

0.5

15.1

1

2 of 22 of 2All within range

Methyl 2-octynoate

111-12-6

0.45

46.6

2.5

87.6

1

2 of 22 of 2All within range

2-amino-phenol

95-55-6

0.4

13.1

1.1

138.2

1

2 of 2

1 of 1

1 run criteria EC1.5 at 33 µM

Chloramine T

127-65-1

0.4

50.2

248.4

404.7

1

2 of 22 of 2All within range

Propyl gallate

121-79-9

0.32

8.2

199.8

650.3

1

2 of 2

1 of 1

1 run criteria EC1.5 at 33 µM

Toluene 2,4-diisocyanate

584-84-9

0.11

4.6

135.0

359.0

1

4 of 44 of 4

All within range

1,4-Hydrochinone

123-31-9

0.1

16.4

9.8

130.7

1

2 of 22 of 2All within range

2,4,6-Trinitrochlorobenzene

88-88-0

0.05

1.6

121.3

616.8

1

2 of 2

1 of 1

1 run criteria 64µM at 1.91-fold

2,4-Dinitrothiocyanatobenzene

1594-56-5

0.047

7.2

2.1

6.4

1

2 of 2

1 of 1

1 run criteria 64µM at 1.91-fold

Tetrachlorsalicylanilide

1154-59-2

0.04

4.9

<0.98

9.15

1

4 of 44 of 4

All within range

4-Aminobenzoic acid150-13-0

>10

1.2

>2000

>2000

0

0 of 20 of 2All within range

1.5

>2000

4.0

0

1 of 4

1 of 4

All within range

Benzalkonium chloride
Fumaric acid110-17-8

>25

1.3

>2000

>2000

0

0 of 20 of 1

1 run criteria EC1.5 at 34 µM

2-hydroxypropyl methacrylate923-26-2

>50

1.95

1025

>2000

0

1 of 4

1 of 2

2 runs criteria EC1.5 at 32/34 µM

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation testing
Page 58 of 77

Table 3. New analysis including new runs for chemicals included in attachment 12c for which the quantitative criteria for cinnamic aldehyde were not fully met according toTable 2 and for which the conclusion is not sufficiently substantiated after excluding these runs.

Name
4-Methylcatechol
Methylanisylidene acetone
Trimellitic anhydride
2-amino-phenol
Propyl gallate
2,4,6-Trinitro-chlorobenzene
2,4-Dinitrothiocyanatobenzene
Fumaric acid
2-hydroxypropyl
methacrylate

Cas-Nr.

Previous resultsPos /EC 1.5
IC50
Neg
19.2
71.7
1
14.8
159.3
1
>2000
>2000
0
1.1
138.2
1
199.8
650.3
1
121.3
616.8
1

reps.
Positive
2 of 22 of 20 of 22 of 22 of 22 of 2

With new results, excluding the runs not meetingthe quantitative criteria
Pos /
reps.
Imax
EC 1.5
IC50
Neg
Positive
9.1
11.3
61.1
1
2 of 2
702.6
15.5
166.6
1
2 of 2
1.0
>2000
>2000
0
0 of 2
19.0
1.7
115.3
1
2 of 2
6.3
136.7
538.6
1
2 of 2
4.4
41.4
791.2
1
2 of 2452-86-8
104-27-8
552-30-7
95-55-6
121-79-9
88-88-0

Imax
8.1
835.8
1.1
13.1
8.2
1.6

1594-56-5
110-17-8

7.2
1.3

2.1
>2000

6.4
>2000

1
0

2 of 20 of 2

8.2
1.1

2.2
>2000

7.5
>2000

1
0

2 of 20 of 4923-26-2

1.95

1025

>2000

0

1 of 4

2.0

410.8

>2000

1

4 of 5ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation testing
Page 59 of 77

EUROPEAN COMMISSIONJOINT RESEARCH CENTRE
Institute for Health and Consumer Protection
European Centre for the Validation of Alternative Methods (ECVAM)

Point 3) and 4) (Ring study Attachments 8b, 10b, 10c, 12b)
We had previously reported the overall occurrence of tests outside of the quantitative criteria forcinnamic aldehyde in the ring study in Attachment 17b and all the data and dose-response data forthe positive control were reported in this attachment. So we were fully transparent on theoccurrence of results not meeting these quantitative criteria and on our approach to accept the threeconsecutive runs performed by the laboratories.
We have now attributed the runs with the criteria outside the range to the individual data points,
and this is shown in the revised attachment 8b and 10b, see files:
Attachment8b_Transferability_Table_wo_non-qualified.xls.
andAttachment10b_BLR_Table_wo_non-qualified.xls
These revised attachments give colour codes for the individual repetitions outside the range, andthey give the results for Imax (green and faint green), EC1.5 and IC50 (red and orange) with andwithout these runs included for each individual lab, and they also give the overall results from alllaboratories, with these runs included or excluded. To see whether the inclusion of the runs affected
the conclusions from the individual labs, it is best to directly compare the two columns for theindividual labs in the Excel files of these attachments. In most cases inclusion or exclusion of theseruns did not affect the quantitative results, but certainly the statistical power is reduced if we have
only 1-2 repetitions in a lab.
To give an overall impression for the quantitative data, below in Figure 2 are shown the overall
results for all chemicals from all labs with and without the runs outside of the quantitative criteria
included.
Figure 2a. Average of the Imax of all five laboratories calculated for all original runs or only thoseruns with the quantitative criteria within the range.
1000

100

10

1
(5 2, 4-ni
- c 4thl Di roboro) nitro enz
te -Me -ch ylbr
tra th lo om
m yl- rob ideti
hy sot enz e
l-t hia enhiM
ur zol eetam ino
hy
ld
di ne
ib
su
ro
lfid
m
4o
P h gl M e
en uta eto
C yle ron linna ne itrim dia leicm
al i n
de e
H
hy
ex
de
yl
C
ci
I
Et
so itrnnhy
am eu allene Imi ic gen
da a
g
ld ol
2M lyco zoli eh
di
y
er
l
ca dim ny delupteo- th rebe ac aC nz rylinna othi ate
m az
yl
o
al l ecoh
G ol
O l yoxa xazlL a ol o
ct ne
ic
ac
D
i
ie
th Gly dcyl
ph eroIs tha l
op l a
So
ro te
S
p
di
um alic an
yli olla c aur
yl cidsu
Phlf
en Eu ategeylM
et ben nolhyz
C l sa oatehl
or licyolSu ben ate
lfa ze
ni ne
la
m
id
e

fold induction

Imax Avg all
Imax w/o non-qual.

2 4ch ,4-D nitrolo
ro init benr)
te -Me o-c zylb
tra th hlo ro
M me yl-i rob mid
et th so e
hy yl- th nz e
ld th ia e
H ibr iur zol ne
ex om am inoylci o g dis nen n l u ul
am tar fidei c oniSo
al tril
de e
di
um
hy
Im la M deid ur etaz ylool su l
id
l
f
in at
C
yl e
in
na
ur
ea
m
i
C
P c
4- he ald itraP h ny e l
en l b hyd
yle en eEt
n e zohy
di ateleam
negl
G i neycol Iso lyo
2di eu xamM
et gen l
er
ha o
ca
c l
pt
o- Ox ryla
be az ten z ol o
ot n
hi e
C
az
in
n a E u ol e
m ge
yl no
al l
co
ho
l

(5
-

conc. (micromolar)

10000.00

Di
n

itr
o

tro
b

H

en

2M

zy

-b
e

in
yl
et
ha

lid

ur

ea

hy
de

l

l
no

itr
a
ge
al
de

eu

C

cr
yl
at
e
nz
ot
h
C
i
a
in
zo
na
le
m
yl
al
co
ho
l
G
ly
ox
al
O
xa
zo
lo
ne

ap
to

er
c

zo

m
ic

Is
o

di
m

id
a

na

gl
yc
ol

Im

in

lc

ex
y

le
ne

ch
lo

lb
ro
m
-c
ro
id
hl
)-M
e
or
ob
et
e
h
nz
yl
te
-is
tra
en
ot
m
e
hi
et
az
hy
ol
l-t
in
hi
on
ur
e
am
di
su
M
et
lfid
hy
e
ld
ib
ro
M
m
et
o
ol
gl
4ut
ar
Ph
on
en
itr
yle
ile
ne
C
di
in
a
na
m
in
m
e
ic
al
de
hy
de

2,
4-

Et
hy

(5
-

4ni

conc. (micromolar)

Figure 2b. Geometric Mean of the EC1.5 values from all five laboratories calculated for all original runs
or only those runs with the quantitative criteria within the range. Chemicals with no significant geneinduction above the 1.5 threshold are excluded from this graph.
1000

EC1.5 GeoMean all
EC1.5 GeoMean w/o non-qual.

100

10

1

Figure 2c. Geometric Mean of the IC50 values from all five laboratories calculated for all original runs
or only those runs with the quantitative criteria within the range. Chemicals with IC50 values over 2000
µM are excluded from this graph.

IC50 GeoMean all

1000.00
IC50 GeoMean w/o non-qual.

100.00

10.00

1.00

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 61 of 77

The revised Attachment 10 b (Attachment10b_BLR_Table_wo_non-qualified.xls) also contains a sheetwith the predictivity Table (Table 2 in the SOP) excluding the runs outside the quantitative criteria. If
we look at this analysis, we certainly obtain significant data-gaps by excluding these runs and thestudy partly becomes invalid due to lack of data. At this point it becomes very important to discuss
the main question:
Are our conclusions regarding transferability and between laboratory reproducibility affectedby our approach of directly using the three consecutive repetitions performed by thelaboratories, even if the quantitative criteria were not fulfilled?
As shown above and in the revised attachments, the quantitative results are not affected by thisapproach, but if we exclude the runs we do have fewer repetitions / fewer laboratories depending on
the chemical. Looking at the Cooper statistics table, we would need to exclude some labs due to data
gaps.
The case of Lab 2 is most dramatic – in this case the data gaps would lead to complete exclusion ofthis laboratory from analysis. On the other hand we can specifically look at the result obtained fromthis lab in comparison to the lead lab, which has fulfilled the criteria in all runs:
Regarding the predictive capacity, this lab obtained the same rating for all chemicals as the lead labwith the exception of Eugenol, which was even correct-positive in Lab2. Based on the statisticalanalysis in Attachment 10f, the best performing external lab is lab 2, and indeed the quantitative
data of Lab 2 are highly comparable to the lead lab as is illustrated in Figure 3.
Figure 3a. Average of the Imax of the lead lab with all repetition within the quantitative range forcinnamic aldehyde and of lab 2 (all results, the majority being outside of the quantitative range).
1000

Imax Avg all Giv
Imax Avg all Lab2
fold induction

100

10

(5
-

2 4ch ,4-D nitrolo
ro init benr)
te -M o-c zylb
tra et hl
hm yl oro rom
et -is b id
hy o en el-t thi ze
M
hi az netur o ehy
am linld
di one
ib
su
ro
lfid
4- mo
Ph g
e
en lut Me
C yl aro tol
in en nna e itm dia rileic
al min
de e
H
ex
hy
yl
de
Et
ci
nn Iso Cit
hy
r
a
le
ne Im mi eug alc
2- gly idaz al eno
M co o de llerhca l di idin yd
pt me yl u eo- th rba e
C enz cry ainona th latem iaz
yla l ol eco
G ho
O lyo lxa x
aLa zol l
ct on
ic e
D
ac
ie
th Gly id
ylc
ph eroIs tha l
op l aSo
di Sa rop teum lic ay nla lic ol
ur ac
yl
su id
Ph
en Eu lfate
M yl b gen
et ehy nz olC l sa oahlor licy teoSu be late
lfa nze
ni n
la e
m
id
e

1

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 62 of 77

Figure 3b. Geometric Mean of the EC1.5 values from of the lead lab with all repetition within thequantitative range for cinnamic aldehyde and of lab 2 (all results, the majority being outside of thequantitative range). Note: tetramethylthiuramdisulfide in the lead lab is at <0.98 µM and could not
be plotted.
1000

100

10

Ci
He
tra
xy
Is
l
oe
lc
in
ug
na
en
m
ol
ic
Et
a
I
l
m
hy
d
e
ida
le
hy
ne
zo
de
l id
gl
yc
in
yl
ol
2ur
di
M
ea
m
er
e
ca
th
ac
pt
ory
la
be
te
nz
ot
Ci
hia
nn
zo
am
le
yl
alc
oh
ol
G
lyo
xa
O
l
xa
zo
lo
ne

1

4ni
tro
2,
be
4nz
Di
(5
yl b
-c
ni
tro
hlo
ro
m
-c
ro
ide
hlo
)-M
ro
et
b
h
en
te
yltra
ze
i so
ne
m
th
et
iaz
hy
ol
l-t
ino
hi
ur
ne
am
di
su
M
et
lfid
hy
e
ldi
br
om
M
et
o
ol
gl
4ut
Ph
ar
o
en
ni
yle
tri
le
n
ed
Ci
nn
ia
m
am
in
e
ic
ald
eh
yd
e

conc. (micromolar)

EC1.5 GeoMean all Giv
EC1.5 GeoMean all Lab2

Based on this specific analysis we would conclude that the assay and the quantitative data weretransferable and reproducible even if the targeted quantitative range of the positive control was notmet (as is especially the case in Lab2), and the assay thus is sufficiently robust to deliver reliable data,
even if the quantitative performance of the positive control was not in the target range.

Overall conclusion to point 2) – 4)

We fully agree that our approach of setting a quantitative target, but then accepting the consecutive
runs if the quantitative criteria were not met are unconventional, to say the least. Starting with themore conservative requirement of a statistically significant positive result for the positive control as
the minimal target would have been better, and then only adding more stringent data requirements
if needed based on experience at a later stage would be a more transparent and more logicalapproach.
Nevertheless, this detailed retrospective analysis looking at runs fulfilling the quantitative target andthose not fulfilling the quantitative target indicates that the study conclusions appear not to beaffected by our unconventional approach. This approach did avoid iterative testing and shows thelevel of reproducibility which can be achieved even without such iterative testing. It thereby gives anindication not only of transferability and reproducibility, but also certain indications of therobustness of the assay.
ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 63 of 77

However, it is clear that going forward this should be handled more strictly. Based on all thisexperience we had recommended in the SOP that one of the two criteria must be fulfilled, andotherwise runs are discarded. See Page 17 of the submitted SOP where we indicated :ACCEPTANCE CRITERIA- Cinnamic aldehyde as positive control must be positive, thus the gene induction by thiscontrol must be statistically significant above the threshold of 1.5 in at least one dose.
- The Imax and the EC 1.5 for cinnamic aldehyde is calculated. The targets are: (i) Average
induction in the three replicates for cinnamic aldehyde at 64 µM should be between 2 and 8,
and (ii) the EC 1.5 value should be between 7.5 µM and 30 µM. At least one of these criteria
must be met, otherwise the run is discarded. If only one criteria is fulfilled, it isrecommended to carefully check the dose-response of cinnamic aldehyde in order to decideon acceptability- For acceptance of the test for a given master plate in a given repetition, the averagevariability in the 3 × 6 solvent control wells for each master plate/repetition should be below
20%. If the variability is higher results are discarded.

References:
Aleksic, M., Thain, E., Roger, D., Saib, O., Davies, M., Li, J., Aptula, A.Zazzeroni, R., 2009. Reactivity
profiling: Covalent modification of single nucleophile peptides for skin sensitization risk
assessment. Toxicological Sciences 108, 401-411.
Gerberick, G. F., Vassallo, J. D., Foertsch, L. M., Price, B. B., Chaney, J. G.Lepoittevin, J. P., 2007.
Quantification of chemical peptide reactivity for screening contact allergens: A classification
tree model approach. Toxicological Sciences 97, 417-427.
Kanerva, L., Jolanki, R.Estlander, T., 1997. 10 years of patch testing with the (meth)acrylate series.
Contact Dermatitis 37, 255-8.
Lazarov, A., 2007. Sensitization to acrylates is a common adverse reaction to artificial fingernails. J
Eur Acad Dermatol Venereol 21, 169-74.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 64 of 77

Annex 2
Meeting of the ESAC working group Sensitisation
1-3 February 2012
EC JRC, Ispra, Italy

Participating:
ESAC members: Erwin Roggen (Chair ESAC WG), Wally Hayes, Walter Pfaller
Experts: Maja Aleksic, Emanuela Corsini, Yong Heo, David Lovell, Michael Woolhiser
ECVAM: Claudius Griesinger (ESAC Coordination/Secretariat), Alexandre Angers (Scientific Support)

1) First discussions concerning the DPRA report.
Not reproduced in the present context (Keratinosens WG report).

2) Further discussions on the Keratinosens report
The WG did not consider that the additional information sent by Givaudan regarding the statisticalanalyses provided the clarification they were requesting. In any case, the WG decided to focus theanalysis of the within laboratory reproducibility (WLR) in terms of concordance of the predictions
obtained within the same laboratory. This means that the only study that can provide thisinformation is in attachment 4c, the evaluation of 14 chemicals at Givaudan, performed in threecomplete experiments.
Concerning the occurrence and impact of the invalid runs, the WG will request from Givaudan a final
analyses of the results in the submission, taking into account three different scenarios:

The final results (in terms of WLR, BLR and Predictive Capacity), accepting all the experiments
generated for the submission

The final results, rejecting all the runs/experiments that did not meet the acceptance criteria
as they were defined prior to the start of the ring trial

The final results, rejecting all the runs/experiments that did not meet the acceptance criteria
the test submitter would like to propose for future use based on the experience gained in the
study.

The WG would also ask Givaudan to clarify the exact details of these acceptance criteria, as they vary
between the different sections of the document.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 65 of 77

Documents will be prepared and sent to Andreas Natsch (AN) in order to make clear what therequests of the WG are. In addition, a teleconference will be set on the 8 th of February (14.00 CET)with AN to explain these requests and the documents.
3) Dates for future meetings
Next meeting for DPRA review: 10/11 May

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 66 of 77

Annex 3
EUROPEAN COMMISSIONJOINT RESEARCH CENTRE
Institute for Health and Consumer Protection
European Union Reference Laboratory for Alternatives to Animal Testing(EURL ECVAM)ESAC Working Group Skin Sensitisation
Draft Minutes Teleconference
April 24th 2012
14:00 – 16:00 CET
Minutes: Claudius Griesinger
1.
Participation
Participating:
ESAC members: Erwin Roggen (Chair ESAC WG), Walter Pfaller
Experts: Maja Aleksic, Emanuela Corsini, Michael Woolhiser
ECVAM: Claudius Griesinger (ESAC Coordination/Secretariat)
Excused:David Lovell, Wally Hayes, Yong Heo
2.

Agenda / purpose of TC

The Secretariat briefly revisited the agenda and purpose of the teleconference:
Agenda:1. Discussion of the material forwarded by Givaudan on 14/3/2012 following request fromESAC WG forwarded on 8/2/2012.
1.1 To which extent does this information (reproducibility expressed as concordance ofpredictions and taking different cases regarding fulfilment of test acceptance criteria into
account ) address the open questions of the WG and is it sufficient for addressing the mainquestion of the review (reliability of the assay)?
1.2 To which extent does the analysis forwarded allow addressing (preliminary) predictive
capacity and inform on possible limitations (including the biological relevance: pathway) of
the test method in view of defining the necessary follow-up work (gaps) required to
sufficiently characterise the test method for possible use within a testing strategy?
2. Organisation/distribution of work - draft ESAC WG report and draft ESAC opinion.
Taking the purpose of the TC into account, it was agreed to discuss the submitted Word Document("Final clarifications…") as well as the more detailed excel spreadsheets providing summary data ofreliability and preliminary predictive capacity as requested by the ESAC WG.
3.

Discussion of material resubmitted by Givaudan

3.1
Section A of the clarification document: Test design
It was agreed that the schematic outline of the test design helped in understanding how a
Keratinosens test is carried out to arrive at a final prediction (Sensitizer / Non-Sensitizer). The
ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 67 of 77

schematic had been initially provided by the ESAC WG/ECVAM and was now updated by Givaudan
for the purpose of this resubmission. The update concerned the addition of the MTT parallel plate toassess cytotoxicity (condition 4 of the Test Acceptance Criteria). It was agreed to include thisschematic figure in the ESAC WG report.
3.2
Section B: Test Acceptance Criteria
The WG established that the Criteria Sets found and described by the ESAC WG corresponded to the
Criteria Sets communicated by Givaudan in the resubmission ("clarification") as shown in table 1. The
ESAC WG agreed that the question of which criteria had been used to analyse in particular the ringtrial data was now sufficiently clear. Moreover, it was transparent which criteria Givaudan
recommends for future use of the assay.
Criteria Set found by WG and communicated in the
document "Final Clarifications…" to Givaudan by the
ESAC Sec. (sent to Givaudan 8.2.2012)
Criteria Set 1 (page 7 of submission).

Criteria Set as described in the resubmission from
Givaudan (received by ECVAM 14.3.2012)
Criteria Set 1 (page 7 of submission)
Additional clarification (in blue) reg. condition 1: CA
positive in the range 4-64uM.
NOTE ESAC SEC: These criteria (and not Criteria 2
communicated in SOP of ring trial) were used to analyse
the ring trial data. However, some ring trial data wereinvalid when applying these criteria. As no rules forretesting in case of unfulfilled TACs had been stipulated
beforehand, this led to some non-qualified test results
being included in the analysis contained in the full
submission to ECVAM. Having realised that non-qualified
test results had been included in the analysis on
reproducibility and predictive capacity, the ESAC WG
requested (on 8/2/2012) a reanalysis of the data using (a)concordance of predictions as a measure for reliabilityand (b) performing this analysis applying or not-applying
the specified test acceptance criteria.

Criteria Set 2 (Att. 7a SOP of ring trial):
contained in the ring trial SOP but neither used byparticipating labs nor by lead lab for final analysis. This set
should be ignored for the purposed of the review.
Criteria Set 3 (page 9 of submission)

Criteria Set 2
Additional clarification (in blue) reg. condition 1: CA
positive in the range 4-64uM
NOTE ESAC SEC: These are the test acceptance criteria as
currently used by Givaudan and as recommended for
future use.

Criteria Set 4 (page 17 of Invittox protocol):This set is identical to set 3, apart from a minor typo (7.5
instead of 7.0 uM in condition 2). This set should be
ignored for the purposes of the review.

3.3
Section C: 1) Within Laboratory Reproducibility
The ESAC WG agreed that the re-analysis resubmitted was satisfying with regard to answering thequestion to which extent non-qualified test results might have influenced the WLR analysis. The
impact was felt to be negligible as in case 2 (most stringent criteria) only 3 individual laboratory
predictions had not qualified. The data were found sufficient to judge WLR.
3.4
Section C: 2) Between Laboratory Reproducibility
Agreement was reached regarding the deletion of the fourth column in the word document as wellas the corresponding columns in the excel files. These columns were intended to allow analysis of
ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 68 of 77

concordance of predictions on the basis of four laboratories only instead of the five laboratories that
had participated in the ring trial. The rationale for this column was that with an increasing number oflabs it may be more likely to get non-concordant results. Givaudan, when planning the ring trial had
been unaware that it is common practice in the context of validation to use three laboratories within
a ring trial for assessing transferability and between-lab reproducibility (BLR). However, as thisanalysis had not been properly conducted in the resubmitted data package, the ESAC WG felt that
this approach should not be followed-up when finalising the ESAC WG / ESAC review.
No agreement was reached with respect to the question whether or not the data on BLR (whentaking non-qualified laboratory predictions into account: case 2 and 3) were sufficient to judge
reproducibility between laboratories. The reason for this uncertainty lies with one of the principal
flaws of the planning and execution of the study, i.e. that the Test Acceptance Criteria (TACs)
provided to the participating laboratories during the ring trial were (a) had not applied whenanalysing the data (a variation of the initial TACs had been employed) as they were found too
stringent, (b) that no provisions for re-testing had been made in case test results would not meet the
TACs and (c) that, as a consequence, the final data matrix contained an appreciable number of nonqualified test results that were included in the analysis.
 With regard to the analysis requested in the ESAC WG's letter to Givaudan requesting
clarification/re-analysis of the data, the Chair (E. Roggen) noted that while the non-qualified
test results had been deleted from the data matrix (case 2 and case 3), the concordance had
nevertheless been calculated on the basis of the remaining laboratory predictions (whichwere in all cases at least 3 lab predictions per chemical). This issue was contentiously
discussed in the group. Some members felt that chemicals for which some individuallaboratory predictions had to be deleted should not be included in the analysis ofconcordance/reliability.
 M. Woolhiser remarked that the approach chosen by Givaudan seemed neverthelessreasonable as the common standard were 3 laboratory predictions per chemical in avalidation ring trial and since for all chemicals at least 3 laboratory predictions were available
after deletion of non-qualified results.
 The ESAC Secretariat remarked that out of 105 individual lab predictions (21 chemicals x 5labs) only 14 were not qualified and had been deleted. Hence there seemed to be sufficient
data for judging whether the test was reliable between laboratories especially when
considering, as pointed out by M. Woolhiser, that in all cases a minimum of three lab
predictions was available. The Secretariat also suggested to the WG to consider separating
the issues of flawed planning/conduct (e.g. no provisions for retesting, criteria appropriately
fixed before ring trial) from the issue of whether or not there were sufficient data for judging
reliability.
It was finally agreed to postpone conclusion of this issue and allow some time to reflect on it.
Members should forward to the Chair and Secretariat a short summary statement describing theirview concerning BLR analysis and in particular whether there are sufficient data to conclude on BLR.
Due to the advanced time, the TC was closed at this point, although transferability as well as
preliminary predictive capacity (including the extended non-sensitizer list: LLNA negatives) had not
yet been discussed.
Finally, the ESAC Secretariat kindly requested that discussions at the next ESAC WG meeting (10/11
May) be limited to the DPRA. Moreover, the Secretariat kindly requested that the Keratinosens
review be finalised as soon as possible, especially in the light of the fact that all critical clarifications
had now been made available and the ESAC WG report was in an advanced state.
ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 69 of 77

4.

Actions

Item Nr.
1

2

3

ActionForward an appraisal of the data on BLR ascontained in the resubmission to Chair andSecretariat
Block date and time for the next TC on
Keratinosens:
22 MAY from 14:00 to 16:30.
With this TC the ESAC WG should come toa conclusion on the main findings of the
Keratinosens review. These should provide
the essential points for the draft ESAC
opinion.
Reformat the submitted spreadsheets(deleting 1 column referring to 4 out of 5
lab predictions).

Actor
All ESAC WG members

Timeline
asap, deadline is May 4 to 7

All ESAC WG members

ESAC Secretariat

asap

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 70 of 77

Annex 4

JOINT RESEARCH CENTRE
Institute for Health and Consumer Protection
EURL-ECVAM
The European Union Reference Laboratory for Alternative Methods to Animal Testing

ECVAM
SCIENTIFIC
ADVISORYCOMMITTEE(ESAC)ESAC Working Group Sensitisation:
Final clarifications regarding the Givaudan/Keratinosens submission(ER 2011-04)

A.

Test design

A complete test, i.e. experiment yielding the intended TEST OUTCOME of the KeratinoSens test
method (i.e. a binary prediction: S=1 or NS=0) is based on the following design: a) Replicate level: Chemicals are tested in 3 replicates (in triplicate). The parameters
measured of the individual replicates are averaged. One replicate parallel plate is run for thecytotoxicity determination.
 b) Repetition level: Using this average value, a preliminary prediction is derived by applying
the prediction model (PM). This result constitutes a so-called "repetition" (consisting of three
replicates for luciferase and one replicate for MTT).
 c) Test outcome level: According to the provisions of the PM, three repetitions are required
to arrive at a final decision/"final call" (=the TEST OUTCOME as foreseen when applying thetest). The final decision is made by using the mode of the predictions of the 3 repetitions
allowing a unequivocal predictions to be made.
N.B: while the algorithm of the PM is applied to average values at run/repetition level to reach aprediction, this result is not equivalent to a final prediction (although being made on the basis of thePM), since the PM requires the two concordant results of three runs/repetitions to arrive at a final
prediction (TEST OUTCOME, TOC). Three repetitions were always made for all the intra- andinterlaboratory studies. As we need two concordant predictions in the PM, two unequivocal
concordant predictions in the first two repetitions were considered sufficient for a TOC in the
additional screenings. In cases of borderline results or non-concordance of the first two repetitions
then 2 – 4 additional repetitions were run for a better assurance of the TOC.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 71 of 77

Figure 1: Schematic depiction of the design of a full TEST, leading to the predefined TEST OUTCOME =a prediction as to whether a chemical is a sensitizer or not.

a) Replicate
level

Replicates 3x(=in triplicate)

Parallel
MTT
plate

Averaging ofparameter
Preliminary
predictionb) Run level(repetition level)(here “repetition”)
acc. to averaging(SOP)”Repetition1” =’rep’c) Final test outcome=prediction using PM

Replicates 3x(=in triplicate)

Parallel
MTT
plate

Preliminary
prediction(repetition level)
acc. to averaging(SOP)”Repetition2” =’rep’

Replicates 3x(=in triplicate)

Parallel
MTT
plate

Preliminary
prediction(repetition level)
acc. to averaging(SOP)”Repetition3” =’rep’

Test Outcome (TOC)
using prediction
model: final decision
based on mode ofrepetitions

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 72 of 77

B.

Clarification on the Test Acceptance Criteria used/suggested

CRITERIA SET 1
Condition:QUALIFIED IF 1 AND 2 AND 3 AND 4 FULFILLED
Criteria (page 7 of submission)

logical operator

1 Cinnamic aldehyde positive in the range 4 - 64uM AND
2 EC1.5 CA 7-30uM
AND
3 Induction at 64uM CA 2-8x
AND
4 Variability in solvent wells 20%

CRITERIA SET 2
Condition:QUALIFIED IF 1 AND (2 AND/OR 3) AND 4 FULFILLED
Criteria (page 9 of submission)
logical operator
1 Cinnamic aldehyde positive in the range 4 - 64uM
AND
2 EC1.5 CA 7-30uM
OR
3 Induction at 64uM CA 2-8x
AND
4 Variability in solvent wells <20%

Criteria set 1 was defined as intralaboratory data on the positive control accumulated and it wasused to evaluate the ring-study data against. However, these criteria were based on the responses
observed experimentally from the positive control in the lead lab, and there was no indication
concerning the importance of these ranges of responses on the validity of the runs, so it was decided
to include all the runs in the ring study irrespective of whether or not they met these criteria.
Based on the experience of the ring study it was proposed in the original test submission to ECVAM
to abandon condition 2 and 3 and only continue with condition 1 and 4. However, during internal
ECVAM review, the ECVAM team recommended to keep minimal quantitative criteria and eventually
modify conditions 2/3. Based on this recommendation and the overall experience, criteria set 2 wasdefined with the “OR” operator for condition 2 and 3 in the revised test submission which was thenaccepted for the peer review.
This criteria are set based on the accumulated experience, namely that this is a practical achievable
criterium – and we thus decided to use it in our Lab on routine use and in the SOP for future use.

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 73 of 77

C.

Performance values (reproducibility, predictive capacity)

1) WITHIN LABORATORY REPRODUCIBILITY SET (N=14)
CASE

EXPLANATION

1ALL
PREDICTIONSINCLUDED
ONLY QUALIFIED 85.7
PREDICTIONS (a)ONLY QUALIFIED 85.7
PREDICTIONS (b)

2
3
a.
b.

Concordance ofPredictions
withinlaboratory
85.7

Sensitivity

Specificity

Accuracy

70.0

100.0

78.6

70.0

100.0

78.6

70.0

100.0

78.6

On the basis of predictions that fulfilled the Test Acceptance Criteria (TAC) as used for the ring trial (Criteria Set 1).
On the basis of predictions that fulfilled the Test Acceptance Criteria (TAC) as specified for future use of the test method(Criteria Set 2).

2) BETWEEN LABORATORY REPRODUCIBILITY SET (N=21)
CASE

EXPLANATION

1ALL
PREDICTIONSINCLUDEDONLY
QUALIFIED
PREDICTIONS(a)ONLY
QUALIFIED
PREDICTIONS
(b)

2

3

1)

Concordanceof predictions
betweenlaboratories:
5/5 =concordant
85.7(n=21)

Concordanceof predictions
betweenlaboratories:
4/5 =concordant
95.2(n=21)

Sensitivity

Specificity

Accuracy

92.9

100.0

95.0
(n=20) 1

90.5 2)(n=21)

Not applicable

86.7

100.0

90.5
(n=21)

85.7 2)(n=21)

100.0(n=11)

86.7

100.0

90.5
(n=21)For one chemical (Eugenol) the status is 2 labs positive, 3 labs negative, thus final call cannot be decide and hence n=20

a. On the basis of predictions that fulfilled the Test Acceptance Criteria (TAC) as used forthe ring trial (Criteria Set 1), 2) concordant for all the labs for which a call could be made,in this case not always 5 labs, see excel file.
b. On the basis of predictions that fulfilled the Test Acceptance Criteria (TAC) as specified
for future use of the test method (Criteria Set 2), 2) concordant for all the labs for which acall could be made, in this case not always 5 labs, see excel file
ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 74 of 77

3) TRANSFERABILITY SET (N=7)
CASE

EXPLANATION

1

ALL
100
100
100
PREDICTIONS(n = 6) 2)
INCLUDED
ONLY
85.7 1)
Not applicable 100
100
100
QUALIFIED(n=7)(n = 6)
PREDICTIONS(a)
ONLY
85.7 1)
Not applicable 100
100
100
QUALIFIED(n=7)(n = 6)
PREDICTIONS(b)
1)
One Lab is excluded due to two repetitions outside of the Criteria Set 1, concordance for
4/42)
For one chemical (Hexyl-cinnamic aldehyde) the status is 2 labs positive, 3 labs negative,
thus final call cannot be decide and hence n=6, see excel file2

3

Concordanceof predictions
betweenlaboratories:
5/5 =concordant
85.7
(n=7)

Concordanceof predictions
betweenlaboratories:
4/5 =concordant
85.7
(n=7)

Sensitivity

Specificity

Accuracya. On the basis of predictions that fulfilled the Test Acceptance Criteria (TAC) as used forthe ring trial (Criteria Set 1).
b. On the basis of predictions that fulfilled the Test Acceptance Criteria (TAC) as specified
for future use of the test method (Criteria Set 2).

4) PREDICTIVE CAPACITY SET BASED ON EXISTING DATA = SILVER LIST (N=67)[USING HISTORICAL DATA ]CASE
1EXPLANATION
ALL PREDICTIONS INCLUDED

Sensitivity
88.4

Specificity
79.2

Accuracy
85.15) PREDICTIVE CAPACITY SET BASED ON NEWLY GENERATED DATA (N=47)
CASE
1EXPLANATION
ALL PREDICTIONS INCLUDED

2

ONLY QUALIFIEDPREDICTIONS (a)ONLY QUALIFIEDPREDICTIONS (b) 2)

3

Sensitivity 1)
67.0

Specificity
100.0

63.9

100.0

67.0

100.0

Accuracy70.2(n= 47)65.8(n= 38)70.2(n= 47)ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 75 of 77

1)

We suggest that this Table should always be viewed along with the discussion and definition of theapplicability domain discussed in detail on page 9-11 of Attachment 12c. This list contains asignificant number of specifically amine-reactive chemicals and phenolic prohaptens, which were
found to be outside of the applicability domain of the assay and which need to be detected byadditional peptide reactivity assays and metabolic activation assays.
2)

Note: The studies in section 5, 6 and 7 only contain data which fulfil the criteria set 2 (i.e. theacceptance criteria suggested for further use of the final SOP as used now in our laboratory, with oneof the two quantitative criteria required positive). Hence in this and the following three Tables thenumerical values for case 1 and case 3 are always identical. In the case 2, data were omitted forwhich one of the two quantitative criteria were not met).

6) PREDICTIVE CAPACITY FOR CHEMICALS WITH NEG. LLNA REFERENCE DATA(SUBMITTED IN JANUARY 2012)
CASE
1

EXPLANATION
ALL PREDICTIONS INCLUDED

2

ONLY QUALIFIED PREDICTIONS (a)3

ONLY QUALIFIED PREDICTIONS (b)Specificity
72.5(n=80)
74.0(n=73)
72.5(n=80)

7) PREDICTIVE CAPACITY FOR CHEMICALS WITH NEG. LLNA REFERENCE DATA (SUBMITTED INJANUARY 2012) AND WHEN CONSIDERING AVAILABLE HUMAN REFERENCE DATACASE
1EXPLANATION
ALL PREDICTIONS INCLUDED

2

ONLY QUALIFIED PREDICTIONS (a)3

ONLY QUALIFIED PREDICTIONS (b)Specificity
76.4(n=80)
77.3(n=73)
76.4(n=80)

Omitting the reactive, peptide alkylating chemicals, for which we do not trust the LLNA despiteabsence of human data:
CASE
1

EXPLANATION
ALL PREDICTIONS INCLUDED

2

ONLY QUALIFIED PREDICTIONS (a)3

ONLY QUALIFIED PREDICTIONS (b)Specificity
83.1(n=73)
83.3(n=67)
83.1(n=73)

8) PREDICTIVE CAPACITY FOR ALL SUBMITTED DATAESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 76 of 77

Compiles all the data of all the submitted attachments (compiled in attached Excel
file ”Keratinosens_SYN_MOD5_all.xls”). Human evidence was taken into account as in section 7
above. Only Case 1 applies, as it includes the historical silver list data.

CASE
1

EXPLANATION
ALL PREDICTIONS INCLUDED

Sensitivity
79.3

Specificity
79.8

Accuracy
79.5(n=220)Omitting the reactive, peptide alkylating chemicals, for which we do not trust the LLNA despiteabsence of human data
CASE
EXPLANATION
Sensitivity
Specificity
Accuracy
1
ALL PREDICTIONS INCLUDED
79.3
84.5
81.7
(n=213)End of document

ESAC WG report on the Givaudan-led validation study of the Keratinosens Test Method for skin sensitisation
testing
Page 77 of 77