-

R Sanchez R. Sanchez -Reillo, R Blanco R. Blanco -Gonz

Gonz

Hurtado, A. Me

University Group for Identification Tec

Carlos III Uni

http

http://idte

rsre

Raul Sanchez-Reillo (UC3M - GUTI IDTestingLab) 
 Introduction 

 Capture Devices  Sample Illustrations 

 Performance Evaluation 

 Specifications Specifications  Parameter Study towards Interoperability  Conclusions (Performance) – 1  Skilled FF orgeries  Conclusions (Performance) – 2 

 “Usability Usability Evaluation (Scenario Evaluation) ” Evaluation (Scenario Evaluation) 

 Specifications  Intra-device and Intra-scenario  Scenario Inteoperability Inteoperability  Expected Interoperability Deployment  Conclusions (Usability) 

Raul Sanchez-Reillo (UC3M - GUTI - IDTestingLab) 2 
 Handwritten Signature: 

 Off-line (static) signature  On-line (dynamic) signature 

 Recommended for non-forensic applications 

 Motivation for using Handwritten Signature: Motivation for using Handwritten Signature: 

 The de-facto authentication method in many operat

scenarios 

 Accepted by users of any age and cultural level  New capture devices are cheap and high quality  Touch sensitive screens can be used as capture devic

Touch sensitive screens can be used as capture devic

 Already in the hands of the final user  “costless” biomet

 Motivation for this study 

 Is th th ere any ii mpact on th th e capture dd evice technolog Is there any impact on the “item” used to sign 

(stylus/finger)? 

 Is there any impact on the position of the user when 

Raul Sanchez-Reillo (UC3M - GUTI - IDTestingLab) 3 
• Asus Eee PC

Touch T101MT 

• Tablet PC • Mixed

• 10.1”

• Samsung Galaxy 

Note (Note-S) 

• Smartphone • Mixed • 5.3”

Raul Sanchez-Reillo (UC3M - GUTI - IDTestingLab) 4 
• BlackBerry 

Playbook 

• Tablet

• 

Tablet Capacitive 

• 7” 

• Samsung Galaxy 

Note (Note-F) 

• Smartphone 

• Mixed 

• 5.3” 

• 

• 

• 

• 

Raul Sanchez-Reillo (UC3M - GUTI - IDTestingLab) 5 
Raul Sanchez-Reillo (UC3M - GUTI - IDTestingLab) 
 All devices 

 Crew: 11 people 

Age: 24 24  – 39 years old

 Other data: Familiar with the technology. Engineers. 

 Sessions: 3  Signatures/session: 20 Signatures/session: 20  All users’ real signatures  Skilled forgeries: 

 Advanced knowledge about the signature to forge Advanced knowledge about the signature to forge  2 x 10 forgeries/user 

 Target: Obtain a preliminary idea on the different paramete

on the different paramete

may affect going towards mobility 

 Algorithm used: 

 DTW -based Only X and Y signals (no pressure!)  Enrolment with the 3 first signatures (not the “typical best” 5)  Genuine Signatures (Random Forgeries): EER = 1.8% (MCYT)  Skilled Forgeries: EER = 7.6% (MCYT) 

Raul Sanchez-Reillo (UC3M - GUTI - IDTestingLab) 8 
 Yes! 

 STU (visual STU (visual fb, EER 1 27%) vs EER=1,27%) vs. Intuos (non-vis EERs for different users: 

 31 79%. STU vs Intuos STU vs Intuos (stylus) (sty lus) 100  12.14% 

0 5 10 15 20 25 30 35 40 

Impostor Attempts Accepted = FAR (%) 

95 

 28.03%  35.04%  19.05%  47 18% .  40.17%  33.16%  56.07% 

Genuine Attempts Accepted = 1-FRR (%) 90 85 80 75 70 65 

60 

 22.90% 55  12.14% 5050

Raul Sanchez-Reillo (UC3M - GUTI - IDTestingLab) 9 
100 

98 96 94 

Raul Sanchez-Reillo (UC3M - GUTI IDTestingLab)

Note stylus vs STU vs Asus 

Genuine Attempts Accepted = 1-FRR (%) 

0 2 4 6 8 10 12 

Impostor Attempts Accepted = FAR (%)-

88

88 92 90 

10

Genuine Attempts Accepted = 1-FRR (%) 94 92 90 

0 2 4 

Impostor Attemp

88

88 

 Technology = Capture technology + capture size + O.S.  Technology matters: 

 With Stylus (EER): With Stylus (EER): 

 Note-S = 0.17; STU = 1.27; ASUS = 3.48

 With finger (EER): 

 Note-F = 0.29; iPad = 0.47; Playbook = 1.39; Tab = 2.38

 Not really sure about which aspect of technology makes the depend

Not really sure about which aspect of technology makes the depend

 Not found a relationship with size (from a minimum size of the Note) or O. About O.S.: which version? Which implementation? 

100 

98 96 

Note finger vs iP
11

STU Template v

s Note stylus vs As

100 

95 90 85 80 

75

750 5 10 15 

Impostor Attempts Accepted = FAR 

Genuine Attempts Accepted = 1-FRR (%) 

 STU as 

reference (office scenario) 

 RESULTS: RESULTS: 

lower performance than with it’ than with it’s own pattern, but acceptable: 

 Note: 

 0.17 

 0.98 

 Asus: 

 3.48 

 6.84 
 Note-F for reference (best result intra-dev

 Tab: 

 2.38  1.45 !!

 iPad: 

 0.47  2.21

 Playbook: 

 1.39  0.93 !!

12

Note finger Template vs iPad vs Pla

100 

NiNNT

98 

Genuine Attempts Accepted = 1-FRR (%) 96 94 92 90 

8888

0 2 4 6 8 

Impostor Attempts Accepted 

= 
 iPad for reference (second best): 

 Tab: 

 2.38  3.06

 Note -F: 

 0.29  3.52

 Playbook: 

 1.39  1.91

ted = 1-FRR (%) 

13

iPad Template vs Note finger vs Play

100 

98 

Genuine Attempts Accep

96 94 92 90 

8888

0 2 4 6 8 

Impostor Attempts Accepted = 

F
 Playbook for reference: 

 Tab: 

 2.38  2.55

 Note -F: 

 0.29  3.65

 iPad: 

 0.47  2.25

Tab:

14

Playbook Template vs Note finger vs 

100 

98 

Genuine Attempts Accepted = 1-FRR (%) 96 94 92 90 

8888

0 2 4 6 8 

Impostor Attempts Accepted = 

F
100 

99 98 97 

Raul Sanchez-Reillo (UC3M - GUTI IDTestingLab)

Note finger vs Note stylus 

Finger Stylus 

0 1 2 3 4 5 6 

Genuine Attempts Accepted = 1-FRR (%) 

94 94 

Impostor Attempts Accepted = FAR (%) -

96 95 

15

% Genuine Attempts Accepted = 1-FRR ( 

99 98.5 98 97.5 

9797 

0 0.5 1 1

Impostor Attempts 

 Comparing within the same device (Note): 

 EER Stylus = 0 17; EER Finger = 0 29 EER Stylus = 0.17; EER Finger = 0.29 

 Comparing “inter-modality” 

 Enrolling with Stylus: 0 34; Enrolling with Finger: 

Enrolling with Stylus: 0.34; Enrolling with Finger: 

 Preliminary Result: Not the same, but compa

Note finger Template 

100 

)

99.5 
 Note-Stylus for reference: 

 T b : a

 2.38  0.55 !! 

 Ntoe-F: 

 0.29  0.34 

16

 Playbook: 

 1.39  1.10 !! 

 iPad: 

 0.47  1.45 

Note stylus Template vs Note finger vs iPa

100 

99 

Genuine Attempts Accepted = 1-FRR (%) 98 97 96 95 

9494

0 1 2 3 

Impostor Attempts Accepted 

=
 iPad for reference: 

 N t o e-Stylus: 

iPad Template vs Note stylus vs ST

 0.17  1.40 

 Asus: 

17

 3.48  7.30 

 STU: 

 1.27  4.80 

100 

Genuine Attempts Accepted = 1-FRR (%) 95 90 85 80 

75750 5 

 F

10 15 

Impostor Attempts Accepted = 
Raul Sanchez-Reillo (UC3M - GUTI IDTestingLab) 18

95.5

Supra-Identity user:ALL 

100 

99 98 97 96 95 

Genuine Att 94empts Accepted = 1-FRR (%) 

-

0 5 10 15 20 25 30 35 40 45 

Impostor Attempts Accepted = FAR (%) 

9393

 Supra-Identity: 

 Each user has 7 different biometric references Each user has 7 different biometric references  Best score is used  EER = 1.66 
19

 Visual Feedback is a major parameter for performance!  No relationship with technology has been found for differ

behaviours behaviours 

 Within genuine samples, interoperability is affordable: 

 Within stylus devices is really good  Within finger Within finger -based devices in affordable

 Improvement of EER in Tab or Playbook, having the template with Note

potential problem in enrolling with Tab or Playbook.

 Habituation? Enrolling with samples 10, 11 and 12 improves EER: 2.3

(Figure below)

 But the problem still exists!

 Results initially shows that a single modality can be consider

 Interoperability is acceptable 

 Enrolling with stylus results may even get better

 Supra-identity shows really good results: 

 But enrolling is tiring for users  And what about future devices being used? And what about future devices being used? 

94

Tab Template:10,11,12 

100 

99 

Genuine Attempts Accepted = 1-FRR (%) 98 97 96 

95 

0 5 

94 10 15

Impostor Attempts Accepted = 
 For each of the stylus-based devices: 

 Note

-Stylus: Stylus: 

 0.17 

 10.1

20

 Asus: 

 3.48 

 19.0

 STU: 

 1.27 

 11.1

Note stylus vs STU vs Asus (skilled 

100 

95 90 

Genuine Attempts Accepted = 1-FRR (%) 85 80 75 70 65 60 

55 

5050

0 5 10 

 F

15 20 25 30 

Impostor Attempts Accepted = 
 For each of the finger-based devices: 

 Tab: 

 2.38 

 17.18

21

 Note 

-F: 

 0.29 

 10.10

 Pla

ybook: 

 1.39 

 9.25

 iPad: 

 0.47 

 7.99

iPad vs Note finger vs Playbook vs Tab 

100 

95 

Genuine Attempts Accepted = 1-FRR (%) 90858075 70 65 60 

55 

5050 0 5 10 

= 

15 20 25 30 

Impostor Attempts Accepted 
 “Intra-modality” interoperability for stylu

based devices (enrolling with STU): based devices (enrolling with STU): 

 Note-Stylus: STU template vs Note stylus vs Asus (s

100 

 0 98. 

 22 9. 

95 

22

 Asus: 

 6.84 

 28.2 

Genuine Attempts Accepted = 1-FRR (%) 90 85 80 75 70 65 60 

55 

50500 5 10 15 20 25 30 

Impostor Attempts Accepted = F
23

100 

iPad template vs Note finger vs Playbook vs 

95 

0 5 10 

5050 

Genuine Attempts Accepted = 1-FRR (%) 90 8580 75 70 65 60 

55 

= 

15 20 25 30 

Impostor Attempts Accepted 

 “Intra-modality” interoperability for finge

based devices (enrolling with iPad): 

 Tab: 

 3.06  29.99 

 Note-F: 

 3.52 

 30.85

 Playbook: 

 1.91 

 30.07
24

 “Inter-modality” interoperability (Note): 

 Reference Note-F: 

 Note-S: 0.98  11.81 

 Reference Note-S: 

 Note-F: 3.52  19.97 

Note finger Template vs Note stylus Template (skilled forgeries) 

100 

95 90 85 

80 75 70 65 60 55 

5050

finger-stylus (sf) stylus-stylus (sf) st l fi f) tylus-finger (sf) finger-finger (sf) 

0 5 10 15 20 25 30 35 40 45 50 

Impostor Attempts Accepted = FAR (%) 

Genuine Attempts Accepted = 1-FRR (%) 
25

50

100 

95 90 85 80 75 70 65 60 55 

5050

0 

Supra-Identity user:ALL (skilled fo

5 10 15 20 25 30 

Impostor Attempts Accepted = F

Genuine Attempts Accepted = 1-FRR (%) 

 Supra-identity: 

 1 66.  15 33 . 
 “Office enrolment / Multiple device 

verification verification scenario (enrolment with ST

” scenario (enrolment with ST

STU Template vs ALL (skilled fo

100 

 Note-Stylus: 

 22.93 

95 

 Asus: 

90 

26

 28.21 

Genuine Attempts Accepted = 1-FRR (%) 

 iPad: 85 

 22.61 80 

 Note-Fin

ger: 75 

 24.55 

70 

 Playbook: 

65

 17 17 . 

60

 Tab: 

 22.53 55

50 50



0 5 10 15 20 25 30 

Impostor Attempts Accepted = F
 Visual Feedback is a major parameter for performance! 

 No relationship with technology has been found for differ

behaviours behaviours 

 Within genuine samples, interoperability is affordable: 

 Within stylus devices is really good  Within finger Within finger -based devices in affordable

 Potential problem in some enrolments with fingers 

 Results initially shows that a single modality can be consider

 Interoperability is acceptable 

 Enrolling with stylus results may even get better 

 Supra-identity shows really good results: 

 But enrolling is tiring for users  A dn what about ff tuture dd evices bi b e ng used? 

 Skilled Forgeries kill modality performance 

 Worst that State-of-the-Art 

 May pressure provide the difference? 

 Intra-device (10% approx.); Intra-modality (>20% for stylus, 30% f

Supra-identity (15.33%); Typical scenario (about 20%) 

 The skilled forgeries were obtained with high knowledge on the u

signature signature 

 Improvement on algorithm? Or real problem with advanced forgery? 

Raul Sanchez-Reillo (UC3M - GUTI - IDTestingLab) 27 
Raul Sanchez-Reillo (UC3M - GUTI - IDTestingLab) 
 Devices: 

 ASUS (Tablet PC)  STU-500 (Peripheral – Reference Device)  iPad (Tablet)  HTC Desire (Smartphone) 

 Crew: 15 people Crew: 15 people 

 Age: 16 – 60 years old  Other data: Broad-spectrum in familiarity with technology, habit

sign, etc. 

 Sessions: 1  Signatures/session: 12  All genuine samples (NO skilled forgeries)  All users’ real signatures  5 Scenarios  Target: Obtain a preliminary idea on the influence of the po

signing 

 First step towards a full usability test (based on HBSI) 

 S l i h Same algorithm used

Raul Sanchez-Reillo (UC3M - GUTI - IDTestingLab) 29 
03 04 0

(Asus,HTC HTC, STU) (iPad, STU) STU) STU) (HTC, 

01 02 

Reference Scenario (HTC, iPad) 

(All devices) (All devices) 

Raul Sanchez-Reillo (UC3M - GUTI - IDTestingLab) 30 
ab) 31

HTC (Usability) 

100 

Genuine Attempts Accepted = 1-FRR (%) 95 90 85 80 

75750 5 10 15 

Impostor Attempts Accepted = FAR 

100 

95 90 

Raul Sanchez-Reillo (UC3M - GUTI IDTestingL

Asus (Usability) 

Asus-01 Asus-03 

Genuine Attempts Accepted = 1-FRR (%) 

75750 5 10 15 20 25

Impostor Attempts Accepted = FAR (%) -

85 80 

 Comparing enrolling with its own scenario (same devic

 ASUS: 

 S i 01 EER 9 Scenario 01: EER = 9.62 ( 3 48) .  Scenario 03: EER = 4.37

 HTC: 

 Scenario 01: EER = 55 Scenario 01: EER .92 ( 2.38 for Tab) 38 for Tab)  Scenario 02: EER = 8.87 Scenario 03: EER = 7.35 Scenario 05: EER = 5.19
100 

95 90 

Raul Sanchez-Reillo (UC3M - GUTI IDTestingLab)

iPad (Usability) 

iPad - 01 iPad - 02 iPad - 04 iPad - 05 

Genuine Attempts Accepted = 1-FRR (%) 

7575 0 5 10 15 20 25 

Impostor Attempts Accepted = FAR (%) -

85 80 

32

STU (Usability) 

100 

Genuine Attempts Accepted = 1-FRR (%) 95 90 85 80 

5750 5 10 15 

Impostor Attempts Accepted = FAR 

 Comparing enrolling with its own scenario (same devic

 iPad: 

 S i 01 EER 4 Scenario 01: EER = 4.44 ( 0 47) .  Scenario 02: EER = 2.23  Scenario 04: EER = 0.14 !!  Scenario 05: EER = 5.84 

 STU: 

 Scenario 01: EER = 5.92 ( 1.27)  Scenario 03: EER = 2.35  Scenario 04: EER = 9.60 

7
100 

95 90 

Raul Sanchez-Reillo (UC3M - GUTI IDTestingLab) 33

 Comparing enrolling with scenario 01 (same devic

 ASUS: 

 Scenario 01: EER = 9.62  Scenario 03: EER = 4.37  8.85 

 HTC: 

Scenario 01: EER EER = 55  01 .92

 Scenario 02: EER = 8.87  5.92 !!  Scenario 03: EER = 7.35  5.96 !!  Scenario 05: EER = 5 19 Scenario 05: EER = 5.19  6 60 . 

Asus, Template 01 (Usability) HTC, Template 01 (Usability) 100 

Asus 01- 01 epted = 1-FRR (%) Asus 01 Asus 01- 03 

nuine Attempts Acce Gen 

Genuine Attempts Accepted = 1-FRR (%) 95 90 

85 

80 80 

7575 7575 0 5 10 15 20 25 0 5 10 15 

Impostor Attempts Accepted = FAR (%) - Impostor Attempts Accepted = FAR 

85 
95 

90 

Raul Sanchez-Reillo (UC3M - GUTI IDTestingLab)

iPad, Template 01 (Usability) 

100 

iPad 01- 01 iPad 01- 02 iPad 01- 04 iPad 01- 05 

Genuine Attempts Accepted = 1-FRR (%) 80 

75750 5 10 15 20 25 

Impostor Attempts Accepted = FAR (%) -

85 

34

 Comparing enrolling with scenario 01 (same device): 

 iPad: 

 S i 01 EER 4 Scenario 01: EER = 4.44  Scenario 02: EER = 2.23  7.42 Scenario 04: EER = 0.14  7.42 Scenario 05: EER = 5.84  7.44

 STU: 

 Scenario 01: EER = 5.92 Scenario 03: EER = 2.35  5.19 Scenario 04: EER = 9.60  5.92 !!

STU, T

100

Genuine Attempts Accepted = 1-FRR (%) 95 90 85 80 

75750 5 1

Impostor Atte
90 

85 80 75 

Raul Sanchez-Reillo (UC3M - GUTI IDTestingLab) 35

 Comparing enrolling with STU (scenario 01): 

 ASUS: 

 Scenario 01: EER = 9.62  17.19  Scenario 03: EER = 4.37  20.75 

 HTC: 

 S i 01 EER 5 92 Scenario 01: EER = 5.92  13 14 .  Scenario 02: EER = 8.87  15.49  Scenario 03: EER = 7.35  13.94  Scenario 05: EER = 5 19 Scenario 05: EER = 5.19  17 04 . 

Asus, STU Template 01 (Usability) 

100 

95 

HTC, ST

100 

95 

STU-Asus01 STU-Asus03 

-

0 5 10 15 20 25 30 35 40 45 50 

Impostor Attempts Accepted = FAR (%) 

Genuine Attempts Accepted = 1-FRR (%) 

Genuine Attempts Accepted = 1-FRR (%) 90 85 80 

75 70 

55 55 

5050 50500 5 10 15 2

Impostor Att

65 60 

70 65 60 
 Comparing enrolling with STU (scenario 01): 

 iPad: 

 Scenario 01: EER = 4.44  14.81 Scenario 02: EER = 2.23  11.76 Scenario 04: EER = 0.14  12.60 Scenario 05: EER = 5.84  12.58

iPad, STU Template 0

Raul Sanchez-Reillo (UC3M - GUTI - IDTestingLab) 36

100 

95 90 85 80 75 

70 65 60 

Genuine Attempts Accepted = 1-FRR (%) 

55 

50500 5 10 

ce

15 20 25 

Impostor Attempts Ac 
 A change in the test crew have a major impact in 

performance 

 Smartphones seem to present a

 l

l ower dd ependenc

the scenario 

 Surprisingly the best results are achieved standin

device on a platform 

 Better rates than the reference scenario  May be due to habituation! 

 Results on the 2nd session may show an improvem

the error rates 

 Enrolment seems to present not stable signatures Enrolment seems to present not stable signatures  Could be improvement in training a solution? 

 Interoperability (both intra-device and inter-devi

show homogeneity on the show homogeneity on the error rates error rates 

 Definitely bad error rates 

 But, ... compared against what?  Same kind of evaluation with other modalities? 

Raul Sanchez-Reillo (UC3M - GUTI - IDTestingLab) 37 
 A lot! 

 At least all the highlighted text on each of th

At least all the highlighted text on each of thconclusions 

 Increasing size of test crews is a must! Increasing size of test crews is a must! 

 For both, Performance and Usability 

 Carrying out a real Usability evaluation Carrying out a real Usability evaluation 

 Following HBSI Methodology 

 This kind of studies will lead to the need o

This kind of studies will lead to the need ofurther improvement in Scenario and Operational Testing Operational Testing 

 May be new standards to arise in the followin

Raul Sanchez-Reillo (UC3M - GUTI - IDTestingLab) 38 
-

R Sanchez R. Sanchez -Reillo, R Blanco R. Blanco -Gonz

Gonz

Hurtado, A. Me

University Group for Identification Tec

Carlos III Uni

http

http://idte

rsre

Raul Sanchez-Reillo (UC3M - GUTI IDTestingLab)