1 

Tattoo Recognition Technology - Evaluation (Tatt-E) 

Tatt 

2 

3 

4 

5 

6 

7 

8 Tattoo Recognition Technology - Evaluation (Tatt-E) 

A Public Evaluation of Tattoo Recognition Algorithms 

Concept, Evaluation Plan, and API 

Version 1.0 

9 

10 

11 

12 

13 

14 Mei Ngan and Patrick Grother 

December 11, 2017 

15 

16 

17 

18 

NIST Concept and Evaluation Plan Page 1 of 24 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Tattoo Recognition Technology - Evaluation (Tatt-E) 

19 

20 

21 

Status of this Document 

This document is intended to be the final specification. Changes are marked in green. Comments 

and questions should be submitted to tatt-e@nist.gov. 

22 

23 

24 Timeline of the Tatt-E Activity 

API Development September 26, 2016 Draft evaluation plan available for public comments 

December 1, 2016 Final evaluation plan published 

Phase 1 December 1, 2016 Participation starts: Algorithms may be sent to NIST 

February March 1, 2017 Last day for submission of algorithms to Phase 1 

March April 14, 2017 Interim results released to Phase 1 participants 

Phase 2 May June 1, 2017 Last day for submission of algorithms to Phase 2 

June July 14, 2017 Interim results released to Phase 2 participants 

Phase 3 August September 15, 2017 Last day for submission of algorithms to Phase 3 

Q4 2017 Release of final public report 

25 

26 

27 Acknowledgements 

28 The organizers would like to thank the sponsor of this activity, the Federal Bureau of Investigation (FBI) 

29 Biometric Center of Excellence (BCOE) for initiating and progressing this work. 

30 

31 Contact Information 

32 Email: tatt-e@nist.gov 

33 Tatt-E Website: 

34 https://www.nist.gov/programs-projects/tattoo-recognition-technology-evaluation-tatt-e 

35 

NIST Concept and Evaluation Plan Page 2 of 24 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Tattoo Recognition Technology - Evaluation (Tatt-E) 

36 Table of Contents 

37 1. Tatt-E .........................................................................................................................................................4 

38 1.1 Background ................................................................................................ ..........................................4 

39 1.2 The Tattoo Recognition Technology Program................................................................ ......................4 

40 1.3 Scope ...................................................................................................................................................5 

41 1.4 Audience...............................................................................................................................................5 

42 1.5 Training Data ........................................................................................................................................5 

43 1.6 Offline Testing ......................................................................................................................................5 

44 1.7 Phased Testing................................ .....................................................................................................5 

45 1.8 Interim reports ......................................................................................................................................6 

46 1.9 Final reports................................................................................................ ..........................................6 

47 1.10 Application scenarios..........................................................................................................................6 

48 1.11 Rules for participation.........................................................................................................................7 

49 1.12 Number and schedule of submissions................................................................................................7 

50 1.13 Core accuracy metrics................................ ........................................................................................7 

51 1.14 Reporting template size................................................................................................ ......................7 

52 1.15 Reporting computational efficiency.....................................................................................................8 

53 1.16 Exploring the accuracy-speed trade-space ........................................................................................8 

54 1.17 Hardware specification ................................................................ .......................................................8 

55 1.18 Operating system, compilation, and linking environment ................................ ...................................8 

56 1.19 Runtime behavior .............................................................................................................................10 

57 1.20 Single-thread Requirement................................ ...............................................................................10 

58 1.21 Time limits ................................................................................................ ........................................10 

59 1.22 Ground truth integrity........................................................................................................................11 

60 2. Data structures supporting the API ..........................................................................................................12 

61 2.1 Data structures ................................ ...................................................................................................12 

62 2.2 File structures for enrolled template collection ...................................................................................15 

63 3. API Specification ......................................................................................................................................16 

64 3.1 Namespace ................................................................................................ ........................................16 

65 3.2 Overview.............................................................................................................................................16 

66 3.3 Detection and Localization (Class D) ................................................................ .................................16 

67 3.4 Identification (Class I).........................................................................................................................18 

68 Annex A Submissions of Implementations to Tatt-E .......................................................................................23 

69 A.1 Submission of implementations to NIST .........................................................................................23 

70 A.2 How to participate ................................................................................................ ...........................23 

71 A.3 Implementation validation ................................ ...............................................................................24 

72 

73 List of Tables 

74 Table 1 – Subtests supported under the Tatt-E activity ....................................................................................6 

75 Table 2 – Tatt-E classes of participation ................................ ...........................................................................7 

76 Table 3 – Cumulative total number of algorithms..............................................................................................7 

77 Table 4 – Implementation library filename convention ................................................................ ......................9 

78 Table 5 – Processing time limits (1 core) in seconds, per 640 x 480 image ...................................................11 

79 Table 6 – Enrollment dataset template manifest .............................................................................................15 

80 Table 7 – Procedural overview of the detection and localization test..............................................................16 

81 Table 8 – Procedural overview of the identification test ..................................................................................18 

82 

NIST Concept and Evaluation Plan Page 3 of 24 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

																																																																		

 

 

 

 

 

 

 

 

 

 

98 

Tattoo Recognition Technology - Evaluation (Tatt-E) 

83 1. Tatt-E 

84 1.1 Background 

85 Tattoos have been used for many years to assist law enforcement in the identification of criminals and 

86 victims and for investigative research purposes. Historically, law enforcement agencies have followed the 

87 ANSI/NIST-ITL 1-20111 standard to collect and assign keyword labels to tattoos. This keyword labeling 

88 approach comes with drawbacks, which include the limited number of ANSI/NIST standard class labels to 

89 able describe the increasing variety of new tattoo designs, the need for multiple keywords to sufficiently 

90 describe some tattoos, and subjectivity in human annotation as the same tattoo can be labeled differently by 

91 examiners. As such, the shortcomings of keyword-based tattoo image retrieval have driven the need for 

92 automated image-based tattoo recognition capabilities. 

93 1.2 The Tattoo Recognition Technology Program 

94 The Tattoo Recognition Technology Program was initiated by NIST to support an operational need for 

95 image-based tattoo recognition to support law enforcement applications. The program provides quantitative 

96 support for tattoo recognition development and best practice guidelines. Program activities to date are 

97 summarized in Figure 1. 

Tattoo Recognition Technology Program 

Tatt-C 

“Open-book” challenge with 

provided dataset to engage research 

community to advance image-based 

tattoo matching technology 

Tatt-BP 

Best practice guidelines and 

training material for the collection 

of tattoo images 

Tatt-E 

Large-scale sequestered 

evaluation of tattoo recognition 

algorithms 

2014 - 2015 2015 - 2016 2016 - 2017 

“Open-book” test with 

provided dataset Best Practice Documents Sequestered Test 

Tatt-Q 

Automated 

image quality assessment 

Future - TBD 

Tatt-L 

Track public literature of 

tattoo recognition related 

publications 

Tatt-S 

Proposed revisions/additions 

to standards 

(e.g.ANSI/NIST) 

Tatt-A 

Tattoo ageing 

99 Figure 1 – Activities under the Tattoo Recognition Technology Program 

100 

101 • Tatt-C was an initial research challenge that provided operational data and use cases to the 

102 research community to advance research and development into automated image-based tattoo 

103 technologies and to assess the state-of-the-art. NIST hosted a culminating industry workshop and 

104 published a public report on the outcomes and recommendations from the Tatt-C activity. Please 

105 visit https://www.nist.gov/programs-projects/tattoo-recognition-technology-challenge-tatt-c for more 

106 information. 

107 

108 • Tatt-BP provides best practice guidance material for the proper collection of tattoo images to support 

109 image-based tattoo recognition. Recognition failure in Tatt-C was often related to the consistency 

110 and quality of image capture, and Tatt-BP aimed to provide guidelines on improving the quality of 

111 tattoo images collected operationally. Please visit https://www.nist.gov/itl/iad/image-group/tattoo112 recognition-technology-best-practices for more information. 

1 The latest version of the ANSI/NIST-ITL 1-2011 standard is available at https://www.nist.gov/programsprojects/ansinist-itl-standard. 

NIST Concept and Evaluation Plan Page 4 of 24 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Tattoo Recognition Technology - Evaluation (Tatt-E) 

113 

114 • Tatt-E is a sequestered evaluation intended to assess tattoo recognition algorithm accuracy and run115 time performance over a large-scale of operational data. The participation details of Tatt-E are 

116 established in this document, also available for download at https://www.nist.gov/programs117 projects/tattoo-recognition-technology-evaluation-tatt-e. 

118 1.3 Scope 

119 The Tattoo Recognition Technology – Evaluation (Tatt-E) is being conducted to assess and measure the 

120 capability of systems to perform automated image-based tattoo recognition. Both comparative and absolute 

121 accuracy measures are of interest, given the goals to determine which algorithms are most effective and 

122 viable for the following primary operational use-cases: 

123 

124 • Tattoo/Region of Interest Identification – matching different instances of the same tattoo image from 

125 the same subject over time. This includes matching with entire and/or partial regions of a tattoo. 

126 • Tattoo detection/localization – determining whether an image contains a tattoo and if so, 

127 segmentation of the tattoo. 

128 • Sketches – matching sketches to tattoo images. 

129 

130 Out of scope: Areas that are out of scope for this evaluation and will not be studied include: matching of 

131 tattoos based on thematically similar content as the definition of “similarity” is ill-defined; tattoo recognition in 

132 video. 

133 

134 This document establishes a concept of operations and an application programming interface (API) for 

135 evaluation of tattoo recognition implementations submitted to NIST’s Tattoo Recognition Technology – 

136 Evaluation. See https://www.nist.gov/programs-projects/tattoo-recognition-technology-evaluation-tatt-e for all 

137 Tatt-E documentation. 

138 1.4 Audience 

139 Any person or organizations with capabilities in any of the following areas are invited to participation in the 

140 Tatt-E test. 

141 • Tattoo matching implementations. 

142 • Tattoo detection and localization algorithms. 

143 • Algorithms with an ability to match sketches to tattoos. 

144 Participants will need to implement the API defined in this document. Participation is open worldwide. There 

145 is no charge for participation. NIST encourages submission of experimental prototypes as well as those that 

146 could be readily made operational. 

147 1.5 Training Data 

148 None of the test data can be provided to participants. Instead prospective participants should leverage public 

149 domain and proprietary datasets as available. 

150 1.6 Offline Testing 

151 While Tatt-E is intended as much as possible to mimic operational reality, this remains an offline test 

152 executed on databases of images. The intent is to assess the core algorithmic capability of tattoo detection, 

153 localization, and recognition algorithms. Offline testing is attractive because it allows uniform, fair, 

154 repeatable, and efficient evaluation of the underlying technologies. Testing of implementations under a fixed 

155 API allows for a detailed set of performance related parameters to be measured. 

156 1.7 Phased Testing 

157 To support development, Tatt-E will run in multiple phases. The final phase will result in the release of public 

158 reports. Providers should not submit revised algorithms to NIST until NIST provides results for the prior 

159 phase. 

NIST Concept and Evaluation Plan Page 5 of 24 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

																																																																		

 

 

Tattoo Recognition Technology - Evaluation (Tatt-E) 

160 For the schedule and number of algorithms of each class that may be submitted for each class, see section 

161 1.12. 

162 1.8 Interim reports 

163 The performance of each implementation in phase 1 and 2 will be reported in a "report card". This will be 

164 provided to the participant. It is intended to facilitate research and development, not for marketing. Report 

165 cards will: be machine generated (i.e. scripted); be provided to participants with coded identification of their 

166 implementation; include timing, accuracy, and other performance results; include results from other 

167 implementations, but will not identify the other providers; be expanded and modified as revised 

168 implementations are tested and as analyses are implemented; be produced independently of the status of 

169 other providers’ implementations; be regenerated on-the-fly, usually whenever any implementation 

170 completes testing, or when new analysis is added. 

171 1.9 Final reports 

172 NIST will publish one or more final public reports. NIST may also publish: additional supplementary reports 

173 (typically as numbered NIST Interagency Reports); in academic journal articles; in conferences and 

174 workshops (typically PowerPoint). 

175 Our intention is that the final test reports will publish results for the best-performing implementation from 

176 each participant. Because “best” is underdefined (accuracy vs. time, for example), the published reports 

177 may include results for other implementations. The intention is to report results for the most capable 

178 implementations (see section 0 on metrics). Other results may be included (e.g. in appendices) to show, for 

179 example, illustration of progress or tradeoffs. 

180 IMPORTANT: All Phase 3 results will be attributed to the providers, publicly associating performance with 

181 organization name. 

182 1.10 Application scenarios 

183 As described in Table 1, the test is intended to represent: 

184 • Use of tattoo recognition technologies in search applications in which the enrolled dataset could 

185 contain images in the hundreds of thousands. 

186 • Tattoo detection and localization with zero or more tattoos in the sample. 

187 

188 Table 1 – Subtests supported under the Tatt-E activity 

# Class label D I 

1. Aspect Detection and Localization 1:N Search 

2. Enrollment dataset None, application to single images N enrolled subjects 

3. Prior NIST test references For detection task, see Detection 

in Tatt-C 20152 See Tattoo Identification, Region of Interest, and Mixed Media matching 

from Tatt-C 20152 

4. Example application Database construction and 

maintenance of large amounts of 

unlabeled, comingled data, e.g. 

given a pile of seized media, 1. 

Detect whether/which images 

contain tattoos and 2. Segment 

tattoos as pre-processing step for 

search against a database. 

Open-set search of a tattoo/sketch 

image against a central tattoo 

database, e.g. a search of a tattoo, 

parts of a tattoo, or a sketch of a 

tattoo against a tattoo database of 

known criminals. 

5. Number of images Variable Enrollment gallery: Up to O(105) 

6. Number of images per 

individual 

N/A Variable: one or more still tattoo 

images 

7. Enrollment image types Tattoo and non-tattoo images Tattoos 

2 See the Tatt-C test report: NIST Interagency Report 8078, linked from https://www.nist.gov/programs-projects/tattoorecognition-technology-challenge-tatt-c 

NIST Concept and Evaluation Plan Page 6 of 24 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Tattoo Recognition Technology - Evaluation (Tatt-E) 

8. Probe image types N/A Tattoos and sketches 

189 

190 1.11 Rules for participation 

191 There is no charge to participate in Tatt-E. A participant must properly follow, complete, and submit the 

192 

193 once, after December 1, 2016. It is not necessary to do this for each submitted software library. 

194 • All participants shall submit at least one class D (detection and localization) algorithm. 

Participation Agreement contained in this document published on the Tatt-E website. This must be done 

195 • Class I (identification) algorithms may be submitted only if at least 1 class D algorithm is also 

196 submitted. 

197 • All submissions shall implement exactly one of the functionalities defined in Table 2. A library shall 

198 not implement the API of more than one class (separate libraries shall be submitted to participate in 

199 separate participation classes). 

200 Table 2 – Tatt-E classes of participation 

Function Detection and Localization Identification 

Class label D I 

Co-requisite class None D 

API requirements 3.3 3.4 

201 

202 

203 

204 

205 

206 

1.12 Number and schedule of submissions 

The test is conducted in three phases, as scheduled on page 2. The maximum total (i.e. cumulative) number 

of submissions is regulated in Table 3. Participation in Phase 1 is not required for algorithm submission in 

Phase 2 and 3. 

Table 3 – Cumulative total number of algorithms 

# Phase 1 Total over Phases 1 + 2 Total over Phases 1 + 2 + 3 

All classes of participation 2 4 6 if at least 1 was successfully 

executed by end of Phase 1 

2 otherwise 

207 

208 

209 

210 

211 

212 

213 

214 

215 

216 

217 

218 

219 

220 

221 

222 

223 

224 

The numbers above may be increased as resources allow. 

1.13 Core accuracy metrics 

For identification testing, the test will target open-universe applications such as searching tattoo databases of 

known criminals (where the subject may or may not exist in the gallery) and closed-set tasks where subject is 

known to be in the database, e.g. in prison or corrections environments. Both score-based and rank-based 

metrics will be considered. Rank-based metrics are appropriate for one-to-many applications that employ 

human examiners to adjudicate candidate lists. Score based metrics are appropriate for cases where 

transaction volumes are too high for human adjudication or when false alarm rates must otherwise be low. 

Metrics include, false positive and negative identification rate (FPIR and FNIR) and cumulative match 

characteristic that can depend on threshold and rank. 

For detection and localization, assessments of overlap between detected and examiner-determined tattoo 

area will be considered along with score-based metrics including false positive and negative detection rate. 

1.14 Reporting template size 

Because template size is influential on storage requirements and computational efficiency, this API supports 

measurement of template size. NIST will report statistics on the actual sizes of templates produced by tattoo 

recognition implementations submitted to Tatt-E. NIST may also report statistics on runtime memory and 

other compute-performance characteristics. 

NIST Concept and Evaluation Plan Page 7 of 24 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Tattoo Recognition Technology - Evaluation (Tatt-E) 

225 1.15 Reporting computational efficiency 

226 As with other tests, NIST will compute and report accuracy. In addition, NIST will also report timing statistics 

227 for all core functions of the submitted API implementations. This includes feature extraction and 1:N 

228 matching. For an example of how efficiency might be reported, see the final report of the FRVT 2013 test3. 

229 1.16 Exploring the accuracy-speed trade-space 

230 NIST will explore the accuracy vs. speed tradeoff for tattoo recognition algorithms running on a fixed 

231 platform. NIST will report both accuracy and speed of the implementations tested. While NIST cannot force 

232 submission of "fast vs. slow" variants, participants may choose to submit variants on some other axis (e.g. 

233 "experimental vs. mature") implementations. 

234 1.17 Hardware specification 

235 NIST intends to support highly optimized algorithms by specifying the runtime hardware. There are several 

236 types of computers that may be used in the testing. The following list gives some details about possible 

237 compute architectures: 

238 • Dual Intel Xeon X5680 3.3 GHz CPUs (6 cores each) 

239 • Dual Intel Xeon X7560 2.3 GHz CPUs (8 cores each) 

240 • Dual Intel Xeon E5-2695 3.3 GHz CPUs (14 cores each; 56 logical CPUs total) with Dual NVIDIA 

241 Tesla K40 GPUs 

242 Each CPU has 512K cache. The bus runs at 667 Mhz. The main memory is 192 GB Memory as 24 8GB 

243 modules. We anticipate that 16 processes can be run without time slicing, though NIST will handle all 

244 multiprocessing work via fork(). Participant-initiated multiprocessing is not permitted. 

245 NIST is requiring use of 64-bit implementations throughout. This will support large memory allocation to 

246 support 1:N identification tasks. Note that while the API allows read access of the disk during the 1:N 

247 search, the disk is relatively slow, and I/O will be included in your run time. 

248 All GPU-enabled machines will be running CUDA version 7.5. cuDNN v5 for CUDA 7.5 will also be installed 

249 on these machines. Implementations that use GPUs will only be run on GPU-enabled machines. 

250 1.18 Operating system, compilation, and linking environment 

251 The operating system that the submitted implementations shall run on will be released as a downloadable file 

252 accessible from http://nigos.nist.gov:8080/evaluations/CentOS-7-x86_64-Everything-1511.iso, which is the 

253 64-bit version of CentOS 7.2 running Linux kernel 3.10.0. 

254 For this test, Windows machines will not be used. Windows-compiled libraries are not permitted. All software 

255 must run under CentOS 7.2. 

256 NIST will link the provided library file(s) to our C++ language test drivers. Participants are required to provide 

257 their library in a format that is dynamically-linkable using the C++11 compiler, g++ version 4.8.5. 

258 A typical link line might be 

259 g++ -std=c++11 -I. -Wall -m64 -o tatte tatte.cpp -L. –ltatte_Company_D_07 

260 The Standard C++ library should be used for development. The prototypes from this document will be 

261 written to a file "tatte.h" which will be included via 

#include <tatte.h> 

262 

263 

264 

265 

266 

The header files will be made available to implementers via https://github.com/usnistgov/tattoo. 

All compilation and testing will be performed on x86_64 platforms. Thus, participants are strongly advised to 

verify library-level compatibility with g++ (on an equivalent platform) prior to submitting their software to NIST 

to avoid linkage problems later on (e.g. symbol name and calling convention mismatches, incorrect binary file 

formats, etc.). 

																																																																		

3 See the FRVT 2013 test report: NIST Interagency Report 8009, linked from http://face.nist.gov/frvt 

NIST Concept and Evaluation Plan Page 8 of 24 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Tattoo Recognition Technology - Evaluation (Tatt-E) 

267 Any and all dependencies on external dynamic/shared libraries not provided by CentOS 7.2 as part of the 

268 built-in “development” package must be provided as a part of the submission to NIST. 

269 1.18.1 Library and Platform Requirements 

270 Participants shall provide NIST with binary code only (i.e. no source code). The implementation should be 

271 submitted in the form of a dynamically-linked library file. 

272 The core library shall be named according to Table 4. Additional dynamic libraries may be submitted that 

273 support this “core” library file (i.e. the “core” library file may have dependencies implemented in these other 

274 libraries). 

275 Intel Integrated Performance Primitives (IPP) ® libraries are permitted if they are delivered as a part of the 

276 developer-supplied library package. It is the provider’s responsibility to establish proper licensing of all 

277 libraries. The use of IPP libraries shall not prevent run on CPUs that do not support IPP. Please take note 

278 that some IPP functions are multithreaded and threaded implementations are prohibited. 

279 NIST will report the size of the supplied libraries. 

280 

281 Table 4 – Implementation library filename convention 

Form libTattE_provider_class_sequence.ending 

Underscore 

delimited parts of 

the filename 

libTattE provider class sequence ending 

Description First part of the 

name, required to 

be this. 

Single word name 

of the main 

provider 

EXAMPLE: Choice 

Function classes 

supported in Table 

2. 

EXAMPLE: D 

A two digit decimal 

identifier to start at 00 

and increment by 1 

every time a library is 

sent to NIST. 

EXAMPLE: 07 

.so 

Example libTattE_Choice_D_07.so 

282 

283 1.18.2 Configuration and developer-defined data 

284 The implementation under test may be supplied with configuration files and supporting data files. NIST will 

285 report the size of the supplied configuration files. 

286 1.18.3 Submission folder hierarchy 

287 Participant submissions should contain the following folders at the top level 

288 • lib/ - contains all participant-supplied software libraries 

289 • config/ - contains all configuration and developer-defined data 

290 • doc/ - contains any participant-provided documentation regarding the submission 

291 • validation/ - contains validation output 

292 1.18.4 Installation and Usage 

293 The implementation shall be installable using simple file copy methods. It shall not require the use of a 

294 separate installation program and shall be executable on any number of machines without requiring 

295 additional machine-specific license control procedures or activation. The implementation shall not use nor 

296 enforce any usage controls or limits based on licenses, number of executions, presence of temporary files, 

297 etc. It shall remain operable with no expiration date. 

298 Hardware (e.g. USB) activation dongles are not acceptable. 

NIST Concept and Evaluation Plan Page 9 of 24 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Tattoo Recognition Technology - Evaluation (Tatt-E) 

299 1.18.5 Modes of operation 

300 Implementations shall not require NIST to switch “modes” of operation or algorithm parameters. For 

301 example, the use of two different feature extractors must either operate automatically or be split across two 

302 separate library submissions. 

303 1.19 Runtime behavior 

304 1.19.1 Interactive behavior, stdout, logging 

305 The implementation will be tested in non-interactive “batch” mode (i.e. without terminal support). Thus, the 

306 submitted library shall: 

307 - Not use any interactive functions such as graphical user interface (GUI) calls, or any other calls, 

308 which require terminal interaction e.g. reads from “standard input”. 

309 - Run quietly, i.e. it should not write messages to "standard error" and shall not write to “standard 

310 output”. 

311 - Only if requested by NIST for debugging, include a logging facility in which debugging messages are 

312 written to a log file whose name includes the provider and library identifiers and the process PID. 

313 Please do not enable this by default. 

314 1.19.2 Exception Handling 

315 The application should include error/exception handling so that in the case of a fatal error, the return code is 

316 still provided to the calling application. 

317 1.19.3 External communication 

318 Processes running on NIST hosts shall not affect the runtime environment in any manner, except for memory 

319 allocation and release. Implementations shall not write any data to external resource (e.g. server, file, 

320 connection, or other process), nor read from such. If detected, NIST will take appropriate steps, including but 

321 not limited to, cessation of evaluation of all implementations from the supplier, notification to the provider, 

322 and documentation of the activity in published reports. 

323 1.19.4 Stateless behavior 

324 All components in this test shall be stateless. Thus, all functions should give identical output, for a given 

325 input, independent of the runtime history. NIST will institute appropriate tests to detect stateful behavior. If 

326 detected, NIST will take appropriate steps, including but not limited to, cessation of evaluation of all 

327 implementations from the supplier, notification to the provider, and documentation of the activity in published 

328 reports. 

329 1.20 Single-thread Requirement 

330 Implementations must run in single-threaded mode, because NIST will parallelize the test by dividing the 

331 workload across many cores and many machines simultaneously. 

332 1.21 Time limits 

333 The elemental functions of the implementations shall execute under the time constraints of Table 5. These 

334 time limits apply to the function call invocations defined in Table 5. Assuming the times are random 

335 variables, NIST cannot regulate the maximum value, so the time limits are 90-th percentiles. This means 

336 that 90% of all operations should take less than the identified duration. 

337 The time limits apply per image. When K tattoo images of a subject are present, the time limits shall be 

338 increased by a factor K. To allow for diversity of algorithms, the time limit to conduct a search has been 

339 increased (see table below). NIST will explore the accuracy vs. speed tradeoff for tattoo recognition 

340 algorithms running on a fixed platform. Both accuracy and speed of the implementations tested will be 

NIST Concept and Evaluation Plan Page 10 of 24 
 

 

 

 

 

 

 

 

 

 

 

 

 

																																																																		

Tattoo Recognition Technology - Evaluation (Tatt-E) 

reported. A number of methods on performing efficient search are available in the literature4 341 , and sub-linear 

342 search implementations are encouraged. 

343 Table 5 – Processing time limits (1 core) in seconds, per 640 x 480 image 

D I 

Function Detection and 1:N identification 

Localization 

Feature extraction for enrollment and 

identification 

5 

5 

Identification of one search template against 

100,000 single-image tattoo records. 

16 

300 

Enrollment finalization of 100,000 single-image 

tattoo records (including disk IO time) 

720 

344 1.22 Ground truth integrity 

345 Some of the test data is derived from operational systems and may contain ground truth errors in which 

346 ― a single tattoo is present under two different identifiers, or 

347 ― two different tattoos are present under one identifier, or 

348 ― in which a tattoo is not present in the image. 

349 If these errors are detected, they will be removed. NIST will use aberrant scores (high impostor scores, low 

350 genuine scores) to detect such errors. This process will be imperfect, and residual errors are likely. For 

351 comparative testing, identical datasets will be used and the presence of errors should give an additive 

352 increment to all error rates. For very accurate implementations this will dominate the error rate. NIST 

353 intends to attach appropriate caveats to the accuracy results. For prediction of operational performance, the 

354 presence of errors gives incorrect estimates of performance. 

355 

4 Jégou, H., Douze, M., & Schmid, C. (2011). Product quantization for nearest neighbor search. In IEEE PAMI. 

NIST Concept and Evaluation Plan Page 11 of 24 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Tattoo Recognition Technology - Evaluation (Tatt-E) 

356 2. Data structures supporting the API 

357 2.1 Data structures 

358 2.1.1 Overview 

359 In this test, a tattoo is represented by K 3 1 two-dimensional tattoo images. 

360 2.1.2 Data structures for encapsulating multiple images 

361 Some of the proposed datasets includes K > 2 same tattoo images per person for some persons. This 

362 affords the possibility to model a recognition scenario in which a new image of a tattoo is compared against 

363 all prior images. Use of multiple images per person has been shown to elevate accuracy over a single image 

364 for other biometric modalities. 

365 For tattoo recognition in this test, NIST will enroll K 3 1 images for each unique tattoo. Both enrolled gallery 

366 and probe samples may consist of multiple images such that a template is the result of applying feature 

367 extraction to a set of K 3 1 images and then integrating information from them. An algorithm might fuse K 

368 feature sets into a single model or might simply maintain them separately. In any case the resulting 

369 proprietary template is contained in a contiguous block of data. All identification functions operate on such 

370 multi-image templates. 

371 The number of images per unique tattoo will vary, and images may not be acquired uniformly over time. 

372 NIST currently estimates that the number of images K will never exceed 100. For the Tatt-E API, K of the 

373 same tattoo images of an individual are contained in data structure of Section 2.1.2.2. 

374 2.1.2.1 TattE::Image Struct Reference 

375 Struct representing a single image. 

376 Public Member Functions 

377 • Image () 

378 • Image (uint16_t widthin, uint16_t heightin, uint8_t depthin, ImageType typein, 

379 std::shared_ptr<uint8_t> datain) 

380 Public Attributes 

381 • uint16_t width 

382 Number of pixels horizontally. 

383 • uint16_t height 

384 Number of pixels vertically. 

385 • uint16_t depth 

386 Number of bits per pixel. Legal values are 8 and 24. 

387 • ImageType imageType 

388 Label describing the type of image. 

389 • std::shared_ptr<uint8_t> data 

390 Managed pointer to raster scanned data. Either RGB color or intensity. If image_depth == 24 

391 this points to 3WH bytes RGBRGBRGB... If image_depth == 8 this points to WH bytes IIIIIII. 

392 2.1.2.2 TattE::MultiTattoo Typedef Reference 

393 typedef std::vector< Image > MultiTattoo 

394 Data structure representing a set of the same tattoo images from a single person. 

395 2.1.3 Data Structure for detected tattoo 

396 Implementations shall return bounding box coordinates of each detected tattoo in an image. 

NIST Concept and Evaluation Plan Page 12 of 24 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Tattoo Recognition Technology - Evaluation (Tatt-E) 

397 2.1.3.1 TattE::BoundingBox Struct Reference 

398 Structure for bounding box around a detected tattoo. 

399 Public Member Functions 

400 • BoundingBox () 

401 • BoundingBox (uint16_t xin, uint16_t yin, uint16_t widthin, uint16_t heightin, double confin) 

402 Public Attributes 

403 • uint16_t x 

404 X-coordinate of top-left corner of bounding box around tattoo. 

405 • uint16_t y 

406 Y-coordinate of top-left corner of bounding box around tattoo. 

407 • uint16_t width 

408 Width, in pixels, of bounding box around tattoo. 

409 • uint16_t height 

410 Height, in pixels, of bounding box around tattoo. 

411 • double confidence 

412 Certainty that this region contains a tattoo. This value shall be on [0, 1]. The higher the 

413 value, the more certain. 

414 2.1.4 Class for representing a tattoo in a MultiTattoo 

415 2.1.4.1 TattE::TattooRep Class Reference 

416 Class representing a tattoo or sketch template from image(s) 

417 Public Member Functions 

418 • TattooRep () 

419 Default Constructor. 

420 • void addBoundingBox (const BoundingBox &bb) 

421 This function should be used to add bounding box entries for each input image provided to 

422 the implementation for template generation. If there are 4 images in the MultiTattoo vector, 

423 then the size of boundingBoxes shall be 4. boundingBoxes[i] is associated with 

424 MultiTattoo[i]. 

425 • std::shared_ptr< uint8_t > resizeTemplate (uint64_t size) 

426 This function takes a size parameter and allocates memory of size and returns a managed 

427 pointer to the newly allocated memory for implementation manipulation. This class will take 

428 care of all memory allocation and de-allocation of its own memory. The implementation shall 

429 not de-allocate memory created by this class. 

430 • const std::shared_ptr< uint8_t > getTattooTemplatePtr () const 

431 • uint64_t getTemplateSize () const 

432 This function returns the size of the template data. 

433 • std::vector< BoundingBox > getBoundingBoxes () const 

434 This function returns the bounding boxes for detected tattoos associated with the input 

435 images. 

436 Private Attributes 

437 • std::shared_ptr< uint8_t > tattooTemplate 

438 Proprietary template data representing a tattoo in images(s) 

439 • uint64_t templateSize 

440 Size of template. 

441 • std::vector< BoundingBox > boundingBoxes 

442 Data structure for capturing bounding boxes around the detected tattoo(s) 

NIST Concept and Evaluation Plan Page 13 of 24 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Tattoo Recognition Technology - Evaluation (Tatt-E) 

443 2.1.5 Data structure for result of an identification search 

444 All identification searches shall return a candidate list of a NIST-specified length. The list shall be sorted with 

445 the most similar matching entries listed first with lowest rank. 

446 2.1.5.1 TattE::Candidate Struct Reference 

447 Data structure for result of an identification search. 

448 Public Member Functions 

449 • Candidate () 

450 • Candidate (bool assignedin, std::string idin, double scorein) 

451 Public Attributes 

452 • bool isAssigned 

453 If the candidate is valid, this should be set to true. If the candidate computation failed, this 

454 should be set to false. 

455 • std::string templateId 

456 The template ID from the enrollment database manifest. 

457 • double similarityScore 

458 Measure of similarity between the identification template and the enrolled candidate. Higher 

459 scores mean more likelihood that the samples are of the same person. An algorithm is free 

460 to assign any value to a candidate. The distribution of values will have an impact on the 

461 appearance of a plot of false-negative and false-positive identification rates. 

462 2.1.6 Data Structure for return value of API function calls 

463 2.1.6.1 TattE::ReturnStatus Struct Reference 

464 A structure to contain information about the success/failure by the software under test. An object of this 

465 class allows the software to return some information from a function call. The string within this object can be 

466 optionally set to provide more information for debugging etc. The status code will be set by the function to 

467 Success on success, or one of the other codes on failure. 

468 Public Member Functions 

469 • ReturnStatus () 

470 • ReturnStatus (const TattE::ReturnCode code, const std::string info="") 

471 Create a ReturnStatus object. 

472 Public Attributes 

473 • TattE::ReturnCode code 

474 Return status code. 

475 • std::string info 

476 Optional information string. 

477 2.1.7 Enumeration Type Documentation 

478 2.1.7.1 enum TattE::ReturnCode[strong] 

479 Return codes for the functions specified by this API. 

480 Enumerator 

481 Success Success 

482 ConfigError Error reading configuration files 

483 ImageTypeNotSupported Image type, e.g., sketches, is not supported by the implementation 

484 RefuseInput Elective refusal to process the input 

NIST Concept and Evaluation Plan Page 14 of 24 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Tattoo Recognition Technology - Evaluation (Tatt-E) 

485 ExtractError Involuntary failure to process the image 

486 ParseError Cannot parse the input data 

487 TemplateCreationError Elective refusal to produce a template 

488 EnrollDirError An operation on the enrollment directory failed (e.g. permission, space) 

489 NumDataError The implementation cannot support the number of input images 

490 TemplateFormatError One or more template files are in an incorrect format or defective 

491 InputLocationError Cannot locate the input data - the input files or names seem incorrect 

492 VendorError Vendor-defined failure 

493 2.1.7.2 enum TattE::TemplateRole[strong] 

494 Labels describing the type/role of the template to be generated (provided as input to template 

495 generation) 

496 Enumerator 

497 Enrollment Enrollment template used to enroll into gallery 

498 Identification Identification template used for search 

499 2.1.7.3 enum TattE::ImageType[strong] 

500 Labels describing the image type. 

501 Enumerator 

502 Tattoo Tattoo image 

503 Sketch Sketch of tattoo 

504 Unknown Unknown or unspecified 

505 2.2 File structures for enrolled template collection 

506 An implementation converts a MultiTattoo into a template, using, for example the createTemplate() function 

507 of section 3.4.1.5.2. To support the Class I identification functions of Table 2, NIST will concatenate 

508 enrollment templates into a single large file, the EDB (for enrollment database). The EDB is a simple binary 

509 concatenation of proprietary templates. There is no header. There are no delimiters. The EDB may be 

510 hundreds of gigabytes in length. 

511 This file will be accompanied by a manifest; this is an ASCII text file documenting the contents of the EDB. 

512 The manifest has the format shown as an example in Table 6. If the EDB contains N templates, the manifest 

513 will contain N lines. The fields are space (ASCII decimal 32) delimited. There are three fields. Strictly 

514 speaking, the third column is redundant. 

515 Important: If a call to the template generation function fails, or does not return a template, NIST will include 

516 the Template ID in the manifest with size 0. Implementations must handle this appropriately. 

517 Table 6 – Enrollment dataset template manifest 

Field name Template ID Template Length Position of first byte in 

EDB 

Datatype required std::string Unsigned decimal 

integer 

Unsigned decimal integer 

Example lines of a manifest file 

appear to the right. Lines 1, 2, 3 

and N appear. 

90201744 1024 0 

Tattoo01 1536 1024 

7456433 512 2560 

... 

Tattoo12 1024 307200000 

518 

519 The EDB scheme avoids the file system overhead associated with storing millions of individual files. 

520 

NIST Concept and Evaluation Plan Page 15 of 24 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Tattoo Recognition Technology - Evaluation (Tatt-E) 

521 3. API Specification 

522 The function prototypes from this document and any other supporting code will be provided in a "tatte.h" file 

523 made available to implementers via https://github.com/usnistgov/tattoo. 

524 3.1 Namespace 

525 All data structures and API interfaces/function calls will be declared in the TattE namespace. 

526 3.2 Overview 

527 This section describes separate APIs for the core tattoo applications described in section 1.10. All 

528 submissions to Tatt-E shall implement the functions required by the rules for participation listed before Table 

529 2. Tatt-E participants shall implement the relevant C++ prototyped interfaces in this section. C++ was 

530 chosen in order to make use of some object-oriented features. 

531 3.3 Detection and Localization (Class D) 

532 This section defines an API for algorithms that can solely perform tattoo detection and localization. The 

533 detection task requires the implementation to detect whether an image contains a tattoo or not, and 

534 localization requires identifying the location of the tattoo within the image. Given an image, an 

535 implementation should 

536 • For detection, classify whether a tattoo was detected in the image or not and provide a real-valued 

537 measure of detection confidence on [0,1], with 1 indicating absolute certainty that the image 

538 contains a tattoo and 0 indicating absolute certainty that the image does not contain a tattoo. 

539 • For localization, report location(s) of one or more tattoos on different body locations in the form of a 

540 bounding box. 

541 Table 7 – Procedural overview of the detection and localization test Phase

Name Description Performance Metrics to be 

reported by NIST 

Detection and Localization 

Initialization initialize() 

Give the implementation the name of a directory where any 

provider-supplied configuration data will have been placed by 

NIST. This location will otherwise be empty. The implementation 

is permitted read-only access to the configuration directory. 

Detection detectTattoo() 

For each of N images, pass single images to the implementation 

for tattoo detection. The implementation will set a boolean 

indicating whether a tattoo was detected or not and a detection 

certainty confidence score. 

Multiple instances of the calling application may run 

simultaneously or sequentially. These may be executing on 

different computers. 

Statistics of detection times. 

Accuracy metrics. 

The incidence of where the 

implementation failed to 

perform detection (nonsuccessful return code). 

Localization localizeTattoos() 

For each of N tattoo images, pass single images to the 

implementation for tattoo localization. The implementation will 

populate a vector with bounding boxes corresponding to the 

tattoos detected from the input image. 

Multiple instances of the calling application may run 

simultaneously or sequentially. These may be executing on 

different computers. 

Statistics of the time needed 

for this operation. 

Accuracy metrics. 

The incidence of where the 

implementation failed to 

perform localization. 

NIST Concept and Evaluation Plan Page 16 of 24 

542 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Tattoo Recognition Technology - Evaluation (Tatt-E) 

543 3.3.1 TattE::DetectAndLocalizeInterface Class Reference 

544 The interface to Class D implementations. 

545 3.3.1.1 Public Member Functions 

546 • virtual ~DetectAndLocalizeInterface () 

547 • virtual ReturnStatus initialize (const std::string &configurationLocation)=0 

548 This function initializes the implementation under test. It will be called by the NIST 

549 application before any call to the functions detectTattoo and localizeTattoos(). 

550 • virtual ReturnStatus detectTattoo (const Image &inputImage, bool &tattooDetected, double 

551 &confidence)=0 

552 This function takes an Image as input and indicates whether a tattoo was detected in the 

553 image or not. 

554 • virtual ReturnStatus localizeTattoos(const Image &inputImage, std::vector< BoundingBox > 

555 &boundingBoxes)=0 

556 This function takes an Image as input, and populates a vector of BoundingBox with the 

557 number of tattoos detected on different body locations from the input image. 

558 3.3.1.2 Static Public Member Functions 

559 • static std::shared_ptr< DetectAndLocalizeInterface > getImplementation () 

560 Factory method to return a managed pointer to the DetectAndLocalizeInterface 

561 object. This function is implemented by the submitted library and must return a managed 

562 pointer to the DetectAndLocalizeInterface object. 

563 3.3.1.3 Detailed Description 

564 The interface to Class D implementations. 

565 The class D detection and localization software under test must implement the interface 

566 DetectAndLocalizeInterface by subclassing this class and implementing each method specified therein. 

567 3.3.1.4 Constructor & Destructor Documentation 

568 • virtual TattE::DetectAndLocalizeInterface::DetectAndLocalizeInterface ()[inline], [virtual] 

569 3.3.1.5 Member Function Documentation 

570 3.3.1.5.1 virtual ReturnStatus TattE::DetectAndLocalizeInterface::initialize (const std::string & 

571 configurationLocation)[pure virtual] 

572 This function initializes the implementation under test. It will be called by the NIST application before any call 

573 to the functions detectTattoo and localizeTattoos. 

574 

575 Parameters: 

in configurationLocation A read-only directory containing any developer-supplied configuration parameters or 

run-time data. The name of this directory is assigned by NIST, not hardwired by the 

provider. The names of the files in this directory are hardwired in the implementation 

and are unrestricted. 

576 3.3.1.5.2 virtual ReturnStatus TattE::DetectAndLocalizeInterface::detectTattoo (const Image & 

577 inputImage, bool & tattooDetected, double & confidence)[pure virtual] 

578 This function takes an Image as input and indicates whether a tattoo was detected in the image or not. 

579 

580 Parameters: 

in inputImage An instance of an Image struct representing a single image 

out tattooDetected true if a tattoo is detected in the image; false otherwise 

NIST Concept and Evaluation Plan Page 17 of 24 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Tattoo Recognition Technology - Evaluation (Tatt-E) 

out confidence A real-valued measure of tattoo detection confidence on [0,1]. A value of 1 indicates 

certainty that the image contains a tattoo, and a value of 0 indicates certainty that the 

image does not contain a tattoo. 

581 3.3.1.5.3 virtual ReturnStatus TattE::DetectAndLocalizeInterface::localizeTattoos(const Image & 

582 inputImage, std::vector< BoundingBox > & boundingBoxes, std::vector< BodyLocation > 

583 & bodyLocations)[pure virtual] 

584 This function takes an Image as input, and populates a vector of BoundingBox with the number of tattoos 

585 detected on different body locations from the input image. 

586 

587 Parameters: 

in inputImage An instance of an Image struct representing a single image 

out boundingBoxes For each tattoo detected in the image, the function shall create a BoundingBox, 

populate it with a confidence score, the x, y, width, height of the bounding box, and 

add it to the vector. 

588 3.3.1.6 static std::shared_ptr<DetectAndLocalizeInterface> 

589 TattE::DetectAndLocalizeInterface::getImplementation ()[static] 

590 Factory method to return a managed pointer to the DetectAndLocalizeInterface object. 

591 This function is implemented by the submitted library and must return a managed pointer to the 

592 DetectAndLocalizeInterface object. 

593 Note: 

594 A possible implementation might be: return(std::make_shared<ImplementationD>()); 

595 3.4 Identification (Class I) 

596 The 1:N application proceeds in two phases, enrollment and identification. The identification phase includes 

597 separate pre-search feature extraction stage, and a search stage. 

598 The design reflects the following testing objectives for 1:N implementations. 

- support distributed enrollment on multiple machines, with multiple processes running in parallel 

- allow recovery after a fatal exception, and measure the number of occurrences 

- allow NIST to copy enrollment data onto many machines to support parallel testing 

- respect the black-box nature of biometric templates 

- extend complete freedom to the provider to use arbitrary algorithms 

- support measurement of duration of core function calls 

- support measurement of template size 

599 Table 8 – Procedural overview of the identification test Phase

# Name Description Performance Metrics to be 

reported by NIST 

Enrollment

E1 Initialization initializeEnrollmentSession() 

Give the implementation the name of a directory where any 

provider-supplied configuration data will have been placed by 

NIST. This location will otherwise be empty. 

NIST Concept and Evaluation Plan Page 18 of 24 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Tattoo Recognition Technology - Evaluation (Tatt-E) 

E2 Parallel 

Enrollment 

createTemplate(TemplateRole=Enrollment) 

The input will be one or more of the same tattoo image. This 

function will pass the input to the implementation for conversion to 

a single template. The implementation will return a template to the 

calling application. 

NIST's calling application will be responsible for storing all 

templates as binary files. These will not be available to the 

implementation during this enrollment phase. 

Multiple instances of the calling application may run 

simultaneously or sequentially. These may be executing on 

different computers. The same tattoo will not be enrolled twice. 

Statistics of the times needed 

to enroll a tattoo. 

Statistics of the sizes of 

created templates. 

The incidence of failed 

template creations. 

E3 Finalization finalizeEnrollment() 

Permanently finalize the enrollment directory. This supports, for 

example, adaptation of the image-processing functions, adaptation 

of the representation, writing of a manifest, indexing, and 

computation of statistical information over the enrollment dataset. 

The implementation is permitted read-write-delete access to the 

enrollment directory during this phase. 

Size of the enrollment 

database as a function of 

population size N and the 

number of images. 

Duration of this operation. 

The time needed to execute 

this function shall be reported 

with the preceding enrollment 

times. 

Pre-search 

S1 Initialization initializeProbeTemplateSession() 

Tell the implementation the location of an enrollment directory. 

The implementation could look at the enrollment data. 

Implementation initialize in preparation for search template 

creation. 

The implementation is permitted read-only access to the 

enrollment directory during this phase. 

Statistics of the time needed 

for this operation. 

Statistics of the time needed 

for this operation. 

S2 Template 

preparation 

createTemplate(TemplateRole=Identification) 

For each probe, create a template from a set of input tattoo(s) or a 

sketch image. This operation will generally be conducted in a 

separate process invocation to step S3. 

The implementation is permitted no access to the enrollment 

directory during this phase. 

The result of this step is a search template. 

Statistics of the time needed 

for this operation. 

Statistics of the size of the 

search template. 

Search 

S3 Initialization initializeIdentificationSession() 

Tell the implementation the location of an enrollment directory. 

The implementation should read all or some of the enrolled data 

into main memory, so that searches can commence. 

The implementation is permitted read-only access to the 

enrollment directory during this phase. 

Statistics of the time needed 

for this operation. 

S4 Search identifyTemplate()

A template is searched against the enrollment database. 

The implementation is permitted read-only access to the 

enrollment directory during this phase. 

Statistics of the time needed 

for this operation. 

Accuracy metrics - Type I + II 

error rates. 

Failure rates. 

600 

601 3.4.1 TattE::IdentificationInterface Class Reference 

602 3.4.1.1 Public Member Functions 

603 • virtual ~IdentificationInterface () 

604 • virtual ReturnStatus initializeEnrollmentSession (const std::string &configurationLocation)=0 

605 This function initializes the implementation under test and sets all needed parameters. 

606 • virtual ReturnStatus createTemplate (const MultiTattoo &inputTattoos, const TemplateRole 

607 &templateType, TattooRep &tattooTemplate, std::vector<double> &quality)=0 

NIST Concept and Evaluation Plan Page 19 of 24 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Tattoo Recognition Technology - Evaluation (Tatt-E) 

608 This function takes a MultiTattoo and outputs a TattooRep object (essentially a template). 

609 • virtual ReturnStatus finalizeEnrollment (const std::string &enrollmentDirectory, const std::string 

610 &edbName, const std::string &edbManifestName)=0 

611 This function will be called after all enrollment templates have been created and freezes the 

612 enrollment data. After this call the enrollment dataset will be forever read-only. 

613 • virtual ReturnStatus initializeProbeTemplateSession (const std::string &configurationLocation, 

614 const std::string &enrollmentDirectory)=0 

615 Before MultiTattoos are sent to the search template creation function, the test harness will 

616 call this initialization function. 

617 • virtual ReturnStatus initializeIdentificationSession (const std::string &configurationLocation, const 

618 std::string &enrollmentDirectory)=0 

619 This function will be called once prior to one or more calls to identifyTemplate. The function 

620 might set static internal variables so that the enrollment database is available to the 

621 subsequent identification searches. 

622 • virtual ReturnStatus identifyTemplate (const TattooRep &idTemplate, const uint32_t 

623 candidateListLength, std::vector< Candidate > &candidateList)=0 

624 This function searches an identification template against the enrollment set, and outputs a 

625 vector containing candidateListLength Candidates. 

626 3.4.1.2 Static Public Member Functions 

627 • static std::shared_ptr< IdentificationInterface > getImplementation () 

628 Factory method to return a managed pointer to the IdentificationInterface object. 

629 

630 3.4.1.3 Detailed Description 

631 The interface to Class I implementations. 

632 The Class I submission software under test will implement this interface by subclassing this class and 

633 implementing each method therein. 

634 3.4.1.4 Constructor & Destructor Documentation 

635 • virtual TattE::IdentificationInterface::~IdentificationInterface ()[inline], [virtual] 

636 

637 3.4.1.5 Member Function Documentation 

638 3.4.1.5.1 virtual ReturnStatus TattE::IdentificationInterface::initializeEnrollmentSession (const 

639 std::string & configurationLocation)[pure virtual] 

640 This function initializes the implementation under test and sets all needed parameters. 

641 This function will be called N=1 times by the NIST application, prior to parallelizing M >= 1 calls to 

642 createTemplate() via fork(). 

643 Parameters: 

in configurationLocation A read-only directory containing any developer-supplied configuration parameters or 

run-time data files. 

644 3.4.1.5.2 virtual ReturnStatus TattE::IdentificationInterface::createTemplate (const MultiTattoo & 

645 inputTattoos, const TemplateRole & templateType, TattooRep & tattooTemplate, 

646 std::vector<double> & quality)[pure virtual] 

647 This function takes a MultiTattoo and outputs a TattooRep object (essentially a template) and a vector of 

648 quality values associated with each tattoo image. 

649 For enrollment templates: If the function executes correctly (i.e. returns a successful exit status), the NIST 

650 calling application will store the template. The NIST application will concatenate the templates and pass the 

651 result to the enrollment finalization function. When the implementation fails to produce a template, it shall still 

NIST Concept and Evaluation Plan Page 20 of 24 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Tattoo Recognition Technology - Evaluation (Tatt-E) 

652 return a blank template (which can be zero bytes in length). The template will be included in the enrollment 

653 database/manifest like all other enrollment templates, but is not expected to contain any feature information. 

654 For identification templates: If the function returns a non-successful return status, the output template will be 

655 not be used in subsequent search operations. 

656 Parameters: 

in inputTattoos An instance of a MultiTattoo structure. Implementations must alter their behavior 

according to the type and number of images/type of image contained in the structure. 

The input image type could be a tattoo or a sketch image. The MultiTattoo will always 

contain the same type of imagery, i.e., no mixing of tattoos and sketch images will 

occur. Note that implementation support for sketch images is OPTIONAL.

Implementation shall return TattE::ImageType::ImageTypeNotSupported if they

do not support sketch images. All algorithms must support tattoo images. 

in templateType A value from the TemplateRole enumeration that indicates the intended usage of the 

template to be generated. In this case, either an enrollment template used for gallery 

enrollment or an identification template used for search. 

out tattooTemplate Tattoo template object. For each tattoo detected in the MultiTattoo, the function shall 

provide the bounding box coordinates in each image. The bounding boxes shall be 

captured in the TattooRep.boundingBoxes variable, which is a vector of 

BoundingBox objects. If there are 4 images in the MultiTattoo vector, then the size 

of boundingBoxes shall be 4. boundingBoxes[i] is associated with MultiTattoo[i]. 

out quality A vector of quality values, one for each input tattoo image. This will be an empty 

vector when passed into this function, and the implementation shall populate a 

quality value corresponding to each input image. quality[i] shall correspond to 

inputTattoos[i]. A measure of tattoo quality on [0,1] is indicative of expected utility to 

the matcher, or matchability. This value could measure tattoo 

distinctiveness/information richness, and would be an indicator of how well the tattoo 

would be expected to match. A value of 1 indicates high quality and that the tattoo 

would be expected to match well, and a value of 0 means low quality, indicative that 

tattoo would not match well. 

657 3.4.1.5.3 virtual ReturnStatus TattE::IdentificationInterface::finalizeEnrollment (const std::string & 

658 enrollmentDirectory, const std::string & edbName, const std::string & 

659 edbManifestName)[pure virtual] 

660 This function will be called after all enrollment templates have been created and freezes the enrollment data. 

661 After this call the enrollment dataset will be forever read-only. 

662 This function allows the implementation to conduct, for example, statistical processing of the feature data, 

663 indexing and data re-organization. The function may create its own data structure. It may increase or 

664 decrease the size of the stored data. No output is expected from this function, except a return code. The 

665 function will generally be called in a separate process after all the enrollment processes are complete. 

666 NOTE: Implementations shall not move the input data. Implementations shall not point to the input data. 

667 Implementations should not assume the input data would be readable after the call. Implementations must, 

668 at a minimum, copy the input data or otherwise extract what is needed for search. 

669 Parameters: 

in enrollmentDirectory The top-level directory in which enrollment data was placed. This variable allows an 

implementation to locate any private initialization data it elected to place in the 

directory. 

in edbName The name of a single file containing concatenated templates, i.e. the EDB described 

in Data Structures Supporting the API. While the file will have read-write-delete 

permission, the implementation should only alter the file if it preserves the necessary 

content, in other files for example. The file may be opened directly. It is not necessary 

to prepend a directory name. This is a NIST-provided input - implementers shall not 

internally hard-code or assume any values. 

in edbManifestName The name of a single file containing the EDB manifest described in Data Structures 

Supporting the API . The file may be opened directly. It is not necessary to prepend a 

directory name. This is a NIST-provided input - implementers shall not internally 

hard-code or assume any values. 

NIST Concept and Evaluation Plan Page 21 of 24 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Tattoo Recognition Technology - Evaluation (Tatt-E) 

670 3.4.1.5.4 virtual ReturnStatus TattE::IdentificationInterface::initializeProbeTemplateSession (const 

671 std::string & configurationLocation, const std::string & enrollmentDirectory)[pure virtual] 

672 Before MultiTattoos are sent to the search template creation function, the test harness will call this 

673 initialization function. 

674 This function initializes the implementation under test and sets all needed parameters. This function will be 

675 called N=1 times by the NIST application, prior to parallelizing M >= 1 calls to createTemplate() via fork(). 

676 Caution: The implementation should tolerate execution of P > 1 processes on the one or more machines 

677 each of which may be reading from this same enrollment directory in parallel. The implementation has read678 only access to its prior enrollment data. 

679 Parameters: 

in configurationLocation A read-only directory containing any developer-supplied configuration parameters or 

run-time data files. 

in enrollmentDirectory The read-only top-level directory in which enrollment data was placed and then 

finalized by the implementation. The implementation can parameterize subsequent 

template production on the basis of the enrolled dataset. 

680 3.4.1.5.5 virtual ReturnStatus TattE::IdentificationInterface::initializeIdentificationSession (const 

681 std::string & configurationLocation, const std::string & enrollmentDirectory)[pure virtual] 

682 This function will be called once prior to one or more calls to identifyTemplate. The function might set static 

683 internal variables so that the enrollment database is available to the subsequent identification searches. 

684 Parameters: 

in configurationLocation A read-only directory containing any developer-supplied configuration parameters or 

run-time data files. 

in enrollmentDirectory The read-only top-level directory in which enrollment data was placed. 

685 3.4.1.5.6 virtual ReturnStatus TattE::IdentificationInterface::identifyTemplate (const TattooRep & 

686 idTemplate, const uint32_t candidateListLength, std::vector< Candidate > & 

687 candidateList)[pure virtual] 

688 This function searches an identification template against the enrollment set, and outputs a vector containing 

689 candidateListLength Candidates. 

690 Each candidate shall be populated by the implementation and added to candidateList. Note that 

691 candidateList will be an empty vector when passed into this function. The candidates shall appear in 

692 descending order of similarity score - i.e. most similar entries appear first. 

693 Parameters: 

in idTemplate A template from createTemplate(). If the value returned by that function was nonsuccessful, the contents of idTemplate will not be used, and this function will not be 

called. 

in candidateListLength The number of candidates the search should return. 

out candidateList Each candidate shall be populated by the implementation. The candidates shall 

appear in descending order of similarity score - i.e. most similar entries appear first. 

694 3.4.1.5.7 static std::shared_ptr<IdentificationInterface> 

695 TattE::IdentificationInterface::getImplementation ()[static] 

696 

697 Factory method to return a managed pointer to the IdentificationInterface object. 

698 This function is implemented by the submitted library and must return a managed pointer to the 

699 IdentificationInterface object. 

700 Note: 

701 A possible implementation might be: return (std::make_shared<ImplementationI>()); 

702 

703 

704 

NIST Concept and Evaluation Plan Page 22 of 24 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

																																																																		

NIST will not register, or establish any kind of membership, on the provided website. 

Tattoo Recognition Technology - Evaluation (Tatt-E) 

705 Annex A 

706 Submissions of Implementations to Tatt-E 

707 A.1 Submission of implementations to NIST 

708 NIST requires that all software, data and configuration files submitted by the participants be signed and 

709 encrypted. Signing is done with the participant's private key, and encryption is done with the NIST public 

710 key. The detailed commands for signing and encrypting are given here: https://www.nist.gov/itl/iad/image711 group/products-and-services/encrypting-softwaredata-transmission-nist. 

712 NIST will validate all submitted materials using the participant's public key, and the authenticity of that key 

713 will be verified using the key fingerprint. This fingerprint must be submitted to NIST by writing it on the 

714 signed Tatt-E Participation Agreement that is published on the Tatt-E website -

715 https://www.nist.gov/programs-projects/tattoo-recognition-technology-evaluation-tatt-e. 

716 By encrypting the submissions, we ensure privacy; by signing the submission, we ensure authenticity (the 

717 software actually belongs to the submitter). NIST will reject any submission that is not signed and encrypted. 

718 NIST accepts no responsibility for anything that is transmitted to NIST that is not signed and encrypted with 

719 the NIST public key. 

720 A.2 How to participate 

721 Those wishing to participate in Tatt-E testing must do all of the following, on the schedule listed on Page 2. 

722 ― IMPORTANT: Follow the instructions for cryptographic protection of your software and data here -

723 https://www.nist.gov/itl/iad/image-group/products-and-services/encrypting-softwaredata-transmission-nist 

724 ― Send a signed and fully completed copy of the Application to Participate in the Tattoo Recognition 

725 Technology - Evaluation (Tatt-E) contained in this document that is published on the Tatt-E website. 

726 This must identify, and include signatures from, the Responsible Parties as defined in the application. 

727 The properly signed Tatt-E Application to Participate shall be sent to NIST as a PDF. 

728 ― Provide a software library that complies with the API (Application Programmer Interface) specified in this 

729 document. 

730 • Encrypted data and libraries below 20MB can be emailed to NIST at tatt-e@nist.gov. 

731 • Encrypted data and libraries above 20MB shall be 

732 EITHER 

733 • Split into sections AFTER the encryption step. Use the unix "split" commands to make 

734 9MB chunks, and then rename to include the filename extension need for passage 

735 through the NIST firewall. 

736 • you% split –a 3 –d –b 9000000 libTattE_Choice_D_07.tgz.gpg 

737 • you% ls -1 x??? | xargs –iQ mv Q libTattE_Choice_D_07_Q.tgz.gpg 

738 • Email each part in a separate email. Upon receipt NIST will 

739 • nist% cat tatte_choice_D07_*.tgz.gpg > libTattE_Choice_D_07.tgz.gpg 

740 OR 

741 • Made available as a file.zip.gpg or file.zip.asc download from a generic http webserver5, 

742 OR 

743 • Mailed as a file.zip.gpg or file.zip.asc on CD / DVD to NIST at this address: 

Tatt-E Test Liaison (A210) 

100 Bureau Drive 

A210/Tech225/Stop 8940 

NIST 

In cases where a courier needs a phone number, 

please use NIST shipping and handling on: 301 --

975 -- 6296. 

5

NIST Concept and Evaluation Plan Page 23 of 24 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Tattoo Recognition Technology - Evaluation (Tatt-E) 

Gaithersburg, MD 20899-8940 

USA 

744 A.3 Implementation validation 

745 Registered Participants will be provided with a small validation dataset and test program upon NIST receipt 

746 of either a signed Participation Agreement or an email to tatt-e@nist.gov indicating participation intentions. 

747 Instructions on obtaining the validation package will be emailed to participants thereafter. 

748 via https://github.com/usnistgov/tattoo shortly after the final evaluation plan is released. An announcement 

749 will be made on the Tatt-E website when the validation package is available. 

750 The validation test programs shall be compiled by the provider. The output of these programs shall be 

751 submitted to NIST. 

752 Prior to submission of the software library and validation data, the Participant must verify that their software 

753 executes on the validation images and produces correct scores and templates. 

754 Software submitted shall implement the Tatt-E API Specification as detailed in the body of this document. 

755 Upon receipt of the software library and validation output, NIST will attempt to reproduce the same output by 

756 executing the software on the validation imagery, using a NIST computer. In the event of disagreement in 

757 the output, or other difficulties, the Participant will be notified. 

758 

759 Please note that the provided validation software and imagery is meant only for validation purposes and 

760 does not reflect how NIST will perform actual testing. The validation images are not representative of the 

761 actual test data that will be used to evaluate implementations. 

762 

763 

764 

NIST Concept and Evaluation Plan Page 24 of 24