INTRODUCTIONNAILS: Neurally Augmented Image Labelling Strategies• Dr. Graham Healy, Dr. Cathal Gurrin, Dr.

Prof. Alan Smeaton• The Insight Centre for Data Analytics• Dublin City University• Topic: P300 signals for image labelling

Tomás Ward,OVERVIEW• Overview• Examples on EEG, P300 & ERPs

• Our Task & Data Plan
• Schedule & PlanQUICKLY: EEG• EEG (Electroencephalography) recording systems come inmany forms

P300 EXAMPLE• This concept is best introduced with an example• Find the image of a can of coke in the following stream ofimages:

P300 EXAMPLE•

P300 EXAMPLE•

P300 EXAMPLE•

P300 EXAMPLE•

P300 EXAMPLE•

P300 EXAMPLE•

P300 EXAMPLE•

P300 EXAMPLE•

P300 EXAMPLE•

P300 EXAMPLE•

P300 EXAMPLE•

P300 EXAMPLE• Everybody noticed it?

P300 RESPONSE• There is nothing special about the canof coke, I could have told you to lookout for the cup instead or used entirely
different images• This works because you do not know

when the can of coke will appear andas a result when it does, your attention
orientates to it (sort of like goingthrough television stations)ANOTHER EXAMPLE (RSVP)P300: A LIFELOG EXAMPLE• We’ve done this where participants are required to searcha stream of visual data for certain targets
images/concepts* Example ICA component activity

P300-RSVP STRATEGY• What this type of approach takes advantage of:• The speed of human visual recognition/processing i.e.

we can often present these images at a very fast rate
(example given) via RSVP (Rapid Serial VisualPresentation)• The on-the-fly capability of defining search tasks forusers (within some constraints)

TASK & DATA PLAN
• For NTCIR-13 we will release a dataset from 10

experiment participants preforming a number of searchtasks• The images used will be derived from the Places 21

dataset and the ImageNet2 dataset

Places 2 dataset made available from MIT , 2ImageNet Stanford vision lab

TASK & DATA PLAN
• The experiment for each participant will involve 8 searchtasks• i.e. “Find photos of cats amongst the following stream ofimages”• Each search task is broken into 3 blocks (24 blocks total)• Each block contains 450 images presented at 5 Hz
• 45 (10%) of these are targets (relevant) and 405 (90%) are

non-targets(non-relevant)TASK & DATA PLAN
• The aim for participating organisations will be tosuccessfully predict from neural and behavioural (keypress) which images are relevant (target) or non-relevant(non-target) for the relevant search query• Participating organisations will be provided withtraining/validation data: 2 of 3 blocks for each search task

TASK & DATA PLAN
• During the evaluation period participating organisations willbenchmark their prediction models using an online REST
API (with some time-based limitations)• We will provide raw/pre-processed feature vector for bothneural / behavioural data for each image (-X task Xsubject) and must submit a prediction (0, 1)• Accuracy will be assessed across all predictions forsubjects X search - tasks on a withheld test set

TYPES OF SEARCH TASKS• *Search-tasks will be modelled around those in ILSVRC• Four scene-centric search-tasks from Places2 i.e. findimages of ‘art_gallery’, ‘nursing_home’, ‘oilrig’,‘television_room’• Four object-centric search tasks from ImageNet i.e. findimages of ‘cat’, ‘car’, ‘mushroom’, ‘desert’. Exact search tasks will be identified during a piloting phase identifying those which a human is capable of sufficiently doin

TYPES OF DATA
• Data will be available in raw and numerous pre-processed

formats to participants to lower the barrier to entry for
those unfamiliar with processing EEG signals• Neural-data feature vectors will include raw, bandpassed,

wavelet, ICA, and other common feature transformations
on which ML methods can be appliedTYPES OF DATA
• Example pipelines will be provided in python enabling

participating organisation to leverage an already working(naive) solution to get started right away
• Before the evaluation period begins code examples will beprovided showing participants how to interact with theonline evaluation system

SCHEDULE & PLAN
• Next Steps:- Internal DCU ethics application to begin conducting
experimentation (July 2016) - Approved (August)- Pilot experiments validating RT & neural responses (Sep2016 - Oct 2016)- Collection of final experimental dataset (Oct 2016 - Nov2016)- Dataset release (Jan 2017)

CONCLUSION• We can better understand the types of search-tasks

where detection of neural events are useful• Amplify insights and perspectives on labelling-behaviour ofusers (i.e. nuance and state-of-mind)• Will help to develop state-of-the-art methods for suchtasks• Will help to understand the connection between

behavioural and neural responses when performing an

THANK YOU• Further information will be available as the task structuredevelops at:•ntcir-nails.computing.dcu.ie• If any further queries please feel free to:ghealy@computing.dcu.ie