SIGIR WORKSHOP REPORT

NeuroIR 2015 – SIGIR 2015 Workshop on Neuro-PhysiologicalMethods in IR Research
Jacek Gwizdka
School of Information
University of Texas atAustin, TX, USA
sigir2015@gwizdka.com

Javed Mostafa
Info. Science & BRIC
UNC at Chapel Hill
NC, USA
jm@unc.edu

Abstract: The first international workshop on NeuroPhysiological (NP) Methods in Information Retrieval
Research took place at SIGIR 2015 in Santiago, Chile on
August 13, 2015. The workshop included a mixture ofshort introductions to selected NP methods, paper
presentations and a panel discussion by invited scholars.
Each session included significant time allotted to open
discussions among the participants. The high attendance
and the lively discussions demonstrated significant
interest in the use of NP methods in information
retrieval research. The panel and discussions pointed out
many advantages of using these methods as a data
source complementary to traditional measures but also
potentially offering deeper insights into humans
interacting with information. Significant challenges,
such as the lack of baseline data and reference tasks, anda need to collaborate with neuroscientists, were also
noted. Overall, the NP-methods were found to be very
promising and worth continued research efforts.

1Introduction

The progress made in neuroscience in the last two decades allows one to expect thatneuroscience will contribute to other disciplines. It has already happened in several social
science disciplines, such as economics and marketing – neuroeconomics and neuromarketing –have been established over a decade ago. More recently, information systems researchers started
a new sub-field NeuroIS (Neuro Information Systems [3]). Information systems research studies
the development and use of information and communication technologies in organizations andsociety. NeuroIS is a subfield of information systems that employs neuroscience andneurophysiological theories and tools to better understand the development, use, and impact of
information technologies [15].

ACM SIGIR Forum83

Vol. 49 No. 2 December 2015

Application of neuro-physiological methods for studying search and browsing tasks, andrelevance judgment has emerged as an important area of active research. The broad intersecting
theme, among IR researchers and those with strong applied neuroscience background, isdevelopment of more robust models for information search – models which go beyond
behavioral data and account for physiological and neurological responses to information asstimuli. This early work by information retrieval researchers resulted in a number of recent
publications that appeared at SIGIR, ECIR, and related conferences. Selected recent publications
include: [2, 5–8, 11, 12]. This work has already received a significant recognition as three ofthese early publications [1, 7, 11] were recognized as a best paper or poster. However, the wider
IR community is still mostly unaware of these new methods. The workshop was organized tobridge this gap.

2Purpose

The objectives of the workshop were to:• introduce neuro-physiological methods to IR researchers (and serve as a mini-tutorial);• make IR researchers aware of potential usefulness of neuro-physiological (NP) methods andtheir applicability to IR research;• enable IR researchers to read and assess papers that employ NP methods;• promote discussion and exchange of experiences among researchers;• build a community and map out future work.

3Structure of the Workshop

The workshop was chaired by Jacek Gwizdka from the School of Information at the Universityof Texas at Austin, and workshop co-organizer. The workshop combined presentations withinteractive discussion sessions. It started with participant introductions and ten lightning talks
given by authors of papers accepted to the workshop1. Next, the workshop session switched to atutorial mode and four introductions to NP modalities were given by experienced researchers.
Each introduction was followed by a discussion. Next, three selected papers were presented by
one of their authors. Workshop ended with a panel session.

4Introductions to Neuro-physiological Modalities

Ioannis Arapakis introduced several physiological modalities, including electro-dermal activity(EDA) and heart-rate variability (HRV) and presented use cases focused on affective andemotional aspects of human information interaction. Jacek Gwizdka introduced eye-tracking andits derived measures. He emphasized the promise for inferring constructs of interest to IR fromdynamic changes in pupil dilation. The usefulness of eye-tracking in IR was illustrated byexamples from his research [6, 9]. Javed Mostafa and Vincent Carrasco introduced EEG andfMRI, respectively. They illustrated these modalities with their own research framework [13].
1Full list of papers accepted to the workshop is available online on the workshop website. See: http://neuroir.org

ACM SIGIR Forum84

Vol. 49 No. 2 December 2015

5Paper Presentations

Tukka Ruotsalo presented “Predicting Relevance of Text from Neuro-Physiology” [4]. On an
example of two case studies he showed the possibilities for detecting relevance using NP-signals,
such as, electro-encephalography (EEG), electro-dermal activity (EDA), and electromyography(EMG). Dawei Song presented “EEG Based Emotion Identification Using Unsupervised Deep
Feature Learning” [10]. The discussed work focused on using Deep Belief Network in automaticfeature extraction from raw EEG signals in order to assess user’s emotional states and use themin implicit relevance feedback. Ioannis Arapakis presented “On Human Information Processing
in Information Retrieval” [14]. In this, somewhat provocative, talk, he proposed that theapplication of psychological knowledge to information retrieval research will not only bebeneficial for IR, but, thanks to large scale testing grounds provided by IR, it will also be usefulfor evaluating psychological hypotheses about human information processing.

6Panel Discussion

6.1 Background on the Panel & Panelists
After the paper presentation session, a panel discussion was held as the concluding event of theworkshop. Three senior scholars in the information retrieval field were invited to participate inthe panel. Representing industry, a highly reputable and established information retrieval
scholar, Sue Dumais participated as a panel member. Another panel member was Diane Kelly,
Professor at the University of North Carolina Chapel Hill’s iSchool, where she conducts researchon human-information interactions. A third member was Paul Kantor, Distinguished Professor(Emeritus) of Information Science, at Rutgers University. Paul has a wide-ranging set ofresearch contributions in information retrieval, covering both theoretical and empirical work.
The panel was moderated by Javed Mostafa, Professor of Information Science at the Universityof North Carolina at Chapel Hill and co-organizer of the workshop. The scope of the panel topics
consisted of three specific dimensions. Panelists were asked to consider the primary drivers andadvantages for integrating neuro-physiological (NP) approaches in information retrieval
research. Specific areas of further development and advances in applying NP approaches in IR
research comprised the second dimension. The last dimension was challenges and barriers
associated with using NP approaches in IR research.

6.2 Summary of Points Discussed
Information retrieval has become ubiquitous and pervasive to a point that we need to rely on
different tools and methods to scrutinize and study user behavior at a more granular level. A
panelist noted that NP methods are likely powerful enough to elicit subtle clues and signals that
other methods are not capable of. It was pointed out that NP methods may identify signals thatare not expressible verbally or through actions. Methods utilizing eye fixation durations, pupildilation, skin conductance response, and even facial expressions can reveal additional
information that may complement other sources of analyzing user behavior in a powerful way. It
may even be possible that such sources of NP evidence and others may reveal “deep signals” thatare less noisy and more immune from external influences such as the person conducting theACM SIGIR Forum85

Vol. 49 No. 2 December 2015

experiment. Some panelists observed that by utilizing portable, less intrusive and sometimes
inexpensive NP measurement tools (e.g., wireless EEG headsets) it may be possible to study
search behavior in more realistic settings and increase the validity of the derived data andobservations. Information retrieval studies that deal with mobile devices may require the use ofsuch easy to use NP measurement tools. A participant pointed out that such NP tools andcapabilities are beginning to be built into some mobile devices such as Apple’s iWatch. Ingeneral, there were a lot of enthusiasms for exploring different types of NP measurement tools
and in a wider variety of search contexts. A participant also commented about the possibility ofexploring the utility of NP methods across different age groups (e.g., children vs. elderly).
A significant amount of discussion time was dedicated to various limitations associated with NP
methods and ways of overcoming some of them. A panelist stated that the major problem
associated with reported results is that typically a large number of tool-oriented parameters andsettings require adjustments before experiments and these parameter-level information areomitted in publications. The omission is seen as a major barrier for other researchers to replicate
the experiments or to even understand the reported NP results deeply. Another participant statedthat although there are potentially advantages to be gained by combining multiple NP modalities
in conducting search experiments, relating results from one modality with another should beconducted carefully, with due attention paid to what the individual modalities are “tuned” tomeasure and if they are set up to measure the same thing (e.g., Are they measuring cognitive
load? Or, are they merely measuring interest or attention?). Another challenge mentioned inconnection with the complexity of theories underlying NP tools was a need to collaborate, or atleast consult with neuroscientists and other specialists. It should be noted that individualmodalities such as eye-tracking, EEG, and fMRI are relatively new ways of measuring human
responses during searching and no norms, baselines, or ranges of expected responses have yet
been established. Therefore, comparison of responses across modalities becomes even moredifficult to make. At the end of the session, all the panelists expressed their concern that it isdifficult for most experts in IR to evaluate NP-based IR papers as there is a lack of awareness ofcritical NP methods and their appropriate applications. As a way to overcome this barrier it wassuggested that a paper should be written targeted to the broad IR community that can be used asa guide to evaluate the rigor and appropriateness of NP-oriented contributions in the IR field.
Finally, development of a standard set of search tasks and baseline results that can be used acrossdifferent NP methods was seen as a useful way to advance this emerging area. Some participants
mentioned that a TREC track on NP-IR (trec.nist.gov) should be seriously considered by theresearch community.

7Conclusions

This was the first workshop on Neuro‐Physiological Methods in IR Research. The intensity ofinterest in the workshop, as represented in the number of paper submissions and in thenumber of workshop participants, marks the growing research engagements in NeuroIR within
the IR community. We believe that this area has a high potential for contributing to IR byenriching it with new ways for collecting data that can yield new and complementary measuresand enable researchers to contribute at theoretical level as well as at applied level. We trustthat the interest in the area will not only continue beyond the workshop, but also increase as
more colleagues become aware of these methods and start using them.

ACM SIGIR Forum86

Vol. 49 No. 2 December 2015

In support of continued conversations, a Neuro-IR mailing list has been created; to subscribe
send email with subject line: SUB neuro_ir-list to: sympa@utlists.utexas.edu . Theworkshop’s website is at: http://neuroir.org.

Acknowledgements
The workshop was co-organized by Jacek Gwizdka, Javed Mostafa, Joemon Jose (Dept. ofComputer Science, University of Glasgow, UK) and Max L. Wilson (School of Computer
Science University of Nottingham, UK). We would like to thank ACM and SIGIR for hosting
this workshop as well as the SIGIR workshop co-chairs and the reviewers. We are further very
thankful to the authors, the members of our program committee (Ioannis Arapakis, YahooResearch, Spain; Vincent N. Carrasco, University of North Carolina at Chapel Hill, USA; AshleeEdwards, University of North Carolina at Chapel Hill, USA; Manuel Eugster, Aalto University,Finland; Roberto González-Ibáñez, Universidad de Santiago, Chile; Hideo Joho, University ofTsukuba, Japan; Yvonne Kammerer, Leibniz-Institut für Wissensmedien - Knowledge Media
Research Center, Germany; Irene Lopatovska, Pratt Institute, USA; Randall Minas, University ofHawaii at Manoa, USA; Yashar Moshfeghi, University of Glasgow, UK; Heather O'Brien,
University of British Columbia, Canada; Luisa Pinto, University of Glasgow, UK; Frank Pollick,
University of Glasgow, UK; Erin Solovey, Drexel, USA), and all participants. They helped toform a very lively, highly interesting and successful workshop.

References
[1]

[2]
[3]

[4]

[5]

[6]
[7]
[8]

Arapakis, I. et al. 2009. Enriching User Profiling with Affective Features for theImprovement of a Multimodal Recommender System. Proceedings of the ACM
International Conference on Image and Video Retrieval (New York, NY, USA), 29:1–29:8.
BEST POSTER
Cole, M.J. et al. 2013. Inferring user knowledge level from eye movement patterns.
Information Processing & Management. 49, 5, 1075–1091.
Dimoka, A. et al. 2011. Research Commentary—NeuroIS: The Potential of Cognitive
Neuroscience for Information Systems Research. Information Systems Research. 22, 4, 687–702.
Eugster, M.J.A. et al. 2015. Predicting Relevance of Text from Neuro-Physiology. 1stInternational Workshop on Neuro-Physiological Methods in IR Research (NeuroIR 2015)(Santiago, Chile).
Eugster, M.J.A. et al. 2014. Predicting Term-relevance from Brain Signals. Proceedings ofthe 37th International ACM SIGIR Conference on Research &#38; Development inInformation Retrieval (New York, NY, USA), 425–434.
Gwizdka, J. 2014. Characterizing Relevance with Eye-tracking Measures. Proceedings of
the 5th Information Interaction in Context Symposium (New York, NY, USA), 58–67.
Gwizdka, J. 2013. Looking for Information Relevance In the Brain. Gmunden Retreat onNeuroIS 2013 (Gmunden, Austria), 14. BEST PAPER
Gwizdka, J. 2014. News Stories Relevance Effects on Eye-movements. Proceedings of theSymposium on Eye Tracking Research and Applications (New York, NY, USA), 283–286.

ACM SIGIR Forum87

Vol. 49 No. 2 December 2015

[9] Gwizdka, J. and Cole, M. 2013. Does interactive search results overview help?: an eyetracking study. CHI’2013 Extended Abstracts on Human Factors in Computing Systems(New York, NY, USA), 1869–1874.
[10] Li, X. et al. 2015. EEG Based Emotion Identification Using Unsupervised Deep Feature
Learning. 1st International Workshop on Neuro-Physiological Methods in IR Research(NeuroIR 2015) (Santiago, Chile).
[11] Moshfeghi, Y. et al. 2013. Understanding Relevance: An fMRI Study. Advances inInformation Retrieval - ECIR'2013. P. Serdyukov et al., eds. Springer Berlin Heidelberg.
14–25. BEST PAPER[12] Moshfeghi, Y. and Jose, J.M. 2013. An effective implicit relevance feedback techniqueusing affective, physiological and behavioural features. Proceedings of the 36thinternational ACM SIGIR conference on Research and development in informationretrieval (New York, NY, USA), 133–142.
[13] Mostafa, J. et al. 2015. Identifying Neurological Patterns Associated with Information
Seeking: A Pilot fMRI Study. Information Systems and Neuroscience. F.D. Davis et al.,eds. Springer International Publishing. 167–173.
[14] Pereda-Baños, A. et al. 2015. On Human Information Processing in Information Retrieval.
1st International Workshop on Neuro-Physiological Methods in IR Research (NeuroIR2015) (Santiago, Chile).
[15] Riedl, R. et al. 2010. The Potential of Neuroscience for Human-Computer Interaction
Research. SIGHCI 2010 Proceedings.

ACM SIGIR Forum88

Vol. 49 No. 2 December 2015