PAPER

Challenges in Designing a Brain-Machine SearchInterface
Yiqun Liu
Tsinghua University
yiqunliu@tsinghua.edu.cn
Xiaohui Xie
Tsinghua University
xiexh thu@163.com

Jiaxin Mao
Renmin University of China
maojiaxin@gmail.com

Min Zhang
Tsinghua University
z-m@tsinghua.edu.cn

Shaoping Ma
Tsinghua University
msp@tsinghua.edu.cn

Abstract
While search engines have reshaped how human beings learn and think, the interaction
paradigm of search has remained relatively stable for decades. With the development ofneural science and biomedical engineering, it is possible to build a direct communication
pathway between a computing device and the human brain via Brain-machine Interfaces(BMIs), which may revolutionize the search paradigm in a predictable future. Therefore,
in this paper, we extensively discuss the possibility, benefits, and potential challenges inusing BMI as a new interface for search, and call for more research efforts in this promising
direction.

1

Introduction

Being adopted in different environments and used by billions of users, search has changed how
humans learn and think. On one hand, Web search engines have become a primary channel
for many people to seek information, acquire knowledge, make decisions, and solve problems.
Augmented by the search engine, the Web has become an external memory for humans [Cerf,2014] and even change the way how humans think and learn [Sparrow et al., 2011]. On the other
hand, search has become ubiquitous. The search function has been incorporated into the operating
systems and applications of all kinds of computing devices including PCs, mobile phones, smart
speakers, and wearables. People can search everywhere and anytime. As a result, it has almost
become an instinct to find the search box when we have an information need because we get soused to searching to locate a file on PCs, launch an app on mobile phones, retrieve a message inthe email/IM apps, etc..
As the search technology is evolving to be more powerful and ubiquitous, the basic interaction
paradigm of search has been relatively stable for decades. As shown in the center part of Figure 1,
when searching, a user will firstly formulate a query, which often consists of a few keywords,
according to her information need and submit it to the system (i.e. a Web search engine). Upon
ACM SIGIR Forum

1

Vol. 54 No. 2 December 2020

receiving the query, the system will return and display a ranked list of search results to the user.
Then the user will interact with the returned search results. She can browse the results and accesses
those which are relevant and useful. With the development of NLP and query understanding
techniques, modern search engines can understand some natural language queries. They can also
provide “direct answers” for the natural language questions that have a short and definite answer.
Recently, with the emergence of search with intelligent assistants and smart speakers, considerable
research efforts have been focused on supporting new types of search interactions such as spoken
search [Trippas et al., 2020] and conversational search (see Anand et al. [2020] for a recent survey
report on this topic). While these works on understanding natural language queries and supportingthe conversational search are promising in innovating the interaction paradigm of search, in thispaper, we want to make bolder speculation of the future and foresee an even more fundamental
transformation in the search interface.
In the science fiction novel Old Man’s War, John Scalzi described a computing device called
BrainPal : “BrainPal is an immensely powerful, compact, semi-organic computer, thoroughly
integrated with the human brain. BrainPal augments the brain’s functions by assisting with mental
ability, memory storage, and communication.” From the perspectives of the current technologies,
it might be difficult to implement a semi-organic computer and integrate it into the human brain.
However, with the rapid development of the Brain-Machine Interfaces (BMIs), it seems possible tocreate a direct pathway between the human brain and the computing machinery that can be used
to augment humans’ cognition ability in the foreseeable future. Since the powerful and ubiquitous
search technology has reshaped how people think and learn, it is likely that the search function
will continue to play a central role in the BMI-based augmentation system and inevitably bringa more fundamental impact on the human s cognition process. Therefore, we argue that the IR
community should be prepared for this potential forthcoming revolution and start thinking about
the potential challenges in implementing a BMI-based or BMI-enhanced search system.
While it is difficult to predict whether and how the practical BrainPal would come to existence in the future, we can envision and discuss how BMIs will augment and improve the current
interaction paradigm of search. As shown in Figure 1, three additional passageways can be implemented with the BMIs that are already available or highly likely to be available in the near
future. First, the BMI signals may help the search engine to understand users’ information needs(IN Understanding). Second, users will be able to interact with the SERP with BMIs (BMIInteraction), which is much more efficient than current paradigms. Third, search engines caninfer users’ emotions and experience with BMI signals, which can be used as nearly real-time
explicit feedback for search effectiveness (Satisfaction Decoding).
In the rest of the paper, we firstly show how it may be possible to implement a BMI-based
or BMI-enhanced search system by introducing the status quo of BMI and some related studies
that have used BMI technology in controlling computer software, inputting text and speech, andunderstanding humans’ information needs as well as emotions (Section 2). Then in Section 3,
we discuss why BMIs would benefit search by showing how the BMI can help to overcome threemajor limitations of modern search interaction paradigms. In Section 4, we summarize three main
challenges for constructing a BMI search system and, finally, in Section 5, we conclude the paperand give some current research directions that may contribute to future BMI search systems, evenif we don’t have a ready-to-use BMI search interface yet.

ACM SIGIR Forum2

Vol. 54 No. 2 December 2020

signals
Information
need

1

ry
Que
BMI

t

inpu

Resultdisplay

Feedback

Interac
tion

2
User
satisfaction

3

1

IN Understanding

2

BMI Interaction

3

Satisfaction Decoding

Figure 1: A traditional Web search interaction process and how BMIs may change it. “Blackdotted arrow”: generation of implicit information that can not be directly decoded in original
search scenarios. “Red solid arrow”: possible new information pathway (e.g. understanding users’
information needs based on BMI signals) introduced by BMIs.

2

BMI Search: Is It Even Possible?

2.1

What is BMI?

BMI, also known as Brain-Computer Interface (BCI), is a direct communication pathway between
an enhanced or wired brain and an external device. Our brain is made up of billions of brain cells
called neurons, which use electricity to communicate with each other. Interfacing the brain withelectronic devices enables us to detect and interpret brain activities that can be used in researching,mapping, assisting, augmenting, or repairing human cognitive or sensory-motor functions [Krucoff
et al., 2016].
Based on how brain signals are collected, current BMI techniques can be grouped into threemajor categories: Invasive, Partially invasive, and Non-invasive. Invasive BMIs require special
devices (e.g. electrodes) to be inserted into the human brain by a critical surgery. It provides
deeper recordings and more accurate signals compared to other two types of BMIs but also suffersfrom the side effects of surgical interventions as well as the limitations in collecting informationfrom the whole neocortex. Recent advances of invasive BMIs include the efforts of the Neuralink

ACM SIGIR Forum

3

Vol. 54 No. 2 December 2020

Corporation by developing a novel technique to implant a large number of tiny electrodes and thin
threads into the brain with a robotic system. The goal is to build a scalable BMI system that has
at least “two orders of magnitude more communication channels than current clinically-approved
devices”.1 Partially invasive BMIs are inserted into the skull on the top of the human brain rather
than within the grey matter. Compared to fully invasive BMIs, it has a lower risk of forming
scar-tissue in the brain (which terribly affects signal collecting effectiveness) but the installation
still needs extracortical surgery. Partially invasive BMIs produce higher spatial resolution, better
signal-to-noise ratio, wider frequency range and less training requirements than non-invasive BMIs.
Electrocorticography (ECoG) is a typical partially invasive BMI which measures the electricalactivity of the brain whose electrodes are embedded in a thin plastic pad and placed above thecortex and beneath the dura mater [Serruya & Donoghue, 2003].
A large part of current BMI researches are focused on non-invasive BMIs. Unlike the fullyor partially invasive BMIs that require implanting a mechanical device in the brain, non-invasive
BMIs employ external devices to record brain signals. In that regard, non-invasive BMIs are
considered a safe and low-cost type of devices and have been used for a broader variety of applications: 1) Communication augmentation: they can help handicapped people to express theiropinions and ideas via a variety of methods such as spelling, semantic categorization, and silent
speech communication; 2) Device controlling: they can be used to control assistive devices suchas robotic devices, virtual keyboards etc.; 3) State Monitoring: they can work as a physiologicalmeasuring tool that collects information about an individual’s emotional, cognitive or affectiveness state. For example, in the educational field, non-invasive BMIs could help identify whether
students are paying attention to the teacher in the classroom.
Despite these promising advantages, non-invasive BMIs can only capture “weaker” brain signals
due to the obstruction of the skull. Among the various kinds of non-invasive BMIs, Electroencephalography (EEG) is one of the most popular choice in related application studies. It can beused to record electrophysiological signals which has high temporal resolution. However, the collected signals are with low spatial resolution and poor signal-to-noise ratio. Hence, noise caused byeye, muscle and movement artifacts should be removed before using the neuronal signals extracted
from EEG. Functional Magnetic Resonance Imaging (fMRI) is another kind of non-invasive BMIs
which can record metabolic signals (i.e., blood-oxygen-level dependent contrasts). Compared toEEG, fMRI has higher spatial resolution but worse temporal resolution. Magnetic signals of brain
can also be obtain using Magnetoencephalography (MEG) which also has high temporal resolution
as fMRI while produces better spatial resolution than EEG. However, MEG is rather expensive andit also has strict requirement on the experimental environment where external magnetic signals
should be shielded.

2.2

BMI Researches related to Search

There are a growing number of research literature using modern BMI techniques to complete tasks
that have IR application potentials, e.g., controlling keyboards and mouses, inputting speech andtext with thoughts, detecting information needs and monitoring human emotions. These works
validate the feasibility of using brain signals to perform related tasks in information retrieval and1

https://neuralink.com/approach/

ACM SIGIR Forum

4

Vol. 54 No. 2 December 2020

provide insight into our proposed future BMI search systems. Hereinafter, we briefly review someof these works.
BMI technology has been applied to perform some simple interactions with PC. Liu et al. [2010]
try to develop a BMI system which is composed of a spelling and a web-browsing tool. They traina SVM to predict the target button based on Electroencephalography (EEG) signals and show thatparticipants can use some basic functions of a Web search engine with the system. Perego et al.
[2011] apply a EEG-based BMI to verify the presence of specific visual perceptual competencies
underpinning the task such as scanning or matching abilities. In this task, participants are trained
to use BMI to play two games shown on computers’ screen, i.e., an astroBrain fight game and amatching game. BMI is also used to perform virtual typing tasks on computers in [Jarosiewicz
et al., 2015]. Recently, Nuyujukian et al. [2018] show that people with tetraplegia can use an
intracortical BMI to control an unmodified commercial tablet computer to use some popularapplications, e.g. web browsing, email and chatting.
BMI technology makes inputting text and speech with thoughts possible. Herff et al. [2015] show
for the first time that continuously spoken speech can be decoded into the expressed words from
BMI recordings. Their system can achieve word error rates as low as 25% and phone error rates
below 50%. Taking a cue from recent advances in machine translation and automatic speech
recognition, Makin et al. [2020] train a recurrent neural network to map neural signals obtained
from Electrocorticography (ECoG) directly to word sequences (sentences). They achieve better
performance compared to results reported in [Herff et al., 2015] in terms of average word error rates.
Furthermore, it has been shown that brain signals can be translated into speech based on a neural
decoder that explicitly leverages kinematic and sound representations encoded in human corticalactivity to synthesize audible speech [Anumanchipalli et al., 2019]. Stavisky et al. [2018] use SVM
to perform a classification analysis to estimate how much information about spoken phonemes
are present in the recorded neural signals. They show that high-fidelity speech prostheses may bepossible using large-scale intracortical recordings in motor cortical areas.
BMI technology has been deployed to explore important concepts in IR. Some important behavior procedures in IR have been studied, such as relevance judgment and realisation of an
information need. Towards relevance judgment, Moshfeghi et al. [2013] use Functional Magnetic
Resonance Imaging (fMRI) to identify brain regions activated by the process of judging the relevance of an image. Kauppi et al. [2015] show that fusion of Magnetoencephalography (MEG)-based
and gaze-based classifiers obtain promising performance on predicting the relevance of visual objects. Besides visual relevance, Eugster et al. [2014] also investigate predicting textual relevanceon the basis of multi-view EEG features. Expanding upon work on relevance, previous work delves
into using brain signals to understand information need in IR tasks. For example, Moshfeghi et al.
[2016] use fMRI to examine the neural processes involved in how Information Need (IN) emerges.
They show that there are clear, detectable, physical manifestations (i.e. neural correlates) of INs
in human brains. Moreover, Moshfeghi et al. [2019] try to predict the realisation of an IN using
brain signals extracted from a common set of predefined brain regions (GM) and a unique setof regions for individual (PM). Their findings show that brain activity could be used to predictwhether participants are in the state of information need.
BMI technology has been utilized to recognize human emotions. EEG-based brain signals (sometimes together with eye movements and facial expressions) are frequently used in sentiment andemotion analysis. Zheng et al. [2014] present an emotion recognition method which combines EEG
ACM SIGIR Forum

5

Vol. 54 No. 2 December 2020

signals and pupillary response collected from an eye tracker. In [Zheng et al., 2014], the combination of deep belief networks and hidden Markov model is used to classify two emotions (positive andnegative) using EEG signals. Soleymani et al. [2011] propose a user-independent emotion recognition method with the goal of recovering affective tags for videos using electroencephalogram(EEG), pupillary response and gaze distance. Petrantonakis & Hadjileontiadis [2010] develop a
new feature extraction method using hybrid adaptive filtering and higher order crossings for classifying six basic emotions with EEG. Recently, Zheng et al. [2018] design a six-electrode placement
above the ears to collect EEG signals. They combine EEG and eye movements for integrating theinternal cognitive states and external subconscious behaviors of users to improve the recognition
accuracy.
These existing research efforts show the potential of how BMI technology can be used in search
scenarios. However, to our best knowledge, there is no systematic research on how BMI may be
integrated into a search system and how this kind of integration can help us solve the challenging
problems faced by modern search engines.

3

BMI Search: What’s the Benefit

While it is hard to fully anticipate what the BMI search would be in the future, in this section,
we demonstrate the potential benefit of introducing BMI into search process. We focus on how it
may help us solve some challenging problems faced by modern search engine systems.

Benefit #1: Understanding Information Needs
While using the current query-based search interface, a user needs to formulate a query to represent
her information need and convey it to the search engine. Due to the effort of inputting longer
queries, users tend to issue short queries that only contains 2-3 keywords. Therefore, the queries
are often unspecific representations of the real information needs and search intents. It is also
well-known that the queries can be ambiguous as the same term may have multiple meanings. And
sometimes, the user may be unable to precisely formulate what she needs as she does not have thenecessary knowledge that is vital for formulating a good query. Therefore, it is sometimes difficult
for the search engine to accurately infer users’ information needs from those short, unspecific,
ambiguous, and clumsily formulated queries.
Different from these text-box based search interfaces, BMIs can provide richer information
about users’ information needs and search intents, in substitute of/in addition to the text query(IN Understanding in Figure 1). When used along with the traditional query-based interface,
BMI may provide information input with thoughts to help disambiguate the query and enrich
the unspecific intent representation. We may also use the BMI signal to detect the Anomalous
State of Knowledge (ASK) [Belkin, 1980] of the users (e.g. by detecting whether they are in
the state of information needs) and whether they have difficulties in query formulation (e.g. by
detecting the emotion of users). When detecting a difficulty in formulating queries, the system canprovide query suggestions or directly infer users’ information needs and retrieve relevant results
by decoding the BMI signals.

ACM SIGIR Forum

6

Vol. 54 No. 2 December 2020

Benefit #2: Collecting Satisfaction Feedback
User satisfaction measures users’ subjective feelings about their interactions with the systemand can be understood as the fulfillment of a specified information requirement [Kelly, 2009].
Satisfaction feedback from real users is valuable for both performance evaluation and system
optimization of search engines. However, asking users to provide explicit satisfaction feedback is
considered impractical for the contemporary search interface because it requires much effort fromusers.
Currently, there are two workarounds to estimate user satisfaction in search. The first one
is using users’ search behavior, such as clicks and dwell times, as implicit satisfaction feedback.
The second one is to estimate user satisfaction with relevance annotations and evaluation metrics
by building a Cranfield-style test collection. Either approach is subject to some limitations. The
implicit feedback based on user behavior is far from being accurate, especially for an individual
search session. And the Cranfield-style evaluation requires relevance annotations from external
assessors, which can be costly and may not necessarily align with the subjective feelings of realusers [Mao et al., 2016].
For the limitation in collecting satisfaction feedback from users, we can use BMIs to monitor
users’ emotions and sentiments unobtrusively (Satisfaction Decoding in Figure 1). By analyzing
how the emotions and sentiments evolve during the search, we can accurately estimate whether
the user feels satisfaction about certain search results and the overall interaction with the system.
Because BMIs can estimate user satisfaction almost in real-time, the collected feedback is not only
valuable for evaluating the search system but also useful for creating a closed-loop to optimize
users’ search experience (e.g. re-ranking results when users are not satisfied).

Benefit #3: Modeling Search Context
Users’ information needs and search behavior are affected by the context, including the physicalenvironment (e.g. at home/office/in commute), their prior knowledge (expert/novice) [Mao et al.,2018; White et al., 2009], as well as the stage of the search [Kuhlthau, 2004; Vakkari, 2001].
Therefore, it is important to take the search context into consideration and provide the right
results in the right context. Currently, the influence of search context is modeled by incorporating
additional signals, such as users’ geolocation, demographic information, and search history, into theranking models. But this kind of approach by no means provides a comprehensive representationof the search context.
With BMIs, we can capture much richer search context information. Compared to the existing
methods that mainly model search context with logged user behavior, BMIs may enable the searchsystem to probe and monitor users’ mental states during the search. By decoding the real-time
BMI signals, the system may even be able to estimate users’ expertise level, detect the changes inusers’ knowledge state, and infer the current stage of the search.
To summarize, regarding the challenging problems faced by the current search interface, BMIs
can:• provide clearer representations of users’ information needs and search intents;• collect satisfaction feedback from users almost in real-time;
ACM SIGIR Forum

7

Vol. 54 No. 2 December 2020

• capture rich context information during the search process.
These advantages over the current search interface make it attractive to integrate the BMI with
future search engine systems.

4

BMI Search: Challenging Problems

After discussing the feasibility and potential advantages of BMI search, in this section, we further
examine the potential challenges in implementing a BMI-based or BMI-enhanced search system.
As our ultimate goal is to build a BMI-based search system to augment humans’ cognition
ability and help them in memorizing, learning, solving problems, and making decisions, the biggest
challenge is to understand these cognition processes. We need to seek answers to the following
questions:• When and how do information needs emerge• How can information needs be fulfilled?
• Why do humans feel (un)satisfied during search?
We understand that these questions may not be perfectly answered with current technical advances
but the research of BMI search may also contribute to a better understanding of these fundamental
aspects of IR researches.
From a more practical perspective, to implement a BMI-enhanced search system as we show
in Figure 1, we will face the following three major challenges:

Challenge #1: How to define the interaction paradigm
The first challenge is to define the paradigm of the interaction between a user and a BMI searchsystem. Specifically, we need to define: 1) when and how the interaction is initiated; 2) how users
would express their information needs (the input of BMI search); 3) how systems would returnand display the search results (the output of BMI search).
In the current interaction paradigm of search engines, it is the user who initiates the search
process when she has an information need by submitting a query. The search engine passively
responses to the user’s query. As we discussed in Section 2, it is possible to detect the emergence of
the information need with BMI techniques [Moshfeghi et al., 2016]. Therefore, the BMI-enhanced
search system may then be able to proactively initiate the search process when it detects the
emerging of information needs from the user.
For the input of the BMI search, previous studies [Anumanchipalli et al., 2019; Herff et al.,2015; Makin et al., 2020] already show that it is feasible to use BMIs to input text and speech. So
the user will be able to issue text queries with BMIs. However, the query language patterns usedby the users of BMI search may be rather different from those used by the current search users.
One would imagine searching with BMIs is like talking to an information assistant with an inner
voice. So the queries would be more like questions and conversations in natural language. On theother hand, it is also possible that using the BMI search is more like recalling information fromACM SIGIR Forum

8

Vol. 54 No. 2 December 2020

one’s own memory. Therefore, the queries could be a few related concepts in the working memory
or even some subconscious thoughts. These differences lead to obvious challenges to the current
search system which are basically designed for keyword-based queries.
In terms of the output, modern search engines return a ranked list of search results when
receiving a query from users. The choice of returning a batch of results is optimal for reducing
users’ efforts and costs when the cost of formulating and submitting a query is several times larger
than examining and accessing a result [Azzopardi, 2014; Azzopardi & Zuccon, 2015]. However, as
the BMI might reduce the cost of querying, the user would issue more queries and examine fewer
results per query when using the BMI search. Therefore, it is possible that returning a single
search result, a short answer, or even a specific fact is optimal for the BMI-enhanced search. This
also poses technical challenges to existing search mechanisms.

Challenge #2: How to separate relevant context from irrelevant noises
The second challenge is to extract relevant information and filter irrelevant noises from the collected BMI signals. We already state in Section 3 that context information may help better
understanding user’s information needs. However, going too far is usually as bad as not going far
enough.
On the one hand, the BMI search system should capture more brain signals to model a richer
search context. For example, because users’ knowledge level may affect their search behavior
and relevance judgment of search results, the search system should monitor the changes in users’
knowledge states. On the other hand, the system needs to filter noisy and irrelevant BMI signals.
The BMI apparatus will inevitably introduce noises into the received signal. As the human brain
is functioning, random thoughts and ideas will constantly emerge. Therefore, the system should be
able to filter the noises caused by the apparatus and the signals corresponding to those irrelevant
thoughts and ideas. Judging which kind of signals is necessary for a certain search request will be
an important and challenging task for BMI search systems.

Challenge #3: How to protect privacy while collecting informationfrom BMI
The third challenge is to resolve the paradox between personalization and preserving privacy during
BMI search. As one would expect, the BMI search will be highly personalized. Different users may
have completely different ways and patterns in thinking and learning. The corresponding brain
activities and the recorded BMI signals would vary greatly across users. Therefore, it is necessary
to build personalized models to decode the BMI signals. Besides, the rich context informationcollected via the BMI would also be beneficial to the personalization of search results.
However, the highly personalized nature of BMI search may undermine the privacy of users.
This issue is more critical for BMI search because the BMI apparatus may “read the user’s mind”,
recording thoughts that she does not want to share, even without her notice or consent. Therefore,
we must consider how to store sensitive data securely and how to balance personalization withuser privacy. Key challenging technical problems include whether and how to share personalized
information with the search system and how to avoid abusing of these information.

ACM SIGIR Forum

9

Vol. 54 No. 2 December 2020

5

BMI Search: How Can We Start?

In this paper, we envision that the Brain-machine Interfaces (BMIs) technology may revolutionize
the existing interaction paradigm of search. A review of the literature suggests that it is potentially possible to use the BMI as a search interface. By inspecting the limitations of current
search engines, we argue that, even with the BMI technologies that are available now or in theforthcoming decade, we can enhance the abilities and performance of search engine systems in intent understanding, context modeling, and satisfaction decoding. We further discuss the potential
challenges of implementing an effective BMI-enhanced search engine.
At the end of this paper, we want to call for a joint effort of the IR community to advance theresearch in this exciting direction. Although we don’t have an off-the-shelf BMI search interface
yet, while waiting for the development in neural science and biomedical engineering, we the IR
researchers can start thinking about how to tackle some of the challenges mentioned earlier. For
example, for the challenge of defining the interaction paradigm of BMI search, we can investigate
how users search with an intelligent assistant. As the interactive mode of BMI search may besimilar to search with an intelligent assistant, such studies may help us to define a more userfriendly interaction paradigm for BMI search. For the challenges of modeling relevant searchcontext and balancing personalization with data privacy, we can start to collect richer user behavior
datasets via field studies and/or lifelogging [Dodge & Kitchin, 2007]. A field study will enable
the researcher to collect a rich and longitudinal user behavior dataset in a naturalistic setting,which will support the study on modeling search context and personalized search. The lifelogging
approach [Dang Nguyen et al., 2017; Dodge & Kitchin, 2007; Gurrin et al., 2016, 2019] leverages
apparatus, such as wearable cameras and activity trackers, to record one’s daily activities in
the forms of photos, videos, audio recordings, and activity levels. As the lifelogging records are
multimodal and may contain sensitive personal information, they are ideal for the study on how
to utilize heterogeneous information to improve the search experience without sacrificing users’
privacy.

Acknowledgements
This work is supported by the Natural Science Foundation of China (Grant No. 61732008,
61532011), Beijing Academy of Artificial Intelligence (BAAI) and Tsinghua University Guoqiang Research Institute. We benefit from discussions with our colleagues including Ji-RongWen, Zhicheng Dou, Jiafeng Guo, Jun Xu and Jie Tang.

References
Anand, A., Cavedon, L., Joho, H., Sanderson, M., & Stein, B. (2020). Conversational search(dagstuhl seminar 19461). In Dagstuhl Reports, volume 9. Schloss Dagstuhl-Leibniz-Zentrum
für Informatik.
Anumanchipalli, G. K., Chartier, J., & Chang, E. F. (2019). Speech synthesis from neural decodingof spoken sentences. Nature, 568 (7753), 493–498.

ACM SIGIR Forum

10

Vol. 54 No. 2 December 2020

Azzopardi, L. (2014). Modelling interaction with economic models of search. In Proceedings of the37th international ACM SIGIR conference on Research & development in information retrieval,(pp. 3–12).
Azzopardi, L. & Zuccon, G. (2015). An analysis of theories of search and search behavior. In
Proceedings of the 2015 International Conference on The Theory of Information Retrieval, (pp.
81–90).
Belkin, N. J. (1980). Anomalous states of knowledge as a basis for information retrieval. Canadianjournal of information science, 5 (1), 133–143.
Cerf, V. G. (2014). Cognitive implants. Commun. ACM, 57 (2), 7.
Dang Nguyen, D. T., Piras, L., Riegler, M., Boato, G., Zhou, L., & Gurrin, C. (2017). Overview
of imageclef lifelog 2017: lifelog retrieval and summarization.
Dodge, M. & Kitchin, R. (2007). ‘outlines of a world coming into existence’: pervasive computing
and the ethics of forgetting. Environment and planning B: planning and design, 34 (3), 431–445.
Eugster, M. J., Ruotsalo, T., Spapé, M. M., Kosunen, I., Barral, O., Ravaja, N., Jacucci, G.,& Kaski, S. (2014). Predicting term-relevance from brain signals. In Proceedings of the 37th
international ACM SIGIR conference on Research & development in information retrieval, (pp.
425–434).
Gurrin, C., Joho, H., Hopfgartner, F., Zhou, L., & Albatal, R. (2016). Ntcir lifelog: The first testcollection for lifelog research. In Proceedings of the 39th International ACM SIGIR conference
on Research and Development in Information Retrieval, (pp. 705–708).
Gurrin, C., Joho, H., Hopfgartner, F., Zhou, L., Ninh, V.-T., Le, T.-K., Albatal, R., Dang-Nguyen,
D.-T., & Healy, G. (2019). Overview of the ntcir-14 lifelog-3 task. In Proceedings of the 14th
NTCIR conference, (pp. 14–26). NII.
Herff, C., Heger, D., De Pesters, A., Telaar, D., Brunner, P., Schalk, G., & Schultz, T. (2015).
Brain-to-text: decoding spoken phrases from phone representations in the brain. Frontiers inneuroscience, 9, 217.
Jarosiewicz, B., Sarma, A. A., Bacher, D., Masse, N. Y., Simeral, J. D., Sorice, B., Oakley, E. M.,
Blabe, C., Pandarinath, C., Gilja, V., et al. (2015). Virtual typing by people with tetraplegia
using a self-calibrating intracortical brain-computer interface. Science translational medicine,
7 (313), 313ra179–313ra179.
Kauppi, J.-P., Kandemir, M., Saarinen, V.-M., Hirvenkari, L., Parkkonen, L., Klami, A., Hari, R.,& Kaski, S. (2015). Towards brain-activity-controlled information retrieval: Decoding image
relevance from meg signals. NeuroImage, 112, 288–298.
Kelly, D. (2009). Methods for evaluating interactive information retrieval systems with users.
Foundations and trends in Information Retrieval, 3 (1—2), 1–224.

ACM SIGIR Forum

11

Vol. 54 No. 2 December 2020

Krucoff, M. O., Rahimpour, S., Slutzky, M. W., Edgerton, V. R., & Turner, D. A. (2016). Enhancing nervous system recovery through neurobiologics, neural interface training, and neurorehabilitation. Frontiers in neuroscience, 10, 584.
Kuhlthau, C. C. (2004). Seeking meaning: A process approach to library and information services,volume 2. Libraries Unlimited Westport, CT.
Liu, T., Goldberg, L., Gao, S., & Hong, B. (2010). An online brain–computer interface using
non-flashing visual evoked potentials. Journal of neural engineering, 7 (3), 036003.
Makin, J. G., Moses, D. A., & Chang, E. F. (2020). Machine translation of cortical activity totext with an encoder–decoder framework. Technical report, Nature Publishing Group.
Mao, J., Liu, Y., Kando, N., Zhang, M., & Ma, S. (2018). How does domain expertise affect users’
search interaction and outcome in exploratory search? ACM Transactions on Information
Systems (TOIS), 36 (4), 1–30.
Mao, J., Liu, Y., Zhou, K., Nie, J.-Y., Song, J., Zhang, M., Ma, S., Sun, J., & Luo, H. (2016).
When does relevance mean usefulness and user satisfaction in web search? In Proceedings ofthe 39th International ACM SIGIR conference on Research and Development in Information
Retrieval, (pp. 463–472).
Moshfeghi, Y., Pinto, L. R., Pollick, F. E., & Jose, J. M. (2013). Understanding relevance: An
fmri study. In European conference on information retrieval, (pp. 14–25). Springer.
Moshfeghi, Y., Triantafillou, P., & Pollick, F. (2019). Towards predicting a realisation of an
information need based on brain signals. In The World Wide Web Conference, (pp. 1300–1309).
Moshfeghi, Y., Triantafillou, P., & Pollick, F. E. (2016). Understanding information need: An
fmri study. In Proceedings of the 39th International ACM SIGIR conference on Research andDevelopment in Information Retrieval, (pp. 335–344).
Nuyujukian, P., Albites Sanabria, J., Saab, J., Pandarinath, C., Jarosiewicz, B., Blabe, C. H.,
Franco, B., Mernoff, S. T., Eskandar, E. N., Simeral, J. D., et al. (2018). Cortical control of atablet computer by people with paralysis. PloS one, 13 (11), e0204566.
Perego, P., Turconi, A., Andreoni, G., Maggi, L., Beretta, E., Parini, S., & Gagliardi, C. (2011).
Cognitive ability assessment by brain–computer interface: Validation of a new assessment
method for cognitive abilities. Journal of neuroscience methods, 201 (1), 239–250.
Petrantonakis, P. C. & Hadjileontiadis, L. J. (2010). Emotion recognition from brain signals using
hybrid adaptive filtering and higher order crossings analysis. IEEE Transactions on affective
computing, 1 (2), 81–97.
Serruya, M. & Donoghue, J. (2003). Chapter iii: Design principles of a neuromotor prosthetic
device in neuroprosthetics: Theory and practice, ed. kenneth w. horch, gurpreet s. dhillon.
Soleymani, M., Pantic, M., & Pun, T. (2011). Multimodal emotion recognition in response tovideos. IEEE transactions on affective computing, 3 (2), 211–223.
ACM SIGIR Forum

12

Vol. 54 No. 2 December 2020

Sparrow, B., Liu, J., & Wegner, D. M. (2011). Google effects on memory: Cognitive consequencesof having information at our fingertips. science, 333 (6043), 776–778.
Stavisky, S. D., Rezaii, P., Willett, F. R., Hochberg, L. R., Shenoy, K. V., & Henderson, J. M.
(2018). Decoding speech from intracortical multielectrode arrays in dorsal “arm/hand areas” ofhuman motor cortex. In 2018 40th Annual International Conference of the IEEE Engineeringin Medicine and Biology Society (EMBC), (pp. 93–97). IEEE.
Trippas, J. R., Spina, D., Thomas, P., Sanderson, M., Joho, H., & Cavedon, L. (2020). Towards amodel for spoken conversational search. Information Processing & Management, 57 (2), 102162.
Vakkari, P. (2001). A theory of the task-based information retrieval process: a summary andgeneralisation of a longitudinal study. Journal of documentation, 57 (1), 44–60.
White, R. W., Dumais, S. T., & Teevan, J. (2009). Characterizing the influence of domain
expertise on web search behavior. In Proceedings of the second ACM international conference
on web search and data mining, (pp. 132–141).
Zheng, W.-L., Dong, B.-N., & Lu, B.-L. (2014). Multimodal emotion recognition using eeg andeye tracking data. In 2014 36th Annual International Conference of the IEEE Engineering in
Medicine and Biology Society, (pp. 5040–5043). IEEE.
Zheng, W.-L., Liu, W., Lu, Y., Lu, B.-L., & Cichocki, A. (2018). Emotionmeter: A multimodal
framework for recognizing human emotions. IEEE transactions on cybernetics, 49 (3), 1110–
1122.
Zheng, W.-L., Zhu, J.-Y., Peng, Y., & Lu, B.-L. (2014). Eeg-based emotion classification using
deep belief networks. In 2014 IEEE International Conference on Multimedia and Expo (ICME),(pp. 1–6). IEEE.

ACM SIGIR Forum

13

Vol. 54 No. 2 December 2020