ISSN 2186-7437

NII Shonan Meeting Report

No. 196

Everlasting Social Signal Processing From Labs into the Real World
Sozo Inoue
Wolfgang Minker
Björn Schuller
March 13–16, 2023

National Institute of Informatics
2-1-2 Hitotsubashi, Chiyoda-Ku, Tokyo, Japan

Everlasting Social Signal Processing - From Labs
into the Real World
Organizers:
Sozo Inoue (Kyushu Institute of Technology, Japan)
Wolfgang Minker (Ulm University, Germany)
Björn Schuller (Imperial College London, UK &University of Augsburg, Germany)
March 13–16, 2023

1

Description of the Meeting

Social signals play an indispensable role in our lives. To understand their importance, we can note the first emotions a child experiences soon after birth.
Negative social signals expressed in shouting, crying, and hand waving, perform
a protective or communication function, reflecting some kind of trouble for ababy: hunger, sickness, desire to sleep or play, etc. When growing up, the rangeof signals experienced by a person expands significantly, while at the same time
improves his or her ability to correctly understand those signals from people
around.
Social Signal Processing (SSP) is a new research domain that attempts tocomprehend and to predict social interactions and to provide computers withsimilar skills in human-computer interaction situations. For computers, thetask of Social Signal Processing remains difficult, especially when it comes toreal-world conditions. Nevertheless, the rapid development of pervasive andubiquitous computing technologies, machine learning algorithms in recent years
suggests that machines will soon be able to reliably perform this task as well.
Nowadays, a tremendous number of intelligent devices and objects, including
smart speakers, watches, fitness trackers, microwaves, smartphones, scales, etc.,
surround us. They all perform calculations and collect information. Every single
day more such devices emerge and form the basis of what we describe as “smart
environments”. These should enable us to perform pervasive and ubiquitous
computing. Thanks to the strong and recent development of artificial intelligence, the range of tasks that machines may be able to solve has expanded
significantly. However, the problem of Social Signal Processing, which is important in human-computer interaction, for instance, in multimodal dialoguesystems, remains challenging since the majority of the experiments are carriedout under clean and neat laboratory conditions.
Moving on to Social Signal Processing from labs in real-world conditions, weare facing novel types of challenges at each of the different stages of data analysis
including (long-term) data collection, evaluation, storing, and protection of theapplied models. All tasks that have successfully been resolved in the laboratory1may encounter significant burdens when transiting into the wild whilst applying
an identical set-up and the same approaches as before. Difficulties may alreadyarise at the stage of setting up the experiment. When conducting experimentsin the wild over a long time, it becomes necessary to take into account all
external factors that may affect the results of such an experiment. This can be
attributed to the stages of feature engineering and selection. Maintenance of theparticipants’ motivation poses another problem. For example, during negative
emotional experiences, participants may not attach proper importance to theexperiment and skip filling in questionnaires, which leads to poor quality of thecollected data. Another weak point seems to be the equipment itself that isused to carry out the research. While using complex and professional devices
in laboratories does not cause any significant problems, the possibility of using
those under real conditions may be rather limited. The question arises aboutfinding analogs to the original signal sources that will provide a comparable
signal quality. The examples described above do not cover all the problems, butshould reflect their complexity.
The aim of this Shonan meeting is therefore to discuss challenges that may
arise when transiting Social Signal Processing from laboratory conditions into
real environments and to propose suitable solutions as well as to establish aroadmap.
To move closer to the solutions of the challenges described above, significant
efforts are required from the scientific community involving specialists in thefield of computer science, medicine, and psychology. Sharing and combining
expertise from various scientific fields may lead to synergies, which will allow
us to create new ways to build solutions or increase the effectiveness of existing
approaches. The Shonan Meeting will help to explore possible challenges andjointly develop a research agenda for main directions. Therefore, we will invite
keynote speakers from the respective research fields, whose contributions willserve as a basis for breakout sessions. In these sessions, participants will work
actively on specific research objectives in small groups. This will help foster aninterdisciplinary understanding and cooperativity. The results of the breakout
sessions will then be discussed with the whole plenum.
We have decided to structure the workshop into four different areas. These
include typical research challenges that need to be addressed when transiting
Social Signal Processing from laboratory conditions into the wild. Secondly,
typical issues of development, testing and evaluation will be discussed. Suitable
use cases and industrial applications will be covered as well. Finally, ethics andsocietal impact need to be considered.

2Research challenges:• Experimental set-up accounting for real-world, real-time, large-scale conditions.
• Data robustness: suppressing noise and handling missing data.
• Data quality: Reliable annotation and motivation of participants.
• Feature engineering and modality selection for Social Signal Processing
taking into account the environment.
• Reliable physiological signal processing.
• Energy, memory, and computing efficient SSP. Model pruning/shrinking
for mobile use.
• From stationary to mobile systems performing SSP. Cloud vs. Edge computing.
• Efficient embedding of SSP.
• Protected data operating, such as federated learning.
• Green social signal processing protecting our environment.
• Virtual and Augmented Reality for SSP to simulate real-world conditions.
• Human-Computer Interaction making use of SSP.
• Data efficient learning and adaptation such as by reinforcement learning.
• Detection of social signal hiding.
• Real-time hiding and conversion of social signals.
• Adversarial attacks and methods to resist.
• Trust and usability.
Use cases and industrial applications:• Computer-mediated human-to-human interaction.
• Security applications.
• Health applications.
• Tutoring and E-learning applications.
• Entertainment and Gaming.
• Social Human-Robot Interaction.
• Success stories, functional systems and industrial challenges.
Development, testing and evaluation, ethics and societal impact:• Experimental design, user studies and evaluation of socially empoweredsystems.
3

• Engineering approaches to SSP: life-cycle, requirement elicitation, robustness to change, standardisation and simulation.
• Development models, tools and strategies: middleware, tools, languages.
• Meaningful and explainable system evaluation• Description, development and sharing of resources: corpora compilation,
annotation tools and approaches, crowdsourcing approaches.
• Pervasive sensing devices and frameworks, the Internet of Things and its
relation to SSP.
• Competitive research challenges planning and organisation
Ethics and societal impact:• Social responsibility.
• Data protection and privacy by design and default.
• Legal issues.

4

2

Meeting Schedule

5Overview of Talks
Adaptive Execution Environments at Scale
Prof. Dr. Stefan Wesner, University of Cologne, Germany
The heterogeneity of computing and data resources has been significantly increasing over the last few years. While this ongoing development delivers visibly
more compute capabilities at a reduced energy footprint it comes with inherent
complexity and a variety of programming models. This applies in particular if
the applications span across the Data Centre or are realised as a distributed
application across the Cloud Continuum. This talk will give a short overview of
the current trends in heterogeneous computing systems, approaches for an efficient use of these systems in High-Performance Computing, Scientific Machine
Learning covering also Secure Execution Environments for processing sensitive
data. Based on the solutions realised for centralised Data Centre/HPC Computing the challenges to realise execution environment across the Cloud Continuum
are presented.

Intelligent Interactive Systems on Mobile, Wearable andUbiquitous Computing
Prof. Dr. Kazuya Murao, Ritsumeikan University, Japan
This talk presents a selection of studies focused on various aspects of computermediated human-to-human interaction, security applications, health applications, and entertainment and gaming. The research encompasses a wide rangeof topics, including activity recognition and gesture behavior analysis utilizing
accelerometers, object recognition techniques, authentication methods employing glasses, health insurance integration with wearables, detection of irregularheart rates as a measure against cheating, muscle activity recognition using
Photoplethysmography (PPG), and the generation of multi-touch interfaces.

From in-lab towards out-of-the-lab, machine learning methods to adapt to an evolving environmentProf. Dr. Francois Portet, Universite Grenoble Alpes, France
Methods for solving real-world problems are typically designed in the lab before
being translated into the real world. Moving out of the lab is challenging, aslab conditions may be very different from real-world conditions. In this talk, wewill argue that moving from the lab to the real world is not only a problem ofmanaging condition mismatch but also a problem of dynamic adaptation. This
will be illustrated by three studies that show how machine learning methods canbe designed to cope with changing environments. The talk will conclude witha brief overview of the many limitations and challenges that remain to build
systems capable of addressing real-world problems.

6

Leveling Up Social Signal Processing through Video Games
Prof. Dr. Georgios N. Yannakakis, University of Malta & modl.ai, Malta &Denmark
Why are games so vital for the advancement of Social Signal Processing (SSP)
and affective computing? How can SSP help us test games, represent the games
we play, design creative AI algorithms, offer reliable agency to AI, and ultimately understand player experience? How have games challenged SSP in vitro
and in vivo (in the wild) settings and, in turn, have advanced research on annotation tools and affect modelling methods? I will attempt to address thesequestions through a series of milestone research studies that led to a numberof key lessons learned over the years. I will conclude the talk by suggesting
directions through which games can reframe the ways we approach SSP at large
and can help us develop both efficient affective computing algorithms and better
games.

Mining new realities of cities - lessons learned and beyond
Prof. Dr. Takuro Yonezawa, Nagoya University, JapanFor many cities, low-cost and sustainable acquisition of valuable spatiotemporalurban data is still the first challenge to be solved. In this talk, we will present
our approach and lessons learned from over 5 years of activities to understandcities by utilizing existing urban resources in any city, involving citizens. On
the other hand, as our reality extends from physical to virtual space (including metaverse), the social signals to be acquired and exchanged must be also
extended. This talk will also present our recent research efforts to understandand connect different kinds of realities.

Bridging the Gap: Challenges and Applications of Social
Signal Processing for Emotion Recognition in the Automotive Context
Dr. Sebastian Zepf, Hochschule Furtwangen University, Germany
Emotions play a central role in our everyday life: They affect several of ourcognitive processes in the short term, such as decision-making, attention andperformance in everyday tasks. In the long term, certain emotions and emotional stress can significantly impact our mental health and be a trigger formental illnesses such as depression and burnout. Due to the high influence ofemotions and the omnipresence of technical devices with integrated sensors suchas cameras and microphones that can provide emotion-relevant information, avast amount of research aims to realize automatic detection and understandingof the user’s emotional state. A generalized goal is to develop such algorithms toenable social interactions between humans and technological devices, with more
specific goals that range from more natural interactions between humans andcomputers, through counteracting short-term consequences of certain emotional7

states, to long-term improvement in mental health and well-being. While there
are various available and relevant signals for emotion analysis as well as technicalapproaches that are promising in laboratory conditions, there are still several
research problems that need to be addressed to bring reliable analysis of thesesocial signals from laboratories to real-world applications. For instance, theseresearch challenges include the varying availability of signals between different
applications, the context-dependency of signal quality, the person-dependencyof feature manifestation, the context-dependency of social signal interpretation,and the gap between signal interpretation and suitable social human-systeminteractions. This talk elaborates on these remaining research challenges andthe application-specificity of social signal processing for mental and emotional
state recognition and interpretation aiming to highlight the need and potentialof application-specific research. As a representative example, the talk mainly
focuses on the application in an automotive context.

Everlasting distributed and ubiquitous sensing - A contactfree sensing perspective
Prof. Dr. Stephan Sigg, Aalto University, Finland
Recent advances in RF-sensing (e.g. WiFi, cellular, LoRa, Bluetooth, etc.)
have demonstrated that communication systems may not only provide connectivity but also ubiquitous sensing and environmental perception capabilities. RFconvergence describes the arising competition for the limited shared spectrum
for ubiquitous sensing as well as communication, which opens a variety of concepts for the ubiquitous integration of sensing capabilities into communication
systems. This symbiosis of (cellular) communication systems and ubiquitous
perception creates new opportunities as it is globally connected and always-on.
On the other hand, awareness arises of the risks surrounding the technology
with respect to ethics. I will talk about our current work on RF-sensing, particularly joint communication and sensing. The talk will touch on challenges in
Ubiquitous perception, such as orientation-dependency, sentiment recognition,energy-neutral sensing and perception, as well as solutions to enhance privacy
in ubiquitous sensing scenarios.

Modeling Human Behavior and Quantifying Data Scarcity
Dr. Tahera Hossain, Aoyama Gakuin University, Japan
During my presentation, my main focus was to talk about utilizing inverse
reinforcement learning (IRL) as a computational modelling approach for modelling human behavior. By using such modelling approaches, we can simulatesituations that people find themselves in and predict the actions that they takein those situations. I discussed the Bayesian experimental design approach tocreating a generalizable sample size determination method based on Uncertainty
Quantification (UQ) for a specific Inverse Reinforcement Learning (IRL) modelof human behavior to deliver behavior-based healthcare interventions to patients with Multiple Sclerosis (MS) at their homes. I discussed the difficulties8

of constructing a computational model that can effectively handle real-world
patient data. Subsequently, I talked about my current project on heatstroke
prevention. In this project, I am collecting data from both controlled laboratory environments and real-world workers to analyze the effects of work stress,
mental stress, and environmental stress on individuals engaged in various activities. By modelling and studying human behavior under different stressors, itwill be possible to gain a better understanding of the impact of different stressfactors. We are investigating not only the effect of the thermal environment
on biological information but also the effect of the work when it overlaps withthe mental or physical burden. I mentioned several challenges in real-life datacollection, data annotation, and data analysis. Furthermore, I described another project that I am currently involved in, which aims to comprehend themental state of office workers by analyzing a multi-modal real-life dataset. I
explained the research challenges of deciphering psychological factors in a realworld work environment. In short, I explained a bit about verification studies
in real-field applications from real nursing care big data collection, data analysis, and challenges for implementing advanced machine learning to understandelderly people’s real-life complex activity data.

93Working Groups

The participants of this meeting were divided into three Working Groups (WG)
and discussed the following three topics.
• Research challenges• Use cases and industrial applications• Development, testing and evaluation, ethics and societal impact

10Summary of Working Group Discussions

11

4

Research Challenges of SSP

Social signals are behaviours (natural or synthetic) that can be sensed (by living beings such as people, by machines or by both) that can be interpreted interms of socially-relevant information (emotions, attitudes, personality, etc.).
Correspondingly, Social Signal Processing is the technological attempt to automatically measure, interpret, convert, and generate observable behaviours. Fig.
1 shows word class visualisation for research challenges in SSP.

Figure 1: Word cloud visualisation for research challenges in SSP.
The Working Groups identified several challenges related to SSP and presented them in Fig. 2. We discussed positioning SSP in the current state of theart. In a sense, SSP is related to affective computing, emotion AI, psychology,
persuasive and pervasive computing and several technological sciences underlying these fields, such as edge computing, IoT, ML/AI, etc. As a key challenge,
we identified the field’s ever-present challenge to obtain “ground truth” (e.g.,by the neurological base), i.e., the true interpretation of behaviour. Rather, thegold standard in most cases is to ask human raters to interpret and form some
sort of agreement such as by (weighted) mean of opinions. This provides thechallenge of subjectivity and therefore uncertainty both in the learning target
and the testing reference, ultimately not allowing machines to surpass humanlevels and rendering the interpretation of evaluation equally uncertain. Related
to this lack of ground truth is the challenge of proving that “something works”.
This comes, as (reproducible) research usually reports results on databases. As
we do not know for sure, however, how reliable the labels are, we cannot be
sure about the meaning of measured accuracy or likes on such databases. While
humans could be asked to assess the quality of outcomes of automatic SSP models, this would a) again lead to uncertainty and b) not be reproducible while c)coming at considerable effort.
Ultimately, this leads to a general challenge of evaluation of SSP systems,
which urgently needs to be addressed before deploying such at large in realworld applications. An alternative could be an indirect measurement of their
quality by task-solving abilities that require social competency.
A further bottleneck remaining a challenge to-date is the lack of access toreal-life data of social signals from diverse and multiple modalities. In particular,such has to be realistic and ideally recorded without an observer paradox andcater for diversity such as various individuals contained in numerous situations
12Figure 2: Hierarchically grouped identified research challenges in SSP.
including multiple environments, allowing training systems to also learn to deal
with noise, reverberation, occlusion, missing data, and likes.
The quality of the measurement and the interpretation of collected data depends on the quality of the data collected but similarly also on how fast andefficiently the data can be analysed and processed. The ability to deliver results
under real-time conditions depends on the available infrastructure as well as on
the efficiency of the implementations. In particular, transferring scenarios froma controlled and limited lab environment towards large-scale deployments imposes significant challenges related to scalability, dependability and robustnessof the underpinning infrastructure. This also comes with the challenge to dynamically adapt to changing situations by changing the amount of data that iscollected (just enough but not less), changing allocation of available resources,using different levels of accuracy and detail depending on the situation or balancing accuracy with time to result. From the computing and communication
infrastructure side, edge computing provides means for building optimized SSP
pipelines, as well as faster and more reliable on-the-air communications. Ideally, a transparent symbiosis between the edge and end-user devices could beestablished. Through the edge, it would be possible to reduce the time requiredfor data processing, model building, and AI/ML. It is highly probable that network virtualization will play a crucial role, as it allows specialized software to be
distributed across nodes in order to optimize resource utilization. By sharing
portions of available data or trained AI/ML models, it is possible to greatly
reduce latency in rapidly changing situations, while ensuring that user qualityof experience is not compromised.
In addition to the above, the Working Groups identified the following main
challenges:
Fairness: There are a large number of factors that can lead SSP approaches
to work better for some users (e.g., caucasian males) than for others (e.g., Asianfemales). In some cases, this depends on biases in the data (the training material does not include examples of certain users), in other it is a matter of
hardware limitations (some digital signal processing chips tend to penalise frequency ranges characteristic, e.g., of male voices) or biases inherent to the data13

(all approaches will tend to work better for the types of users most frequently
represented in the data). The solution to these problems, and the others thatkeep emerging as AI becomes an increasingly more commonly diffused technology, is the focus of Human-Centred AI, an area likely to provide major support
to SSP.
Long-term and short-term Adaptation: Human behaviour is characterized by regularities that can be modelled over long periods of time. However,
it is also deeply contextual and it is always situated, i.e., it always depends onhighly specific aspects of the setting and situation in which it takes place. For
this reason, SSP approaches face the challenge of dealing with two conflicting
needs, namely to reflect long-term tendencies, while being capable to deal withshort-term changes and exceptions.
Ethically Compliant Technology: Different legal frameworks are needed
for SSP when operating in different states to ensure compliance with laws andregulations that are applicable to it and minimizing the risk of noncompliance.
For companies, not following these compliances induce risk in their business andmight lead to considerable sanctions. For ethical frameworks for SSP, different
cultural aspects need to be assessed and then the question is about doing “theright thing” in the social context. In the European Union, the AI Act emphasizes the importance of transparency, explainability, and awareness of bias inintelligent systems. Interpretability pertains to how much humans can understand and anticipate the systems’ decisions. On the other hand, explainability
is the ability to describe precisely how the model operates. It is important tocontinuously update SSP systems, also, to be able to remove the effect of bias in
the models. Developing SSP systems requires the inclusion of individuals withdiverse backgrounds and a variety of needs. It is crucial for both the researchcommunity and industrial player to establish methods and practices that facilitate the participation of vulnerable user groups, minorities, and individuals at
risk of marginalization in the development of these systems. This is particularly
important during the data collection phase, where we aim to minimize bias, increase coverage, and improve data quality and testing phase when the systems
are evaluated.
SSP technologies deal with the inner state of people and promise to know
about people more than what they know about themselves or, at least, morethan what other people know about them. This poses serious ethical challenges,
including who should regulate the field and how (this step must involve legal
experts, philosophers, sociologists, anthropologists and, most importantly, users
expected to benefit from technology), what should be the red line beyond which
SSP should not be pushed (it is necessary to define clear boundaries for theeffectiveness of SSP technologies that should not be capable to access, or at least
to share, information that should remain under the control of people), how toensure that any decision made by an SSP system is explainable (there should
be a clear ground for falsification of SSP technology outcomes), how to make
SSP fair (capable to work equally well for all users irrespective of characteristics
these are not responsible for and cannot change) and inclusive (capable to avoid
excluding any users for characteristics these are not responsible for and cannotchange).
Common-sense Grounding: communication between humans relies on
implicit or tacit information, i.e., a body of shared knowledge that can be given
for granted (often referred to as common sense). A typical example is the14

following dialogue:-Him: “I leave you”
-Her: “Who is she?”Such an exchange is impossible to understand on the sole basis of the words
it contains, but this is all SSP can access for the moment. Humans make senseof the dialogue because they can integrate the words with their experience of thereal world. Machines are still unable to access and store such an experience. For
this reason, endowing machines with common sense is one of the major bottlenecks towards making them more capable to seamlessly integrate our everyday
life [16].
Explainability: Explaining different stages of SSP (data collection methods, analyzation, synthesis and interpretation) to novice users considering recipients’ experience, background and educational level. It might be necessary to
explain such systems in different ways to different audiences. The focus should
be on creating transparency to increase trust, but also to decrease dependency
as well as to increase system robustness and resilience by providing knowledge
and tools to users to carry out the maintenance of such systems as well as toadapt them to changing and arising personal or situational needs.
Dependability: Becoming highly dependent on SSP can cause loss of expertise and knowledge since activities (e.g. decision making, data analysis, observations) are carried out and processed automatically by intelligent systems
without human oversight or collaboration. In the long term, this can create asituation where humans neither have the knowledge nor expertise to evaluate
and judge the quality of the outcome of SSP, increasing dependency on such
systems. It has to be ensured that in the short as well as in the long term humans can still function and operate without the support of SSP and that humanoversight is ensured across all stages of the process.

4.1

Most urgent challenges

One of the most pressing research challenges in AI technologies underlying SSP
is obtaining the ”ground truth” and ”validation” of research findings from clinicians. Additionally, it is essential to carefully select data and employ efficientalgorithms for processing the information, ensuring that it can be effectively applied in real-world scenarios. In addition, it is crucial to strike a balance between
personalization and anonymization while also having access to multimodal data
and addressing data sparseness, to achieve both efficiency and comprehension.
Moreover, striking a balance between personalization and anonymization is alsoessential. On the other hand, as SSP deals with human behavioral data andalso considers the personal factors e.g., emotions and cognition, and therefore,
it is crucial to prioritize ethical compliance and cultural differences as urgent
issues to consider. In SSP, ethical compliance is important in these types of research because it ensures that the study is conducted in a manner that respects
the rights and well-being of participants, as well as the integrity and credibility of the research findings. Compliance with ethical guidelines and principles
will help to prevent potential harm or exploitation of study participants andalso will help to ensure that the research is conducted in a way that is scientifically valid and reliable. Also, cultural factors can have a significant impact
on SSP research, because it is important to know how individuals perceive andrespond to certain issues, which is very much dependent on various cultural and15

social acceptance factors. Failure to consider cultural differences can lead toinaccurate or incomplete data, as well as can potentially hurt participants unintentionally. Understanding and respecting cultural differences can help to build
trust and rapport with study participants, which can lead to more meaningful
and accurate data. Overall, it is important to recognize and address culturaldifferences in a broad way in order to conduct SSP studies that are both ethicaland scientifically valid.
Overall, the main issues can be summarised as follows:
Data quality: Data quality is also an urgent issue in research because theaccuracy, completeness, and reliability of the data directly affect the validity
and usefulness of the research findings. Poor quality data can lead to incorrector misleading conclusions, which can have significant consequences for decisionmaking in various fields, including healthcare. Additionally, data quality isessential for the reproducibility and transparency of research. Other researchers
should be able to reproduce the study findings using the same data, and the data
should be openly available to allow for scrutiny and verification. In summary,data quality is essential for the credibility and usefulness of research findings,
as well as for ensuring that research is conducted rigorously and transparently.
Getting ground truth (reliable/valid data):
Observer’s paradox: ”To obtain the data most important for linguistic theory, we have to observe how people speak when they are not being observed”(Labov, 1972)
Efficiency (careful selection of data – sensors, quality and size, more efficientalgorithms, realistic use cases): Cost of resources (time, monetary, human resources) have to be considered when SSP is employed in the field. There is a
general belief that more data will contribute to higher model accuracy, but forfield studies decisions of what, how much, and for how long data will be collected have to be also based on cost-efficiency aspects. How much burden can
you place on stakeholders, how many resources are available, etc.? Researchers
have to consider how to reach acceptable reliability and validity with a minimal(or limited) set of data. Efficiency here would also refer to using already existing
infrastructures for data collection and considering energy-efficient methods.
Multimodality: SSP deals with human emotions and cognition; therefore,
to interpret these complex behaviours, it is necessary to measure the observable features through several techniques. Visual and aural are natural and mostcommon sources of information about our internal processes. Also, gesturaland linguistic outputs have been thoroughly studied in affective and ubiquitous
computing and other related fields. In the SSP research community, we needboth multimodal data acquisition techniques as well as versatile multimodal userinterfaces to present the outcomes of the analyses to diminish wrong interpretations. Social situations happen fast and the synchronisation of signals needs tobe accurate and fast. The field is closely related to pervasive computing developing novel techniques and devices to measure different events in our everyday
environment. For example, wearables are getting tinier and more powerful inthe capabilities of computing, battery life and sensor integration. The emerging
radio-sensing techniques bring several possibilities for non-invasive observationof humans in groups and individually. Video is also handy in observing groupdynamics, what is the social situation of people in lectures has been of interest.
Holism: Most if not all approaches towards SSP tackle one aspect at a time,
but do not take into account the whole picture. As an example, a system may
16

detect the emotion of a person, but not its sleepiness. However, for a reasonablehigh-level interpretation, states, traits, and behavioural signals of an individual
ideally need to be considered simultaneously - alike, when generating coherent
and convincing behaviour in an agent or alike.
Protection against SSP: All technologies can be used for good and bad.
We cannot prevent misuse but we can try to minimize the risks. In this paper,
we discuss ethical aspects, required transparency of the systems and privacypreserving techniques around SSP. Also, SSP systems need to take responsibility
for any emotional manipulation that might happen in the systems. The companies leveraging SSP to make a profit need to carefully make sure their systems
operate as they are supposed to and back up their results in scientific research[11, 13].

4.2

How to address identified challenges?

Better training in experimental research: To increase the quality of research, especially the quality of data collection thorough training in experimental research is a necessity. Researchers need to have a knowledge foundation of
how to conduct social science research when collecting human behavioural data,
be familiar with measurement concepts such as construct validity, internal validity, reliability, and scaling, and understand the advantages and disadvantages ofdifferent types of research designs (experimental, quasi-experimental, pre-post
designs). This also includes fundamental knowledge of statistics and how toanalyse and interpret data.
Balance between Personalisation and Anonymisation (Clustering/ Groupisation): AI technologies underlying SSP make use of statistics extracted from large numbers of individuals, but they can be adapted to individual
users, thus resulting in highly specific models for every individual. This poses amajor threat to privacy that can be countered by limiting adaptation to clusters
of individuals sharing similar characteristics, but still being sufficiently diverse
to avoid individual targeting (an approach referred to as groupisation).
Data Standards: The scientific community needs to discuss and agree on
basic standards concerning data reliability (measurement accuracy) and validity(how well we measure what we want to measure), acceptable values and how
these values are generated and measured. Naturally, SSP should drive for highquality data that are as close to the true values as possible. However, noise in
the data, uncontrollability of situations and environments, individual differences
as well as measurement errors can contribute to errors in observed values.
Data quality: Inaccurate or incomplete data can arise from a variety offactors, including errors in data collection, measurement, or recording, as wellas biases introduced by the researchers or participants. It is important to ensurethat the data collection methods are standardised, reliable, and valid in orderto minimise these sources of error and bias.
Collecting data must not only address the challenge of ensuring the technicalquality of the measurement such as resolution or accuracy but also requires sufficient provenance data describing how it is collected and the processes applied
to transform and interpret the data. Also the establishment of data standards
not only for the raw data but also for derived data is a pre-condition allowing
exchange and integration of data collected in different experiments and acrossscenarios. Open datasets build the baseline for comparing different approaches
17and methods in terms of accuracy, efficiency and processing time. For achieving
real reproducibility of experiments these data sets must not only be stored incommon or transformable data format but in particular the experiment settings,
assumptions and constraints must be documented.
Experiment documentation: In order to compare different approaches
and solutions experiments must not only collect the data in a standardised format and provide them openly but also the full experiment should be documented
e.g. using Electronic Lab Notebooks or similar means to allow reproduction of
the setting or situation. This includes not only the settings to collect the data
but also the collection of factors that might influence the behaviour of the participants in the experiment. This is besides the openness of the data and applied
procedures a key element for the reproducibility of experiments. Experimentdocumentation includes not only means of data collection but also experimental procedures, instructions given to participants, detailed descriptions of theexperimental setting, as well as procedures for data cleaning and reduction.

4.3

How to put possible solutions into practice?

Transparency: The major challenge to be addressed is to achieve trust in the
SSP system avoiding uncertainty about what and how the data collected areused. This includes clear communication of the benefit for the participant andnot only the system operator. While transparency about the data collected canbe achieved in a controlled setting or an experiment and partially about theprocessing of the data and its use, this cannot be applied to settings including
a public space where persons get involved without explicit consent. In such
scenarios, the collection of data must be controllable by individuals e.g., by accepting to use a mobile application or specific devices communicating acceptanceto participate.
Good communication between all parties involved: Application of
SSP requires a full understanding of how SSP technologies work and how theiroutcomes should be used and interpreted. This requires all parties involved in an
application to develop sufficient literacy. SSP developers should be sufficiently
literate about the problem they are going to address to avoid the main pitfalls of
the technology such as, e.g., spurious correlations that make an approach rightfor the wrong reasons [19]. Users should be sufficiently literate about the way
SSP work to reliably understand when its outcome can be trusted and when
it cannot. This latter form of literacy is particularly necessary for a view ofdesigning human-AI collaboration approaches in which technology is developed
to support humans in their activities rather than replacing them.
“Data Donation” services and gamified socialised annotation sites:
Citizen Science projects involving the greater public may help ease the lack of(labelled) data. What appears important is to provide service back to those
helping in such a framework for ethical reasons, but also to better incentivise
and lend a better understanding of the mechanisms and workings behind those
involved.
Reaching out beyond the SSP community: sparking interest in thecase of SSP in the “core” involved communities such as the core AI, or otherrelated “core” communities in Computer Science and the Humanities may help
reach tailored solutions and answers to the problems of SSP and help get otherperspectives on the problems in the field.
18

Standardisation: in many ways will further foster comparability of findings, but also the applicability of solutions in commercial and mass product
SSP solutions. While some standards such as EmotionML or defined featuresets such as GEMAPS may be a start, many more are needed to cover thebroad and diverse field of SSP.
Collaboration with psychologists / multidisciplinary/intersectional:
one of the main characteristics of SSP is that it requires experts to know about
people as much as they know about machines. In this respect, the developmentof any SSP approach needs interdisciplinary expertise that covers both technicaland human aspects of a problem. In addition, given that SSP technologies canhave an impact on lives of people, especially when bridging the gap between
the laboratory and the real world, it is increasingly more important to involvesociologists capable of assessing the societal changes that might result from theapplication of SSP. In addition, SSP deals with various factors and issues which
require different labels of expertise to make it socially acceptable to a larger
target of users in the end.
For the purposes above, it is essential to collaborate among different disciplines, where it will be possible to get validation of research findings in multiple
ways. Collaboration within multiple disciplines will also be beneficial to split
the research challenges so that different experts can contribute in different ways
and no need to take too much burden from a single point of source.
On the other hand, it is still difficult to conduct genuinely interdisciplinary
research due to the lack of appropriate venues and the reluctance of unidisciplinary research communities to recognize and accept research questions andpractices that come from other areas. Such an attitude can be observed in mostfunding agencies too that, while encouraging interdisciplinarity and collaboration across multiple areas, are still unable to deploy calls, review and selection
processes in which interdisciplinarity is recognized and assessed as such.
Lack of recognized applications: no widely known application of SSP isavailable yet and this makes it difficult to further progress in the area (including
addressing all problems mentioned so far) because there is no input comingfrom real-world experiences. The most promising areas at the moment seemto be personal assistants (there are attempts to make Siri and Alexa more
empathic and capable of sensing the inner state of their users) and social robotics(there are multiple examples of the deployment of social robots in different social
settings). However, SSP is not recognized as a crucial underlying technology
in both cases (the attention tends to be on the interface rather than on theunderlying technology).

4.4

Difficulties in the real world

One of the main challenges of social signal processing is the real world’s difficulties, including issues such as controllability, stakeholder involvement, andshifting ground truth. There are different environments in which social signal
processing can be studied, from in silico simulations to in vivo testing with alimited number of participants, and ultimately in real-life situations that are
unpredictable and complex. In these settings, there is an unclear target forsocial signals, a lack of contextuality in models, and difficulties in machinelearning adaptation and memory. In SSP research, when we move from lab tofield there are numerous other issues, which can make the real-field experiment19difficult, for example, data collection (without interrupting their current workduties), understandability (participants need to familiar with the technologicalenvironment), good network infrastructure (to support the data collection), explainability (participants need to have well understanding of our experiment).
As well as before the experiment it is needed to explain the use case of any
personal label of data, which may cause them a certain level of fear to makeavailable any personal information. These are not the only difficulties for realfield data collection but also evaluating uncontrolled data makes the SSP research more difficult to evaluate for research purposes. On the other hand, in
healthcare research, patient data collection is sometimes difficult and data collection situations using wearables are not always convenient for them. Also,
when it involves cost then we should be aware of optimum data collection as
uner-collection will make machine learning models uncertain to predict certain
symptoms of patients or we will not be able to make robust human behaviourmodels. However, over-collection will take unnecessary costs and increase theburden of data collection.

4.5

New findings

There are many existing categorizations for clustering different human behaviour,e.g. psychological patient assessments, personality traits, etc. However, there isa huge cap on how non-verbal and verbal communication can be mapped to theinner state of humans or human groups. The SSP community is highly multidisciplinary as there are clear needs to find the constraints of different settingsin terms of social behaviour, map the social signals to either existing models ordevelop new categorisations of human behaviour and reach a consensus on whatis actually happening in the situation. Group dynamics also create complex
feedback loops in the system. In this regard, it is crucial to prioritize the controllability, explainability, and dependability of the developed systems, as low
user acceptance can result if the system fails to operate correctly. Another aspect is We have also recognized the potential impact of emerging technologies,
such as brain-computer interfaces, on providing more accurate ground truth
for social signal processing. Additionally, it will be intriguing to observe how
the emergence of quantum computing or the development of general artificial
intelligence will affect SSP.

4.6

Identified issues

One of the main issues in the large-scale adoption of SSP is the high uncertaintyof the interpretations of the social signal. We recognized several already productised applications of SSP (Valossa Labs [22], VentureRadar [21]), medical
applications, etc. ). However, large-scale deployments are still hindered by the
ambiguity of social signals, types of intentions of humans, difficulties in assessing
group dynamics and feedback loops in scenarios, and situational dependencies.
No foundation models of affect/behaviour: There is no general, unified
model of affect and behaviour. A variety of existing models attempt to explain
different aspects of affect and behaviour. These include physiological, neurological, and cognitive theories of emotion. These theories attempt to explain various
aspects of affect, such as when and why we feel emotions, how emotions influence20

our behaviour and thinking, and how emotions have evolved throughout evolution. However, to improve data collection, analysis, and interpretation of SSP,
a comprehensive understanding of the intertwined relationships between affect,
cognition, and behaviour in different social contexts, as well as the underlying
neurological processes, is needed.
No unique theoretical backing: see above
Lack of acceptance: As it is very important to increase transparency andexplainability of the SSP systems, security and privacy are other important aspects to be considered. Edge computing and distributed intelligence will increase
the security and privacy of the systems due to their independence from multitenant clouds and the possibility of serving computing resources near the user.
With decentralized decision-making and trust management, the users could begiven access to locally dynamic consent management, for example.
Thorough testing is necessary for the systems, particularly as we expect
SSP systems to transition from laboratories to living labs and eventually to thereal world. The testing can be conducted in various ways, including ”in use, insitu” experiments, laboratory-based testing (in vitro), or controlled high-fidelity
simulations of the real world (in sitro).
Expectation management / Uncanny valley: Very often, intransparency
or lack of understanding can create the feeling that something is uncanny. Explainability, as well as transparency, are necessary prerequisites to avoid thiseffect. Users’ expectations need to be carefully managed to create a realistic
understanding of what SSP is able to do as well as where its limitations are.
Expectation management is influenced by various factors, starting with design
aspects (if something looks human, we expect it to behave human) and communicating a system’s purpose and functionality.

5

Use Cases and Industrial Applications

Within the discussions around SSP, various applications and use cases havebeen identified, including health applications, safety supervision, fitness andwell-being, customer service, smart homes and cities, office and human resource
management, digital games, learning at all stages, dating, security surveillance,
behaviour analysis on a community level to improve life quality, collaboration
with robots and agents, driver monitoring and entertainment in the automotive
field, and dialogue in various domains where virtual assistants play a role. Regarding bringing these applications to the real world, the main requirements to
be fulfilled are as follows:
1. The development of reliable, valid, and generalizable models.
2. A clear definition of the roles and objectives of the models in the loopof social signal sensing, processing, interpretation, and interaction that
includes in which way and at which time the model shall be utilised.
3. Careful usability engineering that fits the respective use case.
Regarding the current state-of-the-art, there are remarkable differences between
different application areas and application-specific issues remain. There are application areas that are well-explored and already well-established. For instance,21social signal processing in the domain of digital games is mature and rather established when it comes to signals related to gameplaying behaviours such as
game balancing, and player profiling and matching [10]. SSP in games, however,does not normally rely on other forms of social signals or modalities of user input such as eye tracking, physiology, and so on. There are several reasons why
player behaviour-based SSP systems are mature in the domain of games. First,
the underlying SSP technology supporting games is well established. Massive
multiplayer games have been operational since the early 90s through games suchas DOOM (id Software, 1993), which enabled multi-user interaction through thecloud. Second, player data collection and processing through telemetry was introduced as a development service at least 15 years ago and gradually deployed
as a back-end service to the majority of games. Such player behavioural data
has been analysed through simple data aggregation techniques and clustering toinfer groups of players. Those groups can in turn form the basis of player matchmaking and game balancing which are nowadays dominant (and necessary) in
multiplayer games. Microsoft’s TrueSkill [15] and TrueMatch match-making
services are available in Xbox games are indicative examples of such in-game
adaptation systems. In addition, player imitation learning systems such as the
Drivatar AI assistant in the Forza (Xbox Game Studios, 2005-present) series
have been in existence since 2005. Finally, it is worth noting that the reason
behavioural SSP in games is so mature is that it brings low degrees of risk to
its users (designers and players) and is applied in a very controlled (simulated)environment. Low-risk (yet critical) factors include the emergence of toxic behaviours and cyberbullying [4]. In other application domains, the explorations
are already in an advanced state, the proposed solutions however are not wellestablished yet. An example is the domain of healthcare and well-being, inwhich some promising prototypes leveraging social signal processing to address
varying use cases exist already. Studies in controlled environments and clinical
trials indicate that the proposed solutions are promising and have the potential
to improve patients’ health and well-being as well as support healthcare professionals in their daily tasks. However, the step of bringing the prototypes toreal-world usage and achieving user acceptance is still a hurdle that needs tobe taken [20]. Lastly, there are promising applications that still require basic
exploration. For instance, we see great potential to make an impact on safety
by reducing the risk of user harm and accidents in driving and human-robot collaboration use cases. However, these applications need real-time interventions
based on SSP and thus have even higher requirements regarding the reliability
and completeness of user modelling.

5.1

Identified issues

Across the different domains explored and analysed, the remaining issues andchallenges we identified are highly dependent on the application scenario. In thedomain of digital games for example, SSP is required to expand to other modalities of user input so that user states (beyond mere gameplaying behaviour) canbe identified and inferred. Specifically, it is still not clear how player experience models can be integrated not only into the game design and development
process but also directly during gameplay. One potential way to create actionable SSP in games is through the adaptation of game content in the formof experience-driven procedural content generation [24]. However, the effects22

of overpersonalization and filler bubbles need to be considered carefully. The
core ethical concerns of highly engaging and personalised games are that theypotentially can lead to forms of player addiction [25].
For healthcare applications, the transfer from prototype level to actual clinical applications is still a hurdle [23], with the main reason being a lack of trust
between the users and the systems [1]. Some of the reasons why solutions in thisdomain are less mature are the sparseness of the available data, the additional
efforts that need to be invested when collecting behavioural and health-related
data to account for privacy and security aspects, and the limited accessibility tohighly specific user groups such as patients with unique diseases or health carestaff from specific disciplines.
Regarding applications of real-time safety interventions, the available models
still lack sufficient reliability in the detection and interpretation of social signals[26]. One of the main reasons is that we are unable to generate all possible abnormal or dangerous conditions during the experiment due to safety and ethicalconstraints. Another big issue is that these systems are highly user dependentand data from a real-world environment needs to be collected. Therefore, it
is crucial to ensure that the data collection does not interfere with the task to
be done by the user to ensure that the behaviour is not altered and that thecollected data is representative. Further, the data collection and labelling needto consider the specific emotional states that are relevant in a certain context.
Accordingly, achieving a reliable annotation is a challenge [7]. Moreover, evenif we cover all the possible dangerous cases, some other scenarios may occur inreal life as the user may be influenced by unexpected contextual stimuli in their
daily life, personal interests, or many other reasons [27].

5.2

Future Applications

While the applications mentioned above have already attracted a lot of researchattention over the last decade, there are some fairly new potential application
domains and use cases for SSP that emerged recently. The celebrated metaverse
offers a promising future application domain for SSP. The underlying structure,technology (XR, rendering, usability, and interfacing), and functionalities ofthe metaverse are an ongoing design process [6] and, thereby, the exact SSP
uses remain to be found. Despite the immature state of the metaverse, we
argue that SSP in the metaverse will likely be tasked to improve human-human
but also human-to-agent interaction. Further, SSP-enabled agents are expected
to validate, test and enhance the capacities of the metaverse(s) created. The
metaverse is also expected to serve as an immersive version of digital twins of
any type. Think of countries, cities, buildings, rooms, and human digital twins
as indicative examples [17].
State-of-the-art generative AI methods, including diffusion models, largescale language models (LLMs) and, in general, transformer-style architecturesand foundation models [3], will likely be equipped with SSP capacities. HumanAI co-authored processes and outcomes, including text, images, and videos canonly benefit from social signals and enhanced interaction modes. We foresee theintegration of additional user modalities as inputs (prompts) to such foundation
models that will empower their generative capacities. We envision fine-tuned
image-to-image, text-to-image, and text-to-text (GPT variants) models to beable to serve as SSP-to-content methods. Content here refers to text, images,
23

visuals, videos, and even games.
Another potential application area for social signal processing could be general sales and insurance processes. For instance, behavioural information about
driving behaviour [5] could be leveraged to introduce dynamic pricing of insurance [9], which might ultimately lead to safer driving behaviour and increased
fairness of insurance fees. Other use cases might include behavioural-dependent
pricing for health insurance or bringing more transparency into high-stakes sales
dialogues [8]. However, the limits and risks of the applications listed above need
to be examined thoroughly: Estimating the value of individuals and their healthstate [14] yields a high risk of discrimination and can lead to prejudice in various ways. Accordingly, the accessibility of this type of information needs carefulconsideration and transparency of the processing is highly important [2].
Overall, there is a vast amount of promising ideas and applications for SSP.
Within this whole domain of SSP, it needs to be noted that it is crucial tothoroughly consider ethical aspects and boundaries, as well as data privacy
and security issues [18]. Since the definition and perception of ethical aspects
as well as legal constraints differ between countries and cultures, the different
potential user groups and regions need to be taken into account. Social signal
processing can be misused when it comes to such use cases as unethical profiling,interrogation, illegal behaviour forecasting, and “cancelling” individuals basedon behaviour. Therefore, we should avoid them at all costs [12].

6Development, Testing and Evaluation, Ethics
and Societal impact

How to design experimental setups for socially empowered systems?
One of the main questions that arose during the discussion was “How to design
experimental setups for socially empowered systems?”. One of the important
aspects of experimental design is linked to the aim and size of the experiment.
For instance, dyad interaction (a system interacting with a human) will havea different design than a large-scale analysis of social networks. Furthermore,
the timing of the experiment will also impact the design. If an experimentis supposed to last for several months, the complexity and the side effects onparticipants can be more challenging to deal with. Despite the large variety ofexperiments, they all share some kinds of ideal goals. Since SSP systems aresupposed to integrate an ecosystem, experiments should be as close as possible
to real-world conditions (situations, context, environments, population distribution). To preserve this idea of real-world closeness, the experiment should
not be too intrusive into the life of participants. This is for two reasons. The
first is that the participant should feel safe (physically and emotionally). At
the same time she/he should be aware of what the experiment is about (transparency). Another aspect of the participants is to let him/her keep control of
their engagement in the experiment (drop out mechanism). The second reason
for the low disruption of participants’ lives is to collect data as close as possible
to real situations.
The discussion led to a scheme of general practice for designing experiments. A consensus is that there are three main phases: the preparation
phase, the experiment phase and the post-experiment phase. The prepara-24

tion phase covers the question of required data and participants as well as theobjectives. In each experiment, we need to make clear what data is needed
to deduce which mental state/emotion must be tracked and what the benefits for research/society/business are. Experimenters should think about how
to get the necessary and sufficient number of sensors/data to reduce complexity/consumption/cost; this could be done in several steps. Experiments could
consider objective measures (sensors) together with subjective measures (questionnaires and feedback) to have different views on the same situation. In SSP
experiments, particular attention should be paid to ethical approval. In particular, how the system will impact the participant. A general rule is to makethe participant as aware as possible of the aim and impact and give them sufficient time to decide on their participation (informed consent). A mechanism
to stop the experiment should be made clear (drop out any time). In casethe experiment includes some specific population (e.g. “fragile” or dependent
people), a referent team might be included in the experiment (e.g. nurses, relatives, teachers). This team might include a long-term experiment supervisor
who stops/modifies the experiment in case of unforeseen situations.
Once the preparation is over, the experiment phase begins. This phase canbe split into several steps all of them needing proper monitoring (in particular,when the duration is long or when it is performed in the wild). One early step
can be composed of questionnaires to understand the consequences of the experiments. Then some steps could be devoted to prototyping the experimentwith naive participants (colleges), an in-lab experiment with the targeted population and then in the targeted environment. Some of these steps can be revised
and lead to a revisit of the design for further steps. Of course, focus groups
and domain experts could be involved when appropriate. The post-experimentphases consist of getting feedback from the participants and getting a gist of thelong-term consequences. This can be performed with questionnaires, anotherexperiment or subjective analysis (participants commenting on their data).
How can setups be tested - which requirements are needed foruser studies and how to design those? As a preparation for the study,
effective tools for labelling and study design should be developed. In particular,
training for the subjects as well as for the study organizers shall be designed.
Also, scenario-based simulation with identifying some typical ‘persona’ could be
set up and tested. These have to be scheduled in advance of the actual study.
For the selection of users, a sufficient number of users with good diversity isimportant. Some studies might need real users as well as AI-based “users” (e.g.
based on historical data from real users). In particular, such virtual users may
be employed in order to not impose ethical questions. During the training andtowards the study, good motivation of test subjects is to be ensured. For theactual test design, a reasonable number of pilot studies with a smaller user baseshall be executed before entering a large (high-quality) cohort.
In general, an exploratory study has, by definition, does not have a clear
test goal. It is potentially followed by a hypothesis-based experiment based on
the findings from the explorative study. The hypothesis-based experiment shallhave clear test goals.
How could existing metrics and methods be adapted? Another pointof discussion was about the evaluation methods that should be used in SSP experiments. The group reached a consensus on the fact that both objective
and subjective measures should be considered (either qualitative or quantita25

tive). For SSP, usual sensors are composed of wearable, video, physiological,
and speech signals from which inference or ground truth can be extracted. It
is also very common in SSP to ask the participant to perform subjective annotation (whether self or as an observer). For this, many issues arise regarding
how to get user feedback (reliability, time delay, ergonomy: speech interface
or touch screen), how to measure engagement in the experiment, and how tovalidate an annotation (conflicting annotations). Getting ground truth in SSP
experiments is difficult and some social signal measures can be used to correlate participants’ annotation such as proximity, eye contact, eye blinking, faceemotion recognition, etc.
Other measures that should be considered are related to the impact of theexperiment/system such as energy consumption (related to cost and carbon
footprint), and efficiency (how fast/tiring the experiment was). Many current
evaluation methods can be used and adapted. For the annotation by itself,agreement measures such as Kappa, Beta and Gamma can be used to find out
conflicting parts to be solved by expertise. AB tests can also be introduced.
Many other measures from psychology (empathy scale, Godspeed, Russell circumplex model, curiosity, personality trait survey) can also be ported in thefield. However, we need to make clear the quality/validity of the measures.
What ethical and societal implications can arise? We have identified
several ethical and societal implications. From a purely experimental point ofview, engaging participants in experiments must be performed with care. The
participant should not feel expendable (e.g. guinea pig effect) but part of aprocess in which the participant has a role to play and for which motivations
to participate are heard and understood. Specifically, related to the creation ofsocial AI, the experimenters may develop an overtrust towards the AI system,
which may induce bias in the results (e.g. automation bias). Other biases
can be due to the system by itself which makes it more able to interact withsome specific groups better than others leading to discrimination. Furthermore,social AI systems might change human-to-human behaviour so that the results
achieved become unreliable. Additionally, it might happen that a well-behavedsocial AI might overhype the expectations of humans. Finally, some people
may have adverse feelings towards the use of AI and change their behavior to
behave unnaturally (e.g. subjects might hide their affect and behaviour due
to “social AI over-surveillance”). These points are critical because, in Social
Signal Processing, wrong results may have severe consequences (e.g., erroneous
suicidal risk detection, false positive lie detection, etc.). Other implications canbe that the development and multiplication of SSP systems based on computers
and AI will put more pressure on the energy infrastructure.
As discussed earlier, experiments and products must be positioned ethically,legally and socially. Then how to deal with ethical, legal and societal issues?
Regarding ethical issues, the group emphasized that almost all universities andcountries have ethical committees and ethical training. Hence, ethical approval
can be sought for each series of experiments. Regarding legal aspects, of course,
experimenters should increase their awareness of the law (European, American,
Chinese, Japanese, and so on). Some universities and companies do have law
specialists who can be consulted before/during the experiment design. Social
issues are not always fully covered by legal or ethical processes. For instance,some new products might have a big impact on the cultural traits of a population(how disturbing this will impact cultural practice).
26

How to prevent the misuse of socially empowered systems? Generally, in order to proactively prevent misuse of socially empowered systems, theresearcher should refrain from project grant money in dubious cases. The researcher shall further realize that misuse can not be prevented, but only the risk
to misuse a system can be minimized. A good practice of implementation is toenforce transparency. For instance, sharing examples of misuse is important tohelp others to learn from the experience of others. Furthermore, practices should
be enforced to possibly identify misuses in advance. Such measures could be, forinstance, misuse prevention by design (e.g. context-dependent users/situations
such as AI-based adaptive learning content filter). Also, an approach could be
to set out a challenge to break the system in order to learn where possible attack
angles for the system still exist before rolling out the system. Another aspectof misuse prevention is related to the education of all involved actors, such asusers, researchers, but also law-enforcement personnel. Education should startat all levels such as e.g., primary school on the ethics of AI and build up AI
literacy for the upcoming AI-natives. Special attention is also needed for situations in which the AI knows (socially) more about a person than the person
itself (e.g., detects schizophrenia, depression, suicidal risk, criminal tendencies,
break-up, cheating partner, etc.), which may be difficult to cope with for theindividual. Specifically, education also regards preventing humans from gettingsocially “lazy” by leaving it to the AI. Finally, it is important to continue maintenance of the system after deployment. This could regard, e.g. a “guardian AI”
that recognises misuse of socially empowered AI. Specifically, AI can be used todetect possible misuse by recognizing changes in behavior and SSP signals. In
user studies, participants shall always maintain their right to revoke consent.
How to estimate aspects such as adoption, attraction, engagement
or addiction? A good strategy to approach these aspects is to have interaction
logs. Furthermore, cognitive measurements can be used as well as questionnaires
and interviews. In the case of problematic behavior or patterns, we may intend
to induce changes in regular life patterns. One problem can be the accurate
detection of the trade-off between engagement and addiction. How should
we deal with the cultural dependencies of ELSI in SSP? Ethical, legal
and social issues (ELSI): The researcher should be aware that they may havecultural bias and consider adapting their system to other cultures.

6.1

New findings

It turns out that making experiments in SSP can follow most of the steps discussed above. One suggestion for SSP would be to create a supervision agent(nurse, psycho, etc. ) to supervise the experiment for the long and short term.
This agent should be linked to the participants and have some kind of trust
relation and obligation.
How can these setups be tested - which requirements are needed
for such user studies and how to design those? For social signal processing, the preparation phase is very important. It would be good to provide moreguidance to researchers as to how such experiments are designed. In particular,
pre-studies and scenario-based simulation. Also, the impact of the experimenton the participant should be taken into account (impact of the experiment on
the emotional or mental state of the participant). The experimenters should
therefore keep in touch with the participants or set up a group space for discus27

sion and to follow-up with the participants.
The experimental research in SSP faces a lack of standardization mostly
due to the hidden nature of what emotion is. There is also a shortage of data
to analyze emotion-related behavior. So we suggest improving the experimental methodology sharing among the community by favoring open science andsharable resources (sharing measure scripts, data –when no breach of privacy–
system, code when possible). Organization of challenges and benchmarks is
encouraged. The hidden nature of emotion leads to using a mixed strategy
of measurement by using both objective and subjective observations that will
inform on different aspects of the participants’ behavior.
What ethical and societal implications can arise? During the discussion, we realized that there might be an opportunity to use AI as ‘subjects’ more
broadly and for a wider variety of study types. Specifically, the whole interplay
between human participants and AI subjects needs to be investigated, understood and taken into account in the study design. The study designer should
respect the anti-AI sentiment of some people. It could be a consideration toassign AI-free spaces.
Ethical, legal, and social awareness can be improved by inserting in the
curriculum training about these aspects. In the research community as a whole,special sessions, seminars, etc. should be more regular. One way to improve thehandling of ethical proposals would be to share them among the community sothat good practices get better circulated. Another way would be to participate
in ethics committees.
How to prevent misuse of socially empowered systems? SSP is useful
not only for application but also for the prevention of misuse. For instance, SSP
can monitor emotions and thereby detect/recognize emotional changes which
may indicate potential misuse. In this sense, SSP could be used to forecast
prospective misuse.

6.2

Identified issues

Making experiments as close to the real world is sometimes not feasible. Furthermore, experiments in SSP can require several kinds of expertise, which are hard
to find and which augment the complexity of the setting. Finally, experimental
research is not valued in some disciplines.
How can these setups be tested - which requirements are needed
for such user studies and how to design those? In general, the time forthe overall experiment (pre-experiment, experiment, post-experiment) is very
time-consuming and not well aligned with the requirements of PhD studies.
One issue relates to the creation of AI participants. As this is a novel field,
guidelines would be welcome on how such AI shall be designed and created.
Testbeds and shared labs are seldom but are necessary for SSP research.
The main issue regarding evaluation is the lack of evaluation standardization,
the shortage of datasets and the still insufficient collaboration between scientistsof different fields.
How to prevent the misuse of socially empowered systems? Despite
all that researchers/companies can do to prevent it, it must be realized thatmisuse cannot be totally prevented, but at best be only corrected.
What ethical and societal implications can arise? It appears that thepractice of employing AI as a subject could be more explored. Furthermore, the28

relationship between humans and AI changes daily. It is therefore difficult toappropriately design a study taking this into account.
The handling of ethical, legal and social issues is seen as a very timeconsuming task and most of the time people are not trained to do it. Furthermore, it is always not clear what are the boundaries of responsibilities between
research and applications. Suppose research must be aware of the whole implication the research might have. In that case, stakeholders must also be involved
since the implication and responsibility of each of these entities might change
from the design phase to the product phase.

29

List of Participants• Prof. Dr. Sozo Inoue, Kyushu Institute of Technology• Prof. Dr. Wolfgang Minker, Ulm University• Prof. Dr. Björn Schuller, Augsburg University & Imperial College London• Prof. Dr. Takuro Yonezawa, Nagoya University• Prof. Dr. Kazuya Murao, Ritsumeikan University• Prof. Dr. Shuhei Tsuchida, Kobe University• Prof. Dr. Alessandro Vinciarelli, University of Glasgow• Prof. Dr. Georgios N. Yannakakis, University of Malta & modl.ai• Prof. Dr. Susanna Pirttikangas, University of Oulu
• Prof. Dr. Stephan Sigg, Aalto University• Prof. Dr. Hiroki Watanabe, Hokkaido University• Prof. Dr. Atsushi Nakazawa, Kyoto University• Prof. Dr. Francois Portet, Universite Grenoble Alpes• Prof. Dr. Stefan Wesner, University of Cologne• Dr. Jacqueline Urakami, Kyocera• Dr. Tahera Hossain, Aoyama Gakuin University• Dr. Sebastian Zepf, Hochschule Furtwangen University• Mr. Danila Mamontov, Ulm University• Ms. Nazmun Nahid, Kyushu Institute of Technology• Mr. Haru Kaneko, Kyushu Institute of Technology• Ms. Christina Garcia, Kyushu Institute of Technology

30

References[1]

A.S. Albahri et al. “A systematic review of trustworthy and explainableartificial intelligence in healthcare”. In: Information Fusion (Aug. 2023).
issn: 15662535. doi: 10 . 1016 / J . INFFUS . 2023 . 03 . 008. url: https ://dl.acm.org/doi/10.1016/j.inffus.2023.03.008.

[2]

Nicolas Anciaux et al. “Personal Data Management Systems: The security
and functionality standpoint”. In: Information Systems 80 (Feb. 2019),
pp. 13–35. issn: 0306-4379. doi: 10.1016/J.IS.2018.09.002.

[3]

Rishi Bommasani et al. “On the Opportunities and Risks of Foundation
Models”. In: (Aug. 2021). url: https://arxiv.org/abs/2108.07258v3.

[4]

Alessandro Canossa et al. “For Honor, for Toxicity: Detecting Toxic Behavior through Gameplay”. In: Proceedings of the ACM on Human-Computer
Interaction 5 (CHIPLAY Sept. 2021). issn: 25730142. doi: 10 . 1145 /
3474680.

[5]

Rishu Chhabra, Seema Verma, and C. Rama Krishna. “A survey on driver
behavior detection techniques for intelligent transportation systems”. In:
Proceedings of the 7th International Conference Confluence 2017 on Cloud
Computing, Data Science and Engineering (June 2017), pp. 36–41. doi:
10.1109/CONFLUENCE.2017.7943120.

[6]

Fatemeh Daneshfar and Mohammad Behdad Jamshidi. “A Pattern Recognition Framework for Signal Processing in Metaverse”. In: Proceedings 2022 8th International Iranian Conference on Signal Processing and Intelligent Systems, ICSPIS 2022 (2022). doi: 10.1109/ICSPIS56952.2022.
10044059.

[7]

Monique Dittrich and Sebastian Zepf. “Exploring the validity of methods to track emotions behind the wheel”. In: Lecture Notes in Computer
Science (including subseries Lecture Notes in Artificial Intelligence andLecture Notes in Bioinformatics) 11433 LNCS (2019), pp. 115–127. issn:16113349. doi: 10 . 1007 / 978 - 3 - 030 - 17287 - 9 _ 10 / FIGURES / 6. url:
https://link.springer.com/chapter/10.1007/978- 3- 030- 172879_10.

[8]Martin Eling and Mirko Kraft. “The impact of telematics on the insurability of risks”. In: Journal of Risk Finance 21 (2 June 2020), pp. 77–109.
issn: 09657967. doi: 10.1108/JRF-07-2019-0129/FULL/PDF.

[9]

Guangyuan Gao, Shengwang Meng, and Mario V. Wüthrich. “What can
we learn from telematics car driving data: A survey”. In: Insurance: Mathematics and Economics 104 (May 2022), pp. 185–199. issn: 0167-6687.
doi: 10.1016/J.INSMATHECO.2022.02.004.

[10]

Pablo Garcı́a-Sánchez. “Georgios N. Yannakakis and Julian Togelius: Artificial Intelligence and Games: Springer, 2018, Print ISBN: 978-3-31963518-7, Online ISBN: 978-3-319-63519-4, https://doi.org/10.1007/978-3319-63519-4”. In: Genetic Programming and Evolvable Machines 20 (1Mar. 2019), pp. 143–145. issn: 15737632. doi: 10.1007/S10710- 0189337 - 0 / METRICS. url: https : / / link . springer . com / article / 10 .
1007/s10710-018-9337-0.

31

[11]

Alex Hern. Information commissioner warns firms over ‘emotional analysis’ technologies — Biometrics — The Guardian. 2022. url: https://
amp-theguardian-com.cdn.ampproject.org/c/s/amp.theguardian.
com/technology/2022/oct/25/information- commissioner- warnsfirms-over-emotional-analysis-technologies.

[12]

Javier Hernandez et al. “Guidelines for Assessing and Minimizing Risks
of Emotion Recognition Applications”. In: 2021 9th International Conference on Affective Computing and Intelligent Interaction, ACII 2021
(2021). doi: 10.1109/ACII52823.2021.9597452.

[13]

Stanford Behavior Design Lab. The Ethical Use of Persuasive Technology. url: https : / / behaviordesign . stanford . edu / ethical - use persuasive-technology.

[14]

Maria E Marketou et al. “Stroke Risk Prediction with Machine Learning
Techniques”. In: Sensors 2022, Vol. 22, Page 4670 22 (13 June 2022),
p. 4670. issn: 1424-8220. doi: 10.3390/S22134670. url: https://www.
mdpi.com/1424- 8220/22/13/4670/htm%20https://www.mdpi.com/
1424-8220/22/13/4670.

[15]

T. Minka, Ryan Cleven, and Yordan Zaykov. “TrueSkill 2: An improved
Bayesian skill rating system”. In: (2018).

[16]

Melanie Mitchell. “On Crashing the Barrier of Meaning in Artificial Intelligence”. In: AI Magazine 41 (2 June 2020), pp. 86–92. issn: 2371-9621.
doi: 10 . 1609 / AIMAG . V41I2 . 5259. url: https : / / ojs . aaai . org /aimagazine/index.php/aimagazine/article/view/5259.

[17]

Sang Min Park and Young Gab Kim. “A Metaverse: Taxonomy, Components, Applications, and Open Challenges”. In: IEEE Access 10 (2022),
pp. 4209–4251. issn: 21693536. doi: 10.1109/ACCESS.2021.3140175.

[18]

Carson Reynolds and Rosalind Picard. “Affective sensors, privacy, andethical contracts”. In: Conference on Human Factors in Computing Systems - Proceedings (2004), pp. 1103–1106. doi: 10.1145/985921.985999.
url: https://dl.acm.org/doi/10.1145/985921.985999.

[19]

Richard Shiffrin and Melanie Mitchell. “Probing the psychology of AI
models”. In: Proceedings of the National Academy of Sciences of the UnitedStates of America 120 (10 Mar. 2023), e2300963120. issn: 10916490. doi:
10.1073/pnas.2300963120. url: https://www.pnas.org/doi/abs/10.
1073/pnas.2300963120.

[20]

Judith Sligo et al. “A literature review for large-scale health information
system project planning, implementation and evaluation”. In: International Journal of Medical Informatics 97 (Jan. 2017), pp. 86–97. issn:18728243. doi: 10.1016/J.IJMEDINF.2016.09.007.

[21] Top Emotion Recognition companies — VentureRadar. 2019. url: https://www.ventureradar.com/keyword/Emotion%20Recognition.
[22] Video AI — Video Recognition — Image Recognition - Valossa AI makes
sense of the world’s video content. url: https://valossa.com/.
[23] WHO issues first global report on Artificial Intelligence (AI) in health andsix guiding principles for its design and use. url: https://www.who.int/
news/item/28-06-2021-who-issues-first-global-report-on-aiin-health-and-six-guiding-principles-for-its-design-and-use.
32

[24]

Georgios N. Yannakakis and Julian Togelius. “Experience-driven procedural content generation”. In: IEEE Transactions on Affective Computing 2(3 July 2011), pp. 147–161. issn: 19493045. doi: 10.1109/T-AFFC.2011.
6.

[25]

Nicholas Yee. Ariadne Understanding MMORPG Addiction. url: http://www.nickyee.com/hub/addiction/home.html.

[26]

Sebastian Zepf et al. “Driver Emotion Recognition for Intelligent Vehicles”. In: ACM Computing Surveys (CSUR) 53 (3 June 2020). issn:15577341. doi: 10.1145/3388790. url: https://dl.acm.org/doi/10.
1145/3388790.

[27]

Sebastian Zepf et al. “Towards empathetic car interfaces: Emotional triggers while driving”. In: Conference on Human Factors in Computing Systems - Proceedings (May 2019). doi: 10.1145/3290607.3312883. url:
https://dl.acm.org/doi/10.1145/3290607.3312883.

33