Reviewerly/exHarmony huggingface dataset:
Confidential Execution of Deep Learning Inference at the Untrusted Edge with ARM TrustZone

This paper proposes a new confidential deep learning (DL) inference system with ARM TrustZone to provide confidentiality and integrity of DL models and data in an untrusted edge device with limited memory. Although ARM TrustZone supplies a strong, hardware-supported trusted execution environment for protecting sensitive code and data in an edge device against adversaries, resource limitations in typical edge devices have raised significant challenges for protecting on-device DL requiring large memory consumption without sacrificing the security and accuracy of the model. The proposed solution addresses this challenge without modifying the protected DL model, thereby preserving the original prediction accuracy. Comprehensive experiments using different DL architectures and datasets demonstrate that inference services for large and complex DL models can be deployed in edge devices with TrustZone with limited trusted memory, ensuring data confidentiality and preserving the original model's prediction exactness.
IdDecoder: A Face Embedding Inversion Tool and its Privacy and Security Implications on Facial Recognition Systems

Most state-of-the-art facial recognition systems (FRS:s) use face embeddings. In this paper, we present the IdDecoder framework, capable of effectively synthesizing realistic-neutralized face images from face embeddings, and two effective attacks on state-of-the-art facial recognition models using embeddings. The first attack is a black-box version of a model inversion attack that allows the attacker to reconstruct a realistic face image that is both visually and numerically (as determined by the FRS:s) recognized as the same identity as the original face used to create a given face embedding. This attack raises significant privacy concerns regarding the membership of the gallery dataset of these systems and highlights the importance of both the people designing and deploying FRS:s paying greater attention to the protection of the face embeddings than currently done. The second attack is a novel attack that performs the model inversion, so to instead create the face of an alternative identity that is visually different from the original identity but has close identity distance (ensuring that it is recognized as being of the same identity). This attack increases the attacked system's false acceptance rate and raises significant security concerns. Finally, we use IdDecoder to visualize, evaluate, and provide insights into differences between three state-of-the-art facial embedding models.
Engaging with Nature through Technology: A Scoping Review of HCI Research

Technological progress has often been measured by the extent to which it shields and protects us from the harshness of nature. At the same time, it has long been recognised that our resulting disengagement from nature negatively affects our wellbeing and impedes awareness of our vital dependence on natural environments. To understand how HCI has considered the possibilities that digital technology offers for engaging with nature, we conducted a scoping review encompassing more than 20 years of HCI research on nature engagement. We compare the orientations, motivations, and methodologies of different threads within this growing body of work. We show how HCI research has enabled varied forms of direct and indirect engagement with nature, and we develop a typology of the roles proposed for technology in this work. We highlight promising and under-utilised approaches to designing for nature engagement and discuss directions for future research.
Challenges of Moderating Social Virtual Reality

Recent years have seen a rise in social virtual reality (VR) platforms that allow people to interact in real-time through voice and gestures. The ephemeral nature of communication on these platforms can enable new forms of harmful behavior and new challenges for moderators. We performed virtual field research on three VR environments (AltspaceVR, Horizon Worlds, Rec Room). Based on observing 100 scheduled events, our analysis uncovered 13 distinct types of potentially harmful behaviors enabled by real-time voice, embodied interactions, and platform affordances. We witnessed potential harm at 45% of our observed events; only 24% of these incidents were addressed by moderators. To understand moderation practices, we conducted interviews with 11 moderators to investigate how they assess real-time interactions and how they operate within the current state of moderation tools. Our work sheds light on how moderation tools and practices must evolve to meet the new challenges of social VR.
This Watchface Fits with my Tattoos: Investigating Customisation Needs and Preferences in Personal Tracking

People engage in self-tracking with diverse data collection and visualisation needs and preferences. Customisable self-tracking tools offer the potential to support individualized preferences by letting people make changes to the aesthetics and functionality of tracker displays. In this paper, we use the customisation options offered by the displays of commercial fitness smartwatches as a lens to investigate when, why and how 386 self-trackers engage in customisations in their daily lives. We find that people largely customise their trackers’ display frequently, multiple times a day, or not at all, with frequent customisations reflecting situational data, aesthetic and personal meaning needs. We discuss implications for the design of tracking tools aiming to support customisation and discuss the utility of customisations towards goal scaffolding and maintaining interest in tracking.
Sharing Earthquake Narratives: Making Space for Others in our Autobiographical Design Process

As interaction designers are venturing to design for others based on autobiographical experiences, it becomes particularly relevant to critically distinguish the designer's voice from others' experiences. However, few reports go into detail about how self and others mutually shape the design process and how to incorporate external evaluation into these designs. We describe a one-year process involving the design and evaluation of a prototype combining haptics and storytelling, aiming to materialise and share somatic memories of earthquakes experienced by a designer and her partner. We contribute with three strategies for bringing others into our autobiographical processes, avoiding the dilution of first-person voices while critically addressing design flaws that might hinder the representation of our stories.
“I normally wouldn’t talk with strangers”: Introducing a Socio-Spatial Interface for Fostering Togetherness Between Strangers

Interacting with strangers can be beneficial but also challenging. Fortunately, these challenges can lead to design opportunities. In this paper, we present the design and evaluation of a socio-spatial interface, SocialStools, that leverages the human propensity for embodied interaction to foster togetherness between strangers. SocialStools is an installation of three responsive stools on caster wheels that generate sound and imagery in the near environment as three strangers sit on them, move them, and rotate them relative to each other. In our study with 12 groups of three strangers, we found a sense of togetherness emerged through interaction, evidenced by different patterns of socio-spatial movements, verbal communication, non-verbal behavior, and interview responses. We present our findings, articulate reasons for the cultivation of togetherness, consider the unique social affordances of our spatial interface in shifting attention during interpersonal communication, and provide design implications. This research contributes insights toward designing cyber-physical interfaces that foster interaction and togetherness among strangers at a time when cultivating togetherness is especially critical.
Transparency, Fairness, and Coping: How Players Experience Moderation in Multiplayer Online Games

Multiplayer online games seek to address toxic behaviors such as trolling and griefing through behavior moderation, where penalties such as chat restriction or account suspension are issued against toxic players in the hope that punishments create a teachable moment for punished players to reflect and improve future behavior. While punishments impact player experience (PX) in profound ways, little is known regarding how players experience behavior moderation. In this study, we conducted a survey of 291 players to understand their experiences with punishments in online multiplayer games. Through several statistical analyses, we found that moderation explanation plays a critical role in improving players’ perceived transparency and fairness of moderation; and these perceptions significantly affect what players do after punishments. We discuss moderation experience as an important facet of PX, bridge the game and moderation literature, and provide design implications for behavior moderation in multiplayer online games.
Generating Haptic Motion Effects for Multiple Articulated Bodies for Improved 4D Experiences: A Camera Space Approach

Motion effects are indispensable for improving 4D experiences in highly interactive applications, such as amusement parks, 4D theaters, and virtual reality games. Their recent emergence calls for effective algorithms generating motion effects synchronized with audiovisual content. This paper presents an automatic algorithm for synthesizing the object-based motion effects that express the movements of multiple articulated bodies inclusively when the objects' motion trajectories are available in the 3D camera space. By taking the visual velocities and sizes of all object parts, our method computes a motion proxy that represents the objects' movements by one point and converts the motion proxy to a motion command through a motion cueing algorithm. The motion proxy is determined by linearly combining the velocities, and its best combination was selected from several candidates by user studies. The results of user studies indicate that our algorithm can produce compelling object-based motion effects that enhance the multisensory experience.
Playing with Power Tools: Design Toolkits and the Framing of Equity

Design toolkits that aim to to promote equity offer designers simplified approaches to creating more equitable technology. However, it is important to understand how equity is conceptualized in practice. As a curated collection of methods, toolkits signal how equity is imagined in design. In this paper, we perform a qualitative analysis of 17 design toolkits related to equity. We explore alternative design approaches that address inequity in design. We evaluate whether equity toolkits align with calls for changes to design practice, as well as Nancy Fraser's dimensions of justice. Finally, we find that design toolkits focus on the 'digital divide' rather than redistributing world-building power, and thus continue to keep design power with professional designers. We also find that 'design thinking' continues to influence design toolkits. Furthermore, the simplicity of toolkits does not engage with the complexities that shape equity in practice. We conclude with suggestions to help researchers and designers rethink design toolkits.
Libraries of Things: Understanding the Challenges of Sharing Tangible Collections and the Opportunities for HCI

"Libraries of Things" are tangible collections of borrowable objects. There are many benefits to Libraries of Things such as making objects and skill-building accessible, reducing waste through the sharing of items, and saving costs associated with purchasing rarely-used items. We introduce the first HCI study of Library of Things by interviewing 23 librarians who run a variety of collections such as handheld tools, gear, and musical instruments – within public institutions and more grass-roots efforts in the private sector. In our findings, we discuss the challenges these collections experience in changing behavioural patterns from buying to borrowing, helping individuals 'try new things', iterating to find sharable items, training staff, and manual intervention throughout the borrowing cycle. We present 5 opportunities for HCI research to support interactive skill-sharing, self-borrowing, maintenance recognition and cataloguing 'things', organizing non-uniform inventories, and creating public-awareness. Further in-the-wild studies should also consider the tensions between the values of these organizations and low-cost convenient usage.
HCI Research on Agriculture: Competing Sociotechnical Imaginaries, Definitions, and Opportunities

Agriculture is foundational for food security on our planet. Considering climate change and other pressures on food production, HCI scholars have increasingly begun to examine how the field should approach agricultural innovation. We conducted a literature review of HCI research through the lens of competing future visions for good food systems : a "conventional" vision of profit-oriented production, and an "alternative" which prioritizes sustainability and community-led practices. Leveraging the concept of sociotechnical imaginaries, we provide an empirical analysis of how HCI and adjacent applied computing projects align with these competing visions for agriculture. This review reveals, amongst other findings, a prioritization of the perspectives of the Global North and a need for more careful attention to the constraints and aspirations of subsistence farmers. Finally, we note the limits of the conventional-alternative binary that shapes much of contemporary HCI research focused on agriculture and offer opportunities for transcending this framing.
Redlining Maps and Terrains of Sustainability:

We ask how historic redlining, a US government run, racially discriminatory practice of assessing and mapping property values for federally subsidized home loan eligibility in the 1930s, is tied to current issues of sustainability. We frame redlining as a historic data practice, tied to ongoing exposure to environmental harms and difficulty building generational wealth in African American communities in Indianapolis. To address this, we made maps to ground interdisciplinary discourse between the authors: two who research sustainable human computer interaction (SHCI) and one who researches sustainable food systems, including issues of food security. Our maps, which combine historical redlining maps and contemporary sustainability issues facing Indianapolis, helped us explore the ongoing impacts of redlining across our disciplines. We develop the term ‘sustainability’ for HCI across racial, socioeconomic, and environmental tensions and reflect on how SHCI's emerging posthuman emphasis on human/non-human relations are associated with human/human challenges like redlining.
InStitches: Augmenting Sewing Patterns with Personalized Material-Efficient Practice

There is a rapidly growing group of people learning to sew online. Without hands-on instruction, these learners are often left to discover the challenges and pitfalls of sewing through trial and error, which can be a frustrating and wasteful process. We present InStitches, a software tool that augments existing sewing patterns with targeted practice tasks to guide users through the skills needed to complete their chosen project. InStitches analyzes the difficulty of sewing instructions relative to a user's reported expertise in order to determine where practice will be helpful and then solves for a new pattern layout that incorporates additional practice steps while optimizing for efficient use of available materials. Our user evaluation indicates that InStitches can successfully identify challenging sewing tasks and augment existing sewing patterns with practice tasks that users find helpful, showing promise as a tool for helping those new to the craft.
ProObjAR: Prototyping Spatially-aware Interactions of Smart Objects with AR-HMD

The rapid advances in technologies have brought new interaction paradigms of smart objects (e.g., digital devices) beyond digital device screens. By utilizing spatial properties, configurations, and movements of smart objects, designing spatial interaction, which is one of the emerging interaction paradigms, efficiently promotes engagement with digital content and physical facility. However, as an important phase of design, prototyping such interactions still remains challenging, since there is no ad-hoc approach for this emerging paradigm. Designers usually rely on methods that require fixed hardware setup and advanced coding skills to script and validate early-stage concepts. These requirements restrict the design process to a limited group of users in indoor scenes. To facilitate the prototyping to general usages, we aim to figure out the design difficulties and underlying needs of current design processes for spatially-aware object interactions by empirical studies. Besides, we explore the design space of the spatial interaction for smart objects and discuss the design space in an input-output spatial interaction model. Based on these findings, we present ProObjAR, an all-in-one novel prototyping system with an Augmented Reality Head Mounted Display (AR-HMD). Our system allows designers to easily obtain the spatial data of smart objects being prototyped, specify spatially-aware interactive behaviors from an input-output event triggering workflow, and test the prototyping results in situ. From the user study, we find that ProObjAR simplifies the design procedure and increases design efficiency to a large extent and thus advancing the development of spatially-aware applications in smart ecosystems.
A Need-Finding Study with Users of Geospatial Data

Geospatial data is playing an increasingly critical role in the work of Earth and climate scientists, social scientists, and data journalists exploring spatiotemporal change in our environment and societies. However, existing software and programming tools for geospatial analysis and visualization are challenging to learn and difficult to use. The aim of this work is to identify the unmet computing needs of the diverse and expanding community of geospatial data users. We conducted a contextual inquiry study (n = 25) with domain experts using geospatial data in their current work. Through a thematic analysis, we found that participants struggled to (1) find and transform geospatial data to satisfy spatiotemporal constraints, (2) understand the behavior of geospatial operators, (3) track geospatial data provenance, and (4) explore the cartographic design space. These findings suggest design opportunities for developers and designers of geospatial analysis and visualization systems.
Boards Hit Back: Reflecting on Martial Arts Practices Through Soma Design

There is an increasing interest in the HCI community in designing for bodily practices. We report on a soma design process for martial arts and the resulting artifact – an interactive wooden dummy. Through a detailed account of the design process, we show how it enriched and revamped the bodily practice, but also how it changed the martial arts expert in the design team. Based on a phenomenological account of his experiences, we argue that the estrangement methods in soma design may allow practitioners engaging as soma designers, to cultivate and create new artistic habits fused with thought and feeling, changing themselves and their practice in directions they seek.
Lyric App Framework: A Web-based Framework for Developing Interactive Lyric-driven Musical Applications

Lyric videos have become a popular medium to convey lyrical content to listeners, but they present the same content whenever they are played and cannot adapt to listeners' preferences. Lyric apps, as we name them, are a new form of lyric-driven visual art that can render different lyrical content depending on user interaction and address the limitations of static media. To open up this novel design space for programmers and musicians, we present Lyric App Framework, a web-based framework for building interactive graphical applications that play musical pieces and show lyrics synchronized with playback. We designed the framework to provide a streamlined development experience for building production-ready lyric apps with creative coding libraries of choice. We held programming contests twice and collected 52 examples of lyric apps, enabling us to reveal eight representative categories, confirm the framework's effectiveness, and report lessons learned.
Embodying Physics-Aware Avatars in Virtual Reality

Embodiment toward an avatar in virtual reality (VR) is generally stronger when there is a high degree of alignment between the user's and self-avatar's motion. However, one-to-one mapping between the two is not always ideal when user interacts with the virtual environment. On these occasions, the user input often leads to unnatural behavior without physical realism (e.g., objects penetrating virtual body, body unmoved by hitting stimuli). We investigate how adding physics correction to self-avatar motion impacts embodiment. Physics-aware self-avatar preserves the physical meaning of the movement but introduces discrepancies between the user's and self-avatar's motion, whose contingency is a determining factor for embodiment. To understand its impact, we conducted an in-lab study (n = 20) where participants interacted with obstacles on their upper bodies in VR with and without physics correction. Our results showed that, rather than compromising embodiment level, physics-responsive self-avatar improved embodiment compared to no-physics condition in both active and passive interactions.
Tracing and Visualizing Human-ML/AI Collaborative Processes through Artifacts of Data Work

Automated Machine Learning (AutoML) technology can lower barriers in data work yet still requires human intervention to be functional. However, the complex and collaborative process resulting from humans and machines trading off work makes it difficult to trace what was done, by whom (or what), and when. In this research, we construct a taxonomy of data work artifacts that captures AutoML and human processes. We present a rigorous methodology for its creation and discuss its transferability to the visual design process. We operationalize the taxonomy through the development of AutoML Trace a visual interactive sketch showing both the context and temporality of human-ML/AI collaboration in data work. Finally, we demonstrate the utility of our approach via a usage scenario with an enterprise software development team. Collectively, our research process and findings explore challenges and fruitful avenues for developing data visualization tools that interrogate the sociotechnical relationships in automated data work.
DreamVR: Curating an Interactive Exhibition in Social VR Through an Autobiographical Design Study

Virtual exhibitions have long been regarded as an extension of information delivery for physical exhibitions. However, what virtual exhibitions can offer audiences as a novel experience independently from physical exhibitions has been largely unexplored. In this study, we aim to understand the promises and challenges of experiencing and curating exhibitions in VR by interviewing nine expert curators. Drawing from expert insights, we summarized a set of design guidelines to inform what we can learn and adapt from physical exhibitions when curating in VR. Then, using an autobiographical design approach, we curated an interactive exhibition in VRChat to explore novel interaction techniques. We also hosted an open tour guide in the user study to validate our design guidelines with thirty participants. Results show that our approach of curating an exhibition in VRChat provided the participants with engaging and novel experiences interacting with the exhibits and other audiences.