huashen218/convxai-cia-dataset huggingface dataset:
Title: Nuanced Perspectives Toward Disability Simulations from Digital Designers, Blind, Low Vision, and Color Blind People
Authors: family: Tigwell
given: Garreth W.
Venue: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
Keywords: Accessibility, digital design, disability simulations.

Abstract: Designers of digital content have access to various resources that they use to help them meet disabled people’s accessibility needs. Disability simulations are one resource, but often criticized for failing to guide digital designers appropriately, and it is unclear if digital designers are aware of the issues surrounding disability simulations. I surveyed 92 digital designers to understand their perspectives toward disability simulations (both perceived advantages and disadvantages). I then shared work process challenges faced by digital designers and their reasons for using disability simulations with 17 people with vision impairments to facilitate a discussion on this topic. The interviewees discussed ideas that suggest many paths can be explored to connect digital designers and disabled people, in general, to reduce reliance on simulations, and a change is needed within workplace processes, culture, and staffing to further support positive change. There are research opportunities to investigate establishing avenues for connecting digital designers and disabled people in a way that is beneficial to both groups.
Title: “I...Got my Nose-Print. But it Wasn’t Accurate”: How People with Upper Extremity Impairment Authenticate on their Personal Computing Devices
Authors: family: Lewis
given: Brittany; family: Venkatasubramanian
given: Krishna
Venue: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
Keywords: upper extremity impairment, personal computing devices, authentication

Abstract: Authentication has become increasingly ubiquitous for controlling access to personal computing devices (e.g., laptops, tablets, and smartphones). In this paper, we aim to understand the authentication process used by people with upper extremity impairment (UEI). A person with UEI lacks range of motion, strength, endurance, speed, and/or accuracy associated with arms, hands, or fingers. To this end, we conducted semi-structured interviews with eight (8) adults with UEI about their use of authentication for their personal computing devices. We found that our participants primarily use passwords and PINs as a verification credential during authentication. We found the process of authentication to have several accessibility issues for our participants. Consequently, our participants implemented a variety of workarounds that prioritized usability over security throughout the authentication process. Based on these findings, we present six broad subareas of research that should be explored in order to create more accessible authentication for people with UEI.
Title: Digital Design Marginalization: New Perspectives on Designing Inclusive Interfaces
Authors: family: Sin
given: Jaisie; family: L. Franz
given: Rachel; family: Munteanu
given: Cosmin; family: Barbosa Neves
given: Barbara
Venue: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
Keywords: Inclusive Design, Digital Design Marginalization, Social Exclusion, Digital Inclusion, Marginalizing Design, Social Marginalization, Digital Inequality, Digital Divide, Older Adults, Digital Exclusion

Abstract: We conceptualize Digital Design Marginalization (DDM) as the process in which a digital interface design excludes certain users and contributes to marginalization in other areas of their lives. Due to non-inclusive designs, many underrepresented users face barriers in accessing essential services that are moving increasingly, sometimes exclusively, online – services such as personal finance, healthcare, social connectivity, and shopping. This can further perpetuate the “digital divide,” a technology-based form of social inequality that has offline consequences. We introduce the term Marginalizing Design to describe designs that contribute to DDM. In this paper, we focus on the impact of Marginalizing Design on older adults through examples from our research and discussions of services that may have marginalizing designs for older adults. Our aim is to provide a conceptual lens for designers, service providers, and policy makers through which they can use to purposely lessen or avoid digitally marginalizing groups of users.
Title: Investigating the Accessibility of Crowdwork Tasks on Mechanical Turk
Authors: family: Uzor
given: Stephen; family: Jacques
given: Jason T.; family: Dudley
given: John J; family: Kristensson
given: Per Ola
Venue: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
Keywords: AMT, accessibility, disability, crowdwork, MTurk, crowdsourcing

Abstract: Crowdwork can enable invaluable opportunities for people with disabilities, not least the work flexibility and the ability to work from home, especially during the current Covid-19 pandemic. This paper investigates how engagement in crowdwork tasks is affected by individual disabilities and the resulting implications for HCI. We first surveyed 1,000 Amazon Mechanical Turk (AMT) workers to identify demographics of crowdworkers who identify as having various disabilities within the AMT ecosystem—including vision, hearing, cognition/mental, mobility, reading and motor impairments. Through a second focused survey and follow-up interviews, we provide insights into how respondents cope with crowdwork tasks. We found that standard task factors, such as task completion time and presentation, often do not account for the needs of users with disabilities, resulting in anxiety and a feeling of depression on occasion. We discuss how to alleviate barriers to enable effective interaction for crowdworkers with disabilities.
Title: Explainable Automatic Evaluation of the Trail Making Test for Dementia Screening
Authors: family: Prange
given: Alexander; family: Barz
given: Michael; family: Heimann-Steinert
given: Anika; family: Sonntag
given: Daniel
Venue: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
Keywords: digital pen, Trail Making Test, cognitive assessment, explainable, automatic scoring, multimodal

Abstract: The Trail Making Test (TMT) is a frequently used neuropsychological test for assessing cognitive performance. The subject connects a sequence of numbered nodes by using a pen on normal paper. We present an automatic cognitive assessment tool that analyzes samples of the TMT which we record using a digital pen. This enables us to analyze digital pen features that are difficult or impossible to evaluate manually. Our system automatically measures several pen features, including the completion time which is the main performance indicator used by clinicians to score the TMT in practice. In addition, our system provides a structured report of the analysis of the test, for example indicating missed or erroneously connected nodes, thereby offering more objective, transparent and explainable results to the clinician. We evaluate our system with 40 elderly subjects from a geriatrics daycare clinic of a large hospital.
Title: Shing: A Conversational Agent to Alert Customers of Suspected Online-payment Fraud with Empathetical Communication Skills
Authors: family: Guo
given: Jingya; family: Guo
given: Jiajing; family: Yang
given: Changyuan; family: Wu
given: Yanjing; family: Sun
given: Lingyun
Venue: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
Keywords: Trust, Digital finance, Privacy, Customer service, Conversational Agent, Chatbot

Abstract: Alerting customers on suspected online-payment fraud and persuade them to terminate transactions is increasingly requested with the rapid growth of digital finance worldwide. We explored the feasibility of using a conversational agent (CA) to fulfill this request. Shing, a voice-based CA, proactively initializes and repairs the conversation with empathetical communication skills in order to alert customers when a suspected online-payment fraud is detected, collects important information for fraud scrutiny and persuades customers to terminate the transaction once the fraud is confirmed. We evaluated our system by comparing it with a rule-based CA with regards to customer response and perceptions in a real-world context where our systems took 144,795 phone calls in total in which 83,019 (57.3%) natural breakdowns happened. Results showed that more customers stopped risky transactions after conversing with Shing. They seemed more willing to converse with Shing for more dialogue turns and provide transaction details. Our work presents practical implications for the design of proactive CA.
Title: Towards Mutual Theory of Mind in Human-AI Interaction: How Language Reflects What Students Perceive About a Virtual Teaching Assistant
Authors: family: Wang
given: Qiaosi; family: Saha
given: Koustuv; family: Gregori
given: Eric; family: Joyner
given: David; family: Goel
given: Ashok
Venue: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
Keywords: online education, human-AI interaction, language analysis, theory of mind, online community, conversational agent

Abstract: Building conversational agents that can conduct natural and prolonged conversations has been a major technical and design challenge, especially for community-facing conversational agents. We posit Mutual Theory of Mind as a theoretical framework to design for natural long-term human-AI interactions. From this perspective, we explore a community’s perception of a question-answering conversational agent through self-reported surveys and computational linguistic approach in the context of online education. We first examine long-term temporal changes in students’ perception of Jill Watson (JW), a virtual teaching assistant deployed in an online class discussion forum. We then explore the feasibility of inferring students’ perceptions of JW through linguistic features extracted from student-JW dialogues. We find that students’ perception of JW’s anthropomorphism and intelligence changed significantly over time. Regression analyses reveal that linguistic verbosity, readability, sentiment, diversity, and adaptability reflect student perception of JW. We discuss implications for building adaptive community-facing conversational agents as long-term companions and designing towards Mutual Theory of Mind in human-AI interaction.
Title: Planning for Natural Language Failures with the AI Playbook
Authors: family: Hong
given: Matthew K.; family: Fourney
given: Adam; family: DeBellis
given: Derek; family: Amershi
given: Saleema
Venue: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
Keywords: prototyping, natural language technologies, Human-AI interaction, AI failures

Abstract: Prototyping AI user experiences is challenging due in part to probabilistic AI models making it difficult to anticipate, test, and mitigate AI failures before deployment. In this work, we set out to support practitioners with early AI prototyping, with a focus on natural language (NL)-based technologies. Our interviews with 12 NL practitioners from a large technology company revealed that, in addition to challenges prototyping AI, prototyping was often not happening at all or focused only on idealized scenarios due to a lack of tools and tight timelines. These findings informed our design of the AI Playbook, an interactive and low-cost tool we developed to encourage proactive and systematic consideration of AI errors before deployment. Our evaluation of the AI Playbook demonstrates its potential to 1) encourage product teams to prioritize both ideal and failure scenarios, 2) standardize the articulation of AI failures from a user experience perspective, and 3) act as a boundary object between user experience designers, data scientists, and engineers.
Title: Finding the Needle in a Haystack: On the Automatic Identification of Accessibility User Reviews
Authors: family: AlOmar
given: Eman Abdullah; family: Aljedaani
given: Wajdi; family: Tamjeed
given: Murtaza; family: Mkaouer
given: Mohamed Wiem; family: El-Glaly
given: Yasmine N.
Venue: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
Keywords: machine learning., user review, Mobile application, accessibility

Abstract: In recent years, mobile accessibility has become an important trend with the goal of allowing all users the possibility of using any app without many limitations. User reviews include insights that are useful for app evolution. However, with the increase in the amount of received reviews, manually analyzing them is tedious and time-consuming, especially when searching for accessibility reviews. The goal of this paper is to support the automated identification of accessibility in user reviews, to help technology professionals in prioritizing their handling, and thus, creating more inclusive apps. Particularly, we design a model that takes as input accessibility user reviews, learns their keyword-based features, in order to make a binary decision, for a given review, on whether it is about accessibility or not. The model is evaluated using a total of 5,326 mobile app reviews. The findings show that (1) our model can accurately identify accessibility reviews, outperforming two baselines, namely keyword-based detector and a random classifier; (2) our model achieves an accuracy of 85% with relatively small training dataset; however, the accuracy improves as we increase the size of the training dataset.
Title: The Disagreement Deconvolution: Bringing Machine Learning Performance Metrics In Line With Reality
Authors: family: Gordon
given: Mitchell L.; family: Zhou
given: Kaitlyn; family: Patel
given: Kayur; family: Hashimoto
given: Tatsunori; family: Bernstein
given: Michael S.
Venue: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

Abstract: Machine learning classifiers for human-facing tasks such as comment toxicity and misinformation often score highly on metrics such as ROC AUC but are received poorly in practice. Why this gap? Today, metrics such as ROC AUC, precision, and recall are used to measure technical performance; however, human-computer interaction observes that evaluation of human-facing systems should account for people’s reactions to the system. In this paper, we introduce a transformation that more closely aligns machine learning classification metrics with the values and methods of user-facing performance measures. The disagreement deconvolution takes in any multi-annotator (e.g., crowdsourced) dataset, disentangles stable opinions from noise by estimating intra-annotator consistency, and compares each test set prediction to the individual stable opinions from each annotator. Applying the disagreement deconvolution to existing social computing datasets, we find that current metrics dramatically overstate the performance of many human-facing machine learning tasks: for example, performance on a comment toxicity task is corrected from .95 to .73 ROC AUC.
Title: Designing Effective Interview Chatbots: Automatic Chatbot Profiling and Design Suggestion Generation for Chatbot Debugging
Authors: family: Han
given: Xu; family: Zhou
given: Michelle; family: Turner
given: Matthew J.; family: Yeh
given: Tom
Venue: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
Keywords: Chatbot Design Suggestion, Chatbot Debugging, Interview Chatbot, Conversational AI Agents, Chatbot Evaluation Framework, Automatic Chatbot Profiling, Automatic Chatbot Evaluation

Abstract: Recent studies show the effectiveness of interview chatbots for information elicitation. However, designing an effective interview chatbot is non-trivial. Few tools exist to help designers design, evaluate, and improve an interview chatbot iteratively. Based on a formative study and literature reviews, we propose a computational framework for quantifying the performance of interview chatbots. Incorporating the framework, we have developed iChatProfile, an assistive chatbot design tool that can automatically generate a profile of an interview chatbot with quantified performance metrics and offer design suggestions for improving the chatbot based on such metrics. To validate the effectiveness of iChatProfile, we designed and conducted a between-subject study that compared the performance of 10 interview chatbots designed with or without using iChatProfile. Based on the live chats between the 10 chatbots and 1349 users, our results show that iChatProfile helped the designers build significantly more effective interview chatbots, improving both interview quality and user experience.
Title: Soliciting Stakeholders’ Fairness Notions in Child Maltreatment Predictive Systems
Authors: family: Cheng
given: Hao-Fei; family: Stapleton
given: Logan; family: Wang
given: Ruiqi; family: Bullock
given: Paige; family: Chouldechova
given: Alexandra; family: Wu
given: Zhiwei Steven Steven; family: Zhu
given: Haiyi
Venue: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
Keywords: child welfare, algorithmic fairness, machine learning, human-centered AI, algorithm-assisted decision-making

Abstract: Recent work in fair machine learning has proposed dozens of technical definitions of algorithmic fairness and methods for enforcing these definitions. However, we still lack an understanding of how to develop machine learning systems with fairness criteria that reflect relevant stakeholders’ nuanced viewpoints in real-world contexts. To address this gap, we propose a framework for eliciting stakeholders’ subjective fairness notions. Combining a user interface that allows stakeholders to examine the data and the algorithm’s predictions with an interview protocol to probe stakeholders’ thoughts while they are interacting with the interface, we can identify stakeholders’ fairness beliefs and principles. We conduct a user study to evaluate our framework in the setting of a child maltreatment predictive system. Our evaluations show that the framework allows stakeholders to comprehensively convey their fairness viewpoints. We also discuss how our results can inform the design of predictive systems.
Title: Crowdsourcing More Effective Initializations for Single-Target Trackers Through Automatic Re-querying
Authors: family: Lemmer
given: Stephan J.; family: Song
given: Jean Y.; family: Corso
given: Jason J.
Venue: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
Keywords: crowd-AI collaboration, smart replacement, seed rejection, crowdsourcing, single-target video object tracking

Abstract: In single-target video object tracking, an initial bounding box is drawn around a target object and propagated through a video. When this bounding box is provided by a careful human expert, it is expected to yield strong overall tracking performance that can be mimicked at scale by novice crowd workers with the help of advanced quality control methods. However, we show through an investigation of 900 crowdsourced initializations that such quality control strategies are inadequate for this task in two major ways: first, the high level of redundancy in these methods (e.g., averaging multiple responses to reduce error) is unnecessary, as 23% of crowdsourced initializations perform just as well as the gold-standard initialization. Second, even nearly perfect initializations can lead to degraded long-term performance due to the complexity of object tracking. Considering these findings, we evaluate novel approaches for automatically selecting bounding boxes to re-query, and introduce Smart Replacement, an efficient method that decides whether to use the crowdsourced replacement initialization.
Title: A Human-AI Collaborative Approach for Clinical Decision Making on Rehabilitation Assessment
Authors: family: Lee
given: Min Hun; family: Siewiorek
given: Daniel P.; family: Smailagic
given: Asim; family: Bernardino
given: Alexandre; family: Bermúdez i Badia
given: Sergi
Venue: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
Keywords: Stroke Rehabilitation Assessment, Personalization, Decision Support Systems, Human-AI Interaction/Collaboration, Explainable and Interactive Machine Learning

Abstract: Advances in artificial intelligence (AI) have made it increasingly applicable to supplement expert’s decision-making in the form of a decision support system on various tasks. For instance, an AI-based system can provide therapists quantitative analysis on patient’s status to improve practices of rehabilitation assessment. However, there is limited knowledge on the potential of these systems. In this paper, we present the development and evaluation of an interactive AI-based system that supports collaborative decision making with therapists for rehabilitation assessment. This system automatically identifies salient features of assessment to generate patient-specific analysis for therapists, and tunes with their feedback. In two evaluations with therapists, we found that our system supports therapists significantly higher agreement on assessment (0.71 average F1-score) than a traditional system without analysis (0.66 average F1-score, p < 0.05). After tuning with therapist’s feedback, our system significantly improves its performance from 0.8377 to 0.9116 average F1-scores (p < 0.01). This work discusses the potential of a human-AI collaborative system to support more accurate decision making while learning from each other’s strengths.
Title: Directed Diversity: Leveraging Language Embedding Distances for Collective Creativity in Crowd Ideation
Authors: family: Rhys Cox
given: Samuel; family: Wang
given: Yunlong; family: Abdul
given: Ashraf; family: von der Weth
given: Christian; family: Y. Lim
given: Brian
Venue: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
Keywords: Diversity, Creativity Support Tool, Ideation, Motivational messaging, Crowdsourcing, Collective Creativity, Collective Intelligence

Abstract: Crowdsourcing can collect many diverse ideas by prompting ideators individually, but this can generate redundant ideas. Prior methods reduce redundancy by presenting peers’ ideas or peer-proposed prompts, but these require much human coordination. We introduce Directed Diversity, an automatic prompt selection approach that leverages language model embedding distances to maximize diversity. Ideators can be directed towards diverse prompts and away from prior ideas, thus improving their collective creativity. Since there are diverse metrics of diversity, we present a Diversity Prompting Evaluation Framework consolidating metrics from several research disciplines to analyze along the ideation chain — prompt selection, prompt creativity, prompt-ideation mediation, and ideation creativity. Using this framework, we evaluated Directed Diversity in a series of a simulation study and four user studies for the use case of crowdsourcing motivational messages to encourage physical activity. We show that automated diverse prompting can variously improve collective creativity across many nuanced metrics of diversity.