OpenRaiser/Intern-Atlas huggingface dataset:
SpeakEasy: A Conversational Intelligence Chatbot for Enhancing College Students' Communication Skills
Venue: arXiv cs.HC 2023
Year: 2023

Social interactions and conversation skills separate the successful from the rest and the confident from the shy. For college students in particular, the ability to converse can be an outlet for the stress and anxiety experienced on a daily basis along with a foundation for all-important career skills. In light of this, we designed SpeakEasy: a chatbot with some degree of intelligence that provides feedback to the user on their ability to engage in free-form conversations with the chatbot. SpeakEasy attempts to help college students improve their communication skills by engaging in a seven-minute spoken conversation with the user, analyzing the user's responses with metrics designed based on previous psychology and linguistics research, and providing feedback to the user on how they can improve their conversational ability. To simulate natural conversation, SpeakEasy converses with the user on a wide assortment of topics that two people meeting for the first time might discuss: travel, sports, and entertainment. Unlike most other chatbots with the goal of improving conversation skills, SpeakEasy actually records the user speaking, transcribes the audio into tokens, and uses macros-e.g., sequences that calculate the pace of speech, determine if the user has an over-reliance on certain words, and identifies awkward transitions-to evaluate the quality of the conversation. Based on the evaluation, SpeakEasy provides elaborate feedback on how the user can improve their conversations. In turn, SpeakEasy updates its algorithms based on a series of questions that the user responds to regarding SpeakEasy's performance.
GRLib: An Open-Source Hand Gesture Detection and Recognition Python Library
Venue: arXiv cs.CV 2023
Year: 2023

Hand gesture recognition systems provide a natural way for humans to interact with computer systems. Although various algorithms have been designed for this task, a host of external conditions, such as poor lighting or distance from the camera, make it difficult to create an algorithm that performs well across a range of environments. In this work, we present GRLib: an open-source Python library able to detect and classify static and dynamic hand gestures. Moreover, the library can be trained on existing data for improved classification robustness. The proposed solution utilizes a feed from an RGB camera. The retrieved frames are then subjected to data augmentation and passed on to MediaPipe Hands to perform hand landmark detection. The landmarks are then classified into their respective gesture class. The library supports dynamic hand gestures through trajectories and keyframe extraction. It was found that the library outperforms another publicly available HGR system - MediaPipe Solutions, on three diverse, real-world datasets. The library is available at https://github.com/mikhail-vlasenko/grlib and can be installed with pip.
Robust Outlier Detection Method Based on Local Entropy and Global Density
Venue: arXiv cs.IT 2023
Year: 2023

By now, most outlier-detection algorithms struggle to accurately detect both point anomalies and cluster anomalies simultaneously. Furthermore, a few K-nearest-neighbor-based anomaly-detection methods exhibit excellent performance on many datasets, but their sensitivity to the value of K is a critical issue that needs to be addressed. To address these challenges, we propose a novel robust anomaly detection method, called Entropy Density Ratio Outlier Detection (EDROD). This method incorporates the probability density of each sample as the global feature, and the local entropy around each sample as the local feature, to obtain a comprehensive indicator of abnormality for each sample, which is called Entropy Density Ratio (EDR) for short in this paper. By comparing several competing anomaly detection methods on both synthetic and real-world datasets, it is found that the EDROD method can detect both point anomalies and cluster anomalies simultaneously with accurate performance. In addition, it is also found that the EDROD method exhibits strong robustness to the number of selected neighboring samples, the dimension of samples in the dataset, and the size of the dataset. Therefore, the proposed EDROD method can be applied to a variety of real-world datasets to detect anomalies with accurate and robust performances.
My Lockdown Escape: Sparking Self-Empathy in the Context of the Covid-19 Pandemic
Venue: arXiv cs.HC 2023
Year: 2023

During the Covid-19 pandemic, research communities focused on collecting and understanding people's behaviours and feelings to study and tackle the pandemic indirect effects. Despite its consequences are slowly starting to fade away, such an interest is still alive. In this article, we propose a hybrid, gamified, story-driven data collection approach to spark self-empathy, hence resurfacing people's past feelings. The game is designed to include a physical board, decks of cards, and a digital application. As the player plays through the game, they customize and escape from their lockdown room by completing statements and answering a series of questions that define their story. The decoration of the lockdown room and the storytelling-driven approach are targeted at sparking people's emotions and self-empathy towards their past selves. Ultimately, the proposed approach was proven effective in sparking and collecting feelings, while a few improvements are still necessary.
A Universal Anti-Spoofing Approach for Contactless Fingerprint Biometric Systems
Venue: arXiv cs.CV 2023
Year: 2023

With the increasing integration of smartphones into our daily lives, fingerphotos are becoming a potential contactless authentication method. While it offers convenience, it is also more vulnerable to spoofing using various presentation attack instruments (PAI). The contactless fingerprint is an emerging biometric authentication but has not yet been heavily investigated for anti-spoofing. While existing anti-spoofing approaches demonstrated fair results, they have encountered challenges in terms of universality and scalability to detect any unseen/unknown spoofed samples. To address this issue, we propose a universal presentation attack detection method for contactless fingerprints, despite having limited knowledge of presentation attack samples. We generated synthetic contactless fingerprints using StyleGAN from live finger photos and integrating them to train a semi-supervised ResNet-18 model. A novel joint loss function, combining the Arcface and Center loss, is introduced with a regularization to balance between the two loss functions and minimize the variations within the live samples while enhancing the inter-class variations between the deepfake and live samples. We also conducted a comprehensive comparison of different regularizations' impact on the joint loss function for presentation attack detection (PAD) and explored the performance of a modified ResNet-18 architecture with different activation functions (i.e., leaky ReLU and RelU) in conjunction with Arcface and center loss. Finally, we evaluate the performance of the model using unseen types of spoof attacks and live data. Our proposed method achieves a Bona Fide Classification Error Rate (BPCER) of 0.12\%, an Attack Presentation Classification Error Rate (APCER) of 0.63\%, and an Average Classification Error Rate (ACER) of 0.37\%.
Robot Skill Generalization via Keypoint Integrated Soft Actor-Critic Gaussian Mixture Models
Venue: arXiv cs.RO 2023
Year: 2023

A long-standing challenge for a robotic manipulation system operating in real-world scenarios is adapting and generalizing its acquired motor skills to unseen environments. We tackle this challenge employing hybrid skill models that integrate imitation and reinforcement paradigms, to explore how the learning and adaptation of a skill, along with its core grounding in the scene through a learned keypoint, can facilitate such generalization. To that end, we develop Keypoint Integrated Soft Actor-Critic Gaussian Mixture Models (KIS-GMM) approach that learns to predict the reference of a dynamical system within the scene as a 3D keypoint, leveraging visual observations obtained by the robot's physical interactions during skill learning. Through conducting comprehensive evaluations in both simulated and real-world environments, we show that our method enables a robot to gain a significant zero-shot generalization to novel environments and to refine skills in the target environments faster than learning from scratch. Importantly, this is achieved without the need for new ground truth data. Moreover, our method effectively copes with scene displacements.
Affective and Dynamic Beam Search for Story Generation
Venue: arXiv cs.CL 2023
Year: 2023

Storytelling's captivating potential makes it a fascinating research area, with implications for entertainment, education, therapy, and cognitive studies. In this paper, we propose Affective Story Generator (AffGen) for generating interesting narratives. AffGen introduces "intriguing twists" in narratives by employing two novel techniques-Dynamic Beam Sizing and Affective Reranking. Dynamic Beam Sizing encourages less predictable, more captivating word choices using a contextual multi-arm bandit model. Affective Reranking prioritizes sentence candidates based on affect intensity. Our empirical evaluations, both automatic and human, demonstrate AffGen's superior performance over existing baselines in generating affectively charged and interesting narratives. Our ablation study and analysis provide insights into the strengths and weaknesses of AffGen.
Quantifying the Dialect Gap and its Correlates Across Languages
Venue: arXiv cs.CL 2023
Year: 2023

Historically, researchers and consumers have noticed a decrease in quality when applying NLP tools to minority variants of languages (i.e. Puerto Rican Spanish or Swiss German), but studies exploring this have been limited to a select few languages. Additionally, past studies have mainly been conducted in a monolingual context, so cross-linguistic trends have not been identified and tied to external factors. In this work, we conduct a comprehensive evaluation of the most influential, state-of-the-art large language models (LLMs) across two high-use applications, machine translation and automatic speech recognition, to assess their functionality on the regional dialects of several high- and low-resource languages. Additionally, we analyze how the regional dialect gap is correlated with economic, social, and linguistic factors. The impact of training data, including related factors like dataset size and its construction procedure, is shown to be significant but not consistent across models or languages, meaning a one-size-fits-all approach cannot be taken in solving the dialect gap. This work will lay the foundation for furthering the field of dialectal NLP by laying out evident disparities and identifying possible pathways for addressing them through mindful data collection.
AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models
Venue: arXiv cs.CR 2023
Year: 2023

Safety alignment of Large Language Models (LLMs) can be compromised with manual jailbreak attacks and (automatic) adversarial attacks. Recent studies suggest that defending against these attacks is possible: adversarial attacks generate unlimited but unreadable gibberish prompts, detectable by perplexity-based filters; manual jailbreak attacks craft readable prompts, but their limited number due to the necessity of human creativity allows for easy blocking. In this paper, we show that these solutions may be too optimistic. We introduce AutoDAN, an interpretable, gradient-based adversarial attack that merges the strengths of both attack types. Guided by the dual goals of jailbreak and readability, AutoDAN optimizes and generates tokens one by one from left to right, resulting in readable prompts that bypass perplexity filters while maintaining high attack success rates. Notably, these prompts, generated from scratch using gradients, are interpretable and diverse, with emerging strategies commonly seen in manual jailbreak attacks. They also generalize to unforeseen harmful behaviors and transfer to black-box LLMs better than their unreadable counterparts when using limited training data or a single proxy model. Furthermore, we show the versatility of AutoDAN by automatically leaking system prompts using a customized objective. Our work offers a new way to red-team LLMs and understand jailbreak mechanisms via interpretability.
DEsignBench: Exploring and Benchmarking DALL-E 3 for Imagining Visual Design
Venue: arXiv cs.CV 2023
Year: 2023

We introduce DEsignBench, a text-to-image (T2I) generation benchmark tailored for visual design scenarios. Recent T2I models like DALL-E 3 and others, have demonstrated remarkable capabilities in generating photorealistic images that align closely with textual inputs. While the allure of creating visually captivating images is undeniable, our emphasis extends beyond mere aesthetic pleasure. We aim to investigate the potential of using these powerful models in authentic design contexts. In pursuit of this goal, we develop DEsignBench, which incorporates test samples designed to assess T2I models on both "design technical capability" and "design application scenario." Each of these two dimensions is supported by a diverse set of specific design categories. We explore DALL-E 3 together with other leading T2I models on DEsignBench, resulting in a comprehensive visual gallery for side-by-side comparisons. For DEsignBench benchmarking, we perform human evaluations on generated images in DEsignBench gallery, against the criteria of image-text alignment, visual aesthetic, and design creativity. Our evaluation also considers other specialized design capabilities, including text rendering, layout composition, color harmony, 3D design, and medium style. In addition to human evaluations, we introduce the first automatic image generation evaluator powered by GPT-4V. This evaluator provides ratings that align well with human judgments, while being easily replicable and cost-efficient. A high-resolution version is available at https://github.com/design-bench/design-bench.github.io/raw/main/designbench.pdf?download=
Linear Representations of Sentiment in Large Language Models
Venue: arXiv cs.LG 2023
Year: 2023

Sentiment is a pervasive feature in natural language text, yet it is an open question how sentiment is represented within Large Language Models (LLMs). In this study, we reveal that across a range of models, sentiment is represented linearly: a single direction in activation space mostly captures the feature across a range of tasks with one extreme for positive and the other for negative. Through causal interventions, we isolate this direction and show it is causally relevant in both toy tasks and real world datasets such as Stanford Sentiment Treebank. Through this case study we model a thorough investigation of what a single direction means on a broad data distribution. We further uncover the mechanisms that involve this direction, highlighting the roles of a small subset of attention heads and neurons. Finally, we discover a phenomenon which we term the summarization motif: sentiment is not solely represented on emotionally charged words, but is additionally summarized at intermediate positions without inherent sentiment, such as punctuation and names. We show that in Stanford Sentiment Treebank zero-shot classification, 76% of above-chance classification accuracy is lost when ablating the sentiment direction, nearly half of which (36%) is due to ablating the summarized sentiment direction exclusively at comma positions.
Towards Subject Agnostic Affective Emotion Recognition
Venue: arXiv cs.LG 2023
Year: 2023

This paper focuses on affective emotion recognition, aiming to perform in the subject-agnostic paradigm based on EEG signals. However, EEG signals manifest subject instability in subject-agnostic affective Brain-computer interfaces (aBCIs), which led to the problem of distributional shift. Furthermore, this problem is alleviated by approaches such as domain generalisation and domain adaptation. Typically, methods based on domain adaptation confer comparatively better results than the domain generalisation methods but demand more computational resources given new subjects. We propose a novel framework, meta-learning based augmented domain adaptation for subject-agnostic aBCIs. Our domain adaptation approach is augmented through meta-learning, which consists of a recurrent neural network, a classifier, and a distributional shift controller based on a sum-decomposable function. Also, we present that a neural network explicating a sum-decomposable function can effectively estimate the divergence between varied domains. The network setting for augmented domain adaptation follows meta-learning and adversarial learning, where the controller promptly adapts to new domains employing the target data via a few self-adaptation steps in the test phase. Our proposed approach is shown to be effective in experiments on a public aBICs dataset and achieves similar performance to state-of-the-art domain adaptation methods while avoiding the use of additional computational resources.
Towards Possibilities & Impossibilities of AI-generated Text Detection: A Survey
Venue: arXiv cs.CL 2023
Year: 2023

Large Language Models (LLMs) have revolutionized the domain of natural language processing (NLP) with remarkable capabilities of generating human-like text responses. However, despite these advancements, several works in the existing literature have raised serious concerns about the potential misuse of LLMs such as spreading misinformation, generating fake news, plagiarism in academia, and contaminating the web. To address these concerns, a consensus among the research community is to develop algorithmic solutions to detect AI-generated text. The basic idea is that whenever we can tell if the given text is either written by a human or an AI, we can utilize this information to address the above-mentioned concerns. To that end, a plethora of detection frameworks have been proposed, highlighting the possibilities of AI-generated text detection. But in parallel to the development of detection frameworks, researchers have also concentrated on designing strategies to elude detection, i.e., focusing on the impossibilities of AI-generated text detection. This is a crucial step in order to make sure the detection frameworks are robust enough and it is not too easy to fool a detector. Despite the huge interest and the flurry of research in this domain, the community currently lacks a comprehensive analysis of recent developments. In this survey, we aim to provide a concise categorization and overview of current work encompassing both the prospects and the limitations of AI-generated text detection. To enrich the collective knowledge, we engage in an exhaustive discussion on critical and challenging open questions related to ongoing research on AI-generated text detection.
Inclusion in Virtual Reality Technology: A Scoping Review
Venue: arXiv cs.HC 2023
Year: 2023

Despite the significant growth in virtual reality applications and research, the notion of inclusion in virtual reality is not well studied. Inclusion refers to the active involvement of different groups of people in the adoption, use, design, and development of VR technology and applications. In this review, we provide a scoping analysis of existing virtual reality research literature about inclusion. We categorize the literature based on target group into ability, gender, and age, followed by those that study community-based design of VR experiences. In the latter group, we focus mainly on Indigenous Peoples as a clearer and more important example. We also briefly review the approaches to model and consider the role of users in technology adoption and design as a background for inclusion studies. We identify a series of generic barriers and research gaps and some specific ones for each group, resulting in suggested directions for future research.
Exploring the Potential of Large Language Models in Generating Code-Tracing Questions for Introductory Programming Courses
Venue: arXiv cs.CL 2023
Year: 2023

In this paper, we explore the application of large language models (LLMs) for generating code-tracing questions in introductory programming courses. We designed targeted prompts for GPT4, guiding it to generate code-tracing questions based on code snippets and descriptions. We established a set of human evaluation metrics to assess the quality of questions produced by the model compared to those created by human experts. Our analysis provides insights into the capabilities and potential of LLMs in generating diverse code-tracing questions. Additionally, we present a unique dataset of human and LLM-generated tracing questions, serving as a valuable resource for both the education and NLP research communities. This work contributes to the ongoing dialogue on the potential uses of LLMs in educational settings.
Specialist or Generalist? Instruction Tuning for Specific NLP Tasks
Venue: arXiv cs.CL 2023
Year: 2023

The potential of large language models (LLMs) to simultaneously perform a wide range of natural language processing (NLP) tasks has been the subject of extensive research. Although instruction tuning has proven to be a data-efficient method for transforming LLMs into such generalist models, their performance still lags behind specialist models trained exclusively for specific tasks. In this paper, we investigate whether incorporating broad-coverage generalist instruction tuning can contribute to building a specialist model. We hypothesize that its efficacy depends on task specificity and skill requirements. Our experiments assess four target tasks with distinct coverage levels, revealing that integrating generalist instruction tuning consistently enhances model performance when the task coverage is broad. The effect is particularly pronounced when the amount of task-specific training data is limited. Further investigation into three target tasks focusing on different capabilities demonstrates that generalist instruction tuning improves understanding and reasoning abilities. However, for tasks requiring factual knowledge, generalist data containing hallucinatory information may negatively affect the model's performance. Overall, our work provides a systematic guide for developing specialist models with general instruction tuning. Our code and other related resources can be found at https://github.com/DavidFanzz/Generalist_or_Specialist.
Burgers' pinns with implicit euler transfer learning
Venue: arXiv cs.LG 2023
Year: 2023

The Burgers equation is a well-established test case in the computational modeling of several phenomena such as fluid dynamics, gas dynamics, shock theory, cosmology, and others. In this work, we present the application of Physics-Informed Neural Networks (PINNs) with an implicit Euler transfer learning approach to solve the Burgers equation. The proposed approach consists in seeking a time-discrete solution by a sequence of Artificial Neural Networks (ANNs). At each time step, the previous ANN transfers its knowledge to the next network model, which learns the current time solution by minimizing a loss function based on the implicit Euler approximation of the Burgers equation. The approach is tested for two benchmark problems: the first with an exact solution and the other with an alternative analytical solution. In comparison to the usual PINN models, the proposed approach has the advantage of requiring smaller neural network architectures with similar accurate results and potentially decreasing computational costs.