Reviewerly/exHarmony huggingface dataset:
Audiovisual Inputs for Learning Robust, Real-time Facial Animation with Lip Sync

We present an approach for generating facial animation that combines video and audio input data in real time for low-end devices through deep learning. Our method produces control signals from audiovisual inputs separately, and mixes them to animate a character rig. The architecture relies on two specialized networks that are trained on a combination of synthetic and real world data and are highly engineered to be efficient in order to support quality avatar faces even on low-end devices. In addition, the system supports several levels of detail that degrade gracefully for additional scaling and efficiency. We showcase how user testing has been employed to improve performance and a comparison with state of the art.
Multimodal Video Captioning using Object-Auditory Information Fusion with Transformers

Video captioning aims to generate natural language sentences of an input video. Generating coherent natural language sentences is a challenging task due to the complex nature of video content such as object and scene understanding, extraction of object- and event-specific auditory information, and acquisition of relationships among objects. In this study, we address the problem of efficient modeling of object interactions in scenes, as they include crucial information regarding the events in the visual scene. To this end, we propose to use object features along with auditory information to better model the audio-visual scene appearing within the video. Specifically, we extract Faster R-CNN as the object features and VGGish as the auditory features and design a transformer encoder-decoder architecture in the multimodal setup. Experiments on MSR-VTT show encouraging results and object features better model the object interactions along with the auditory information in comparison to the ResNet features.
Effect of Avatar Clothing and User Personality on Group Dynamics in Virtual Reality

Virtual reality (VR) has emerged as a promising technology for remote teamwork; it allows individuals to collaborate in real-time via virtual avatars that convey body language and other nonverbal cues not available via traditional communication mechanisms. However, there is not much known about how the characteristics of these avatars affect the social interactions taking place in these environments. To bridge this gap, we investigate how avatar clothing influences group dynamics during team collaboration in VR. Particularly, we observe the difference between team dynamics when groups are dressed formally and casually. We asked groups of participants to report on their personality, complete two decision-making tasks while wearing different clothing, and report on their perceptions of work group inclusion and entitativity afterwards. We found that clothing had no influence on inclusion and entitativity, but personality did.
Real-time Computational Cinematographic Editing for Broadcasting of Volumetric-captured events: an Application to Ultimate Fighting

The capacity to capture and broadcast sports events with close to real-time volumetric reconstruction techniques opens exciting perspectives in how audiences can consume and interact with these contents. In this work, we propose the design of an real-time cinematography system that is capable of generating qualitative framing and editing of volumetric-captured content, by mimicking real broadcast footage. To illustrate our approach, we focus on the specific problem of cinematography for ring-based events such as Ultimate Fighting Championships (UFC). We start by extracting statistical features from hours of real footage to understand the specific framing and cutting behaviors of real broadcast directors. We then exploit these features in a real-time editing system, not only to replicate the behaviors in existing broadcasting, but also to generalize to novel camera layouts. We demonstrate our approach on the volumetric reconstruction of UFC fights, compare our results with baseline methods, and report different qualitative and quantitative evaluations.
Story-to-Images Translation: Leveraging Diffusion Models and Large Language Models for Sequence Image Generation

Diffusion models are catalyzing breakthroughs in creative fields, with a notable impact on text-to-image generation. This study centers on the transformation of textual narratives into coherent sequences of images - a process currently hampered by issues of consistency and contextual fidelity. To address these challenges, we propose a method utilizing a large language model, with an emphasis on context and character information. Empirical evaluations, carried out using Hollywood movie scripts, clearly indicate that our approach improves both the consistency and contextual fidelity of the resulting image sequences.
Human Action Recognition Method Based on Spatio-temporal Relationship

Traditional human motion recognition algorithms cannot fully utilize the spatio-temporal information in the video. In this paper, we propose a human motion recognition method with spatio-temporal modeling from local to global perspective. For local spatio-temporal information, the motion feature extraction module and the multi-scale spatio-temporal feature extraction module are designed. The local motion features are extracted by establishing cross-frame level correspondence within the network and combined into spatial features to obtain local spatio-temporal information. For global spatio-temporal information, the segmented network fusion module is designed. The representative recognition results are filtered from the video segments, and the fused filtered recognition results predict the final classification results. The results show that the model can effectively obtain the spatio-temporal dynamic information in the video for action recognition, achieve high recognition accuracy, and greatly reduce the model’s complexity.
Efficient Few-Shot Image Generation via Lightweight Octave Generative Adversarial Networks

Generating high-quality images from few-shot image is always a challenging task. FastGAN achieves great success in few-shot image generation task by using a lightweight network structure with incorporating a self-supervised discriminator and skip-layer channel-wise excitation module. However, FastGAN still has defect that the generated image is severe distortion. To tackle this problem, this work proposes a novel few-shot generation method called Octave Generative Adversarial Network (OctGAN) which using octave convolution in the unconditional generation task. Specifically, octave convolution is utilized to directly decouple high and low frequency features at the feature level, thereby high frequency features are used to enhance the generator’s image generation quality and low frequency features are used to increase the discriminator’s receptive field. Through qualitative and quantitative experimental analysis on multiple datasets covering a wide variety of image domains demonstrate that this work proves OctGAN not only has better image generation ability but also has less parameters and computational complexity.
Narrative Graph for Narrative Generation from Long Videos

Advancements in camera technology and cloud storage have led to a surge in video content creation, making videos more accessible. However, consuming raw, unprocessed, and lengthy videos can be unengaging. While videos with human-authored narratives (such as videos on YouTube) are captivating, creating such video requires a tremendous amount of effort and skill, and its scalability remains a bottleneck. To address this, we propose an algorithmic narrator that generates topic-specific narratives in real-time from raw videos, inspired by ChatGPT's natural language processing capabilities. Specifically, we proposed a novel narrative graph structure that captures narrative-worthy and semantically enriched factual information, as well as establishes temporal and causal links between narrative segments. The narrative graph is then fed to the algorithmic narrator to generate a textual narrative summary. Our comprehensive empirical study demonstrates the potential of algorithmic narrators and narrative graphs in creating engaging and coherent narratives, offering insights for the future of video content consumption.
Attribute Space Analysis for Image Editing

Image editing is a widely studied topic in computer vision, which enables the modification of specific attributes in images without altering other crucial information. One popular unsupervised technique currently used is feature decomposition in the latent space of Generative Adversarial Networks (GANs), which provides editing directions that can control attribute changes to achieve desired image editing results. However, this method often does not allow for the direct acquisition of the desired editing direction by setting the target attribute in advance. In this work, we propose a method to finding editing directions in the attribute space by analyzing image differences. This enables users to obtain target directions by actively defining the attribute they want to change. Specifically, this method discovers semantic directions suitable for target attribute editing by applying Principal Component Analysis (PCA) on the difference of image latent codes embedded in the latent space. Through experiments, our method can effectively find the target editing direction according to user needs and achieve satisfactory editing effects at the same time.
Graph neural network based approach to automatically assigning common weakness enumeration identifiers for vulnerabilities

Abstract Vulnerability reports are essential for improving software security since they record key information on vulnerabilities. In a report, CWE denotes the weakness of the vulnerability and thus helps quickly understand the cause of the vulnerability. Therefore, CWE assignment is useful for categorizing newly discovered vulnerabilities. In this paper, we propose an automatic CWE assignment method with graph neural networks. First, we prepare a dataset that contains 3394 real world vulnerabilities from Linux, OpenSSL, Wireshark and many other software programs. Then, we extract statements with vulnerability syntax features from these vulnerabilities and use program slicing to slice them according to the categories of syntax features. On top of slices, we represent these slices with graphs that characterize the data dependency and control dependency between statements. Finally, we employ the graph neural networks to learn the hidden information from these graphs and leverage the Siamese network to compute the similarity between vulnerability functions, thereby assigning CWE IDs for these vulnerabilities. The experimental results show that the proposed method is effective compared to existing methods.
Learning Discriminative Proposal Representation for Multi-object Tracking

Multiple object tracking (MOT) by tracklets rather than discrete detections has received more attention in recent years. Following the tracking-by-detection paradigm, many approaches treat tracklets as individual units in data association, aiming at exploiting local or global relationships among them. However, the problem of fragmentations still remains. When severe occlusions occur, adjacent trajectories will collapse into many ambiguous tracklets, which renders tracklet representations to be unreliable. To address this, we treat potential tracklets to be linked as a proposal and propose a trainable tracklet-to-proposal embedding framework based on graph attention network (GAT). Guided by tracklet-wise information, our framework mainly designs two tracklet-embedding modules to extract intra- and inter-tracklet features to generate discriminative representations of tracklet-based proposals, enhancing the accuracy of proposal classification. We experimentally demonstrate that the proposed method significantly outperforms previous state-of-the-art techniques on MOT17 public benchmarks.
Unsupervised Vehicle Re-Identification via Raw UAV Videos

For matching vehicles across different camera views, vehicle Re-Identification has made great progress in supervised learning. However, supervised approach would require extensive manual labeling which is costly and unfeasible for large-scale vehicle Re-ID dataset. Therefore, we propose an unsupervised method to overcome the difficulty of vehicle ID labeling. Inspired by self-supervised methods in object tracking, we utilize self-supervised tracker to associate vehicle images in each unlabeled raw videos. We also utilize object sequence clustering method to associate vehicles from different videos and ensure the quality of the predicted pseudo labels. Based on these vehicle images and predicted labels discriminative vehicle features can be learned. In this paper, we construct a large-scale Unmanned Aerial Vehicle (UAV) vehicle video dataset to facilitate the study of video-based unsupervised vehicle Re-ID. Extensive experiments show that our method is effective and achieves competitive performance compared with recent unsupervised works. In addition, using the data obtained by the proposed method as the pre-training data can further improve the performance of the fully-supervised methods.
Generalized Zero-Shot Space Target Recognition Based on Global-Local Visual Feature Embedding Network

Existing deep learning-based space target recognition methods rely on abundantly labeled samples and are not capable of recognizing samples from unseen classes without training. In this article, based on generalized zero-shot learning (GZSL), we propose a space target recognition framework to simultaneously recognize space targets from both seen and unseen classes. First, we defined semantic attributes to describe the characteristics of different categories of space targets. Second, we constructed a dual-branch neural network, termed the global-local visual feature embedding network (GLVFENet), which jointly learns global and local visual features to obtain discriminative feature representations, thereby achieving GZSL for space targets with higher accuracy. Specifically, the global visual feature embedding subnetwork (GVFE-Subnet) calculates the compatibility score by measuring the cosine similarity between the projection of global visual features in the semantic space and various semantic vectors, thereby obtaining global visual embeddings. The local visual feature embedding subnetwork (LVFE-Subnet) introduces soft space attention, and an encoder discovers the semantic-guided local regions in the image to then generate local visual embeddings. Finally, the visual embeddings from both branches were combined and matched with semantics. The calibrated stacking method is introduced to achieve GZSL recognition of space targets. Extensive experiments were conducted on an electromagnetic simulation dataset of nine categories of space targets, and the effectiveness of our GLVFENet is confirmed.
Efficient Federated Learning with Pre-Trained Large Language Model Using Several Adapter Mechanisms

Recent advancements in deep learning have led to various challenges, one of which is the issue of data privacy in training data. To address this issue, federated learning, a technique that merges models trained by clients on servers, has emerged as an attractive solution. However, federated learning faces challenges related to data heterogeneity and system heterogeneity. Recent observations suggest that incorporating pre-trained models into federated learning can mitigate some of these challenges. Nonetheless, the main drawback of pre-trained models lies in their typically large model size, leading to excessive data transmission when clients send these models to the server. Additionally, federated learning involves multiple global steps, which means transmitting a large language model to multiple clients results in too much data exchange. In this paper, we propose a novel approach to address this challenge using adapters. Adapters demonstrate training efficiency by training a small capacity adapter layer alongside a large language model. This unique characteristic reduces the volume of data transmission, offering a practical solution to the problem. The evaluation results demonstrate that the proposed method achieves a reduction in training time of approximately 20–40% and a transmission speed improvement of over 98% compared to previous approaches.
Fault Diagnosis of Vibration Sensors Based on Triage Loss Function-Improved XGBoost

Vibration sensors are prone to bias, drift, and other failures. To avoid misjudgments in state monitoring systems and potential safety accidents caused by vibration sensor failures, it is significant to diagnose the faults of vibration sensors. Existing methods for vibration sensor fault diagnosis are primarily based on Deep Learning, but Extreme Gradient Boosting stands out due to its excellent interpretability, and compared to other ensemble learning algorithms, it boasts superior accuracy and efficiency. Therefore, a vibration sensor fault diagnosis method based on Extreme Gradient Boosting is proposed to diagnose seven common types of faults in vibration sensors. To prevent the model from being overwhelmed by simple negative cases during training, a new loss function named Triage Loss is designed to improve the classifier’s performance. The vibration sensor fault diagnosis has confirmed the efficacy and practicality of the suggested approach. The experimental results indicate that the training of the model done using Triage Loss outperforms the training model done using the default loss function, with a maximum improvement of 5.4% accuracy, 5.45% in the F1-score, and 9.87% in the mean Average Precision under different fault rates.
The Paradigm Shift to UX and the Durability of Usability in TPC

The past two decades have experienced a paradigm shift from a narrow conception of usability to a broader process of user experience. We argue that durable connections to usability remain in TPC. In this perspectives piece, we highlight the paradigm shift and share traces of how the usability paradigm remains durable, primarily in the service course. We share savvy practices of instructors embracing the UX paradigm, even in the face of constraints.
A Holistic Model for Disciplinary Professional Development—Overcoming the Disciplinary Barriers to Implementing ICT in Teaching

What helps teachers overcome technological barriers to Information and Communications Technology (ICT) implementation in their subject domains? Can a disciplinary professional development model help reduce these disciplinary barriers? To answer these questions, this mixed-methods case study examines the change in digital ICT implementation and its barriers among secondary school first language teachers after the utilization of a holistic model for disciplinary professional development (language, science, etc.). Ninety-two first language teachers participated in a professional development program that included a unique instructor profile, theoretical and practical training, technical support, and ongoing individual and school-wide didactic and pedagogical support. The teachers’ experiences and changes in their personal general experience during their professional development were then assessed via interviews and open-ended questionnaires. The findings indicate that the unique professional development model largely removed the teachers’ disciplinary barriers. Specifically, during their professional development, the teachers changed their perceptions of the purposes of ICT in first language teaching in general and teaching writing in particular, indicating that they realized that ICT promotes their discipline and is not merely a tool for lesson diversification or intrigue. Furthermore, they recognized that ICT is not an option but an integral part of teaching in the 21st century.
A Dual-Stage Solar Power Prediction Model That Reflects Uncertainties in Weather Forecasts

Renewable energy sources are being expanded globally in response to global warming. Solar power generation is closely related to solar radiation and typically experiences significant fluctuations in solar radiation hours during periods of high solar radiation, leading to substantial inaccuracies in power generation predictions. In this paper, we suggest a solar power generation prediction method aimed at minimizing prediction errors during solar time. The proposed method comprises two stages. The first stage is the construction of the Solar Base Model by extracting characteristics from input variables. In the second stage, the prediction error period is detected using the Solar Change Point, which measures the difference between the predicted output from the Solar Base Model and the actual power generation. Subsequently, the probability of a weather forecast state change within the error occurrence period is calculated, and this information is used to update the power generation forecast value. The performance evaluation was restricted to July and August. The average improvement rate in predicted power generation was 24.5%. Using the proposed model, updates to weather forecast status information were implemented, leading to enhanced accuracy in predicting solar power generation.
The Unequal Opportunities of Large Language Models: Examining Demographic Biases in Job Recommendations by ChatGPT and LLaMA

Warning: This paper discusses and contains content that is offensive or upsetting. Large Language Models (LLMs) have seen widespread deployment in various real-world applications. Understanding these biases is crucial to comprehend the potential downstream consequences when using LLMs to make decisions, particularly for historically disadvantaged groups. In this work, we propose a simple method for analyzing and comparing demographic bias in LLMs, through the lens of job recommendations. We demonstrate the effectiveness of our method by measuring intersectional biases within ChatGPT and LLaMA, two cutting-edge LLMs. Our experiments primarily focus on uncovering gender identity and nationality bias; however, our method can be extended to examine biases associated with any intersection of demographic identities. We identify distinct biases in both models toward various demographic identities, such as both models consistently suggesting low-paying jobs for Mexican workers or preferring to recommend secretarial roles to women. Our study highlights the importance of measuring the bias of LLMs in downstream applications to understand the potential for harm and inequitable outcomes. Our code is available at https://github.com/Abel2Code/Unequal-Opportunities-of-LLMs.
Machine Learning-Based Hand Sign Recognition

Machine Learning-Based Hand Sign Recognition is a project aimed at developing a robust and efficient system for recognizing hand signs and gestures using state-of-the-art machine learning techniques. Hand sign recognition holds significant potential in a wide range of applications, including communication assistance for individuals with hearing impairments, human-computer interaction, and automation in various industries. This project leverages deep learning models, particularly convolutional neural networks (CNNs) and recurrent neural networks (RNNs), to analyze and classify hand signs and gestures. The system is trained on a diverse dataset of hand signs, encompassing different gestures, poses, and lighting conditions, to ensure its adaptability and reliability in real-world scenarios. The project's success is contingent on the fine-tuning of machine learning models, thorough testing, and the incorporation of efficient computer vision techniques. Through rigorous evaluation and validation, we aim to achieve high accuracy and real-time performance. As a result, Machine Learning-Based Hand Sign Recognition has the potential to improve accessibility and convenience for those with hearing impairments, enhance human-computer interaction, and contribute to advancements in automation and robotics across multiple domains