Miracle-dz/UniFAR-FTUs huggingface dataset:
SmartMeeting: Automatic Meeting Transcription and Summarization for In-Person Conversations

Meetings are a necessary part of the operations of any institution, whether they are held online or in-person. However, meeting transcription and summarization are always painful requirements since they involve tedious human effort. This drives the need for automatic meeting transcription and summarization (AMTS) systems. A successful AMTS system relies on systematic integration of multiple natural language processing (NLP) techniques, such as automatic speech recognition, speaker identification, and meeting summarization, which are traditionally developed separately and validated offline with standard datasets. In this demonstration, we provide a novel productive meeting tool named SmartMeeting, which enables users to automatically record, transcribe, summarize, and manage the information in an in-person meeting. SmartMeeting transcribes every word on the fly, enriches the transcript with speaker identification and voice separation, and extracts essential decisions and crucial insights automatically. In our demonstration, the audience can experience the great potential of the state-of-the-art NLP techniques in this real-life application.
Augmenting TV Shows via Uncalibrated Camera Small Motion Tracking in Dynamic Scene

To augment the TV show in post-production, we propose a novel solution to uncalibrated camera small motion tracking in a dynamic scene that simultaneously reconstructs the sparse 3D scene and computes camera poses and focal lengths of each frame. The critical elements of our approach are a robust image feature tracking strategy in dynamic scenes followed by automatic local-window frames slicing, local and global bundle adjustment optimization initialized by a homography-based uncalibrated relative rotation solver. The proposed method allows us to add the virtual objects (elements) into the reconstructed 3D scene, then composite them back into the original shot while perfectly matched perspective and appear seamless. The evaluation of a large variety of real TV show sequences demonstrates the merits of our method against state-of-the-art works and commercial software products.
CARE: Cloudified Android OSes on the Cloud Rendering

GPUs have become ubiquitous in the Cloud-rendering areas due to the outstanding rendering performance. However, many existing Cloud-rendering systems suffer from low GPU utilization caused by the CPU bottleneck. Recent proposals (e.g., API-forwarding and c-GPU) for GPU-usage optimization are promising but fail to address the system-resource redundancy issues (i.e., each instance tends to occupy all the system resources exceeding their requirements), leading to unnecessary CPU consumption and lowering GPU utilization. We conducted an experiment by testing real-world applications on the percentage of unused resources to demonstrate the severity of this issue. Nearly 50% of resources are unused. To solve this problem, we present CARE, the first framework intended to reduce the system-level redundancy by cloudifying the system from monolithic to Cloud-native. To allow users to configure their own required services, CARE puts forward a functional unit called Configurable Android (CA). To allow multiple instances to share certain types of resources, CARE innovates Sharing Resource (SR). To reduce the unused services, CARE introduces Pruning Resources (PR). Last but not least, to further reduce CPU consumption, CARE proposes a new isolation unit called CiC. So far, CARE primarily focuses on Android systems due to the great popularity of Android Cloud-gaming frameworks. Furthermore, CARE can handle 60 heavyweight instances (e.g., KOG (King of Glory)) on Intel® SG1.
TransFusion: Multi-Modal Fusion for Video Tag Inference via Translation-based Knowledge Embedding

Tag inference is an important task in the business of video platforms with wide applications such as recommendation, interpretation, and more. Existing works are mainly based on extracting video information from multiple modalities such as frames or music, and then infer tags through classification or object detection. This, however, does not apply to inferring generic tags or taxonomy that are less relevant to video contents, such as video originality or its broader category, which are important in practice. In this paper, we claim that these generic tags can be modeled through the semantic relations between videos and tags, and can be utilized simultaneously with the multi-modal features to achieve better video tagging. We propose TransFusion, an end-to-end supervised learning framework that fuses multi-modal embeddings (e.g., vision, audio, texts, etc.) with the knowledge embedding to derive the video representation. To infer the diverse tags following heterogeneous relations, TransFusion adopts a dual attentive approach to learn both the modality importance in fusion and relation importance in inference. Besides, it is general enough and can be used with the existing translation-based knowledge embedding approaches. Extensive experiments show that TransFusion outperforms the baseline methods with lowered mean rank and at least 9.59% improvement in HITS@10 on the real-world video knowledge graph.
Wearable Interactions for Users with Motor Impairments: Systematic Review, Inventory, and Research Implications

We conduct a systematic literature review on wearable interactions for users with motor impairments and report results from a meta-analysis of 57 scientific articles identified in the ACM DL and IEEE Xplore databases. Our findings show limited research conducted on accessible wearable interactions (e.g., just four papers addressing smartwatch input), a disproportionate interest for hand gestures compared to other input modalities for wearable devices, and low numbers of participants with motor impairments involved in user studies about wearable interactions (a median of 6.0 and average of 8.2 participants per study). We compile an inventory of 92 finger, hand, head, shoulder, eye gaze, and foot gesture commands for smartwatches, smartglasses, headsets, earsets, fitness trackers, data gloves, and armband wearable devices extracted from the scientific literature that we surveyed. Based on our findings, we propose four directions for future research on accessible wearable interactions for users with motor impairments.
Searching a Hierarchically Aggregated Fusion Architecture for Fast Multi-Modality Image Fusion

Multi-modality image fusion refers to generating a complementary image that integrates typical characteristics from source images. In recent years, we have witnessed the remarkable progress of deep learning models for multi-modality fusion. Existing CNN-based approaches strain every nerve to design various architectures for realizing these tasks in an end-to-end manner. However, these handcrafted designs are unable to cope with the high demanding fusion tasks, resulting in blurred targets and lost textural details. To alleviate these issues, in this paper, we propose a novel approach, aiming at searching effective architectures according to various modality principles and fusion mechanisms. Specifically, we construct a hierarchically aggregated fusion architecture to extract and refine fused features from feature-level and object-level fusion perspectives, which is responsible for obtaining complementary target/detail representations. Then by investigating diverse effective practices, we composite a more flexible fusion-specific search space. Motivated by the collaborative principle, we employ a new search strategy with different principled losses and hardware constraints for sufficient discovery of components. As a result, we can obtain a task-specific architecture with fast inference time. Extensive quantitative and qualitative results demonstrate the superiority and versatility of our method against state-of-the-art methods.
Camera-Agnostic Person Re-Identification via Adversarial Disentangling Learning

Despite the success of single-domain person re-identification (ReID), current supervised models degrade dramatically when deployed to unseen domains, mainly due to the discrepancy across cameras. To tackle this issue, we propose an Adversarial Disentangling Learning (ADL) framework to decouple camera-related and ID-related features, which can be readily used for camera-agnostic person ReID. ADL adopts a discriminative way instead of the mainstream generative styles in disentangling methods, eg. , GAN or VAE based, because for person ReID task only the information to discriminate IDs is needed, and more information to generate images are redundant and may be noisy. Specifically, our model involves a feature separation module that encodes images into two separate feature spaces and a disentangled feature learning module that performs adversarial training to minimize mutual information. We design an effective solution to approximate and minimize mutual information by transforming it into a discrimination problem. The two modules are co-designed to obtain strong generalization ability by only using source dataset. Extensive experiments on three public benchmarks show that our method outperforms the state-of-the-art generalizable person ReID model by a large margin. Our code is publicly available at https://github.com/luckyaci/ADL_ReID.
GPS: A Global Publish-Subscribe Model for Multi-GPU Memory Management

Suboptimal management of memory and bandwidth is one of the primary causes of low performance on systems comprising multiple GPUs. Existing memory management solutions like Unified Memory (UM) offer simplified programming but come at the cost of performance: applications can even exhibit slowdown with increasing GPU count due to their inability to leverage system resources effectively. To solve this challenge, we propose GPS, a HW/SW multi-GPU memory management technique that efficiently orchestrates inter-GPU communication using proactive data transfers. GPS offers the programmability advantage of multi-GPU shared memory with the performance of GPU-local memory. To enable this, GPS automatically tracks the data accesses performed by each GPU, maintains duplicate physical replicas of shared regions in each GPU’s local memory, and pushes updates to the replicas in all consumer GPUs. GPS is compatible within the existing NVIDIA GPU memory consistency model but takes full advantage of its relaxed nature to deliver high performance. We evaluate GPS in the context of a 4-GPU system with varying interconnects and show that GPS achieves an average speedup of 3.0 × relative to the performance of a single GPU, outperforming the next best available multi-GPU memory management technique by 2.3 × on average. In a 16-GPU system, using a future PCIe 6.0 interconnect, we demonstrate a 7.9 × average strong scaling speedup over single-GPU performance, capturing 80% of the available opportunity.
MeronymNet: A Hierarchical Model for Unified and Controllable Multi-Category Object Generation

We introduce MeronymNet, a novel hierarchical approach for controllable, part-based generation of multi-category objects using a single unified model. We adopt a guided coarse-to-fine strategy involving semantically conditioned generation of bounding box layouts, pixel-level part layouts and ultimately, the object depictions themselves. We use Graph Convolutional Networks, Deep Recurrent Networks along with custom-designed Conditional Variational Autoencoders to enable flexible, diverse and category-aware generation of 2-D objects in a controlled manner. The performance scores for generated objects reflect MeronymNet's superior performance compared to multiple strong baselines and ablative variants. We also showcase MeronymNet's suitability for controllable object generation and interactive object editing at various levels of structural and semantic granularity.
What difference does tech make? Conceptualizations of Disability and Assistive Technology among Kenyan Youth: Conceptualizations of Disability and AT

Most research which investigates stigma towards with people with disabilities and the use of Assistive Technology (AT) are based in the Global North and focus on the experiences of people with disabilities and the consequences that stigma has on choices surrounding AT. However, stigma is a societal construct rooted in the attitude and beliefs that people without disabilities hold on disability and AT. Furthermore, the portrayal of people with disabilities and AT is dependent on the social context. In this paper, we examine how young Kenyans without disabilities view people with disabilities and AT users. Findings show that while the portrayal of disability is often shaped by negative emotion, participants felt that many of the barriers affecting people with disabilities were created by society. Perceptions of AT differed –devices were not only seen as a mark of disability but also as a sign of access to resources. Therefore, what we see is an emergent picture where social barriers can be reinforced by poverty, and where poverty reinforces social barriers faced by people with disabilities. We conclude that access to appropriate technology alongside societal interventions tackling incorrect beliefs about disability can help to overcome the stigma faced by people with disabilities.
CAA: Candidate-Aware Aggregation for Temporal Action Detection

Temporal action detection aims to locate specific segments of action instances in an untrimmed video. Most existing approaches commonly extract the features of all candidate video segments and then classify them separately. However, they may neglect the underlying relationship among candidates unconsciously. In this paper, we propose a novel model termed Candidate-Aware Aggregation (CAA) to tackle this problem. In CAA, we design the Global Awareness (GA) module to exploit long-range relations among all candidates from a global perspective, which enhances the features of action instances. The GA module is then embedded into a multi-level hierarchical network named FENet, to aggregate local features in adjacent candidates to suppress background noise. As a result, the relationship among candidates is explicitly captured from both local and global perspectives, which ensures more accurate prediction results for the candidates. Extensive experiments conducted on two popular benchmarks ActivityNet-1.3 and THUMOS-14 demonstrate the superiority of CAA comparing to the state-of-the-art methods.
Zero-shot Video Emotion Recognition via Multimodal Protagonist-aware Transformer Network

Recognizing human emotions from videos has attracted significant attention in numerous computer vision and multimedia applications, such as human-computer interaction and health care. It aims to understand the emotional response of humans, where candidate emotion categories are generally defined by specific psychological theories. However, with the development of psychological theories, emotion categories become increasingly diverse and fine-grained, samples are also increasingly difficult to collect. In this paper, we investigate a new task of zero-shot video emotion recognition, which aims to recognize rare unseen emotions. Specifically, we propose a novel multimodal protagonist-aware transformer network, which is composed of two branches: one is equipped with a novel dynamic emotional attention mechanism and a visual transformer to learn better visual representations; the other is an acoustic transformer for learning discriminative acoustic representations. We manage to align the visual and acoustic representations with semantic embeddings of fine-grained emotion labels through jointly mapping them into a common space under a noise contrastive estimation objective. Extensive experimental results on three datasets demonstrate the effectiveness of the proposed method.
D³Net: Dual-Branch Disturbance Disentangling Network for Facial Expression Recognition

One of the main challenges in facial expression recognition (FER) is to address the disturbance caused by various disturbing factors, including common ones (such as identity, pose, and illumination) and potential ones (such as hairstyle, accessory, and occlusion). Recently, a number of FER methods have been developed to explicitly or implicitly alleviate the disturbance involved in facial images. However, these methods either consider only a few common disturbing factors or neglect the prior information of these disturbing factors, thus resulting in inferior recognition performance. In this paper, we propose a novel Dual-branch Disturbance Disentangling Network (D3Net), mainly consisting of an expression branch and a disturbance branch, to perform effective FER. In the disturbance branch, a label-aware sub-branch (LAS) and a label-free sub-branch (LFS) are elaborately designed to cope with different types of disturbing factors. On the one hand, LAS explicitly captures the disturbance due to some common disturbing factors by transfer learning on a pretrained model. On the other hand, LFS implicitly encodes the information of potential disturbing factors in an unsupervised manner. In particular, we introduce an Indian buffet process (IBP) prior to model the distribution of potential disturbing factors in LFS. Moreover, we leverage adversarial training to increase the differences between disturbance features and expression features, thereby enhancing the disentanglement of disturbing factors. By disentangling the disturbance from facial images, we are able to extract discriminative expression features. Extensive experiments demonstrate that our proposed method performs favorably against several state-of-the-art FER methods on both in-the-lab and in-the-wild databases.
VLAD-VSA: Cross-Domain Face Presentation Attack Detection with Vocabulary Separation and Adaptation

For face presentation attack detection (PAD), most of the spoofing cues are subtle, local image patterns (e.g., local image distortion, 3D mask edge and cut photo edges). The representations of existing PAD works with simple global pooling method, however, lose the local feature discriminability. In this paper, the VLAD aggregation method is adopted to quantize local features with visual vocabulary locally partitioning the feature space, and hence preserve the local discriminability. We further propose the vocabulary separation and adaptation method to modify VLAD for cross-domain PAD task. The proposed vocabulary separation method divides vocabulary into domain-shared and domain-specific visual words to cope with the diversity of live and attack faces under the cross-domain scenario.The proposed vocabulary adaptation method imitates the maximization step of the k-means algorithm in the end-to-end training, which guarantees the visual words be close to the center of assigned local features and thus brings robust similarity measurement. We give illustrations and extensive experiments to demonstrate the effectiveness of VLAD with the proposed vocabulary separation and adaptation method on standard cross-domain PAD benchmarks. The codes are available at https://github.com/Liubinggunzu/VLAD-VSA.
Beyond Fun: Players’ Experiences of Accessible Rehabilitation Gaming for Spinal Cord Injury

Rehabilitation gaming—play of digital games that incorporate rehabilitation exercises—is a well-known and broadly applicable way to make physical rehabilitation more fun. It can motivate patients with spinal cord injury to engage in exercises that they find boring and can be as effective as traditional physiotherapy. However, patients’ needs are not only physical. Rehabilitation also needs to help patients overcome the psychological trauma of spinal cord injury. For patients coping with disability, hopelessness, depression, anxiety, or a loss of identity, rehabilitation gaming may provide benefits beyond making exercise more fun. We asked six participants with spinal cord injury to play three cycling-based rehabilitation games to determine how play might change their experiences of rehabilitation. They said that rehabilitation games may be able to help patients to actively participate in their rehabilitation, help them to rediscover who they are, and show them a better future living with spinal cord injury.
Exploring Contextual-Aware Representation and Linguistic-Diverse Expression for Visual Dialog

Visual dialog is a fundamental vision-language task where an AI agent holds a meaningful dialogue about visual content with humans in nature. However, this task remains challenging, since there is still no consensus way to capture rich visual contextual information contained in the environment rather than only focusing on visual objects. Furthermore, conventional methods suffer from the single-answer learning strategy, where it only accepts one correct answer without considering the diverse expressions of the language (i.e., one identical meaning but multiple expressions via rephrasing or adopting synonyms etc). In this paper, we introduce Contextual-Aware Representation and linguistic-diverse Expression (CARE), a novel plug-and-play framework with contextual-based graph embedding and curriculum contrastive learning to solve the above two issues. Specifically, the contextual-based graph embedding (CGE) module aims to integrate the environmental context information with visual objects to improve the answer quality. In addition, we propose a curriculum contrastive learning (CCL) paradigm to imitate the learning habits of humans when facing a question with multiple correct answers sharing the same meaning but with diverse expressions. To support CCL, a CCL loss is designed to progressively strengthen the model's ability in identifying the answers with correct semantics. Extensive experiments are conducted on two benchmark datasets, and our proposed method outperforms the state-of-the-arts by a considerable margin on VisDial V1.0 (4.63% NDCG) and VisDial V0.9 (1.27% MRR, 1.74% R@1, 0.87% R@5, 1.28% R@10, 0.26 Mean.
JokerGAN: Memory-Efficient Model for Handwritten Text Generation with Text Line Awareness

Collecting labeled data for training of models for image recognition problems, including handwritten text recognition (HTR), is a tedious and expensive task. Recent work on handwritten text generation shows that generative models can be used as a data augmentation method to improve the performance of HTR systems. We propose a new method for handwritten text generation that uses generative adversarial networks with multi-class conditional batch normalization, which enables us to use character sequences with variable lengths as conditional input. Compared to existing methods, it has significantly lower memory requirements which are almost constant regardless of the size of the character set. This allows us to train a generative model for languages with a large number of characters, such as Japanese. We also introduce an additional condition that makes the generator aware of vertical properties of the characters in the generated sequence, which helps generate text with well-aligned characters in the text line. Experiments on handwritten text datasets show that our proposed model can be used to boost the performance of HTR, particularly when we only have access to partially annotated data and train our generative model in a semi-supervised fashion. The results also show that our model outperforms the current state-of-the-art for handwritten text generation. In addition, we perform a human evaluation study that indicates that the proposed method generates handwritten text images that look more realistic and natural. Our source code will be available at https://github.com/janzd/jokergan.