Reviewerly/exHarmony huggingface dataset:
Nonverbal Communication Cue Recognition: A Pathway to More Accessible Communication

Nonverbal communication, such as body language, facial expressions, and hand gestures, is crucial to human communication as it conveys more information about emotions and attitudes than spoken words. However, individuals who are blind or have low-vision (BLV) may not have access to this method of communication, leading to asymmetry in conversations. Developing systems to recognize nonverbal communication cues (NVCs) for the BLV community would enhance communication and understanding for both parties. This paper focuses on developing a multimodal computer vision system to recognize and detect NVCs. To accomplish our objective, we are collecting a dataset focused on nonverbal communication cues. Here, we propose a baseline model for recognizing NVCs and present initial results on the Aff-Wild2 dataset. Our baseline model achieved an accuracy of 68% and a F1-Score of 64% on the Aff-Wild2 validation set, making it comparable with previous state of the art results. Furthermore, we discuss the various challenges associated with NVC recognition as well as the limitations of our current work.
FlexiCurve: Flexible Piecewise Curves Estimation for Photo Retouching

This paper presents a new method, called FlexiCurve, for photo retouching. Unlike most existing methods that perform image-to-image mapping, which requires expensive pixel-wise reconstruction, FlexiCurve takes an input image and estimates global curves to adjust the image. The adjustment curves are specially designed for performing piecewise mapping, taking nonlinear adjustment and differentiability into account. To cope with challenging and diverse properties in real-world photos, FlexiCurve is formulated to produce diverse estimations. The spatial dependencies among these estimations are implicitly modeled by a Transformer structure to improve local retouching of different regions. Thanks to the image-to-curve formulation, FlexiCurve only needs a lightweight network. Our method improves efficiency without compromising the retouching quality and losing details in the original image. The method is also appealing as it is not limited to paired training data, thus it can flexibly learn rich retouching styles from unpaired data. Extensive experiments demonstrate the efficiency, retouching performance, and flexibility of our method quantitatively and qualitatively.
All Keypoints You Need: Detecting Arbitrary Keypoints on the Body of Triple, High, and Long Jump Athletes

Performance analyses based on videos are commonly used by coaches of athletes in various sports disciplines. In individual sports, these analyses mainly comprise the body posture. This paper focuses on the disciplines of triple, high, and long jump, which require fine-grained locations of the athlete’s body. Typical human pose estimation datasets provide only a very limited set of keypoints, which is not sufficient in this case. Therefore, we propose a method to detect arbitrary keypoints on the whole body of the athlete by leveraging the limited set of annotated keypoints and auto-generated segmentation masks of body parts. Evaluations show that our model is capable of detecting keypoints on the head, torso, hands, feet, arms, and legs, including also bent elbows and knees. We analyze and compare different techniques to encode desired keypoints as the model’s input and their embedding for the Transformer backbone.
ZEBRA: Explaining rare cases through outlying interpretable concepts

Anomaly detection methods can detect outliers, but what are the properties of an outlierƒ In this paper, we propose ZEBRA, a novel framework for generating explanations of an outlier based on the analysis of feature rarity in an interpretable feature space. The contributions of our work include: (a) a modular model-agnostic framework for explanations of outliers; (b) a statistical explanation method based on a rarity score and weighted aggregation functions; (c) multimodal explanations combining visual, textual, and numeric explanations. ZEBRA simplifies the mapping of low-level features to high-level concepts to generate multimodal and human-readable explanations of outliers.
Internal Diverse Image Completion

Image completion is widely used in photo restoration and editing applications, e.g. for object removal. Recently, there has been a surge of research on generating diverse completions for missing regions. However, existing methods require large training sets from a specific domain of interest, and often fail on general-content images. In this paper, we propose a diverse completion method that does not require a training set and can thus treat arbitrary images from any domain. Our internal diverse completion (IDC) approach draws inspiration from recent single-image generative models that are trained on multiple scales of a single image, adapting them to the extreme setting in which only a small portion of the image is available for training. We illustrate the strength of IDC on several datasets, using both user studies and quantitative comparisons.
AutoSplice: A Text-prompt Manipulated Image Dataset for Media Forensics

Recent advancements in language-image models have led to the development of highly realistic images that can be generated from textual descriptions. However, the increased visual quality of these generated images poses a potential threat to the field of media forensics. This paper aims to investigate the level of challenge that language-image generation models pose to media forensics. To achieve this, we propose a new approach that leverages the DALL-E2 language-image model to automatically generate and splice masked regions guided by a text prompt. To ensure the creation of realistic manipulations, we have designed an annotation platform with human checking to verify reasonable text prompts. This approach has resulted in the creation of a new image dataset called AutoSplice, containing 5,894 manipulated and authentic images. Specifically, we have generated a total of 3,621 images by locally or globally manipulating real-world image-caption pairs, which we believe will provide a valuable resource for developing generalized detection methods in this area <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> . The dataset is evaluated under two media forensic tasks: forgery detection and localization. Our extensive experiments show that most media forensic models struggle to detect the AutoSplice dataset as an unseen manipulation. However, when fine-tuned models are used, they exhibit improved performance in both tasks.
A Global-Local Tracking Framework Driven by Both Motion and Appearance for Infrared Anti-UAV

Unmanned aerial vehicles (UAVs) have been widely used in various application domains, but unauthorized UAVs may pose a threat to public safety due to violation of aviation regulations. Therefore, how to design an effective UAV tracking method for anti-UAV is a crucial part of the UAV-defense system. In this paper, we propose a Global-Local Tracking Framework driven by both Motion and Appearance (GLTF-MA) including four modules to deal with the practical difficulties in infrared anti-UAV. Firstly, a Periodic Global Detection (PGD) module is periodically performed to re-locate UAVs in the whole image to account for frequent appearance/disappearance and unstable flight paths of UAVs. Meanwhile, a Multi-stage Local Tracking (MLT) module containing a priori stage switching mechanism, motion-appearance matching mechanism, and a motion estimation punisher is routinely implemented to deal with the tiny size of UAVs and background interference. Next, a Target Disappearance Judgement (TDJ) module is performed to give a robust target disappearance flag, followed by a Bounding Box Refinement (BBR) module to refine the target box when the TDJ module thinks the target exists. Extensive experiments demonstrate the superiority of GLTF-MA over other competing counterparts, especially when the UAV is low resolution and moves quickly.
Weakly Supervised Visual Question Answer Generation

Growing interest in conversational agents promote two-way human-computer communications involving asking and answering visual questions have become an active area of research in AI. Thus, generation of visual question-answer pair(s) becomes an important and challenging task. To address this issue, we propose a weakly-supervised visual question answer generation method that generates a relevant question-answer pairs for a given input image and associated caption. Most of the prior works are supervised and depend on the annotated question-answer datasets. In our work, we present a weakly supervised method that synthetically generates question-answer pairs procedurally from visual information and captions. The proposed method initially extracts list of answer words, then does nearest question generation that uses the caption and answer word to generate synthetic question. Next, the relevant question generator converts the nearest question to relevant language question by dependency parsing and in-order tree traversal, finally, fine-tune a ViLBERT model with the question-answer pair(s) generated at end. We perform an exhaustive experimental analysis on VQA dataset and see that our model significantly outperform SOTA methods on BLEU scores. We also show the results wrt baseline models and ablation study.
Improving Rare Classes on nuScenes LiDAR segmentation Through Targeted Domain Adaptation

We generate synthetic data in order to target improvement on specific rare classes in LiDAR segmentation without regressing performance on the existing, plentiful classes. While using auxiliary data to improve performance on a domain is not new with respect to classification, there is limited research on targeting specific classes with this technique. It is currently unclear how to extend those methods to work well on more complicated but realistic use cases for autonomous driving such as LiDAR segmentation.By upsampling specific classes in the auxiliary domain, mixing data between domains, and splitting representation building and fine-tuning, we are able to see impressive improvements on a targeted rare class without losing performance on the other classes. On the popular autonomous driving benchmark nuScenes, we use this procedure to improve performance on the rare class of cyclists by 18%, resulting in the best Cylinder3D model on the LiDAR segmentation benchmark. We also show that these techniques extend to other classes (debris) and other tasks (LiDAR object detection), giving strong evidence that this methodology generalizes well to other autonomous perception tasks.
Analyzing Results of Depth Estimation Models with Monocular Criteria

Monocular depth estimation is an essential but ill-posed (computer) vision task. While human visual perception of depth relies on several monocular depth clues, such as occlusion of objects, relative height, usual object size, linear perspective, deep learning models have to implicitly learn these cues from labeled training data to determine depth. In this paper, we investigate whether monocular depth criteria from human vision are violated for certain image instances given a model’s predictions. We consider the task of depth estimation as a ranking problem, i.e., for a given pair of points, we estimate which point is nearer to the camera. In particular, we model four monocular depth criteria to automatically predict a subset of point pairs and infer their depth relation. Our experiments show that the implemented depth criteria achieve comparable performance to deep learning models. This allows the investigation of models with regard to the plausibility of predictions by finding image instances where the prediction is incorrect according to modeled human visual perception.
Efficient Structure Search for Person Re-identification

Person re-identification (re-ID) has been a research hotspot in the field of computer vision, and in recent years, with the development of deep learning, combined re-ID with deep learning has achieved promising performance. Pedestrian features, however, can be easily affected by various factors such as illumination, occlusion, and viewpoint changes, which cannot be fully expressed. In this paper, we use ResNet-50 network as the backbone network for feature extraction. Firstly, we embed a lightweight attention mechanism to the residual blocks in it to enhance the feature information extraction. Secondly, the training techniques of random erasure using Cutout, downsampling using generalized mean pooling, and adding ArcFace loss improve the performance of the model. Finally, the features are processed by convolution, batch normalization, and Relu activation to enhance the feature representation of the model. Through extensive experiments on Market1501 and DukeMTMC-ReID datasets, Rank1 is improved by 5.9% and 5.5%, and mAP is improved by 9.8% and 6.8%, respectively, and the results show that the improved model can achieve better re-ID performance.
Continual Domain Adaptation through Pruning-aided Domain-specific Weight Modulation

In this paper, we propose to develop a method to address unsupervised domain adaptation (UDA) in a practical setting of continual learning (CL). The goal is to update the model on continually changing domains while preserving domain-specific knowledge to prevent catastrophic forgetting of past-seen domains. To this end, we build a framework for preserving domain-specific features utilizing the inherent model capacity via pruning. We also perform effective inference using a novel batch-norm based metric to predict the final model parameters to be used accurately. Our approach achieves not only state-of-the-art performance but also prevents catastrophic forgetting of past domains significantly. Our code is made publicly available. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Unmasking Your Expression: Expression-Conditioned GAN for Masked Face Inpainting

As face masks continue to be a part of our daily lives, the challenge of reconstructing occluded faces remains relevant. While several approaches have been proposed for removing masks from neutral facial images, few have explored the use of facial expressions as a dominant feature for reconstruction of expressive faces. To address this gap, we propose an expression-conditioned GAN (ECGAN) for reconstructing masked faces with a specified expression. Our approach leverages both the binary segmentation map of the mask and an expression label to generate high-quality images. To train our ECGAN in a supervised manner, we synthesize masked images using the RAFDB dataset to create non-masked-masked pairs of images for training. We evaluate of our approach on the RAFDB test set, demonstrating its effectiveness in generating realistic images that convincingly belong to the given expression class. This is further highlighted by comparing it to a baseline model and a state-of-the-art approach without expression-input. The code is available at https://github.com/SridharSola/ECGAN.
K-DUMBs IoRT: Knowledge Driven Unified Model Block Sharing in the Internet of Robotic Things

6G is expected to revolutionize the Internet of things (IoT) applications toward a future of completely intelligent and autonomous systems. Conventional machine-learning approaches involve centralizing training data in a data center, where the algorithms can be used for data analysis and inference. To promote green computing in IoT applications, Machine-2-Machine (M2M) technologies are largely focused on lowering energy consumption and creating effective IT infrastructure. In this paper, we introduce an AI-enabled One-Shot Interference(O-SI) Knowledge-Driven unified model block sharing (K-Dumbs) framework in which actionable knowledge is aggregated from the training perception robots to facilitate others at the Edge in the vicinity. To demonstrate the practicality of the proposed concept, we explore a K-Dumb Fed-Average (FedAvg) algorithm to meet the massively distributed and unbalanced pattern and privacy requirement of the Internet of Robotic Things(IoRT). Simulation results show that, when compared to traditional Federated Learning (FL) systems, the proposed K-Dumb FedAvg architecture delivers higher information-sharing and learning quality. In addition, we validate our method using MNIST handwritten digits for training image processing with an accuracy that is close to the centralized solution for up to 80% reduction in the amount of exchange data with the O-SI method. Furthermore, the suggested solution reduces IoRT energy consumption by up to 10 times and protects privacy.
Posture-based Infant Action Recognition in the Wild with Very Limited Data

Automatic detection of infant actions from home videos could aid medical and behavioral specialists in the early detection of motor impairments in infancy. However, most computer vision approaches for action recognition are centered around adult subjects, following datasets and benchmarks in the field. In this work, we present a data-efficient pipeline for infant action recognition based on the idea of modeling an action as a time sequence consisting of two different stable postures with a transition period between them. The postures are detected frame-wise from the estimated 2D and 3D infant body poses and the action sequence is segmented based on the posture-driven low-dimensional features of each frame. To spur further research in the field, we also created and release the first-of-its-kind infant action dataset—InfAct—consisting of 200 fully annotated home videos representing a wide range of common infant actions, intended as a public benchmark. Among the ten more common classes of infant actions, our action recognition model achieved 78.0% accuracy when tested on InfAct, highlighting the promise of video-based infant action recognition as a viable monitoring tool for infant motor development <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Green Placement of Data Centers over Optical Networks for Minimizing Carbon Emissions

This paper proposes a low-carbon data center placement model based on integer linear programming to reduce carbon emissions. Simulations show the model can effectively reduce carbon emissions while meeting the latency and power constraints.
Wildlife Image Generation from Scene Graphs

Image generation from natural language descriptions is an exciting and challenging task in computer vision and natural language processing. In this work, we propose a novel method to generate synthetic images from scene graphs in the context of wildlife scenarios. Given a scene graph, our method uses a graph convolutional network to predict semantic layouts, and a semi-parametric approach based on a cascade refinement network to synthesize the final image. We test our approach on a subset of COCO dataset, which we call COCO-Wildlife. Our results outperform the baselines, both quantitatively and qualitatively, and the visual results show the ability of our approach to generate stunning images with natural interaction between the different objects. Our findings show the potential to expand the use case of the proposed method to other contexts where scale and realism is fundamental.
Seg-XRes-CAM: Explaining Spatially Local Regions in Image Segmentation

While many post-hoc model interpretability techniques exist for image classification, image segmentation has not received the same attention. An extension of Grad-CAM, Seg-Grad-CAM was proposed as a local interpretability technique for image segmentation. In this paper, we highlight that by virtue of its design, Seg-Grad-CAM does not utilize spatial information when it comes to generating explanations for regions within a segmentation map. Taking inspiration from HiResCAM, we propose Seg-XRes-CAM in order to solve this problem. We verify the utility of our proposed method by visually comparing explanations generated from Seg-Grad-CAM and Seg-XRes-CAM against a model-agnostic, perturbation-based method, RISE. The code is available at https://github.com/Nouman97/Seg_XRes_CAM.
A 7-bit Precision Linearized Mach-Zehnder Interferometer for High Accuracy Optical Neural Networks

A full-scale linearized MZI is proposed and experimentally validated by introducing an extremely overcoupled microring. It exhibits triangular-like transmission with a 3-bit improvement in precision while maintaining the same small number of phase shifters. The linearized MZI-based ONN demonstrates higher training accuracy.
DynaShare: Task and Instance Conditioned Parameter Sharing for Multi-Task Learning

Multi-task networks rely on effective parameter sharing to achieve robust generalization across tasks. In this paper, we present a novel parameter sharing method for multitask learning that conditions parameter sharing on both the task and the intermediate feature representations at inference time. In contrast to traditional parameter sharing approaches, which fix or learn a deterministic sharing pattern during training and apply the same pattern to all examples during inference, we propose to dynamically decide which parts of the network to activate based on both the task and the input instance. Our approach learns a hierarchical gating policy consisting of a task-specific policy for coarse layer selection and gating units for individual input instances, which work together to determine the execution path at inference time. Experiments on the NYU v2, Cityscapes and MIMIC-III datasets demonstrate the potential of the proposed approach and its applicability across problem domains.
Real-time Segmenting Human Portrait at Anywhere

Real-time portrait segmentation is an important task for a wide range of human-centered applications. With the increase of mobile devices, such as mobile phones and personal computers, more and more human-centered applications are transferred to running on these devices to provide users with a better experience. So, lightweight model designing becomes indispensable for building applications on these limited-resource platforms. In this work, we propose a real-time segmentation U-shape architecture with a Re-parameter Compress Residual module (RCR module) and a bypass branch that can further improve the segmentation efficiency. In order to speed up during the inference phase, the RCR module is compressed during inference, and the bypass branch adds the missing edge information improving the learning skill of the network. Based on the experiments on the EG1800 and P3M-10K dataset compared with the state-of-the-art methods, the proposed method achieves better performance with less number of parameters. Specifically, our method reduces the number of parameters around 50% while maintaining comparable high accuracy, and the details will be described in the experiment part.
Uncovering the Inner Workings of STEGO for Safe Unsupervised Semantic Segmentation

Self-supervised pre-training strategies have recently shown impressive results for training general-purpose feature extraction backbones in computer vision. In combination with the Vision Transformer architecture, the DINO self-distillation technique has interesting emerging proper-ties, such as unsupervised clustering in the latent space and semantic correspondences of the produced features without using explicit human-annotated labels. The STEGO method for unsupervised semantic segmentation contrastively distills feature correspondences of a DINO-pre-trained Vision Transformer and recently set a new state of the art. However, the detailed workings of STEGO have yet to be disentangled, preventing its usage in safety-critical applications.This paper provides a deeper understanding of the STEGO architecture and training strategy by conducting studies that uncover the working mechanisms behind STEGO, reproduce and extend its experimental validation, and investigate the ability of STEGO to transfer to different datasets. Results demonstrate that the STEGO architecture can be interpreted as a semantics-preserving dimensionality reduction technique.