Reviewerly/exHarmony huggingface dataset:
BeCAPTCHA-Type: Biometric Keystroke Data Generation for Improved Bot Detection

This work proposes a data driven learning model for the synthesis of keystroke biometric data. The proposed method is compared with two statistical approaches based on Universal and User-dependent models. These approaches are validated on a bot detection task, using the keystroke synthetic data to improve the training process of keystroke-based bot detection systems. Our experimental framework considers a dataset with 136 million keystroke events from 168 thousand subjects. We have analyzed the performance of the three synthesis approaches through qualitative and quantitative experiments. Different bot detectors are considered based on several supervised classifiers (Support Vector Machine, Random Forest, Gaussian Naive Bayes and a Long Short-Term Memory network) and a learning framework including human and synthetic samples. The experiments demonstrate the realism of the synthetic samples. The classification results suggest that in scenarios with large labeled data, these synthetic samples can be detected with high accuracy. However, if the proposed synthetic data is nor properly modelled using massive data by bot detectors, then that data will be very difficult to detect even for the most sophisticate bot detectors. Furthermore, these results show the great potential of the presented models for improving the training of bot detection technology.
Network Specialization via Feature-level Knowledge Distillation

State-of-the-art model specialization methods are mainly based on fine-tuning a pre-trained machine learning model to fit the specific needs of a particular task or application. Or by modifying the architecture of the model itself. However, these methods are not preferable in industrial applications because of the model’s large size and the complexity of the training process. In this paper, the difficulty of network specialization is attributed to overfitting caused by a lack of data, and we propose a novel model specialization method by Knowledge Distillation (SKD). The proposed methods merge transfer learning and model compression into one stage. Specifically, we distill and transfer knowledge at the feature map level, circumventing logit-level inconsistency between teacher and student. We empirically investigate and prove the effects of the three parts: Models can be specialized to customer use cases by knowledge distillation. Knowledge distillation can effectively regularize the knowledge transfer process to a smaller, task-specific model. Compared with classical methods such as training a model from scratch and model fine-tuning, our methods achieve comparable and much better results and have better training efficiency on the CIFAR-100 dataset for image classification tasks. This paper proves the great potential of model specialization by knowledge distillation.
Generating Adversarial Attacks in the Latent Space

Adversarial attacks in the input (pixel) space typically incorporate noise margins such as L <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</inf> or L <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</inf> -norm to produce imperceptibly perturbed data that can confound deep learning networks. Such noise margins confine the magnitude of permissible noise. In this work, we propose injecting adversarial perturbations in the latent (feature) space using a generative adversarial network, removing the need for margin-based priors. Experiments on MNIST, CIFAR10, Fashion-MNIST, CIFAR100 and Stanford Dogs datasets support the effectiveness of the proposed method in generating adversarial attacks in the latent space while ensuring a high degree of visual realism with respect to pixel-based adversarial attack methods.
Modulation Recognition with Enhanced Constellation Based on Convolutional Neural Network

Modulation recognition is essential for cognitive radio and non-cooperative communications. Recently, convolutional neural network (CNN) has been applied in modulation recognition and achieved great progresses. However, the current methods using constellation as signal features usually ignore the time correlation information between adjacent samples. Motivated by this deficiency, in this paper, we propose a CNN-based modulation recognition method with enhanced constellation. To eliminate the redundancy in the original constellation, only the information representing whether there exist any constellation points is extracted from the original constellation. Then, the phases carrying sample correlation information are superimposed to synthesize the enhanced constellation as the representative features. The enhanced constellation is further improved by incorporating the amplitude information as pixel values. In addition, to adapt to the varying levels of correlation between and within rows in enhanced constellation, we optimize the existing CNN network with the proper convolution kernel. Simulation results of the proposed method verify its effectiveness and robustness in modulation recognition.
Localized Latent Updates for Fine-Tuning Vision-Language Models

Although massive pre-trained vision-language models like CLIP show impressive generalization capabilities for many tasks, still it often remains necessary to fine-tune them for improved performance on specific datasets. When doing so, it is desirable that updating the model is fast and that the model does not lose its capabilities on data outside of the dataset, as is often the case with classical fine-tuning approaches. In this work we suggest a lightweight adapter that only updates the models predictions close to seen datapoints. We demonstrate the effectiveness and speed of this relatively simple approach in the context of few-shot learning, where our results both on classes seen and unseen during training are comparable with or improve on the state of the art.
MMSE Threshold-based Power Control for Wireless Federated Learning

We put forward a novel minimum mean square error (MMSE) threshold-based power control scheme for wireless federated learning in digital communication systems. The proposed approach uses pulse-amplitude modulation to attain digital over-the-air aggregation of local machine learning models. To reduce the communication cost, we design a novel threshold-based power control strategy that minimizes the mean squared error of parameter estimation and satisfies a constraint on the average power. Simulation results show that the proposed approach is superior to one-bit broadband digital aggregation (OBDA) in terms of the testing accuracy of machine learning and the power consumption of wireless communications. Furthermore, in comparison with broadband analog aggregation (BAA), the proposed approach reduces the power consumption of wireless communications without sacrificing the testing accuracy.
MIMMO: Multi-Input Massive Multi-Output Neural Network

Neural networks (NNs) have achieved superhuman accuracy in multiple tasks, but NNs predictions’ certainty is often debatable, especially if confronted with out of training distribution data. Averaging predictions of an ensemble of NNs can recalibrate the certainty of the predictions, but an ensemble is computationally expensive to deploy in practice. Recently, a new hardware-efficient multi-input multi-output (MIMO) NN was proposed to fit an ensemble of independent NNs into a single NN. In this work, we propose the addition of early-exits to the MIMO architecture with inferred depth-wise weightings to produce multiple predictions for the same input, giving a more diverse ensemble. We denote this combination as MIMMO: a multi-input, massive multi-output NN and we show that it can achieve better accuracy and calibration compared to the MIMO NN, simultaneously fit more NNs and be similarly hardware efficient as MIMO or the early-exit ensemble.
What does a Panoptic Perception Model Learn?

Panoptic driving perception is a promising technology for autonomous driving, which aims to provide a comprehensive understanding of the surrounding environment by processing multiple tasks together. Given the constrained resource on autonomous vehicles, it is essential to develop panoptic perception models with high precision and low resource consumption to assist autonomous driving in real-time. To achieve this goal, it is urgent to understand what a panoptic perception model learns, and get insights to efficiently improve the model design. In this work, we focus on analyzing the explainability of panoptic perception models. Specifically, we use YOLOP as an example, analyze the model explainability and propose several insights for designing the panoptic perception model in the future. To facilitate further research, the source codes are released at https://github.com/lori930/panoptic_visualization.
ODIN: An OmniDirectional INdoor dataset capturing Activities of Daily Living from multiple synchronized modalities

We introduce ODIN (the OmniDirectional INdoor dataset), the first large-scale multi-modal dataset aimed at spurring research using top-view omnidirectional cameras in challenges related to human behaviour understanding. Recorded in real-life indoor environments with varying levels of occlusion, the dataset contains images of participants performing various activities of daily living. Along with omnidirectional images, additional synchronized modalities of data are provided. These include (1) RGB, infrared, and depth images from multiple RGB-D cameras, (2) egocentric videos, (3) physiological signals and accelerometer readings from a smart bracelet, and (4) 3D scans of the recording environments. To the best of our knowledge, ODIN is also the first dataset to provide camera-frame 3D human pose estimates for omnidirectional images, which are obtained using our novel pipeline. The project is open sourced and available at https://odin-dataset.github.io.
Is Multimodal Vision Supervision Beneficial to Language?

Vision (image & video) - Language (VL) pre-training is the recent popular paradigm that achieved state-of-the-art results on multi-modal tasks like image-retrieval, video-retrieval, visual question answering etc. These models are trained in an unsupervised way and greatly benefit from the complementary modality supervision. In this paper, we explore if the language representations trained using vision supervision perform better than vanilla language representations on Natural Language Understanding and commonsense reasoning benchmarks. We experiment with a diverse set of image-text models such as ALBEF, BLIP, METER and video-text models like ALPRO, Frozen in Time, VIOLET. We compare the performance of language representations of stand-alone text encoders of these models to the language representations of text encoders learnt through vision supervision. Our experiments suggest that vanilla language representations show superior performance on most of the tasks. These results shed light on the current drawbacks of the vision-language models. The code is available at https://github.com/avinashsai/MML
Parameter-less Asynchronous Federated Learning under Computation and Communication Constraints

Federated Learning is a fast-developing distributed learning scheme that has promising applications in vertical domains such as industrial automation and connected automated driving. In this paper we address the heterogeneity of the participation of devices in federated learning caused by: i) non-uniform distribution of local data; ii) uneven and varying computational resources across the devices; and iii) dynamic communication link. We propose a quasi-dynamic simulation scheme allowing realistic approximation of these three factors of heterogeneity. Aggregation schemes at the server based on the clients’ work status are implemented. We show that the new asynchronous aggregation algorithm does not require tuning of hyper-parameters such as the round time in synchronous federated learning and the aggregation weight in classic asynchronous aggregation, while providing better or comparable performance in terms of accuracy and convergence speed.
VisiTherS: Visible-thermal infrared stereo disparity estimation of human silhouette

This paper presents a novel approach for visible-thermal infrared stereoscopy, focusing on the estimation of disparities of human silhouettes. Visible-thermal infrared stereo poses several challenges, including occlusions and differently textured matching regions in both spectra. Finding matches between two spectra with varying colors, textures, and shapes adds further complexity to the task. To address the aforementioned challenges, this paper proposes a novel approach where a high-resolution convolutional neural network is used to better capture relationships between the two spectra. To do so, a modified HRNet backbone is used for feature extraction. This HRNet backbone is capable of capturing fine details and textures as it extracts features at multiple scales, thereby enabling the utilization of both local and global information. For matching visible and thermal infrared regions, our method extracts features on each patch using two modified HRNet streams. Features from the two streams are then combined for predicting the disparities by concatenation and correlation. Results on public datasets demonstrate the effectiveness of the proposed approach by improving the results by approximately 18 percentage points on the ≤ 1 pixel error, highlighting its potential for improving accuracy in this task. The code of VisiTherS is available on GitHub at the following link: https://github.com/philippeDG/VisiTherS.
ETAD: Training Action Detection End to End on a Laptop

Temporal action detection (TAD) with end-to-end training often suffers from the pain of huge demand for computing resources due to long video duration. In this work, we propose an efficient temporal action detector (ETAD) that can train directly from video frames with extremely low GPU memory consumption. Our main idea is to minimize and balance the heavy computation among features and gradients in each training iteration. We propose to sequentially forward the snippet frame through the video encoder, and backward only a small necessary portion of gradients to update the encoder. To further alleviate the computational redundancy in training, we propose to dynamically sample only a small subset of proposals during training. Various sampling strategies and ratios are studied for both the encoder and detector. ETAD achieves state-of-the-art performance on TAD benchmarks with remarkable efficiency. On ActivityNet-1.3, training ETAD in 18 hours can reach 38.25% average mAP with only 1.3 GB memory per video under end-to-end training. Code is available at https://github.com/sming256/ETAD.
Fuzzy Logic-based Adaptive Multimedia Streaming for Internet of Vehicles

Multimedia streaming for the Internet of Vehicles has the potential to enhance road safety and transport efficiency for autonomous vehicles, and the in-car experience for passengers. MPEG-Dynamic Adaptive Streaming over HTTP (MPEG-DASH) framework has been widely deployed to optimize video streaming with respect to end-user Quality of Experience (QoE). However, existing heuristic-based, reinforcement learning-based, or fuzzy-based adaptive algorithms, which use complex control laws and decision-making processes, are not well-suited to handle the non-stationary nature of road traffic environments. Consequently, these solutions often struggle to deliver optimal performance across multiple QoE objectives and under diverse network conditions. In this paper, we introduce FLAME, a novel adaptive multimedia streaming solution based on advanced fuzzy logic. FLAME incorporates interactive membership functions and fuzzy rules in its two variants, FLAME7 and FLAME5, resulting in reduced model complexities and training overheads. FLAME is adaptable to diverse video client settings and QoE goals. Our trace-driven experimental results demonstrate that FLAME solutions offer an improved uninterrupted streaming experience for connected vehicles. On average, FLAME outperforms other state-of-the-art solutions such as PENSIEVE, BOLA, FESTIVE, BBA, and ELASTIC by achieving 11.7% higher QoE.
Disentangling Local and Global Information for Light Field Depth Estimation

Accurate depth estimation from light field images is essential for various applications. Deep learning-based techniques have shown great potential in addressing this problem while still face challenges such as sensitivity to occlusions and difficulties in handling untextured areas. To overcome these limitations, we propose a novel approach that utilizes both local and global features in the cost volume for depth estimation. Specifically, our hybrid cost volume network consists of two complementary sub-modules: a 2D ContextNet for global context information and a matching cost volume for local feature information. We also introduce an occlusion-aware loss that accounts for occlusion areas to improve depth estimation quality. We demonstrate the effectiveness of our approach on the UrbanLF and HCInew datasets, showing significant improvements over existing methods, especially in occluded and untextured regions. Our method disentangles local feature and global semantic information explicitly, reducing the occlusion and untextured area reconstruction error and improving the accuracy of depth estimation.
Localized Shortcut Removal

Machine learning is a data-driven field, and the quality of the underlying datasets plays a crucial role in learning success. However, high performance on held-out test data does not necessarily indicate that a model generalizes or learns anything meaningful. This is often due to the existence of machine learning shortcuts - features in the data that are predictive but unrelated to the problem at hand. To address this issue for datasets where the shortcuts are smaller and more localized than true features, we propose a novel approach to detect and remove them. We use an adversarially trained lens to detect and eliminate highly predictive but semantically unconnected clues in images. In our experiments on both synthetic and real-world data, we show that our proposed approach reliably identifies and neutralizes such shortcuts without causing degradation of model performance on clean data. We believe that our approach can lead to more meaningful and generalizable machine learning models, especially in scenarios where the quality of the underlying datasets is crucial.
Moving Object Detection Based on Clustering and Event-Based Camera

The Moving object detection as a problem in computer vision, has the attention of researchers for its need of different applications. Event cameras are used to support moving object detection missions depending on the event camera efficiently capturing the events of moving objects compared to classic cameras. In this paper, we proposed a model that used a clustering algorithm as a machine learning approach to help detect moving objects that were captured using an event camera. The proposed model used Hierarchical clustering algorithm called “Agglomerative” and compared to partitioning and density-based clustering algorithms for the mission of detecting moving objects. Moreover, the model shows better results compared to others in previous studies with “92.07” F1-score as a performance measure.
Vision DiffMask: Faithful Interpretation of Vision Transformers with Differentiable Patch Masking

The lack of interpretability of the Vision Transformer may hinder its use in critical real-world applications despite its effectiveness. To overcome this issue, we propose a post-hoc interpretability method called Vision DiffMask, which uses the activations of the model’s hidden layers to predict the relevant parts of the input that contribute to its final predictions. Our approach uses a gating mechanism to identify the minimal subset of the original input that preserves the predicted distribution over classes. We demonstrate the faithfulness of our method, by introducing a faithfulness task, and comparing it to other state-of-the-art attribution methods on CIFAR-10 and ImageNet-1K, achieving compelling results. To aid reproducibility and further extension of our work, we open source our implementation here.
DNA: Deformable Neural Articulations Network for Template-free Dynamic 3D Human Reconstruction from Monocular RGB-D Video

In this paper, we present a novel Deformable Neural Articulations Network (DNA-Net), which is a template-free learning-based method for dynamic 3D human reconstruction from a single RGB-D sequence. Our proposed DNA-Net includes a Neural Articulation Prediction Network (NAP-Net), which is capable of representing non-rigid motions of a human by learning to predict a set of articulated bones to follow movements of the human in the input sequence. Moreover, DNA-Net also include Signed Distance Field Network (SDF-Net) and Apearance Network (Color-Net), which take advantage of the powerful neural implicit functions in modeling 3D geometries and appearance. Finally, to avoid the reliance on external optical flow estimators to obtain deformation cues like previous related works, we propose a novel training loss, namely Easy-to-Hard Geometric-based, which is a simple strategy that inherits the merits of Chamfer distance to achieve good deformation guidance while still avoiding its limitation of local mismatches sensitivity. DNA-Net is trained end-to-end in a self-supervised manner directly on the input sequence to obtain 3D reconstructions of the input objects. Quantitative results on videos of DeepDeform dataset show that DNA-Net outperforms related state-of-the-art methods with an adequate gaps, qualitative results additionally prove that our method can reconstruct human shapes with high fidelity and details.
A Smart Interactive Behavioral Chatbot: A Theoretical Prototype

In recent years, significant advancements have been made in speech and language understanding research, with a particular focus on the development of conversational dialog systems between humans and machines. These systems, also recognized as chatbots, interactive conversational agents, or dialog systems, have broad applications, including technical support services, language entertainment, and learning. The emergence of deep neural networks has surged research efforts aimed at building data-driven dialog models. In this study, a novel approach is introduced that incorporates emotions extracted from various sources to design a smart and interactive chatbot.
Camera based Eye State Estimation for ICU Patients: A Pilot Clinical Study

In the Intensive Care Unit (ICU), the awakening of patients from comas is indicative of recovery. This article investigates the feasibility of using conventional and deep learning-based methods for eye state estimation based on videos recorded by a CCTV-camera installed in the ICU. For handcrafted feature-based methods, geometric, HOG and RGB features are combined as the input of the SVM classifier to classify the eye state as open and closed. For deep learning-based methods, the eye and face images were used as joint input for classification. The clinical trial involved 48 critically-ill ICU patients. The results show that the HOG-RGB based method achieved an accuracy of 91.39%, while the deep learning-based method achieved an accuracy of 89.35%. These findings highlight the chances of using CCTV cameras to estimate the eye state of ICU patients, which can be a useful mean to provide information regarding the patient consciousness for clinicians to assess their recovery.