Reviewerly/exHarmony huggingface dataset:
TSOSVNet: Teacher-student collaborative knowledge distillation for Online Signature Verification

Online signature verification (OSV) is a standardized personal authentication scheme with wide social acceptance in critical real-time applications include access control, m-commerce, etc. Even though the current advances in Deep learning (DL) technologies catalysed state-of-the-art frameworks for challenging domains like computer vision, speech recognition, etc., the DL-based frameworks are voluminous with huge trainable parameters and are hard to deploy in real-time systems demanding faster inference. To adopt DL into OSV for improved performance, we propose an OSV framework made up of teacher-student collaborative knowledge distillation (TSKD) technique. A heavy Transformer based teacher is trained first and the teacher knowledge is distilled into a very lightweight Convolutional Neural Network (CNN) based student. A well trained teacher network results in an efficient deep representative feature learning by the student and results in a performance improvement. In a thorough set of experiments with three popular and standard datasets, i.e., the MCYT-100, SUSIG, and SVC, TSOSVNet framework, with a CNN based student model requiring only 3266 trainable parameters results in an EER of 12.42% compared to the recent SOTA 13.38% by a model with 206277 parameters in skilled_01 category of MCYT-100 dataset. In comparison to cutting-edge CNN-based OSV models, the proposed TSOSVNet produced a state-of-the-art EER in the most of the test categories with an average of 90% lesser trainable parameters.
Real-Time Optimisation-Based Path Planning for Visually Impaired People in Dynamic Environments

Most existing outdoor assistive mobility solutions notify Visually Impaired People (VIP) about potential collisions but fail to provide Optimal Local Collision-Free Path Planning (OLCFPP) to enable the VIP to get out of the way effectively. In this paper, we propose MinD, the first VIP OLCFPP scheme that notifies the VIP of the shortest path required to avoid Critical Moving Objects (CMOs), like cars, motorcycles, etc. This simultaneously accounts for the VIP’s mobility constraints, the different CMO types and movement patterns, and predicted collision times, conducting a safety prediction trajectory analysis of the optimal path for the VIP to move in. We implement a real-world prototype to conduct extensive outdoor experiments that record the aforementioned parameters, and this populates our simulations for evaluation against the state-of-the-art. Experimental results demonstrate that MinD outperforms the Artificial Potential Field (APF) approach in effectively planning a short collision-free route, requiring only 1.69m of movement on average, shorter than APF by 90.23%, with a 0% collision rate; adapting to the VIP’s mobility limitations and provides a high safe time separation (> 5.35s on average compared to APF). MinD also shows near real-time performance, with decisions taking only 0.04s processing time on a standard off-the-shelf laptop.
On Moving Object Segmentation from Monocular Video with Transformers

Moving object detection and segmentation from a single moving camera is a challenging task, requiring an understanding of recognition, motion and 3D geometry. Combining both recognition and reconstruction boils down to a fusion problem, where appearance and motion features need to be combined for classification and segmentation.In this paper, we present a novel fusion architecture for monocular motion segmentation - M <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> Former, which leverages the strong performance of transformers for segmentation and multi-modal fusion. As reconstructing motion from monocular video is ill-posed, we systematically analyze different 2D and 3D motion representations for this problem and their importance for segmentation performance. Finally, we analyze the effect of training data and show that diverse datasets are required to achieve SotA performance on Kitti and Davis. Code will be released upon publication.
Post Training Mixed Precision Quantization of Neural Networks using First-Order Information

Quantization is an efficient way of downsizing both memory footprints and inference time of large size Deep Neural Networks (DNNs) and makes their application feasible on resource-constrained devices. However, quantizing all layers uniformly with ultra-low precision bits results in significant degradation in performance. A promising approach to address this problem is mixed-precision quantization where higher bit precisions are assigned to layers that are more sensitive. In this study, we introduce the method that uses first-order information (i.e. gradient) only for determining the neural network layers’ sensitivity for mixed-precision quantization and shows that the proposed method is equally effective in performance and better in computation complexity with its counterpart methods which use second order information (i.e. hessian). Finally, we formulate the mixed precision problem as an Integer linear programming problem which uses proposed sensitivity metric and allocate the number of bits for each layer efficiently for a given model size. Furthermore, we only use post training quantization techniques to achieve the state of the art results in comparison to the popular methods for mixed precision quantization which fine-tunes the model with large training data. Extensive experiments conducted on benchmark vision neural network architectures using ImageNet dataset demonstrates the superiority over existing mixed-precision approaches. Our proposed method achieves better or comparable results for ResNet18 (0.65% accuracy-drop, for 8× weight compression), ResNet50 (0.69% accuracy-drop, for 8× weight compression), MobileNet-V2 (0.49% accuracy-drop, for 8× weight compression) and Inception-V3 (1.30% accuracy-drop, for 8× weight compression), compared to other state-of-the-art methods which requires retraining or uses hessian as a sensitivity metric for mixed precision quantization.
Extending TrOCR for Text Localization-Free OCR of Full-Page Scanned Receipt Images

Digitization of scanned receipts aims to extract text from receipt images and save it into structured documents. This is usually split into two sub-tasks: text localization and optical character recognition (OCR). Most existing OCR models only focus on the cropped text instance images, which require the bounding box information provided by a text region detection model. Introducing an additional detector to identify the text instance images in advance adds complexity, however instance-level OCR models have very low accuracy when processing the whole image for the document-level OCR, such as receipt images containing multiple text lines arranged in various layouts. To this end, we propose a localization-free document-level OCR model for transcribing all the characters in a receipt image into an ordered sequence end-to-end. Specifically, we finetune the pretrained instance-level model TrOCR with randomly cropped image chunks, and gradually increase the image chunk size to generalize the recognition ability from instance images to full-page images. In our experiments on the SROIE receipt OCR dataset, the model finetuned with our strategy achieved 64.4 F1-score and a 22.8% character error rate (CER), respectively, which outperforms the baseline results with 48.5 F1-score and 50.6% CER. The best model, which splits the full image into 15 equally sized chunks, gives 87.8 F1-score and 4.98% CER with minimal additional pre or post-processing of the output. Moreover, the characters in the generated document-level sequences are arranged in the reading order, which is practical for real-world applications.
Domain Adversarial Learning Towards Underwater Image Enhancement

Underwater images are degraded due to the absorption and scattering of light inside the water. The underwater degradation causes the loss of information in terms of texture, style, color, and minute detail of edges and hence the degraded images are not useful in many higher-order applications. Several deep learning techniques are explored by researchers across the globe for the same. Further, deep learning networks learn the distribution of degraded training samples and fail when there is a deviation due to a change in water type. This paper proposes an encoder-decoder network that preserves the image content, texture, and style while maintaining overall global similarity by capturing the inherent distribution of the training samples. To overcome the deviation due to a change in water type, a classifier network is induced in the latent space of encoder-decoder architecture. The classifier loss and adversarial loss in the classifier network ensure the learning across domains and avoid setting priors on captured distribution. Hence, the proposed model is robust against the change of water type and can be deployed in real-life without retraining. To train the model, we use attenuation coefficients of underwater environments at different depths to recreate 5430 paired underwater images from the Underwater Image Enhancement Benchmark (UIEB) dataset with six distinct types of water. Our proposed model enhanced the degraded images among different degradation levels due to depth and water type variations. The proposed model is evaluated on UIEB and EUVP benchmark databases. The performance of the proposed model is verified against twenty-two state-of-the-art methods in terms of underwater reference and no-reference image quality assessment metrics.
MMTF: Multi-Modal Temporal Fusion for Commonsense Video Question Answering

Video question answering is a challenging task that requires understanding the video and question in the same context. This becomes even harder when the questions involve reasoning, such as predicting future events or explaining counterfactual events, because they need knowledge not explicitly shown. Existing methods use coarse-grained fusion of video and language features, ignoring temporal information. To address this, we propose a novel vision-text fusion module that learns the temporal context of the video and question. Our module expands question tokens along the video’s temporal axis and fuses them with video features to generate new representations with local and global context. We evaluated our method on four VideoQA datasets, including MSVD-QA, NExT-QA, Causal-VidQA, and AGQA-2.0.
Kinship Representation Learning with Face Componential Relation

Kinship recognition aims to determine whether the subjects in two facial images are kin or non-kin, which is an emerging and challenging problem. However, most previous methods focus on heuristic designs without considering the spatial correlation between face images. In this paper, we aim to learn discriminative kinship representations embedded with the relation information between face components. To achieve this goal, we propose the Face Componential Relation Network (FaCoRNet), which learns the relationship between face components among images with a cross-attention mechanism, to automatically learn the important facial regions for kinship recognition. Moreover, we propose Relation-Guided Contrastive Learning, which adapts the loss function by the guidance from cross-attention to learn more discriminative feature representations. The proposed FaCoRNet outperforms previous state-of-the-art methods by large margins for experiments on multiple public kinship recognition benchmarks. Our code is available at https://github.com/wtnthu/FaCoR.
Detection and Classification of Smart Jamming in Wi-Fi Networks Using Machine Learning

Smart adversaries can exploit the publicly known frame structure of OFDM-based Wi-Fi protocols to disrupt communications by strategically jamming specific time samples or specific subcarriers. Such attacks are very difficult to detect by traditional techniques like spectral analysis and signal strength indicators. Machine learning (ML) based methods have been proposed to tackle this problem. However, existing ML methods are computationally intensive and perform well only at low signal-to-jamming power ratios (SJRs). In this paper, we propose a computationally efficient deep convolutional neural network (DCNN) consisting of only four convolution layers to detect and classify several smart jamming attacks in Wi-Fi networks. To deal with the time-frequency selectivity of smart jamming, we apply the continuous wavelet transform (CWT) to partially overlapped segments of the received I/Q samples to extract features. The scalogram of the CWT is used as input to the DCNN. We focus on three smart jamming attacks: preamble jamming, pilot jamming, and interleaving jamming. These attacks share similar characteristics, making their differentiation particularly challenging. Our proposed classifier achieves high accuracy in detecting and classifying these jamming attacks across a range of SJRs, from −6 dB to 15 dB, with an overall classification accuracy of 98%. Even at high SJR levels, the accuracy remains high at around 90%. We also train the classifier to be robust against partial preamble jamming and pilot jamming, The resulting classification accuracy is over 90% at SJRs up to 12 dB. Additionally, we compare our classifier with one that uses the spectrogram (short-time Fourier transform) as input to the DCNN, and demonstrate the superior performance of the proposed scalogram-based classifier, particularly in the high SJR regime.
Fine-Tuned but Zero-Shot 3D Shape Sketch View Similarity and Retrieval

Recently, encoders like ViT (vision transformer) and ResNet have been trained on vast datasets and utilized as perceptual metrics for comparing sketches and images, as well as multi-domain encoders in a zero-shot setting. However, there has been limited effort to quantify the granularity of these encoders. Our work addresses this gap by focusing on multi-modal 2D projections of individual 3D instances. This task holds crucial implications for retrieval and sketch-based modeling. We show that in a zero-shot setting (without retraining on a specific shape category or sketch style), the more abstract the sketch, the higher the likelihood of incorrect image matches. Even within the same sketch domain, sketches of the same object drawn in different styles, for example by distinct individuals, might not be accurately matched. One of the key findings of our research is that meticulous fine-tuning on one class of 3D shapes leads to improved performance on other shape classes (fine-tuned but zero-shot), reaching or surpassing the accuracy of supervised methods. We compare and discuss several fine-tuning strategies. Additionally, we delve deeply into how the scale of an object in a sketch influences the similarity of features at different network layers, helping us identify which network layers provide the most accurate matching. Significantly, we discover that ViT and ResNet perform best when dealing with similar object scales. We believe that our work will have a significant impact on research in the sketch domain, providing insights and guidance on how to adopt large pre-trained models as perceptual losses. Our code is available at https://github.com/GBerardi/ZS-SBSR.
MINDFL: Mitigating the Impact of Imbalanced and Noisy-labeled Data in Federated Learning with Quality and Fairness-Aware Client Selection

Federated Learning (FL) has been acclaimed for enhancing machine learning privacy, but it also faces criticism for its high communication overhead due to the need for a large number of interactions between participants and the server. The conventional approach of randomly selecting clients for FL participation can reduce communication costs, but it often slows down the convergence of the global model and lowering its overall quality. Although various advanced methods have been proposed for client selection, focusing on the selection of the most informative local models, these solutions frequently assume clean training data. They often overlook the impact of noisy-labeled and imbalanced local data among clients, which can significantly hinder training efficiency.To address these challenges, we present MINDFL, a client selection mechanism that prioritizes quality and fairness considerations. MINDFL tackles the problem of utility divergence among local models caused by noisy-labeled and imbalanced data. It also ensures fairness in the selection process. In particular, we introduce a novel metric called Quality-of-Model (QoM), which assesses the contribution of local models to the aggregated global model. MINDFL selects clients with high QoM, representing the most informative model updates, in each FL round to maximize learning efficiency. Further, to enhance participant diversity while maintaining clients’ model quality, we utilize a sortition-inspired selection method to choose clients with high QoM randomly. Our comprehensive experimental evaluations demonstrate the effectiveness of MINDFL in improving learning speed and reducing communication overhead.
Quality Assurance of Facial Recognition Software for Arabic Social Media Celebrities

In this study, we developed a system to test the quality assurance of face detection for Arabic celebrities in social media using the Haar detection technique with the OpenCV Python package. In our developed system, we used 64 images to determine and find the quality measures, like the error rate of recognizing faces in the images, by employing various testing methods. Our work resulted in creating a data set called Yasmin Nadia Arabc Social Media Images containing faces for Arabic social media celebrations, which is available for public use on the websites Kaggle and GitHub. The developed system can identify faces with an accuracy rate of more than 90% in most cases.
No Data Augmentation? Alternative Regularizations for Effective Training on Small Datasets

Solving image classification tasks given small training datasets remains an open challenge for modern computer vision. Aggressive data augmentation and generative models are among the most straightforward approaches to overcoming the lack of data. However, the first fails to be agnostic to varying image domains, while the latter requires additional compute and careful design.In this work, we study alternative regularization strategies to push the limits of supervised learning on small image classification datasets. In particular, along with the model size and training schedule scaling, we employ a heuristic to select (semi) optimal learning rate and weight decay couples via the norm of model parameters. By training on only 1% of the original CIFAR-10 training set (i.e., 50 images per class) and testing on ciFAIR-10, a variant of the original CIFAR without duplicated images, we reach a test accuracy of 66.5%, on par with the best state-of-the-art methods.
ProVLA: Compositional Image Search with Progressive Vision-Language Alignment and Multimodal Fusion

Traditional image-to-image and text-to-image search struggle with comprehending complex user intentions, particularly in fashion e-commerce, where users search for similar products with text modifications to a reference image. This paper introduces Progressive Vision-Language Alignment and Multimodal Fusion (ProVLA), a novel approach which utilizes a transformer-based vision and language model to generate multimodal embeddings. Our method involves a two-step learning process and a cross-attention-based fusion encoder to facilitate robust information fusion, and a momentum queue-based hard negative mining mechanism to handle noisy training data. Extensive evaluations on the Fashion 200k and Shoes benchmark datasets demonstrate that our model outperforms state-of-the-art methods.
Graph Machine Learning based Cyber Attack Detection for Mobile Tactical Networks

First responders and other tactical teams rely on mobile tactical networks to coordinate and accomplish emergent time-critical tasks. The information exchanged through these networks is vulnerable to various strategic cyber network attacks. Detecting and mitigating them is a challenging problem due to the volatile and mobile nature of an ad hoc environment. This paper proposes MalCAD, a graph machine learning-based framework for detecting cyber attacks in mobile tactical software-defined networks. Mal-CAD operates based on observing connectivity features among various nodes obtained using graph theory, instead of collecting information at each node. The MalCAD framework is based on the XGBOOST classification algorithm and is evaluated for lost versus wasted connectivity and random versus targeted cyber attacks. Results show that, while the initial cyber attacks create a loss of 30%–60% throughput, MalCAD results in a gain of average throughput by 25%–50%, demonstrating successful attack mitigation.
Generalizing Machine Learning Models for Zero-Day Encrypted Messaging Applications

Access to fully-labeled mobile network traffic data is challenging to come by and harder still to keep up-to-date. Utilizing positive and unlabeled (PU) learning allows us to focus on labeling only a small subset of classes while using large amounts of unlabeled data. Furthermore, the positive labeled data for these few essential classes can be combined to represent a shared genre and then used to classify other applications belonging to the same genre, even when no training data is present for those new applications. We show that PU learning models, trained on several of the most popular encrypted messaging applications on the Android OS, can effectively detect zero-day applications, i.e., those that were not present during the training of the models, with a combined average accuracy of almost 60%. In contrast, random guessing yields less than 1%.
Adaptive Self-Training for Object Detection

Deep learning has emerged as an effective solution for solving the task of object detection in images but at the cost of requiring large labeled datasets. To mitigate this cost, semi-supervised object detection methods, which consist in leveraging abundant unlabeled data, have been proposed and have already shown impressive results. These methods however often rely on a thresholding mechanism to allocate pseudo-labels. This threshold value is usually determined empirically for a dataset, which is time consuming and requires a new and costly parameter search when the domain changes. In this work, we introduce a new teacher-student method, named Adaptive Self-Training for Object Detection (ASTOD), which is simple and effective. ASTOD selects pseudo-labels adaptively by examining the score histogram. In addition, we also introduce the idea to systematically refine the student, after training, with the labeled data only to improve its performance. While the teacher and the student of ASTOD are trained separately, in the end, the refined student replaces the teacher in an iterative fashion.Our experiments show that, on the MS-COCO dataset, our method consistently outperforms other adaptive state-of-the-art methods, and performs equally with respect to methods that require a manual parameter sweep search, and are therefore of limited use in practice. Additional experiments with respect to a supervised baseline on the DIOR dataset containing satellite images lead to similar conclusions, and prove that it is possible to adapt the score threshold automatically in self-training, regardless of the data distribution. The code is available at https://github.com/rvandeghen/ASTOD.