Reviewerly/exHarmony huggingface dataset:
Employing An Efficient Technique with Deep Neural Network for Speaker Identification

A speaker identification system based on Mel-frequency cepstral coefficients (MFCC) and a convolutional neural network (CNN) is proposed in this paper. This paper aims to identify speakers using very short time durations (0.5sec and 1sec) of speech utterances. The system proposed consists of three main steps: preprocessing, feature extraction, and classification. Several techniques are applied to the speech data in the preprocessing step to obtain a better speech representation. Mel-frequency cepstral coefficients (MFCC) are employed in the feature extraction step to extract the intrinsic features of the preprocessed speech and reduce dimension. Finally, a convolutional neural network (CNN) is used for the classification task. Different databases are employed to evaluate the proposed system: SALU-AC, ELSDSR, RA VDESS, and TIMIT. The results obtained by the proposed system are 95.74%, 99.01%, 87.30%, and 88.03% for 1 sec of the SALU-AC, ELSDSR, RAVDESS, and TIMIT databases, respectively. For 0.5sec, the SALU-AC, ELSDSR, RAVDESS, and TIMIT databases obtained 92.57%, 97.45%, 85.26%, and 74.82%, respectively. The system's high performance is due to the combinations of the powerful techniques used in the preprocessing steps, the MFCC algorithm that leads to the substantial features extracted from the speech signal, and the exhaustion search to choose the adequate parameters for the CNN structure.
Arabic Document Clustering: A Survey

Arabic language significance is continuously growing due to economic advancements and the textual content of the Internet nowadays. As Arabic documents become more widely available on the Internet, Arabic document clustering becomes an essential topic in information retrieval and mining. Electronic Arabic documents are growing daily, making a clustering procedure based on humans impractical. Effective and efficient electronic search algorithms are necessary to make clustering easier and more accurate information retrieval possible. The purpose of this work is to provide a thorough examination of the algorithms and strategies utilized for clustering digitized Arabic texts. The paper presents a critical and constructive analysis of existing published literature in Arabic document clustering through summary, classification, analysis, and comparison. It collects the aims of surveys, such as literature organization and evaluation, identifying patterns and trends in the literature, synthesizing literature, identifying research gaps, and recommending new research areas.
Class-Incremental Learning of Plant and Disease Detection: Growing Branches with Knowledge Distillation

This paper investigates the problem of class-incremental object detection for agricultural applications where a model needs to learn new plant species and diseases incrementally without forgetting the previously learned ones. We adapt two public datasets to include new categories over time, simulating a more realistic and dynamic scenario. We then compare three class-incremental learning methods that leverage different forms of knowledge distillation to mitigate catastrophic forgetting. Our experiments show that all three methods suffer from catastrophic forgetting, but the Dynamic Y-KD approach, which additionally uses a dynamic architecture that grows new branches to learn new tasks, outperforms ILOD and Faster-ILOD in most settings both on new and old classes.These results highlight the challenges and opportunities of continual object detection for agricultural applications. In particular, we hypothesize that the large intra-class and small inter-class variability that is typical of plant images exacerbate the difficulty of learning new categories without interfering with previous knowledge. We publicly release our code to encourage future work. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
InFusion: Inject and Attention Fusion for Multi Concept Zero-Shot Text-based Video Editing

Large text-to-image diffusion models have achieved remarkable success in generating diverse, high-quality images. Additionally, these models have been successfully leveraged to edit input images by just changing the text prompt. But when these models are applied to videos, the main challenge is to ensure temporal consistency and coherence across frames. In this paper, we propose InFusion, a framework for zero-shot text-based video editing leveraging large pre-trained image diffusion models. Our framework specifically supports editing of multiple concepts with pixel-level control over diverse concepts mentioned in the editing prompt. Specifically, we inject the difference in features obtained with source and edit prompts from U-Net residual blocks of decoder layers. When these are combined with injected attention features, it becomes feasible to query the source contents and scale edited concepts along with the injection of unedited parts. The editing is further controlled in a fine-grained manner with mask extraction and attention fusion, which cut the edited part from the source and paste it into the denoising pipeline for the editing prompt. Our framework is a low-cost alternative to one-shot tuned models for editing since it does not require training. We demonstrated complex concept editing with a generalised image model (Stable Diffusion v1.5) using LoRA. Adaptation is compatible with all the existing image diffusion techniques. Extensive experimental results demonstrate the effectiveness of existing methods in rendering high-quality and temporally consistent videos.
On the unreasonable vulnerability of transformers for image restoration – and an easy fix

Following their success in visual recognition tasks, Vision Transformers(ViTs) are being increasingly employed for image restoration. As a few recent works claim that ViTs for image classification also have better robustness properties, we investigate whether the improved adversarial robustness of ViTs extends to image restoration. We consider the recently proposed Restormer model, as well as NAFNet and the "Baseline network" which are both simplified versions of a Restormer. We use Projected Gradient Descent (PGD) and CosPGD for our robustness evaluation. Our experiments are performed on real-world images from the GoPro dataset for image deblurring. Our analysis indicates that contrary to as advocated by ViTs in image classification works, these models are highly susceptible to adversarial attacks. We attempt to find an easy fix and improve their robustness through adversarial training. While this yields a significant increase in robustness for Restormer, results on other networks are less promising. Interestingly, we find that the design choices in NAFNet and Baselines, which were based on iid performance, and not on robust generalization, seem to be at odds with the model robustness.
On the Interplay of Convolutional Padding and Adversarial Robustness

It is common practice to apply padding prior to convolution operations to preserve the resolution of feature-maps in Convolutional Neural Networks (CNN). While many alternatives exist, this is often achieved by adding a border of zeros around the inputs. In this work, we show that adversarial attacks often result in perturbation anomalies at the image boundaries, which are the areas where padding is used. Consequently, we aim to provide an analysis of the interplay between padding and adversarial attacks and seek an answer to the question of how different padding modes (or their absence) affect adversarial robustness in various scenarios.
Accelerating Deep Neural Networks via Semi-Structured Activation Sparsity

The demand for efficient processing of deep neural networks (DNNs) on embedded devices is a significant challenge limiting their deployment. Exploiting sparsity in the network’s feature maps is one of the ways to reduce its inference latency. It is known that unstructured sparsity results in lower accuracy degradation with respect to structured sparsity but the former needs extensive inference engine changes to get latency benefits. To tackle this challenge, we propose a solution to induce semi-structured activation sparsity exploitable through minor runtime modifications. To attain high speedup levels at inference time, we design a sparse training procedure with awareness of the final position of the activations while computing the General Matrix Multiplication (GEMM). We extensively evaluate the proposed solution across various models for image classification and object detection tasks. Remarkably, our approach yields a speed improvement of 1.25× with a minimal accuracy drop of 1.1% for the ResNet18 model on the ImageNet dataset. Furthermore, when combined with a state-of-the-art structured pruning method, the resulting models provide a good latency-accuracy trade-off, outperforming models that solely employ structured pruning techniques. The code is available at https://github.com/Deeplite/activ-sparse.
Softmax Bias Correction for Quantized Generative Models

Post-training quantization (PTQ) is the go-to compression technique for large generative models, such as stable diffusion or large language models. PTQ methods commonly keep the softmax activation in higher precision as it has been shown to be very sensitive to quantization noise. However, this can lead to a significant runtime and power overhead during inference on resource-constraint edge devices. In this work, we investigate the source of the softmax sensitivity to quantization and show that the quantization operation leads to a large bias in the softmax output, causing accuracy degradation. To overcome this issue, we propose an offline bias correction technique that improves the quatizability of softmax without additional compute during deployment, as it can be readily absorbed into the quantization parameters. We demonstrate the effectiveness of our method on stable diffusion v1.5 and 125M-size OPT language model, achieving significant accuracy improvement for 8-bit quantized softmax.
Efficient, Self-Supervised Human Pose Estimation with Inductive Prior Tuning

The goal of 2D human pose estimation (HPE) is to localize anatomical landmarks, given an image of a person in a pose. SOTA techniques make use of thousands of labeled figures (finetuning transformers or training deep CNNs), acquired using labor-intensive crowdsourcing. On the other hand, self-supervised methods re-frame the HPE task as a reconstruction problem, enabling them to leverage the vast amount of unlabeled visual data, though at the present cost of accuracy. In this work, we explore ways to improve self-supervised HPE. We (1) analyze the relationship between reconstruction quality and pose estimation accuracy, (2) develop a model pipeline that outperforms the baseline which inspired our work, using less than one-third the amount of training data, and (3) offer a new metric suitable for self-supervised settings that measures the consistency of predicted body part length proportions. We show that a combination of well-engineered reconstruction losses and inductive priors can help coordinate pose learning alongside reconstruction in a self-supervised paradigm.
Continuous Hand Gesture Recognition for Human-Robot Collaborative Assembly

In this work, we present a framework for dynamic hand gesture recognition on RGB images acquired by an overhead camera. The recognition is realized for Methods Time Measurement-based planning of human-robot collaborative workspace. The 3D hand posture is estimated by MediaPipe. The recognition is done by a neural network in which a layer-wise feature combination takes place. We combine features extracted by basic blocks of Spatio-Temporal Adaptive Graph Convolutional Neural Network and by basic spatio-temporal self-attention blocks. We recorded and manually annotated 12 videos consisting of 54,659 RGB images with five basic motion sequences: grasp, move, position, release, and reach. We demonstrate experimentally that results of our networks are superior to results achieved by RNNs, ST-GCN, ST-AGCN, and CTR-GCN networks.
Improving Replay Sample Selection and Storage for Less Forgetting in Continual Learning

Continual learning seeks to enable deep learners to train on a series of tasks of unknown length without suffering from the catastrophic forgetting of previous tasks. One effective solution is replay, which involves storing few previous experiences in memory and replaying them when learning the current task. However, there is still room for improvement when it comes to selecting the most informative samples for storage and determining the optimal number of samples to be stored. This study aims to address these issues with a novel comparison of the commonly used reservoir sampling to various alternative population strategies and providing a novel detailed analysis of how to find the optimal number of stored samples.
Experience Replay as an Effective Strategy for Optimizing Decentralized Federated Learning

Federated and continual learning are training paradigms addressing data distribution shift in space and time. More specifically, federated learning tackles non-i.i.d data in space as information is distributed in multiple nodes, while continual learning faces with temporal aspect of training as it deals with continuous streams of data. Distribution shifts over space and time is what it happens in real federated learning scenarios that show multiple challenges. First, the federated model needs to learn sequentially while retaining knowledge from the past training rounds. Second, the model has also to deal with concept drift from the distributed data distributions. To address these complexities, we attempt to combine continual and federated learning strategies by proposing a solution inspired by experience replay and generative adversarial concepts for supporting decentralized distributed training. In particular, our approach relies on using limited memory buffers of synthetic privacy-preserving samples and interleaving training on local data and on buffer data. By translating the CL formulation into the task of integrating distributed knowledge with local knowledge, our method enables models to effectively integrate learned representation from local nodes, providing models the capability to generalize across multiple datasets.We test our integrated strategy on two realistic medical image analysis tasks — tuberculosis and melanoma classification — using multiple datasets in order to simulate realistic non-i.i.d. medical data scenarios. Results show that our approach achieves performance comparable to standard (non-federated) learning and significantly outperforms state-of-the-art federated methods in their centralized (thus, more favourable) formulation.
Temporal DINO: A Self-supervised Video Strategy to Enhance Action Prediction

The emerging field of action prediction - the task of forecasting action in a video sequence - plays a vital role in various computer vision applications such as autonomous driving, activity analysis and human-computer interaction. Despite significant advancements, accurately predicting future actions remains a challenging problem due to high dimensionality, complex dynamics and uncertainties inherent in video data. Traditional supervised approaches require large amounts of labelled data, which is expensive and time-consuming to obtain. This paper introduces a novel self-supervised video strategy for enhancing action prediction inspired by DINO (self-distillation with no labels). The approach, named Temporal-DINO, employs two models; a ‘student’ processing past frames; and a ‘teacher’ processing both past and future frames, enabling a broader temporal context. During training, the teacher guides the student to learn future context by only observing past frames. The strategy is evaluated on ROAD dataset for the action prediction downstream task using 3D-ResNet, Transformer, and LSTM architectures. The experimental results showcase significant improvements in prediction performance across these architectures, with our method achieving an average enhancement of 9.9% Precision Points (PP), which highlights its effectiveness in enhancing the backbones’ capabilities of capturing long-term dependencies. Furthermore, our approach demonstrates efficiency in terms of the pretraining dataset size and the number of epochs required. This method overcomes limitations present in other approaches, including the consideration of various backbone architectures, addressing multiple prediction horizons, reducing reliance on hand-crafted augmentations, and streamlining the pretraining process into a single stage. These findings highlight the potential of our approach in diverse video-based tasks such as activity recognition, motion planning, and scene understanding. Code can be found at https://github.com/IzzeddinTeeti/ssl_pred.
Coarse to Fine Frame Selection for Online Open-ended Video Question Answering

The central aim of Video Question Answering (VideoQA) is to provide answers to questions posed in natural language, relying on the content of the given videos. However, when applied to video streams like CCTV recordings and live broadcasts, the solver encounters more intricate challenges. In such scenarios, the segment of the video needed to answer a specific question is often a small component of the entire video. To address these complexities, a recent and innovative problem domain called Online Open-ended Video Question Answering (O <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> VQA) has been introduced[18].In this paper, we propose an architecture based on multi-modal foundational transformers for the O <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> VQA task. The architecture comprises three modules. The first module is responsible for the coarse selection of the target video segment relevant to answering the question. The second module refines this coarse segment by leveraging a Temporal Concept Spotting mechanism, enabling the capture of temporal saliency and resulting in the identification of frames most critical for addressing the question. Lastly, we employ an end-to-end Video-Language Pre-training model to provide the answer. To evaluate our proposed model, we conduct experiments on the publicly available ATBS dataset[18]. The results showcase the superiority of our approach over current state-of-the-art models.
TrainFors: A Large Benchmark Training Dataset for Image Manipulation Detection and Localization

The evaluation datasets and metrics for image manipulation detection and localization (IMDL) research have been standardized. But the training dataset for such a task is still nonstandard. Previous researchers have used unconventional and deviating datasets to train neural networks for detecting image forgeries and localizing pixel maps of manipulated regions. For a fair comparison, the training set, test set, and evaluation metrics should be persistent. Hence, comparing the existing methods may not seem fair as the results depend heavily on the training datasets as well as the model architecture. Moreover, none of the previous works release the synthetic training dataset used for the IMDL task. We propose a standardized benchmark training dataset for image splicing, copy-move forgery, removal forgery, and image enhancement forgery. Furthermore, we identify the problems with the existing IMDL datasets and propose the required modifications. We also train the state-of-the-art IMDL methods on our proposed TrainFors <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> dataset for a fair evaluation and report the actual performance of these methods under similar conditions.
Reconstruction of 3D Interaction Models from Images using Shape Prior

We investigate the reconstruction of 3D human-object interactions from images, encompassing 3D human shape and pose estimation as well as object shape and pose estimation. To address this task, we introduce an autoregressive transformer-based variational autoencoder capable of learning a robust shape prior from extensive 3D shape datasets. Additionally, we leverage the reconstructed 3D human body as supplementary features for object shape and pose estimation. In contrast, prior methods only predict object pose and rely on shape templates for shape prediction. Experimental findings on the BEHAVE dataset underscore the effectiveness of our proposed approach, achieving a 40.7cm Chamfer distance and demonstrating the advantages of learning a shape prior.
Scalable MAV Indoor Reconstruction with Neural Implicit Surfaces

Many previous works achieved impressive reconstruction results on room-scale indoor scenes from multi-view RGB images, but capturing and reconstructing multistory, complex indoor scenes is still a challenging problem. In this paper, we propose a fully automated pipeline for reconstructing large and complex indoor scenes with drone-captured RGB images. First, we leverage traditional structure-from-motion methods to obtain camera poses and reconstruct an initial point cloud. Next, we devise a divide-and-conquer strategy to utilize neural surface reconstruction under the Manhattan-world assumption. Our method reduces the point cloud’s outliers and significantly improves reconstruction quality on low-textured regions. We simultaneously predict point-wise semantic logits for walls, floors, and ceilings. The semantic segmentation enables category-wise plane fitting and improves reconstruction quality on polygonal geometry. To validate our method, we use a drone to capture videos inside a large-scale, complex indoor scene. Experimental results showed our method achieved better PSNR in view synthesis tasks and higher floor plan IOU than traditional reconstruction solutions such as COLMAP.
CLIPath: Fine-tune CLIP with Visual Feature Fusion for Pathology Image Analysis Towards Minimizing Data Collection Efforts

Contrastive Language-Image Pre-training (CLIP) has shown its ability to learn distinctive visual representations and generalize to various downstream vision tasks. However, its applicability in the classification of pathology images with limited labeled data is still under study due to the giant domain shift (between large natural image datasets in the source domain and small-scale target pathology images) and overfitting issues. In this work, we first explore the zero-shot transferability of CLIP on pathology classification tasks and benchmark the performance. Then, we propose Residual Feature Connection (RFC) to fine-tune CLIP with a small amount of trainable parameters. RFC aims to fuse the task-specific knowledge learned from the target domain and the original knowledge pre-trained from CLIP. We show that RFC can adapt pre-trained CLIP to downstream pathology tasks and achieve good performance with just a few annotated samples. Specifically, RFC achieves over 19% improvement in accuracy when only using 0.1% of labeled data in PCam with only 10 minutes of fine-tuning while running on a single GPU.
Looking through the past: better knowledge retention for generative replay in continual learning

In this work, we improve the generative replay in a continual learning setting. We notice that in VAE-based generative replay, the generated features are quite far from the original ones when mapped to the latent space. Therefore, we propose modifications that allow the model to learn and generate complex data. More specifically, we incorporate the distillation in latent space between the current and previous models to reduce feature drift. Additionally, a latent matching for the reconstruction and original data is proposed to improve generated features alignment. Further, based on the observation that the reconstructions are better for preserving knowledge, we add the cycling of generations through the previously trained model to make them closer to the original data. Our method outperforms other generative replay methods in various scenarios.