Reviewerly/exHarmony huggingface dataset:
Small object detection leveraging density‐aware scale adaptation

Abstract Albeit general object detection has made impressive progress in the last decade, as a significant subfield, small object detection still performs far from satisfactorily, which is impeded by several challenges, such as small size, severe occlusion and variant scales. To tackle these challenges, we propose a coarse‐to‐fine small object detection method leveraging density‐aware scale adaptation. Firstly, we employ global sketchy prediction via a coarse network in large scenes and generate adaptively scaled block regions with potential targets. Subsequently, we perform local accurate detection by a fine network for instances in densely packed areas with approximately unified scales. In particular, a density map with object distribution information is utilised to provide a scene classification auxiliary to instruct scale transformation. Extensive experiments on the popular remote sensing benchmark AI‐TOD and representative small object datasets VisDrone and UAVDT demonstrate the superiority of our method for small object detection, achieving an improvement of 2.9% mAP‐vt and 2.1% mAP on AI‐TOD, and outperforming the state‐of‐the‐art methods on VisDrone and UAVDT with an enhancement of 1.7% mAP and 2.0% mAP50, respectively.
An intelligent energy minimization algorithm with virtual machine consolidation for sensor-based decision support system

Cloud computing has many benefits for businesses because of its distinctive qualities, including scalability, flexibility, on-demand service, and security. A good task scheduler is required to increase the efficiency of a cloud system, which performs numerous jobs simultaneously. On-demand access to resources that have been virtualized is made available as a service without the need for further waiting. For task scheduling issues based on a makespan limitation, energy consumption is decreased, significantly reducing energy cost. Additionally, the complexity of scheduling issues has increased primarily due to the application's lack of a makespan constraint. Unfortunately, reducing the energy cloud services use presents special research issues and difficulties. More precisely, because of the diversity of servers found in cloud centers, it is challenging to choose the best servers for cloud-based decision support systems to reduce energy usage. The presented approach is innovative and could be applied for complex applications while maintaining an average energy consumption for the running resources, which is a big challenge. The current process finds an optimized energy minimization in the cloud with a combination of Virtual Machine consolidation. The outputs were considered in terms of Energy, Virtual Machine Migration, Performance Degradation, Aggregated Ideal Time Factor, and Aggregated Overload Time Fraction. The obtained results in terms of the existing state of art approaches are far better than the competing approaches.
Ultraviolet-photoluminescence and trace element analyses in Ga-rich sphalerite from the Djebel Gustar Zn-Pb deposit, Algeria

The Djebel Gustar Zn-Pb deposit in northeastern Algeria is one of many carbonate-hosted base metal deposits in the Atlas metallogenic province of North Africa. The sphalerite in this deposit occurs in five distinct types that differ in texture, mineral assemblage, chemical composition, and UV photoluminescence signature. Detailed investigation of trace elements by laser ablation inductively coupled plasma mass spectrometry (LA-ICP-MS) showed that the sphalerite is significantly enriched in the critical metal Ga. The trace element concentrations of the different sphalerite types correlated well with their photoluminescence color; with Mn2+ (orange-yellow), Cu+ (green), and Ga3+ (red) being the dominant activators. We suggest that the Ga enrichment in the Djebel Gustar deposit is the result of formation conditions that include the mixing of a Cl-rich fluid carrying mainly Zn and Pb with a more reduced and Cl-poor fluid containing considerable organic matter and/or calcite with adsorbed Ga(OH)3. This Ga was subsequently incorporated into sphalerite, rather than forming a stoichiometric Ga phase. Furthermore, our study demonstrates that UV photoluminescence of low Fe sphalerite has the potential to become a simple and affordable tool for preliminary characterization of sphalerite ores in mining and exploration with respect to critical elements such as Ga.
NeatSankey: Sankey diagrams with improved readability based on node positioning and edge bundling

Sankey diagrams are widely used to visualize event sequence data. However, when the data volume is large, its readability is affected by dense edge crossings, excessive swing amplitude, and small crossover angles, while it is computationally intensive to obtain an optimal layout. In this paper, we propose NeatSankey, a balanced method that generates Sankey diagrams smoothly. It can be laid out quickly with good readability when Sankey diagrams are very complex. At the same time, to comprehensively evaluate the readability of Sankey diagrams, we use three evaluation metrics: crossing number, swing amplitude, and layout coverage. Firstly, we use a heuristic layout algorithm and a force-directed algorithm to adjust the node layout to minimize the edge crossings and swing amplitude with edge widths considered. Secondly, to better reduce the dense confusion caused by edge crossings, we introduce a edge bundling algorithm based on attribute similarity. We present three evaluations: a comprehensive comparison of our results with state-of-the-art techniques, user studies with thirty volunteers, and a case study of two datasets. Our evaluations demonstrate the effectiveness and practicability of the NeatSankey.
Lifelong learning on evolving graphs under the constraints of imbalanced classes and new classes

Lifelong graph learning deals with the problem of continually adapting graph neural network (GNN) models to changes in evolving graphs. We address two critical challenges of lifelong graph learning in this work: dealing with new classes and tackling imbalanced class distributions. The combination of these two challenges is particularly relevant since newly emerging classes typically resemble only a tiny fraction of the data, adding to the already skewed class distribution. We make several contributions: First, we show that the amount of unlabeled data does not influence the results, which is an essential prerequisite for lifelong learning on a sequence of tasks. Second, we experiment with different label rates and show that our methods can perform well with only a tiny fraction of annotated nodes. Third, we propose the gDOC method to detect new classes under the constraint of having an imbalanced class distribution. The critical ingredient is a weighted binary cross-entropy loss function to account for the class imbalance. Moreover, we demonstrate combinations of gDOC with various base GNN models such as GraphSAGE, Simplified Graph Convolution, and Graph Attention Networks. Lastly, our k-neighborhood time difference measure provably normalizes the temporal changes across different graph datasets. With extensive experimentation, we find that the proposed gDOC method is consistently better than a naive adaption of DOC to graphs. Specifically, in experiments using the smallest history size, the out-of-distribution detection score of gDOC is 0.09 compared to 0.01 for DOC. Furthermore, gDOC achieves an Open-F1 score, a combined measure of in-distribution classification and out-of-distribution detection, of 0.33 compared to 0.25 of DOC (32% increase).
Price discovery between Bitcoin spot markets and exchange traded products

We examine price discovery dynamics between Bitcoin exchange-traded products (ETPs) and spot markets on centralised cryptocurrency exchanges. We apply four popular price discovery measures to ETP and spot transaction data between August 2021 and July 2022. Our results show that price discovery is dominated by the spot market across all measures and sampling frequencies. This implies that ETP markets play a smaller role in the incorporation of new information about Bitcoin prices, and that informed investors largely prefer to trade on spot markets that offer significantly deeper liquidity and operate round the clock.
Preciser comparison: Augmented multi-layer dynamic contrastive strategy for text2text question classification

Text2text question classification (TQC), as a particular application case of question classification (QC), is of great practical value. Traditional QC methods usually label categories of questions using one or multiple keywords provided by users. In contrast, in TQC, each question in natural language is automatically categorized into pre-designed standard question classes, which are coded in the form of short text. Because of this unique characteristic, TQC relies on a specifically designed framework and should be trained and validated based on customized experimental datasets. Previous TQC-related work mainly utilized textual similarity-matching methods. However, no effective pairwise learning paradigm has been proposed in TQC to model correlations between input text and classes; and the influence of distance metrics and loss function in TQC has not been investigated. In this work, we propose a novel and comprehensive strategy, Augmented Dynamic Multi-layer Contrastive (ADMC), to resolve the challenge of TQC. Our framework consists of (1) an optional data augmentation module, (2) one stage for dynamic negative sampling, and (3) one stage for precise matching. The comprehensive TQC framework with ADMC strategy in this work resolves data imbalance and explores distance metrics learning via multiple augmentation options and dynamic negative sampling based on multi-layer contrastive learning. To compensate for the shortage of public datasets for this task, we collected two real-world datasets and adaptively expanded three existing public datasets, which will be available after data masking. The results show that our ADMC outperformed other baseline methods investigated in this paper. The codes are available at https://github.com/WJULYW/ADMC.
Robust Acoustic And Semantic Contextual Biasing In Neural Transducers For Speech Recognition

Attention-based contextual biasing approaches have shown significant improvements in the recognition of generic and/or personal rare-words in End-to-End Automatic Speech Recognition (E2E ASR) systems like neural transducers. These approaches employ crossattention to bias the model towards specific contextual entities injected as bias-phrases to the model. Prior approaches typically relied on subword encoders for encoding the bias phrases. However, subword tokenizations are coarse and fail to capture granular pronunciation information which is crucial for biasing based on acoustic similarity. In this work, we propose to use lightweight character representations to encode fine-grained pronunciation features to improve contextual biasing guided by acoustic similarity between the audio and the contextual entities (termed acoustic biasing). We further integrate pretrained neural language model (NLM) based encoders to encode the utterance's semantic context along with contextual entities to perform biasing informed by the utterance’s semantic context (termed semantic biasing). Experiments using a Conformer Transducer model on the Librispeech dataset show a 4.62% - 9.26% relative WER improvement on different biasing list sizes over the baseline contextual model when incorporating our proposed acoustic and semantic biasing approach. On a large-scale in-house dataset, we observe 7.91% relative WER improvement compared to our baseline model. On tail utterances, the improvements are even more pronounced with 36.80% and 23.40% relative WER improvements on Librispeech rare words and an in-house testset respectively.
Massively Multilingual Shallow Fusion with Large Language Models

While large language models (LLM) have made impressive progress in natural language processing, it remains unclear how to utilize them in improving automatic speech recognition (ASR). In this work, we propose to train a single multilingual language model (LM) for shallow fusion in multiple languages. We push the limits of the multilingual LM to cover up to 84 languages by scaling up using a mixture-of-experts LLM, i.e., generalist language model (GLaM). When the number of experts increases, GLaM dynamically selects only two at each decoding step to keep the inference computation roughly constant. We then apply GLaM to a multilingual shallow fusion task based on a state-of-the-art end-to-end model. Compared to a dense LM of similar computation during inference, GLaM reduces the WER of an English long-tail test set by 4.4% relative. In a multilingual shallow fusion task, GLaM improves 41 out of 50 languages with an average relative WER reduction of 3.85%, and a maximum reduction of 10%. Compared to the baseline model, GLaM achieves an average WER reduction of 5.53% over 43 languages.
CB-Conformer: Contextual Biasing Conformer for Biased Word Recognition

Due to the mismatch between the source and target domains, how to better utilize the biased word information to improve the performance of the automatic speech recognition model in the target domain becomes a hot research topic. Previous approaches either decode with a fixed external language model or introduce a sizeable biasing module, which leads to poor adaptability and slow inference. In this work, we propose CB-Conformer to improve biased word recognition by introducing the Contextual Biasing Module and the Self-Adaptive Language Model to vanilla Conformer. The Contextual Biasing Module combines audio fragments and contextual information, with only 0.2% model parameters of the original Conformer. The Self-Adaptive Language Model modifies the internal weights of biased words based on their recall and precision, resulting in a greater focus on biased words and more successful integration with the automatic speech recognition model than the standard fixed language model. In addition, we construct and release an open-source Mandarin biased-word dataset based on WenetSpeech. Experiments indicate that our proposed method brings a 15.34% character error rate reduction, a 14.13% biased word recall increase, and a 6.80% biased word F1-score increase compared with the base Conformer.
Lattice-Free Sequence Discriminative Training for Phoneme-Based Neural Transducers

Recently, RNN-Transducers have achieved remarkable results on various automatic speech recognition tasks. However, lattice-free sequence discriminative training methods, which obtain superior performance in hybrid models, are rarely investigated in RNN-Transducers. In this work, we propose three lattice-free training objectives, namely lattice-free maximum mutual information, lattice-free segment-level minimum Bayes risk, and lattice-free minimum Bayes risk, which are used for the final posterior output of the phoneme-based neural transducer with a limited context dependency. Compared to criteria using N-best lists, lattice-free methods eliminate the decoding step for hypotheses generation during training, which leads to more efficient training. Experimental results show that lattice-free methods gain up to 6.5% relative improvement in word error rate compared to a sequence-level cross-entropy trained model. Compared to the N-best-list based minimum Bayes risk objectives, lattice-free methods gain 40% - 70% relative training time speedup with a small degradation in performance.
An ASR-Free Fluency Scoring Approach with Self-Supervised Learning

A typical fluency scoring system generally relies on an automatic speech recognition (ASR) system to obtain time stamps in input speech for the subsequent calculation of fluency-related features or directly modeling speech fluency with an end-to-end approach. This paper describes a novel ASR-free approach for automatic fluency assessment using self-supervised learning (SSL). Specifically, wav2vec2.0 is used to extract frame-level speech features, followed by K-means clustering to assign a pseudo label (cluster index) to each frame. A BLSTM-based model is trained to predict an utterance-level fluency score from frame-level SSL features and the corresponding cluster indexes. Neither speech transcription nor time stamp information is required in the proposed system. It is ASR-free and can potentially avoid the ASR errors effect in practice. Experimental results carried out on non-native English databases show that the proposed approach significantly improves the performance in the "open response" scenario as compared to previous methods and matches the recently reported performance in the "read aloud" scenario.
Two-Branch Multi-Scale Deep Neural Network for Generalized Document Recapture Attack Detection

The image recapture attack is an effective image manipulation method to erase certain forensic traces, and when targeting on personal document images, it poses a great threat to the security of e-commerce and other web applications. Considering the current learning-based methods suffer from serious over-fitting problem, in this paper, we propose a novel two-branch deep neural network by mining better generalized recapture artifacts with a designed frequency filter bank and multi-scale cross-attention fusion module. In the extensive experiment, we show that our method can achieve better generalization capability compared with state-of-the-art techniques on different scenarios.
AMPose: Alternately Mixed Global-Local Attention Model for 3D Human Pose Estimation

The graph convolutional networks (GCNs) have been applied to model the physically connected and non-local relations among human joints for 3D human pose estimation (HPE). In addition, the purely Transformer-based models recently show promising results in video-based 3D HPE. However, the single-frame method still needs to model the physically connected relations among joints because the feature representations transformed only by global relations via the Transformer neglect information on the human skeleton. To deal with this problem, we propose a novel method in which the Transformer encoder and GCN blocks are alternately stacked, namely AMPose, to combine the global and physically connected relations among joints towards HPE. In the AMPose, the Transformer encoder is applied to connect each joint with all the other joints, while GCNs are applied to capture information on physically connected relations. The effectiveness of our proposed method is evaluated on the Human3.6M dataset. Our model also shows better generalization ability by testing on the MPI-INF-3DHP dataset. Code can be retrieved at https://github.com/erikervalid/AMPose.
Enhancing Speech-To-Speech Translation with Multiple TTS Targets

It has been known that direct speech-to-speech translation (S2ST) models usually suffer from the data scarcity issue because of the limited existing parallel materials for both source and target speech. Therefore to train a direct S2ST system, previous works usually utilize text-to-speech (TTS) systems to generate samples in the target language by augmenting the data from speech-to-text translation (S2TT). However, there is a limited investigation into how the synthesized target speech would affect the S2ST models. In this work, we analyze the effect of changing synthesized target speech for direct S2ST models. We find that simply combining the target speech from different TTS systems can potentially improve the S2ST performances. Following that, we also propose a multi-task framework that jointly optimizes the S2ST system with multiple targets from different TTS systems. Extensive experiments demonstrate that our proposed framework achieves consistent improvements (2.8 BLEU) over the baselines on the Fisher Spanish-English dataset.
A Token-Level Contrastive Framework for Sign Language Translation

Sign Language Translation (SLT) is a promising technology to bridge the communication gap between the deaf and the hearing people. Recently, researchers have adopted Neural Machine Translation (NMT) methods, which usually require large-scale corpus for training, to achieve SLT. However, the publicly available SLT corpus is very limited, which causes the collapse of the token representations and the inaccuracy of the generated tokens. To alleviate this issue, we propose Con-SLT, a novel token-level Contrastive learning framework for Sign Language Translation , which learns effective token representations by incorporating token-level contrastive learning into the SLT decoding process. Concretely, ConSLT treats each token and its counterpart generated by different dropout masks as positive pairs during decoding, and then randomly samples K tokens in the vocabulary that are not in the current sentence to construct negative examples. We conduct comprehensive experiments on two benchmarks (PHOENIX14T and CSL-Daily) for both end-to-end and cascaded settings. The experimental results demonstrate that ConSLT can achieve better translation quality than the strong baselines <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Importance of Different Temporal Modulations of Speech: a Tale of two Perspectives

How important are different temporal speech modulations for speech recognition? We answer this question from two complementary perspectives. Firstly, we quantify the amount of phonetic information in the modulation spectrum of speech by computing the mutual information between temporal modulations with frame-wise phoneme labels. Looking from another perspective, we ask - which speech modulations an Automatic Speech Recognition (ASR) system prefers for its operation. Data-driven weights are learned over the modulation spectrum and optimized for an end-to-end ASR task. Both methods unanimously agree that speech information is mostly contained in slow modulation. Maximum mutual information occurs around 3-6 Hz which also happens to be the range of modulations most preferred by the ASR. In addition, we show that the incorporation of this knowledge into ASRs significantly reduces their dependency on the amount of training data.
Untargeted Backdoor Attack Against Object Detection

Recent studies revealed that deep neural networks (DNNs) are exposed to backdoor threats when training with third-party resources (such as training samples or backbones). The back-doored model has promising performance in predicting benign samples, whereas its predictions can be maliciously manipulated by adversaries based on activating its backdoors with pre-defined trigger patterns. Currently, most of the existing backdoor attacks were conducted on the image classification under the targeted manner. In this paper, we reveal that these threats could also happen in object detection, posing threatening risks to many mission-critical applications (e.g., pedestrian detection and intelligent surveillance systems). Specifically, we design a simple yet effective poison-only backdoor attack in an untargeted manner, based on task characteristics. We show that, once the backdoor is embedded into the target model by our attack, it can trick the model to lose detection of any object stamped with our trigger patterns. We conduct extensive experiments on the benchmark dataset, showing its effectiveness in both digital and physical-world settings and its resistance to potential defenses.