Reviewerly/exHarmony huggingface dataset:
Towards Domain Generalisation in ASR with Elitist Sampling and Ensemble Knowledge Distillation

Knowledge distillation (KD) has widely been used for model compression and domain adaptation for speech applications. In the presence of multiple teachers, knowledge can easily be transferred to the student by averaging the models output. However, previous research shows that the student do not adapt well with such combination. This paper propose to use an elitist sampling strategy at the output of ensemble teacher models to select the best-decoded utterance generated by completely out-of-domain teacher models for generalizing unseen domain. The teacher models are trained on AMI, LibriSpeech and WSJ while the student is adapted for the Switchboard data. The results show that with the selection strategy based on the individual model’s posteriors the student model achieves a better WER compared to all the teachers and baselines with a minimum absolute improvement of about 8.4%. Furthermore, an insights on the model adaptation with out-of-domain data has also been studied via correlation analysis.
Towards a Robust and Efficient Classifier for Real World Radio Signal Modulation Classification

Automatic modulation classification for radio signals is an important task in many applications, including cognitive radio, radio spectrum monitoring and signal decoding in non-cooperative communications. Recent studies in this area apply various deep learning methods to achieve accurate classification. However, due to the nature of radio signals, distortions during transmission are often unforeseen and unpredictable, which poses a need for robust learning models. At the same time, there is the need for fast real-time modulation classification to meet strict timing requirements. In this work, we propose a lightweight deep learning model that accurately and quickly classifies the modulation of signals having different types of distortions, without the need to be trained using distorted signals. Our model trains 25% faster and classifies 36% faster compared to the state-of-the-art [1], with smaller accuracy degradation on datasets generated using distortion parameters that do not appear in the training set.
DepthFormer: Multimodal Positional Encodings and Cross-Input Attention for Transformer-based Segmentation Networks

Most approaches for semantic segmentation use only information from color cameras to parse the scenes, yet recent advancements show that using depth data allows to further improve performances. In this work, we focus on transformer-based deep learning architectures, that have achieved state-of-the-art performances on the segmentation task, and we propose to employ depth information by embedding it in the positional encoding. Effectively, we extend the network to multimodal data without adding any parameters and in a natural way that exploits the strength of transformers’ self-attention modules. We also investigate the idea of performing cross-modality operations inside the attention module, swapping the key inputs between the depth and color branches. Our approach consistently improves performances on the Cityscapes benchmark.
A Unified Interpretation of the Semantics of Relevance Logic

Abstract I introduce a novel and quite intuitive interpretation of the ternary relation that figures in the relational semantics of many relevance logics. Conceptually, my interpretation makes use only of incompatibility and parthood relations, defined over a set of states. In this way, the proposed interpretation—of the ternary relation and the conditional—extends Dunn’s and Restall’s works on negation and the Routley star operator. Therefore, the interpretation is unified, and hence not only intuitive but also parsimonious. Additionally, the interpretation provides us with a cogent argument to the effect that a ternary relation is indispensable, and we cannot merely do with a binary relation.
Avoid Overthinking in Self-Supervised Models for Speech Recognition

Self-supervised learning (SSL) models reshaped our approach to speech, language and vision. However their huge size and the opaque relations between their layers and tasks result in slow inference and network overthinking, where predictions made from the last layer of large models is worse than those made from intermediate layers. Early exit (EE) strategies can solve both issues by dynamically reducing computations at inference time for certain samples. Although popular for classification tasks in vision and language, EE has seen less use for sequence-to-sequence speech recognition (ASR) tasks where outputs from early layers are often degenerate. This challenge is further compounded when speech SSL models are applied on out-of-distribution (OOD) data. This paper first shows that SSL models do overthinking in ASR. We then motivate further research in EE by computing an optimal bound for performance versus speed trade-offs. To approach this bound we propose two new strategies for ASR: (1) we adapt the recently proposed patience strategy to ASR; and (2) we design a new EE strategy specific to ASR that performs better than all strategies previously introduced.
SAMO: Speaker Attractor Multi-Center One-Class Learning For Voice Anti-Spoofing

Voice anti-spoofing systems are crucial auxiliaries for automatic speaker verification (ASV) systems. A major challenge is caused by unseen attacks empowered by advanced speech synthesis technologies. Our previous research on one-class learning has improved the generalization ability to unseen attacks by compacting the bona fide speech in the embedding space. However, such compactness lacks consideration of the diversity of speakers. In this work, we propose speaker attractor multi-center one-class learning (SAMO), which clusters bona fide speech around a number of speaker attractors and pushes away spoofing attacks from all the attractors in a high-dimensional embedding space. For training, we propose an algorithm for the co-optimization of bona fide speech clustering and bona fide/spoof classification. For inference, we propose strategies to enable anti-spoofing for speakers without enrollment. Our proposed system outperforms existing state-of-the-art single systems with a relative improvement of 38% on equal error rate (EER) on the ASVspoof2019 LA evaluation set.
Inv-Senet: Invariant Self Expression Network for Clustering Under Biased Data

Subspace clustering algorithms are used for understanding the cluster structure that explains the patterns prevalent in the dataset well. These methods are extensively used for data-exploration tasks in various areas of Natural Sciences. However, most of these methods fail to handle confounding attributes in the dataset. For datasets where a data sample represent multiple attributes, naively applying any clustering approach can result in undesired output. To this end, we propose a novel framework for jointly removing confounding attributes while learning to cluster data points in individual subspaces. Assuming we have label information about these confounding attributes, we regularize the clustering method by adversarially learning to minimize the mutual information between the data representation and the confounding attribute labels. Our experimental result on synthetic and real-world datasets demonstrate the effectiveness of our approach.
Wave-U-Net Discriminator: Fast and Lightweight Discriminator for Generative Adversarial Network-Based Speech Synthesis

In speech synthesis, a generative adversarial network (GAN), training a generator (speech synthesizer) and a discriminator in a min-max game, is widely used to improve speech quality. An ensemble of discriminators is commonly used in recent neural vocoders (e.g., HiFi-GAN) and end-to-end text-to-speech (TTS) systems (e.g., VITS) to scrutinize waveforms from multiple perspectives. Such discriminators allow synthesized speech to adequately approach real speech; however, they require an increase in the model size and computation time according to the increase in the number of discriminators. Alternatively, this study proposes a Wave-U-Net discriminator, which is a single but expressive discriminator with Wave-U-Net architecture. This discriminator is unique; it can assess a waveform in a sample-wise manner with the same resolution as the input signal, while extracting multilevel features via an encoder and decoder with skip connections. This architecture provides a generator with sufficiently rich information for the synthesized speech to be closely matched to the real speech. During the experiments, the proposed ideas were applied to a representative neural vocoder (HiFi-GAN) and an end-to-end TTS system (VITS). The results demonstrate that the proposed models can achieve comparable speech quality with a 2.31 times faster and 14.5 times more lightweight discriminator when used in HiFi-GAN and a 1.90 times faster and 9.62 times more lightweight discriminator when used in VITS. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
KG-ECO: Knowledge Graph Enhanced Entity Correction For Query Rewriting

Query Rewriting (QR) plays a critical role in large-scale dialogue systems for reducing frictions. When there is an entity error, it imposes extra challenges for a dialogue system to produce satisfactory responses. In this work, we propose KG-ECO: Knowledge Graph enhanced Entity COrrection for query rewriting, an entity correction system with corrupt entity span detection and entity retrieval/re-ranking functionalities.To boost the model performance, we incorporate Knowledge Graph (KG) to provide entity structural information (neighboring entities encoded by graph neural networks) and textual information (KG entity descriptions encoded by RoBERTa). Experimental results show that our approach yields a clear performance gain over two baselines: utterance level QR and entity correction without utilizing KG information. The proposed system is particularly effective for few-shot learning cases where target entities are rarely seen in training or there is a KG relation between the target entity and other contextual entities in the query.
Topological Slepians: Maximally Localized Representations of Signals Over Simplicial Complexes

This paper introduces topological Slepians, i.e., a novel class of signals defined over topological spaces (e.g., simplicial complexes) that are maximally concentrated on the topological domain (e.g., over a set of nodes, edges, triangles, etc.) and perfectly localized on the dual domain (e.g., a set of frequencies).These signals are obtained as the principal eigenvectors of a matrix built from proper localization operators acting over topology and frequency domains.Then, we suggest a principled procedure to build dictionaries of topological Slepians, which theoretically provide non-degenerate frames.Finally, we evaluate the effectiveness of the proposed topological Slepian dictionary in two applications, i.e., sparse signal representation and denoising of edge flows.
MLP-GAN for Brain Vessel Image Segmentation

Brain vessel image segmentation can be used as a promising biomarker for better prevention and treatment of different diseases. One successful approach is to consider the segmentation as an image-to-image translation task and perform a conditional Generative Adversarial Network (cGAN) to learn a transformation between two distributions. In this paper, we present a novel multi-view approach, MLP-GAN, which splits a 3D volumetric brain vessel image into three different dimensional 2D images (i.e., sagittal, coronal, axial) and then feed them into three different 2D cGANs. The proposed MLP-GAN not only alleviates the memory issue which exists in the original 3D neural networks but also retains 3D spatial information. Specifically, we utilize U-Net as the backbone for our generator and redesign the pattern of skip connection integrated with the MLP-Mixer [1] which has attracted lots of attention recently. Our model obtains the ability to capture cross-patch information to learn global information with the MLP-Mixer. Extensive experiments are performed on the public brain vessel dataset [2] that show our MLP-GAN outperforms other state-of-the-art methods.
Transadapt: A Transformative Framework for Online Test Time Adaptive Semantic Segmentation

Test-time adaptive (TTA) semantic segmentation adapts a source pre-trained image semantic segmentation model to unlabeled batches of target domain test images, different from real-world, where samples arrive one-by-one in an online fashion. To tackle online settings, we propose TransAdapt, a framework that uses transformer and input transformations to improve segmentation performance. Specifically, we pre-train a transformer-based module on a segmentation network that transforms unsupervised segmentation output to a more reliable supervised output, without requiring test-time online training. To also facilitate test-time adaptation, we propose an unsupervised loss based on the transformed input that enforces the model to be invariant and equivariant to photometric and geometric perturbations, respectively. Overall, our framework produces higher quality segmentation masks with up to 17.6% and 2.8% mIOU improvement over no-adaptation and competitive baselines, respectively.
Continuous Action Space-Based Spoken Language Acquisition Agent Using Residual Sentence Embedding and Transformer Decoder

Studies on spoken language acquisition agents aim to understand the mechanism of human language learning and to realize it on computers. Existing open vocabulary agents first perform unsupervised word learning from speech signals to construct a word dictionary as a discrete action space and then conduct reinforcement learning to understand the use of the words in the dictionary through interaction with dialogue partners. A limitation is that they have difficulty pronouncing multi-word utterances. This study proposes an agent that generates multi-word waveform utterances using a continuous action space. The conventional agent uses a vision-focusing mechanism to accelerate dialogue-based learning by guiding the agent’s attention to those concepts in its eyesight. In contrast, the proposed agent replaces it with residual sentence embedding combined with vision features used as the action space. The agent consists of speech and image input front-ends, a transformer language model of pseudo-action space. Experimental results show that the agent learns multi-word utterances assisted by unsupervised learning algorithms using unlabeled speech and image data sets.
Neural Crypto-Coding Based Approach to Enhance the Security of Images over the Untrusted Cloud Environment

The cloud provides on-demand, high-quality services to its users without the burden of managing hardware and software. Though the users benefit from the remote services provided by the cloud, they do not have their personal data in their physical possession. This certainly poses new security threats for personal and confidential data, bringing the focus back on trusting the use of the cloud for sensitive data. The benefits of the cloud outweigh the concerns raised earlier, and with an increase in cloud usage, it becomes more important for security services to evolve in order to address the ever-changing threat landscape. Advanced encryption standard (AES), being one of the most widely used encryption techniques, has inherent disadvantages related to the secret key that is shared, and predictable patterns in subkey generation. In addition, since cloud storage involves data transfer over a wireless channel, it is important to address the effect of noise and multipath propagation on the transmitted data. Catering to this problem, we propose a new approach—the secure and reliable neural cryptcoding (SARNC) technique—which provides a superior algorithm, dealing with better encryption techniques combined with channel coding. A chain is as strong as the weakest link and, in the case of symmetric key encryption, the weakest link is the shared key. In order to overcome this limitation, we propose an approach wherein the key used for cryptographic purposes is different from the key shared between the sender and the receiver. The shared key is used to derive the secret private key, which is generated by the neural key exchange protocol. In addition, the proposed approach emphasizes strengthening the sub-key generation process and integrating advanced encryption standard (AES) with low-density parity check (LDPC) codes to provide end-to-end security and reliability over wireless channels. The proposed technique was tested against research done in related areas. A comparative study shows a significant improvement in PSNR, MSE, and the structural similarity index (SSIM). The key strength analysis was carried out to understand the strength and weaknesses of the keys generated.
HA-FPN: Hierarchical Attention Feature Pyramid Network for Object Detection

The goals of object detection are to accurately detect and locate objects of various sizes in digital images. Multi-scale processing technology can improve the detection accuracy of the detector. Feature pyramid networks (FPNs) have been proven to be effective in extracting multi-scaled features. However, most existing object detection methods recognize objects in isolation, without considering contextual information between objects. Moreover, for the sake of computational efficiency, a significant reduction in the channel dimension may lead to the loss of semantic information. This study explores the utilization of attention mechanisms to augment the representational power and efficiency of features, ultimately improving the accuracy and efficiency of object detection. The study proposed a novel hierarchical attention feature pyramid network (HA-FPN), which comprises two key components: transformer feature pyramid networks (TFPNs) and channel attention modules (CAMs). In TFPNs, multi-scaled convolutional features are embedded as tokens and self-attention is applied to across both the intra- and inter-scales to capture contextual information between the tokens. CAMs are employed to select the channels with rich channel information to alleviate massive channel information losses. By introducing contextual information and attention mechanisms, the HA-FPN significantly improves the accuracy of bounding box detection, leading to more precise identification and localization of target objects. Extensive experiments conducted on the challenging MS COCO dataset demonstrate that the proposed HA-FPN outperforms existing multi-object detection models, while incurring minimal computational overhead.
GaPP: Multi-Target Tracking with Gaussian Processes

Multi-target tracking of agile targets can be limited by choice of dynamical models. This is typically overcome by using sophisticated non-Gaussian and/or nonlinear motion models, and complex data association schemes. Here we aim to tackle scenarios when tracking (semi-)autonomous systems, such as drones, which often follow smooth optimised trajectories and undertake rapid manoeuvres when needed. This paper introduces a novel, flexible, multi-target tracking approach based upon a Gaussian process as a dynamical model, coupled with a non-homogeneous Poisson process for the observation model. It applies a particle filtering inference method for state estimation (including data association) and online parameter learning. The promising performance of the proposed technique is demonstrated on both synthetic data and real drone surveillance radar measurements, compared with a selection of more standard approaches.
Learning Hypergraphs From Signals With Dual Smoothness Prior

Hypergraph structure learning, which aims to learn the hypergraph structures from the observed signals to capture the intrinsic high-order relationships among the entities, becomes crucial when a hypergraph topology is not readily available in the datasets. There are two challenges that lie at the heart of this problem: 1) how to handle the huge search space of potential hyperedges, and 2) how to define meaningful criteria to measure the relationship between the signals observed on nodes and the hypergraph structure. In this paper, for the first challenge, we adopt the assumption that the ideal hypergraph structure can be derived from a learnable graph structure that captures the pairwise relations within signals. Further, we propose a hypergraph structure learning framework HGSL with a novel dual smoothness prior that reveals a mapping between the observed node signals and the hypergraph structure, whereby each hyperedge corresponds to a subgraph with both node signal smoothness and edge signal smoothness in the learnable graph structure. Finally, we conduct extensive experiments to evaluate HGSL on both synthetic and real world datasets. Experiments show that HGSL can efficiently infer meaningful hypergraph topologies from observed signals.
Dynamic Multi-View Scene Reconstruction Using Neural Implicit Surface

Reconstructing general dynamic scenes is important for many computer vision and graphics applications. Recent works represent the dynamic scene with neural radiance fields for photorealistic view synthesis, while their surface geometry is under-constrained and noisy. Other works introduce surface constraints to the implicit neural representation to disentangle the ambiguity of geometry and appearance field for static scene reconstruction. To bridge the gap between rendering dynamic scenes and recovering static surface geometry, we propose a template-free method to reconstruct surface geometry and appearance using neural implicit representations from multi-view videos. We leverage topology-aware deformation and the signed distance field to learn complex dynamic surfaces via differentiable volume rendering without scene-specific prior knowledge like template models. Furthermore, we propose a novel mask-based ray selection strategy to significantly boost the optimization on challenging time-varying regions. Experiments on different multi-view video datasets demonstrate that our method achieves high-fidelity surface reconstruction as well as photorealistic novel view synthesis.
Adapted Multimodal Bert with Layer-Wise Fusion for Sentiment Analysis

Multimodal learning pipelines have benefited from the success of pretrained language models. However, this comes at the cost of increased model parameters. In this work, we propose Adapted Multimodal BERT (AMB), a BERT-based architecture for multimodal tasks that uses a combination of adapter modules and intermediate fusion layers. The adapter adjusts the pretrained language model for the task at hand, while the fusion layers perform task-specific, layer-wise fusion of audio-visual information with textual BERT representations. During the adaptation process the pre-trained language model parameters remain frozen, allowing for fast, parameter-efficient training. In our ablations we see that this approach leads to efficient models, that can outperform their fine-tuned counterparts and are robust to input noise. Our experiments on sentiment analysis with CMU-MOSEI show that AMB outperforms the current state-of-the-art across metrics, with 3.4% relative reduction in the resulting error and 2.1% relative improvement in 7−class classification accuracy.
E2E Segmentation in a Two-Pass Cascaded Encoder ASR Model

We explore unifying a neural segmenter with two-pass cascaded encoder ASR into a single model. A key challenge is allowing the segmenter (which runs in real-time, synchronously with the decoder) to finalize the non-causal 2nd pass (which runs 900 ms behind real-time) without introducing user-perceived latency or deletion errors during inference. We propose a design where the neural segmenter is integrated with the causal 1st pass decoder to emit a end-of-segment (EOS) signal in real-time. The EOS signal is then used to finalize the non-causal 2nd pass. We experiment with different ways to finalize the 2nd pass, and find that a dummy frame injection strategy allows for simultaneous high quality 2nd pass results and low finalization latency. On a real-world long-form captioning task (YouTube), we achieve 2.4% relative WER and 140 ms EOS latency gains over a baseline VAD-based segmenter with the same cascaded encoder.