Reviewerly/exHarmony huggingface dataset:
Explicit and Implicit Knowledge Distillation via Unlabeled Data

Data-free knowledge distillation is a challenging model lightweight task for scenarios in which the original dataset is not available. Previous methods require a lot of extra computational costs to update one or more generators and their naive imitate-learning lead to lower distillation efficiency. Based on these observations, we first propose an efficient unlabeled sample selection method to replace high computational generators and focus on improving the training efficiency of the selected samples. Then, a class-dropping mechanism is designed to suppress the label noise caused by the data domain shifts. Finally, we propose a distillation method that incorporates explicit features and implicit structured relations to improve the effect of distillation. Experimental results show that our method can quickly converge and obtain higher accuracy than other state-of-the-art methods.
Ensemble Knowledge Distillation of Self-Supervised Speech Models

Distilled self-supervised models have shown competitive performance and efficiency in recent years. However, there is a lack of experience in jointly distilling multiple self-supervised speech models. In our work, we performed Ensemble Knowledge Distillation (EKD) on various self-supervised speech models such as HuBERT, RobustHuBERT, and WavLM. We tried two different aggregation techniques, layerwise-average and layerwise-concatenation, to the representations of different teacher models and found that the former was more effective. On top of that, we proposed a multiple prediction head method for student models to predict different layer outputs of multiple teacher models simultaneously. The experimental results show that our method improves the performance of the distilled models on four downstream speech processing tasks, Phoneme Recognition, Speaker Identification, Emotion Recognition, and Automatic Speech Recognition in the hidden-set track of the SUPERB benchmark.
Improving Knowledge Distillation for Non-Intrusive Load Monitoring Through Explainability Guided Learning

Knowledge distillation (KD) is a machine learning technique widely used in recent years for the task of domain adaptation and complexity reduction. It relies on a Student-Teacher mechanism to transfer the knowledge of a large and complex Teacher network into a smaller Student model. Given the inherent complexity of large Deep Neural Network (DNN) models, and the need for deployment on edge devices with limited resources, complexity reduction techniques have become a hot topic in the Non-intrusive Load Monitoring (NILM) community. Recent literature in NILM has devoted increased effort to domain adaptation and architecture reduction via KD. However, the mechanism behind the transfer of knowledge from the Teacher to the Student is not clearly understood. In this work, we aim to address the aforementioned issue by placing the KD NILM approach in a framework of explainable AI (XAI). We identify the main inconsistency in the transfer of explainable knowledge, and exploit this information to propose a method for improvement of KD through explainability guided learning. We evaluate our approach on a variety of appliances and domain adaptation scenarios and demonstrate that solving inconsistencies in the transfer of explainable knowledge can lead to improvement in predictive performance.
Jointly Visual- and Semantic-Aware Graph Memory Networks for Temporal Sentence Localization in Videos

Temporal sentence localization in videos (TSLV) aims to retrieve the most interested segment in an untrimmed video according to a given sentence query. However, almost of existing TSLV approaches suffer from the same limitations: (1) They only focus on either frame-level or object-level visual representation learning and corresponding correlation reasoning, but fail to integrate them both; (2) They neglect to leverage the rich semantic contexts to further benefit the query reasoning. To address these issues, in this paper, we propose a novel Hierarchical Visual- and Semantic-Aware Reasoning Network (HVSARN), which enables both visual- and semantic-aware query reasoning from object-level to frame-level. Specifically, we present a new graph memory mechanism to perform visual-semantic query reasoning: For visual reasoning, we design a visual graph memory to leverage visual information of video; For semantic reasoning, a semantic graph memory is also introduced to explicitly leverage semantic knowledge contained in the classes and attributes of video objects, and perform correlation reasoning in the semantic space. Experiments on three datasets demonstrate that our HVSARN achieves a new state-of-the-art performance.
Audio Coding With Unified Noise Shaping And Phase Contrast Control

Over the past decade, audio coding technology has seen standardization and the development of many frameworks incorporated with linear predictive coding (LPC). As LPC reduces information in the frequency domain, LP-based frequency-domain noise-shaping (FDNS) was previously proposed. To code transient signals effectively, FDNS with temporal noise shaping (TNS) has emerged. However, these mainly operated in the modified discrete cosine transform domain, which essentially accompanies time domain aliasing. In this paper, a unified noise-shaping (UNS) framework including FDNS and complex LPC-based TNS (CTNS) in the DFT domain is proposed to overcome the aliasing issues. Additionally, a modified polar quantizer with phase contrast control is proposed, which saves phase bits depending on the frequency envelope information. The core coding feasibility at low bit rates is verified through various objective metrics and subjective listening evaluations.
Self Supervised Bert for Legal Text Classification

Critical BERT-based text classification tasks, such as legal text classification, require huge amounts of accurately labeled data. Legal text classification faces two trivial problems: labeling legal data is a sensitive process and can only be carried out by skilled professionals, and legal text is prone to privacy issues hence not all the data can be made available in the public domain. This means that we have limited diversity in the textual data, and to account for this data paucity, we propose a self-supervision approach to train Legal-BERT classifiers. We use the BERT text classifier’s knowledge of the class boundaries and perform gradient ascent w.r.t. class logits. Synthetic latent texts are generated through activation maximization. The main advantages over existing SOTAs are that our model: is easy to train, does not require much data but instead uses the synthesized data as fake samples; has less variance that helps to generate texts with good sample quality and diversity. We show the efficacy of the proposed method on the ECHR Violation (Multi-Label) Dataset and the Over-ruling Task Dataset.
Hybrid Neural Network with Cross- and Self-Module Attention Pooling for Text-Independent Speaker Verification

Extraction of a speaker embedding vector plays an important role in deep learning-based speaker verification. In this contribution, to extract speaker discriminant utterance level embeddings, we propose a hybrid neural network that employs both cross- and self-module attention pooling mechanisms. More specifically, the proposed system incorporates a 2D-Convolution Neural Network (CNN)-based feature extraction module in cascade with a frame-level network, which is composed of a fully Time Delay Neural Network (TDNN) network and a TDNN-Long Short Term Memory (TDNN-LSTM) hybrid network in a parallel manner. The proposed system also employs a multi-level cross- and self-module attention pooling for aggregating the speaker information within an utterance-level context by capturing the complementarity between two parallelly connected modules. In order to evaluate the proposed system, we conduct a set of experiments on the Voxceleb corpus, and the proposed hybrid network is able to outperform the conventional approaches trained on the same dataset.
D<sup>2</sup>Q-DETR: Decoupling and Dynamic Queries for Oriented Object Detection with Transformers

Despite the promising results, existing oriented object detection methods usually involve heuristically designed rules, e.g., RRoI generation, rotated NMS. In this paper, we propose an end-to-end framework for oriented object detection, which simplifies the model pipeline and obtains superior performance. Our framework is based on DETR, with the box regression head replaced with a points prediction head. The learning of points is more flexible, and the distribution of points can reflect the angle and size of the target rotated box. We further propose to decouple the query features into classification and regression features, which significantly improves the model precision. Aerial images usually contain thousands of instances. To better balance model precision and efficiency, we propose a novel dynamic query design, which reduces the number of object queries in stacked decoder layers without sacrificing model performance. Finally, we rethink the label assignment strategy of existing DETR-like detectors and propose an effective label re-assignment strategy for improved performance. We name our method D <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> Q-DETR. Experiments on the largest and challenging DOTA-v1.0 and DOTA-v1.5 datasets show that D <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> Q-DETR outperforms existing NMS-based and NMS-free oriented object detection methods and achieves the new state-of-the-art.
SUFF: Accelerating Subgraph Matching with Historical Data

Subgraph matching is a fundamental problem in graph theory and has wide applications in areas like sociology, chemistry, and social networks. Due to its NP-hardness, the basic approach is a brute-force search over the whole search space. Some pruning strategies have been proposed to reduce the search space. However, they are either space-inefficient or based on assumptions that the graph has specific properties. In this paper, we propose SUFF, a general and powerful structure filtering framework, which can accelerate most of the existing approaches with slight modifications. Specifically, it builds a set of filters using matching results of past queries, and uses them to prune the search space for future queries. By fully utilizing the relationship between matches of two queries, it ensures that such pruning is sound. Furthermore, several optimizations are proposed to reduce the computation and space cost for building, storing, and using filters. Extensive experiments are conducted on multiple real-world data sets and representative existing approaches. The results show that SUFF can achieve up to 15X speedup with small overheads.
Distributed Graph Embedding with Information-Oriented Random Walks

Graph embedding maps graph nodes to low-dimensional vectors, and is widely adopted in machine learning tasks. The increasing availability of billion-edge graphs underscores the importance of learning efficient and effective embeddings on large graphs, such as link prediction on Twitter with over one billion edges. Most existing graph embedding methods fall short of reaching high data scalability. In this paper, we present a general-purpose, distributed, information-centric random walk-based graph embedding framework, DistGER, which can scale to embed billion-edge graphs. DistGER incrementally computes information-centric random walks. It further leverages a multi-proximity-aware, streaming, parallel graph partitioning strategy, simultaneously achieving high local partition quality and excellent workload balancing across machines. DistGER also improves the distributed Skip-Gram learning model to generate node embeddings by optimizing the access locality, CPU throughput, and synchronization efficiency. Experiments on real-world graphs demonstrate that compared to state-of-the-art distributed graph embedding frameworks, including KnightKing, DistDGL, and Pytorch-BigGraph, DistGER exhibits 2.33×--129× acceleration, 45% reduction in cross-machines communication, and &gt;10% effectiveness improvement in downstream tasks.
Time-Aware Recommender Systems: A Comprehensive Survey and Quantitative Assessment of Literature

Recommender systems (RS) are among the most widely used applications in data mining and machine-learning technologies. These technologies recommend relevant products to customers, such as movies to watch, items to buy, and books to read. The difference in user preferences over time is one of the most significant issues faced by recommender systems. Researchers have focused on time-sensitive recommender systems, and numerous studies have been conducted in this field. These studies aim to consider the time factor while offering recommendations to users by incorporating and utilizing temporal data in recommendations. In this work, we review existing works in this field and present the most prominent techniques and algorithms that have the ability to capture changes in user preferences over time, and the most important application areas for these recommendations. In addition, we present a quantitative assessment of comprehensive literature that investigates publications in terms of publication time, publication type, and datasets used. Finally, we highlight a range of findings and conclusions and provide the reader with insights based on a general analysis of time-sensitive recommender systems.
Hybrid Nature-Inspired Based Oversampling and Feature Selection Approach for Imbalance Data Streams Classification

A big data stream is described using 5 $\mathrm{V}prime$s (Volume, Variety, Velocity, Variability, and Veracity). These characteristics impose various challenges. In many cases, data streams are unbalanced, making traditional data mining approaches impossible to employ. The standard data mining approaches are not suitable for imbalanced data streams for achieving analytical efficiency because they require periodic analyses, but big data requires real-time analytics. Additionally, the induction model must be re-run and rebuilt each time to add up the most recent data. Mining these unique streams, on the other hand, is one of the most intriguing research areas. Deep learning (DL) algorithms were developed to increase classification performance for issues requiring large data sets with varying types and characteristics. Feature selection (F.S.) is a critical stage in any classification application. F.S. entails eliminating superfluous and redundant characteristics, resulting in a prediction model that is more efficient, interpretable, and fast. While complete solutions are available for F.S., managing massive data streams that need instantaneous processing is challenging by its own nature. This work provides a hybrid metaheuristic strategy with FFSMOTE for oversampling imbalanced data stream and Honey Bee algorithm for feature selection. Hybridization aims to improve feature selection processes by combing the advantage of both the algorithms. Furthermore, the Ensemble Deep classifiers are used for data stream classification.
A Bibliometric Analysis of Network Security on Blockchain

In the present technological era, blockchain is a buzzword for security. And one of the most often discussed approaches for protecting data storage and transfer through trustworthy, decentralized peer-to-peer systems is blockchain. It is not only limited to cryptocurrency but also for secure data transfer within the network because blockchain provides a transparent, tamperproof, and distributed ledger for data transactions over the network. Any user inside the blockchain may validate data integrity, which satisfies the requirements of the network. Additionally, the blockchain allows any user to verify the accuracy of data, meeting the needs of the network. This paper is based on the examination of peer-reviewed research based on blockchain’s potential for network security. This paper also provides a scientific examination of the most widely used blockchain security solution based on the bibliometric analysis. The objective of this work is to combine a bibliometric analysis with a thorough blockchain investigation over network security. The VOS viewer program software is created by Leiden University, was used to evaluate the data once it was retrieved from the Scopus database. For bibliometric analysis more than 200 papers has been selected from the Scopus database, based on the appropriate keywords.
Understanding Deep MIMO Detection

Incorporating deep learning (DL) into multiple-input multiple-output (MIMO) detection has been deemed as a promising technique for future wireless communications. However, most of the DL-based MIMO detection algorithms are lack of interpretation on internal mechanisms. In this paper, we analyze the performance of the DL-based MIMO detection to better understand its strengths and weaknesses. We investigate and compare two different models: data-driven DL detector with neural networks activated by rectifier linear unit (ReLU) function and model-driven DL detector based on traditional detection algorithms. We show that the data-driven DL detector asymptotically approaches to the maximum a posterior (MAP) detector in various scenarios but requires a large amount of training samples to converge in time-varying channels. On the other hand, the model-driven DL detector utilizes the expert knowledge to alleviate the impact of channels and achieves relatively high detection accuracy with a small set of training data. Simulation results confirm our analytical results and demonstrate the effectiveness of the DL-based MIMO detection for both linear and nonlinear signal systems.
A Benchmark for Hierarchical Emotion Cause Extraction in Spoken Dialogues

Emotion cause extraction (ECE) seeks to find out what causes a given emotion, which has drawn much attention in natural language and signal processing. Conventional ECE normally focuses on single level (i.e., either word-level or clause-level) in a document scenario. However, as we known, single level ECE can not satisfy the wide applications, compared with both levels. Besides, the existing dialogue systems increasingly need empathy support. Therefore, in this paper, we propose to hierarchically extract both word and utterance-level emotion causes in the spoken dialogue scenario (HECE). We first construct two datasets (i.e., HECE-DD and HECE-IE) based on previous studies, then propose a hierarchical framework, which consists of a feature extractor, an utterance-level cause extractor, and a word-level cause extractor. In this framework, utterance and word levels can naturally preform interaction. Detailed experiments on two HECE datasets demonstrate that hierarchical extraction performs better than extraction on a single level.
Resource Constrained Vehicular Edge Federated Learning With Highly Mobile Connected Vehicles

This paper proposes a vehicular edge federated learning (VEFL) solution, where an edge server leverages highly mobile connected vehicles’ (CVs’) onboard central processing units (CPUs) and local datasets to train a global model. Convergence analysis reveals that the VEFL training loss depends on the successful receptions of the CVs’ trained models over the intermittent vehicle-to-infrastructure (V2I) wireless links. Owing to high mobility, in the full device participation case (FDPC), the edge server aggregates client model parameters based on a weighted combination according to the CVs’ dataset sizes and sojourn periods, while it selects a subset of CVs in the partial device participation case (PDPC). We then devise joint VEFL and radio access technology (RAT) parameters optimization problems under delay, energy and cost constraints to maximize the probability of successful reception of the locally trained models. Considering that the optimization problem is NP-hard, we decompose it into a VEFL parameter optimization sub-problem, given the estimated worst-case sojourn period, delay and energy expense, and an online RAT parameter optimization sub-problem. Finally, extensive simulations are conducted to validate the effectiveness of the proposed solutions with a practical 5G new radio (5G-NR) RAT under a realistic microscopic mobility model.
Fast Learning Radiance Fields by Shooting Much Fewer Rays

Learning radiance fields has shown remarkable results for novel view synthesis. The learning procedure usually costs lots of time, which motivates the latest methods to speed up the learning procedure by learning without neural networks or using more efficient data structures. However, these specially designed approaches do not work for most of radiance fields based methods. To resolve this issue, we introduce a general strategy to speed up the learning procedure for almost all radiance fields based methods. Our key idea is to reduce the redundancy by shooting much fewer rays in the multi-view volume rendering procedure which is the base for almost all radiance fields based methods. We find that shooting rays at pixels with dramatic color change not only significantly reduces the training burden but also barely affects the accuracy of the learned radiance fields. In addition, we also adaptively subdivide each view into a quadtree according to the average rendering error in each node in the tree, which makes us dynamically shoot more rays in more complex regions with larger rendering error. We evaluate our method with different radiance fields based methods under the widely used benchmarks. Experimental results show that our method achieves comparable accuracy to the state-of-the-art with much faster training.
Accurate Anomaly Detection With Energy Efficiency in IoT–Edge–Cloud Collaborative Networks

With the applicability of edge intelligence in various domains, anomaly detection, which aims to identify unusual and infrequent circumstances, is regarded as a regularly performed task to guarantee the health of the Internet of Things (IoT) applications. Generally, sensory data are gathered at the network edge and completely transmitted to the cloud, where computational-heavy algorithms are mostly adopted to determine the locations of anomaly. Considering the occurrence infrequency of anomalies, this strategy may transmit relatively huge volume of sensory data, which may reflect a healthy situation indeed, to the cloud. To mitigate this problem, this article proposes an accurate anomaly detection mechanism with energy efficiency in three-tier IoT–edge–cloud collaborative networks. Specifically, after gathering sensory data provided by IoT nodes in certain edge networks, the edge node applies the marching squares algorithm to generate isopleths, where an isopleth may capture the boundary of anomaly. A sensory data filtering mechanism is conducted at the edge tier, such that anomaly-relevant sensory data are transmitted to the cloud and, thus, the network traffic is decreased significantly. Thereafter, the boundary of anomaly is obtained, and the locations of candidate boundary nodes are determined by adopting the Kriging spatial interpolation algorithm at the cloud tier. These locations are traversed by mobile sensing nodes at edge networks, and their sensory data are gathered for boundary refinement. Extensive experiments are conducted on an air quality hazardous gas data set from Toward Data Science, and evaluation results show that our technique outperforms the state-of-the-art counterparts in boundary accuracy and energy consumption.