Reviewerly/exHarmony huggingface dataset:
Pros and Cons of Weight Pruning for Out-of-Distribution Detection: An Empirical Survey

Deep neural networks (DNNs) perform well on samples from the training distribution. However, DNNs deployed in the real world are exposed to out-of-distribution (OOD) samples, which refer to the samples from distributions that differ from the training distribution. OOD detection is indispensable to the DNNs as OOD samples can cause unexpected behaviors for them. This paper empirically explores the effectiveness of weight pruning of DNNs for OOD detection in a post-hoc setting (i.e., performing OOD detection based on pretrained DNN models). We conduct experiments on image, text, and tabular datasets to thoroughly evaluate OOD detection performance of weight-pruned DNNs. Our experimental results bring the following three novel findings: (i) Weight pruning improves OOD detection per-formance more significantly with a Mahalanobis distance-based detection approach, which performs OOD detection on DNN hidden representations using the Mahalanobis distance, than with logit-based detection approaches. (ii) Weight-pruned DNNs tend to extract global features of inputs, which improves the OOD detection on samples much dissimilar to the in-distribution samples. (iii) The weights that are useless for classification are often useful for OOD detection, and thus weight importance should not be quantified as the sensitivity of weights only to classification error. On the basis of these findings, we advocate practical techniques of DNN weight pruning that enable weight-pruned DNNs to maintain both OOD detection and classification capabilities.
Spectral Batch Normalization: Normalization in the Frequency Domain

Regularization is a set of techniques that are used to improve the generalization ability of deep neural networks. In this paper, we introduce spectral batch normalization (SBN), a novel effective method to improve generalization by normalizing feature maps in the frequency (spectral) domain. The activations of residual networks without batch normalization (BN) tend to explode exponentially in the depth of the network at initialization. This leads to extremely large feature map norms even though the parameters are relatively small. These explosive dynamics can be very detrimental to learning. BN makes weight decay regularization on the scaling factors <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\gamma,\beta$</tex> approximately equivalent to an additive penalty on the norm of the feature maps, which prevents extremely large feature map norms to a certain degree. It was previously shown that preventing explosive growth at the final layer at initialization and during training in ResNets can recover a large part of Batch Normalization's generalization boost. However, we show experimentally that, despite the approximate additive penalty of BN, feature maps in deep neural networks (DNNs) tend to explode at the beginning of the training and that feature maps of DNNs contain large values during the whole training. This phenomenon also occurs in a weakened form in non-residual networks. Intuitively, it is not preferred to have large values in feature maps since they have too much influence on the prediction in contrast to other parts of the feature map. SBN addresses large feature maps by normalizing them in the frequency domain. In our experiments, we empirically show that SBN prevents exploding feature maps at initialization and large feature map values during the training. Moreover, the normalization of feature maps in the frequency domain leads to more uniform distributed frequency components. This discourages the DNNs to rely on single frequency components of feature maps. These, together with other effects (e.g. noise injection, scaling and shifting of the feature map) of SBN, have a regularizing effect on the training of residual and non-residual networks. We show experimentally that using SBN in addition to standard regularization methods improves the performance of DNNs by a relevant margin, e.g. ResNet50 on CIFAR-100 by 2.31%, on ImageNet by 0.71% (from 76.80% to 77.51%) and VGG19 on CIFAR-100 by 0.66%.
Real-Time Photonic Deep Reservoir Computing for Speech Recognition

Reservoir computing is a bio-inspired computing paradigm with the main advantage of being less computationally demanding than state-of-the-art neural networks. However, the performance of reservoir computing is limited comparing to other power-hungry machine learning algorithms. Deep reservoir computing addresses this problem by using multiple interconnected reservoirs to provide greater capacity for processing information, leading to improved performance. In this work we propose a photonic deep reservoir computer: we test it on the speech recognition task, and we experimentally confirm its effectiveness. We show how some design choices can simplify the realization of a reservoir computer. Our photonic system is the first experimental deep reservoir computer that can process speech recordings in real-time. We thus hope that this work can pave the way for efficient dedicated hardware for high-speed signal processing.
Augmenting Glosses with Geometrical Inflection Parameters for the Animation of Sign Language Avatars

We present a new machine-readable symbolic representation of sign language based on the pairing of glosses with parameters that can be used for the inflection of motion captured sign animation clips. With respect to existing representations, this approach detaches from a purely linguistic point of view and provides a solution to the problem from a lower-level of abstraction, aiming at generic body-motion manipulation. Early experiments show the effectiveness in manipulating hand trajectories and their potential in modulating the expressivity and communicative emotion of pre-recorded signs.
CTVAE: Current Task Variational Auto-Encoder with Dynamic Ensemble Loss for Few-Shot Learning

Few-shot learning is a challenging task in which a classifier needs to quickly adapt to new classes. These new classes are unseen in the training stage, and there are only very few samples (e.g., five images) provided for learning each new class in the testing stage. When the existing methods learn with such a small amount of samples, they could easily be affected by the outliers. Moreover, the category center calculated from those few samples may deviate from the true center. To address these issues, we propose a novel approach called Current Task Variational Auto-Encoder (CTVAE) for few-shot learning. In our framework, a trained feature extractor first produces the features of the current task, and these features are used to repeatedly train the generator in CTVAE. After that, we can use CTVAE to generate additional features of samples, and then find a new center of the category based on these newly generated features. Compared with the original center, the new center tends to be closer to the true center in vector space. CTVAE can break the limitation of traditional few-shot learning methods, which can only fine-tune the model with very few samples in the testing stage. Moreover, by generating the features directly without producing the images first, the training process of the generator in CTVAE is simplified and becomes more efficient, and the features can be generated faster and more precisely. According to the experiments on benchmark datasets (i.e., Mini-ImageNet, CUB, and CIFAR-FS), our proposed framework is able to outperform the state-of-the-art methods and improves the accuracy by 1–4%. We also conduct experiments on the cross-domain tasks, and the results show that the proposed framework can bring 1-5% accuracy improvements.
Multi-hop Attention GNN with Answer-Evidence Contrastive Loss for Multi-hop QA

Multi-hop question answering (QA) is a challenging task in natural language processing (NLP), which requires multi-step reasoning over the sentences from several passages and finding out the answer as well as the scattered evidence sentences. The existing QA models that are based on Graph Neural Network (GNN) have exhibited good performance, however, the advantages of GNN have not been brought into full play. In this paper, we incorporate an effective multi-hop attention mechanism into GNN to aggregate richer information from high-order nodes of the graph. In addition, when multiple tasks are jointly optimized, the performance of all tasks is usually unable to improve together. To address this problem, we design a novel answer-evidence contrastive learning loss, which encourages models to learn better shared representation and distinguish the evidence sentences from other confusing ones through answer-evidence similarity. Our experiments on HotpotQA dataset demonstrate that the proposed method achieves comparable results to the state-of-the-art models and helps the baseline model gain significant performance improvement.
OOA-UADS: Offline, Online, Analysis-an Unsupervised Anomaly Detection Solution for Multivariate Time Series

In the era of the Industrial Internet of Things, anomaly detection is important for real-world applications. However, most streaming data lack meaningful labels. Furthermore, some anomalies of streaming data may be concept drift, but few methods can deal with it. To address these challenges, we propose an unsupervised anomaly detection solution that can deal with streaming data, called OOA-UADS (Offline, Online, Analysis-an Unsupervised Anomaly Detection Solution for Multivariate Time Series). The solution consists of three stages: offline training, online prediction and anomaly analysis. Time convolutional networks and variational autoencoders are used to deconstruct and reconstruct the multivariate time series data to learn the normal patterns. The anomaly inversion mechanism identifies concept drift in the anomaly prediction stage by dynamically updating the classification thresholds. Intelligent anomaly analysis then provides anomaly dimensions to help engineers better analyse the anomalous behaviour. Our experiments show that OOA-UADS performs satisfactorily. On seven streaming datasets, OOA-UADS outperforms 11 baselines in terms of AUC and provides state-of-the-art F1 scores on three batch datasets.
Cross Domain Deep Collaborative Filtering without Overlapping Data

Cross-domain collaborative filtering (CDCF) is an effective method to alleviate the data sparsity problem by transferring knowledge from a source domain to assist the learning of a target domain. However, most of the existing CDCF approaches require that the two domains have at least one overlapping side (either on user or item) and the raw data can be fully shared across domains, which is difficult to be satisfied in reality due to corporate barriers and the risk of user privacy leakage. Although there are some attempts on applying CDCF to the scenario without overlapping data by transferring cluster-level rating patterns, these methods fail to mine the complex connections between the two domains, which makes their performance still not satisfactory. To address these problems, we propose a novel deep Interaction Distribution Transfer (IDT) framework, which extracts and transfers knowledge from the feature distribution formed by the whole dataset rather than specific data. In this way, the knowledge is embedded into high-order features for transfer, which can effectively avoid privacy leakage during the data sharing process. Moreover, as a flexible framework, IDT obtains powerful feature extraction ability from the base model, which guarantees its superior performance. Extensive experiments on three benchmark datasets are conducted and the results verify the effectiveness of the proposed framework.
Decomposing Source Codes by Program Slicing for Bug Localization

Bug localization, which aims to automatically locate buggy source code files based on the given bug report, is a critical yet time-consuming task in the software engineering field. Existing advanced bug localization methods have successfully leveraged deep learning to bridge the lexical gap between bug reports and source code files at the semantic level. These methods usually first build the entire source code file semantic representation and then match it with the bug report. However, the bug described in the bug report may be related to only part of the source code file semantics. Directly constructing a semantic representation of the entire source code file would increase the difficulty of semantic matching between bug reports and source code files. In this paper, we propose a novel model named S-BugLocator, which decomposes source code file with the help of program slicing. Especially, our proposed S-BugLocator incorporates two distinctly structured slice feature extraction components in processing source code files to cope with the significant discrepancy between multi-row slices and single-row slices. For each multi-row slice, a CNN and Bi-LSTM network is firstly employed to extract its semantics and then a keywords supervised attention mechanism is designed to build its semantic representation by focusing on slices that have strong relevance with the bug report. For each single-row slice, the semantic representation is obtained by fusing word embeddings in single-row slices. The experimental results on four real-world large-scale projects indicate that our proposed model outperforms existing state-of-the-art bug localization methods.
Improving 3D Pose Estimation For Sign Language

This work addresses 3D human pose reconstruction in single images. We present a method that combines Forward Kinematics (FK) with neural networks to ensure a fast and valid prediction of 3D pose. Pose is represented as a hierarchical tree/graph with nodes corresponding to human joints that model their physical limits. Given a 2D detection of keypoints in the image, we lift the skeleton to 3D using neural networks to predict both the joint rotations and bone lengths. These predictions are then combined with skeletal constraints using an FK layer implemented as a network layer in PyTorch. The result is a fast and accurate approach to the estimation of 3D skeletal pose. Through quantitative and qualitative evaluation, we demonstrate the method is significantly more accurate than MediaPipe in terms of both per joint positional error and visual appearance. Furthermore, we demonstrate generalization over different datasets and sign languages. The implementation in PyTorch runs at between 100-200 milliseconds per image (including CNN detection) using CPU only.
LeanStereo: A Leaner Backbone based Stereo Network

Recently, end-to-end deep networks based stereo matching methods, mainly because of their performance, have gained popularity. However, this improvement in performance comes at the cost of increased computational and memory bandwidth requirements, thus necessitating specialized hardware (GPUs); even then, these methods have large inference times compared to classical methods. This limits their applicability in real-world applications. Although we desire high accuracy stereo methods albeit with reasonable inference time. To this end, we propose a fast end-to-end stereo matching method. Majority of this speedup comes from integrating a leaner backbone. To recover the performance lost because of a leaner backbone, we propose to use learned attention weights based cost volume combined with LogL1 loss for stereo matching. Using LogL1 loss not only improves the overall performance of the proposed network but also leads to faster convergence. We do a detailed empirical evaluation of different design choices and show that our method requires <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$4\times$</tex> less operations and is also about 9 to <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$14\times$</tex> faster compared to the state of the art methods like ACVNet [1], LEAStereo [2] and CFNet [3] while giving comparable performance <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Code: https://github.com/cogsys-tuebingen/LeanStereo
A Novel Control-Variates Approach for Performative Gradient-Based Learners with Missing Data

We propose a new, principled approach to tackling missing data problems that can reduce both bias and variance of any (stochastic) gradient descent-based predictive model that is learned on such data. The proposed method can use an arbitrary (and potentially biased) imputation model to fill in the missing values, as it corrects the biases introduced by imputation with a control variates method, leading to an unbiased estimation for gradient updates. Theoretically, we prove that our control variates approach improves the convergence of stochastic gradient descent under common missing data settings. Empirically, we show that our method yields superior performance as compared to the results obtained using competing imputation methods, on various applications, across different missing data patterns.
A Clustered Federated Learning Approach for Estimating the Quality of Experience of Web Users

In this paper, we present FedCLWAvg, a novel clustered Federated Learning (FL) approach for estimating the Quality of Experience (QoE) of Web users. FedCLWAvg performs clustering on the weights of the trained local models (CLW stands for clustered weights) to identify users with similar data distribution. In the context of QoE modelling, the hypothesis is that personal differences (in terms of perceived QoE for the same stimuli) between groups of users are reflected in different weights of the trained local models. Then, each identified cluster learns its own model using the FedAvg algorithm. To validate our approach, we used the Web QoE dataset including the subjective quality of 3,400 Web browsing sessions identified by the measurement of 9 Web session features. Experimental results have shown that FedCLWAvg achieved greater QoE estimation performance than the classical FedAvg algorithm in terms of mean accuracy and recall, F1-score, and precision computed for the single quality scores.
A Method for Generating a Non-Manual Feature Model for Sign Language Processing

While recent approaches to sign language processing have shifted to the domain of Machine Learning (ML), the treatment of Non-Manual Features (NMFs) remains an open question. The principal challenge facing this method is the comparatively small sign language corpora available for training machine learning models.This study produces a statistical model which may be used in future ML, rules-based, and hybrid-learning approaches for sign language processing tasks. In doing so, this research explores the emerging patterns of non-manual articulation concerning grammatical classes in Irish Sign Language (ISL). The experimental method applied here is a novel implementation of an association rules mining approach to a sign language dataset consisting of NMF and grammatical class data from the Signs of Ireland corpus.Our analysis of association rules has identified patterns between grammatical classes and various non-manual articulations. One such pattern discovery is the strong correlation between various NMFs and depicting verbs. Indeed, this study reports that the less lexicalised a sign is, the more likely it is to use NMFs.Findings from this work will inform future research on NMF treatment in sign language processing, while the statistical model may be utilised by such systems in the future.
EARP: Integration with Entity Attribute and Relation Path for Event Knowledge Graph Representation Learning

Event knowledge graph (EKG) as a special case of knowledge graph (KG) can realize the goal of event prediction, and has been proved useful in medical diagnosis and intelligent recommendation. To successfully build an EKG, knowledge representation learning is often required to compute the semantic links of entities and relationships in a low-dimensional space and solve the data sparsity issue in knowledge acquisition, fusion and reasoning. This paper proposes a new EKG representation learning model featuring the integration of event entity attributes and relation paths. By utilizing the knowledge of entity attribute, which contains entity type and entity description, and the knowledge about relation paths, the entity initial vector is obtained by multiplying entity semantic vector, entity description representation vector and entity type representation vector, and the representation of relation path is obtained according to the relation between event pairs, a translation-based model framework is used to integrate and train all vectors to obtain the entity learning vector and the relation learning vector. our method can generate more expressive learning representations, and consequently, enhance the inference performance of EKG. Experiments on publicly available real-world EKG datasets show that our method achieves better performance than the state-of-the-art models on two typical tasks.
Discourse Parsing on Multi-Granularity Interaction

Discourse parsing aims to construct a discourse structure tree to reflect the internal structure of a document. Most existing work only considers parsing documents from the paragraph-level or sentence-level granularity, ignoring the inter-action of different levels of granularity. Therefore, we propose a Multi-Granularity Interaction Method (MGIM) that facilitates discourse parsing through bidirectional information interaction at multi-granularity. We first introduce the structural information at the sentence level to boost paragraph-level parsing, and then use the functional pragmatics information at the paragraph level to guide sentence-level parsing. Moreover, we introduce an auxiliary task, discourse functional pragmatics recognition, to improve sentence-level parsing, which can guide sentence-level discourse tree construction from a macro perspective. Meanwhile, since the research field still lacks data for studying unified Chi-nese discourse parsing, we construct a Unified Chinese Discourse TreeBank UCDTB. Experimental results on both the Chinese UCDTB and the English RST-DT demonstrate the effectiveness of our proposed method.
Investigating Random Variations of the Forward-Forward Algorithm for Training Neural Networks

The Forward-forward (FF) algorithm is a new method for training neural networks, proposed as an alternative to the traditional Backpropagation (BP) algorithm by Hinton. The FF algorithm replaces the backward computations in the learning process with another forward pass. Each layer has an objective function, which aims to be high for positive data and low for negative ones. This paper presents a preliminary investigation into variations of the FF algorithm, such as incorporating a local Backpropagation to create a hybrid network that robustly converges while preserving the ability to avoid backward computations when needed, for example, in non-differentiable areas of the network. Additionally, a pseudo-random logic for selecting trainable stacks of layers at each epoch is proposed to speed up the learning process.