OpenRaiser/Intern-Atlas huggingface dataset:
Zeroth-Order Topological Insights into Iterative Magnitude Pruning
Venue: arXiv cs.LG 2022
Year: 2022

Modern-day neural networks are famously large, yet also highly redundant and compressible; there exist numerous pruning strategies in the deep learning literature that yield over 90% sparser sub-networks of fully-trained, dense architectures while still maintaining their original accuracies. Amongst these many methods though -- thanks to its conceptual simplicity, ease of implementation, and efficacy -- Iterative Magnitude Pruning (IMP) dominates in practice and is the de facto baseline to beat in the pruning community. However, theoretical explanations as to why a simplistic method such as IMP works at all are few and limited. In this work, we leverage the notion of persistent homology to gain insights into the workings of IMP and show that it inherently encourages retention of those weights which preserve topological information in a trained network. Subsequently, we also provide bounds on how much different networks can be pruned while perfectly preserving their zeroth order topological features, and present a modified version of IMP to do the same.
FreeTransfer-X: Safe and Label-Free Cross-Lingual Transfer from Off-the-Shelf Models
Venue: arXiv cs.CL 2022
Year: 2022

Cross-lingual transfer (CLT) is of various applications. However, labeled cross-lingual corpus is expensive or even inaccessible, especially in the fields where labels are private, such as diagnostic results of symptoms in medicine and user profiles in business. Nevertheless, there are off-the-shelf models in these sensitive fields. Instead of pursuing the original labels, a workaround for CLT is to transfer knowledge from the off-the-shelf models without labels. To this end, we define a novel CLT problem named FreeTransfer-X that aims to achieve knowledge transfer from the off-the-shelf models in rich-resource languages. To address the problem, we propose a 2-step knowledge distillation (KD, Hinton et al., 2015) framework based on multilingual pre-trained language models (mPLM). The significant improvement over strong neural machine translation (NMT) baselines demonstrates the effectiveness of the proposed method. In addition to reducing annotation cost and protecting private labels, the proposed method is compatible with different networks and easy to be deployed. Finally, a range of analyses indicate the great potential of the proposed method.
Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search
Venue: arXiv cs.IR 2022
Year: 2022

Improving the quality of search results can significantly enhance users experience and engagement with search engines. In spite of several recent advancements in the fields of machine learning and data mining, correctly classifying items for a particular user search query has been a long-standing challenge, which still has a large room for improvement. This paper introduces the "Shopping Queries Dataset", a large dataset of difficult Amazon search queries and results, publicly released with the aim of fostering research in improving the quality of search results. The dataset contains around 130 thousand unique queries and 2.6 million manually labeled (query,product) relevance judgements. The dataset is multilingual with queries in English, Japanese, and Spanish. The Shopping Queries Dataset is being used in one of the KDDCup'22 challenges. In this paper, we describe the dataset and present three evaluation tasks along with baseline results: (i) ranking the results list, (ii) classifying product results into relevance categories, and (iii) identifying substitute products for a given query. We anticipate that this data will become the gold standard for future research in the topic of product search.
Plug-and-Play Pseudo Label Correction Network for Unsupervised Person Re-identification
Venue: arXiv cs.CV 2022
Year: 2022

Clustering-based methods, which alternate between the generation of pseudo labels and the optimization of the feature extraction network, play a dominant role in both unsupervised learning (USL) and unsupervised domain adaptive (UDA) person re-identification (Re-ID). To alleviate the adverse effect of noisy pseudo labels, the existing methods either abandon unreliable labels or refine the pseudo labels via mutual learning or label propagation. However, a great many erroneous labels are still accumulated because these methods mostly adopt traditional unsupervised clustering algorithms which rely on certain assumptions on data distribution and fail to capture the distribution of complex real-world data. In this paper, we propose the plug-and-play graph-based pseudo label correction network (GLC) to refine the pseudo labels in the manner of supervised clustering. GLC is trained to perceive the varying data distribution at each epoch of the self-training with the supervision of initial pseudo labels generated by any clustering method. It can learn to rectify the initial noisy labels by means of the relationship constraints between samples on the k Nearest Neighbor (kNN) graph and early-stop training strategy. Specifically, GLC learns to aggregate node features from neighbors and predict whether the nodes should be linked on the graph. Besides, GLC is optimized with 'early stop' before the noisy labels are severely memorized to prevent overfitting to noisy pseudo labels. Consequently, GLC improves the quality of pseudo labels though the supervision signals contain some noise, leading to better Re-ID performance. Extensive experiments in USL and UDA person Re-ID on Market-1501 and MSMT17 show that our method is widely compatible with various clustering-based methods and promotes the state-of-the-art performance consistently.
MDS Codes with Euclidean and Hermitian Hulls of Flexible Dimensions and Their Applications to EAQECCs
Venue: arXiv cs.IT 2022
Year: 2022

The hull of a linear code is the intersection of itself with its dual code with respect to certain inner product. Both Euclidean and Hermitian hulls are of theorical and practical significance. In this paper, we construct several new classes of MDS codes via (extended) generalized Reed-Solomon (GRS) codes and determine their Euclidean or Hermitian hulls. Specifically, four new classes of MDS codes with Hermitian hulls of flexible dimensions and six new classes of MDS codes with Euclidean hulls of flexible dimensions are constructed. For the former, we further construct four new classes of entanglement-assisted quantum error-correcting codes (EAQECCs) and four new classes of MDS EAQECCs of length $n>q+1$. For the latter, we also give some examples on Euclidean self-orthogonal and one-dimensional Euclidean hull MDS codes.
Effect of money heterogeneity on resource dependency in complex networks
Venue: arXiv cs.SI 2022
Year: 2022

Exchange of resources among individual components of a system is fundamental to systems like a social network of humans and a network of cities and villages. For various reasons, the human society has come up with the notion of money as a proxy for the resources. Here we extend the model of resource dependencies in networks that was recently proposed by one of us, by incorporating the concept of money so that the vertices of a network can sell and buy required resources among themselves. We simulate the model using the configuration model as a substrate for homogeneous as well as heterogeneous degree distributions and using various exchange strategies. We show that a moderate amount of initial heterogeneity in the money on the vertices can significantly improve the survivability of Scale-free networks but not that of homogeneous networks like the Erdos-Renyi network. Our work is a step towards understanding the effect of presence of money on the resource distribution dynamics in complex networks.
Automatic compile-time synthesis of entropy-optimal Boltzmann samplers
Venue: arXiv cs.SE 2022
Year: 2022

We present a famework for the automatic compilation of multi-parametric Boltzmann samplers for algebraic data types in Haskell. Our framework uses Template Haskell to synthesise efficient, entropy-optimal samplers generating random instances of user-declared algebraic data types. Users can control the outcome distribution through a pure, declarative interface. For instance, users can control the mean size and constructor frequencies of generated objects. We illustrate the effectiveness of our framework through a prototype generic-boltzmann-brain library showing that it is possible to control thousands of different parameters in systems of tens of thousands of ADTs. Our prototype framework synthesises Boltzmann samplers capable of rapidly generating random objects of sizes in the millions.
Severe Damage Recovery in Evolving Soft Robots through Differentiable Programming
Venue: arXiv cs.NE 2022
Year: 2022

Biological systems are very robust to morphological damage, but artificial systems (robots) are currently not. In this paper we present a system based on neural cellular automata, in which locomoting robots are evolved and then given the ability to regenerate their morphology from damage through gradient-based training. Our approach thus combines the benefits of evolution to discover a wide range of different robot morphologies, with the efficiency of supervised training for robustness through differentiable update rules. The resulting neural cellular automata are able to grow virtual robots capable of regaining more than 80\% of their functionality, even after severe types of morphological damage.
Efficient Private Storage of Sparse Machine Learning Data
Venue: arXiv cs.IT 2022
Year: 2022

We consider the problem of maintaining sparsity in private distributed storage of confidential machine learning data. In many applications, e.g., face recognition, the data used in machine learning algorithms is represented by sparse matrices which can be stored and processed efficiently. However, mechanisms maintaining perfect information-theoretic privacy require encoding the sparse matrices into randomized dense matrices. It has been shown that, under some restrictions on the storage nodes, sparsity can be maintained at the expense of relaxing the perfect information-theoretic privacy requirement, i.e., allowing some information leakage. In this work, we lift the restrictions imposed on the storage nodes and show that there exists a trade-off between sparsity and the achievable privacy guarantees. We focus on the setting of non-colluding nodes and construct a coding scheme that encodes the sparse input matrices into matrices with the desired sparsity level while limiting the information leakage.
Modelling Guidance in Software Engineering: A Systematic Literature Review
Venue: arXiv cs.SE 2022
Year: 2022

Despite potential benefits in Software Engineering (SE), adoption of software modelling in industry is low. Technical issues such as tool support have gained significant research before, but individual guidance and training have received little attention. As a first step towards providing the necessary guidance in modelling, we conduct a systematic literature review (SLR) to explore the current state of the art. We searched academic literature for modelling guidance, and selected 25 papers for full-text screening through three rounds of selection. We find research on modelling guidance to be fragmented, with inconsistent usage of terminology, and a lack of empirical validation or supporting evidence. We outline the different dimensions commonly used to provide guidance on software modelling. Additionally, we provide definitions of the three terms modelling method, style, and guideline as current literature lacks a well-defined distinction between them. These definitions can help distinguishing between important concepts and provide precise modelling guidance.
How to Agree to Disagree: Managing Ontological Perspectives using Standpoint Logic
Venue: arXiv cs.AI 2022
Year: 2022

The importance of taking individual, potentially conflicting perspectives into account when dealing with knowledge has been widely recognised. Many existing ontology management approaches fully merge knowledge perspectives, which may require weakening in order to maintain consistency; others represent the distinct views in an entirely detached way. As an alternative, we propose Standpoint Logic, a simple, yet versatile multi-modal logic "add-on" for existing KR languages intended for the integrated representation of domain knowledge relative to diverse, possibly conflicting standpoints, which can be hierarchically organised, combined and put in relation to each other. Starting from the generic framework of First-Order Standpoint Logic (FOSL), we subsequently focus our attention on the fragment of sentential formulas, for which we provide a polytime translation into the standpoint-free version. This result yields decidability and favourable complexities for a variety of highly expressive decidable fragments of first-order logic. Using some elaborate encoding tricks, we then establish a similar translation for the very expressive description logic SROIQb_s underlying the OWL 2 DL ontology language. By virtue of this result, existing highly optimised OWL reasoners can be used to provide practical reasoning support for ontology languages extended by standpoint modelling.
Variance Reduction for Policy-Gradient Methods via Empirical Variance Minimization
Venue: arXiv cs.LG 2022
Year: 2022

Policy-gradient methods in Reinforcement Learning(RL) are very universal and widely applied in practice but their performance suffers from the high variance of the gradient estimate. Several procedures were proposed to reduce it including actor-critic(AC) and advantage actor-critic(A2C) methods. Recently the approaches have got new perspective due to the introduction of Deep RL: both new control variates(CV) and new sub-sampling procedures became available in the setting of complex models like neural networks. The vital part of CV-based methods is the goal functional for the training of the CV, the most popular one is the least-squares criterion of A2C. Despite its practical success, the criterion is not the only one possible. In this paper we for the first time investigate the performance of the one called Empirical Variance(EV). We observe in the experiments that not only EV-criterion performs not worse than A2C but sometimes can be considerably better. Apart from that, we also prove some theoretical guarantees of the actual variance reduction under very general assumptions and show that A2C least-squares goal functional is an upper bound for EV goal. Our experiments indicate that in terms of variance reduction EV-based methods are much better than A2C and allow stronger variance reduction.
Robust Reinforcement Learning with Distributional Risk-averse formulation
Venue: arXiv cs.LG 2022
Year: 2022

Robust Reinforcement Learning tries to make predictions more robust to changes in the dynamics or rewards of the system. This problem is particularly important when the dynamics and rewards of the environment are estimated from the data. In this paper, we approximate the Robust Reinforcement Learning constrained with a $Φ$-divergence using an approximate Risk-Averse formulation. We show that the classical Reinforcement Learning formulation can be robustified using standard deviation penalization of the objective. Two algorithms based on Distributional Reinforcement Learning, one for discrete and one for continuous action spaces are proposed and tested in a classical Gym environment to demonstrate the robustness of the algorithms.
An Accurate HDDL Domain Learning Algorithm from Partial and Noisy Observations
Venue: arXiv cs.AI 2022
Year: 2022

The Hierarchical Task Network ({\sf HTN}) formalism is very expressive and used to express a wide variety of planning problems. In contrast to the classical {\sf STRIPS} formalism in which only the action model needs to be specified, the {\sf HTN} formalism requires to specify, in addition, the tasks of the problem and their decomposition into subtasks, called {\sf HTN} methods. For this reason, hand-encoding {\sf HTN} problems is considered more difficult and more error-prone by experts than classical planning problem. To tackle this problem, we propose a new approach (HierAMLSI) based on grammar induction to acquire {\sf HTN} planning domain knowledge, by learning action models and {\sf HTN} methods with their preconditions. Unlike other approaches, HierAMLSI is able to learn both actions and methods with noisy and partial inputs observation with a high level or accuracy.
Manifold Alignment-Based Multi-Fidelity Reduced-Order Modeling Applied to Structural Analysis
Venue: arXiv cs.LG 2022
Year: 2022

This work presents the application of a recently developed parametric, non-intrusive, and multi-fidelity reduced-order modeling method on high-dimensional displacement and stress fields arising from the structural analysis of geometries that differ in the size of discretization and structural topology.The proposed approach leverages manifold alignment to fuse inconsistent field outputs from high- and low-fidelity simulations by individually projecting their solution onto a common subspace. The effectiveness of the method is demonstrated on two multi-fidelity scenarios involving the structural analysis of a benchmark wing geometry. Results show that outputs from structural simulations using incompatible grids, or related yet different topologies, are easily combined into a single predictive model, thus eliminating the need for additional pre-processing of the data. The new multi-fidelity reduced-order model achieves a relatively higher predictive accuracy at a lower computational cost when compared to a single-fidelity model.
A Multi-task Framework for Infrared Small Target Detection and Segmentation
Venue: arXiv cs.CV 2022
Year: 2022

Due to the complicated background and noise of infrared images, infrared small target detection is one of the most difficult problems in the field of computer vision. In most existing studies, semantic segmentation methods are typically used to achieve better results. The centroid of each target is calculated from the segmentation map as the detection result. In contrast, we propose a novel end-to-end framework for infrared small target detection and segmentation in this paper. First, with the use of UNet as the backbone to maintain resolution and semantic information, our model can achieve a higher detection accuracy than other state-of-the-art methods by attaching a simple anchor-free head. Then, a pyramid pool module is used to further extract features and improve the precision of target segmentation. Next, we use semantic segmentation tasks that pay more attention to pixel-level features to assist in the training process of object detection, which increases the average precision and allows the model to detect some targets that were previously not detectable. Furthermore, we develop a multi-task framework for infrared small target detection and segmentation. Our multi-task learning model reduces complexity by nearly half and speeds up inference by nearly twice compared to the composite single-task model, while maintaining accuracy. The code and models are publicly available at https://github.com/Chenastron/MTUNet.
Private Collection Matching Protocols
Venue: arXiv cs.CR 2022
Year: 2022

We introduce Private Collection Matching (PCM) problems, in which a client aims to determine whether a collection of sets owned by a server matches their interests. Existing privacy-preserving cryptographic primitives cannot solve PCM problems efficiently without harming privacy. We propose a modular framework that enables designers to build privacy-preserving PCM systems that output one bit: whether a collection of server sets matches the client's set. The communication cost of our protocols scales linearly with the size of the client's set and is independent of the number of server elements. We demonstrate the potential of our framework by designing and implementing novel solutions for two real-world PCM problems: determining whether a dataset has chemical compounds of interest, and determining whether a document collection has relevant documents. Our evaluation shows that we offer a privacy gain with respect to existing works at a reasonable communication and computation cost.
Cross-Chain State Machine Replication
Venue: arXiv cs.DC 2022
Year: 2022

This paper considers the classical state machine replication (SMR) problem in a distributed system model inspired by cross-chain exchanges. We propose a novel SMR protocol adapted for this model. Each state machine transition takes $O(n)$ message delays, where $n$ is the number of active participants, of which any number may be Byzantine. This protocol makes novel use of path signatures to keep replicas consistent. This protocol design cleanly separates application logic from fault-tolerance, providing a systematic way to replace complex ad-hoc cross-chain protocols with a more principled approach.
Lazy Queries Can Reduce Variance in Zeroth-order Optimization
Venue: arXiv cs.LG 2022
Year: 2022

A major challenge of applying zeroth-order (ZO) methods is the high query complexity, especially when queries are costly. We propose a novel gradient estimation technique for ZO methods based on adaptive lazy queries that we term as LAZO. Different from the classic one-point or two-point gradient estimation methods, LAZO develops two alternative ways to check the usefulness of old queries from previous iterations, and then adaptively reuses them to construct the low-variance gradient estimates. We rigorously establish that through judiciously reusing the old queries, LAZO can reduce the variance of stochastic gradient estimates so that it not only saves queries per iteration but also achieves the regret bound for the symmetric two-point method. We evaluate the numerical performance of LAZO, and demonstrate the low-variance property and the performance gain of LAZO in both regret and query complexity relative to several existing ZO methods. The idea of LAZO is general, and can be applied to other variants of ZO methods.
Learning Hands-On Electronics from Home: A Simulator for Fritzing
Venue: arXiv cs.RO 2022
Year: 2022

The recent pandemic has forced us to teach online, which is especially difficult for hand-on courses in robotics, like basic electronics. In this paper, I present a simulator which tries to replicate the same experience students will encounter during the exercises in the laboratory. The simulator has been developed in Fritzing, uses realistic multimeters for measuring and checks for common mistakes. Results show not only that the simulator was extremely useful during the pandemic, but also that it can supplement laboratory exercises when teaching in-classroom.
Frequency-centroid features for word recognition of non-native English speakers
Venue: arXiv cs.SD 2022
Year: 2022

The objective of this work is to investigate complementary features which can aid the quintessential Mel frequency cepstral coefficients (MFCCs) in the task of closed, limited set word recognition for non-native English speakers of different mother-tongues. Unlike the MFCCs, which are derived from the spectral energy of the speech signal, the proposed frequency-centroids (FCs) encapsulate the spectral centres of the different bands of the speech spectrum, with the bands defined by the Mel filterbank. These features, in combination with the MFCCs, are observed to provide relative performance improvement in English word recognition, particularly under varied noisy conditions. A two-stage Convolution Neural Network (CNN) is used to model the features of the English words uttered with Arabic, French and Spanish accents.