OpenRaiser/Intern-Atlas huggingface dataset:
A Novel ISAC Transmission Framework based on Spatially-Spread Orthogonal Time Frequency Space Modulation
Venue: arXiv cs.IT 2021
Year: 2021

In this paper, we propose a novel integrated sensing and communication (ISAC) transmission framework based on the spatially-spread orthogonal time frequency space (SS-OTFS) modulation by considering the fact that communication channel strengths cannot be directly obtained from radar sensing. We first propose the concept of SS-OTFS modulation, where the key novelty is the angular domain discretization enabled by the spatial-spreading/de-spreading. This discretization gives rise to simple and insightful effective models for both radar sensing and communication, which result in simplified designs for the related estimation and detection problems. In particular, we design simple beam tracking, angle estimation, and power allocation schemes for radar sensing, by utilizing the special structure of the effective radar sensing matrix. Meanwhile, we provide a detailed analysis on the pair-wise error probability (PEP) for communication, which unveils the key conditions for both precoding and power allocation designs. Based on those conditions, we design a symbol-wise precoding scheme for communication based only on the delay, Doppler, and angle estimates from radar sensing, without the a priori knowledge of the communication channel fading coefficients, and also introduce the power allocation for communication. Furthermore, we notice that radar sensing and communication requires different power allocations. Therefore, we discuss the performances of both the radar sensing and communication with different power allocations and show that the power allocation should be designed leaning towards radar sensing in practical scenarios. The effectiveness of the proposed ISAC transmission framework is verified by our numerical results, which also agree with our analysis and discussions.
Position Masking for Improved Layout-Aware Document Understanding
Venue: arXiv cs.CL 2021
Year: 2021

Natural language processing for document scans and PDFs has the potential to enormously improve the efficiency of business processes. Layout-aware word embeddings such as LayoutLM have shown promise for classification of and information extraction from such documents. This paper proposes a new pre-training task called that can improve performance of layout-aware word embeddings that incorporate 2-D position embeddings. We compare models pre-trained with only language masking against models pre-trained with both language masking and position masking, and we find that position masking improves performance by over 5% on a form understanding task.
Planning from video game descriptions
Venue: arXiv cs.AI 2021
Year: 2021

This project proposes a methodology for the automatic generation of action models from video game dynamics descriptions, as well as its integration with a planning agent for the execution and monitoring of the plans. Planners use these action models to get the deliberative behaviour for an agent in many different video games and, combined with a reactive module, solve deterministic and no-deterministic levels. Experimental results validate the methodology and prove that the effort put by a knowledge engineer can be greatly reduced in the definition of such complex domains. Furthermore, benchmarks of the domains has been produced that can be of interest to the international planning community to evaluate planners in international planning competitions.
From Movement Kinematics to Object Properties: Online Recognition of Human Carefulness
Venue: arXiv cs.RO 2021
Year: 2021

When manipulating objects, humans finely adapt their motions to the characteristics of what they are handling. Thus, an attentive observer can foresee hidden properties of the manipulated object, such as its weight, temperature, and even whether it requires special care in manipulation. This study is a step towards endowing a humanoid robot with this last capability. Specifically, we study how a robot can infer online, from vision alone, whether or not the human partner is careful when moving an object. We demonstrated that a humanoid robot could perform this inference with high accuracy (up to 81.3%) even with a low-resolution camera. Only for short movements without obstacles, carefulness recognition was insufficient. The prompt recognition of movement carefulness from observing the partner's action will allow robots to adapt their actions on the object to show the same degree of care as their human partners.
Towards Learning a Vocabulary of Visual Concepts and Operators using Deep Neural Networks
Venue: arXiv cs.CV 2021
Year: 2021

Deep neural networks have become the default choice for many applications like image and video recognition, segmentation and other image and video related tasks.However, a critical challenge with these models is the lack of explainability.This requirement of generating explainable predictions has motivated the research community to perform various analysis on trained models.In this study, we analyze the learned feature maps of trained models using MNIST images for achieving more explainable predictions.Our study is focused on deriving a set of primitive elements, here called visual concepts, that can be used to generate any arbitrary sample from the data generating distribution.We derive the primitive elements from the feature maps learned by the model.We illustrate the idea by generating visual concepts from a Variational Autoencoder trained using MNIST images.We augment the training data of MNIST dataset by adding about 60,000 new images generated with visual concepts chosen at random.With this we were able to reduce the reconstruction loss (mean square error) from an initial value of 120 without augmentation to 60 with augmentation.Our approach is a first step towards the final goal of achieving trained deep neural network models whose predictions, features in hidden layers and the learned filters can be well explained.Such a model when deployed in production can easily be modified to adapt to new data, whereas existing deep learning models need a re training or fine tuning. This process again needs a huge number of data samples that are not easy to generate unless the model has good explainability.
Multistage Pruning of CNN Based ECG Classifiers for Edge Devices
Venue: arXiv cs.LG 2021
Year: 2021

Using smart wearable devices to monitor patients electrocardiogram (ECG) for real-time detection of arrhythmias can significantly improve healthcare outcomes. Convolutional neural network (CNN) based deep learning has been used successfully to detect anomalous beats in ECG. However, the computational complexity of existing CNN models prohibits them from being implemented in low-powered edge devices. Usually, such models are complex with lots of model parameters which results in large number of computations, memory, and power usage in edge devices. Network pruning techniques can reduce model complexity at the expense of performance in CNN models. This paper presents a novel multistage pruning technique that reduces CNN model complexity with negligible loss in performance compared to existing pruning techniques. An existing CNN model for ECG classification is used as a baseline reference. At 60% sparsity, the proposed technique achieves 97.7% accuracy and an F1 score of 93.59% for ECG classification tasks. This is an improvement of 3.3% and 9% for accuracy and F1 Score respectively, compared to traditional pruning with fine-tuning approach. Compared to the baseline model, we also achieve a 60.4% decrease in run-time complexity.
Fight Fire with Fire: Fine-tuning Hate Detectors using Large Samples of Generated Hate Speech
Venue: arXiv cs.CL 2021
Year: 2021

Automatic hate speech detection is hampered by the scarcity of labeled datasetd, leading to poor generalization. We employ pretrained language models (LMs) to alleviate this data bottleneck. We utilize the GPT LM for generating large amounts of synthetic hate speech sequences from available labeled examples, and leverage the generated data in fine-tuning large pretrained LMs on hate detection. An empirical study using the models of BERT, RoBERTa and ALBERT, shows that this approach improves generalization significantly and consistently within and across data distributions. In fact, we find that generating relevant labeled hate speech sequences is preferable to using out-of-domain, and sometimes also within-domain, human-labeled examples.
Learning compositional programs with arguments and sampling
Venue: arXiv cs.PL 2021
Year: 2021

One of the most challenging goals in designing intelligent systems is empowering them with the ability to synthesize programs from data. Namely, given specific requirements in the form of input/output pairs, the goal is to train a machine learning model to discover a program that satisfies those requirements. A recent class of methods exploits combinatorial search procedures and deep learning to learn compositional programs. However, they usually generate only toy programs using a domain-specific language that does not provide any high-level feature, such as function arguments, which reduces their applicability in real-world settings. We extend upon a state of the art model, AlphaNPI, by learning to generate functions that can accept arguments. This improvement will enable us to move closer to real computer programs. Moreover, we investigate employing an Approximate version of Monte Carlo Tree Search (A-MCTS) to speed up convergence. We showcase the potential of our approach by learning the Quicksort algorithm, showing how the ability to deal with arguments is crucial for learning and generalization.
Algorithme de recherche approximative dans un dictionnaire fondé sur une distance d'édition définie par blocs
Venue: arXiv cs.CL 2021
Year: 2021

We propose an algorithm for approximative dictionary lookup, where altered strings are matched against reference forms. The algorithm makes use of a divergence function between strings -- broadly belonging to the family of edit distances; it finds dictionary entries whose distance to the search string is below a certain threshold. The divergence function is not the classical edit distance (DL distance); it is adaptable to a particular corpus, and is based on elementary alteration costs defined on character blocks, rather than on individual characters. Nous proposons un algorithme de recherche approximative de chaînes dans un dictionnaire à partir de formes altérées. Cet algorithme est fondé sur une fonction de divergence entre chaînes~ -- une sorte de distance d'édition: il recherche des entrées pour lesquelles la distance à la chaîne cherchée est inférieure à un certain seuil. La fonction utilisée n'est pas la distance d'édition classique (distance DL); elle est adaptée à un corpus, et se fonde sur la prise en compte de coûts d'altération élémentaires définis non pas sur des caractères, mais sur des sous-chaînes (des blocs de caractères).
Searching for Efficient Multi-Stage Vision Transformers
Venue: arXiv cs.CV 2021
Year: 2021

Vision Transformer (ViT) demonstrates that Transformer for natural language processing can be applied to computer vision tasks and result in comparable performance to convolutional neural networks (CNN), which have been studied and adopted in computer vision for years. This naturally raises the question of how the performance of ViT can be advanced with design techniques of CNN. To this end, we propose to incorporate two techniques and present ViT-ResNAS, an efficient multi-stage ViT architecture designed with neural architecture search (NAS). First, we propose residual spatial reduction to decrease sequence lengths for deeper layers and utilize a multi-stage architecture. When reducing lengths, we add skip connections to improve performance and stabilize training deeper networks. Second, we propose weight-sharing NAS with multi-architectural sampling. We enlarge a network and utilize its sub-networks to define a search space. A super-network covering all sub-networks is then trained for fast evaluation of their performance. To efficiently train the super-network, we propose to sample and train multiple sub-networks with one forward-backward pass. After that, evolutionary search is performed to discover high-performance network architectures. Experiments on ImageNet demonstrate that ViT-ResNAS achieves better accuracy-MACs and accuracy-throughput trade-offs than the original DeiT and other strong baselines of ViT. Code is available at https://github.com/yilunliao/vit-search.
Agon: A Scalable Competitive Scheduler for Large Heterogeneous Systems
Venue: arXiv cs.AR 2021
Year: 2021

This work proposes a competitive scheduling approach, designed to scale to large heterogeneous multicore systems. This scheduler overcomes the challenges of (1) the high computation overhead of near-optimal schedulers, and (2) the error introduced by inaccurate performance predictions. This paper presents Agon, a neural network-based classifier that selects from a range of schedulers, from simple to very accurate, and learns which scheduler provides the right balance of accuracy and overhead for each scheduling interval. Agon also employs a de-noising frontend allowing the individual schedulers to be tolerant towards noise in performance predictions, producing better overall schedules. By avoiding expensive scheduling overheads, Agon improves average system performance by 6\% on average, approaching the performance of an oracular scheduler (99.1% of oracle performance).
TabFairGAN: Fair Tabular Data Generation with Generative Adversarial Networks
Venue: arXiv cs.LG 2021
Year: 2021

With the increasing reliance on automated decision making, the issue of algorithmic fairness has gained increasing importance. In this paper, we propose a Generative Adversarial Network for tabular data generation. The model includes two phases of training. In the first phase, the model is trained to accurately generate synthetic data similar to the reference dataset. In the second phase we modify the value function to add fairness constraint, and continue training the network to generate data that is both accurate and fair. We test our results in both cases of unconstrained, and constrained fair data generation. In the unconstrained case, i.e. when the model is only trained in the first phase and is only meant to generate accurate data following the same joint probability distribution of the real data, the results show that the model beats state-of-the-art GANs proposed in the literature to produce synthetic tabular data. Also, in the constrained case in which the first phase of training is followed by the second phase, we train the network and test it on four datasets studied in the fairness literature and compare our results with another state-of-the-art pre-processing method, and present the promising results that it achieves. Comparing to other studies utilizing GANs for fair data generation, our model is comparably more stable by using only one critic, and also by avoiding major problems of original GAN model, such as mode-dropping and non-convergence, by implementing a Wasserstein GAN.
An Empirical Exploration in Quality Filtering of Text Data
Venue: arXiv cs.CL 2021
Year: 2021

While conventional wisdom suggests that more aggressively filtering data from low-quality sources like Common Crawl always monotonically improves the quality of training data, we find that aggressive filtering can in fact lead to a decrease in model quality on a wide array of downstream tasks for a GPT-like language model. We speculate that this is because optimizing sufficiently strongly for a proxy metric harms performance on the true objective, suggesting a need for more robust filtering objectives when attempting to filter more aggressively. We hope this work leads to detailed analysis of the effects of dataset filtering design choices on downstream model performance in future work.
An Information-Theoretic View of Stochastic Localization
Venue: arXiv cs.IT 2021
Year: 2021

Given a probability measure $μ$ over ${\mathbb R}^n$, it is often useful to approximate it by the convex combination of a small number of probability measures, such that each component is close to a product measure. Recently, Ronen Eldan used a stochastic localization argument to prove a general decomposition result of this type. In Eldan's theorem, the `number of components' is characterized by the entropy of the mixture, and `closeness to product' is characterized by the covariance matrix of each component. We present an elementary proof of Eldan's theorem which makes use of an information theory (or estimation theory) interpretation. The proof is analogous to the one of an earlier decomposition result known as the `pinning lemma.'
ConQX: Semantic Expansion of Spoken Queries for Intent Detection based on Conditioned Text Generation
Venue: arXiv cs.CL 2021
Year: 2021

Intent detection of spoken queries is a challenging task due to their noisy structure and short length. To provide additional information regarding the query and enhance the performance of intent detection, we propose a method for semantic expansion of spoken queries, called ConQX, which utilizes the text generation ability of an auto-regressive language model, GPT-2. To avoid off-topic text generation, we condition the input query to a structured context with prompt mining. We then apply zero-shot, one-shot, and few-shot learning. We lastly use the expanded queries to fine-tune BERT and RoBERTa for intent detection. The experimental results show that the performance of intent detection can be improved by our semantic expansion method.
Quaternions over Galois rings and their codes
Venue: arXiv cs.IT 2021
Year: 2021

It is shown in this paper that, if $R$ is a Frobenius ring, then the quaternion ring $\mathcal{H}_{a,b}(R)$ is a Frobenius ring for all units $a,b \in R$. In particular, if $q$ is an odd prime power then $\mathcal{H}_{a,b}(\mathbb{F}_q)$ is the semisimple non-commutative matrix ring $M_2(\mathbb{F}_q)$. Consequently, a homogeneous weight that depends on the field size $q$ is obtained. On the other hand, the homogeneous weight of a finite Frobenius ring with a unique minimal ideal is derived in terms of the size of the ideal. This is illustrated by the quaternions over the Galois ring $GR(2^r,m)$. Finally, one-sided linear block codes over the quaternions over Galois rings are constructed, and certain bounds on the homogeneous distance of the images of these codes are proved. These bounds are based on the Hamming distance of the quaternion code and the parameters of the Galois ring. Good examples of one-sided rate-2/6, 3-quasi-cyclic quaternion codes and their images are generated. One of these codes meets the Singleton bound and is therefore a maximum distance separable code.
dbcsp: User-friendly R package for Distance-Based Common Spacial Patterns
Venue: arXiv cs.MS 2021
Year: 2021

Common Spacial Patterns (CSP) is a widely used method to analyse electroencephalography (EEG) data, concerning the supervised classification of brain's activity. More generally, it can be useful to distinguish between multivariate signals recorded during a time span for two different classes. CSP is based on the simultaneous diagonalization of the average covariance matrices of signals from both classes and it allows to project the data into a low-dimensional subspace. Once data are represented in a low-dimensional subspace, a classification step must be carried out. The original CSP method is based on the Euclidean distance between signals and here, we extend it so that it can be applied on any appropriate distance for data at hand. Both, the classical CSP and the new Distance-Based CSP (DB-CSP) are implemented in an R package, called dbcsp.
Transfer of Pretrained Model Weights Substantially Improves Semi-Supervised Image Classification
Venue: arXiv cs.CV 2021
Year: 2021

Deep neural networks produce state-of-the-art results when trained on a large number of labeled examples but tend to overfit when small amounts of labeled examples are used for training. Creating a large number of labeled examples requires considerable resources, time, and effort. If labeling new data is not feasible, so-called semi-supervised learning can achieve better generalisation than purely supervised learning by employing unlabeled instances as well as labeled ones. The work presented in this paper is motivated by the observation that transfer learning provides the opportunity to potentially further improve performance by exploiting models pretrained on a similar domain. More specifically, we explore the use of transfer learning when performing semi-supervised learning using self-learning. The main contribution is an empirical evaluation of transfer learning using different combinations of similarity metric learning methods and label propagation algorithms in semi-supervised learning. We find that transfer learning always substantially improves the model's accuracy when few labeled examples are available, regardless of the type of loss used for training the neural network. This finding is obtained by performing extensive experiments on the SVHN, CIFAR10, and Plant Village image classification datasets and applying pretrained weights from Imagenet for transfer learning.
Built Year Prediction from Buddha Face with Heterogeneous Labels
Venue: arXiv cs.CV 2021
Year: 2021

Buddha statues are a part of human culture, especially of the Asia area, and they have been alongside human civilisation for more than 2,000 years. As history goes by, due to wars, natural disasters, and other reasons, the records that show the built years of Buddha statues went missing, which makes it an immense work for historians to estimate the built years. In this paper, we pursue the idea of building a neural network model that automatically estimates the built years of Buddha statues based only on their face images. Our model uses a loss function that consists of three terms: an MSE loss that provides the basis for built year estimation; a KL divergence-based loss that handles the samples with both an exact built year and a possible range of built years (e.g., dynasty or centuries) estimated by historians; finally a regularisation that utilises both labelled and unlabelled samples based on manifold assumption. By combining those three terms in the training process, we show that our method is able to estimate built years for given images with 37.5 years of a mean absolute error on the test set.
Adherence and Constancy in LIME-RS Explanations for Recommendation
Venue: arXiv cs.IR 2021
Year: 2021

Explainable Recommendation has attracted a lot of attention due to a renewed interest in explainable artificial intelligence. In particular, post-hoc approaches have proved to be the most easily applicable ones to increasingly complex recommendation models, which are then treated as black-boxes. The most recent literature has shown that for post-hoc explanations based on local surrogate models, there are problems related to the robustness of the approach itself. This consideration becomes even more relevant in human-related tasks like recommendation. The explanation also has the arduous task of enhancing increasingly relevant aspects of user experience such as transparency or trustworthiness. This paper aims to show how the characteristics of a classical post-hoc model based on surrogates is strongly model-dependent and does not prove to be accountable for the explanations generated.