OpenRaiser/Intern-Atlas huggingface dataset:
Distilling EEG Representations via Capsules for Affective Computing
Venue: arXiv cs.LG 2021
Year: 2021

Affective computing with Electroencephalogram (EEG) is a challenging task that requires cumbersome models to effectively learn the information contained in large-scale EEG signals, causing difficulties for real-time smart-device deployment. In this paper, we propose a novel knowledge distillation pipeline to distill EEG representations via capsule-based architectures for both classification and regression tasks. Our goal is to distill information from a heavy model to a lightweight model for subject-specific tasks. To this end, we first pre-train a large model (teacher network) on large number of training samples. Then, we employ the teacher network to learn the discriminative features embedded in capsules by adopting a lightweight model (student network) to mimic the teacher using the privileged knowledge. Such privileged information learned by the teacher contain similarities among capsules and are only available during the training stage of the student network. We evaluate the proposed architecture on two large-scale public EEG datasets, showing that our framework consistently enables student networks with different compression ratios to effectively learn from the teacher, even when provided with limited training samples. Lastly, our method achieves state-of-the-art results on one of the two datasets.
Cross-Modal Self-Attention with Multi-Task Pre-Training for Medical Visual Question Answering
Venue: arXiv cs.MM 2021
Year: 2021

Due to the severe lack of labeled data, existing methods of medical visual question answering usually rely on transfer learning to obtain effective image feature representation and use cross-modal fusion of visual and linguistic features to achieve question-related answer prediction. These two phases are performed independently and without considering the compatibility and applicability of the pre-trained features for cross-modal fusion. Thus, we reformulate image feature pre-training as a multi-task learning paradigm and witness its extraordinary superiority, forcing it to take into account the applicability of features for the specific image comprehension task. Furthermore, we introduce a cross-modal self-attention~(CMSA) module to selectively capture the long-range contextual relevance for more effective fusion of visual and linguistic features. Experimental results demonstrate that the proposed method outperforms existing state-of-the-art methods. Our code and models are available at https://github.com/haifangong/CMSA-MTPT-4-MedicalVQA.
Theoretical Analysis for Determining Geographical Route of Cable Network with Various Disaster-Endurance Levels
Venue: arXiv cs.NI 2021
Year: 2021

This paper theoretically analyzes cable network disconnection due to randomly occurring natural disasters, where the disaster-endurance (DE) levels of the network are determined by a network entity such as the type of shielding method used for a duct containing cables. The network operator can determine which parts have a high DE level. When a part of a network can be protected, the placement of that part can be specified to decrease the probability of disconnecting two given nodes. The maximum lower bound of the probability of connecting two given nodes is explicitly derived. Conditions decreasing (not decreasing) the probability of connecting two given nodes with a partially protected network are provided.
Probabilistic Analysis of Operating Modes in Cache-Enabled Full-Duplex D2D Networks
Venue: arXiv cs.NI 2021
Year: 2021

With the extensive acquisition of various mobile applications, cellular networks are facing challenges due to exponentially growing demand for high data rate, which causes a great burden on mobile core networks and backhaul links. Cache-enabled Device-to-Device (D2D) communication, which is recognized as one of the key enablers of the fifth generation (5G) cellular network, is a promising solution to alleviate this problem. However, conventional half-duplex (HD) communication may not be sufficient to provide fast enough content delivery over D2D links in order to meet strict latency targets of emerging D2D applications. In-band full-duplex (FD), with its capability of allowing simultaneous transmission and reception, can provide more content delivery opportunities, thus resulting improved spectral efficiency and latency reduction. However, given the random nature of the cached contents in user devices and users' random requests, it is unlikely to consider all involving nodes in content exchange collaborations as a pure HD or FD network. In this paper, we aim to analyze the caching perspective of a finite network of D2D nodes in which each node is endowed with FD capability and utilize a more realistic caching policy. We model and analyze all possible operating modes for an arbitrary device, which we compute the probability of occurrence of each mode along with the Probability Mass Functions (PMFs) of nodes that are operating in all possible modes. Our analysis concretely quantize all possible outcomes that strongly depend on the random nature of the caching parameters, yielding to have an accurate insight on the caching performance and all possible outcomes of the cache-enabled FD-D2D network.
A Brief Survey And Investigation Of Hybrid Beamforming For Millimeter Waves In 5G Massive MIMO Systems
Venue: arXiv cs.IT 2021
Year: 2021

Millimeter-wave (mm-wave) is a promising technique to enhance the network capacity and coverage of next-generation (5G) based on utilizing a great number of available spectrum resources in mobile communication. Improving the 5G network requires enhancing and employing mm-wave beamforming channel propagation characteristics. To achieve high data rates, system performance remains a challenge given the impact of propagation channels in mm-wave that is insufficient in both path loss, delay spread, and penetration loss. Additional challenges arise due to high cost and energy consumption, which require combining both analog and digital beamforming (hybrid beamforming) to reduce the number of radio frequency (RF) chains. In this paper, the distributed powers in the small cell to suppress path loss by specifying a considerable power and controlling the distributed power to reduce the high cost and energy consumption was proposed. The hybrid beamforming in mm-wave exploits a large bandwidth which reduces the large path loss in Rayleigh fading channel. Also, the trade-off between the energy consumption of RF chains and cost efficiency depends on reducing the number of RF chains and the distributed number of users. This paper finds that hybrid beamforming for massive multiple-input multiple-output (MIMO) systems constitute a promising platform for advancing and capitalizing on 5G networks
Stochastic Mutual Information Gradient Estimation for Dimensionality Reduction Networks
Venue: arXiv cs.LG 2021
Year: 2021

Feature ranking and selection is a widely used approach in various applications of supervised dimensionality reduction in discriminative machine learning. Nevertheless there exists significant evidence on feature ranking and selection algorithms based on any criterion leading to potentially sub-optimal solutions for class separability. In that regard, we introduce emerging information theoretic feature transformation protocols as an end-to-end neural network training approach. We present a dimensionality reduction network (MMINet) training procedure based on the stochastic estimate of the mutual information gradient. The network projects high-dimensional features onto an output feature space where lower dimensional representations of features carry maximum mutual information with their associated class labels. Furthermore, we formulate the training objective to be estimated non-parametrically with no distributional assumptions. We experimentally evaluate our method with applications to high-dimensional biological data sets, and relate it to conventional feature selection algorithms to form a special case of our approach.
Detecting Deletions and Insertions in Concatenated Strings with Optimal Redundancy
Venue: arXiv cs.IT 2021
Year: 2021

We study codes that can detect the exact number of deletions and insertions in concatenated binary strings. We construct optimal codes for the case of detecting up to $\del$ deletions. We prove the optimality of these codes by deriving a converse result which shows that the redundancy of our codes is asymptotically optimal in $\del$ among all families of deletion detecting codes, and particularly optimal among all block-by-block decodable codes. For the case of insertions, we construct codes that can detect up to $2$ insertions in each concatenated binary string.
Signature-Based Abduction with Fresh Individuals and Complex Concepts for Description Logics (Extended Version)
Venue: arXiv cs.AI 2021
Year: 2021

Given a knowledge base and an observation as a set of facts, ABox abduction aims at computing a hypothesis that, when added to the knowledge base, is sufficient to entail the observation. In signature-based ABox abduction, the hypothesis is further required to use only names from a given set. This form of abduction has applications such as diagnosis, KB repair, or explaining missing entailments. It is possible that hypotheses for a given observation only exist if we admit the use of fresh individuals and/or complex concepts built from the given signature, something most approaches for ABox abduction so far do not support or only support with restrictions. In this paper, we investigate the computational complexity of this form of abduction -- allowing either fresh individuals, complex concepts, or both -- for various description logics, and give size bounds on the hypotheses if they exist.
Non-asymptotic Excess Risk Bounds for Classification with Deep Convolutional Neural Networks
Venue: arXiv cs.LG 2021
Year: 2021

In this paper, we consider the problem of binary classification with a class of general deep convolutional neural networks, which includes fully-connected neural networks and fully convolutional neural networks as special cases. We establish non-asymptotic excess risk bounds for a class of convex surrogate losses and target functions with different modulus of continuity. An important feature of our results is that we clearly define the prefactors of the risk bounds in terms of the input data dimension and other model parameters and show that they depend polynomially on the dimensionality in some important models. We also show that the classification methods with CNNs can circumvent the curse of dimensionality if the input data is supported on an approximate low-dimensional manifold. To establish these results, we derive an upper bound for the covering number for the class of general convolutional neural networks with a bias term in each convolutional layer, and derive new results on the approximation power of CNNs for any uniformly-continuous target functions. These results provide further insights into the complexity and the approximation power of general convolutional neural networks, which are of independent interest and may have other applications. Finally, we apply our general results to analyze the non-asymptotic excess risk bounds for four widely used methods with different loss functions using CNNs, including the least squares, the logistic, the exponential and the SVM hinge losses.
PREDICT: Persian Reverse Dictionary
Venue: arXiv cs.CL 2021
Year: 2021

Finding the appropriate words to convey concepts (i.e., lexical access) is essential for effective communication. Reverse dictionaries fulfill this need by helping individuals to find the word(s) which could relate to a specific concept or idea. To the best of our knowledge, this resource has not been available for the Persian language. In this paper, we compare four different architectures for implementing a Persian reverse dictionary (PREDICT). We evaluate our models using (phrase,word) tuples extracted from the only Persian dictionaries available online, namely Amid, Moein, and Dehkhoda where the phrase describes the word. Given the phrase, a model suggests the most relevant word(s) in terms of the ability to convey the concept. The model is considered to perform well if the correct word is one of its top suggestions. Our experiments show that a model consisting of Long Short-Term Memory (LSTM) units enhanced by an additive attention mechanism is enough to produce suggestions comparable to (or in some cases better than) the word in the original dictionary. The study also reveals that the model sometimes produces the synonyms of the word as its output which led us to introduce a new metric for the evaluation of reverse dictionaries called Synonym Accuracy accounting for the percentage of times the event of producing the word or a synonym of it occurs. The assessment of the best model using this new metric also indicates that at least 62% of the times, it produces an accurate result within the top 100 suggestions.
MARL: Multimodal Attentional Representation Learning for Disease Prediction
Venue: arXiv cs.CV 2021
Year: 2021

Existing learning models often utilise CT-scan images to predict lung diseases. These models are posed by high uncertainties that affect lung segmentation and visual feature learning. We introduce MARL, a novel Multimodal Attentional Representation Learning model architecture that learns useful features from multimodal data under uncertainty. We feed the proposed model with both the lung CT-scan images and their perspective historical patients' biological records collected over times. Such rich data offers to analyse both spatial and temporal aspects of the disease. MARL employs Fuzzy-based image spatial segmentation to overcome uncertainties in CT-scan images. We then utilise a pre-trained Convolutional Neural Network (CNN) to learn visual representation vectors from images. We augment patients' data with statistical features from the segmented images. We develop a Long Short-Term Memory (LSTM) network to represent the augmented data and learn sequential patterns of disease progressions. Finally, we inject both CNN and LSTM feature vectors to an attention layer to help focus on the best learning features. We evaluated MARL on regression of lung disease progression and status classification. MARL outperforms state-of-the-art CNN architectures, such as EfficientNet and DenseNet, and baseline prediction models. It achieves a 91% R^2 score, which is higher than the other models by a range of 8% to 27%. Also, MARL achieves 97% and 92% accuracy for binary and multi-class classification, respectively. MARL improves the accuracy of state-of-the-art CNN models with a range of 19% to 57%. The results show that combining spatial and sequential temporal features produces better discriminative feature.
Neko: a Library for Exploring Neuromorphic Learning Rules
Venue: arXiv cs.LG 2021
Year: 2021

The field of neuromorphic computing is in a period of active exploration. While many tools have been developed to simulate neuronal dynamics or convert deep networks to spiking models, general software libraries for learning rules remain underexplored. This is partly due to the diverse, challenging nature of efforts to design new learning rules, which range from encoding methods to gradient approximations, from population approaches that mimic the Bayesian brain to constrained learning algorithms deployed on memristor crossbars. To address this gap, we present Neko, a modular, extensible library with a focus on aiding the design of new learning algorithms. We demonstrate the utility of Neko in three exemplar cases: online local learning, probabilistic learning, and analog on-device learning. Our results show that Neko can replicate the state-of-the-art algorithms and, in one case, lead to significant outperformance in accuracy and speed. Further, it offers tools including gradient comparison that can help develop new algorithmic variants. Neko is an open source Python library that supports PyTorch and TensorFlow backends.
Two-terminal Erasure Source-Broadcast with Feedback
Venue: arXiv cs.IT 2021
Year: 2021

We study the effects of introducing a feedback channel in the two-receiver erasure source-broadcast problem in which a binary equiprobable source is to be sent over an erasure broadcast channel to two receivers subject to erasure distortion constraints. The receivers each require a certain fraction of a source sequence, and we are interested in the minimum latency, or transmission time, required to serve them all. We first show that for a two-user broadcast channel, a point-to-point outer bound can always be achieved. We further show that the point-to-point outer bound can also be achieved if only one of the users, the stronger user, has a feedback channel. Our coding scheme relies on a hybrid approach that combines transmitting both random linear combinations of source symbols as well as a retransmission strategy.
AI-enabled Efficient and Safe Food Supply Chain
Venue: arXiv cs.LG 2021
Year: 2021

This paper provides a review of an emerging field in the food processing sector, referring to efficient and safe food supply chains, from farm to fork, as enabled by Artificial Intelligence (AI). Recent advances in machine and deep learning are used for effective food production, energy management and food labeling. Appropriate deep neural architectures are adopted and used for this purpose, including Fully Convolutional Networks, Long Short-Term Memories and Recurrent Neural Networks, Auto-Encoders and Attention mechanisms, Latent Variable extraction and clustering, as well as Domain Adaptation. Three experimental studies are presented, illustrating the ability of these AI methodologies to produce state-of-the-art performance in the whole food supply chain. In particular, these concern: (i) predicting plant growth and tomato yield in greenhouses, thus matching food production to market needs and reducing food waste or food unavailability; (ii) optimizing energy consumption across large networks of food retail refrigeration systems, through optimal selection of systems that can get shut-down and through prediction of the respective food de-freezing times, during peaks of power demand load; (iii) optical recognition and verification of food consumption expiry date in automatic inspection of retail packaged food, thus ensuring safety of food and people's health.
Three-terminal Erasure Source-Broadcast with Feedback
Venue: arXiv cs.IT 2021
Year: 2021

We study the effects of introducing a feedback channel in the erasure source-broadcast problem for the case of three receivers. In our problem formulation, we wish to transmit a binary source to three users over the erasure broadcast channel when causal feedback is universally available to the transmitter and all three receivers. Each receiver requires a certain fraction of the source sequence, and we are interested in the minimum latency, or transmission time, required to serve them all. We propose a queue-based hybrid digital-analog coding scheme that achieves optimal performance for the duration of analog transmissions. We characterize the number of analog transmissions that can be sent as the solution to a linear program and furthermore give sufficient conditions for which optimal performance can be achieved by all users. In some cases, we find that users can be point-to-point optimal regardless of their distortion constraints. When the analog transmissions are insufficient in meeting user demands, we propose two subsequent coding schemes. The first uses a queue preprocessing strategy for channel coding. The second method is a novel chaining algorithm, which involves the transmitter targeting point-to-point optimal performance for two users as if they were the only users in the network. Meanwhile, the third user simultaneously builds "chains" of symbols that he may at times be able to decode based on the channel conditions, and the subsequent reception of additional symbols.
Subspace Representation Learning for Few-shot Image Classification
Venue: arXiv cs.CV 2021
Year: 2021

In this paper, we propose a subspace representation learning (SRL) framework to tackle few-shot image classification tasks. It exploits a subspace in local CNN feature space to represent an image, and measures the similarity between two images according to a weighted subspace distance (WSD). When K images are available for each class, we develop two types of template subspaces to aggregate K-shot information: the prototypical subspace (PS) and the discriminative subspace (DS). Based on the SRL framework, we extend metric learning based techniques from vector to subspace representation. While most previous works adopted global vector representation, using subspace representation can effectively preserve the spatial structure, and diversity within an image. We demonstrate the effectiveness of the SRL framework on three public benchmark datasets: MiniImageNet, TieredImageNet and Caltech-UCSD Birds-200-2011 (CUB), and the experimental results illustrate competitive/superior performance of our method compared to the previous state-of-the-art.
pyBKT: An Accessible Python Library of Bayesian Knowledge Tracing Models
Venue: arXiv cs.MS 2021
Year: 2021

Bayesian Knowledge Tracing, a model used for cognitive mastery estimation, has been a hallmark of adaptive learning research and an integral component of deployed intelligent tutoring systems (ITS). In this paper, we provide a brief history of knowledge tracing model research and introduce pyBKT, an accessible and computationally efficient library of model extensions from the literature. The library provides data generation, fitting, prediction, and cross-validation routines, as well as a simple to use data helper interface to ingest typical tutor log dataset formats. We evaluate the runtime with various dataset sizes and compare to past implementations. Additionally, we conduct sanity checks of the model using experiments with simulated data to evaluate the accuracy of its EM parameter learning and use real-world data to validate its predictions, comparing pyBKT's supported model variants with results from the papers in which they were originally introduced. The library is open source and open license for the purpose of making knowledge tracing more accessible to communities of research and practice and to facilitate progress in the field through easier replication of past approaches.
Billion-scale Pre-trained E-commerce Product Knowledge Graph Model
Venue: arXiv cs.AI 2021
Year: 2021

In recent years, knowledge graphs have been widely applied to organize data in a uniform way and enhance many tasks that require knowledge, for example, online shopping which has greatly facilitated people's life. As a backbone for online shopping platforms, we built a billion-scale e-commerce product knowledge graph for various item knowledge services such as item recommendation. However, such knowledge services usually include tedious data selection and model design for knowledge infusion, which might bring inappropriate results. Thus, to avoid this problem, we propose a Pre-trained Knowledge Graph Model (PKGM) for our billion-scale e-commerce product knowledge graph, providing item knowledge services in a uniform way for embedding-based models without accessing triple data in the knowledge graph. Notably, PKGM could also complete knowledge graphs during servicing, thereby overcoming the common incompleteness issue in knowledge graphs. We test PKGM in three knowledge-related tasks including item classification, same item identification, and recommendation. Experimental results show PKGM successfully improves the performance of each task.