OpenRaiser/Intern-Atlas huggingface dataset:
Does QA-based intermediate training help fine-tuning language models for text classification?
Venue: arXiv cs.CL 2021
Year: 2021

Fine-tuning pre-trained language models for downstream tasks has become a norm for NLP. Recently it is found that intermediate training based on high-level inference tasks such as Question Answering (QA) can improve the performance of some language models for target tasks. However it is not clear if intermediate training generally benefits various language models. In this paper, using the SQuAD-2.0 QA task for intermediate training for target text classification tasks, we experimented on eight tasks for single-sequence classification and eight tasks for sequence-pair classification using two base and two compact language models. Our experiments show that QA-based intermediate training generates varying transfer performance across different language models, except for similar QA tasks.
A general technique for the estimation of farm animal body part weights from CT scans and its applications in a rabbit breeding program
Venue: arXiv cs.CV 2021
Year: 2021

Various applications of farm animal imaging are based on the estimation of weights of certain body parts and cuts from the CT images of animals. In many cases, the complexity of the problem is increased by the enormous variability of postures in CT images due to the scanning of non-sedated, living animals. In this paper, we propose a general and robust approach for the estimation of the weights of cuts and body parts from the CT images of (possibly) living animals. We adapt multi-atlas based segmentation driven by elastic registration and joint feature and model selection for the regression component to cape with the large number of features and low number of samples. The proposed technique is evaluated and illustrated through real applications in rabbit breeding programs, showing r^2 scores 12% higher than previous techniques and methods that used to drive the selection so far. The proposed technique is easily adaptable to similar problems, consequently, it is shared in an open source software package for the benefit of the community.
Chatbot for fitness management using IBM Watson
Venue: arXiv cs.SE 2021
Year: 2021

Chatbots have revolutionized the way humans interact with computer systems and they have substituted the use of service agents, call-center representatives etc. Fitness industry has always been a growing industry although it has not adapted to the latest technologies like AI, ML and cloud computing. In this paper, we propose an idea to develop a chatbot for fitness management using IBM Watson and integrate it with a web application. We proposed using Natural Language Processing (NLP) and Natural Language Understanding (NLU) along with frameworks of IBM Cloud Watson provided for the Chatbot Assistant. This software uses a serverless architecture to combine the services of a professional by offering diet plans, home exercises, interactive counseling sessions, fitness recommendations.
First order linear logic and tensor type calculus for categorial grammars
Venue: arXiv cs.CL 2021
Year: 2021

We study relationship between first order multiplicative linear logic (MLL1), which has been known to provide representations to different categorial grammars, and the recently introduced extended tensor type calculus (ETTC). We identify a fragment of MLL1, which seems sufficient for many grammar representations, and establish a correspondence between ETTC and this fragment. The system ETTC, thus, can be seen as an alternative syntax and intrinsic deductive system together with a geometric representation for the latter. We also give a natural deduction formulation of ETTC, which might be convenient.
An Intelligent Self-driving Truck System For Highway Transportation
Venue: arXiv cs.RO 2021
Year: 2021

Recently, there have been many advances in autonomous driving society, attracting a lot of attention from academia and industry. However, existing works mainly focus on cars, extra development is still required for self-driving truck algorithms and models. In this paper, we introduce an intelligent self-driving truck system. Our presented system consists of three main components, 1) a realistic traffic simulation module for generating realistic traffic flow in testing scenarios, 2) a high-fidelity truck model which is designed and evaluated for mimicking real truck response in real-world deployment, 3) an intelligent planning module with learning-based decision making algorithm and multi-mode trajectory planner, taking into account the truck's constraints, road slope changes, and the surrounding traffic flow. We provide quantitative evaluations for each component individually to demonstrate the fidelity and performance of each part. We also deploy our proposed system on a real truck and conduct real world experiments which shows our system's capacity of mitigating sim-to-real gap. Our code is available at https://github.com/InceptioResearch/IITS
SplitBrain: Hybrid Data and Model Parallel Deep Learning
Venue: arXiv cs.LG 2021
Year: 2021

The recent success of deep learning applications has coincided with those widely available powerful computational resources for training sophisticated machine learning models with huge datasets. Nonetheless, training large models such as convolutional neural networks using model parallelism (as opposed to data parallelism) is challenging because the complex nature of communication between model shards makes it difficult to partition the computation efficiently across multiple machines with an acceptable trade-off. This paper presents SplitBrain, a high performance distributed deep learning framework supporting hybrid data and model parallelism. Specifically, SplitBrain provides layer-specific partitioning that co-locates compute intensive convolutional layers while sharding memory demanding layers. A novel scalable group communication is proposed to further improve the training throughput with reduced communication overhead. The results show that SplitBrain can achieve nearly linear speedup while saving up to 67\% of memory consumption for data and model parallel VGG over CIFAR-10.
Improved Algorithm for the Network Alignment Problem with Application to Binary Diffing
Venue: arXiv cs.LG 2021
Year: 2021

In this paper, we present a novel algorithm to address the Network Alignment problem. It is inspired from a previous message passing framework of Bayati et al. [2] and includes several modifications designed to significantly speed up the message updates as well as to enforce their convergence. Experiments show that our proposed model outperforms other state-of-the-art solvers. Finally, we propose an application of our method in order to address the Binary Diffing problem. We show that our solution provides better assignment than the reference differs in almost all submitted instances and outline the importance of leveraging the graphical structure of binary programs.
Statistical Device Activity Detection for OFDM-based Massive Grant-Free Access
Venue: arXiv cs.IT 2021
Year: 2021

Existing works on grant-free access, proposed to support massive machine-type communication (mMTC) for the Internet of things (IoT), mainly concentrate on narrow band systems under flat fading. However, little is known about massive grant-free access for wideband systems under frequency-selective fading. This paper investigates massive grant-free access in a wideband system under frequency-selective fading. First, we present an orthogonal frequency division multiplexing (OFDM)-based massive grant-free access scheme. Then, we propose two different but equivalent models for the received pilot signal, which are essential for designing various device activity detection and channel estimation methods for OFDM-based massive grant-free access. One directly models the received signal for actual devices, whereas the other can be interpreted as a signal model for virtual devices. Next, we investigate statistical device activity detection under frequency-selective Rayleigh fading based on the two signal models. We first model device activities as unknown deterministic quantities and propose three maximum likelihood (ML) estimation-based device activity detection methods with different detection accuracies and computation times. We also model device activities as random variables with a known joint distribution and propose three maximum a posterior probability (MAP) estimation-based device activity methods, which further enhance the accuracies of the corresponding ML estimation-based methods. Optimization techniques and matrix analysis are applied in designing and analyzing these methods. Finally, numerical results show that the proposed statistical device activity detection methods outperform existing state-of-the-art device activity detection methods under frequency-selective Rayleigh fading.
Robust Entropy-regularized Markov Decision Processes
Venue: arXiv cs.LG 2021
Year: 2021

Stochastic and soft optimal policies resulting from entropy-regularized Markov decision processes (ER-MDP) are desirable for exploration and imitation learning applications. Motivated by the fact that such policies are sensitive with respect to the state transition probabilities, and the estimation of these probabilities may be inaccurate, we study a robust version of the ER-MDP model, where the stochastic optimal policies are required to be robust with respect to the ambiguity in the underlying transition probabilities. Our work is at the crossroads of two important schemes in reinforcement learning (RL), namely, robust MDP and entropy regularized MDP. We show that essential properties that hold for the non-robust ER-MDP and robust unregularized MDP models also hold in our settings, making the robust ER-MDP problem tractable. We show how our framework and results can be integrated into different algorithmic schemes including value or (modified) policy iteration, which would lead to new robust RL and inverse RL algorithms to handle uncertainties. Analyses on computational complexity and error propagation under conventional uncertainty settings are also provided.
Efficient and Reliable Overlay Networks for Decentralized Federated Learning
Venue: arXiv cs.NI 2021
Year: 2021

We propose near-optimal overlay networks based on $d$-regular expander graphs to accelerate decentralized federated learning (DFL) and improve its generalization. In DFL a massive number of clients are connected by an overlay network, and they solve machine learning problems collaboratively without sharing raw data. Our overlay network design integrates spectral graph theory and the theoretical convergence and generalization bounds for DFL. As such, our proposed overlay networks accelerate convergence, improve generalization, and enhance robustness to clients failures in DFL with theoretical guarantees. Also, we present an efficient algorithm to convert a given graph to a practical overlay network and maintaining the network topology after potential client failures. We numerically verify the advantages of DFL with our proposed networks on various benchmark tasks, ranging from image classification to language modeling using hundreds of clients.
Centralized and Distributed Power Allocation for Max-Min Fairness in Cell-Free Massive MIMO
Venue: arXiv cs.IT 2021
Year: 2021

Cell-free Massive MIMO systems consist of a large number of geographically distributed access points (APs) that serve users by coherent joint transmission. Downlink power allocation is important in these systems, to determine which APs should transmit to which users and with what power. If the system is implemented correctly, it can deliver a more uniform user performance than conventional cellular networks. To this end, previous works have shown how to perform system-wide max-min fairness power allocation when using maximum ratio precoding. In this paper, we first generalize this method to arbitrary precoding, and then train a neural network to perform approximately the same power allocation but with reduced computational complexity. Finally, we train one neural network per AP to mimic system-wide max-min fairness power allocation, but using only local information. By learning the structure of the local propagation environment, this method outperforms the state-of-the-art distributed power allocation method from the Cell-free Massive MIMO literature.
Inferring perceptual decision making parameters from behavior in production and reproduction tasks
Venue: arXiv cs.LG 2021
Year: 2021

Bayesian models of behavior have provided computational level explanations in a range of psychophysical tasks. One fundamental experimental paradigm is the production or reproduction task, in which subjects are instructed to generate an action that either reproduces a previously sensed stimulus magnitude or achieves a target response. This type of task therefore distinguishes itself from other psychophysical tasks in that the responses are on a continuum and effort plays an important role with increasing response magnitude. Based on Bayesian decision theory we present an inference method to recover perceptual uncertainty, response variability, and the cost function underlying human responses. Crucially, the cost function is parameterized such that effort is explicitly included. We present a hybrid inference method employing MCMC sampling utilizing appropriate proposal distributions and an inner loop utilizing amortized inference with a neural network that approximates the mode of the optimal response distribution. We show how this model can be utilized to avoid unidentifiability of experimental designs and that parameters can be recovered through validation on synthetic and application to experimental data. Our approach will enable behavioral scientists to perform Bayesian inference of decision making parameters in production and reproduction tasks.
Actor Loss of Soft Actor Critic Explained
Venue: arXiv cs.LG 2021
Year: 2021

This technical report is devoted to explaining how the actor loss of soft actor critic is obtained, as well as the associated gradient estimate. It gives the necessary mathematical background to derive all the presented equations, from the theoretical actor loss to the one implemented in practice. This necessitates a comparison of the reparameterization trick used in soft actor critic with the nabla log trick, which leads to open questions regarding the most efficient method to use.
PCACE: A Statistical Approach to Ranking Neurons for CNN Interpretability
Venue: arXiv cs.CV 2021
Year: 2021

In this paper we introduce a new problem within the growing literature of interpretability for convolution neural networks (CNNs). While previous work has focused on the question of how to visually interpret CNNs, we ask what it is that we care to interpret, that is, which layers and neurons are worth our attention? Due to the vast size of modern deep learning network architectures, automated, quantitative methods are needed to rank the relative importance of neurons so as to provide an answer to this question. We present a new statistical method for ranking the hidden neurons in any convolutional layer of a network. We define importance as the maximal correlation between the activation maps and the class score. We provide different ways in which this method can be used for visualization purposes with MNIST and ImageNet, and show a real-world application of our method to air pollution prediction with street-level images.
Device Activity Detection for Massive Grant-Free Access Under Frequency-Selective Rayleigh Fading
Venue: arXiv cs.IT 2021
Year: 2021

Device activity detection and channel estimation for massive grant-free access under frequency-selective fading have unfortunately been an outstanding problem. This paper aims to address the challenge. Specifically, we present an orthogonal frequency division multiplexing (OFDM)-based massive grant-free access scheme for a wideband system with one M-antenna base station (BS), N single-antenna Internet of Things (IoT) devices, and P channel taps. We obtain two different but equivalent models for the received pilot signals under frequency-selective Rayleigh fading. Based on each model, we formulate device activity detection as a non-convex maximum likelihood estimation (MLE) problem and propose an iterative algorithm to obtain a stationary point using optimal techniques. The two proposed MLE-based methods have the identical computational complexity order O(NPL^2), irrespective of M, and degrade to the existing MLE-based device activity detection method when P=1. Conventional channel estimation methods can be readily applied for channel estimation of detected active devices under frequency-selective Rayleigh fading, based on one of the derived models for the received pilot signals. Numerical results show that the two proposed methods have different preferable system parameters and complement each other to offer promising device activity detection design for grant-free massive access under frequency-selective Rayleigh fading.
Jekyll RDF: Template-Based Linked Data Publication with Minimized Effort and Maximum Scalability
Venue: arXiv cs.DB 2021
Year: 2021

Over the last decades the Web has evolved from a human-human communication network to a network of complex human-machine interactions. An increasing amount of data is available as Linked Data which allows machines to "understand" the data, but RDF is not meant to be understood by humans. With Jekyll RDF we present a method to close the gap between structured data and human accessible exploration interfaces by publishing RDF datasets as customizable static HTML sites. It consists of an RDF resource mapping system to serve the resources under their respective IRI, a template mapping based on schema classes, and a markup language to define templates to render customized resource pages. Using the template system, it is possible to create domain specific browsing interfaces for RDF data next to the Linked Data resources. This enables content management and knowledge management systems to serve datasets in a highly customizable, low effort, and scalable way to be consumed by machines as well as humans.
Automatic Pharma News Categorization
Venue: arXiv cs.IR 2021
Year: 2021

We use a text dataset consisting of 23 news categories relevant to pharma information science, in order to compare the fine-tuning performance of multiple transformer models in a classification task. Using a well-balanced dataset with multiple autoregressive and autocoding transformation models, we compare their fine-tuning performance. To validate the winning approach, we perform diagnostics of model behavior on mispredicted instances, including inspection of category-wise metrics, evaluation of prediction certainty and assessment of latent space representations. Lastly, we propose an ensemble model consisting of the top performing individual predictors and demonstrate that this approach offers a modest improvement in the F1 metric.
MetaChain: A Novel Blockchain-based Framework for Metaverse Applications
Venue: arXiv cs.CR 2021
Year: 2021

Metaverse has recently attracted paramount attention due to its potential for future Internet. However, to fully realize such potential, Metaverse applications have to overcome various challenges such as massive resource demands, interoperability among applications, and security and privacy concerns. In this paper, we propose MetaChain, a novel blockchain-based framework to address emerging challenges for the development of Metaverse applications. In particular, by utilizing the smart contract mechanism, MetaChain can effectively manage and automate complex interactions among the Metaverse Service Provider (MSP) and the Metaverse users (MUs). In addition, to allow the MSP to efficiently allocate its resources for Metaverse applications and MUs' demands, we design a novel sharding scheme to improve the underlying blockchain's scalability. Moreover, to leverage MUs' resources as well as to attract more MUs to support Metaverse operations, we develop an incentive mechanism using the Stackelberg game theory that rewards MUs' contributions to the Metaverse. Through numerical experiments, we clearly show the impacts of the MUs' behaviors and how the incentive mechanism can attract more MUs and resources to the Metaverse.
Runway Extraction and Improved Mapping from Space Imagery
Venue: arXiv cs.CV 2021
Year: 2021

Change detection methods applied to monitoring key infrastructure like airport runways represent an important capability for disaster relief and urban planning. The present work identifies two generative adversarial networks (GAN) architectures that translate reversibly between plausible runway maps and satellite imagery. We illustrate the training capability using paired images (satellite-map) from the same point of view and using the Pix2Pix architecture or conditional GANs. In the absence of available pairs, we likewise show that CycleGAN architectures with four network heads (discriminator-generator pairs) can also provide effective style transfer from raw image pixels to outline or feature maps. To emphasize the runway and tarmac boundaries, we experimentally show that the traditional grey-tan map palette is not a required training input but can be augmented by higher contrast mapping palettes (red-black) for sharper runway boundaries. We preview a potentially novel use case (called "sketch2satellite") where a human roughly draws the current runway boundaries and automates the machine output of plausible satellite images. Finally, we identify examples of faulty runway maps where the published satellite and mapped runways disagree but an automated update renders the correct map using GANs.
A Deep Learning Approach to Integrate Human-Level Understanding in a Chatbot
Venue: arXiv cs.CL 2021
Year: 2021

In recent times, a large number of people have been involved in establishing their own businesses. Unlike humans, chatbots can serve multiple customers at a time, are available 24/7 and reply in less than a fraction of a second. Though chatbots perform well in task-oriented activities, in most cases they fail to understand personalized opinions, statements or even queries which later impact the organization for poor service management. Lack of understanding capabilities in bots disinterest humans to continue conversations with them. Usually, chatbots give absurd responses when they are unable to interpret a user's text accurately. Extracting the client reviews from conversations by using chatbots, organizations can reduce the major gap of understanding between the users and the chatbot and improve their quality of products and services.Thus, in our research we incorporated all the key elements that are necessary for a chatbot to analyse and understand an input text precisely and accurately. We performed sentiment analysis, emotion detection, intent classification and named-entity recognition using deep learning to develop chatbots with humanistic understanding and intelligence. The efficiency of our approach can be demonstrated accordingly by the detailed analysis.