OpenRaiser/Intern-Atlas huggingface dataset:
Modelling the COVID-19 virus evolution with Incremental Machine Learning
Venue: arXiv cs.LG 2021
Year: 2021

The investment of time and resources for better strategies and methodologies to tackle a potential pandemic is key to deal with potential outbreaks of new variants or other viruses in the future. In this work, we recreated the scene of a year ago, 2020, when the pandemic erupted across the world for the fifty countries with more COVID-19 cases reported. We performed some experiments in which we compare state-of-the-art machine learning algorithms, such as LSTM, against online incremental machine learning algorithms to adapt them to the daily changes in the spread of the disease and predict future COVID-19 cases. To compare the methods, we performed three experiments: In the first one, we trained the models using only data from the country we predicted. In the second one, we use data from all fifty countries to train and predict each of them. In the first and second experiment, we used a static hold-out approach for all methods. In the third experiment, we trained the incremental methods sequentially, using a prequential evaluation. This scheme is not suitable for most state-of-the-art machine learning algorithms because they need to be retrained from scratch for every batch of predictions, causing a computational burden. Results show that incremental methods are a promising approach to adapt to changes of the disease over time; they are always up to date with the last state of the data distribution, and they have a significantly lower computational cost than other techniques such as LSTMs.
DANICE: Domain adaptation without forgetting in neural image compression
Venue: arXiv cs.CV 2021
Year: 2021

Neural image compression (NIC) is a new coding paradigm where coding capabilities are captured by deep models learned from data. This data-driven nature enables new potential functionalities. In this paper, we study the adaptability of codecs to custom domains of interest. We show that NIC codecs are transferable and that they can be adapted with relatively few target domain images. However, naive adaptation interferes with the solution optimized for the original source domain, resulting in forgetting the original coding capabilities in that domain, and may even break the compatibility with previously encoded bitstreams. Addressing these problems, we propose Codec Adaptation without Forgetting (CAwF), a framework that can avoid these problems by adding a small amount of custom parameters, where the source codec remains embedded and unchanged during the adaptation process. Experiments demonstrate its effectiveness and provide useful insights on the characteristics of catastrophic interference in NIC.
A Multi-Task Deep Learning Framework for Building Footprint Segmentation
Venue: arXiv cs.CV 2021
Year: 2021

The task of building footprint segmentation has been well-studied in the context of remote sensing (RS) as it provides valuable information in many aspects, however, difficulties brought by the nature of RS images such as variations in the spatial arrangements and in-consistent constructional patterns require studying further, since it often causes poorly classified segmentation maps. We address this need by designing a joint optimization scheme for the task of building footprint delineation and introducing two auxiliary tasks; image reconstruction and building footprint boundary segmentation with the intent to reveal the common underlying structure to advance the classification accuracy of a single task model under the favor of auxiliary tasks. In particular, we propose a deep multi-task learning (MTL) based unified fully convolutional framework which operates in an end-to-end manner by making use of joint loss function with learnable loss weights considering the homoscedastic uncertainty of each task loss. Experimental results conducted on the SpaceNet6 dataset demonstrate the potential of the proposed MTL framework as it improves the classification accuracy considerably compared to single-task and lesser compounded tasks.
A Joint Energy and Latency Framework for Transfer Learning over 5G Industrial Edge Networks
Venue: arXiv cs.NI 2021
Year: 2021

In this paper, we propose a transfer learning (TL)-enabled edge-CNN framework for 5G industrial edge networks with privacy-preserving characteristic. In particular, the edge server can use the existing image dataset to train the CNN in advance, which is further fine-tuned based on the limited datasets uploaded from the devices. With the aid of TL, the devices that are not participating in the training only need to fine-tune the trained edge-CNN model without training from scratch. Due to the energy budget of the devices and the limited communication bandwidth, a joint energy and latency problem is formulated, which is solved by decomposing the original problem into an uploading decision subproblem and a wireless bandwidth allocation subproblem. Experiments using ImageNet demonstrate that the proposed TL-enabled edge-CNN framework can achieve almost 85% prediction accuracy of the baseline by uploading only about 1% model parameters, for a compression ratio of 32 of the autoencoder.
SIGIR 2021 E-Commerce Workshop Data Challenge
Venue: arXiv cs.IR 2021
Year: 2021

The 2021 SIGIR workshop on eCommerce is hosting the Coveo Data Challenge for "In-session prediction for purchase intent and recommendations". The challenge addresses the growing need for reliable predictions within the boundaries of a shopping session, as customer intentions can be different depending on the occasion. The need for efficient procedures for personalization is even clearer if we consider the e-commerce landscape more broadly: outside of giant digital retailers, the constraints of the problem are stricter, due to smaller user bases and the realization that most users are not frequently returning customers. We release a new session-based dataset including more than 30M fine-grained browsing events (product detail, add, purchase), enriched by linguistic behavior (queries made by shoppers, with items clicked and items not clicked after the query) and catalog meta-data (images, text, pricing information). On this dataset, we ask participants to showcase innovative solutions for two open problems: a recommendation task (where a model is shown some events at the start of a session, and it is asked to predict future product interactions); an intent prediction task, where a model is shown a session containing an add-to-cart event, and it is asked to predict whether the item will be bought before the end of the session.
Artificial Intelligence in Open Radio Access Network
Venue: arXiv cs.NI 2021
Year: 2021

This tutorial seeks to outline the proposed Open Radio Access Network (O-RAN) deployment for Fifth generation (5G) wireless networks. O-RAN seeks to supplant hardware-specific Radio Access Network (RAN) components (e.g., the mobility management entity (MME) or base station (gNB)) with generic hardware, specialized software, and open signaling interfaces. The virtualization and network slicing features of 5G allow for software to replace previously hardware specific functions. Software further provides faster analytics, thus supporting 5Gs latency requirements and advanced usage scenarios (i.e., enhanced mobile broadband (eMBB), massive machine type communications (mMTC), and ultra-reliable low latency communications (uRLLC)). Furthermore, as software annexes control of the RAN, there is freedom to integrate Artificial Intelligence/Machine Learning (AI/ML) algorithms into RAN management (particularly at the user plane). This integration is one of the goals of O-RAN. Lastly, relying on generic hardware and specialized, open-source software eliminates reliance upon specific device manufacturers. This paper will provide questions regarding the future of O-RAN, with a focus on 5G network device security.
Arithmetic-Intensity-Guided Fault Tolerance for Neural Network Inference on GPUs
Venue: arXiv cs.DC 2021
Year: 2021

Neural networks (NNs) are increasingly employed in safety-critical domains and in environments prone to unreliability (e.g., soft errors), such as on spacecraft. Therefore, it is critical to impart fault tolerance to NN inference. Algorithm-based fault tolerance (ABFT) is emerging as an efficient approach for fault tolerance in NNs. We propose an adaptive approach to ABFT for NN inference that exploits untapped opportunities in emerging deployment scenarios. GPUs have high compute-to-memory-bandwidth ratios, while NN layers have a wide range of arithmetic intensities. This leaves some layers compute bound and others memory-bandwidth bound, but current approaches to ABFT do not consider these differences. We first investigate ABFT schemes best suited for each of these scenarios. We then propose intensity-guided ABFT, an adaptive, arithmetic-intensity-guided approach that selects the most efficient ABFT scheme for each NN layer. Intensity-guided ABFT reduces execution-time overhead by 1.09--5.3$\times$ across many NNs compared to traditional approaches to ABFT.
Machine learning-assisted surrogate construction for full-core fuel performance analysis
Venue: arXiv cs.CE 2021
Year: 2021

Accurately predicting the behavior of a nuclear reactor requires multiphysics simulation of coupled neutronics, thermal-hydraulics and fuel thermo-mechanics. The fuel thermo-mechanical response provides essential information for operational limits and safety analysis. Traditionally, fuel performance analysis is performed standalone, using calculated spatial-temporal power distribution and thermal boundary conditions from the coupled neutronics-thermal-hydraulics simulation as input. Such one-way coupling is result of the high cost induced by the full-core fuel performance analysis, which provides more realistic and accurate prediction of the core-wide response than the "peak rod" analysis. It is therefore desirable to improve the computational efficiency of full-core fuel performance modeling by constructing fast-running surrogate, such that fuel performance modeling can be utilized in the core reload design optimization. This work presents methodologies for full-core surrogate construction based on several realistic equilibrium PWR core designs. As a fast and conventional approach, look-up tables are only effective for certain fuel performance quantities of interest (QoIs). Several representative machine-learning algorithms are introduced to capture the complicated physics for other fuel performance QoIs. Rule-based model is useful as a feature extraction technique to account for the spatial-temporal complexity of operating conditions. Constructed surrogates achieve at least ten thousand time acceleration with satisfying prediction accuracy. Current work lays foundation for tighter coupling of fuel performance modeling into the core design optimization framework. It also sets stage for full-core fuel performance analysis with BISON where the computational cost becomes more burdensome.
Learning to Communicate with Strangers via Channel Randomisation Methods
Venue: arXiv cs.LG 2021
Year: 2021

We introduce two methods for improving the performance of agents meeting for the first time to accomplish a communicative task. The methods are: (1) `message mutation' during the generation of the communication protocol; and (2) random permutations of the communication channel. These proposals are tested using a simple two-player game involving a `teacher' who generates a communication protocol and sends a message, and a `student' who interprets the message. After training multiple agents via self-play we analyse the performance of these agents when they are matched with a stranger, i.e. their zero-shot communication performance. We find that both message mutation and channel permutation positively influence performance, and we discuss their effects.
Political Discussion is Abundant in Non-political Subreddits (and Less Toxic)
Venue: arXiv cs.SI 2021
Year: 2021

Research on online political communication has primarily focused on content in explicitly political spaces. In this work, we set out to determine the amount of political talk missed using this approach. Focusing on Reddit, we estimate that nearly half of all political talk takes place in subreddits that host political content less than 25% of the time. In other words, cumulatively, political talk in non-political spaces is abundant. We further examine the nature of political talk and show that political conversations are less toxic in non-political subreddits. Indeed, the average toxicity of political comments replying to a out-partisan in non-political subreddits is even less than the toxicity of co-partisan replies in explicitly political subreddits.
Detecting Vehicle Type and License Plate Number of different Vehicles on Images
Venue: arXiv cs.CV 2021
Year: 2021

With ever increasing number of vehicles, vehicular tracking is one of the major challenges faced by urban areas. In this paper we try to develop a model that can locate a particular vehicle that the user is looking for depending on two factors 1. the Type of vehicle and the 2. License plate number of the car. The proposed system uses a unique mixture consisting of Mask R-CNN model for vehicle type detection, WpodNet and pytesseract for License Plate detection and Prediction of letters in it.
Vectorized Secure Evaluation of Decision Forests
Venue: arXiv cs.CR 2021
Year: 2021

As the demand for machine learning-based inference increases in tandem with concerns about privacy, there is a growing recognition of the need for secure machine learning, in which secret models can be used to classify private data without the model or data being leaked. Fully Homomorphic Encryption (FHE) allows arbitrary computation to be done over encrypted data, providing an attractive approach to providing such secure inference. While such computation is often orders of magnitude slower than its plaintext counterpart, the ability of FHE cryptosystems to do \emph{ciphertext packing} -- that is, encrypting an entire vector of plaintexts such that operations are evaluated elementwise on the vector -- helps ameliorate this overhead, effectively creating a SIMD architecture where computation can be vectorized for more efficient evaluation. Most recent research in this area has targeted regular, easily vectorizable neural network models. Applying similar techniques to irregular ML models such as decision forests remains unexplored, due to their complex, hard-to-vectorize structures. In this paper we present COPSE, the first system that exploits ciphertext packing to perform decision-forest inference. COPSE consists of a staging compiler that automatically restructures and compiles decision forest models down to a new set of vectorizable primitives for secure inference. We find that COPSE's compiled models outperform the state of the art across a range of decision forest models, often by more than an order of magnitude, while still scaling well.
Operationalizing a National Digital Library: The Case for a Norwegian Transformer Model
Venue: arXiv cs.CL 2021
Year: 2021

In this work, we show the process of building a large-scale training set from digital and digitized collections at a national library. The resulting Bidirectional Encoder Representations from Transformers (BERT)-based language model for Norwegian outperforms multilingual BERT (mBERT) models in several token and sequence classification tasks for both Norwegian Bokmål and Norwegian Nynorsk. Our model also improves the mBERT performance for other languages present in the corpus such as English, Swedish, and Danish. For languages not included in the corpus, the weights degrade moderately while keeping strong multilingual properties. Therefore, we show that building high-quality models within a memory institution using somewhat noisy optical character recognition (OCR) content is feasible, and we hope to pave the way for other memory institutions to follow.
The mixed deep energy method for resolving concentration features in finite strain hyperelasticity
Venue: arXiv cs.CE 2021
Year: 2021

The introduction of Physics-informed Neural Networks (PINNs) has led to an increased interest in deep neural networks as universal approximators of PDEs in the solid mechanics community. Recently, the Deep Energy Method (DEM) has been proposed. DEM is based on energy minimization principles, contrary to PINN which is based on the residual of the PDEs. A significant advantage of DEM, is that it requires the approximation of lower order derivatives compared to formulations that are based on strong form residuals. However both DEM and classical PINN formulations struggle to resolve fine features of the stress and displacement fields, for example concentration features in solid mechanics applications. We propose an extension to the Deep Energy Method (DEM) to resolve these features for finite strain hyperelasticity. The developed framework termed mixed Deep Energy Method (mDEM) introduces stress measures as an additional output of the NN to the recently introduced pure displacement formulation. Using this approach, Neumann boundary conditions are approximated more accurately and the accuracy around spatial features which are typically responsible for high concentrations is increased. In order to make the proposed approach more versatile, we introduce a numerical integration scheme based on Delaunay integration, which enables the mDEM framework to be used for random training point position sets commonly needed for computational domains with stress concentrations. We highlight the advantages of the proposed approach while showing the shortcomings of classical PINN and DEM formulations. The method is offering comparable results to Finite-Element Method (FEM) on the forward calculation of challenging computational experiments involving domains with fine geometric features and concentrated loads.
Quaternion Generative Adversarial Networks
Venue: arXiv cs.LG 2021
Year: 2021

Latest Generative Adversarial Networks (GANs) are gathering outstanding results through a large-scale training, thus employing models composed of millions of parameters requiring extensive computational capabilities. Building such huge models undermines their replicability and increases the training instability. Moreover, multi-channel data, such as images or audio, are usually processed by realvalued convolutional networks that flatten and concatenate the input, often losing intra-channel spatial relations. To address these issues related to complexity and information loss, we propose a family of quaternion-valued generative adversarial networks (QGANs). QGANs exploit the properties of quaternion algebra, e.g., the Hamilton product, that allows to process channels as a single entity and capture internal latent relations, while reducing by a factor of 4 the overall number of parameters. We show how to design QGANs and to extend the proposed approach even to advanced models.We compare the proposed QGANs with real-valued counterparts on several image generation benchmarks. Results show that QGANs are able to obtain better FID scores than real-valued GANs and to generate visually pleasing images. Furthermore, QGANs save up to 75% of the training parameters. We believe these results may pave the way to novel, more accessible, GANs capable of improving performance and saving computational resources.
Hypervolume-Optimal $μ$-Distributions on Line/Plane-based Pareto Fronts in Three Dimensions
Venue: arXiv cs.NE 2021
Year: 2021

Hypervolume is widely used in the evolutionary multi-objective optimization (EMO) field to evaluate the quality of a solution set. For a solution set with $μ$ solutions on a Pareto front, a larger hypervolume means a better solution set. Investigating the distribution of the solution set with the largest hypervolume is an important topic in EMO, which is the so-called hypervolume optimal $μ$-distribution. Theoretical results have shown that the $μ$ solutions are uniformly distributed on a linear Pareto front in two dimensions. However, the $μ$ solutions are not always uniformly distributed on a single-line Pareto front in three dimensions. They are only uniform when the single-line Pareto front has one constant objective. In this paper, we further investigate the hypervolume optimal $μ$-distribution in three dimensions. We consider the line- and plane-based Pareto fronts. For the line-based Pareto fronts, we extend the single-line Pareto front to two-line and three-line Pareto fronts, where each line has one constant objective. For the plane-based Pareto fronts, the linear triangular and inverted triangular Pareto fronts are considered. First, we show that the $μ$ solutions are not always uniformly distributed on the line-based Pareto fronts. The uniformity depends on how the lines are combined. Then, we show that a uniform solution set on the plane-based Pareto front is not always optimal for hypervolume maximization. It is locally optimal with respect to a $(μ+1)$ selection scheme. Our results can help researchers in the community to better understand and utilize the hypervolume indicator.
CoDR: Computation and Data Reuse Aware CNN Accelerator
Venue: arXiv cs.AR 2021
Year: 2021

Computation and Data Reuse is critical for the resource-limited Convolutional Neural Network (CNN) accelerators. This paper presents Universal Computation Reuse to exploit weight sparsity, repetition, and similarity simultaneously in a convolutional layer. Moreover, CoDR decreases the cost of weight memory access by proposing a customized Run-Length Encoding scheme and the number of memory accesses to the intermediate results by introducing an input and output stationary dataflow. Compared to two recent compressed CNN accelerators with the same area of 2.85 mm^2, CoDR decreases SRAM access by 5.08x and 7.99x, and consumes 3.76x and 6.84x less energy.
Addressing the Vulnerability of NMT in Input Perturbations
Venue: arXiv cs.CL 2021
Year: 2021

Neural Machine Translation (NMT) has achieved significant breakthrough in performance but is known to suffer vulnerability to input perturbations. As real input noise is difficult to predict during training, robustness is a big issue for system deployment. In this paper, we improve the robustness of NMT models by reducing the effect of noisy words through a Context-Enhanced Reconstruction (CER) approach. CER trains the model to resist noise in two steps: (1) perturbation step that breaks the naturalness of input sequence with made-up words; (2) reconstruction step that defends the noise propagation by generating better and more robust contextual representation. Experimental results on Chinese-English (ZH-EN) and French-English (FR-EN) translation tasks demonstrate robustness improvement on both news and social media text. Further fine-tuning experiments on social media text show our approach can converge at a higher position and provide a better adaptation.