OpenRaiser/Intern-Atlas huggingface dataset:
Silicon Dating
Venue: arXiv cs.CR 2020
Year: 2020

In order to service an ever-growing base of legacy electronics, both government and industry customers must turn to third-party brokers for components in short supply or discontinued by the original manufacturer. Sourcing equipment from a third party creates an opportunity for unscrupulous gray market suppliers to insert counterfeit devices: failed, knock-off, or otherwise inferior to the original product. This increases the supplier's profits at the expense of reduced performance/reliability of the customer's system. The most challenging class of counterfeit devices to detect is recycled counterfeits: recovered genuine devices which are re-sold as new. Such devices are difficult to detect because they typically pass performance and parametric tests but fail prematurely due to age-related wear. To address the challenge of detecting recycled devices pre-deployment, we develop Silicon Dating: a low-overhead classifier for detecting recycled integrated circuits using Static Random-Access Memory (SRAM) power-on states. Silicon Dating targets devices with no known-new record or purpose-built anti-recycling hardware. We observe that over time, software running on a device imprints its unique data patterns into SRAM through analog-domain changes; we measure the level and direction of this change through SRAM power-on state statistics. In contrast to highly symmetric power-on states produced by variation during SRAM fabrication, we show that embedded software data is generally highly asymmetric and that the degree of power-on state asymmetry imprinted by software reveals device use. Using empirical results from embedded benchmarks running on several microcontrollers, we show that Silicon Dating identifies recycled devices with 84.1% accuracy with no software-specific knowledge and with 92.0% accuracy by incorporating software knowledge---without prior device enrollment or modification.
Opportunistic Beamforming using an Intelligent Reflecting Surface Without Instantaneous CSI
Venue: arXiv cs.IT 2020
Year: 2020

While intelligent reflecting surface (IRS) assisted wireless communication has emerged as an important research paradigm, channel state information (CSI) acquisition remains a critical challenge to design the IRS phase-shifts and yield the promised coherent beamforming gains. In this paper, we propose an IRS-assisted opportunistic beamforming (OBF) scheme under proportional fair scheduling, which does not require instantaneous CSI to design the IRS parameters. In a slow-fading environment, we show that with only random rotations at the IRS, the proposed scheme can capitalize on the multi-user (MU)-diversity effect to approach the performance of coherent beamforming as the number of users grows large. Next we study the sum-rate scaling of IRS-assisted OBF in the correlated Rayleigh fast fading environment under a deterministic beamforming scheme that results in a considerable sum-rate improvement.
Unconstrained Text Detection in Manga: a New Dataset and Baseline
Venue: arXiv cs.CV 2020
Year: 2020

The detection and recognition of unconstrained text is an open problem in research. Text in comic books has unusual styles that raise many challenges for text detection. This work aims to binarize text in a comic genre with highly sophisticated text styles: Japanese manga. To overcome the lack of a manga dataset with text annotations at a pixel level, we create our own. To improve the evaluation and search of an optimal model, in addition to standard metrics in binarization, we implement other special metrics. Using these resources, we designed and evaluated a deep network model, outperforming current methods for text binarization in manga in most metrics.
Improved Trainable Calibration Method for Neural Networks on Medical Imaging Classification
Venue: arXiv cs.CV 2020
Year: 2020

Recent works have shown that deep neural networks can achieve super-human performance in a wide range of image classification tasks in the medical imaging domain. However, these works have primarily focused on classification accuracy, ignoring the important role of uncertainty quantification. Empirically, neural networks are often miscalibrated and overconfident in their predictions. This miscalibration could be problematic in any automatic decision-making system, but we focus on the medical field in which neural network miscalibration has the potential to lead to significant treatment errors. We propose a novel calibration approach that maintains the overall classification accuracy while significantly improving model calibration. The proposed approach is based on expected calibration error, which is a common metric for quantifying miscalibration. Our approach can be easily integrated into any classification task as an auxiliary loss term, thus not requiring an explicit training round for calibration. We show that our approach reduces calibration error significantly across various architectures and datasets.
Generalizing Complex/Hyper-complex Convolutions to Vector Map Convolutions
Venue: arXiv cs.NE 2020
Year: 2020

We show that the core reasons that complex and hypercomplex valued neural networks offer improvements over their real-valued counterparts is the weight sharing mechanism and treating multidimensional data as a single entity. Their algebra linearly combines the dimensions, making each dimension related to the others. However, both are constrained to a set number of dimensions, two for complex and four for quaternions. Here we introduce novel vector map convolutions which capture both of these properties provided by complex/hypercomplex convolutions, while dropping the unnatural dimensionality constraints they impose. This is achieved by introducing a system that mimics the unique linear combination of input dimensions, such as the Hamilton product for quaternions. We perform three experiments to show that these novel vector map convolutions seem to capture all the benefits of complex and hyper-complex networks, such as their ability to capture internal latent relations, while avoiding the dimensionality restriction.
On the Orthogonality of Knowledge Distillation with Other Techniques: From an Ensemble Perspective
Venue: arXiv cs.LG 2020
Year: 2020

To put a state-of-the-art neural network to practical use, it is necessary to design a model that has a good trade-off between the resource consumption and performance on the test set. Many researchers and engineers are developing methods that enable training or designing a model more efficiently. Developing an efficient model includes several strategies such as network architecture search, pruning, quantization, knowledge distillation, utilizing cheap convolution, regularization, and also includes any craft that leads to a better performance-resource trade-off. When combining these technologies together, it would be ideal if one source of performance improvement does not conflict with others. We call this property as the orthogonality in model efficiency. In this paper, we focus on knowledge distillation and demonstrate that knowledge distillation methods are orthogonal to other efficiency-enhancing methods both analytically and empirically. Analytically, we claim that knowledge distillation functions analogous to a ensemble method, bootstrap aggregating. This analytical explanation is provided from the perspective of implicit data augmentation property of knowledge distillation. Empirically, we verify knowledge distillation as a powerful apparatus for practical deployment of efficient neural network, and also introduce ways to integrate it with other methods effectively.
A Security Architecture for Railway Signalling
Venue: arXiv cs.CR 2020
Year: 2020

We present the proposed security architecture Deutsche Bahn plans to deploy to protect its trackside safety-critical signalling system against cyber-attacks. We first present the existing reference interlocking system that is built using standard components. Next, we present a taxonomy to help model the attack vectors relevant for the railway environment. Building upon this, we present the proposed "compartmentalized" defence concept for securing the upcoming signalling systems.
DyNODE: Neural Ordinary Differential Equations for Dynamics Modeling in Continuous Control
Venue: arXiv cs.LG 2020
Year: 2020

We present a novel approach (DyNODE) that captures the underlying dynamics of a system by incorporating control in a neural ordinary differential equation framework. We conduct a systematic evaluation and comparison of our method and standard neural network architectures for dynamics modeling. Our results indicate that a simple DyNODE architecture when combined with an actor-critic reinforcement learning (RL) algorithm that uses model predictions to improve the critic's target values, outperforms canonical neural networks, both in sample efficiency and predictive performance across a diverse range of continuous tasks that are frequently used to benchmark RL algorithms. This approach provides a new avenue for the development of models that are more suited to learn the evolution of dynamical systems, particularly useful in the context of model-based reinforcement learning. To assist related work, we have made code available at https://github.com/vmartinezalvarez/DyNODE .
How much does an interlibrary loan request cost? A review of the literature
Venue: arXiv cs.DL 2020
Year: 2020

Interlibrary loan (ILL) services are used to fill the gap between academic libraries' collections and the information needs of their users. Today's trend toward the cancellation of serials "Big Deals" has increased the importance of clear information around ILL to support decision-making. In order to plan the cancellation of a journal package, academic libraries need to be able to forecast their total spendings on ILL, which requires to have an appropriate estimate of what it costs to fulfill an individual ILL request. This paper aims to help librarians answer this question by reviewing the most recent academic literature related to these costs. There are several factors that may affect the cost of an ILL service, including the cost of labour, the geographic location of the library, the use of a ILL software, and membership to a library consortium. We find that there is a wide range of estimates for ILL cost, from $3.75 (USD) to $100.00 (USD). However, Jackson's (2004) figure of $17.50 (USD) per transaction remains the guideline for most researchers and librarians.
Online trajectory recovery from offline handwritten Japanese kanji characters
Venue: arXiv cs.CV 2020
Year: 2020

In general, it is straightforward to render an offline handwriting image from an online handwriting pattern. However, it is challenging to reconstruct an online handwriting pattern given an offline handwriting image, especially for multiple-stroke character as Japanese kanji. The multiple-stroke character requires not only point coordinates but also stroke orders whose difficulty is exponential growth by the number of strokes. Besides, several crossed and touch points might increase the difficulty of the recovered task. We propose a deep neural network-based method to solve the recovered task using a large online handwriting database. Our proposed model has two main components: Convolutional Neural Network-based encoder and Long Short-Term Memory Network-based decoder with an attention layer. The encoder focuses on feature extraction while the decoder refers to the extracted features and generates the time-sequences of coordinates. We also demonstrate the effect of the attention layer to guide the decoder during the reconstruction. We evaluate the performance of the proposed method by both visual verification and handwritten character recognition. Although the visual verification reveals some problems, the recognition experiments demonstrate the effect of trajectory recovery in improving the accuracy of offline handwritten character recognition when online recognition for the recovered trajectories are combined.
Assessing Game Balance with AlphaZero: Exploring Alternative Rule Sets in Chess
Venue: arXiv cs.AI 2020
Year: 2020

It is non-trivial to design engaging and balanced sets of game rules. Modern chess has evolved over centuries, but without a similar recourse to history, the consequences of rule changes to game dynamics are difficult to predict. AlphaZero provides an alternative in silico means of game balance assessment. It is a system that can learn near-optimal strategies for any rule set from scratch, without any human supervision, by continually learning from its own experience. In this study we use AlphaZero to creatively explore and design new chess variants. There is growing interest in chess variants like Fischer Random Chess, because of classical chess's voluminous opening theory, the high percentage of draws in professional play, and the non-negligible number of games that end while both players are still in their home preparation. We compare nine other variants that involve atomic changes to the rules of chess. The changes allow for novel strategic and tactical patterns to emerge, while keeping the games close to the original. By learning near-optimal strategies for each variant with AlphaZero, we determine what games between strong human players might look like if these variants were adopted. Qualitatively, several variants are very dynamic. An analytic comparison show that pieces are valued differently between variants, and that some variants are more decisive than classical chess. Our findings demonstrate the rich possibilities that lie beyond the rules of modern chess.
Routing Networks with Co-training for Continual Learning
Venue: arXiv cs.LG 2020
Year: 2020

The core challenge with continual learning is catastrophic forgetting, the phenomenon that when neural networks are trained on a sequence of tasks they rapidly forget previously learned tasks. It has been observed that catastrophic forgetting is most severe when tasks are dissimilar to each other. We propose the use of sparse routing networks for continual learning. For each input, these network architectures activate a different path through a network of experts. Routing networks have been shown to learn to route similar tasks to overlapping sets of experts and dissimilar tasks to disjoint sets of experts. In the continual learning context this behaviour is desirable as it minimizes interference between dissimilar tasks while allowing positive transfer between related tasks. In practice, we find it is necessary to develop a new training method for routing networks, which we call co-training which avoids poorly initialized experts when new tasks are presented. When combined with a small episodic memory replay buffer, sparse routing networks with co-training outperform densely connected networks on the MNIST-Permutations and MNIST-Rotations benchmarks.
Phasic Policy Gradient
Venue: arXiv cs.LG 2020
Year: 2020

We introduce Phasic Policy Gradient (PPG), a reinforcement learning framework which modifies traditional on-policy actor-critic methods by separating policy and value function training into distinct phases. In prior methods, one must choose between using a shared network or separate networks to represent the policy and value function. Using separate networks avoids interference between objectives, while using a shared network allows useful features to be shared. PPG is able to achieve the best of both worlds by splitting optimization into two phases, one that advances training and one that distills features. PPG also enables the value function to be more aggressively optimized with a higher level of sample reuse. Compared to PPO, we find that PPG significantly improves sample efficiency on the challenging Procgen Benchmark.
Nondiagonal Mixture of Dirichlet Network Distributions for Analyzing a Stock Ownership Network
Venue: arXiv cs.SI 2020
Year: 2020

Block modeling is widely used in studies on complex networks. The cornerstone model is the stochastic block model (SBM), widely used over the past decades. However, the SBM is limited in analyzing complex networks as the model is, in essence, a random graph model that cannot reproduce the basic properties of many complex networks, such as sparsity and heavy-tailed degree distribution. In this paper, we provide an edge exchangeable block model that incorporates such basic features and simultaneously infers the latent block structure of a given complex network. Our model is a Bayesian nonparametric model that flexibly estimates the number of blocks and takes into account the possibility of unseen nodes. Using one synthetic dataset and one real-world stock ownership dataset, we show that our model outperforms state-of-the-art SBMs for held-out link prediction tasks.
Physics-Informed Neural Networks for Nonhomogeneous Material Identification in Elasticity Imaging
Venue: arXiv cs.LG 2020
Year: 2020

We apply Physics-Informed Neural Networks (PINNs) for solving identification problems of nonhomogeneous materials. We focus on the problem with a background in elasticity imaging, where one seeks to identify the nonhomogeneous mechanical properties of soft tissue based on the full-field displacement measurements under quasi-static loading. In our model, we apply two independent neural networks, one for approximating the solution of the corresponding forward problem, and the other for approximating the unknown material parameter field. As a proof of concept, we validate our model on a prototypical plane strain problem for incompressible hyperelastic tissue. The results show that the PINNs are effective in accurately recovering the unknown distribution of mechanical properties. By employing two neural networks in our model, we extend the capability of material identification of PINNs to include nonhomogeneous material parameter fields, which enables more flexibility of PINNs in representing complex material properties.
Discovering Textual Structures: Generative Grammar Induction using Template Trees
Venue: arXiv cs.CL 2020
Year: 2020

Natural language generation provides designers with methods for automatically generating text, e.g. for creating summaries, chatbots and game content. In practise, text generators are often either learned and hard to interpret, or created by hand using techniques such as grammars and templates. In this paper, we introduce a novel grammar induction algorithm for learning interpretable grammars for generative purposes, called Gitta. We also introduce the novel notion of template trees to discover latent templates in corpora to derive these generative grammars. By using existing human-created grammars, we found that the algorithm can reasonably approximate these grammars using only a few examples. These results indicate that Gitta could be used to automatically learn interpretable and easily modifiable grammars, and thus provide a stepping stone for human-machine co-creation of generative models.
Attention based Writer Independent Handwriting Verification
Venue: arXiv cs.CV 2020
Year: 2020

The task of writer verification is to provide a likelihood score for whether the queried and known handwritten image samples belong to the same writer or not. Such a task calls for the neural network to make it's outcome interpretable, i.e. provide a view into the network's decision making process. We implement and integrate cross-attention and soft-attention mechanisms to capture the highly correlated and salient points in feature space of 2D inputs. The attention maps serve as an explanation premise for the network's output likelihood score. The attention mechanism also allows the network to focus more on relevant areas of the input, thus improving the classification performance. Our proposed approach achieves a precision of 86\% for detecting intra-writer cases in CEDAR cursive "AND" dataset. Furthermore, we generate meaningful explanations for the provided decision by extracting attention maps from multiple levels of the network.
Multimodal Deep Learning for Flaw Detection in Software Programs
Venue: arXiv cs.LG 2020
Year: 2020

We explore the use of multiple deep learning models for detecting flaws in software programs. Current, standard approaches for flaw detection rely on a single representation of a software program (e.g., source code or a program binary). We illustrate that, by using techniques from multimodal deep learning, we can simultaneously leverage multiple representations of software programs to improve flaw detection over single representation analyses. Specifically, we adapt three deep learning models from the multimodal learning literature for use in flaw detection and demonstrate how these models outperform traditional deep learning models. We present results on detecting software flaws using the Juliet Test Suite and Linux Kernel.
RoIFusion: 3D Object Detection from LiDAR and Vision
Venue: arXiv cs.CV 2020
Year: 2020

When localizing and detecting 3D objects for autonomous driving scenes, obtaining information from multiple sensor (e.g. camera, LIDAR) typically increases the robustness of 3D detectors. However, the efficient and effective fusion of different features captured from LIDAR and camera is still challenging, especially due to the sparsity and irregularity of point cloud distributions. This notwithstanding, point clouds offer useful complementary information. In this paper, we would like to leverage the advantages of LIDAR and camera sensors by proposing a deep neural network architecture for the fusion and the efficient detection of 3D objects by identifying their corresponding 3D bounding boxes with orientation. In order to achieve this task, instead of densely combining the point-wise feature of the point cloud and the related pixel features, we propose a novel fusion algorithm by projecting a set of 3D Region of Interests (RoIs) from the point clouds to the 2D RoIs of the corresponding the images. Finally, we demonstrate that our deep fusion approach achieves state-of-the-art performance on the KITTI 3D object detection challenging benchmark.
Developing and Improving Risk Models using Machine-learning Based Algorithms
Venue: arXiv cs.LG 2020
Year: 2020

The objective of this study is to develop a good risk model for classifying business delinquency by simultaneously exploring several machine learning based methods including regularization, hyper-parameter optimization, and model ensembling algorithms. The rationale under the analyses is firstly to obtain good base binary classifiers (include Logistic Regression ($LR$), K-Nearest Neighbors ($KNN$), Decision Tree ($DT$), and Artificial Neural Networks ($ANN$)) via regularization and appropriate settings of hyper-parameters. Then two model ensembling algorithms including bagging and boosting are performed on the good base classifiers for further model improvement. The models are evaluated using accuracy, Area Under the Receiver Operating Characteristic Curve (AUC of ROC), recall, and F1 score via repeating 10-fold cross-validation 10 times. The results show the optimal base classifiers along with the hyper-parameter settings are $LR$ without regularization, $KNN$ by using 9 nearest neighbors, $DT$ by setting the maximum level of the tree to be 7, and $ANN$ with three hidden layers. Bagging on $KNN$ with $K$ valued 9 is the optimal model we can get for risk classification as it reaches the average accuracy, AUC, recall, and F1 score valued 0.90, 0.93, 0.82, and 0.89, respectively.