OpenRaiser/Intern-Atlas huggingface dataset:
On the comparability of Pre-trained Language Models
Venue: arXiv cs.CL 2020
Year: 2020

Recent developments in unsupervised representation learning have successfully established the concept of transfer learning in NLP. Mainly three forces are driving the improvements in this area of research: More elaborated architectures are making better use of contextual information. Instead of simply plugging in static pre-trained representations, these are learned based on surrounding context in end-to-end trainable models with more intelligently designed language modelling objectives. Along with this, larger corpora are used as resources for pre-training large language models in a self-supervised fashion which are afterwards fine-tuned on supervised tasks. Advances in parallel computing as well as in cloud computing, made it possible to train these models with growing capacities in the same or even in shorter time than previously established models. These three developments agglomerate in new state-of-the-art (SOTA) results being revealed in a higher and higher frequency. It is not always obvious where these improvements originate from, as it is not possible to completely disentangle the contributions of the three driving forces. We set ourselves to providing a clear and concise overview on several large pre-trained language models, which achieved SOTA results in the last two years, with respect to their use of new architectures and resources. We want to clarify for the reader where the differences between the models are and we furthermore attempt to gain some insight into the single contributions of lexical/computational improvements as well as of architectural changes. We explicitly do not intend to quantify these contributions, but rather see our work as an overview in order to identify potential starting points for benchmark comparisons. Furthermore, we tentatively want to point at potential possibilities for improvement in the field of open-sourcing and reproducible research.
A Framework for Democratizing AI
Venue: arXiv cs.LG 2020
Year: 2020

Machine Learning and Artificial Intelligence are considered an integral part of the Fourth Industrial Revolution. Their impact, and far-reaching consequences, while acknowledged, are yet to be comprehended. These technologies are very specialized, and few organizations and select highly trained professionals have the wherewithal, in terms of money, manpower, and might, to chart the future. However, concentration of power can lead to marginalization, causing severe inequalities. Regulatory agencies and governments across the globe are creating national policies, and laws around these technologies to protect the rights of the digital citizens, as well as to empower them. Even private, not-for-profit organizations are also contributing to democratizing the technologies by making them \emph{accessible} and \emph{affordable}. However, accessibility and affordability are all but a few of the facets of democratizing the field. Others include, but not limited to, \emph{portability}, \emph{explainability}, \emph{credibility}, \emph{fairness}, among others. As one can imagine, democratizing AI is a multi-faceted problem, and it requires advancements in science, technology and policy. At \texttt{mlsquare}, we are developing scientific tools in this space. Specifically, we introduce an opinionated, extensible, \texttt{Python} framework that provides a single point of interface to a variety of solutions in each of the categories mentioned above. We present the design details, APIs of the framework, reference implementations, road map for development, and guidelines for contributions.
Biometric and Physical Identifiers with Correlated Noise for Controllable Private Authentication
Venue: arXiv cs.IT 2020
Year: 2020

The problem of secret-key based authentication under privacy and storage constraints on the source sequence is considered. The identifier measurement channels during authentication are assumed to be controllable via a cost-constrained action sequence. Single-letter inner and outer bounds for the key-leakage-storage-cost regions are derived for a generalization of a classic two-terminal key agreement model with an eavesdropper that observes a sequence that is correlated with the sequences observed by the legitimate terminals. The additions to the model are that the encoder observes a noisy version of a remote source, and the noisy output and the remote source output together with an action sequence are given as inputs to the measurement channel at the decoder. Thus, correlation is introduced between the noise components on the encoder and decoder measurements. The model with a secret key generated by an encoder is extended to the randomized models, where a secret-key is embedded to the encoder. The results are relevant for several user and device authentication scenarios including physical and biometric identifiers with multiple measurements that provide diversity and multiplexing gains. To illustrate the behavior of the rate region, achievable (secret-key rate, storage-rate, cost) tuples are given for binary identifiers and measurement channels that can be represented as a mixture of binary symmetric subchannels. The gains from using an action sequence such as a large secret-key rate at a significantly small hardware cost, are illustrated to motivate the use of low-complexity transform-coding algorithms with cost-constrained actions.
Improving Grid Computing Performance by Optimally Reducing Checkpointing Effect
Venue: arXiv cs.DC 2020
Year: 2020

Grid computing is a collection of computer resources that are gathered together from various areas to give computational resources such as storage, data or application services. This is to permit clients to access this huge measure of processing resources without the need to know where these might be found and what technology such as, hardware equipment and operating system was used. Dependability and performance are among the key difficulties faced in a grid computing environment. Various systems have been proposed in the literature to handle recouping from resource failure in Grid computing environment. One case of such system is checkpointing. Checkpointing is a system that endures faults when resources failed. Checkpointing method has the upside of lessening the work lost because of resource faults. However, checkpointing presents a huge runtime overhead. In this paper, we propose an improved checkpointing system to bring down runtime overhead. A replica is added to ensure the availability of resources. This replicates all checkpointing files to other machines as opposed to having dedicated checkpointing machine. The results of simulation employing GridSim noted that retaining the number of resources fixed and changing the number of gridlets, gains of up to 11%, 9%, and 11% on makespan, throughput, and turnaround time respectively were realized while changing the number of resources and preserving the number of gridlets fixed, increases of up to 11%, 8%, and 9% on makespan, throughput, and turnaround time respectively, were realized
A phase field model for cohesive fracture in micropolar continua
Venue: arXiv cs.CE 2020
Year: 2020

While crack nucleation and propagation in the brittle or quasi-brittle regime can be predicted via variational or material-force-based phase field fracture models, these models often assume that the underlying elastic response of the material is non-polar and yet a length scale parameter must be introduced to enable the sharp cracks represented by a regularized implicit function. However, many materials with internal microstructures that contain surface tension, micro-cracks, micro-fracture, inclusion, cavity or those of particulate nature often exhibit size-dependent behaviors in both the path-independent and path-dependent regimes. This paper is intended to introduce a unified treatment that captures the size effect of the materials in both elastic and damaged states. By introducing a cohesive micropolar phase field fracture theory, along with the computational model and validation exercises, we explore the interacting size-dependent elastic deformation and fracture mechanisms exhibits in materials of complex microstructures. To achieve this goal, we introduce the distinctive degradation functions of the force-stress-strain and couple-stress-micro-rotation energy-conjugated pairs for a given regularization profile such that the macroscopic size-dependent responses of the micropolar continua is insensitive to the length scale parameter of the regularized interface. Then, we apply the variational principle to derive governing equations from the micropolar stored energy and dissipative functionals. Numerical examples are introduced to demonstrate the proper way to identify material parameters and the capacity of the new formulation to simulate complex crack patterns in the quasi-static regime.
Root Cause Detection Among Anomalous Time Series Using Temporal State Alignment
Venue: arXiv cs.LG 2020
Year: 2020

The recent increase in the scale and complexity of software systems has introduced new challenges to the time series monitoring and anomaly detection process. A major drawback of existing anomaly detection methods is that they lack contextual information to help stakeholders identify the cause of anomalies. This problem, known as root cause detection, is particularly challenging to undertake in today's complex distributed software systems since the metrics under consideration generally have multiple internal and external dependencies. Significant manual analysis and strong domain expertise is required to isolate the correct cause of the problem. In this paper, we propose a method that isolates the root cause of an anomaly by analyzing the patterns in time series fluctuations. Our method considers the time series as observations from an underlying process passing through a sequence of discretized hidden states. The idea is to track the propagation of the effect when a given problem causes unaligned but homogeneous shifts of the underlying states. We evaluate our approach by finding the root cause of anomalies in Zillows clickstream data by identifying causal patterns among a set of observed fluctuations.
Recognizing and tracking outdoor objects by using ARToolKit markers
Venue: arXiv cs.HC 2020
Year: 2020

We created an augmented reality platform for spatial exploration that recognizes buildings facades and displays various multimedia for different time points. In order to provide the user with the best user experience fast recognition and stable tracking are the key elements of any augmented reality app. In an outdoor environment, lighting, reflective surfaces and occlusion can drastically affect the user experience. In a setup where these conditions are similar, marker creation methodology and the app parameters are key. In this paper we focus on resizing the photo prior marker creating and the importance of camera calibration and resolution and their effect on the recognition speed and quality of tracking outdoor objects.
RPR: Random Partition Relaxation for Training; Binary and Ternary Weight Neural Networks
Venue: arXiv cs.CV 2020
Year: 2020

We present Random Partition Relaxation (RPR), a method for strong quantization of neural networks weight to binary (+1/-1) and ternary (+1/0/-1) values. Starting from a pre-trained model, we quantize the weights and then relax random partitions of them to their continuous values for retraining before re-quantizing them and switching to another weight partition for further adaptation. We demonstrate binary and ternary-weight networks with accuracies beyond the state-of-the-art for GoogLeNet and competitive performance for ResNet-18 and ResNet-50 using an SGD-based training method that can easily be integrated into existing frameworks.
MushroomRL: Simplifying Reinforcement Learning Research
Venue: arXiv cs.LG 2020
Year: 2020

MushroomRL is an open-source Python library developed to simplify the process of implementing and running Reinforcement Learning (RL) experiments. Compared to other available libraries, MushroomRL has been created with the purpose of providing a comprehensive and flexible framework to minimize the effort in implementing and testing novel RL methodologies. Indeed, the architecture of MushroomRL is built in such a way that every component of an RL problem is already provided, and most of the time users can only focus on the implementation of their own algorithms and experiments. The result is a library from which RL researchers can significantly benefit in the critical phase of the empirical analysis of their works. MushroomRL stable code, tutorials and documentation can be found at https://github.com/MushroomRL/mushroom-rl.
Timely Status Updating Through Intermittent Sensing and Transmission
Venue: arXiv cs.IT 2020
Year: 2020

We consider a novel intermittent status updating model where an energy harvesting node with an intermittent energy source performs status updating to a receiver through non-preemptive sensing and transmission operations. Each operation costs a single energy recharge of the node and the node cannot harvest energy while in operation. The sensing time for each update is independent with a general distribution. The transmission queue has a single server receiving packets generated after sensing operation, general service time distribution and a single data buffer to save the latest arriving update packet. Once energy is harvested, the node has to decide whether to activate sensing to generate a new update or transmission to send the existing update (if any) to the receiver. We prove that average peak age of information (AoI) at the receiver is minimized by a threshold-based stopping rule that accepts only young packets to the transmission server. We then use this result to address average AoI optimization over the considered stopping rules through novel hybrid waiting and thresholding schemes. Our numerical results show the improvements in average AoI maintained by hybrid schemes.
TCM-ICP: Transformation Compatibility Measure for Registering Multiple LIDAR Scans
Venue: arXiv cs.CV 2020
Year: 2020

Rigid registration of multi-view and multi-platform LiDAR scans is a fundamental problem in 3D mapping, robotic navigation, and large-scale urban modeling applications. Data acquisition with LiDAR sensors involves scanning multiple areas from different points of view, thus generating partially overlapping point clouds of the real world scenes. Traditionally, ICP (Iterative Closest Point) algorithm is used to register the acquired point clouds together to form a unique point cloud that captures the scanned real world scene. Conventional ICP faces local minima issues and often needs a coarse initial alignment to converge to the optimum. In this work, we present an algorithm for registering multiple, overlapping LiDAR scans. We introduce a geometric metric called Transformation Compatibility Measure (TCM) which aids in choosing the most similar point clouds for registration in each iteration of the algorithm. The LiDAR scan most similar to the reference LiDAR scan is then transformed using simplex technique. An optimization of the transformation using gradient descent and simulated annealing techniques are then applied to improve the resulting registration. We evaluate the proposed algorithm on four different real world scenes and experimental results shows that the registration performance of the proposed method is comparable or superior to the traditionally used registration methods. Further, the algorithm achieves superior registration results even when dealing with outliers.
CNNTOP: a CNN-based Trajectory Owner Prediction Method
Venue: arXiv cs.LG 2020
Year: 2020

Trajectory owner prediction is the basis for many applications such as personalized recommendation, urban planning. Although much effort has been put on this topic, the results archived are still not good enough. Existing methods mainly employ RNNs to model trajectories semantically due to the inherent sequential attribute of trajectories. However, these approaches are weak at Point of Interest (POI) representation learning and trajectory feature detection. Thus, the performance of existing solutions is far from the requirements of practical applications. In this paper, we propose a novel CNN-based Trajectory Owner Prediction (CNNTOP) method. Firstly, we connect all POI according to trajectories from all users. The result is a connected graph that can be used to generate more informative POI sequences than other approaches. Secondly, we employ the Node2Vec algorithm to encode each POI into a low-dimensional real value vector. Then, we transform each trajectory into a fixed-dimensional matrix, which is similar to an image. Finally, a CNN is designed to detect features and predict the owner of a given trajectory. The CNN can extract informative features from the matrix representations of trajectories by convolutional operations, Batch normalization, and $K$-max pooling operations. Extensive experiments on real datasets demonstrate that CNNTOP substantially outperforms existing solutions in terms of macro-Precision, macro-Recall, macro-F1, and accuracy.
Throughput Maximization of Mixed FSO/RF UAV-aided Mobile Relaying with a Buffer
Venue: arXiv cs.IT 2020
Year: 2020

In this paper, we consider an unmanned aerial vehicle (UAV) aided mobile relaying system under a buffer constraint. We propose a new relaying protocol employing mixed free-space optical/radio frequency (FSO/RF) communication, i.e., the source-relay and relay-destination links utilize FSO and RF links, respectively, under the buffer constraint at the UAV relay node. Taking the conditions of an imbalance in transmission rate between RF and FSO links into consideration, we study the trajectory optimization problem of buffer-constrained UAV relay node in order to maximize the end-to-end data throughput. Especially, we classify two relaying transmission schemes according to the delay requirements, i.e., i) delay-limited transmission and ii) delay-tolerant transmission. We solve the locally optimal trajectory problem of the UAV to maximize the throughput of ground user terminal. As a result, we propose an iterative algorithm that efficiently finds a local optimum solution for the throughput maximization problems. Through this algorithm, we present the resulting trajectories over the the atmospheric condition, the buffer size, and the delay requirement. Also, we show the optimum buffer size and the throughput-delay tradeoff for a given system. Our numerical results validate that the proposed buffer-aided mobile relaying scheme achieves 65.55% throughput gains compared to conventional static relaying scheme.
Spatial-Scale Aligned Network for Fine-Grained Recognition
Venue: arXiv cs.CV 2020
Year: 2020

Existing approaches for fine-grained visual recognition focus on learning marginal region-based representations while neglecting the spatial and scale misalignments, leading to inferior performance. In this paper, we propose the spatial-scale aligned network (SSANET) and implicitly address misalignments during the recognition process. Especially, SSANET consists of 1) a self-supervised proposal mining formula with Morphological Alignment Constraints; 2) a discriminative scale mining (DSM) module, which exploits the feature pyramid via a circulant matrix, and provides the Fourier solver for fast scale alignments; 3) an oriented pooling (OP) module, that performs the pooling operation in several pre-defined orientations. Each orientation defines one kind of spatial alignment, and the network automatically determines which is the optimal alignments through learning. With the proposed two modules, our algorithm can automatically determine the accurate local proposal regions and generate more robust target representations being invariant to various appearance variances. Extensive experiments verify that SSANET is competent at learning better spatial-scale invariant target representations, yielding superior performance on the fine-grained recognition task on several benchmarks.
Flexible Log File Parsing using Hidden Markov Models
Venue: arXiv cs.LG 2020
Year: 2020

We aim to model unknown file processing. As the content of log files often evolves over time, we established a dynamic statistical model which learns and adapts processing and parsing rules. First, we limit the amount of unstructured text by focusing only on those frequent patterns which lead to the desired output table similar to Vaarandi [10]. Second, we transform the found frequent patterns and the output stating the parsed table into a Hidden Markov Model (HMM). We use this HMM as a specific, however, flexible representation of a pattern for log file processing. With changes in the raw log file distorting learned patterns, we aim the model to adapt automatically in order to maintain high quality output. After training our model on one system type, applying the model and the resulting parsing rule to a different system with slightly different log file patterns, we achieve an accuracy over 99%.
FDFtNet: Facing Off Fake Images using Fake Detection Fine-tuning Network
Venue: arXiv cs.CV 2020
Year: 2020

Creating fake images and videos such as "Deepfake" has become much easier these days due to the advancement in Generative Adversarial Networks (GANs). Moreover, recent research such as the few-shot learning can create highly realistic personalized fake images with only a few images. Therefore, the threat of Deepfake to be used for a variety of malicious intents such as propagating fake images and videos becomes prevalent. And detecting these machine-generated fake images has been quite challenging than ever. In this work, we propose a light-weight robust fine-tuning neural network-based classifier architecture called Fake Detection Fine-tuning Network (FDFtNet), which is capable of detecting many of the new fake face image generation models, and can be easily combined with existing image classification networks and finetuned on a few datasets. In contrast to many existing methods, our approach aims to reuse popular pre-trained models with only a few images for fine-tuning to effectively detect fake images. The core of our approach is to introduce an image-based self-attention module called Fine-Tune Transformer that uses only the attention module and the down-sampling layer. This module is added to the pre-trained model and fine-tuned on a few data to search for new sets of feature space to detect fake images. We experiment with our FDFtNet on the GANsbased dataset (Progressive Growing GAN) and Deepfake-based dataset (Deepfake and Face2Face) with a small input image resolution of 64x64 that complicates detection. Our FDFtNet achieves an overall accuracy of 90.29% in detecting fake images generated from the GANs-based dataset, outperforming the state-of-the-art.