OpenRaiser/Intern-Atlas huggingface dataset:
Multi-head or Single-head? An Empirical Comparison for Transformer Training
Venue: arXiv cs.CL 2021
Year: 2021

Multi-head attention plays a crucial role in the recent success of Transformer models, which leads to consistent performance improvements over conventional attention in various applications. The popular belief is that this effectiveness stems from the ability of jointly attending multiple positions. In this paper, we first demonstrate that jointly attending multiple positions is not a unique feature of multi-head attention, as multi-layer single-head attention also attends multiple positions and is more effective. Then, we suggest the main advantage of the multi-head attention is the training stability, since it has less number of layers than the single-head attention, when attending the same number of positions. For example, 24-layer 16-head Transformer (BERT-large) and 384-layer single-head Transformer has the same total attention head number and roughly the same model size, while the multi-head one is significantly shallower. Meanwhile, we show that, with recent advances in deep learning, we can successfully stabilize the training of the 384-layer Transformer. As the training difficulty is no longer a bottleneck, substantially deeper single-head Transformer achieves consistent performance improvements without tuning hyper-parameters.
Adaptive Low-Rank Regularization with Damping Sequences to Restrict Lazy Weights in Deep Networks
Venue: arXiv cs.LG 2021
Year: 2021

Overfitting is one of the critical problems in deep neural networks. Many regularization schemes try to prevent overfitting blindly. However, they decrease the convergence speed of training algorithms. Adaptive regularization schemes can solve overfitting more intelligently. They usually do not affect the entire network weights. This paper detects a subset of the weighting layers that cause overfitting. The overfitting recognizes by matrix and tensor condition numbers. An adaptive regularization scheme entitled Adaptive Low-Rank (ALR) is proposed that converges a subset of the weighting layers to their Low-Rank Factorization (LRF). It happens by minimizing a new Tikhonov-based loss function. ALR also encourages lazy weights to contribute to the regularization when epochs grow up. It uses a damping sequence to increment layer selection likelihood in the last generations. Thus before falling the training accuracy, ALR reduces the lazy weights and regularizes the network substantially. The experimental results show that ALR regularizes the deep networks well with high training speed and low resource usage.
Comparing the behavior of OpenMP Implementations with various Applications on two different Fujitsu A64FX platforms
Venue: arXiv cs.PF 2021
Year: 2021

The development of the A64FX processor by Fujitsu has been a massive innovation in vectorized processors and led to Fugaku: the current world's fastest supercomputer. We use a variety of tools to analyze the behavior and performance of several OpenMP applications with different compilers, and how these applications scale on the different A64FX processors on clusters at Stony Brook University and RIKEN.
Topological Indoor Mapping through WiFi Signals
Venue: arXiv cs.NI 2021
Year: 2021

The ubiquitous presence of WiFi access points and mobile devices capable of measuring WiFi signal strengths allow for real-world applications in indoor localization and mapping. In particular, no additional infrastructure is required. Previous approaches in this field were, however, often hindered by problems such as effortful map-building processes, changing environments and hardware differences. We tackle these problems focussing on topological maps. These represent discrete locations, such as rooms, and their relations, e.g., distances and transition frequencies. In our unsupervised method, we employ WiFi signal strength distributions, dimension reduction and clustering. It can be used in settings where users carry mobile devices and follow their normal routine. We aim for applications in short-lived indoor events such as conferences.
A Distance-based Separability Measure for Internal Cluster Validation
Venue: arXiv cs.LG 2021
Year: 2021

To evaluate clustering results is a significant part of cluster analysis. Since there are no true class labels for clustering in typical unsupervised learning, many internal cluster validity indices (CVIs), which use predicted labels and data, have been created. Without true labels, to design an effective CVI is as difficult as to create a clustering method. And it is crucial to have more CVIs because there are no universal CVIs that can be used to measure all datasets and no specific methods of selecting a proper CVI for clusters without true labels. Therefore, to apply a variety of CVIs to evaluate clustering results is necessary. In this paper, we propose a novel internal CVI -- the Distance-based Separability Index (DSI), based on a data separability measure. We compared the DSI with eight internal CVIs including studies from early Dunn (1974) to most recent CVDD (2019) and an external CVI as ground truth, by using clustering results of five clustering algorithms on 12 real and 97 synthetic datasets. Results show DSI is an effective, unique, and competitive CVI to other compared CVIs. We also summarized the general process to evaluate CVIs and created the rank-difference metric for comparison of CVIs' results.
Smoothed Multi-View Subspace Clustering
Venue: arXiv cs.CV 2021
Year: 2021

In recent years, multi-view subspace clustering has achieved impressive performance due to the exploitation of complementary imformation across multiple views. However, multi-view data can be very complicated and are not easy to cluster in real-world applications. Most existing methods operate on raw data and may not obtain the optimal solution. In this work, we propose a novel multi-view clustering method named smoothed multi-view subspace clustering (SMVSC) by employing a novel technique, i.e., graph filtering, to obtain a smooth representation for each view, in which similar data points have similar feature values. Specifically, it retains the graph geometric features through applying a low-pass filter. Consequently, it produces a ``clustering-friendly" representation and greatly facilitates the downstream clustering task. Extensive experiments on benchmark datasets validate the superiority of our approach. Analysis shows that graph filtering increases the separability of classes.
Anomaly Detection in Dynamic Graphs via Transformer
Venue: arXiv cs.LG 2021
Year: 2021

Detecting anomalies for dynamic graphs has drawn increasing attention due to their wide applications in social networks, e-commerce, and cybersecurity. Recent deep learning-based approaches have shown promising results over shallow methods. However, they fail to address two core challenges of anomaly detection in dynamic graphs: the lack of informative encoding for unattributed nodes and the difficulty of learning discriminate knowledge from coupled spatial-temporal dynamic graphs. To overcome these challenges, in this paper, we present a novel Transformer-based Anomaly Detection framework for DYnamic graphs (TADDY). Our framework constructs a comprehensive node encoding strategy to better represent each node's structural and temporal roles in an evolving graphs stream. Meanwhile, TADDY captures informative representation from dynamic graphs with coupled spatial-temporal patterns via a dynamic graph transformer model. The extensive experimental results demonstrate that our proposed TADDY framework outperforms the state-of-the-art methods by a large margin on six real-world datasets.
Generating various airfoil shapes with required lift coefficient using conditional variational autoencoders
Venue: arXiv cs.CE 2021
Year: 2021

Multiple shapes must be obtained in the mechanical design process to satisfy the required design specifications. The inverse design problem has been analyzed in previous studies to obtain such shapes. However, finding multiple shapes in a short computation period is difficult while using the conventional methods. This paper proposes the use of the conditional variational autoencoders (CVAE) with normal distribution, denoted by N-CVAE, along with the von Mises-Fischer distribution, denoted by S-CVAE, to find multiple solutions for the inverse design problems. Both the CVAE models embed shapes into a latent space. The S-CVAE enables the separation of data in the latent space, whereas the N-CVAE embeds the data in a narrow space. These different features are used for various tasks in this study. In one of the tasks, the dataset consists of only one type of data and generates similar airfoils. Here, S-CVAE outperforms N-CVAE because it can separate the data. Another task involves combining different types of airfoils and generating new types of data. N-CVAE is useful in this instance since it embeds different shapes in the same latent area, due to which, the model outputs intermediate shapes of different types. The shape-generation capability of S-CVAE and N-CVAE are experimentally compared in this study.
Goal-Directed Planning by Reinforcement Learning and Active Inference
Venue: arXiv cs.LG 2021
Year: 2021

What is the difference between goal-directed and habitual behavior? We propose a novel computational framework of decision making with Bayesian inference, in which everything is integrated as an entire neural network model. The model learns to predict environmental state transitions by self-exploration and generating motor actions by sampling stochastic internal states ${z}$. Habitual behavior, which is obtained from the prior distribution of ${z}$, is acquired by reinforcement learning. Goal-directed behavior is determined from the posterior distribution of ${z}$ by planning, using active inference which optimizes the past, current and future ${z}$ by minimizing the variational free energy for the desired future observation constrained by the observed sensory sequence. We demonstrate the effectiveness of the proposed framework by experiments in a sensorimotor navigation task with camera observations and continuous motor actions.
Labelling Drifts in a Fault Detection System for Wind Turbine Maintenance
Venue: arXiv cs.LG 2021
Year: 2021

A failure detection system is the first step towards predictive maintenance strategies. A popular data-driven method to detect incipient failures and anomalies is the training of normal behaviour models by applying a machine learning technique like feed-forward neural networks (FFNN) or extreme learning machines (ELM). However, the performance of any of these modelling techniques can be deteriorated by the unexpected rise of non-stationarities in the dynamic environment in which industrial assets operate. This unpredictable statistical change in the measured variable is known as concept drift. In this article a wind turbine maintenance case is presented, where non-stationarities of various kinds can happen unexpectedly. Such concept drift events are desired to be detected by means of statistical detectors and window-based approaches. However, in real complex systems, concept drifts are not as clear and evident as in artificially generated datasets. In order to evaluate the effectiveness of current drift detectors and also to design an appropriate novel technique for this specific industrial application, it is essential to dispose beforehand of a characterization of the existent drifts. Under the lack of information in this regard, a methodology for labelling concept drift events in the lifetime of wind turbines is proposed. This methodology will facilitate the creation of a drift database that will serve both as a training ground for concept drift detectors and as a valuable information to enhance the knowledge about maintenance of complex systems.
A System-Level Voltage/Frequency Scaling Characterization Framework for Multicore CPUs
Venue: arXiv cs.AR 2021
Year: 2021

Supply voltage scaling is one of the most effective techniques to reduce the power consumption of microprocessors. However, technology limitations such as aging and process variability enforce microprocessor designers to apply pessimistic voltage guardbands to guarantee correct operation in the field for any foreseeable workload. This worst-case design practice makes energy efficiency hard to scale with technology evolution. Improving energy-efficiency requires the identification of the chip design margins through time-consuming and comprehensive characterization of its operational limits. Such a characterization of state-of-the-art multi-core CPUs fabricated in aggressive technologies is a multi-parameter process, which requires statistically significant information. In this paper, we present an automated framework to support system-level voltage and frequency scaling characterization of Applied Micro's state-of-the-art ARMv8-based multicore CPUs used in the X-Gene 2 micro-server family. The fully automated framework can provide fine-grained information of the system's state by monitoring any abnormal behavior that may occur during reduced supply voltage conditions. We also propose a new metric to quantify the behavior of a microprocessor when it operates beyond nominal conditions. Our experimental results demonstrate potential uses of the characterization framework to identify the limits of operation for improved energy efficiency.
Weakly Supervised Pre-Training for Multi-Hop Retriever
Venue: arXiv cs.CL 2021
Year: 2021

In multi-hop QA, answering complex questions entails iterative document retrieval for finding the missing entity of the question. The main steps of this process are sub-question detection, document retrieval for the sub-question, and generation of a new query for the final document retrieval. However, building a dataset that contains complex questions with sub-questions and their corresponding documents requires costly human annotation. To address the issue, we propose a new method for weakly supervised multi-hop retriever pre-training without human efforts. Our method includes 1) a pre-training task for generating vector representations of complex questions, 2) a scalable data generation method that produces the nested structure of question and sub-question as weak supervision for pre-training, and 3) a pre-training model structure based on dense encoders. We conduct experiments to compare the performance of our pre-trained retriever with several state-of-the-art models on end-to-end multi-hop QA as well as document retrieval. The experimental results show that our pre-trained retriever is effective and also robust on limited data and computational resources.
Exploring Counterfactual Explanations Through the Lens of Adversarial Examples: A Theoretical and Empirical Analysis
Venue: arXiv cs.LG 2021
Year: 2021

As machine learning (ML) models become more widely deployed in high-stakes applications, counterfactual explanations have emerged as key tools for providing actionable model explanations in practice. Despite the growing popularity of counterfactual explanations, a deeper understanding of these explanations is still lacking. In this work, we systematically analyze counterfactual explanations through the lens of adversarial examples. We do so by formalizing the similarities between popular counterfactual explanation and adversarial example generation methods identifying conditions when they are equivalent. We then derive the upper bounds on the distances between the solutions output by counterfactual explanation and adversarial example generation methods, which we validate on several real-world data sets. By establishing these theoretical and empirical similarities between counterfactual explanations and adversarial examples, our work raises fundamental questions about the design and development of existing counterfactual explanation algorithms.
SPBERT: An Efficient Pre-training BERT on SPARQL Queries for Question Answering over Knowledge Graphs
Venue: arXiv cs.CL 2021
Year: 2021

In this paper, we propose SPBERT, a transformer-based language model pre-trained on massive SPARQL query logs. By incorporating masked language modeling objectives and the word structural objective, SPBERT can learn general-purpose representations in both natural language and SPARQL query language. We investigate how SPBERT and encoder-decoder architecture can be adapted for Knowledge-based QA corpora. We conduct exhaustive experiments on two additional tasks, including SPARQL Query Construction and Answer Verbalization Generation. The experimental results show that SPBERT can obtain promising results, achieving state-of-the-art BLEU scores on several of these tasks.
Software-Defined Networking for Data Centre Network Management: A Survey
Venue: arXiv cs.NI 2021
Year: 2021

Data centres are growing in numbers and size, and their networks expanding to carry larger amounts of traffic. The traffic profile is constantly varying, particularly in cloud data centres where tenants arrive, leave, and may change their resource requirements in between, and so the network configuration must change at a commensurate rate. Software-Defined Networking - programmatic control of network configuration - has been critical to meeting the demands of modern data centre network management, and has been the subject of intense focus by the research community, working in conjunction with industry. In this survey, we review Software-Defined Networking research targeting the management and operation of data centre networks.
V-Edge: Virtual Edge Computing as an Enabler for Novel Microservices and Cooperative Computing
Venue: arXiv cs.NI 2021
Year: 2021

As we move from 5G to 6G, edge computing is one of the concepts that needs revisiting. Its core idea is still intriguing: instead of sending all data and tasks from an end user's device to the cloud, possibly covering thousands of kilometers and introducing delays that are just owed to limited propagation speed, edge servers deployed in close proximity to the user, e.g., at some 5G gNB, serve as proxy for the cloud. Yet this promising idea is hampered by the limited availability of such edge servers. In this paper, we discuss a way forward, namely the virtual edge computing (V-Edge) concept. V-Edge bridges the gap between cloud, edge, and fog by virtualizing all available resources including the end users' devices and making these resources widely available using well-defined interfaces. V-Edge also acts as an enabler for novel microservices as well as cooperative computing solutions. We introduce the general V-Edge architecture and we characterize some of the key research challenges to overcome, in order to enable wide-spread and even more powerful edge services.
Discerning Generic Event Boundaries in Long-Form Wild Videos
Venue: arXiv cs.CV 2021
Year: 2021

Detecting generic, taxonomy-free event boundaries invideos represents a major stride forward towards holisticvideo understanding. In this paper we present a technique forgeneric event boundary detection based on a two stream in-flated 3D convolutions architecture, which can learn spatio-temporal features from videos. Our work is inspired from theGeneric Event Boundary Detection Challenge (part of CVPR2021 Long Form Video Understanding- LOVEU Workshop).Throughout the paper we provide an in-depth analysis ofthe experiments performed along with an interpretation ofthe results obtained.
Challenges and Limitations with the Metrics Measuring the Complexity of Code-Mixed Text
Venue: arXiv cs.CL 2021
Year: 2021

Code-mixing is a frequent communication style among multilingual speakers where they mix words and phrases from two different languages in the same utterance of text or speech. Identifying and filtering code-mixed text is a challenging task due to its co-existence with monolingual and noisy text. Over the years, several code-mixing metrics have been extensively used to identify and validate code-mixed text quality. This paper demonstrates several inherent limitations of code-mixing metrics with examples from the already existing datasets that are popularly used across various experiments.
Towards Robotic Laboratory Automation Plug & Play: The "LAPP" Framework
Venue: arXiv cs.RO 2021
Year: 2021

Increasing the level of automation in pharmaceutical laboratories and production facilities plays a crucial role in delivering medicine to patients. However, the particular requirements of this field make it challenging to adapt cutting-edge technologies present in other industries. This article provides an overview of relevant approaches and how they can be utilized in the pharmaceutical industry, especially in development laboratories. Recent advancements include the application of flexible mobile manipulators capable of handling complex tasks. However, integrating devices from many different vendors into an end-to-end automation system is complicated due to the diversity of interfaces. Therefore, various approaches for standardization are considered in this article, and a concept is proposed for taking them a step further. This concept enables a mobile manipulator with a vision system to "learn" the pose of each device and - utilizing a barcode - fetch interface information from a universal cloud database. This information includes control and communication protocol definitions and a representation of robot actions needed to operate the device. In order to define the movements in relation to the device, devices have to feature - besides the barcode - a fiducial marker as standard. The concept will be elaborated following appropriate research activities in follow-up papers.
Enhancing user creativity: Semantic measures for idea generation
Venue: arXiv cs.CL 2021
Year: 2021

Human creativity generates novel ideas to solve real-world problems. This thereby grants us the power to transform the surrounding world and extend our human attributes beyond what is currently possible. Creative ideas are not just new and unexpected, but are also successful in providing solutions that are useful, efficient and valuable. Thus, creativity optimizes the use of available resources and increases wealth. The origin of human creativity, however, is poorly understood, and semantic measures that could predict the success of generated ideas are currently unknown. Here, we analyze a dataset of design problem-solving conversations in real-world settings by using 49 semantic measures based on WordNet 3.1 and demonstrate that a divergence of semantic similarity, an increased information content, and a decreased polysemy predict the success of generated ideas. The first feedback from clients also enhances information content and leads to a divergence of successful ideas in creative problem solving. These results advance cognitive science by identifying real-world processes in human problem solving that are relevant to the success of produced solutions and provide tools for real-time monitoring of problem solving, student training and skill acquisition. A selected subset of information content (IC Sánchez-Batet) and semantic similarity (Lin/Sánchez-Batet) measures, which are both statistically powerful and computationally fast, could support the development of technologies for computer-assisted enhancements of human creativity or for the implementation of creativity in machines endowed with general artificial intelligence.