OpenRaiser/Intern-Atlas huggingface dataset:
To tune or not to tune? An Approach for Recommending Important Hyperparameters
Venue: arXiv cs.LG 2021
Year: 2021

Novel technologies in automated machine learning ease the complexity of algorithm selection and hyperparameter optimization. Hyperparameters are important for machine learning models as they significantly influence the performance of machine learning models. Many optimization techniques have achieved notable success in hyperparameter tuning and surpassed the performance of human experts. However, depending on such techniques as blackbox algorithms can leave machine learning practitioners without insight into the relative importance of different hyperparameters. In this paper, we consider building the relationship between the performance of the machine learning models and their hyperparameters to discover the trend and gain insights, with empirical results based on six classifiers and 200 datasets. Our results enable users to decide whether it is worth conducting a possibly time-consuming tuning strategy, to focus on the most important hyperparameters, and to choose adequate hyperparameter spaces for tuning. The results of our experiments show that gradient boosting and Adaboost outperform other classifiers across 200 problems. However, they need tuning to boost their performance. Overall, the results obtained from this study provide a quantitative basis to focus efforts toward guided automated hyperparameter optimization and contribute toward the development of better-automated machine learning frameworks.
An Introduction to Variational Inference
Venue: arXiv cs.LG 2021
Year: 2021

Approximating complex probability densities is a core problem in modern statistics. In this paper, we introduce the concept of Variational Inference (VI), a popular method in machine learning that uses optimization techniques to estimate complex probability densities. This property allows VI to converge faster than classical methods, such as, Markov Chain Monte Carlo sampling. Conceptually, VI works by choosing a family of probability density functions and then finding the one closest to the actual probability density -- often using the Kullback-Leibler (KL) divergence as the optimization metric. We introduce the Evidence Lower Bound to tractably compute the approximated probability density and we review the ideas behind mean-field variational inference. Finally, we discuss the applications of VI to variational auto-encoders (VAE) and VAE-Generative Adversarial Network (VAE-GAN). With this paper, we aim to explain the concept of VI and assist in future research with this approach.
Investigating Vulnerabilities of Deep Neural Policies
Venue: arXiv cs.LG 2021
Year: 2021

Reinforcement learning policies based on deep neural networks are vulnerable to imperceptible adversarial perturbations to their inputs, in much the same way as neural network image classifiers. Recent work has proposed several methods to improve the robustness of deep reinforcement learning agents to adversarial perturbations based on training in the presence of these imperceptible perturbations (i.e. adversarial training). In this paper, we study the effects of adversarial training on the neural policy learned by the agent. In particular, we follow two distinct parallel approaches to investigate the outcomes of adversarial training on deep neural policies based on worst-case distributional shift and feature sensitivity. For the first approach, we compare the Fourier spectrum of minimal perturbations computed for both adversarially trained and vanilla trained neural policies. Via experiments in the OpenAI Atari environments we show that minimal perturbations computed for adversarially trained policies are more focused on lower frequencies in the Fourier domain, indicating a higher sensitivity of these policies to low frequency perturbations. For the second approach, we propose a novel method to measure the feature sensitivities of deep neural policies and we compare these feature sensitivity differences in state-of-the-art adversarially trained deep neural policies and vanilla trained deep neural policies. We believe our results can be an initial step towards understanding the relationship between adversarial training and different notions of robustness for neural policies.
CSDS: A Fine-Grained Chinese Dataset for Customer Service Dialogue Summarization
Venue: arXiv cs.CL 2021
Year: 2021

Dialogue summarization has drawn much attention recently. Especially in the customer service domain, agents could use dialogue summaries to help boost their works by quickly knowing customer's issues and service progress. These applications require summaries to contain the perspective of a single speaker and have a clear topic flow structure, while neither are available in existing datasets. Therefore, in this paper, we introduce a novel Chinese dataset for Customer Service Dialogue Summarization (CSDS). CSDS improves the abstractive summaries in two aspects: (1) In addition to the overall summary for the whole dialogue, role-oriented summaries are also provided to acquire different speakers' viewpoints. (2) All the summaries sum up each topic separately, thus containing the topic-level structure of the dialogue. We define tasks in CSDS as generating the overall summary and different role-oriented summaries for a given dialogue. Next, we compare various summarization methods on CSDS, and experiment results show that existing methods are prone to generate redundant and incoherent summaries. Besides, the performance becomes much worse when analyzing the performance on role-oriented summaries and topic structures. We hope that this study could benchmark Chinese dialogue summarization and benefit further studies.
DuTrust: A Sentiment Analysis Dataset for Trustworthiness Evaluation
Venue: arXiv cs.CL 2021
Year: 2021

While deep learning models have greatly improved the performance of most artificial intelligence tasks, they are often criticized to be untrustworthy due to the black-box problem. Consequently, many works have been proposed to study the trustworthiness of deep learning. However, as most open datasets are designed for evaluating the accuracy of model outputs, there is still a lack of appropriate datasets for evaluating the inner workings of neural networks. The lack of datasets obviously hinders the development of trustworthiness research. Therefore, in order to systematically evaluate the factors for building trustworthy systems, we propose a novel and well-annotated sentiment analysis dataset to evaluate robustness and interpretability. To evaluate these factors, our dataset contains diverse annotations about the challenging distribution of instances, manual adversarial instances and sentiment explanations. Several evaluation metrics are further proposed for interpretability and robustness. Based on the dataset and metrics, we conduct comprehensive comparisons for the trustworthiness of three typical models, and also study the relations between accuracy, robustness and interpretability. We release this trustworthiness evaluation dataset at \url{https://github/xyz} and hope our work can facilitate the progress on building more trustworthy systems for real-world applications.
Towards Secure Wireless Mesh Networks for UAV Swarm Connectivity: Current Threats, Research, and Opportunities
Venue: arXiv cs.NI 2021
Year: 2021

UAVs are increasingly appearing in swarms or formations to leverage cooperative behavior, forming flying ad hoc networks. These UAV-enabled networks can meet several complex mission requirements and are seen as a potential enabler for many of the emerging use-cases in future communication networks. Such networks, however, are characterized by a highly dynamic and mobile environment with no guarantee of a central network infrastructure which can cause both connectivity and security issues. While wireless mesh networks are envisioned as a solution for such scenarios, these networks come with their own challenges and security vulnerabilities. In this paper, we analyze the key security and resilience issues resulting from the application of wireless mesh networks within UAV swarms. Specifically, we highlight the main challenges of applying current mesh technologies within the domain of UAV swarms and expose existing vulnerabilities across the communication stack. Based on this analysis, we present a security-focused architecture for UAV mesh communications. Finally, from the identification of these vulnerabilities, we discuss research opportunities posed by the unique challenges of UAV swarm connectivity.
Exploring Channel Probing to Determine Coherent Optical Transponder Configurations in a Long-Haul Network
Venue: arXiv cs.NI 2021
Year: 2021

We use channel probing to determine the best transponder configurations for spectral services in a long-haul production network. An estimation accuracy better than +/- 0,7dB in GSNR margin is obtained for lightpaths up to 5738km.
A Dynamic Game Approach to Designing Secure Interdependent IoT-Enabled Infrastructure Network
Venue: arXiv cs.NI 2021
Year: 2021

The emerging Internet of Things (IoT) applications that leverage ubiquitous connectivity and big data are facilitating the realization of smart everything initiatives. IoT-enabled infrastructures have naturally a multi-layer system architecture with an overlaid or underlaid device network and its coexisting infrastructure network. The connectivity between different components in these two heterogeneous interdependent networks plays an important role in delivering real-time information and ensuring a high-level situational awareness. However, IoT-enabled infrastructures face cyber threats due to the wireless nature of communications. Therefore, maintaining network connectivity in the presence of adversaries is a critical task for infrastructure network operators. In this paper, we establish a three-player three-stage dynamic game-theoretic framework including two network operators and one attacker to capture the secure design of multi-layer interdependent infrastructure networks by allocating limited resources. We use subgame perfect Nash equilibrium (SPE) to characterize the strategies of players with sequential moves. In addition, we assess the efficiency of the equilibrium network by comparing with its team optimal solution counterparts in which two network operators can coordinate. We further design a scalable algorithm to guide the construction of the equilibrium IoT-enabled infrastructure networks. Finally, we use case studies on the emerging paradigm of the Internet of Battlefield Things (IoBT) to corroborate the obtained results.
NOMA Assisted Multi-MEC Offloading for IoVT Networks
Venue: arXiv cs.NI 2021
Year: 2021

Nowadays, Internet of Video Things (IoVT) grows rapidly in terms of quantity and computation demands. In spite of the higher local computation capability on visual processing compared with conventional Internet of Things devices, IoVT devices need to offload partial visual processing tasks to the mobile edge computing (MEC) server wirelessly due to its larger computation demands. However, visual processing task offloading is limited by uplink throughput and computation capability of the MEC server. To break through these limitations, a novel non-orthogonal multiple access (NOMA) assisted IoVT framework with multiple MEC servers is proposed, where NOMA is exploited to improve uplink throughput and MEC servers are co-located with base stations to provide enough computation capability for offloading. In the proposed framework, the association strategy, uplink visual data transmission assisted by NOMA and division of the visual processing tasks as well as computation resource allocation at the MEC servers are jointly optimized to minimize the total delay of all visual processing tasks, while meeting the delay requirements of all IoVT devices. Simulation results demonstrate that significant performance gains can be achieved by proposed joint optimization with NOMA transmission and multi-MEC offloading in the heterogeneous IoVT network.
Enterprise Architecture Model Transformation Engine
Venue: arXiv cs.DC 2021
Year: 2021

With increasing linkage within value chains, the IT systems of different companies are also being connected with each other. This enables the integration of services within the movement of Industry 4.0 in order to improve the quality and performance of the processes. Enterprise architecture models form the basis for this with a better buisness IT-alignment. However, the heterogeneity of the modeling frameworks and description languages makes a concatenation considerably difficult, especially differences in syntax, semantic and relations. Therefore, this paper presents a transformation engine to convert enterprise architecture models between several languages. We developed the first generic translation approach that is free of specific meta-modeling, which is flexible adaptable to arbitrary modeling languages. The transformation process is defined by various pattern matching techniques using a rule-based description language. It uses set theory and first-order logic for an intuitive description as a basis. The concept is practical evaluated using an example in the area of a large German IT-service provider. Anyhow, the approach is applicable between a wide range of enterprise architecture frameworks.
LUAI Challenge 2021 on Learning to Understand Aerial Images
Venue: arXiv cs.CV 2021
Year: 2021

This report summarizes the results of Learning to Understand Aerial Images (LUAI) 2021 challenge held on ICCV 2021, which focuses on object detection and semantic segmentation in aerial images. Using DOTA-v2.0 and GID-15 datasets, this challenge proposes three tasks for oriented object detection, horizontal object detection, and semantic segmentation of common categories in aerial images. This challenge received a total of 146 registrations on the three tasks. Through the challenge, we hope to draw attention from a wide range of communities and call for more efforts on the problems of learning to understand aerial images.
Small Number of Communities in Twitter Keyword Networks
Venue: arXiv cs.SI 2021
Year: 2021

We investigate networks formed by keywords in tweets and study their community structure. Based on datasets of tweets mined from over seven hundred political figures in the U.S. and Canada, we hypothesize that such Twitter keyword networks exhibit a small number of communities. Our results are further reinforced by considering via so-called pseudo-tweets generated randomly and using AI-based language generation software. We speculate as to the possible origins of the small community hypothesis and further attempts at validating it.
Thermal Management in Large Data Centers: Security Threats and Mitigation
Venue: arXiv cs.CR 2021
Year: 2021

Data centres are experiencing significant growth in their scale, especially, with the ever-increasing demand for cloud and IoT services. However, this rapid growth has raised numerous security issues and vulnerabilities; new types of strategic cyber-attacks are aimed at specific physical components of data centres that keep them operating. Attacks against temperature monitoring and cooling systems of data centres, also known as thermal attacks, can cause a complete meltdown and are generally considered difficult to address. In this paper, we focus on this issue by analysing the potential security threats to these systems and their impact on the overall data center safety and performance. We also present current thermal anomaly detection methods and their limitations. Finally, we propose a hybrid method that uses multi-variant anomaly detection to prevent thermal attacks, as well as a fuzzy-based health factor to enhance data center thermal awareness and security
Incremental Learning for Personalized Recommender Systems
Venue: arXiv cs.IR 2021
Year: 2021

Ubiquitous personalized recommender systems are built to achieve two seemingly conflicting goals, to serve high quality content tailored to individual user's taste and to adapt quickly to the ever changing environment. The former requires a complex machine learning model that is trained on a large amount of data; the latter requires frequent update to the model. We present an incremental learning solution to provide both the training efficiency and the model quality. Our solution is based on sequential Bayesian update and quadratic approximation. Our focus is on large-scale personalized logistic regression models, with extensions to deep learning models. This paper fills in the gap between the theory and the practice by addressing a few implementation challenges that arise when applying incremental learning to large personalized recommender systems. Detailed offline and online experiments demonstrated our approach can significantly shorten the training time while maintaining the model accuracy. The solution is deployed in LinkedIn and directly applicable to industrial scale recommender systems.
A Theoretical Framework for Online Information Search
Venue: arXiv cs.IR 2021
Year: 2021

A significant part of human activity today consists of searching for a piece of information online, utilizing knowledge repositories. This endeavor may be time-consuming if the individual searching for the information is unfamiliar with the subject matter of that information. However, experts can aid individuals find relevant information by searching online. This paper describes a theoretical framework to model the dynamic process by which requests for information come to a system of experts, who then answer the requests by searching for those pieces of information.
On the Multilingual Capabilities of Very Large-Scale English Language Models
Venue: arXiv cs.CL 2021
Year: 2021

Generative Pre-trained Transformers (GPTs) have recently been scaled to unprecedented sizes in the history of machine learning. These models, solely trained on the language modeling objective, have been shown to exhibit outstanding few-shot learning capabilities in a number of different tasks. Nevertheless, aside from anecdotal experiences, little is known regarding their multilingual capabilities, given the fact that the pre-training corpus is almost entirely composed of English text. In this work, we investigate the multilingual skills of GPT-3, focusing on one language that barely appears in the pre-training corpus, Catalan, which makes the results especially meaningful; we assume that our results may be relevant for other languages as well. We find that the model shows an outstanding performance, particularly in generative tasks, with predictable limitations mostly in language understanding tasks but still with remarkable results given the zero-shot scenario. We investigate its potential and limits in extractive question-answering and natural language generation, as well as the effect of scale in terms of model size.
MultPIM: Fast Stateful Multiplication for Processing-in-Memory
Venue: arXiv cs.AR 2021
Year: 2021

Processing-in-memory (PIM) seeks to eliminate computation/memory data transfer using devices that support both storage and logic. Stateful logic techniques such as IMPLY, MAGIC and FELIX can perform logic gates within memristive crossbar arrays with massive parallelism. Multiplication via stateful logic is an active field of research due to the wide implications. Recently, RIME has become the state-of-the-art algorithm for stateful single-row multiplication by using memristive partitions, reducing the latency of the previous state-of-the-art by 5.1x. In this paper, we begin by proposing novel partition-based computation techniques for broadcasting and shifting data. Then, we design an in-memory multiplication algorithm based on the carry-save add-shift (CSAS) technique. Finally, we develop a novel stateful full-adder that significantly improves the state-of-the-art (FELIX) design. These contributions constitute MultPIM, a multiplier that reduces state-of-the-art time complexity from quadratic to linear-log. For 32-bit numbers, MultPIM improves latency by an additional 4.2x over RIME, while even slightly reducing area overhead. Furthermore, we optimize MultPIM for full-precision matrix-vector multiplication and improve latency by 25.5x over FloatPIM matrix-vector multiplication.
Exploiting the Electrothermal Timescale in PrMnO3 RRAM for a compact, clock-less neuron exhibiting biological spiking patterns
Venue: arXiv cs.ET 2021
Year: 2021

Spiking Neural Networks (SNNs) are gaining widespread momentum in the field of neuromorphic computing. These network systems integrated with neurons and synapses provide computational efficiency by mimicking the human brain. It is desired to incorporate the biological neuronal dynamics, including complex spiking patterns which represent diverse brain activities within the neural networks. Earlier hardware realization of neurons was (1) area intensive because of large capacitors in the circuit design, (2) neuronal spiking patterns were demonstrated with clocked neurons at the device level. To achieve more realistic biological neuron spiking behavior, emerging memristive devices are considered promising alternatives. In this paper, we propose, PrMnO3(PMO) -RRAM device-based neuron. The voltage-controlled electrothermal timescales of the compact PMO RRAM device replace the electrical timescales of charging a large capacitor. The electrothermal timescale is used to implement an integration block with multiple voltage-controlled timescales coupled with a refractory block to generate biological neuronal dynamics. Here, first, a Verilog-A implementation of the thermal device model is demonstrated, which captures the current-temperature dynamics of the PMO device. Second, a driving circuitry is designed to mimic different spiking patterns of cortical neurons, including Intrinsic bursting (IB) and Chattering (CH). Third, a neuron circuit model is simulated, which includes the PMO RRAM device model and the driving circuitry to demonstrate the asynchronous neuron behavior. Finally, a hardware-software hybrid analysis is done in which the PMO RRAM device is experimentally characterized to mimic neuron spiking dynamics. The work presents a realizable and more biologically comparable hardware-efficient solution for large-scale SNNs.
Generalizing Weighted Trees: A Bridge from Bitcoin to GHOST
Venue: arXiv cs.CR 2021
Year: 2021

Despite the tremendous interest in cryptocurrencies like Bitcoin and Ethereum today, many aspects of the underlying consensus protocols are poorly understood. Therefore, the search for protocols that improve either throughput or security (or both) continues. Bitcoin always selects the longest chain (i.e., the one with most work). Forks may occur when two miners extend the same block simultaneously, and the frequency of forks depends on how fast blocks are propagated in the network. In the GHOST protocol, used by Ethereum, all blocks involved in the fork contribute to the security. However, the greedy chain selection rule of GHOST does not consider the full information available in the block tree, which has led to some concerns about its security. This paper introduces a new family of protocols, called Medium, which takes the structure of the whole block tree into account, by weighting blocks differently according to their depths. Bitcoin and GHOST result as special cases. This protocol leads to new insights about the security of Bitcoin and GHOST and paves the way for developing network- and application-specific protocols, in which the influence of forks on the chain-selection process can be controlled. It is shown that almost all protocols in this family achieve strictly greater throughput than Bitcoin (at the same security level) and resist attacks that can be mounted against GHOST.