OpenRaiser/Intern-Atlas huggingface dataset:
Optical Networks for Composable Data Centers
Venue: arXiv cs.NI 2021
Year: 2021

Composable data centers (DCs) have been proposed to enable greater efficiencies as the uptake of on-demand computing services grows. In this article we give an overview of composable DCs by discussing their enabling technologies, benefits, challenges, and research directions. We then describe a network for composable DCs that leverages optical communication technologies and components to implement a targeted design. Relative to the implementation of a generic design that requires a (high capacity) dedicated transceiver on each point-to-point link on a mesh optical fabric in a composable DC rack, the targeted design can significantly reduce capital expenditure (by up to 34 times) because fewer transceivers are used. This is achieved with little or no degradation of expected performance in composable DCs.
Ghosts in Neural Networks: Existence, Structure and Role of Infinite-Dimensional Null Space
Venue: arXiv cs.LG 2021
Year: 2021

Overparametrization has been remarkably successful for deep learning studies. This study investigates an overlooked but important aspect of overparametrized neural networks, that is, the null components in the parameters of neural networks, or the ghosts. Since deep learning is not explicitly regularized, typical deep learning solutions contain null components. In this paper, we present a structure theorem of the null space for a general class of neural networks. Specifically, we show that any null element can be uniquely written by the linear combination of ridgelet transforms. In general, it is quite difficult to fully characterize the null space of an arbitrarily given operator. Therefore, the structure theorem is a great advantage for understanding a complicated landscape of neural network parameters. As applications, we discuss the roles of ghosts on the generalization performance of deep learning.
Sentence Embeddings using Supervised Contrastive Learning
Venue: arXiv cs.CL 2021
Year: 2021

Sentence embeddings encode sentences in fixed dense vectors and have played an important role in various NLP tasks and systems. Methods for building sentence embeddings include unsupervised learning such as Quick-Thoughts and supervised learning such as InferSent. With the success of pretrained NLP models, recent research shows that fine-tuning pretrained BERT on SNLI and Multi-NLI data creates state-of-the-art sentence embeddings, outperforming previous sentence embeddings methods on various evaluation benchmarks. In this paper, we propose a new method to build sentence embeddings by doing supervised contrastive learning. Specifically our method fine-tunes pretrained BERT on SNLI data, incorporating both supervised crossentropy loss and supervised contrastive loss. Compared with baseline where fine-tuning is only done with supervised cross-entropy loss similar to current state-of-the-art method SBERT, our supervised contrastive method improves 2.8% in average on Semantic Textual Similarity (STS) benchmarks and 1.05% in average on various sentence transfer tasks.
Joint System-Wise Optimization for Pipeline Goal-Oriented Dialog System
Venue: arXiv cs.CL 2021
Year: 2021

Recent work (Takanobu et al., 2020) proposed the system-wise evaluation on dialog systems and found that improvement on individual components (e.g., NLU, policy) in prior work may not necessarily bring benefit to pipeline systems in system-wise evaluation. To improve the system-wise performance, in this paper, we propose new joint system-wise optimization techniques for the pipeline dialog system. First, we propose a new data augmentation approach which automates the labeling process for NLU training. Second, we propose a novel stochastic policy parameterization with Poisson distribution that enables better exploration and offers a principled way to compute policy gradient. Third, we propose a reward bonus to help policy explore successful dialogs. Our approaches outperform the competitive pipeline systems from Takanobu et al. (2020) by big margins of 12% success rate in automatic system-wise evaluation and of 16% success rate in human evaluation on the standard multi-domain benchmark dataset MultiWOZ 2.1, and also outperform the recent state-of-the-art end-to-end trained model from DSTC9.
Tracking by Joint Local and Global Search: A Target-aware Attention based Approach
Venue: arXiv cs.CV 2021
Year: 2021

Tracking-by-detection is a very popular framework for single object tracking which attempts to search the target object within a local search window for each frame. Although such local search mechanism works well on simple videos, however, it makes the trackers sensitive to extremely challenging scenarios, such as heavy occlusion and fast motion. In this paper, we propose a novel and general target-aware attention mechanism (termed TANet) and integrate it with tracking-by-detection framework to conduct joint local and global search for robust tracking. Specifically, we extract the features of target object patch and continuous video frames, then we concatenate and feed them into a decoder network to generate target-aware global attention maps. More importantly, we resort to adversarial training for better attention prediction. The appearance and motion discriminator networks are designed to ensure its consistency in spatial and temporal views. In the tracking procedure, we integrate the target-aware attention with multiple trackers by exploring candidate search regions for robust tracking. Extensive experiments on both short-term and long-term tracking benchmark datasets all validated the effectiveness of our algorithm. The project page of this paper can be found at \url{https://sites.google.com/view/globalattentiontracking/home/extend}.
A Case Study: Using Genetic Algorithm for Job Scheduling Problem
Venue: arXiv cs.NE 2021
Year: 2021

Nowadays, DevOps pipelines of huge projects are getting more and more complex. Each job in the pipeline might need different requirements including specific hardware specifications and dependencies. To achieve minimal makespan, developers always apply as much machines as possible. Consequently, others may be stalled for waiting resource released. Minimizing the makespan of each job using a few resource is a challenging problem. In this study, it is aimed to 1) automatically determine the priority of jobs to reduce the waiting time in the line, 2) automatically allocate the machine resource to each job. In this work, the problem is formulated as a multi-objective optimization problem. We use GA algorithm to automatically determine job priorities and resource demand for minimizing individual makespan and resource usage. Finally, the experimental results show that our proposed priority list generation algorithm is more effective than current priority list producing method in the aspects of makespan and allocated machine count.
A Blockchain-Based Trust Management Framework with Verifiable Interactions
Venue: arXiv cs.CR 2021
Year: 2021

There has been tremendous interest in the development of formal trust models and metrics through the use of analytics (e.g., Belief Theory and Bayesian models), logics (e.g., Epistemic and Subjective Logic) and other mathematical models. The choice of trust metric will depend on context, circumstance and user requirements and there is no single best metric for use in all circumstances. Where different users require different trust metrics to be employed the trust score calculations should still be based on all available trust evidence. Trust is normally computed using past experiences but, in practice (especially in centralised systems), the validity and accuracy of these experiences are taken for granted. In this paper, we provide a formal framework and practical blockchain-based implementation that allows independent trust providers to implement different trust metrics in a distributed manner while still allowing all trust providers to base their calculations on a common set of trust evidence. Further, our design allows experiences to be provably linked to interactions without the need for a central authority. This leads to the notion of evidence-based trust with provable interactions. Leveraging blockchain allows the trust providers to offer their services in a competitive manner, charging fees while users are provided with payments for recording experiences. Performance details of the blockchain implementation are provided.
Interaction-Grounded Learning
Venue: arXiv cs.LG 2021
Year: 2021

Consider a prosthetic arm, learning to adapt to its user's control signals. We propose Interaction-Grounded Learning for this novel setting, in which a learner's goal is to interact with the environment with no grounding or explicit reward to optimize its policies. Such a problem evades common RL solutions which require an explicit reward. The learning agent observes a multidimensional context vector, takes an action, and then observes a multidimensional feedback vector. This multidimensional feedback vector has no explicit reward information. In order to succeed, the algorithm must learn how to evaluate the feedback vector to discover a latent reward signal, with which it can ground its policies without supervision. We show that in an Interaction-Grounded Learning setting, with certain natural assumptions, a learner can discover the latent reward and ground its policy for successful interaction. We provide theoretical guarantees and a proof-of-concept empirical evaluation to demonstrate the effectiveness of our proposed approach.
Erratum: Leveraging Flexible Tree Matching to Repair Broken Locators in Web Automation Scripts
Venue: arXiv cs.SE 2021
Year: 2021

Web applications are constantly evolving to integrate new features and fix reported bugs. Even an imperceptible change can sometimes entail significant modifications of the Document Object Model (DOM), which is the underlying model used by browsers to render all the elements included in a web application. Scripts that interact with web applications (e.g. web test scripts, crawlers, or robotic process automation) rely on this continuously evolving DOM which means they are often particularly fragile. More precisely, the major cause of breakages observed in automation scripts are element locators, which are identifiers used by automation scripts to navigate across the DOM. When the DOM evolves, these identifiers tend to break, thus causing the related scripts to no longer locate the intended target elements. For this reason, several contributions explored the idea of automatically repairing broken locators on a page. These works attempt to repair a given broken locator by scanning all elements in the new DOM to find the most similar one. Unfortunately, this approach fails to scale when the complexity of web pages grows, leading either to long computation times or incorrect element repairs. This article, therefore, adopts a different perspective on this problem by introducing a new locator repair solution that leverages tree matching algorithms to relocate broken locators. This solution, named Erratum, implements a holistic approach to reduce the element search space, which greatly eases the locator repair task and drastically improves repair accuracy. We compare the robustness of Erratum on a large-scale benchmark composed of realistic and synthetic mutations applied to popular web applications currently deployed in production. Our empirical results demonstrate that Erratum outperforms the accuracy of WATER, a state-of-the-art solution, by 67%.
Towards Deep Industrial Transfer Learning for Anomaly Detection on Time Series Data
Venue: arXiv cs.LG 2021
Year: 2021

Deep learning promises performant anomaly detection on time-variant datasets, but greatly suffers from low availability of suitable training datasets and frequently changing tasks. Deep transfer learning offers mitigation by letting algorithms built upon previous knowledge from different tasks or locations. In this article, a modular deep learning algorithm for anomaly detection on time series datasets is presented that allows for an easy integration of such transfer learning capabilities. It is thoroughly tested on a dataset from a discrete manufacturing process in order to prove its fundamental adequacy towards deep industrial transfer learning - the transfer of knowledge in industrial applications' special environment.
Unifying Behavioral and Response Diversity for Open-ended Learning in Zero-sum Games
Venue: arXiv cs.MA 2021
Year: 2021

Measuring and promoting policy diversity is critical for solving games with strong non-transitive dynamics where strategic cycles exist, and there is no consistent winner (e.g., Rock-Paper-Scissors). With that in mind, maintaining a pool of diverse policies via open-ended learning is an attractive solution, which can generate auto-curricula to avoid being exploited. However, in conventional open-ended learning algorithms, there are no widely accepted definitions for diversity, making it hard to construct and evaluate the diverse policies. In this work, we summarize previous concepts of diversity and work towards offering a unified measure of diversity in multi-agent open-ended learning to include all elements in Markov games, based on both Behavioral Diversity (BD) and Response Diversity (RD). At the trajectory distribution level, we re-define BD in the state-action space as the discrepancies of occupancy measures. For the reward dynamics, we propose RD to characterize diversity through the responses of policies when encountering different opponents. We also show that many current diversity measures fall in one of the categories of BD or RD but not both. With this unified diversity measure, we design the corresponding diversity-promoting objective and population effectivity when seeking the best responses in open-ended learning. We validate our methods in both relatively simple games like matrix game, non-transitive mixture model, and the complex \textit{Google Research Football} environment. The population found by our methods reveals the lowest exploitability, highest population effectivity in matrix game and non-transitive mixture model, as well as the largest goal difference when interacting with opponents of various levels in \textit{Google Research Football}.
Auto-tagging of Short Conversational Sentences using Natural Language Processing Methods
Venue: arXiv cs.CL 2021
Year: 2021

In this study, we aim to find a method to auto-tag sentences specific to a domain. Our training data comprises short conversational sentences extracted from chat conversations between company's customer representatives and web site visitors. We manually tagged approximately 14 thousand visitor inputs into ten basic categories, which will later be used in a transformer-based language model with attention mechanisms for the ultimate goal of developing a chatbot application that can produce meaningful dialogue. We considered three different state-of-the-art models and reported their auto-tagging capabilities. We achieved the best performance with the bidirectional encoder representation from transformers (BERT) model. Implementation of the models used in these experiments can be cloned from our GitHub repository and tested for similar auto-tagging problems without much effort.
Information Avoidance and Overvaluation in Sequential Decision Making under Epistemic Constraints
Venue: arXiv cs.AI 2021
Year: 2021

Decision makers involved in the management of civil assets and systems usually take actions under constraints imposed by societal regulations. Some of these constraints are related to epistemic quantities, as the probability of failure events and the corresponding risks. Sensors and inspectors can provide useful information supporting the control process (e.g. the maintenance process of an asset), and decisions about collecting this information should rely on an analysis of its cost and value. When societal regulations encode an economic perspective that is not aligned with that of the decision makers, the Value of Information (VoI) can be negative (i.e., information sometimes hurts), and almost irrelevant information can even have a significant value (either positive or negative), for agents acting under these epistemic constraints. We refer to these phenomena as Information Avoidance (IA) and Information OverValuation (IOV). In this paper, we illustrate how to assess VoI in sequential decision making under epistemic constraints (as those imposed by societal regulations), by modeling a Partially Observable Markov Decision Processes (POMDP) and evaluating non optimal policies via Finite State Controllers (FSCs). We focus on the value of collecting information at current time, and on that of collecting sequential information, we illustrate how these values are related and we discuss how IA and IOV can occur in those settings.
Salient Positions based Attention Network for Image Classification
Venue: arXiv cs.CV 2021
Year: 2021

The self-attention mechanism has attracted wide publicity for its most important advantage of modeling long dependency, and its variations in computer vision tasks, the non-local block tries to model the global dependency of the input feature maps. Gathering global contextual information will inevitably need a tremendous amount of memory and computing resources, which has been extensively studied in the past several years. However, there is a further problem with the self-attention scheme: is all information gathered from the global scope helpful for the contextual modelling? To our knowledge, few studies have focused on the problem. Aimed at both questions this paper proposes the salient positions-based attention scheme SPANet, which is inspired by some interesting observations on the attention maps and affinity matrices generated in self-attention scheme. We believe these observations are beneficial for better understanding of the self-attention. SPANet uses the salient positions selection algorithm to select only a limited amount of salient points to attend in the attention map computing. This approach will not only spare a lot of memory and computing resources, but also try to distill the positive information from the transformation of the input feature maps. In the implementation, considering the feature maps with channel high dimensions, which are completely different from the general visual image, we take the squared power of the feature maps along the channel dimension as the saliency metric of the positions. In general, different from the non-local block method, SPANet models the contextual information using only the selected positions instead of all, along the channel dimension instead of space dimension. Our source code is available at https://github.com/likyoo/SPANet.
Dual-Modality Vehicle Anomaly Detection via Bilateral Trajectory Tracing
Venue: arXiv cs.CV 2021
Year: 2021

Traffic anomaly detection has played a crucial role in Intelligent Transportation System (ITS). The main challenges of this task lie in the highly diversified anomaly scenes and variational lighting conditions. Although much work has managed to identify the anomaly in homogenous weather and scene, few resolved to cope with complex ones. In this paper, we proposed a dual-modality modularized methodology for the robust detection of abnormal vehicles. We introduced an integrated anomaly detection framework comprising the following modules: background modeling, vehicle tracking with detection, mask construction, Region of Interest (ROI) backtracking, and dual-modality tracing. Concretely, we employed background modeling to filter the motion information and left the static information for later vehicle detection. For the vehicle detection and tracking module, we adopted YOLOv5 and multi-scale tracking to localize the anomalies. Besides, we utilized the frame difference and tracking results to identify the road and obtain the mask. In addition, we introduced multiple similarity estimation metrics to refine the anomaly period via backtracking. Finally, we proposed a dual-modality bilateral tracing module to refine the time further. The experiments conducted on the Track 4 testset of the NVIDIA 2021 AI City Challenge yielded a result of 0.9302 F1-Score and 3.4039 root mean square error (RMSE), indicating the effectiveness of our framework.
Fuzzy propositional configuration logics
Venue: arXiv cs.LO 2021
Year: 2021

We introduce and investigate a weighted propositional configuration logic over De Morgan algebras. This logic is able to describe software architectures with quantitative features such as the uncertainty of the interactions that occur in the architecture. We deal with the equivalence problem of formulas in our logic by proving that every formula can be written in a specific form. To our surprise, there are formulas which are equivalent only over specific De Morgan algebras. We provide examples of formulas in our logic which describe well-known software architectures equipped with quantitative features.
On the Cover and Pombra Gaussian Feedback Capacity: Complete Sequential Characterizations via a Sufficient Statistic
Venue: arXiv cs.IT 2021
Year: 2021

The main objective of this paper is to derive a new sequential characterization of the Cover and Pombra \cite{cover-pombra1989} characterization of the $n-$finite block or transmission feedback information ($n$-FTFI) capacity, which clarifies several issues of confusion and incorrect interpretation of results in literature. The optimal channel input processes of the new equivalent sequential characterizations are expressed as functionals of a sufficient statistic and a Gaussian orthogonal innovations process. From the new representations follows that the Cover and Pombra characterization of the $n-$FTFI capacity is expressed as a functional of two generalized matrix difference Riccati equations (DRE) of filtering theory of Gaussian systems. This contradicts results which are redundant in the literature, and illustrates the fundamental complexity of the feedback capacity formula.
A Communication Layer for Integrated Sensors and Robotic ecology Solutions to Ambient Intelligence
Venue: arXiv cs.RO 2021
Year: 2021

This paper presents a communication framework built to simplify the construction of robotic ecologies, i.e., networks of heterogeneous computational nodes interfaced with sensors, actuators, and mobile robots. Building integrated ambient intelligence (AmI) solutions out of such a wide range of heterogeneous devices is a key requirement for a range of application domains, such as home automation, logistic, security and Ambient Assisted Living (AAL). This goal is challenging since these ecologies need to adapt to changing environments and especially when they include tiny embedded devices with limited computational resources. We discuss a number of requirements characterizing this type of systems and illustrate how they have been addressed in the design of the new communication framework. The most distinguishing aspect of our frameworks is the transparency with which the same communication features are offered across heterogeneous programming languages and operating systems under a consistent API. Finally, we illustrate how the framework has been used to bind together and to support the operations of all the components of adaptive robotic ecologies in two real-world test-beds.