OpenRaiser/Intern-Atlas huggingface dataset:
Automatic Standardization of Colloquial Persian
Venue: arXiv cs.CL 2020
Year: 2020

The Iranian Persian language has two varieties: standard and colloquial. Most natural language processing tools for Persian assume that the text is in standard form: this assumption is wrong in many real applications especially web content. This paper describes a simple and effective standardization approach based on sequence-to-sequence translation. We design an algorithm for generating artificial parallel colloquial-to-standard data for learning a sequence-to-sequence model. Moreover, we annotate a publicly available evaluation data consisting of 1912 sentences from a diverse set of domains. Our intrinsic evaluation shows a higher BLEU score of 62.8 versus 61.7 compared to an off-the-shelf rule-based standardization model in which the original text has a BLEU score of 46.4. We also show that our model improves English-to-Persian machine translation in scenarios for which the training data is from colloquial Persian with 1.4 absolute BLEU score difference in the development data, and 0.8 in the test data.
Self-Supervised Learning of Lidar Segmentation for Autonomous Indoor Navigation
Venue: arXiv cs.RO 2020
Year: 2020

We present a self-supervised learning approach for the semantic segmentation of lidar frames. Our method is used to train a deep point cloud segmentation architecture without any human annotation. The annotation process is automated with the combination of simultaneous localization and mapping (SLAM) and ray-tracing algorithms. By performing multiple navigation sessions in the same environment, we are able to identify permanent structures, such as walls, and disentangle short-term and long-term movable objects, such as people and tables, respectively. New sessions can then be performed using a network trained to predict these semantic labels. We demonstrate the ability of our approach to improve itself over time, from one session to the next. With semantically filtered point clouds, our robot can navigate through more complex scenarios, which, when added to the training pool, help to improve our network predictions. We provide insights into our network predictions and show that our approach can also improve the performances of common localization techniques.
Data-driven Method for Estimating Aircraft Mass from Quick Access Recorder using Aircraft Dynamics and Multilayer Perceptron Neural Network
Venue: arXiv cs.LG 2020
Year: 2020

Accurate aircraft-mass estimation is critical to airlines from the safety-management and performance-optimization viewpoints. Overloading an aircraft with passengers and baggage might result in a safety hazard. In contrast, not fully utilizing an aircraft's payload-carrying capacity undermines its operational efficiency and airline profitability. However, accurate determination of the aircraft mass for each operating flight is not feasible because it is impractical to weigh each aircraft component, including the payload. The existing methods for aircraft-mass estimation are dependent on the aircraft- and engine-performance parameters, which are usually considered proprietary information. Moreover, the values of these parameters vary under different operating conditions while those of others might be subject to large estimation errors. This paper presents a data-driven method involving use of the quick access recorder (QAR)-a digital flight-data recorder-installed on all aircrafts to record the initial aircraft climb mass during each flight. The method requires users to select appropriate parameters among several thousand others recorded by the QAR using physical models. The selected data are subsequently processed and provided as input to a multilayer perceptron neural network for building the model for initial-climb aircraft-mass prediction. Thus, the proposed method offers the advantages of both the model-based and data-driven approaches for aircraft-mass estimation. Because this method does not explicitly rely on any aircraft or engine parameter, it is universally applicable to all aircraft types. In this study, the proposed method was applied to a set of Boeing 777-300ER aircrafts, the results of which demonstrated reasonable accuracy. Airlines can use this tool to better utilize aircraft's payload.
Path Optimization Sheaves
Venue: arXiv cs.NI 2020
Year: 2020

Motivated by efforts to incorporate sheaves into networking, we seek to reinterpret pathfinding algorithms in terms of cellular sheaves, using Dijkstra's algorithm as an example. We construct sheaves on a graph with distinguished source and sink vertices, in which paths are represented by sections. The first sheaf is a very general construction that can be applied to other algorithms, while the second is created specifically to capture the decision making of Dijkstra's algorithm. In both cases, Dijkstra's algorithm can be described as a systematic process of extending local sections to global sections. We discuss the relationship between the two sheaves and summarize how other pathfinding algorithms can be interpreted in a similar way. While the sheaves presented here address paths and pathfinding algorithms, we suggest that future work could explore connections to other concepts from graph theory and other networking algorithms. This work was supported by the NASA Internship Project and SCaN Internship Project during the summer of 2020.
CommPOOL: An Interpretable Graph Pooling Framework for Hierarchical Graph Representation Learning
Venue: arXiv cs.LG 2020
Year: 2020

Recent years have witnessed the emergence and flourishing of hierarchical graph pooling neural networks (HGPNNs) which are effective graph representation learning approaches for graph level tasks such as graph classification. However, current HGPNNs do not take full advantage of the graph's intrinsic structures (e.g., community structure). Moreover, the pooling operations in existing HGPNNs are difficult to be interpreted. In this paper, we propose a new interpretable graph pooling framework - CommPOOL, that can capture and preserve the hierarchical community structure of graphs in the graph representation learning process. Specifically, the proposed community pooling mechanism in CommPOOL utilizes an unsupervised approach for capturing the inherent community structure of graphs in an interpretable manner. CommPOOL is a general and flexible framework for hierarchical graph representation learning that can further facilitate various graph-level tasks. Evaluations on five public benchmark datasets and one synthetic dataset demonstrate the superior performance of CommPOOL in graph representation learning for graph classification compared to the state-of-the-art baseline methods, and its effectiveness in capturing and preserving the community structure of graphs.
Overcoming Catastrophic Forgetting in Graph Neural Networks
Venue: arXiv cs.LG 2020
Year: 2020

Catastrophic forgetting refers to the tendency that a neural network "forgets" the previous learned knowledge upon learning new tasks. Prior methods have been focused on overcoming this problem on convolutional neural networks (CNNs), where the input samples like images lie in a grid domain, but have largely overlooked graph neural networks (GNNs) that handle non-grid data. In this paper, we propose a novel scheme dedicated to overcoming catastrophic forgetting problem and hence strengthen continual learning in GNNs. At the heart of our approach is a generic module, termed as topology-aware weight preserving~(TWP), applicable to arbitrary form of GNNs in a plug-and-play fashion. Unlike the main stream of CNN-based continual learning methods that rely on solely slowing down the updates of parameters important to the downstream task, TWP explicitly explores the local structures of the input graph, and attempts to stabilize the parameters playing pivotal roles in the topological aggregation. We evaluate TWP on different GNN backbones over several datasets, and demonstrate that it yields performances superior to the state of the art. Code is publicly available at \url{https://github.com/hhliu79/TWP}.
Uncertainty-Aware Deep Calibrated Salient Object Detection
Venue: arXiv cs.CV 2020
Year: 2020

Existing deep neural network based salient object detection (SOD) methods mainly focus on pursuing high network accuracy. However, those methods overlook the gap between network accuracy and prediction confidence, known as the confidence uncalibration problem. Thus, state-of-the-art SOD networks are prone to be overconfident. In other words, the predicted confidence of the networks does not reflect the real probability of correctness of salient object detection, which significantly hinder their real-world applicability. In this paper, we introduce an uncertaintyaware deep SOD network, and propose two strategies from different perspectives to prevent deep SOD networks from being overconfident. The first strategy, namely Boundary Distribution Smoothing (BDS), generates continuous labels by smoothing the original binary ground-truth with respect to pixel-wise uncertainty. The second strategy, namely Uncertainty-Aware Temperature Scaling (UATS), exploits a relaxed Sigmoid function during both training and testing with spatially-variant temperature scaling to produce softened output. Both strategies can be incorporated into existing deep SOD networks with minimal efforts. Moreover, we propose a new saliency evaluation metric, namely dense calibration measure C, to measure how the model is calibrated on a given dataset. Extensive experimental results on seven benchmark datasets demonstrate that our solutions can not only better calibrate SOD models, but also improve the network accuracy.
Provable Defense against Privacy Leakage in Federated Learning from Representation Perspective
Venue: arXiv cs.LG 2020
Year: 2020

Federated learning (FL) is a popular distributed learning framework that can reduce privacy risks by not explicitly sharing private data. However, recent works demonstrated that sharing model updates makes FL vulnerable to inference attacks. In this work, we show our key observation that the data representation leakage from gradients is the essential cause of privacy leakage in FL. We also provide an analysis of this observation to explain how the data presentation is leaked. Based on this observation, we propose a defense against model inversion attack in FL. The key idea of our defense is learning to perturb data representation such that the quality of the reconstructed data is severely degraded, while FL performance is maintained. In addition, we derive certified robustness guarantee to FL and convergence guarantee to FedAvg, after applying our defense. To evaluate our defense, we conduct experiments on MNIST and CIFAR10 for defending against the DLG attack and GS attack. Without sacrificing accuracy, the results demonstrate that our proposed defense can increase the mean squared error between the reconstructed data and the raw data by as much as more than 160X for both DLG attack and GS attack, compared with baseline defense methods. The privacy of the FL system is significantly improved.
Smooth Bandit Optimization: Generalization to Hölder Space
Venue: arXiv cs.LG 2020
Year: 2020

We consider bandit optimization of a smooth reward function, where the goal is cumulative regret minimization. This problem has been studied for $α$-Hölder continuous (including Lipschitz) functions with $0<α\leq 1$. Our main result is in generalization of the reward function to Hölder space with exponent $α>1$ to bridge the gap between Lipschitz bandits and infinitely-differentiable models such as linear bandits. For Hölder continuous functions, approaches based on random sampling in bins of a discretized domain suffices as optimal. In contrast, we propose a class of two-layer algorithms that deploy misspecified linear/polynomial bandit algorithms in bins. We demonstrate that the proposed algorithm can exploit higher-order smoothness of the function by deriving a regret upper bound of $\tilde{O}(T^\frac{d+α}{d+2α})$ for when $α>1$, which matches existing lower bound. We also study adaptation to unknown function smoothness over a continuous scale of Hölder spaces indexed by $α$, with a bandit model selection approach applied with our proposed two-layer algorithms. We show that it achieves regret rate that matches the existing lower bound for adaptation within the $α\leq 1$ subset.
A Dark Flash Normal Camera
Venue: arXiv cs.CV 2020
Year: 2020

Casual photography is often performed in uncontrolled lighting that can result in low quality images and degrade the performance of downstream processing. We consider the problem of estimating surface normal and reflectance maps of scenes depicting people despite these conditions by supplementing the available visible illumination with a single near infrared (NIR) light source and camera, a so-called "dark flash image". Our method takes as input a single color image captured under arbitrary visible lighting and a single dark flash image captured under controlled front-lit NIR lighting at the same viewpoint, and computes a normal map, a diffuse albedo map, and a specular intensity map of the scene. Since ground truth normal and reflectance maps of faces are difficult to capture, we propose a novel training technique that combines information from two readily available and complementary sources: a stereo depth signal and photometric shading cues. We evaluate our method over a range of subjects and lighting conditions and describe two applications: optimizing stereo geometry and filling the shadows in an image.
Document-aligned Japanese-English Conversation Parallel Corpus
Venue: arXiv cs.CL 2020
Year: 2020

Sentence-level (SL) machine translation (MT) has reached acceptable quality for many high-resourced languages, but not document-level (DL) MT, which is difficult to 1) train with little amount of DL data; and 2) evaluate, as the main methods and data sets focus on SL evaluation. To address the first issue, we present a document-aligned Japanese-English conversation corpus, including balanced, high-quality business conversation data for tuning and testing. As for the second issue, we manually identify the main areas where SL MT fails to produce adequate translations in lack of context. We then create an evaluation set where these phenomena are annotated to alleviate automatic evaluation of DL systems. We train MT models using our corpus to demonstrate how using context leads to improvements.
DevOps A Historical Review and Future Works
Venue: arXiv cs.SE 2020
Year: 2020

DevOps is an emerging practice to be followed in Software Development life cycle. The name DevOps indicates that its an integration of Development and Operations team. It is followed to integrate the various stages of the development lifecycle. DevOps is an extended version of the existing Agile method. DevOps aims at Continuous Integration, Continuous Delivery, Continuous Improvement, faster Feedback and Security. This paper reviews the building blocks of DevOps, challenges in adopting DevOps, models to improve DevOps practices and Future works on DevOps.
Improving Task-Agnostic BERT Distillation with Layer Mapping Search
Venue: arXiv cs.CL 2020
Year: 2020

Knowledge distillation (KD) which transfers the knowledge from a large teacher model to a small student model, has been widely used to compress the BERT model recently. Besides the supervision in the output in the original KD, recent works show that layer-level supervision is crucial to the performance of the student BERT model. However, previous works designed the layer mapping strategy heuristically (e.g., uniform or last-layer), which can lead to inferior performance. In this paper, we propose to use the genetic algorithm (GA) to search for the optimal layer mapping automatically. To accelerate the search process, we further propose a proxy setting where a small portion of the training corpus are sampled for distillation, and three representative tasks are chosen for evaluation. After obtaining the optimal layer mapping, we perform the task-agnostic BERT distillation with it on the whole corpus to build a compact student model, which can be directly fine-tuned on downstream tasks. Comprehensive experiments on the evaluation benchmarks demonstrate that 1) layer mapping strategy has a significant effect on task-agnostic BERT distillation and different layer mappings can result in quite different performances; 2) the optimal layer mapping strategy from the proposed search process consistently outperforms the other heuristic ones; 3) with the optimal layer mapping, our student model achieves state-of-the-art performance on the GLUE tasks.
Soft Compression for Lossless Image Coding
Venue: arXiv cs.IT 2020
Year: 2020

Soft compression is a lossless image compression method, which is committed to eliminating coding redundancy and spatial redundancy at the same time by adopting locations and shapes of codebook to encode an image from the perspective of information theory and statistical distribution. In this paper, we propose a new concept, compressible indicator function with regard to image, which gives a threshold about the average number of bits required to represent a location and can be used for revealing the performance of soft compression. We investigate and analyze soft compression for binary image, gray image and multi-component image by using specific algorithms and compressible indicator value. It is expected that the bandwidth and storage space needed when transmitting and storing the same kind of images can be greatly reduced by applying soft compression.
Trash Talk: Accelerating Garbage Collection on Integrated GPUs is Worthless
Venue: arXiv cs.DC 2020
Year: 2020

Systems integrating heterogeneous processors with unified memory provide seamless integration among these processors with minimal development complexity. These systems integrate accelerators such as GPUs on the same die with CPU cores to accommodate running parallel applications with varying levels of parallelism. Such integration is becoming very common on modern chip architectures, and it places a burden (or opportunity) on application and system programmers to utilize the full potential of such integrated chips. In this paper we evaluate whether we can obtain any performance benefits from running garbage collection on integrated GPU systems, and discuss how difficult it would be to realize these gains for the programmer. Proliferation of garbage-collected languages running on a variety of platforms from handheld mobile devices to data centers makes garbage collection an interesting target to examine on such platforms and can offer valuable lessons for other applications. We present our analysis of running garbage collection on integrated systems and find that the current state of these systems does not provide an advantage for accelerating such a task. We build a framework that allows us to offload garbage collection tasks on integrated GPU systems from within the JVM. We identify dominant phases of garbage collection and study the viability of offloading them to the integrated GPU. We show that performance advantages are limited, partly because an integrated GPU has limited advantage in memory bandwidth over the CPU, and partly because of costly atomic operations.
Market2Dish: Health-aware Food Recommendation
Venue: arXiv cs.IR 2020
Year: 2020

With the rising incidence of some diseases, such as obesity and diabetes, a healthy diet is arousing increasing attention. However, most existing food-related research efforts focus on recipe retrieval, user preference-based food recommendation, cooking assistance, or the nutrition and calorie estimation of dishes, ignoring the personalized health-aware food recommendation. Therefore, in this work, we present a personalized health-aware food recommendation scheme, namely Market2Dish, mapping the ingredients displayed in the market to the healthy dishes eaten at home. The proposed scheme comprises three components, namely recipe retrieval, user-health profiling, and health-aware food recommendation. In particular, recipe retrieval aims to acquire the ingredients available to the users, and then retrieve recipe candidates from a large-scale recipe dataset. User health profiling is to characterize the health conditions of users by capturing the textual health-related information crawled from social networks. Specifically, to solve the issue that the health-related information is extremely sparse, we incorporate a word-class interaction mechanism into the proposed deep model to learn the fine-grained correlations between the textual tweets and pre-defined health concepts. For the health-aware food recommendation, we present a novel category-aware hierarchical memory network-based recommender to learn the health-aware user-recipe interactions for better food recommendation. Moreover, extensive experiments demonstrate the effectiveness of the health-aware food recommendation scheme.
A New Neural Network Architecture Invariant to the Action of Symmetry Subgroups
Venue: arXiv cs.LG 2020
Year: 2020

We propose a computationally efficient $G$-invariant neural network that approximates functions invariant to the action of a given permutation subgroup $G \leq S_n$ of the symmetric group on input data. The key element of the proposed network architecture is a new $G$-invariant transformation module, which produces a $G$-invariant latent representation of the input data. Theoretical considerations are supported by numerical experiments, which demonstrate the effectiveness and strong generalization properties of the proposed method in comparison to other $G$-invariant neural networks.
Orthogonal Language and Task Adapters in Zero-Shot Cross-Lingual Transfer
Venue: arXiv cs.CL 2020
Year: 2020

Adapter modules, additional trainable parameters that enable efficient fine-tuning of pretrained transformers, have recently been used for language specialization of multilingual transformers, improving downstream zero-shot cross-lingual transfer. In this work, we propose orthogonal language and task adapters (dubbed orthoadapters) for cross-lingual transfer. They are trained to encode language- and task-specific information that is complementary (i.e., orthogonal) to the knowledge already stored in the pretrained transformer's parameters. Our zero-shot cross-lingual transfer experiments, involving three tasks (POS-tagging, NER, NLI) and a set of 10 diverse languages, 1) point to the usefulness of orthoadapters in cross-lingual transfer, especially for the most complex NLI task, but also 2) indicate that the optimal adapter configuration highly depends on the task and the target language. We hope that our work will motivate a wider investigation of usefulness of orthogonality constraints in language- and task-specific fine-tuning of pretrained transformers.