OpenRaiser/Intern-Atlas huggingface dataset:
Syntactic Perturbations Reveal Representational Correlates of Hierarchical Phrase Structure in Pretrained Language Models
Venue: arXiv cs.CL 2021
Year: 2021

While vector-based language representations from pretrained language models have set a new standard for many NLP tasks, there is not yet a complete accounting of their inner workings. In particular, it is not entirely clear what aspects of sentence-level syntax are captured by these representations, nor how (if at all) they are built along the stacked layers of the network. In this paper, we aim to address such questions with a general class of interventional, input perturbation-based analyses of representations from pretrained language models. Importing from computational and cognitive neuroscience the notion of representational invariance, we perform a series of probes designed to test the sensitivity of these representations to several kinds of structure in sentences. Each probe involves swapping words in a sentence and comparing the representations from perturbed sentences against the original. We experiment with three different perturbations: (1) random permutations of n-grams of varying width, to test the scale at which a representation is sensitive to word position; (2) swapping of two spans which do or do not form a syntactic phrase, to test sensitivity to global phrase structure; and (3) swapping of two adjacent words which do or do not break apart a syntactic phrase, to test sensitivity to local phrase structure. Results from these probes collectively suggest that Transformers build sensitivity to larger parts of the sentence along their layers, and that hierarchical phrase structure plays a role in this process. More broadly, our results also indicate that structured input perturbations widens the scope of analyses that can be performed on often-opaque deep learning systems, and can serve as a complement to existing tools (such as supervised linear probes) for interpreting complex black-box models.
SISA: Set-Centric Instruction Set Architecture for Graph Mining on Processing-in-Memory Systems
Venue: arXiv cs.AR 2021
Year: 2021

Simple graph algorithms such as PageRank have been the target of numerous hardware accelerators. Yet, there also exist much more complex graph mining algorithms for problems such as clustering or maximal clique listing. These algorithms are memory-bound and thus could be accelerated by hardware techniques such as Processing-in-Memory (PIM). However, they also come with nonstraightforward parallelism and complicated memory access patterns. In this work, we address this problem with a simple yet surprisingly powerful observation: operations on sets of vertices, such as intersection or union, form a large part of many complex graph mining algorithms, and can offer rich and simple parallelism at multiple levels. This observation drives our cross-layer design, in which we (1) expose set operations using a novel programming paradigm, (2) express and execute these operations efficiently with carefully designed set-centric ISA extensions called SISA, and (3) use PIM to accelerate SISA instructions. The key design idea is to alleviate the bandwidth needs of SISA instructions by mapping set operations to two types of PIM: in-DRAM bulk bitwise computing for bitvectors representing high-degree vertices, and near-memory logic layers for integer arrays representing low-degree vertices. Set-centric SISA-enhanced algorithms are efficient and outperform hand-tuned baselines, offering more than 10x speedup over the established Bron-Kerbosch algorithm for listing maximal cliques. We deliver more than 10 SISA set-centric algorithm formulations, illustrating SISA's wide applicability.
SummVis: Interactive Visual Analysis of Models, Data, and Evaluation for Text Summarization
Venue: arXiv cs.CL 2021
Year: 2021

Novel neural architectures, training strategies, and the availability of large-scale corpora haven been the driving force behind recent progress in abstractive text summarization. However, due to the black-box nature of neural models, uninformative evaluation metrics, and scarce tooling for model and data analysis, the true performance and failure modes of summarization models remain largely unknown. To address this limitation, we introduce SummVis, an open-source tool for visualizing abstractive summaries that enables fine-grained analysis of the models, data, and evaluation metrics associated with text summarization. Through its lexical and semantic visualizations, the tools offers an easy entry point for in-depth model prediction exploration across important dimensions such as factual consistency or abstractiveness. The tool together with several pre-computed model outputs is available at https://github.com/robustness-gym/summvis.
Camera View Adjustment Prediction for Improving Image Composition
Venue: arXiv cs.CV 2021
Year: 2021

Image composition plays an important role in the quality of a photo. However, not every camera user possesses the knowledge and expertise required for capturing well-composed photos. While post-capture cropping can improve the composition sometimes, it does not work in many common scenarios in which the photographer needs to adjust the camera view to capture the best shot. To address this issue, we propose a deep learning-based approach that provides suggestions to the photographer on how to adjust the camera view before capturing. By optimizing the composition before a photo is captured, our system helps photographers to capture better photos. As there is no publicly-available dataset for this task, we create a view adjustment dataset by repurposing existing image cropping datasets. Furthermore, we propose a two-stage semi-supervised approach that utilizes both labeled and unlabeled images for training a view adjustment model. Experiment results show that the proposed semi-supervised approach outperforms the corresponding supervised alternatives, and our user study results show that the suggested view adjustment improves image composition 79% of the time.
Approximation algorithms for 1-Wasserstein distance between persistence diagrams
Venue: arXiv cs.CG 2021
Year: 2021

Recent years have witnessed a tremendous growth using topological summaries, especially the persistence diagrams (encoding the so-called persistent homology) for analyzing complex shapes. Intuitively, persistent homology maps a potentially complex input object (be it a graph, an image, or a point set and so on) to a unified type of feature summary, called the persistence diagrams. One can then carry out downstream data analysis tasks using such persistence diagram representations. A key problem is to compute the distance between two persistence diagrams efficiently. In particular, a persistence diagram is essentially a multiset of points in the plane, and one popular distance is the so-called 1-Wasserstein distance between persistence diagrams. In this paper, we present two algorithms to approximate the 1-Wasserstein distance for persistence diagrams in near-linear time. These algorithms primarily follow the same ideas as two existing algorithms to approximate optimal transport between two finite point-sets in Euclidean spaces via randomly shifted quadtrees. We show how these algorithms can be effectively adapted for the case of persistence diagrams. Our algorithms are much more efficient than previous exact and approximate algorithms, both in theory and in practice, and we demonstrate its efficiency via extensive experiments. They are conceptually simple and easy to implement, and the code is publicly available in github.
Automated Evolution of Feature Logging Statement Levels Using Git Histories and Degree of Interest
Venue: arXiv cs.SE 2021
Year: 2021

Logging -- used for system events and security breaches to describe more informational yet essential aspects of software features -- is pervasive. Given the high transactionality of today's software, logging effectiveness can be reduced by information overload. Log levels help alleviate this problem by correlating a priority to logs that can be later filtered. As software evolves, however, levels of logs documenting surrounding feature implementations may also require modification as features once deemed important may have decreased in urgency and vice-versa. We present an automated approach that assists developers in evolving levels of such (feature) logs. The approach, based on mining Git histories and manipulating a degree of interest (DOI) model, transforms source code to revitalize feature log levels based on the "interestingness" of the surrounding code. Built upon JGit and Mylyn, the approach is implemented as an Eclipse IDE plug-in and evaluated on 18 Java projects with $\sim$3 million lines of code and $\sim$4K log statements. Our tool successfully analyzes 99.22% of logging statements, increases log level distributions by $\sim$20%, and increases the focus of logs in bug fix contexts $\sim$83% of the time. Moreover, pull (patch) requests were integrated into large and popular open-source projects. The results indicate that the approach is promising in assisting developers in evolving feature log levels.
Demystifying Memory Access Patterns of FPGA-Based Graph Processing Accelerators
Venue: arXiv cs.AR 2021
Year: 2021

Recent advances in reprogrammable hardware (e.g., FPGAs) and memory technology (e.g., DDR4, HBM) promise to solve performance problems inherent to graph processing like irregular memory access patterns on traditional hardware (e.g., CPU). While several of these graph accelerators were proposed in recent years, it remains difficult to assess their performance and compare them on common graph workloads and accelerator platforms, due to few open source implementations and excessive implementation effort. In this work, we build on a simulation environment for graph processing accelerators, to make several existing accelerator approaches comparable. This allows us to study relevant performance dimensions such as partitioning schemes and memory technology, among others. The evaluation yields insights into the strengths and weaknesses of current graph processing accelerators along these dimensions, and features a novel in-depth comparison.
A survey of active learning algorithms for supervised remote sensing image classification
Venue: arXiv cs.CV 2021
Year: 2021

Defining an efficient training set is one of the most delicate phases for the success of remote sensing image classification routines. The complexity of the problem, the limited temporal and financial resources, as well as the high intraclass variance can make an algorithm fail if it is trained with a suboptimal dataset. Active learning aims at building efficient training sets by iteratively improving the model performance through sampling. A user-defined heuristic ranks the unlabeled pixels according to a function of the uncertainty of their class membership and then the user is asked to provide labels for the most uncertain pixels. This paper reviews and tests the main families of active learning algorithms: committee, large margin and posterior probability-based. For each of them, the most recent advances in the remote sensing community are discussed and some heuristics are detailed and tested. Several challenging remote sensing scenarios are considered, including very high spatial resolution and hyperspectral image classification. Finally, guidelines for choosing the good architecture are provided for new and/or unexperienced user.
Learning User's confidence for active learning
Venue: arXiv cs.CV 2021
Year: 2021

In this paper, we study the applicability of active learning in operative scenarios: more particularly, we consider the well-known contradiction between the active learning heuristics, which rank the pixels according to their uncertainty, and the user's confidence in labeling, which is related to both the homogeneity of the pixel context and user's knowledge of the scene. We propose a filtering scheme based on a classifier that learns the confidence of the user in labeling, thus minimizing the queries where the user would not be able to provide a class for the pixel. The capacity of a model to learn the user's confidence is studied in detail, also showing the effect of resolution is such a learning task. Experiments on two QuickBird images of different resolutions (with and without pansharpening) and considering committees of users prove the efficiency of the filtering scheme proposed, which maximizes the number of useful queries with respect to traditional active learning.
A Novel Hybrid Deep Learning Approach for Non-Intrusive Load Monitoring of Residential Appliance Based on Long Short Term Memory and Convolutional Neural Networks
Venue: arXiv cs.LG 2021
Year: 2021

Energy disaggregation or nonintrusive load monitoring (NILM), is a single-input blind source discrimination problem, aims to interpret the mains user electricity consumption into appliance level measurement. This article presents a new approach for power disaggregation by using a deep recurrent long short term memory (LSTM) network combined with convolutional neural networks (CNN). Deep neural networks have been shown to be a significant way for these types of problems because of their complexity and huge number of trainable paramters. Hybrid method that proposed in the article could significantly increase the overall accuracy of NILM because it benefits from both network advantages. The proposed method used sequence-to-sequence learning, where the input is a window of the mains and the output is a window of the target appliance. The proposed deep neural network approach has been applied to real-world household energy dataset "REFIT". The REFIT electrical load measurements dataset described in this paper includes whole house aggregate loads and nine individual appliance measurements at 8-second intervals per house, collected continuously over a period of two years from 20 houses around the UK. The proposed method achieve significant performance, improving accuracy and F1-score measures by 95.93% and 80.93% ,respectively which demonstrates the effectiveness and superiority of the proposed approach for home energy monitoring. Comparison of proposed method and other recently published method has been presented and discussed based on accuracy, number of considered appliances and size of the deep neural network trainable parameters. The proposed method shows remarkable performance compare to other previous methods.
Exploring software developers' work practices: Task differences, participation, engagement, and speed of task resolution
Venue: arXiv cs.SE 2021
Year: 2021

In seeking to understand the processes enacted during software development, an increasing number of studies have mined software repositories. In particular, studies have endeavored to show how teams resolve software defects. Although much of this work has been useful, we contend that large-scale examinations across the range of activities that are commonly performed, beyond defect-related issues alone, would help us to more fully understand the reasons why defects occur as well as their consequences. More generally, these explorations would reveal how team processes occur during all software development efforts. We thus extend such studies by investigating how software practitioners work while undertaking the range of software tasks that are typically performed. Multiple forms of analyses of a longitudinal case study reveal that software practitioners were mostly involved in fixing defects, and that their engagement covaried depending on the nature of the work they were performing. Furthermore, multiple external factors affected speed of task resolution. Our outcomes suggest that behavioral and intrinsic issues may interact with extrinsic factors becoming significant predictors of the speed of software task resolution.
Matching-oriented Product Quantization For Ad-hoc Retrieval
Venue: arXiv cs.CL 2021
Year: 2021

Product quantization (PQ) is a widely used technique for ad-hoc retrieval. Recent studies propose supervised PQ, where the embedding and quantization models can be jointly trained with supervised learning. However, there is a lack of appropriate formulation of the joint training objective; thus, the improvements over previous non-supervised baselines are limited in reality. In this work, we propose the Matching-oriented Product Quantization (MoPQ), where a novel objective Multinoulli Contrastive Loss (MCL) is formulated. With the minimization of MCL, we are able to maximize the matching probability of query and ground-truth key, which contributes to the optimal retrieval accuracy. Given that the exact computation of MCL is intractable due to the demand of vast contrastive samples, we further propose the Differentiable Cross-device Sampling (DCS), which significantly augments the contrastive samples for precise approximation of MCL. We conduct extensive experimental studies on four real-world datasets, whose results verify the effectiveness of MoPQ. The code is available at https://github.com/microsoft/MoPQ.
SGL: Spectral Graph Learning from Measurements
Venue: arXiv cs.LG 2021
Year: 2021

This work introduces a highly scalable spectral graph densification framework for learning resistor networks with linear measurements, such as node voltages and currents. We prove that given $O(\log N)$ pairs of voltage and current measurements, it is possible to recover ultra-sparse $N$-node resistor networks which can well preserve the effective resistance distances on the graph. Also, the learned graphs preserve the structural (spectral) properties of the original graph, which can potentially be leveraged in many circuit design and optimization tasks. We show that the proposed graph learning approach is equivalent to solving the classical graphical Lasso problems with Laplacian-like precision matrices. Through extensive experiments for a variety of real-world test cases, we show that the proposed approach is highly scalable for learning ultra-sparse resistor networks without sacrificing solution quality.
Histopathology WSI Encoding based on GCNs for Scalable and Efficient Retrieval of Diagnostically Relevant Regions
Venue: arXiv cs.CV 2021
Year: 2021

Content-based histopathological image retrieval (CBHIR) has become popular in recent years in the domain of histopathological image analysis. CBHIR systems provide auxiliary diagnosis information for pathologists by searching for and returning regions that are contently similar to the region of interest (ROI) from a pre-established database. While, it is challenging and yet significant in clinical applications to retrieve diagnostically relevant regions from a database that consists of histopathological whole slide images (WSIs) for a query ROI. In this paper, we propose a novel framework for regions retrieval from WSI-database based on hierarchical graph convolutional networks (GCNs) and Hash technique. Compared to the present CBHIR framework, the structural information of WSI is preserved through graph embedding of GCNs, which makes the retrieval framework more sensitive to regions that are similar in tissue distribution. Moreover, benefited from the hierarchical GCN structures, the proposed framework has good scalability for both the size and shape variation of ROIs. It allows the pathologist defining query regions using free curves according to the appearance of tissue. Thirdly, the retrieval is achieved based on Hash technique, which ensures the framework is efficient and thereby adequate for practical large-scale WSI-database. The proposed method was validated on two public datasets for histopathological WSI analysis and compared to the state-of-the-art methods. The proposed method achieved mean average precision above 0.857 on the ACDC-LungHP dataset and above 0.864 on the Camelyon16 dataset in the irregular region retrieval tasks, which are superior to the state-of-the-art methods. The average retrieval time from a database within 120 WSIs is 0.802 ms.
Probing Across Time: What Does RoBERTa Know and When?
Venue: arXiv cs.CL 2021
Year: 2021

Models of language trained on very large corpora have been demonstrated useful for NLP. As fixed artifacts, they have become the object of intense study, with many researchers "probing" the extent to which linguistic abstractions, factual and commonsense knowledge, and reasoning abilities they acquire and readily demonstrate. Building on this line of work, we consider a new question: for types of knowledge a language model learns, when during (pre)training are they acquired? We plot probing performance across iterations, using RoBERTa as a case study. Among our findings: linguistic knowledge is acquired fast, stably, and robustly across domains. Facts and commonsense are slower and more domain-sensitive. Reasoning abilities are, in general, not stably acquired. As new datasets, pretraining protocols, and probes emerge, we believe that probing-across-time analyses can help researchers understand the complex, intermingled learning that these models undergo and guide us toward more efficient approaches that accomplish necessary learning faster.
Coverage Probability and Energy Efficiency of Reconfigurable Intelligent Surface-Assisted mmWave Networks
Venue: arXiv cs.IT 2021
Year: 2021

Reconfigurable intelligent surface (RIS) emerges as a promising technology for the next generation networks. In this paper, we utilize the tools from stochastic geometry to study the performance of a RIS-assisted millimeter wave (mmWave) cellular network. Specifically, the locations of the base stations (BSs) and the midpoints of the blockages are modeled as two independent Poisson point processes (PPP) where the blockages are modeled by line boolean model and a fraction of the blockages are coated with RISs. The distinguish characteristics of mmWave communications, i.e., the directional beamforming and different path loss laws for line-of-sight (LOS) and non-line-of-sight (NLOS), are incorporated into the analysis. We derive the expressions of the coverage probability and the area spectral efficiency. The coverage probability under the special case where the blockage parameter is sufficiently small is also derived. Numerical results demonstrate that better coverage performance and higher energy efficiency can be achieved by the large-scale deployment of RISs. In addition, the tradeoff between the BS and RIS densities is investigated and the results show that the RISs are excellent supplementary for the traditional networks to improve the coverage probability with limited power consumption.