OpenRaiser/Intern-Atlas huggingface dataset:
DeepQAMVS: Query-Aware Hierarchical Pointer Networks for Multi-Video Summarization
Venue: arXiv cs.CV 2021
Year: 2021

The recent growth of web video sharing platforms has increased the demand for systems that can efficiently browse, retrieve and summarize video content. Query-aware multi-video summarization is a promising technique that caters to this demand. In this work, we introduce a novel Query-Aware Hierarchical Pointer Network for Multi-Video Summarization, termed DeepQAMVS, that jointly optimizes multiple criteria: (1) conciseness, (2) representativeness of important query-relevant events and (3) chronological soundness. We design a hierarchical attention model that factorizes over three distributions, each collecting evidence from a different modality, followed by a pointer network that selects frames to include in the summary. DeepQAMVS is trained with reinforcement learning, incorporating rewards that capture representativeness, diversity, query-adaptability and temporal coherence. We achieve state-of-the-art results on the MVS1K dataset, with inference time scaling linearly with the number of input video frames.
A Unified Metamodel for NoSQL and Relational Databases
Venue: arXiv cs.DB 2021
Year: 2021

The Database field is undergoing significant changes. Although relational systems are still predominant, the interest in NoSQL systems is continuously increasing. In this scenario, polyglot persistence is envisioned as the database architecture to be prevalent in the future. Multi-model database tools normally use a generic or unified metamodel to represent schemas of the data model that they support. Such metamodels facilitate developing utilities, as they can be built on a common representation. Also, the number of mappings required to migrate databases from a data model to another is reduced, and integrability is favored. In this paper, we present the U-Schema unified metamodel able to represent logical schemas for the four most popular NoSQL paradigms (columnar, document, key-value, and graph) as well as relational schemas. We will formally define the mappings between U-Schema and the data model defined for each paradigm. How these mappings have been implemented and validated will be discussed, and some applications of U-Schema will be shown. To achieve flexibility to respond to data changes, most of NoSQL systems are "schema-on-write," and the declaration of schemas is not required. Such an absence of schema declaration makes structural variability possible, i.e., stored data of the same entity type can have different structure. Moreover, data relationships supported by each data model are different. We will show how all these issues have been tackled in our approach. Our metamodel goes beyond the existing proposals by distinguishing entity types and relationship types, representing aggregation and reference relationships, and including the notion of structural variability. Our contributions also include developing schema extraction strategies for schemaless systems of each NoSQL data model, and tackling performance and scalability in the implementation for each store.
Reliable and Secure Short-Packet Communications
Venue: arXiv cs.IT 2021
Year: 2021

Exploiting short packets for communications is one of the key technologies for realizing emerging application scenarios such as massive machine type communications (mMTC) and ultra-reliable low-latency communications (uRLLC). In this paper, we investigate short-packet communications to provide both reliability and security guarantees simultaneously with an eavesdropper. In particular, an outage probability considering both reliability and secrecy is defined according to the characteristics of short-packet transmission, while the effective throughput in the sense of outage is established as the performance metric. Specifically, a general analytical framework is proposed to approximate the outage probability and effective throughput. Furthermore, closed-form expressions for these quantities are derived for the high signal-to-noise ratio (SNR) regime. Both effective throughput obtained via a general analytical framework and a high-SNR approximation are maximized under an outage-probability constraint by searching for the optimal blocklength. Numerical results verify the feasibility and accuracy of the proposed analytical framework, and illustrate the influence of the main system parameters on the blocklength and system performance under the outage-probability constraint.
Counterexample-Guided Repair for Symbolic-Geometric Action Abstractions
Venue: arXiv cs.RO 2021
Year: 2021

Integrated Task and Motion Planning (TMP) provides a promising class of approaches for solving robot planning problems with intricate symbolic and geometric constraints. However, the practical usefulness of TMP planners is limited by their need for symbolic abstractions of robot actions, which are difficult to construct even for experts. We propose an approach to automatically construct and continuously improve a symbolic abstraction of a robot action via observations of the robot performing the action. This approach, called automatic abstraction repair, allows symbolic abstractions to be initially incorrect or incomplete and converge toward a correct model over time. Abstraction repair uses constrained polynomial zonotopes (CPZs), an efficient non-convex set representation, to model predicates over joint symbolic and geometric state, and performs an optimizing search over symbolic edit operations to predicate formulae to improve the correspondence of a symbolic abstraction to the behavior of a physical robot controller. In this work, we describe the aforementioned predicate model, introduce the symbolic-geometric abstraction repair problem, and present an anytime algorithm for automatic abstraction repair. We then demonstrate that abstraction repair can improve realistic action abstractions for common mobile manipulation actions from a handful of observations.
Toward Real-time Analysis of Experimental Science Workloads on Geographically Distributed Supercomputers
Venue: arXiv cs.DC 2021
Year: 2021

Massive upgrades to science infrastructure are driving data velocities upwards while stimulating adoption of increasingly data-intensive analytics. While next-generation exascale supercomputers promise strong support for I/O-intensive workflows, HPC remains largely untapped by live experiments, because data transfers and disparate batch-queueing policies are prohibitive when faced with scarce instrument time. To bridge this divide, we introduce Balsam: a distributed orchestration platform enabling workflows at the edge to securely and efficiently trigger analytics tasks across a user-managed federation of HPC execution sites. We describe the architecture of the Balsam service, which provides a workflow management API, and distributed sites that provision resources and schedule scalable, fault-tolerant execution. We demonstrate Balsam in efficiently scaling real-time analytics from two DOE light sources simultaneously onto three supercomputers (Theta, Summit, and Cori), while maintaining low overheads for on-demand computing, and providing a Python library for seamless integration with existing ecosystems of data analysis tools.
Handwriting Recognition with Novelty
Venue: arXiv cs.CV 2021
Year: 2021

This paper introduces an agent-centric approach to handle novelty in the visual recognition domain of handwriting recognition (HWR). An ideal transcription agent would rival or surpass human perception, being able to recognize known and new characters in an image, and detect any stylistic changes that may occur within or across documents. A key confound is the presence of novelty, which has continued to stymie even the best machine learning-based algorithms for these tasks. In handwritten documents, novelty can be a change in writer, character attributes, writing attributes, or overall document appearance, among other things. Instead of looking at each aspect independently, we suggest that an integrated agent that can process known characters and novelties simultaneously is a better strategy. This paper formalizes the domain of handwriting recognition with novelty, describes a baseline agent, introduces an evaluation protocol with benchmark data, and provides experimentation to set the state-of-the-art. Results show feasibility for the agent-centric approach, but more work is needed to approach human-levels of reading ability, giving the HWR community a formal basis to build upon as they solve this challenging problem.
Meta Auxiliary Learning for Facial Action Unit Detection
Venue: arXiv cs.CV 2021
Year: 2021

Despite the success of deep neural networks on facial action unit (AU) detection, better performance depends on a large number of training images with accurate AU annotations. However, labeling AU is time-consuming, expensive, and error-prone. Considering AU detection and facial expression recognition (FER) are two highly correlated tasks, and facial expression (FE) is relatively easy to annotate, we consider learning AU detection and FER in a multi-task manner. However, the performance of the AU detection task cannot be always enhanced due to the negative transfer in the multi-task scenario. To alleviate this issue, we propose a Meta Auxiliary Learning method (MAL) that automatically selects highly related FE samples by learning adaptative weights for the training FE samples in a meta learning manner. The learned sample weights alleviate the negative transfer from two aspects: 1) balance the loss of each task automatically, and 2) suppress the weights of FE samples that have large uncertainties. Experimental results on several popular AU datasets demonstrate MAL consistently improves the AU detection performance compared with the state-of-the-art multi-task and auxiliary learning methods. MAL automatically estimates adaptive weights for the auxiliary FE samples according to their semantic relevance with the primary AU detection task.
RC2020 Report: Learning De-biased Representations with Biased Representations
Venue: arXiv cs.LG 2021
Year: 2021

As part of the ML Reproducibility Challenge 2020, we investigated the ICML 2020 paper "Learning De-biased Representations with Biased Representations" by Bahng et al., where the authors formalize and attempt to tackle the so called "cross bias generalization" problem with a new approach they introduce called ReBias. This report contains results of our attempts at reproducing the work in the application area of Image Recognition, specifically on the datasets biased MNIST and ImageNet. We compare ReBias with other methods - Vanilla, Biased, RUBi (as implemented by the authors), and conclude with a discussion concerning the validity of the claims made by the paper. We were able to reproduce results reported for the biased MNIST dataset to within 1% of the original values reported in the paper. Like the authors, we report results averaged over 3 runs. However, in a later section, we provide some additional results that appear to weaken the central claim of the paper with regards to the biased MNIST dataset. We were not able to reproduce results for ImageNet as in the original paper, but based on communication with the authors, provide a discussion as to the reasons for the same. This work attempts to be useful to other researchers aiming to use ReBias for their own research purposes, advising on certain possible pitfalls that may be encountered in the process.
Verification of Size Invariance in DNN Activations using Concept Embeddings
Venue: arXiv cs.CV 2021
Year: 2021

The benefits of deep neural networks (DNNs) have become of interest for safety critical applications like medical ones or automated driving. Here, however, quantitative insights into the DNN inner representations are mandatory. One approach to this is concept analysis, which aims to establish a mapping between the internal representation of a DNN and intuitive semantic concepts. Such can be sub-objects like human body parts that are valuable for validation of pedestrian detection. To our knowledge, concept analysis has not yet been applied to large object detectors, specifically not for sub-parts. Therefore, this work first suggests a substantially improved version of the Net2Vec approach (arXiv:1801.03454) for post-hoc segmentation of sub-objects. Its practical applicability is then demonstrated on a new concept dataset by two exemplary assessments of three standard networks, including the larger Mask R-CNN model (arXiv:1703.06870): (1) the consistency of body part similarity, and (2) the invariance of internal representations of body parts with respect to the size in pixels of the depicted person. The findings show that the representation of body parts is mostly size invariant, which may suggest an early intelligent fusion of information in different size categories.
A Separator Theorem for Hypergraphs and a CSP-SAT Algorithm
Venue: arXiv cs.LO 2021
Year: 2021

We show that for every $r \ge 2$ there exists $ε_r > 0$ such that any $r$-uniform hypergraph with $m$ edges and maximum vertex degree $o(\sqrt{m})$ contains a set of at most $(\frac{1}{2} - ε_r)m$ edges the removal of which breaks the hypergraph into connected components with at most $m/2$ edges. We use this to give an algorithm running in time $d^{(1 - ε_r)m}$ that decides satisfiability of $m$-variable $(d, k)$-CSPs in which every variable appears in at most $r$ constraints, where $ε_r$ depends only on $r$ and $k\in o(\sqrt{m})$. Furthermore our algorithm solves the corresponding #CSP-SAT and Max-CSP-SAT of these CSPs. We also show that CNF representations of unsatisfiable $(2, k)$-CSPs with variable frequency $r$ can be refuted in tree-like resolution in size $2^{(1 - ε_r)m}$. Furthermore for Tseitin formulas on graphs with degree at most $k$ (which are $(2, k)$-CSPs) we give a deterministic algorithm finding such a refutation.
Long Short-term Memory RNN
Venue: arXiv cs.LG 2021
Year: 2021

This paper is based on a machine learning project at the Norwegian University of Science and Technology, fall 2020. The project was initiated with a literature review on the latest developments within time-series forecasting methods in the scientific community over the past five years. The paper summarizes the essential aspects of this research. Furthermore, in this paper, we introduce an LSTM cell's architecture, and explain how different components go together to alter the cell's memory and predict the output. Also, the paper provides the necessary formulas and foundations to calculate a forward iteration through an LSTM. Then, the paper refers to some practical applications and research that emphasize the strength and weaknesses of LSTMs, shown within the time-series domain and the natural language processing (NLP) domain. Finally, alternative statistical methods for time series predictions are highlighted, where the paper outline ARIMA and exponential smoothing. Nevertheless, as LSTMs can be viewed as a complex architecture, the paper assumes that the reader has some knowledge of essential machine learning aspects, such as the multi-layer perceptron, activation functions, overfitting, backpropagation, bias, over- and underfitting, and more.
Partitioned Deep Learning of Fluid-Structure Interaction
Venue: arXiv cs.CE 2021
Year: 2021

We present a partitioned neural network-based framework for learning of fluid-structure interaction (FSI) problems. We decompose the simulation domain into two smaller sub-domains, i.e., fluid and solid domains, and incorporate an independent neural network for each. A library is used to couple the two networks which takes care of boundary data communication, data mapping and equation coupling. Simulation data are used for training of the both neural networks. We use a combination of convolutional and recurrent neural networks (CNN and RNN) to account for both spatial and temporal connectivity. A quasi-Newton method is used to accelerate the FSI coupling convergence. We observe a very good agreement between the results of the presented framework and the classical numerical methods for simulation of 1d fluid flow inside an elastic tube. This work is a preliminary step for using neural networks to speed-up the FSI coupling convergence by providing an accurate initial guess in each time step for classical numerical solvers
Waste detection in Pomerania: non-profit project for detecting waste in environment
Venue: arXiv cs.CV 2021
Year: 2021

Waste pollution is one of the most significant environmental issues in the modern world. The importance of recycling is well known, either for economic or ecological reasons, and the industry demands high efficiency. Our team conducted comprehensive research on Artificial Intelligence usage in waste detection and classification to fight the world's waste pollution problem. As a result an open-source framework that enables the detection and classification of litter was developed. The final pipeline consists of two neural networks: one that detects litter and a second responsible for litter classification. Waste is classified into seven categories: bio, glass, metal and plastic, non-recyclable, other, paper and unknown. Our approach achieves up to 70% of average precision in waste detection and around 75% of classification accuracy on the test dataset. The code used in the studies is publicly available online.
Multi-task Graph Convolutional Neural Network for Calcification Morphology and Distribution Analysis in Mammograms
Venue: arXiv cs.CV 2021
Year: 2021

The morphology and distribution of microcalcifications in a cluster are the most important characteristics for radiologists to diagnose breast cancer. However, it is time-consuming and difficult for radiologists to identify these characteristics, and there also lacks of effective solutions for automatic characterization. In this study, we proposed a multi-task deep graph convolutional network (GCN) method for the automatic characterization of morphology and distribution of microcalcifications in mammograms. Our proposed method transforms morphology and distribution characterization into node and graph classification problem and learns the representations concurrently. Through extensive experiments, we demonstrate significant improvements with the proposed multi-task GCN comparing to the baselines. Moreover, the achieved improvements can be related to and enhance clinical understandings. We explore, for the first time, the application of GCNs in microcalcification characterization that suggests the potential of graph learning for more robust understanding of medical images.
Sparsity-Probe: Analysis tool for Deep Learning Models
Venue: arXiv cs.LG 2021
Year: 2021

We propose a probe for the analysis of deep learning architectures that is based on machine learning and approximation theoretical principles. Given a deep learning architecture and a training set, during or after training, the Sparsity Probe allows to analyze the performance of intermediate layers by quantifying the geometrical features of representations of the training set. We show how the Sparsity Probe enables measuring the contribution of adding depth to a given architecture, to detect under-performing layers, etc., all this without any auxiliary test data set.
From Multisets over Distributions to Distributions over Multisets
Venue: arXiv cs.LO 2021
Year: 2021

A well-known challenge in the semantics of programming languages is how to combine non-determinism and probability. At a technical level, the problem arises from the fact that there is a no distributive law between the powerset monad and the distribution monad - as noticed some twenty years ago by Plotkin. More recently, it has become clear that there is a distributive law of the multiset monad over the distribution monad. This article elaborates the details of this distributivity and shows that there is a rich underlying theory relating multisets and probability distributions. It is shown that the new distributive law, called parallel multinomial law, can be defined in (at least) four equivalent ways. It involves putting multinomial distributions in parallel and commutes with hypergeometric distributions. Further, it is shown that this distributive law commutes with a new form of zipping for multisets. Abstractly, this can be described in terms of monoidal structure for a fixed-size multiset functor, when lifted to the Kleisli category of the distribution monad. Concretely, an application of the theory to sampling semantics is included.
The Greedy Algorithm is \emph{not} Optimal for On-Line Edge Coloring
Venue: arXiv cs.DS 2021
Year: 2021

Nearly three decades ago, Bar-Noy, Motwani and Naor showed that no online edge-coloring algorithm can edge color a graph optimally. Indeed, their work, titled "the greedy algorithm is optimal for on-line edge coloring", shows that the competitive ratio of $2$ of the naïve greedy algorithm is best possible online. However, their lower bound required bounded-degree graphs, of maximum degree $Δ= O(\log n)$, which prompted them to conjecture that better bounds are possible for higher-degree graphs. While progress has been made towards resolving this conjecture for restricted inputs and arrivals or for random arrival orders, an answer for fully general \emph{adversarial} arrivals remained elusive. We resolve this thirty-year-old conjecture in the affirmative, presenting a $(1.9+o(1))$-competitive online edge coloring algorithm for general graphs of degree $Δ= ω(\log n)$ under vertex arrivals. At the core of our results, and of possible independent interest, is a new online algorithm which rounds a fractional bipartite matching $x$ online under vertex arrivals, guaranteeing that each edge $e$ is matched with probability $(1/2+c)\cdot x_e$, for a constant $c>0.027$.
Plot and Rework: Modeling Storylines for Visual Storytelling
Venue: arXiv cs.CL 2021
Year: 2021

Writing a coherent and engaging story is not easy. Creative writers use their knowledge and worldview to put disjointed elements together to form a coherent storyline, and work and rework iteratively toward perfection. Automated visual storytelling (VIST) models, however, make poor use of external knowledge and iterative generation when attempting to create stories. This paper introduces PR-VIST, a framework that represents the input image sequence as a story graph in which it finds the best path to form a storyline. PR-VIST then takes this path and learns to generate the final story via an iterative training process. This framework produces stories that are superior in terms of diversity, coherence, and humanness, per both automatic and human evaluations. An ablation study shows that both plotting and reworking contribute to the model's superiority.