OpenRaiser/Intern-Atlas huggingface dataset:
A Label Dependence-aware Sequence Generation Model for Multi-level Implicit Discourse Relation Recognition
Venue: arXiv cs.CL 2021
Year: 2021

Implicit discourse relation recognition (IDRR) is a challenging but crucial task in discourse analysis. Most existing methods train multiple models to predict multi-level labels independently, while ignoring the dependence between hierarchically structured labels. In this paper, we consider multi-level IDRR as a conditional label sequence generation task and propose a Label Dependence-aware Sequence Generation Model (LDSGM) for it. Specifically, we first design a label attentive encoder to learn the global representation of an input instance and its level-specific contexts, where the label dependence is integrated to obtain better label embeddings. Then, we employ a label sequence decoder to output the predicted labels in a top-down manner, where the predicted higher-level labels are directly used to guide the label prediction at the current level. We further develop a mutual learning enhanced training method to exploit the label dependence in a bottomup direction, which is captured by an auxiliary decoder introduced during training. Experimental results on the PDTB dataset show that our model achieves the state-of-the-art performance on multi-level IDRR. We will release our code at https://github.com/nlpersECJTU/LDSGM.
Security Risks of Porting C Programs to WebAssembly
Venue: arXiv cs.CR 2021
Year: 2021

WebAssembly is a compilation target for cross-platform applications that is increasingly being used. In this paper, we investigate whether one can transparently cross-compile C programs to WebAssembly, and if not, what impact porting can have on their security. We compile 17,802 programs that exhibit common vulnerabilities to 64-bit x86 and to WebAssembly binaries, and we observe that the execution of 4,911 binaries produces different results across these platforms. Through manual inspection, we identify three classes of root causes for such differences: the use of a different standard library implementation, the lack of security measures in WebAssembly, and the different semantics of the execution environments. We describe our observations and discuss the ones that are critical from a security point of view and need most attention from developers. We conclude that compiling an existing C program to WebAssembly for cross-platform distribution may require source code adaptations; otherwise, the security of the WebAssembly application may be at risk.
Class-aware Sounding Objects Localization via Audiovisual Correspondence
Venue: arXiv cs.CV 2021
Year: 2021

Audiovisual scenes are pervasive in our daily life. It is commonplace for humans to discriminatively localize different sounding objects but quite challenging for machines to achieve class-aware sounding objects localization without category annotations, i.e., localizing the sounding object and recognizing its category. To address this problem, we propose a two-stage step-by-step learning framework to localize and recognize sounding objects in complex audiovisual scenarios using only the correspondence between audio and vision. First, we propose to determine the sounding area via coarse-grained audiovisual correspondence in the single source cases. Then visual features in the sounding area are leveraged as candidate object representations to establish a category-representation object dictionary for expressive visual character extraction. We generate class-aware object localization maps in cocktail-party scenarios and use audiovisual correspondence to suppress silent areas by referring to this dictionary. Finally, we employ category-level audiovisual consistency as the supervision to achieve fine-grained audio and sounding object distribution alignment. Experiments on both realistic and synthesized videos show that our model is superior in localizing and recognizing objects as well as filtering out silent ones. We also transfer the learned audiovisual network into the unsupervised object detection task, obtaining reasonable performance.
The Importance of the Current Input in Sequence Modeling
Venue: arXiv cs.CL 2021
Year: 2021

The last advances in sequence modeling are mainly based on deep learning approaches. The current state of the art involves the use of variations of the standard LSTM architecture, combined with several tricks that improve the final prediction rates of the trained neural networks. However, in some cases, these adaptations might be too much tuned to the particular problems being addressed. In this article, we show that a very simple idea, to add a direct connection between the input and the output, skipping the recurrent module, leads to an increase of the prediction accuracy in sequence modeling problems related to natural language processing. Experiments carried out on different problems show that the addition of this kind of connection to a recurrent network always improves the results, regardless of the architecture and training-specific details. When this idea is introduced into the models that lead the field, the resulting networks achieve a new state-of-the-art perplexity in language modeling problems.
Exploring Inter-frequency Guidance of Image for Lightweight Gaussian Denoising
Venue: arXiv cs.CV 2021
Year: 2021

Image denoising is of vital importance in many imaging or computer vision related areas. With the convolutional neural networks showing strong capability in computer vision tasks, the performance of image denoising has also been brought up by CNN based methods. Though CNN based image denoisers show promising results on this task, most of the current CNN based methods try to learn the mapping from noisy image to clean image directly, which lacks the explicit exploration of prior knowledge of images and noises. Natural images are observed to obey the reciprocal power law, implying the low-frequency band of image tend to occupy most of the energy. Thus in the condition of AGWN (additive gaussian white noise) deterioration, low-frequency band tend to preserve a higher PSNR than high-frequency band. Considering the spatial morphological consistency of different frequency bands, low-frequency band with more fidelity can be used as a guidance to refine the more contaminated high-frequency bands. Based on this thought, we proposed a novel network architecture denoted as IGNet, in order to refine the frequency bands from low to high in a progressive manner. Firstly, it decomposes the feature maps into high- and low-frequency subbands using DWT (discrete wavelet transform) iteratively, and then each low band features are used to refine the high band features. Finally, the refined feature maps are processed by a decoder to recover the clean result. With this design, more inter-frequency prior and information are utilized, thus the model size can be lightened while still perserves competitive results. Experiments on several public datasets show that our model obtains competitive performance comparing with other state-of-the-art methods yet with a lightweight structure.
Shape Fragments
Venue: arXiv cs.DB 2021
Year: 2021

In constraint languages for RDF graphs, such as ShEx and SHACL, constraints on nodes and their properties in RDF graphs are known as "shapes". Schemas in these languages list the various shapes that certain targeted nodes must satisfy for the graph to conform to the schema. Using SHACL, we propose in this paper a novel use of shapes, by which a set of shapes is used to extract a subgraph from an RDF graph, the so-called shape fragment. Our proposed mechanism fits in the framework of Linked Data Fragments. In this paper, (i) we define our extraction mechanism formally, building on recently proposed SHACL formalizations; (ii) we establish correctness properties, which relate shape fragments to notions of provenance for database queries; (iii) we compare shape fragments with SPARQL queries; (iv) we discuss implementation options; and (v) we present initial experiments demonstrating that shape fragments are a feasible new idea.
Dynamics of senses of new physics discourse: co-keywords analysis
Venue: arXiv cs.DL 2021
Year: 2021

The paper presents a longitudinal analysis of the evolution of new physics keywords co-occurrence patterns. For that, we explore the documents indexed in the INSPIRE database from 1989 to 2018. Our purpose is to quantify the knowledge structure of the fast-growing subfield of new physics. The development of a novel approach to keywords co-occurrence analysis is the main point of the paper. In contrast to traditional co-keyword network analysis, we investigate structures that unite physics concepts in different documents and bind different documents with the same physics concepts. We consider the structures that reveal relationships among concepts as topological and call them "physics senses". Based on the notion of trajectory mutual information, the paper offers clustering of physics senses, determines their period of life, and constructs a classification of senses' "authority".
Bottom-up approaches for multi-person pose estimation and it's applications: A brief review
Venue: arXiv cs.CV 2021
Year: 2021

Human Pose Estimation (HPE) is one of the fundamental problems in computer vision. It has applications ranging from virtual reality, human behavior analysis, video surveillance, anomaly detection, self-driving to medical assistance. The main objective of HPE is to obtain the person's posture from the given input. Among different paradigms for HPE, one paradigm is called bottom-up multi-person pose estimation. In the bottom-up approach, initially, all the key points of the targets are detected, and later in the optimization stage, the detected key points are associated with the corresponding targets. This review paper discussed the recent advancements in bottom-up approaches for the HPE and listed the possible high-quality datasets used to train the models. Additionally, a discussion of the prominent bottom-up approaches and their quantitative results on the standard performance matrices are given. Finally, the limitations of the existing methods are highlighted, and guidelines of the future research directions are given.
A Discriminative Single-Shot Segmentation Network for Visual Object Tracking
Venue: arXiv cs.CV 2021
Year: 2021

Template-based discriminative trackers are currently the dominant tracking paradigm due to their robustness, but are restricted to bounding box tracking and a limited range of transformation models, which reduces their localization accuracy. We propose a discriminative single-shot segmentation tracker -- D3S2, which narrows the gap between visual object tracking and video object segmentation. A single-shot network applies two target models with complementary geometric properties, one invariant to a broad range of transformations, including non-rigid deformations, the other assuming a rigid object to simultaneously achieve robust online target segmentation. The overall tracking reliability is further increased by decoupling the object and feature scale estimation. Without per-dataset finetuning, and trained only for segmentation as the primary output, D3S2 outperforms all published trackers on the recent short-term tracking benchmark VOT2020 and performs very close to the state-of-the-art trackers on the GOT-10k, TrackingNet, OTB100 and LaSoT. D3S2 outperforms the leading segmentation tracker SiamMask on video object segmentation benchmarks and performs on par with top video object segmentation algorithms.
Blockchain and Beyond: Understanding Blockchains through Prototypes and Public Engagement
Venue: arXiv cs.HC 2021
Year: 2021

This paper presents an annotated portfolio of projects that seek to understand and communicate the social and societal implications of blockchains, distributed ledgers and smart contracts. These complex technologies rely on human and technical factors to deliver cryptocurrencies, shared computation and trustless protocols but have a secondary benefit in providing a moment to re-think many aspects of society, and imagine alternative possibilities. The projects use design and HCI methods to relate blockchains to a range of topics, including global supply chains, delivery infrastructure, smart grids, volunteering and charitable giving, through engaging publics, exploring ideas and speculating on possible futures. Based on an extensive annotated portfolio we draw out learning for the design of blockchain systems, broadening participation and surfacing questions around imaginaries, social implications and engagement with new technology. This paints a comprehensive picture of how HCI and design can shape understandings of the future of complex technologies.
Few-shot Font Generation with Weakly Supervised Localized Representations
Venue: arXiv cs.CV 2021
Year: 2021

Automatic few-shot font generation aims to solve a well-defined, real-world problem because manual font designs are expensive and sensitive to the expertise of designers. Existing methods learn to disentangle style and content elements by developing a universal style representation for each font style. However, this approach limits the model in representing diverse local styles, because it is unsuitable for complicated letter systems, for example, Chinese, whose characters consist of a varying number of components (often called "radical") -- with a highly complex structure. In this paper, we propose a novel font generation method that learns localized styles, namely component-wise style representations, instead of universal styles. The proposed style representations enable the synthesis of complex local details in text designs. However, learning component-wise styles solely from a few reference glyphs is infeasible when a target script has a large number of components, for example, over 200 for Chinese. To reduce the number of required reference glyphs, we represent component-wise styles by a product of component and style factors, inspired by low-rank matrix factorization. Owing to the combination of strong representation and a compact factorization strategy, our method shows remarkably better few-shot font generation results (with only eight reference glyphs) than other state-of-the-art methods. Moreover, strong locality supervision, for example, location of each component, skeleton, or strokes, was not utilized. The source code is available at https://github.com/clovaai/lffont and https://github.com/clovaai/fewshot-font-generation.
An eXtended Finite Element Method Implementation in COMSOL Multiphysics: Thermo-Hydro-Mechanical Modeling of Fluid Flow in Discontinuous Porous Media
Venue: arXiv cs.MS 2021
Year: 2021

This paper presents the implementation of the eXtended Finite Element Method (XFEM) in the general-purpose commercial software package COMSOL Multiphysics for multi-field thermo-hydro-mechanical problems in discontinuous porous media. To this end, an exclusive enrichment strategy is proposed in compliance with the COMSOL modeling structure. COMSOL modules and physics interfaces are adopted to take account of the relevant physical processes involved in thermo-hydro-mechanical coupling analysis, namely: the mechanical deformation, fluid flow in porous media and heat transfer. Essential changes are made to the internal variables of the physics interfaces to ensure consistency in the evaluation of enriched solution fields. The model preprocessing, level-set updates, coupling of the relevant physics and postprocessing procedures are performed adopting a coherent utilization of the COMSOL built-in features along with the COMSOL LiveLink for MATLAB functions. The implementation process, remedies for the treatment of the enriched zones, XFEM framework setup, multiphysics coupling, numerical integration and numerical solution strategy are described in detail. The capabilities and performance of the proposed approach are investigated by examining several multi-field thermo-hydro-mechanical simulations involving single/multiple discontinuities in 2D/3D porous rock settings.
Continual learning of longitudinal health records
Venue: arXiv cs.LG 2021
Year: 2021

Continual learning denotes machine learning methods which can adapt to new environments while retaining and reusing knowledge gained from past experiences. Such methods address two issues encountered by models in non-stationary environments: ungeneralisability to new data, and the catastrophic forgetting of previous knowledge when retrained. This is a pervasive problem in clinical settings where patient data exhibits covariate shift not only between populations, but also continuously over time. However, while continual learning methods have seen nascent success in the imaging domain, they have been little applied to the multi-variate sequential data characteristic of critical care patient recordings. Here we evaluate a variety of continual learning methods on longitudinal ICU data in a series of representative healthcare scenarios. We find that while several methods mitigate short-term forgetting, domain shift remains a challenging problem over large series of tasks, with only replay based methods achieving stable long-term performance. Code for reproducing all experiments can be found at https://github.com/iacobo/continual
Barely-Supervised Learning: Semi-Supervised Learning with very few labeled images
Venue: arXiv cs.CV 2021
Year: 2021

This paper tackles the problem of semi-supervised learning when the set of labeled samples is limited to a small number of images per class, typically less than 10, problem that we refer to as barely-supervised learning. We analyze in depth the behavior of a state-of-the-art semi-supervised method, FixMatch, which relies on a weakly-augmented version of an image to obtain supervision signal for a more strongly-augmented version. We show that it frequently fails in barely-supervised scenarios, due to a lack of training signal when no pseudo-label can be predicted with high confidence. We propose a method to leverage self-supervised methods that provides training signal in the absence of confident pseudo-labels. We then propose two methods to refine the pseudo-label selection process which lead to further improvements. The first one relies on a per-sample history of the model predictions, akin to a voting scheme. The second iteratively updates class-dependent confidence thresholds to better explore classes that are under-represented in the pseudo-labels. Our experiments show that our approach performs significantly better on STL-10 in the barely-supervised regime, e.g. with 4 or 8 labeled images per class.
Can Deep Neural Networks be Converted to Ultra Low-Latency Spiking Neural Networks?
Venue: arXiv cs.CV 2021
Year: 2021

Spiking neural networks (SNNs), that operate via binary spikes distributed over time, have emerged as a promising energy efficient ML paradigm for resource-constrained devices. However, the current state-of-the-art (SOTA) SNNs require multiple time steps for acceptable inference accuracy, increasing spiking activity and, consequently, energy consumption. SOTA training strategies for SNNs involve conversion from a non-spiking deep neural network (DNN). In this paper, we determine that SOTA conversion strategies cannot yield ultra low latency because they incorrectly assume that the DNN and SNN pre-activation values are uniformly distributed. We propose a new training algorithm that accurately captures these distributions, minimizing the error between the DNN and converted SNN. The resulting SNNs have ultra low latency and high activation sparsity, yielding significant improvements in compute efficiency. In particular, we evaluate our framework on image recognition tasks from CIFAR-10 and CIFAR-100 datasets on several VGG and ResNet architectures. We obtain top-1 accuracy of 64.19% with only 2 time steps on the CIFAR-100 dataset with ~159.2x lower compute energy compared to an iso-architecture standard DNN. Compared to other SOTA SNN models, our models perform inference 2.5-8x faster (i.e., with fewer time steps).
Multi-modal 3D Human Pose Estimation with 2D Weak Supervision in Autonomous Driving
Venue: arXiv cs.CV 2021
Year: 2021

3D human pose estimation (HPE) in autonomous vehicles (AV) differs from other use cases in many factors, including the 3D resolution and range of data, absence of dense depth maps, failure modes for LiDAR, relative location between the camera and LiDAR, and a high bar for estimation accuracy. Data collected for other use cases (such as virtual reality, gaming, and animation) may therefore not be usable for AV applications. This necessitates the collection and annotation of a large amount of 3D data for HPE in AV, which is time-consuming and expensive. In this paper, we propose one of the first approaches to alleviate this problem in the AV setting. Specifically, we propose a multi-modal approach which uses 2D labels on RGB images as weak supervision to perform 3D HPE. The proposed multi-modal architecture incorporates LiDAR and camera inputs with an auxiliary segmentation branch. On the Waymo Open Dataset, our approach achieves a 22% relative improvement over camera-only 2D HPE baseline, and 6% improvement over LiDAR-only model. Finally, careful ablation studies and parts based analysis illustrate the advantages of each of our contributions.
Real-Time Multi-Convex Model Predictive Control for Occlusion Free Target Tracking
Venue: arXiv cs.RO 2021
Year: 2021

This paper proposes a Model Predictive Control (MPC) algorithm for target tracking amongst static and dynamic obstacles. Our main contribution lies in improving the computational tractability and reliability of the underlying non-convex trajectory optimization. The result is an MPC algorithm that runs real-time on laptops and embedded hardware devices such as Jetson TX2. Our approach relies on novel reformulations for the tracking, collision, and occlusion constraints that induce a multi-convex structure in the resulting trajectory optimization. We exploit these mathematical structures using the split Bregman Iteration technique, eventually reducing our MPC to a series of convex Quadratic Programs solvable in a few milliseconds. The fast re-planning of our MPC allows for occlusion and collision-free tracking in complex environments even while considering a simple constant-velocity prediction for the target trajectory and dynamic obstacles. We perform extensive bench-marking in a realistic physics engine and show that our MPC outperforms the state-of-the-art algorithms in visibility, smoothness, and computation-time metrics.
Evolution and trade-off dynamics of functional load
Venue: arXiv cs.CL 2021
Year: 2021

Function Load (FL) quantifies the contributions by phonological contrasts to distinctions made across the lexicon. Previous research has linked particularly low values of FL to sound change. Here we broaden the scope of enquiry into FL, to its evolution at all values. We apply phylogenetic methods to examine the diachronic evolution of FL across 90 languages of the Pama-Nyungan (PN) family of Australia. We find a high degree of phylogenetic signal in FL. Though phylogenetic signal has been reported for phonological structures, such as phonotactics, its detection in measures of phonological function is novel. We also find a significant, negative correlation between the FL of vowel length and of the following consonant, that is, a deep-time historical trade-off dynamic, which we relate to known allophony in modern PN languages and compensatory sound changes in their past. The finding reveals a historical dynamic, similar to transphonologization, which we characterize as a flow of contrastiveness between subsystems of the phonology. Recurring across a language family which spans a whole continent and many millennia of time depth, our finding provides one of the most compelling examples yet of Sapir's 'drift' hypothesis, of non-accidentally parallel development in historically related languages.