OpenRaiser/Intern-Atlas huggingface dataset:
Respecting causality is all you need for training physics-informed neural networks
Venue: arXiv cs.LG 2022
Year: 2022

While the popularity of physics-informed neural networks (PINNs) is steadily rising, to this date PINNs have not been successful in simulating dynamical systems whose solution exhibits multi-scale, chaotic or turbulent behavior. In this work we attribute this shortcoming to the inability of existing PINNs formulations to respect the spatio-temporal causal structure that is inherent to the evolution of physical systems. We argue that this is a fundamental limitation and a key source of error that can ultimately steer PINN models to converge towards erroneous solutions. We address this pathology by proposing a simple re-formulation of PINNs loss functions that can explicitly account for physical causality during model training. We demonstrate that this simple modification alone is enough to introduce significant accuracy improvements, as well as a practical quantitative mechanism for assessing the convergence of a PINNs model. We provide state-of-the-art numerical results across a series of benchmarks for which existing PINNs formulations fail, including the chaotic Lorenz system, the Kuramoto-Sivashinsky equation in the chaotic regime, and the Navier-Stokes equations in the turbulent regime. To the best of our knowledge, this is the first time that PINNs have been successful in simulating such systems, introducing new opportunities for their applicability to problems of industrial complexity.
Unsupervised Clustering of Roman Potsherds via Variational Autoencoders
Venue: arXiv cs.CV 2022
Year: 2022

In this paper we propose an artificial intelligence imaging solution to support archaeologists in the classification task of Roman commonware potsherds. Usually, each potsherd is represented by its sectional profile as a two dimensional black-white image and printed in archaeological books related to specific archaeological excavations. The partiality and handcrafted variance of the fragments make their matching a challenging problem: we propose to pair similar profiles via the unsupervised hierarchical clustering of non-linear features learned in the latent space of a deep convolutional Variational Autoencoder (VAE) network. Our contribution also include the creation of a ROman COmmonware POTtery (ROCOPOT) database, with more than 4000 potsherds profiles extracted from 25 Roman pottery corpora, and a MATLAB GUI software for the easy inspection of shape similarities. Results are commented both from a mathematical and archaeological perspective so as to unlock new research directions in both communities.
Robust Dynamic Walking for a 3D Dual-SLIP Model under One-Step Unilateral Stiffness Perturbations: Towards Bipedal Locomotion over Compliant Terrain
Venue: arXiv cs.RO 2022
Year: 2022

Bipedal walking is one of the most important hallmarks of human that robots have been trying to mimic for many decades. Although previous control methodologies have achieved robot walking on some terrains, there is a need for a framework allowing stable and robust locomotion over a wide range of compliant surfaces. This work proposes a novel biomechanics-inspired controller that adjusts the stiffness of the legs in support for robust and dynamic bipedal locomotion over compliant terrains. First, the 3D Dual-SLIP model is extended to support for the first time locomotion over compliant surfaces with variable stiffness and damping parameters. Then, the proposed controller is compared to a Linear-Quadratic Regulator (LQR) controller, in terms of robustness on stepping on soft terrain. The LQR controller is shown to be robust only up to a moderate ground stiffness level of 174 kN/m, while it fails in lower stiffness levels. On the contrary, the proposed controller can produce stable gait in stiffness levels as low as 30 kN/m, which results in a vertical ground penetration of the leg that is deeper than 10% of its rest length. The proposed framework could advance the field of bipedal walking, by generating stable walking trajectories for a wide range of compliant terrains useful for the control of bipeds and humanoids, as well as by improving controllers for prosthetic devices with tunable stiffness.
Simplicial Attention Neural Networks
Venue: arXiv cs.LG 2022
Year: 2022

The aim of this work is to introduce simplicial attention networks (SANs), i.e., novel neural architectures that operate on data defined on simplicial complexes leveraging masked self-attentional layers. Hinging on formal arguments from topological signal processing, we introduce a proper self-attention mechanism able to process data components at different layers (e.g., nodes, edges, triangles, and so on), while learning how to weight both upper and lower neighborhoods of the given topological domain in a totally task-oriented fashion. The proposed SANs generalize most of the current architectures available for processing data defined on simplicial complexes. The proposed approach compares favorably with other methods when applied to different (inductive and transductive) tasks such as trajectory prediction and missing data imputations in citation complexes.
Skydiver: A Spiking Neural Network Accelerator Exploiting Spatio-Temporal Workload Balance
Venue: arXiv cs.AR 2022
Year: 2022

Spiking Neural Networks (SNNs) are developed as a promising alternative to Artificial Neural networks (ANNs) due to their more realistic brain-inspired computing models. SNNs have sparse neuron firing over time, i.e., spatio-temporal sparsity; thus, they are useful to enable energy-efficient hardware inference. However, exploiting spatio-temporal sparsity of SNNs in hardware leads to unpredictable and unbalanced workloads, degrading the energy efficiency. In this work, we propose an FPGA-based convolutional SNN accelerator called Skydiver that exploits spatio-temporal workload balance. We propose the Approximate Proportional Relation Construction (APRC) method that can predict the relative workload channel-wisely and a Channel-Balanced Workload Schedule (CBWS) method to increase the hardware workload balance ratio to over 90%. Skydiver was implemented on a Xilinx XC7Z045 FPGA and verified on image segmentation and MNIST classification tasks. Results show improved throughput by 1.4X and 1.2X for the two tasks. Skydiver achieved 22.6 KFPS throughput, and 42.4 uJ/Image prediction energy on the classification task with 98.5% accuracy.
Choose Your QA Model Wisely: A Systematic Study of Generative and Extractive Readers for Question Answering
Venue: arXiv cs.CL 2022
Year: 2022

While both extractive and generative readers have been successfully applied to the Question Answering (QA) task, little attention has been paid toward the systematic comparison of them. Characterizing the strengths and weaknesses of the two readers is crucial not only for making a more informed reader selection in practice but also for developing a deeper understanding to foster further research on improving readers in a principled manner. Motivated by this goal, we make the first attempt to systematically study the comparison of extractive and generative readers for question answering. To be aligned with the state-of-the-art, we explore nine transformer-based large pre-trained language models (PrLMs) as backbone architectures. Furthermore, we organize our findings under two main categories: (1) keeping the architecture invariant, and (2) varying the underlying PrLMs. Among several interesting findings, it is important to highlight that (1) the generative readers perform better in long context QA, (2) the extractive readers perform better in short context while also showing better out-of-domain generalization, and (3) the encoder of encoder-decoder PrLMs (e.g., T5) turns out to be a strong extractive reader and outperforms the standard choice of encoder-only PrLMs (e.g., RoBERTa). We also study the effect of multi-task learning on the two types of readers varying the underlying PrLMs and perform qualitative and quantitative diagnosis to provide further insights into future directions in modeling better readers.
Practical Detectability for Persistent Lock-Free Data Structures
Venue: arXiv cs.PL 2022
Year: 2022

Persistent memory (PM) is an emerging class of storage technology that combines the benefits of DRAM and SSD. This characteristic inspires research on persistent objects in PM with fine-grained concurrency control. Among such objects, persistent lock-free data structures (DSs) are particularly interesting thanks to their efficiency and scalability. One of the most widely used correctness criteria for persistent lock-free DSs is durable linearizability (Izraelevitz et. al., DISC 2016). However, durable linearizability is insufficient to use persistent DSs for fault-tolerant systems requiring exactly-once semantics for storage systems, because we may not be able to detect whether an operation is performed when a crash occurs. We present a practical programming framework for persistent lock-free DSs with detectability. In contrast to the prior work on such DSs, our framework supports (1) primitive detectable operations such as space-efficient compare-and-swap, insertion, and deletion; (2) systematic transformation of lock-free DSs in DRAM into those in PM requiring modest efforts; (3) comparable performance with non-detectable DSs by DRAM scratchpad optimization; and (4) recovery from both full system and thread crashes. The key idea is memento objects serving as a lightweight, precise, and per-thread checkpoints in PM. As a case study, we implement lock-free and combining queues and hash tables with detectability that outperform (and perform comparably) the state-of-the-art DSs with (and without, respectively) detectability.
Graph Representation Learning for Popularity Prediction Problem: A Survey
Venue: arXiv cs.SI 2022
Year: 2022

The online social platforms, like Twitter, Facebook, LinkedIn and WeChat, have grown really fast in last decade and have been one of the most effective platforms for people to communicate and share information with each other. Due to the "word of mouth" effects, information usually can spread rapidly on these social media platforms. Therefore, it is important to study the mechanisms driving the information diffusion and quantify the consequence of information spread. A lot of efforts have been focused on this problem to help us better understand and achieve higher performance in viral marketing and advertising. On the other hand, the development of neural networks has blossomed in the last few years, leading to a large number of graph representation learning (GRL) models. Compared to traditional models, GRL methods are often shown to be more effective. In this paper, we present a comprehensive review for existing works using GRL methods for popularity prediction problem, and categorize related literatures into two big classes, according to their mainly used model and techniques: embedding-based methods and deep learning methods. Deep learning method is further classified into six small classes: convolutional neural networks, graph convolutional networks, graph attention networks, graph neural networks, recurrent neural networks, and reinforcement learning. We compare the performance of these different models and discuss their strengths and limitations. Finally, we outline the challenges and future chances for popularity prediction problem.
What's in the Black Box? The False Negative Mechanisms Inside Object Detectors
Venue: arXiv cs.CV 2022
Year: 2022

In object detection, false negatives arise when a detector fails to detect a target object. To understand why object detectors produce false negatives, we identify five 'false negative mechanisms', where each mechanism describes how a specific component inside the detector architecture failed. Focusing on two-stage and one-stage anchor-box object detector architectures, we introduce a framework for quantifying these false negative mechanisms. Using this framework, we investigate why Faster R-CNN and RetinaNet fail to detect objects in benchmark vision datasets and robotics datasets. We show that a detector's false negative mechanisms differ significantly between computer vision benchmark datasets and robotics deployment scenarios. This has implications for the translation of object detectors developed for benchmark datasets to robotics applications. Code is publicly available at https://github.com/csiro-robotics/fn_mechanisms
Supervised Contrastive Learning with Structure Inference for Graph Classification
Venue: arXiv cs.LG 2022
Year: 2022

Advanced graph neural networks have shown great potentials in graph classification tasks recently. Different from node classification where node embeddings aggregated from local neighbors can be directly used to learn node labels, graph classification requires a hierarchical accumulation of different levels of topological information to generate discriminative graph embeddings. Still, how to fully explore graph structures and formulate an effective graph classification pipeline remains rudimentary. In this paper, we propose a novel graph neural network based on supervised contrastive learning with structure inference for graph classification. First, we propose a data-driven graph augmentation strategy that can discover additional connections to enhance the existing edge set. Concretely, we resort to a structure inference stage based on diffusion cascades to recover possible connections with high node similarities. Second, to improve the contrastive power of graph neural networks, we propose to use a supervised contrastive loss for graph classification. With the integration of label information, the one-vs-many contrastive learning can be extended to a many-vs-many setting, so that the graph-level embeddings with higher topological similarities will be pulled closer. The supervised contrastive loss and structure inference can be naturally incorporated within the hierarchical graph neural networks where the topological patterns can be fully explored to produce discriminative graph embeddings. Experiment results show the effectiveness of the proposed method compared with recent state-of-the-art methods.
Benchmarking and Interpreting End-to-end Learning of MIMO and Multi-User Communication
Venue: arXiv cs.IT 2022
Year: 2022

End-to-end autoencoder (AE) learning has the potential of exceeding the performance of human-engineered transceivers and encoding schemes, without a priori knowledge of communication-theoretic principles. In this work, we aim to understand to what extent and for which scenarios this claim holds true when comparing with fair benchmarks. Our particular focus is on memoryless multiple-input multiple-output (MIMO) and multi-user (MU) systems. Four case studies are considered: two point-to-point (closed-loop and open-loop MIMO) and two MU scenarios (MIMO broadcast and interference channels). For the point-to-point scenarios, we explain some of the performance gains observed in prior work through the selection of improved baseline schemes that include geometric shaping as well as bit and power allocation. For the MIMO broadcast channel, we demonstrate the feasibility of a novel AE method with centralized learning and decentralized execution. Interestingly, the learned scheme performs close to nonlinear vector-perturbation precoding and significantly outperforms conventional zero-forcing. Lastly, we highlight potential pitfalls when interpreting learned communication schemes. In particular, we show that the AE for the considered interference channel learns to avoid interference, albeit in a rotated reference frame. After de-rotating the learned signal constellation of each user, the resulting scheme corresponds to conventional time sharing with geometric shaping.
Bio-inspired Multi-robot Autonomy
Venue: arXiv cs.RO 2022
Year: 2022

Increasingly, high value industrial markets are driving trends for improved functionality and resilience from resident autonomous systems. This led to an increase in multi-robot fleets that aim to leverage the complementary attributes of the diverse platforms. In this paper we introduce a novel bio-inspired Symbiotic System of Systems Approach (SSOSA) for designing the operational governance of a multi-robot fleet consisting of ground-based quadruped and wheeled platforms. SSOSA couples the MR-fleet to the resident infrastructure monitoring systems into one collaborative digital commons. The hyper visibility of the integrated distributed systems, achieved through a latency bidirectional communication network, supports collaboration, coordination and corroboration (3C) across the integrated systems. In our experiment, we demonstrate how an operator can activate a pre-determined autonomous mission and utilize SSOSA to overcome intrinsic and external risks to the autonomous missions. We demonstrate how resilience can be enhanced by local collaboration between SPOT and Husky wherein we detect a replacement battery, and utilize the manipulator arm of SPOT to support a Clearpath Husky A200 wheeled robotic platform. This allows for increased resilience of an autonomous mission as robots can collaborate to ensure the battery state of the Husky robot. Overall, these initial results demonstrate the value of a SSOSA approach in addressing a key operational barrier to scalable autonomy, the resilience.
SPA-VAE: Similar-Parts-Assignment for Unsupervised 3D Point Cloud Generation
Venue: arXiv cs.CV 2022
Year: 2022

This paper addresses the problem of unsupervised parts-aware point cloud generation with learned parts-based self-similarity. Our SPA-VAE infers a set of latent canonical candidate shapes for any given object, along with a set of rigid body transformations for each such candidate shape to one or more locations within the assembled object. In this way, noisy samples on the surface of, say, each leg of a table, are effectively combined to estimate a single leg prototype. When parts-based self-similarity exists in the raw data, sharing data among parts in this way confers numerous advantages: modeling accuracy, appropriately self-similar generative outputs, precise in-filling of occlusions, and model parsimony. SPA-VAE is trained end-to-end using a variational Bayesian approach which uses the Gumbel-softmax trick for the shared part assignments, along with various novel losses to provide appropriate inductive biases. Quantitative and qualitative analyses on ShapeNet demonstrate the advantage of SPA-VAE.
A Survey of fault models and fault tolerance methods for 2D bus-based multi-core systems and TSV based 3D NOC many-core systems
Venue: arXiv cs.AR 2022
Year: 2022

Reliability has taken centre stage in the development of high-performance computing processors. A Surge of interest is noticeable in recent times in formulating fault and failure models, understanding failure mechanism and strategizing fault mitigation methods for improving the reliability of the system. The article presents a congregation of concepts illustrated one after the other for a better understanding of damages caused by radiation, relevant fault models, and effects of faults. We examine the state of art fault mitigation techniques at the logical layer for digital CMOS based design and SRAM based FPGA. CMOS SRAM structure is the same for both digital CMOS and FPGA. Understanding of resilient SRAM based FPGA is necessary for developing resilient prototypes and it facilitates a faster integration of digital CMOS designs. At the micro-architectural and architectural layer, error detection and recovery methods are discussed for bus-based multi-core systems. The Through silicon via based 3D Network on chip is the prospective solution for integrating many cores on single die. A suitable interconnection approach for petascale computing on many-core systems. The article presents an elaborate discussion on fault models, failure mechanisms, resilient 3D routers, defect tolerance methods for the TSV based 3D NOC many-core systems. Core redundancy, self-diagnosis and distributed diagnosis at the hardware level are examined for many-core systems. The article presents a gamut of fault tolerance solutions from logic level to processor core level in a multi-core and many-core scenario.
Magnetic Field Prediction Using Generative Adversarial Networks
Venue: arXiv cs.LG 2022
Year: 2022

Plenty of scientific and real-world applications are built on magnetic fields and their characteristics. To retrieve the valuable magnetic field information in high resolution, extensive field measurements are required, which are either time-consuming to conduct or even not feasible due to physical constraints. To alleviate this problem, we predict magnetic field values at a random point in space from a few point measurements by using a generative adversarial network (GAN) structure. The deep learning (DL) architecture consists of two neural networks: a generator, which predicts missing field values of a given magnetic field, and a critic, which is trained to calculate the statistical distance between real and generated magnetic field distributions. By minimizing this statistical distance, a reconstruction loss as well as physical losses, our trained generator has learned to predict the missing field values with a median reconstruction test error of 5.14%, when a single coherent region of field points is missing, and 5.86%, when only a few point measurements in space are available and the field measurements around are predicted. We verify the results on an experimentally validated field.
Dynamic Time Warping Under Translation: Approximation Guided by Space-Filling Curves
Venue: arXiv cs.CG 2022
Year: 2022

The Dynamic Time Warping (DTW) distance is a popular measure of similarity for a variety of sequence data. For comparing polygonal curves $π, σ$ in $\mathbb{R}^d$, it provides a robust, outlier-insensitive alternative to the Fréchet distance. However, like the Fréchet distance, the DTW distance is not invariant under translations. Can we efficiently optimize the DTW distance of $π$ and $σ$ under arbitrary translations, to compare the curves' shape irrespective of their absolute location? There are surprisingly few works in this direction, which may be due to its computational intricacy: For the Euclidean norm, this problem contains as a special case the geometric median problem, which provably admits no exact algebraic algorithm (that is, no algorithm using only addition, multiplication, and $k$-th roots). We thus investigate exact algorithms for non-Euclidean norms as well as approximation algorithms for the Euclidean norm: - For the $L_1$ norm in $\mathbb{R}^d$, we provide an $\mathcal{O}(n^{2(d+1)})$-time algorithm, i.e., an exact polynomial-time algorithm for constant $d$. Here and below, $n$ bounds the curves' complexities. - For the Euclidean norm in $\mathbb{R}^2$, we show that a simple problem-specific insight leads to a $(1+\varepsilon)$-approximation in time $\mathcal{O}(n^3/\varepsilon^2)$. We then show how to obtain a subcubic $\widetilde{\mathcal{O}}(n^{2.5}/\varepsilon^2)$ time algorithm with significant new ideas; this time comes close to the well-known quadratic time barrier for computing DTW for fixed translations. Technically, the algorithm is obtained by speeding up repeated DTW distance estimations using a dynamic data structure for maintaining shortest paths in weighted planar digraphs. Crucially, we show how to traverse a candidate set of translations using space-filling curves in a way that incurs only few updates to the data structure.