OpenRaiser/Intern-Atlas huggingface dataset:
NeuralPDE: Modelling Dynamical Systems from Data
Venue: arXiv cs.LG 2022
Year: 2022

Many physical processes such as weather phenomena or fluid mechanics are governed by partial differential equations (PDEs). Modelling such dynamical systems using Neural Networks is an active research field. However, current methods are still very limited, as they do not exploit the knowledge about the dynamical nature of the system, require extensive prior knowledge about the governing equations or are limited to linear or first-order equations. In this work we make the observation that the Method of Lines used to solve PDEs can be represented using convolutions which makes convolutional neural networks (CNNs) the natural choice to parametrize arbitrary PDE dynamics. We combine this parametrization with differentiable ODE solvers to form the NeuralPDE Model, which explicitly takes into account the fact that the data is governed by differential equations. We show in several experiments on toy and real-world data that our model consistently outperforms state-of-the-art models used to learn dynamical systems.
The Choice of Knowledge Base in Automated Claim Checking
Venue: arXiv cs.CL 2022
Year: 2022

Automated claim checking is the task of determining the veracity of a claim given evidence found in a knowledge base of trustworthy facts. While previous work has taken the knowledge base as given and optimized the claim-checking pipeline, we take the opposite approach - taking the pipeline as given, we explore the choice of knowledge base. Our first insight is that a claim-checking pipeline can be transferred to a new domain of claims with access to a knowledge base from the new domain. Second, we do not find a "universally best" knowledge base - higher domain overlap of a task dataset and a knowledge base tends to produce better label accuracy. Third, combining multiple knowledge bases does not tend to improve performance beyond using the closest-domain knowledge base. Finally, we show that the claim-checking pipeline's confidence score for selecting evidence can be used to assess whether a knowledge base will perform well for a new set of claims, even in the absence of ground-truth labels.
Deep Semantic Manipulation of Facial Videos
Venue: arXiv cs.CV 2022
Year: 2022

Editing and manipulating facial features in videos is an interesting and important field of research with a plethora of applications, ranging from movie post-production and visual effects to realistic avatars for video games and virtual assistants. Our method supports semantic video manipulation based on neural rendering and 3D-based facial expression modelling. We focus on interactive manipulation of the videos by altering and controlling the facial expressions, achieving promising photorealistic results. The proposed method is based on a disentangled representation and estimation of the 3D facial shape and activity, providing the user with intuitive and easy-to-use control of the facial expressions in the input video. We also introduce a user-friendly, interactive AI tool that processes human-readable semantic labels about the desired expression manipulations in specific parts of the input video and synthesizes photorealistic manipulated videos. We achieve that by mapping the emotion labels to points on the Valence-Arousal space (where Valence quantifies how positive or negative is an emotion and Arousal quantifies the power of the emotion activation), which in turn are mapped to disentangled 3D facial expressions through an especially-designed and trained expression decoder network. The paper presents detailed qualitative and quantitative experiments, which demonstrate the effectiveness of our system and the promising results it achieves.
CoLLIE: Continual Learning of Language Grounding from Language-Image Embeddings
Venue: arXiv cs.CL 2022
Year: 2022

This paper presents CoLLIE: a simple, yet effective model for continual learning of how language is grounded in vision. Given a pre-trained multimodal embedding model, where language and images are projected in the same semantic space (in this case CLIP by OpenAI), CoLLIE learns a transformation function that adjusts the language embeddings when needed to accommodate new language use. This is done by predicting the difference vector that needs to be applied, as well as a scaling factor for this vector, so that the adjustment is only applied when needed. Unlike traditional few-shot learning, the model does not just learn new classes and labels, but can also generalize to similar language use and leverage semantic compositionality. We verify the model's performance on two different tasks of identifying the targets of referring expressions, where it has to learn new language use. The results show that the model can efficiently learn and generalize from only a few examples, with little interference with the model's original zero-shot performance.
Hybrid Beam Alignment for Multi-Path Channels: A Group Testing Viewpoint
Venue: arXiv cs.IT 2022
Year: 2022

High-frequency bands such as millimeter-wave and terahertz require narrow beams due to path loss and shadowing. Beam alignment (BA) methods allow the transceivers to adjust the directions of these beams efficiently by exploiting the channel sparsity at high frequencies. This paper investigates BA for an uplink scenario, where the channel between the user equipment (UE) and base station (BS) consists of multiple paths. The BS wishes to localize the angle of arrival of each of these paths with a given resolution using the least number of time slots. At each time slot of the BA, the UE transmits a BA packet and the BS uses hybrid beamforming to scan its angular region. To minimize the expected BA duration, a group testing framework is devised, and the associated novel analog and hybrid BA strategies are described. Simulation studies show the performance improvement both in noiseless and realistic 5G mmWave BA settings.
Free Will Belief as a consequence of Model-based Reinforcement Learning
Venue: arXiv cs.LG 2022
Year: 2022

The debate on whether or not humans have free will has been raging for centuries. Although there are good arguments based on our current understanding of the laws of nature for the view that it is not possible for humans to have free will, most people believe they do. This discrepancy begs for an explanation. If we accept that we do not have free will, we are faced with two problems: (1) while freedom is a very commonly used concept that everyone intuitively understands, what are we actually referring to when we say that an action or choice is "free" or not? And, (2) why is the belief in free will so common? Where does this belief come from, and what is its purpose, if any? In this paper, we examine these questions from the perspective of reinforcement learning (RL). RL is a framework originally developed for training artificial intelligence agents. However, it can also be used as a computational model of human decision making and learning, and by doing so, we propose that the first problem can be answered by observing that people's common sense understanding of freedom is closely related to the information entropy of an RL agent's normalized action values, while the second can be explained by the necessity for agents to model themselves as if they could have taken decisions other than those they actually took, when dealing with the temporal credit assignment problem. Put simply, we suggest that by applying the RL framework as a model for human learning it becomes evident that in order for us to learn efficiently and be intelligent we need to view ourselves as if we have free will.
Patent Data for Engineering Design: A Critical Review and Future Directions
Venue: arXiv cs.DL 2022
Year: 2022

Patent data have long been used for engineering design research because of its large and expanding size, and widely varying massive amount of design information contained in patents. Recent advances in artificial intelligence and data science present unprecedented opportunities to develop data-driven design methods and tools, as well as advance design science, using the patent database. Herein, we survey and categorize the patent-for-design literature based on its contributions to design theories, methods, tools, and strategies, as well as the types of patent data and data-driven methods used in respective studies. Our review highlights promising future research directions in patent data-driven design research and practice.
On the Potential of Execution Traces for Batch Processing Workload Optimization in Public Clouds
Venue: arXiv cs.DC 2022
Year: 2022

With the growing amount of data, data processing workloads and the management of their resource usage becomes increasingly important. Since managing a dedicated infrastructure is in many situations infeasible or uneconomical, users progressively execute their respective workloads in the cloud. As the configuration of workloads and resources is often challenging, various methods have been proposed that either quickly profile towards a good configuration or determine one based on data from previous runs. Still, performance data to train such methods is often lacking and must be costly collected. In this paper, we propose a collaborative approach for sharing anonymized workload execution traces among users, mining them for general patterns, and exploiting clusters of historical workloads for future optimizations. We evaluate our prototype implementation for mining workload execution graphs on a publicly available trace dataset and demonstrate the predictive value of workload clusters determined through traces only.
Self-Learning Tuning for Post-Silicon Validation
Venue: arXiv cs.LG 2022
Year: 2022

Increasing complexity of modern chips makes design validation more difficult. Existing approaches are not able anymore to cope with the complexity of tasks such as robust performance tuning in post-silicon validation. Therefore, we propose a novel approach based on learn-to-optimize and reinforcement learning in order to solve complex and mixed-type tuning tasks in a efficient and robust way.
A Neural Solver for Variational Problems on CAD Geometries with Application to Electric Machine Simulation
Venue: arXiv cs.CE 2022
Year: 2022

This work presents a deep learning-based framework for the solution of partial differential equations on complex computational domains described with computer-aided design tools. To account for the underlying distribution of the training data caused by spline-based projections from the reference to the physical domain, a variational neural solver equipped with an importance sampling scheme is developed, such that the loss function based on the discretized energy functional obtained after the weak formulation is modified according to the sample distribution. To tackle multi-patch domains possibly leading to solution discontinuities, the variational neural solver is additionally combined with a domain decomposition approach based on the Discontinuous Galerkin formulation. The proposed neural solver is verified on a toy problem and then applied to a real-world engineering test case, namely that of electric machine simulation. The numerical results show clearly that the neural solver produces physics-conforming solutions of significantly improved accuracy.
ReaLPrune: ReRAM Crossbar-aware Lottery Ticket Pruned CNNs
Venue: arXiv cs.AR 2022
Year: 2022

Training machine learning (ML) models at the edge (on-chip training on end user devices) can address many pressing challenges including data privacy/security, increase the accessibility of ML applications to different parts of the world by reducing the dependence on the communication fabric and the cloud infrastructure, and meet the real-time requirements of AR/VR applications. However, existing edge platforms do not have sufficient computing capabilities to support complex ML tasks such as training large CNNs. ReRAM-based architectures offer high-performance yet energy efficient computing platforms for on-chip CNN training/inferencing. However, ReRAM-based architectures are not scalable with the size of the CNN. Larger CNNs have more weights, which requires more ReRAM cells that cannot be integrated in a single chip. Moreover, training larger CNNs on-chip will require higher power, which cannot be afforded by these smaller devices. Pruning is an effective way to solve this problem. However, existing pruning techniques are either targeted for inferencing only, or they are not crossbar-aware. This leads to sub-optimal hardware savings and performance benefits for CNN training on ReRAM-based architectures. In this paper, we address this problem by proposing a novel crossbar-aware pruning strategy, referred as ReaLPrune, which can prune more than 90% of CNN weights. The pruned model can be trained from scratch without any accuracy loss. Experimental results indicate that ReaLPrune reduces hardware requirements by 77.2% and accelerates CNN training by ~20X compared to unpruned CNNs. ReaLPrune also outperforms other crossbar-aware pruning techniques in terms of both performance and hardware savings. In addition, ReaLPrune is equally effective for diverse datasets and more complex CNNs
Punyo-1: Soft tactile-sensing upper-body robot for large object manipulation and physical human interaction
Venue: arXiv cs.RO 2022
Year: 2022

The manipulation of large objects and safe operation in the vicinity of humans are key capabilities of a general purpose domestic robotic assistant. We present the design of a soft, tactile-sensing humanoid upper-body robot and demonstrate whole-body rich-contact manipulation strategies for handling large objects. We demonstrate our hardware design philosophy for outfitting off-the-shelf hard robot arms and other components with soft tactile-sensing modules, including: (i) low-cost, cut-resistant, contact pressure localizing coverings for the arms, (ii) paws based on TRI's Soft-bubble sensors for the end effectors, and (iii) compliant force/geometry sensors for the coarse geometry sensing chest. We leverage the mechanical intelligence and tactile sensing of these modules to develop and demonstrate motion primitives for whole-body grasping. We evaluate the hardware's effectiveness in achieving grasps of varying strengths over a variety of large domestic objects. Our results demonstrate the importance of exploiting softness and tactile sensing in contact-rich manipulation strategies, as well as a path forward for whole-body force-controlled interactions with the world. (The supplemental video is available publicly at https://youtu.be/G8ZYgPRV5LY).
MPF6D: Masked Pyramid Fusion 6D Pose Estimation
Venue: arXiv cs.CV 2022
Year: 2022

Object pose estimation has multiple important applications, such as robotic grasping and augmented reality. We present a new method to estimate the 6D pose of objects that improves upon the accuracy of current proposals and can still be used in real-time. Our method uses RGB-D data as input to segment objects and estimate their pose. It uses a neural network with multiple heads to identify the objects in the scene, generate the appropriate masks and estimate the values of the translation vectors and the quaternion that represents the objects' rotation. These heads leverage a pyramid architecture used during feature extraction and feature fusion. We conduct an empirical evaluation using the two most common datasets in the area, and compare against state-of-the-art approaches, illustrating the capabilities of MPF6D. Our method can be used in real-time with its low inference time and high accuracy.
Automatic Neural Network Pruning that Efficiently Preserves the Model Accuracy
Venue: arXiv cs.CV 2022
Year: 2022

Neural networks performance has been significantly improved in the last few years, at the cost of an increasing number of floating point operations per second (FLOPs). However, more FLOPs can be an issue when computational resources are limited. As an attempt to solve this problem, pruning filters is a common solution, but most existing pruning methods do not preserve the model accuracy efficiently and therefore require a large number of finetuning epochs. In this paper, we propose an automatic pruning method that learns which neurons to preserve in order to maintain the model accuracy while reducing the FLOPs to a predefined target. To accomplish this task, we introduce a trainable bottleneck that only requires one single epoch with 25.6% (CIFAR-10) or 7.49% (ILSVRC2012) of the dataset to learn which filters to prune. Experiments on various architectures and datasets show that the proposed method can not only preserve the accuracy after pruning but also outperform existing methods after finetuning. We achieve a 52.00% FLOPs reduction on ResNet-50, with a Top-1 accuracy of 47.51% after pruning and a state-of-the-art (SOTA) accuracy of 76.63% after finetuning on ILSVRC2012. Code available at https://github.com/nota-github/autobot_AAAI23.
Enhanced Membership Inference Attacks against Machine Learning Models
Venue: arXiv cs.LG 2022
Year: 2022

How much does a machine learning algorithm leak about its training data, and why? Membership inference attacks are used as an auditing tool to quantify this leakage. In this paper, we present a comprehensive \textit{hypothesis testing framework} that enables us not only to formally express the prior work in a consistent way, but also to design new membership inference attacks that use reference models to achieve a significantly higher power (true positive rate) for any (false positive rate) error. More importantly, we explain \textit{why} different attacks perform differently. We present a template for indistinguishability games, and provide an interpretation of attack success rate across different instances of the game. We discuss various uncertainties of attackers that arise from the formulation of the problem, and show how our approach tries to minimize the attack uncertainty to the one bit secret about the presence or absence of a data point in the training set. We perform a \textit{differential analysis} between all types of attacks, explain the gap between them, and show what causes data points to be vulnerable to an attack (as the reasons vary due to different granularities of memorization, from overfitting to conditional memorization). Our auditing framework is openly accessible as part of the \textit{Privacy Meter} software tool.
Resilience from Diversity: Population-based approach to harden models against adversarial attacks
Venue: arXiv cs.LG 2022
Year: 2022

Traditional deep learning networks (DNN) exhibit intriguing vulnerabilities that allow an attacker to force them to fail at their task. Notorious attacks such as the Fast Gradient Sign Method (FGSM) and the more powerful Projected Gradient Descent (PGD) generate adversarial samples by adding a magnitude of perturbation $ε$ to the input's computed gradient, resulting in a deterioration of the effectiveness of the model's classification. This work introduces a model that is resilient to adversarial attacks. Our model leverages an established mechanism of defense which utilizes randomness and a population of DNNs. More precisely, our model consists of a population of $n$ diverse submodels, each one of them trained to individually obtain a high accuracy for the task at hand, while forced to maintain meaningful differences in their weights. Each time our model receives a classification query, it selects a submodel from its population at random to answer the query. To counter the attack transferability, diversity is introduced and maintained in the population of submodels. Thus introducing the concept of counter linking weights. A Counter-Linked Model (CLM) consists of a population of DNNs of the same architecture where a periodic random similarity examination is conducted during the simultaneous training to guarantee diversity while maintaining accuracy. Though the randomization technique proved to be resilient against adversarial attacks, we show that by retraining the DNNs ensemble or training them from the start with counter linking would enhance the robustness by around 20\% when tested on the MNIST dataset and at least 15\% when tested on the CIFAR-10 dataset. When CLM is coupled with adversarial training, this defense mechanism achieves state-of-the-art robustness.
Reliable Distributed Computing for Metaverse: A Hierarchical Game-Theoretic Approach
Venue: arXiv cs.NI 2022
Year: 2022

The metaverse is regarded as a new wave of technological transformation that provides a virtual space for people to interact through digital avatars. To achieve immersive user experiences in the metaverse, real-time rendering is the key technology. However, computing-intensive tasks of real-time rendering from metaverse service providers cannot be processed efficiently on a single resource-limited mobile device. Alternatively, such mobile devices can offload the metaverse rendering tasks to other mobile devices by adopting the collaborative computing paradigm based on Coded Distributed Computing (CDC). Therefore, this paper introduces a hierarchical game-theoretic CDC framework for the metaverse services, especially for the vehicular metaverse. In the framework, idle resources from vehicles, acting as CDC workers, are aggregated to handle intensive computation tasks in the vehicular metaverse. Specifically, in the upper layer, a miner coalition formation game is formulated based on a reputation metric to select reliable workers. To guarantee the reliable management of reputation values, the reputation values calculated based on the subjective logical model are maintained in a blockchain database. In the lower layer, a Stackelberg game-based incentive mechanism is considered to attract reliable workers selected in the upper layer to participate in rendering tasks. The simulation results illustrate that the proposed framework is resistant to malicious workers. Compared with the best-effort worker selection scheme, the proposed scheme can improve the utility of metaverse service provider and the average profit of CDC workers.
Towards Graph Self-Supervised Learning with Contrastive Adjusted Zooming
Venue: arXiv cs.LG 2022
Year: 2022

Graph representation learning (GRL) is critical for graph-structured data analysis. However, most of the existing graph neural networks (GNNs) heavily rely on labeling information, which is normally expensive to obtain in the real world. Although some existing works aim to effectively learn graph representations in an unsupervised manner, they suffer from certain limitations, such as the heavy reliance on monotone contrastiveness and limited scalability. To overcome the aforementioned problems, we introduce a novel self-supervised graph representation learning algorithm via Graph Contrastive Adjusted Zooming, namely G-Zoom, to learn node representations by leveraging the proposed adjusted zooming scheme. Specifically, this mechanism enables G-Zoom to explore and extract self-supervision signals from a graph from multiple scales: micro (i.e., node-level), meso (i.e., neighborhood-level), and macro (i.e., subgraph-level). Firstly, we generate two augmented views of the input graph via two different graph augmentations. Then, we establish three different contrastiveness on the above three scales progressively, from node, neighboring, to subgraph level, where we maximize the agreement between graph representations across scales. While we can extract valuable clues from a given graph on the micro and macro perspectives, the neighboring-level contrastiveness offers G-Zoom the capability of a customizable option based on our adjusted zooming scheme to manually choose an optimal viewpoint that lies between the micro and macro perspectives to better understand the graph data. Additionally, to make our model scalable to large graphs, we employ a parallel graph diffusion approach to decouple model training from the graph size. We have conducted extensive experiments on real-world datasets, and the results demonstrate that our proposed model outperforms state-of-the-art methods consistently.