OpenRaiser/Intern-Atlas huggingface dataset:
Building Spatio-temporal Transformers for Egocentric 3D Pose Estimation
Venue: arXiv cs.CV 2022
Year: 2022

Egocentric 3D human pose estimation (HPE) from images is challenging due to severe self-occlusions and strong distortion introduced by the fish-eye view from the head mounted camera. Although existing works use intermediate heatmap-based representations to counter distortion with some success, addressing self-occlusion remains an open problem. In this work, we leverage information from past frames to guide our self-attention-based 3D HPE estimation procedure -- Ego-STAN. Specifically, we build a spatio-temporal Transformer model that attends to semantically rich convolutional neural network-based feature maps. We also propose feature map tokens: a new set of learnable parameters to attend to these feature maps. Finally, we demonstrate Ego-STAN's superior performance on the xR-EgoPose dataset where it achieves a 30.6% improvement on the overall mean per-joint position error, while leading to a 22% drop in parameters compared to the state-of-the-art.
Securing AI-based Healthcare Systems using Blockchain Technology: A State-of-the-Art Systematic Literature Review and Future Research Directions
Venue: arXiv cs.CR 2022
Year: 2022

Healthcare systems are increasingly incorporating Artificial Intelligence into their systems, but it is not a solution for all difficulties. AI's extraordinary potential is being held back by challenges such as a lack of medical datasets for training AI models, adversarial attacks, and a lack of trust due to its black box working style. We explored how blockchain technology can improve the reliability and trustworthiness of AI-based healthcare. This paper has conducted a Systematic Literature Review to explore the state-of-the-art research studies conducted in healthcare applications developed with different AI techniques and Blockchain Technology. This systematic literature review proceeds with three different paths as natural language processing-based healthcare systems, computer vision-based healthcare systems and acoustic AI-based healthcare systems. We found that 1) Defence techniques for adversarial attacks on AI are available for specific kind of attacks and even adversarial training is AI based technique which in further prone to different attacks. 2) Blockchain can address security and privacy issues in healthcare fraternity. 3) Medical data verification and user provenance can be enabled with Blockchain. 4) Blockchain can protect distributed learning on heterogeneous medical data. 5) The issues like single point of failure, non-transparency in healthcare systems can be resolved with Blockchain. Nevertheless, it has been identified that research is at the initial stage. As a result, we have synthesized a conceptual framework using Blockchain Technology for AI-based healthcare applications that considers the needs of each NLP, Computer Vision, and Acoustic AI application. A global solution for all sort of adversarial attacks on AI based healthcare. However, this technique has significant limits and challenges that need to be addressed in future studies.
Provable Guarantees for Sparsity Recovery with Deterministic Missing Data Patterns
Venue: arXiv cs.LG 2022
Year: 2022

We study the problem of consistently recovering the sparsity pattern of a regression parameter vector from correlated observations governed by deterministic missing data patterns using Lasso. We consider the case in which the observed dataset is censored by a deterministic, non-uniform filter. Recovering the sparsity pattern in datasets with deterministic missing structure can be arguably more challenging than recovering in a uniformly-at-random scenario. In this paper, we propose an efficient algorithm for missing value imputation by utilizing the topological property of the censorship filter. We then provide novel theoretical results for exact recovery of the sparsity pattern using the proposed imputation strategy. Our analysis shows that, under certain statistical and topological conditions, the hidden sparsity pattern can be recovered consistently with high probability in polynomial time and logarithmic sample complexity.
ABCDE: An Agent-Based Cognitive Development Environment
Venue: arXiv cs.AI 2022
Year: 2022

Children's cognitive abilities are sometimes cited as AI benchmarks. How can the most common 1,000 concepts (89\% of everyday use) be learnt in a naturalistic children's setting? Cognitive development in children is about quality, and new concepts can be conveyed via simple examples. Our approach of knowledge scaffolding uses simple objects and actions to convey concepts, like how children are taught. We introduce ABCDE, an interactive 3D environment modeled after a typical playroom for children. It comes with 300+ unique 3D object assets (mostly toys), and a large action space for child and parent agents to interact with objects and each other. ABCDE is the first environment aimed at mimicking a naturalistic setting for cognitive development in children; no other environment focuses on high-level concept learning through learner-teacher interactions. The simulator can be found at https://pypi.org/project/ABCDESim/1.0.0/
NSSIA: A New Self-Sovereign Identity Scheme with Accountability
Venue: arXiv cs.CR 2022
Year: 2022

Self-Sovereign Identity (SSI) is a new distributed method for identity management, commonly used to address the problem that users are lack of control over their identities. However, the excessive pursuit of self-sovereignty in the most existing SSI schemes hinders sanctions against attackers. To deal with the malicious behavior, a few SSI schemes introduce accountability mechanisms, but they sacrifice users' privacy. What's more, the digital identities (static strings or updatable chains) in the existing SSI schemes are as inputs to a third-party executable program (mobile app, smart contract, etc.) to achieve identity reading, storing and proving, users' self-sovereignty are weakened. To solve the above problems, we present a new self-sovereign identity scheme to strike a balance between privacy and accountability and get rid of the dependence on the third-party program. In our scheme, one and only individual-specific executable code is generated as a digital avatar-i for each human to interact with others in cyberspace without a third-party program, in which the embedding of biometrics enhances uniqueness and user control over their identity. In addition, a joint accountability mechanism, which is based on the shamir (t, n) threshold algorithm and a consortium blockchain, is designed to restrict the power of each regulatory authority and protect users' privacy. Finally, we analyze the security, SSI properties and conduct detailed experiments in term of the cost of computation, storage and blockchain gas. The analysis results indicate that our scheme resists the known attacks and fulfills all the six SSI properties. Compared with the state-of-the-art schemes, the extensive experiment results show that the cost is larger in server storage, blockchain storage and blockchain gas, but is still low enough for practical situations.
RuCoCo: a new Russian corpus with coreference annotation
Venue: arXiv cs.CL 2022
Year: 2022

We present a new corpus with coreference annotation, Russian Coreference Corpus (RuCoCo). The goal of RuCoCo is to obtain a large number of annotated texts while maintaining high inter-annotator agreement. RuCoCo contains news texts in Russian, part of which were annotated from scratch, and for the rest the machine-generated annotations were refined by human annotators. The size of our corpus is one million words and around 150,000 mentions. We make the corpus publicly available.
NR-DFERNet: Noise-Robust Network for Dynamic Facial Expression Recognition
Venue: arXiv cs.CV 2022
Year: 2022

Dynamic facial expression recognition (DFER) in the wild is an extremely challenging task, due to a large number of noisy frames in the video sequences. Previous works focus on extracting more discriminative features, but ignore distinguishing the key frames from the noisy frames. To tackle this problem, we propose a noise-robust dynamic facial expression recognition network (NR-DFERNet), which can effectively reduce the interference of noisy frames on the DFER task. Specifically, at the spatial stage, we devise a dynamic-static fusion module (DSF) that introduces dynamic features to static features for learning more discriminative spatial features. To suppress the impact of target irrelevant frames, we introduce a novel dynamic class token (DCT) for the transformer at the temporal stage. Moreover, we design a snippet-based filter (SF) at the decision stage to reduce the effect of too many neutral frames on non-neutral sequence classification. Extensive experimental results demonstrate that our NR-DFERNet outperforms the state-of-the-art methods on both the DFEW and AFEW benchmarks.
Conjunctive Queries With Self-Joins, Towards a Fine-Grained Complexity Analysis
Venue: arXiv cs.DB 2022
Year: 2022

Even though query evaluation is a fundamental task in databases, known classifications of conjunctive queries by their fine-grained complexity only apply to queries without self-joins. We study how self-joins affect enumeration complexity, with the aim of building upon the known results to achieve general classifications. We do this by examining the extension of two known dichotomies: one with respect to linear delay, and one with respect to constant delay after linear preprocessing. As this turns out to be an intricate investigation, this paper is structured as an example-driven discussion that initiates this analysis. We show enumeration algorithms that rely on self-joins to efficiently evaluate queries that otherwise cannot be answered with the same guarantees. Due to these additional tractable cases, the hardness proofs are more complex than the self-join-free case. We show how to harness a known tagging technique to prove hardness of queries with self-joins. Our study offers sufficient conditions and necessary conditions for tractability and settles the cases of queries of low arity and queries with cyclic cores. Nevertheless, many cases remain open.
Spatial Cross-Attention Improves Self-Supervised Visual Representation Learning
Venue: arXiv cs.CV 2022
Year: 2022

Unsupervised representation learning methods like SwAV are proved to be effective in learning visual semantics of a target dataset. The main idea behind these methods is that different views of a same image represent the same semantics. In this paper, we further introduce an add-on module to facilitate the injection of the knowledge accounting for spatial cross correlations among the samples. This in turn results in distilling intra-class information including feature level locations and cross similarities between same-class instances. The proposed add-on can be added to existing methods such as the SwAV. We can later remove the add-on module for inference without any modification of the learned weights. Through an extensive set of empirical evaluations, we verify that our method yields an improved performance in detecting the class activation maps, top-1 classification accuracy, and down-stream tasks such as object detection, with different configuration settings.
Social Network Structure Shapes Innovation: Experience-sharing in RL with SAPIENS
Venue: arXiv cs.AI 2022
Year: 2022

Human culture relies on innovation: our ability to continuously explore how existing elements can be combined to create new ones. Innovation is not solitary, it relies on collective search and accumulation. Reinforcement learning (RL) approaches commonly assume that fully-connected groups are best suited for innovation. However, human laboratory and field studies have shown that hierarchical innovation is more robustly achieved by dynamic social network structures. In dynamic settings, humans oscillate between innovating individually or in small clusters, and then sharing outcomes with others. To our knowledge, the role of social network structure on innovation has not been systematically studied in RL. Here, we use a multi-level problem setting (WordCraft), with three different innovation tasks to test the hypothesis that the social network structure affects the performance of distributed RL algorithms. We systematically design networks of DQNs sharing experiences from their replay buffers in varying structures (fully-connected, small world, dynamic, ring) and introduce a set of behavioral and mnemonic metrics that extend the classical reward-focused evaluation framework of RL. Comparing the level of innovation achieved by different social network structures across different tasks shows that, first, consistent with human findings, experience sharing within a dynamic structure achieves the highest level of innovation in tasks with a deceptive nature and large search spaces. Second, experience sharing is not as helpful when there is a single clear path to innovation. Third, the metrics we propose, can help understand the success of different social network structures on different tasks, with the diversity of experiences on an individual and group level lending crucial insights.
Diffeomorphic Counterfactuals with Generative Models
Venue: arXiv cs.LG 2022
Year: 2022

Counterfactuals can explain classification decisions of neural networks in a human interpretable way. We propose a simple but effective method to generate such counterfactuals. More specifically, we perform a suitable diffeomorphic coordinate transformation and then perform gradient ascent in these coordinates to find counterfactuals which are classified with great confidence as a specified target class. We propose two methods to leverage generative models to construct such suitable coordinate systems that are either exactly or approximately diffeomorphic. We analyze the generation process theoretically using Riemannian differential geometry and validate the quality of the generated counterfactuals using various qualitative and quantitative measures.
Deep Multi-Agent Reinforcement Learning with Hybrid Action Spaces based on Maximum Entropy
Venue: arXiv cs.LG 2022
Year: 2022

Multi-agent deep reinforcement learning has been applied to address a variety of complex problems with either discrete or continuous action spaces and achieved great success. However, most real-world environments cannot be described by only discrete action spaces or only continuous action spaces. And there are few works having ever utilized deep reinforcement learning (drl) to multi-agent problems with hybrid action spaces. Therefore, we propose a novel algorithm: Deep Multi-Agent Hybrid Soft Actor-Critic (MAHSAC) to fill this gap. This algorithm follows the centralized training but decentralized execution (CTDE) paradigm, and extend the Soft Actor-Critic algorithm (SAC) to handle hybrid action space problems in Multi-Agent environments based on maximum entropy. Our experiences are running on an easy multi-agent particle world with a continuous observation and discrete action space, along with some basic simulated physics. The experimental results show that MAHSAC has good performance in training speed, stability, and anti-interference ability. At the same time, it outperforms existing independent deep hybrid learning method in cooperative scenarios and competitive scenarios.
A Proof of the Tree of Shapes in n-D
Venue: arXiv cs.DM 2022
Year: 2022

In this paper, we prove that the self-dual morphological hierarchical structure computed on a n-D gray-level wellcomposed image u by the algorithm of G{é}raud et al. [1] is exactly the mathematical structure defined to be the tree of shape of u in Najman et al [2]. We recall that this algorithm is in quasi-linear time and thus considered to be optimal. The tree of shapes leads to many applications in mathematical morphology and in image processing like grain filtering, shapings, image segmentation, and so on.
Bayesian calibration of coupled computational mechanics models under uncertainty based on interface deformation
Venue: arXiv cs.CE 2022
Year: 2022

Calibration or parameter identification is used with computational mechanics models related to observed data of the modeled process to find model parameters such that good similarity between model prediction and observation is achieved. We present a Bayesian calibration approach for surface coupled problems in computational mechanics based on measured deformation of an interface when no displacement data of material points is available. The interpretation of such a calibration problem as a statistical inference problem, in contrast to deterministic model calibration, is computationally more robust and allows the analyst to find a posterior distribution over possible solutions rather than a single point estimate. The proposed framework also enables the consideration of unavoidable uncertainties that are present in every experiment and are expected to play an important role in the model calibration process. To mitigate the computational costs of expensive forward model evaluations, we propose to learn the log-likelihood function from a controllable amount of parallel simulation runs using Gaussian process regression. We introduce and specifically study the effect of three different discrepancy measures for deformed interfaces between reference data and simulation. We show that a statistically based discrepancy measure results in the most expressive posterior distribution. We further apply the approach to numerical examples in higher model parameter dimensions and interpret the resulting posterior under uncertainty. In the examples, we investigate coupled multi-physics models of fluid-structure interaction effects in biofilms and find that the model parameters affect the results in a coupled manner.
Lightweight Conditional Model Extrapolation for Streaming Data under Class-Prior Shift
Venue: arXiv cs.LG 2022
Year: 2022

We introduce LIMES, a new method for learning with non-stationary streaming data, inspired by the recent success of meta-learning. The main idea is not to attempt to learn a single classifier that would have to work well across all occurring data distributions, nor many separate classifiers, but to exploit a hybrid strategy: we learn a single set of model parameters from which a specific classifier for any specific data distribution is derived via classifier adaptation. Assuming a multi-class classification setting with class-prior shift, the adaptation step can be performed analytically with only the classifier's bias terms being affected. Another contribution of our work is an extrapolation step that predicts suitable adaptation parameters for future time steps based on the previous data. In combination, we obtain a lightweight procedure for learning from streaming data with varying class distribution that adds no trainable parameters and almost no memory or computational overhead compared to training a single model. Experiments on a set of exemplary tasks using Twitter data show that LIMES achieves higher accuracy than alternative approaches, especially with respect to the relevant real-world metric of lowest within-day accuracy.
Nominal Metaphor Generation with Multitask Learning
Venue: arXiv cs.CL 2022
Year: 2022

Metaphor generation is a challenging task which can impact many downstream tasks such as improving user satisfaction with dialogue systems and story generation. This paper tackles the problem of Chinese nominal metaphor generation by introducing a multitask metaphor generation framework with self-training and metaphor identification mechanisms. Self-training addresses the data scarcity issue of metaphor datasets. That is, instead of solely relying on labelled metaphor datasets which are usually small in size, self-training helps identify potential metaphors from a large-scale unlabelled corpus for metaphor generation. The metaphor weighting mechanism enables our model to focus on the metaphor-related parts of the input (e.g., the comparison of the metaphor and comparator) during model learning and thus improves the metaphoricity of the generated metaphors. Our model is trained on an annotated corpus consisting of 6.3k sentences that contain diverse metaphorical expressions. Experimental results show that our model is able to generate metaphors with better readability and creativity compared to the baseline models, even in the situation where training data is insufficient.
EigenFairing: 3D Model Fairing using Image Coherence
Venue: arXiv cs.CV 2022
Year: 2022

A surface is often modeled as a triangulated mesh of 3D points and textures associated with faces of the mesh. The 3D points could be either sampled from range data or derived from a set of images using a stereo or Structure-from-Motion algorithm. When the points do not lie at critical points of maximum curvature or discontinuities of the real surface, faces of the mesh do not lie close to the modeled surface. This results in textural artifacts, and the model is not perfectly coherent with a set of actual images -- the ones that are used to texture-map its mesh. This paper presents a technique for perfecting the 3D surface model by repositioning its vertices so that it is coherent with a set of observed images of the object. The textural artifacts and incoherence with images are due to the non-planarity of a surface patch being approximated by a planar face, as observed from multiple viewpoints. Image areas from the viewpoints are used to represent texture for the patch in Eigenspace. The Eigenspace representation captures variations of texture, which we seek to minimize. A coherence measure based on the difference between the face textures reconstructed from Eigenspace and the actual images is used to reposition the vertices so that the model is improved or faired. We refer to this technique of model refinement as EigenFairing, by which the model is faired, both geometrically and texturally, to better approximate the real surface.
Object Detection, Recognition, Deep Learning, and the Universal Law of Generalization
Venue: arXiv cs.LG 2022
Year: 2022

Object detection and recognition are fundamental functions underlying the success of species. Because the appearance of an object exhibits a large variability, the brain has to group these different stimuli under the same object identity, a process of generalization. Does the process of generalization follow some general principles or is it an ad-hoc "bag-of-tricks"? The Universal Law of Generalization provided evidence that generalization follows similar properties across a variety of species and tasks. Here we test the hypothesis that the internal representations underlying generalization reflect the natural properties of object detection and recognition in our environment rather than the specifics of the system solving these problems. By training a deep-neural-network with images of "clear" and "camouflaged" animals, we found that with a proper choice of category prototypes, the generalization functions are monotone decreasing, similar to the generalization functions of biological systems. Our findings support the hypothesis of the study.