OpenRaiser/Intern-Atlas huggingface dataset:
Ego2HandsPose: A Dataset for Egocentric Two-hand 3D Global Pose Estimation
Venue: arXiv cs.CV 2022
Year: 2022

Color-based two-hand 3D pose estimation in the global coordinate system is essential in many applications. However, there are very few datasets dedicated to this task and no existing dataset supports estimation in a non-laboratory environment. This is largely attributed to the sophisticated data collection process required for 3D hand pose annotations, which also leads to difficulty in obtaining instances with the level of visual diversity needed for estimation in the wild. Progressing towards this goal, a large-scale dataset Ego2Hands was recently proposed to address the task of two-hand segmentation and detection in the wild. The proposed composition-based data generation technique can create two-hand instances with quality, quantity and diversity that generalize well to unseen domains. In this work, we present Ego2HandsPose, an extension of Ego2Hands that contains 3D hand pose annotation and is the first dataset that enables color-based two-hand 3D tracking in unseen domains. To this end, we develop a set of parametric fitting algorithms to enable 1) 3D hand pose annotation using a single image, 2) automatic conversion from 2D to 3D hand poses and 3) accurate two-hand tracking with temporal consistency. We provide incremental quantitative analysis on the multi-stage pipeline and show that training on our dataset achieves state-of-the-art results that significantly outperforms other datasets for the task of egocentric two-hand global 3D pose estimation.
Response to: Significance and stability of deep learning-based identification of subtypes within major psychiatric disorders. Molecular Psychiatry (2022)
Venue: arXiv cs.LG 2022
Year: 2022

Recently, Winter and Hahn [1] commented on our work on identifying subtypes of major psychiatry disorders (MPDs) based on neurobiological features using machine learning [2]. They questioned the generalizability of our methods and the statistical significance, stability, and overfitting of the results, and proposed a pipeline for disease subtyping. We appreciate their earnest consideration of our work, however, we need to point out their misconceptions of basic machine-learning concepts and delineate some key issues involved.
Semantic Technology based Usage Control for Decentralized Systems
Venue: arXiv cs.CR 2022
Year: 2022

The sharing of data and digital assets in a decentralized settling is associated with various legislative challenges, including, but not limited to, the need to adhere to legal requirements with respect to privacy (e.g. data protection legislation) and copyright (e.g. copyright legislation). In order to enable software platform providers to manage data and digital assets appropriately and to provide more control to data and digital asset owners, usage control technologies could be used to make sure that consumers handle data according to privacy preferences, licenses, regulatory requirements, among others. In this research proposal, we explore the application of usage control in decentralized environments. In particular, we address the challenges related to the specification of usage control policies, the enforcement of the respective policies, and the usability of the tools that are used to administer them.
Zero-Shot Audio Classification using Image Embeddings
Venue: arXiv cs.SD 2022
Year: 2022

Supervised learning methods can solve the given problem in the presence of a large set of labeled data. However, the acquisition of a dataset covering all the target classes typically requires manual labeling which is expensive and time-consuming. Zero-shot learning models are capable of classifying the unseen concepts by utilizing their semantic information. The present study introduces image embeddings as side information on zero-shot audio classification by using a nonlinear acoustic-semantic projection. We extract the semantic image representations from the Open Images dataset and evaluate the performance of the models on an audio subset of AudioSet using semantic information in different domains; image, audio, and textual. We demonstrate that the image embeddings can be used as semantic information to perform zero-shot audio classification. The experimental results show that the image and textual embeddings display similar performance both individually and together. We additionally calculate the semantic acoustic embeddings from the test samples to provide an upper limit to the performance. The results show that the classification performance is highly sensitive to the semantic relation between test and training classes and textual and image embeddings can reach up to the semantic acoustic embeddings when the seen and unseen classes are semantically similar.
An Open Framework to Model Diffraction by Dynamic Blockers in Millimeter Wave Simulations
Venue: arXiv cs.NI 2022
Year: 2022

The millimeter wave (mmWave) band will be exploited to address the growing demand for high data rates and low latency. The higher frequencies, however, are prone to limitations on the propagation of the signal in the environment. Thus, highly directional beamforming is needed to increase the antenna gain. Another crucial problem of the mmWave frequencies is their vulnerability to blockage by physical obstacles. To this aim, we studied the problem of modeling the impact of second-order effects on mmWave channels, specifically the susceptibility of the mmWave signals to physical blockers. With respect to existing works on this topic, our project focuses on scenarios where mmWaves interact with multiple, dynamic blockers. Our open source software includes diffraction-based blockage models and interfaces directly with an open source Radio Frequency (RF) ray-tracing software.
Going Beyond the Cookie Theft Picture Test: Detecting Cognitive Impairments using Acoustic Features
Venue: arXiv cs.SD 2022
Year: 2022

Standardized tests play a crucial role in the detection of cognitive impairment. Previous work demonstrated that automatic detection of cognitive impairment is possible using audio data from a standardized picture description task. The presented study goes beyond that, evaluating our methods on data taken from two standardized neuropsychological tests, namely the German SKT and a German version of the CERAD-NB, and a semi-structured clinical interview between a patient and a psychologist. For the tests, we focus on speech recordings of three sub-tests: reading numbers (SKT 3), interference (SKT 7), and verbal fluency (CERAD-NB 1). We show that acoustic features from standardized tests can be used to reliably discriminate cognitively impaired individuals from non-impaired ones. Furthermore, we provide evidence that even features extracted from random speech samples of the interview can be a discriminator of cognitive impairment. In our baseline experiments, we use OpenSMILE features and Support Vector Machine classifiers. In an improved setup, we show that using wav2vec 2.0 features instead, we can achieve an accuracy of up to 85%.
Divide (CPU Load) and Conquer: Semi-Flexible Cloud Resource Allocation
Venue: arXiv cs.DC 2022
Year: 2022

Cloud resource management is often modeled by two-dimensional bin packing with a set of items that correspond to tasks having fixed CPU and memory requirements. However, applications running in clouds are much more flexible: modern frameworks allow to (horizontally) scale a single application to dozens, even hundreds of instances; and then the load balancer can precisely divide the workload between them. We analyze a model that captures this (semi)-flexibility of cloud resource management. Each cloud application is characterized by its memory footprint and its momentary CPU load. Combining the scheduler and the load balancer, the resource manager decides how many instances of each application will be created and how the CPU load will be balanced between them. In contrast to the divisible load model, each instance of the application requires a certain amount of memory, independent of the number of instances. Thus, the resource manager effectively trades additional memory for more evenly balanced load. We study two objectives: the bin-packing-like minimization of the number of machines used; and the makespan-like minimization of the maximum load among all the machines. We prove NP-hardness of the general problems, but also propose polynomial-time exact algorithms for boundary special cases. Notably, we show that (semi)-flexibility may result in reducing the required number of machines by a tight factor of $2-\varepsilon$. For the general case, we propose heuristics that we validate by simulation on instances derived from the Azure trace.
Image Generation with Multimodal Priors using Denoising Diffusion Probabilistic Models
Venue: arXiv cs.CV 2022
Year: 2022

Image synthesis under multi-modal priors is a useful and challenging task that has received increasing attention in recent years. A major challenge in using generative models to accomplish this task is the lack of paired data containing all modalities (i.e. priors) and corresponding outputs. In recent work, a variational auto-encoder (VAE) model was trained in a weakly supervised manner to address this challenge. Since the generative power of VAEs is usually limited, it is difficult for this method to synthesize images belonging to complex distributions. To this end, we propose a solution based on a denoising diffusion probabilistic models to synthesise images under multi-model priors. Based on the fact that the distribution over each time step in the diffusion model is Gaussian, in this work we show that there exists a closed-form expression to the generate the image corresponds to the given modalities. The proposed solution does not require explicit retraining for all modalities and can leverage the outputs of individual modalities to generate realistic images according to different constraints. We conduct studies on two real-world datasets to demonstrate the effectiveness of our approach
Glyph from Icon -- Automated Generation of Metaphoric Glyphs
Venue: arXiv cs.HC 2022
Year: 2022

Metaphoric glyphs enhance the readability and learnability of abstract glyphs used for the visualization of quantitative multidimensional data by building upon graphical entities that are intuitively related to the underlying problem domain. Their construction is, however, a predominantly manual process. In this paper, we introduce the Glyph-from-Icon (GfI) approach that allows the automated generation of metaphoric glyphs from user specified icons. Our approach modifies the icon's visual appearance using up to seven quantifiable visual variables, three of which manipulate its geometry while four affect its color. Depending on the visualization goal, specific combinations of these visual variables define the glyphs's variables used for data encoding. Technically, we propose a diffusion-curve based parametric icon representation, which comprises the degrees-of-freedom related to the geometric and color-based visual variables. Moreover, we extend our GfI approach to achieve scalability of the generated glyphs. Based on a user study we evaluate the perception of the glyph's main variables, i.e., amplitude and frequency of geometric and color modulation, as function of the stimuli and deduce functional relations as well as quantization levels to achieve perceptual monotonicity and readability. Finally, we propose a robustly perceivable combination of visual variables, which we apply to the visualization of COVID-19 data.
A Fine Line: Total Least-Squares Line Fitting as QCQP Optimization
Venue: arXiv cs.RO 2022
Year: 2022

This note uses the Total Least-Squares (TLS) line-fitting problem as a canvas to explore some modern optimization tools. The contribution is meant to be tutorial in nature. The TLS problem has a lot of mathematical similarities to important problems in robotics and computer vision but is easier to visualize and understand. We demonstrate how to turn this problem into a Quadratically Constrained Quadratic Program (QCQP) so that it can be cast either as an eigenproblem or a Semi-Definite Program (SDP). We then turn to the more challenging situation where a Geman-McClure cost function and M-estimation are used to reject outlier datapoints. Using Black-Rangarajan duality, we show this can also be cast as a QCQP and solved as an SDP; however, with a lot of data the SDP can be slow and as such we show how we can construct a certificate of optimality for a faster method such as Iteratively Reweighted Least-Squares (IRLS).
REKnow: Enhanced Knowledge for Joint Entity and Relation Extraction
Venue: arXiv cs.CL 2022
Year: 2022

Relation extraction is an important but challenging task that aims to extract all hidden relational facts from the text. With the development of deep language models, relation extraction methods have achieved good performance on various benchmarks. However, we observe two shortcomings of previous methods: first, there is no unified framework that works well under various relation extraction settings; second, effectively utilizing external knowledge as background information is absent. In this work, we propose a knowledge-enhanced generative model to mitigate these two issues. Our generative model is a unified framework to sequentially generate relational triplets under various relation extraction settings and explicitly utilizes relevant knowledge from Knowledge Graph (KG) to resolve ambiguities. Our model achieves superior performance on multiple benchmarks and settings, including WebNLG, NYT10, and TACRED.
Global Internet public peering capacity of interconnection: a complex network analysis
Venue: arXiv cs.NI 2022
Year: 2022

A massive and growing part of Autonomous System (AS)-level traffic exchanges takes place at Internet Exchange Points (IXPs). This paper leverages PeeringDB, a database providing a partial but reasonable view of the global interconnection of ASes at IXPs, to model a complex graph enabling the characterization of the key Internet peering players and their interactions over time. We model a PeeringDB snapshot as a weighted directed bipartite graph, called the pDB c-graph, that captures the port size ASes possess at IXPs using available metadata. This novel model of the Internet is shown to picture relevant features of a complex network that groups ASes and IXPs in geographical areas of influence. From this model, we extract central players of public peering such as hypergiant AS content providers and major regional traffic receivers. Most importantly, this graph model opens the way to apply spectral analysis using reduced Google matrix in order to retrieve the intensity of possible interactions between ASes on the basis of pure connectivity information. As an illustration, we retrieve the timely evolution of the peering network to show how the central content and cloud providers have increased their reach to eyeball networks during Covid-19 pandemic.
An Image Processing Pipeline for Camera Trap Time-Lapse Recordings
Venue: arXiv cs.CV 2022
Year: 2022

A new open-source image processing pipeline for analyzing camera trap time-lapse recordings is described. This pipeline includes machine learning models to assist human-in-the-loop video segmentation and animal re-identification. We present some performance results and observations on the utility of this pipeline after using it in a year-long project studying the spatial ecology and social behavior of the gopher tortoise.
Learning the Space of Deep Models
Venue: arXiv cs.CV 2022
Year: 2022

Embedding of large but redundant data, such as images or text, in a hierarchy of lower-dimensional spaces is one of the key features of representation learning approaches, which nowadays provide state-of-the-art solutions to problems once believed hard or impossible to solve. In this work, in a plot twist with a strong meta aftertaste, we show how trained deep models are as redundant as the data they are optimized to process, and how it is therefore possible to use deep learning models to embed deep learning models. In particular, we show that it is possible to use representation learning to learn a fixed-size, low-dimensional embedding space of trained deep models and that such space can be explored by interpolation or optimization to attain ready-to-use models. We find that it is possible to learn an embedding space of multiple instances of the same architecture and of multiple architectures. We address image classification and neural representation of signals, showing how our embedding space can be learnt so as to capture the notions of performance and 3D shape, respectively. In the Multi-Architecture setting we also show how an embedding trained only on a subset of architectures can learn to generate already-trained instances of architectures it never sees instantiated at training time.
A new distance measurement and its application in K-Means Algorithm
Venue: arXiv cs.LG 2022
Year: 2022

K-Means clustering algorithm is one of the most commonly used clustering algorithms because of its simplicity and efficiency. K-Means clustering algorithm based on Euclidean distance only pays attention to the linear distance between samples, but ignores the overall distribution structure of the dataset (i.e. the fluid structure of dataset). Since it is difficult to describe the internal structure of two data points by Euclidean distance in high-dimensional data space, we propose a new distance measurement, namely, view-distance, and apply it to the K-Means algorithm. On the classical manifold learning datasets, S-curve and Swiss roll datasets, not only this new distance can cluster the data according to the structure of the data itself, but also the boundaries between categories are neat dividing lines. Moreover, we also tested the classification accuracy and clustering effect of the K-Means algorithm based on view-distance on some real-world datasets. The experimental results show that, on most datasets, the K-Means algorithm based on view-distance has a certain degree of improvement in classification accuracy and clustering effect.
CONTINUER: Maintaining Distributed DNN Services During Edge Failures
Venue: arXiv cs.DC 2022
Year: 2022

Partitioning and deploying Deep Neural Networks (DNNs) across edge nodes may be used to meet performance objectives of applications. However, the failure of a single node may result in cascading failures that will adversely impact the delivery of the service and will result in failure to meet specific objectives. The impact of these failures needs to be minimised at runtime. Three techniques are explored in this paper, namely repartitioning, early-exit and skip-connection. When an edge node fails, the repartitioning technique will repartition and redeploy the DNN thus avoiding the failed nodes. The early-exit technique makes provision for a request to exit (early) before the failed node. The skip connection technique dynamically routes the request by skipping the failed nodes. This paper will leverage trade-offs in accuracy, end-to-end latency and downtime for selecting the best technique given user-defined objectives (accuracy, latency and downtime thresholds) when an edge node fails. To this end, CONTINUER is developed. Two key activities of the framework are estimating the accuracy and latency when using the techniques for distributed DNNs and selecting the best technique. It is demonstrated on a lab-based experimental testbed that CONTINUER estimates accuracy and latency when using the techniques with no more than an average error of 0.28% and 13.06%, respectively and selects the suitable technique with a low overhead of no more than 16.82 milliseconds and an accuracy of up to 99.86%.
Spatial-temporal Concept based Explanation of 3D ConvNets
Venue: arXiv cs.CV 2022
Year: 2022

Recent studies have achieved outstanding success in explaining 2D image recognition ConvNets. On the other hand, due to the computation cost and complexity of video data, the explanation of 3D video recognition ConvNets is relatively less studied. In this paper, we present a 3D ACE (Automatic Concept-based Explanation) framework for interpreting 3D ConvNets. In our approach: (1) videos are represented using high-level supervoxels, which is straightforward for human to understand; and (2) the interpreting framework estimates a score for each voxel, which reflects its importance in the decision procedure. Experiments show that our method can discover spatial-temporal concepts of different importance-levels, and thus can explore the influence of the concepts on a target task, such as action classification, in-depth. The codes are publicly available.
Less Is More: Linear Layers on CLIP Features as Powerful VizWiz Model
Venue: arXiv cs.CV 2022
Year: 2022

Current architectures for multi-modality tasks such as visual question answering suffer from their high complexity. As a result, these architectures are difficult to train and require high computational resources. To address these problems we present a CLIP-based architecture that does not require any fine-tuning of the feature extractors. A simple linear classifier is used on the concatenated features of the image and text encoder. During training an auxiliary loss is added which operates on the answer types. The resulting classification is then used as an attention gate on the answer class selection. On the VizWiz 2022 Visual Question Answering Challenge we achieve 60.15 % accuracy on Task 1: Predict Answer to a Visual Question and AP score of 83.78 % on Task 2: Predict Answerability of a Visual Question.
Cross-TOP: Zero-Shot Cross-Schema Task-Oriented Parsing
Venue: arXiv cs.CL 2022
Year: 2022

Deep learning methods have enabled task-oriented semantic parsing of increasingly complex utterances. However, a single model is still typically trained and deployed for each task separately, requiring labeled training data for each, which makes it challenging to support new tasks, even within a single business vertical (e.g., food-ordering or travel booking). In this paper we describe Cross-TOP (Cross-Schema Task-Oriented Parsing), a zero-shot method for complex semantic parsing in a given vertical. By leveraging the fact that user requests from the same vertical share lexical and semantic similarities, a single cross-schema parser is trained to service an arbitrary number of tasks, seen or unseen, within a vertical. We show that Cross-TOP can achieve high accuracy on a previously unseen task without requiring any additional training data, thereby providing a scalable way to bootstrap semantic parsers for new tasks. As part of this work we release the FoodOrdering dataset, a task-oriented parsing dataset in the food-ordering vertical, with utterances and annotations derived from five schemas, each from a different restaurant menu.
Research Software Publication Policy Case Study
Venue: arXiv cs.DL 2022
Year: 2022

Research software is increasingly recognized as a vital component of the scholarly record. Journals offer authors the opportunity to publish research software papers, but often have different requirements for how these publications should be structured and how code should be verified. In this short case study we gather data from 20 Physical Science journals to trace the frequency, quality control, and publishing criteria for software papers. Our goal with the case study is to provide a proof-of-concept for doing descriptive empirical work with software publication policies across numerous domains of science and engineering. In the narrative we therefore provide descriptive statistics showing how these journals differ in criteria required for archiving, linking, verifying, and documenting software as part of a formal publication. The contribution of this preliminary work is twofold: 1. We provide case study of Physical Science research software publications over time; 2. We demonstrate the use of a new survey method for analyzing research software publication policies. In our conclusion, we describe how comparative research into software publication policies can provide better criteria and requirements for an emerging software publication landscape.