OpenRaiser/Intern-Atlas huggingface dataset:
LocFormer: Enabling Transformers to Perform Temporal Moment Localization on Long Untrimmed Videos With a Feature Sampling Approach
Venue: arXiv cs.CV 2021
Year: 2021

We propose LocFormer, a Transformer-based model for video grounding which operates at a constant memory footprint regardless of the video length, i.e. number of frames. LocFormer is designed for tasks where it is necessary to process the entire long video and at its core lie two main contributions. First, our model incorporates a new sampling technique that splits the input feature sequence into a fixed number of sections and selects a single feature per section using a stochastic approach, which allows us to obtain a feature sample set that is representative of the video content for the task at hand while keeping the memory footprint constant. Second, we propose a modular design that separates functionality, enabling us to learn an inductive bias via supervising the self-attention heads, while also effectively leveraging pre-trained text and video encoders. We test our proposals on relevant benchmark datasets for video grounding, showing that not only LocFormer can achieve excellent results including state-of-the-art performance on YouCookII, but also that our sampling technique is more effective than competing counterparts and that it consistently improves the performance of prior work, by up to 3.13\% in the mean temporal IoU, ultimately leading to a new state-of-the-art performance on Charades-STA.
Managing dataset shift by adversarial validation for credit scoring
Venue: arXiv cs.LG 2021
Year: 2021

Dataset shift is common in credit scoring scenarios, and the inconsistency between the distribution of training data and the data that actually needs to be predicted is likely to cause poor model performance. However, most of the current studies do not take this into account, and they directly mix data from different time periods when training the models. This brings about two problems. Firstly, there is a risk of data leakage, i.e., using future data to predict the past. This can result in inflated results in offline validation, but unsatisfactory results in practical applications. Secondly, the macroeconomic environment and risk control strategies are likely to be different in different time periods, and the behavior patterns of borrowers may also change. The model trained with past data may not be applicable to the recent stage. Therefore, we propose a method based on adversarial validation to alleviate the dataset shift problem in credit scoring scenarios. In this method, partial training set samples with the closest distribution to the predicted data are selected for cross-validation by adversarial validation to ensure the generalization performance of the trained model on the predicted samples. In addition, through a simple splicing method, samples in the training data that are inconsistent with the test data distribution are also involved in the training process of cross-validation, which makes full use of all the data and further improves the model performance. To verify the effectiveness of the proposed method, comparative experiments with several other data split methods are conducted with the data provided by Lending Club. The experimental results demonstrate the importance of dataset shift in the field of credit scoring and the superiority of the proposed method.
MoCaNet: Motion Retargeting in-the-wild via Canonicalization Networks
Venue: arXiv cs.CV 2021
Year: 2021

We present a novel framework that brings the 3D motion retargeting task from controlled environments to in-the-wild scenarios. In particular, our method is capable of retargeting body motion from a character in a 2D monocular video to a 3D character without using any motion capture system or 3D reconstruction procedure. It is designed to leverage massive online videos for unsupervised training, needless of 3D annotations or motion-body pairing information. The proposed method is built upon two novel canonicalization operations, structure canonicalization and view canonicalization. Trained with the canonicalization operations and the derived regularizations, our method learns to factorize a skeleton sequence into three independent semantic subspaces, i.e., motion, structure, and view angle. The disentangled representation enables motion retargeting from 2D to 3D with high precision. Our method achieves superior performance on motion transfer benchmarks with large body variations and challenging actions. Notably, the canonicalized skeleton sequence could serve as a disentangled and interpretable representation of human motion that benefits action analysis and motion retrieval.
Evaluating System Identification Methods for Predicting Thermal Dissipation of Heterogeneous SoCs
Venue: arXiv cs.LG 2021
Year: 2021

In this paper we evaluate the use of system identification methods to build a thermal prediction model of heterogeneous SoC platforms that can be used to quickly predict the temperature of different configurations without the need of hardware. Specifically, we focus on modeling approaches that can predict the temperature based on the clock frequency and the utilization percentage of each core. We investigate three methods with respect to their prediction accuracy: a linear state-space identification approach using polynomial regressors, a NARX neural network approach and a recurrent neural network approach configured in an FIR model structure. We evaluate the methods on an Odroid-XU4 board featuring an Exynos 5422 SoC. The results show that the model based on polynomial regressors significantly outperformed the other two models when trained with 1 hour and 6 hours of data.
Blockchain-based Platform for Secure Sharing and Validation of Vaccination Certificates
Venue: arXiv cs.CR 2021
Year: 2021

The COVID-19 pandemic has recently emerged as a worldwide health emergency that necessitates coordinated international measures. To contain the virus's spread, governments and health organisations raced to develop vaccines that would lower Covid-19 morbidity, relieve pressure on healthcare systems, and allow economies to open. As a way forward after the COVID-19 vaccination, the Vaccination certificate has been adopted to help the authorities formulate policies by controlling cross-border travelling. To resolve significant privacy concerns and remove the need for relying on third parties to maintain trust and control the user's data, in this paper, we leverage blockchain technologies in developing a secure and verifiable vaccination certificate. Our approach has the advantage of utilising a hybrid architecture that implements different advanced technologies, such as smart contracts, interPlanetary File System (IPFS), and Self-sovereign Identity (SSI). We will rely on verifiable credentials paired with smart contracts to implement on-chain access control decisions and provide on-chain verification and validation of the user and issuer DIDs. The usability of this approach was further analysed, particularly concerning performance and security. Our analysis proved that our approach satisfies vaccination certificate security requirements.
Driver Drowsiness Detection Using Ensemble Convolutional Neural Networks on YawDD
Venue: arXiv cs.CV 2021
Year: 2021

Driver drowsiness detection using videos/images is one of the most essential areas in today's time for driver safety. The development of deep learning techniques, notably Convolutional Neural Networks (CNN), applied in computer vision applications such as drowsiness detection, has shown promising results due to the tremendous increase in technology in the recent few decades. Eyes that are closed or blinking excessively, yawning, nodding, and occlusion are all key aspects of drowsiness. In this work, we have applied four different Convolutional Neural Network (CNN) techniques on the YawDD dataset to detect and examine the extent of drowsiness depending on the yawning frequency with specific pose and occlusion variation. Preliminary computational results show that our proposed Ensemble Convolutional Neural Network (ECNN) outperformed the traditional CNN-based approach by achieving an F1 score of 0.935, whereas the other three CNN, such as CNN1, CNN2, and CNN3 approaches gained 0.92, 0.90, and 0.912 F1 scores, respectively.
Deriving Semantics-Aware Fuzzers from Web API Schemas
Venue: arXiv cs.CR 2021
Year: 2021

Fuzzing -- whether generating or mutating inputs -- has found many bugs and security vulnerabilities in a wide range of domains. Stateful and highly structured web APIs present significant challenges to traditional fuzzing techniques, as execution feedback is usually limited to a response code instead of code coverage and vulnerabilities of interest include silent information-disclosure in addition to explicit errors. Our tool, Schemathesis, derives structure- and semantics-aware fuzzers from web API schemas in the OpenAPI or GraphQL formats, using property-based testing tools. Derived fuzzers can be incorporated into unit-test suites or run directly, with or without end-user customisation of data generation and semantic checks. We construct the most comprehensive evaluation of web API fuzzers to date, running eight fuzzers against sixteen real-world open source web services. OpenAPI schemas found in the wild have a long tail of rare features and complex structures. Of the tools we evaluated, Schemathesis was the only one to handle more than two-thirds of our target services without a fatal internal error. Schemathesis finds 1.4 times to 4.5 times more unique defects than the respectively second-best fuzzer for each target, and is the only fuzzer to find defects in four targets.
Long term behavior of dynamic equilibria in fluid queuing networks
Venue: arXiv cs.GT 2021
Year: 2021

A fluid queuing network constitutes one of the simplest models in which to study flow dynamics over a network. In this model we have a single source-sink pair and each link has a per-time-unit capacity and a transit time. A dynamic equilibrium (or equilibrium flow over time) is a flow pattern over time such that no flow particle has incentives to unilaterally change its path. Although the model has been around for almost fifty years, only recently results regarding existence and characterization of equilibria have been obtained. In particular the long term behavior remains poorly understood. Our main result in this paper is to show that, under a natural (and obviously necessary) condition on the queuing capacity, a dynamic equilibrium reaches a steady state (after which queue lengths remain constant) in finite time. Previously, it was not even known that queue lengths would remain bounded. The proof is based on the analysis of a rather non-obvious potential function that turns out to be monotone along the evolution of the equilibrium. Furthermore, we show that the steady state is characterized as an optimal solution of a certain linear program. When this program has a unique solution, which occurs generically, the long term behavior is completely predictable. On the contrary, if the linear program has multiple solutions the steady state is more difficult to identify as it depends on the whole temporal evolution of the equilibrium.
Concurrence for well-formed CAFs: Naive Semantics
Venue: arXiv cs.LO 2021
Year: 2021

In the area of claim-based reasoning in abstract argumentation, a claim-based semantics is said to be concurrent in a given framework if all its variants yield the same extensions. In this note, we show that the concurrence problem with respect to naive semantics is coNP-hard for well-formed CAFs. This solves a problem that has been left open in Dvorák et al. (2021).
Fusion and Orthogonal Projection for Improved Face-Voice Association
Venue: arXiv cs.CV 2021
Year: 2021

We study the problem of learning association between face and voice, which is gaining interest in the computer vision community lately. Prior works adopt pairwise or triplet loss formulations to learn an embedding space amenable for associated matching and verification tasks. Albeit showing some progress, such loss formulations are, however, restrictive due to dependency on distance-dependent margin parameter, poor run-time training complexity, and reliance on carefully crafted negative mining procedures. In this work, we hypothesize that enriched feature representation coupled with an effective yet efficient supervision is necessary in realizing a discriminative joint embedding space for improved face-voice association. To this end, we propose a light-weight, plug-and-play mechanism that exploits the complementary cues in both modalities to form enriched fused embeddings and clusters them based on their identity labels via orthogonality constraints. We coin our proposed mechanism as fusion and orthogonal projection (FOP) and instantiate in a two-stream pipeline. The overall resulting framework is evaluated on a large-scale VoxCeleb dataset with a multitude of tasks, including cross-modal verification and matching. Results show that our method performs favourably against the current state-of-the-art methods and our proposed supervision formulation is more effective and efficient than the ones employed by the contemporary methods.
Scale-Net: Learning to Reduce Scale Differences for Large-Scale Invariant Image Matching
Venue: arXiv cs.CV 2021
Year: 2021

Most image matching methods perform poorly when encountering large scale changes in images. To solve this problem, firstly, we propose a scale-difference-aware image matching method (SDAIM) that reduces image scale differences before local feature extraction, via resizing both images of an image pair according to an estimated scale ratio. Secondly, in order to accurately estimate the scale ratio, we propose a covisibility-attention-reinforced matching module (CVARM) and then design a novel neural network, termed as Scale-Net, based on CVARM. The proposed CVARM can lay more stress on covisible areas within the image pair and suppress the distraction from those areas visible in only one image. Quantitative and qualitative experiments confirm that the proposed Scale-Net has higher scale ratio estimation accuracy and much better generalization ability compared with all the existing scale ratio estimation methods. Further experiments on image matching and relative pose estimation tasks demonstrate that our SDAIM and Scale-Net are able to greatly boost the performance of representative local features and state-of-the-art local feature matching methods.
Low-Complexity Resource Allocation for Dense Cellular Vehicle-to-Everything (C-V2X) Communications
Venue: arXiv cs.IT 2021
Year: 2021

Vehicular communications are the key enabler of traffic reduction and road safety improvement referred to as cellular vehicle-to-everything (C-V2X) communications. Considering the numerous transmitting entities in next generation cellular networks, most existing resource allocation algorithms are impractical or non-effective to ensure reliable C-V2X communications which lead to safe intelligent transportation systems. We study a centralized framework to develop a low-complexity, scalable, and practical resource allocation scheme for dense C-V2X communications. The NP-hard sum-rate maximization resource allocation problem is formulated as a mixed-integer non-linear non-convex optimization problem considering both cellular vehicular links (CVLs) and non-cellular VLs (NCVLs) quality-of-service (QoS) constraints. By assuming that multiple NCVLs can simultaneously reuse a single cellular link (CL), we propose two low-complexity sub-optimal matching-based algorithms in four steps. The first two steps provide a channel gain-based CVL priority and CL assignment followed by an innovative scalable min-max channel-gain-based CVL-NCVL matching. We propose an analytically proven closed-form fast feasibility check theorem as the third step. The objective function is transformed into a difference of convex (DC) form and the power allocation problem is solved optimally using majorization-minimization (MaMi) method and interior point methods as the last step. Numerical results verify that our schemes are scalable and effective for dense C-V2X communications. The low-complexity and practicality of the proposed schemes for dense cellular networks is also shown. Furthermore, it is shown that the proposed schemes outperform other methods up to %6 in terms of overall sum-rate in dense scenarios and have a near optimal performance.
Sample-Efficient Reinforcement Learning via Conservative Model-Based Actor-Critic
Venue: arXiv cs.LG 2021
Year: 2021

Model-based reinforcement learning algorithms, which aim to learn a model of the environment to make decisions, are more sample efficient than their model-free counterparts. The sample efficiency of model-based approaches relies on whether the model can well approximate the environment. However, learning an accurate model is challenging, especially in complex and noisy environments. To tackle this problem, we propose the conservative model-based actor-critic (CMBAC), a novel approach that achieves high sample efficiency without the strong reliance on accurate learned models. Specifically, CMBAC learns multiple estimates of the Q-value function from a set of inaccurate models and uses the average of the bottom-k estimates -- a conservative estimate -- to optimize the policy. An appealing feature of CMBAC is that the conservative estimates effectively encourage the agent to avoid unreliable "promising actions" -- whose values are high in only a small fraction of the models. Experiments demonstrate that CMBAC significantly outperforms state-of-the-art approaches in terms of sample efficiency on several challenging tasks, and the proposed method is more robust than previous methods in noisy environments.
Real-Time Optical Flow for Vehicular Perception with Low- and High-Resolution Event Cameras
Venue: arXiv cs.CV 2021
Year: 2021

Event cameras capture changes of illumination in the observed scene rather than accumulating light to create images. Thus, they allow for applications under high-speed motion and complex lighting conditions, where traditional framebased sensors show their limits with blur and over- or underexposed pixels. Thanks to these unique properties, they represent nowadays an highly attractive sensor for ITS-related applications. Event-based optical flow (EBOF) has been studied following the rise in popularity of these neuromorphic cameras. The recent arrival of high-definition neuromorphic sensors, however, challenges the existing approaches, because of the increased resolution of the events pixel array and a much higher throughput. As an answer to these points, we propose an optimized framework for computing optical flow in real-time with both low- and high-resolution event cameras. We formulate a novel dense representation for the sparse events flow, in the form of the "inverse exponential distance surface". It serves as an interim frame, designed for the use of proven, state-of-the-art frame-based optical flow computation methods. We evaluate our approach on both low- and high-resolution driving sequences, and show that it often achieves better results than the current state of the art, while also reaching higher frame rates, 250Hz at 346 x 260 pixels and 77Hz at 1280 x 720 pixels.
A Multi-user Oriented Live Free-viewpoint Video Streaming System Based On View Interpolation
Venue: arXiv cs.MM 2021
Year: 2021

As an important application form of immersive multimedia services, free-viewpoint video(FVV) enables users with great immersive experience by strong interaction. However, the computational complexity of virtual view synthesis algorithms poses a significant challenge to the real-time performance of an FVV system. Furthermore, the individuality of user interaction makes it difficult to serve multiple users simultaneously for a system with conventional architecture. In this paper, we novelly introduce a CNN-based view interpolation algorithm to synthesis dense virtual views in real time. Based on this, we also build an end-to-end live free-viewpoint system with a multi-user oriented streaming strategy. Our system can utilize a single edge server to serve multiple users at the same time without having to bring a large view synthesis load on the client side. We analyze the whole system and show that our approaches give the user a pleasant immersive experience, in terms of both visual quality and latency.
Turbo-Sim: a generalised generative model with a physical latent space
Venue: arXiv cs.LG 2021
Year: 2021

We present Turbo-Sim, a generalised autoencoder framework derived from principles of information theory that can be used as a generative model. By maximising the mutual information between the input and the output of both the encoder and the decoder, we are able to rediscover the loss terms usually found in adversarial autoencoders and generative adversarial networks, as well as various more sophisticated related models. Our generalised framework makes these models mathematically interpretable and allows for a diversity of new ones by setting the weight of each loss term separately. The framework is also independent of the intrinsic architecture of the encoder and the decoder thus leaving a wide choice for the building blocks of the whole network. We apply Turbo-Sim to a collider physics generation problem: the transformation of the properties of several particles from a theory space, right after the collision, to an observation space, right after the detection in an experiment.
Complete Decentralization of Distributed Data Storages Based on Blockchain Technology
Venue: arXiv cs.DC 2021
Year: 2021

The work presents a solution for completely decentralized data management systems in geographically distributed environments with administratively unrelated or loosely related user groups and in conditions of partial or complete lack of trust between them. The solution is based on the integration of blockchain technology, smart contracts and provenance metadata driven data management. Architecture, operation principles and algorithms developed provides fault-tolerant, safe and reliable management of provenance metadata, control of operations with data files, as well as resource access management in collaborative distributed computing systems. The latter refer to distributed systems formed by combining into a single pool of computer resources of various organizations (institutions) to work together in the framework of some project.
Learning Spatio-Temporal Specifications for Dynamical Systems
Venue: arXiv cs.LG 2021
Year: 2021

Learning dynamical systems properties from data provides important insights that help us understand such systems and mitigate undesired outcomes. In this work, we propose a framework for learning spatio-temporal (ST) properties as formal logic specifications from data. We introduce SVM-STL, an extension of Signal Signal Temporal Logic (STL), capable of specifying spatial and temporal properties of a wide range of dynamical systems that exhibit time-varying spatial patterns. Our framework utilizes machine learning techniques to learn SVM-STL specifications from system executions given by sequences of spatial patterns. We present methods to deal with both labeled and unlabeled data. In addition, given system requirements in the form of SVM-STL specifications, we provide an approach for parameter synthesis to find parameters that maximize the satisfaction of such specifications. Our learning framework and parameter synthesis approach are showcased in an example of a reaction-diffusion system.