OpenRaiser/Intern-Atlas huggingface dataset:
Removing Word-Level Spurious Alignment between Images and Pseudo-Captions in Unsupervised Image Captioning
Venue: arXiv cs.CL 2021
Year: 2021

Unsupervised image captioning is a challenging task that aims at generating captions without the supervision of image-sentence pairs, but only with images and sentences drawn from different sources and object labels detected from the images. In previous work, pseudo-captions, i.e., sentences that contain the detected object labels, were assigned to a given image. The focus of the previous work was on the alignment of input images and pseudo-captions at the sentence level. However, pseudo-captions contain many words that are irrelevant to a given image. In this work, we investigate the effect of removing mismatched words from image-sentence alignment to determine how they make this task difficult. We propose a simple gating mechanism that is trained to align image features with only the most reliable words in pseudo-captions: the detected object labels. The experimental results show that our proposed method outperforms the previous methods without introducing complex sentence-level learning objectives. Combined with the sentence-level alignment method of previous work, our method further improves its performance. These results confirm the importance of careful alignment in word-level details.
Does Publicity in the Science Press Drive Citations?
Venue: arXiv cs.DL 2021
Year: 2021

We study how publicity in the science press, in the form of highlighting, affects the citations of research papers. Using multiple linear regression, we quantify the citation advantage associated with several highlighting platforms for papers published in Physical Review Letters (PRL) from 2008-2018. We thus find that the strongest predictor of citation accrual is a Viewpoint in Physics magazine, followed by a Research Highlight in Nature, an Editors' Suggestion in PRL, and a Research Highlight in Nature Physics. A similar hierarchical pattern is found when we search for extreme, not average, citation accrual, in the form a paper being listed among the top-1% cited papers in physics by Clarivate Analytics. The citation advantage of each highlighting platform is stratified according to the degree of vetting for importance that the manuscript received during peer review. This implies that we can view highlighting platforms as predictors of citation accrual, with varying degrees of strength that mirror each platform's vetting level.
Motion-guided Non-local Spatial-Temporal Network for Video Crowd Counting
Venue: arXiv cs.CV 2021
Year: 2021

We study video crowd counting, which is to estimate the number of objects (people in this paper) in all the frames of a video sequence. Previous work on crowd counting is mostly on still images. There has been little work on how to properly extract and take advantage of the spatial-temporal correlation between neighboring frames in both short and long ranges to achieve high estimation accuracy for a video sequence. In this work, we propose Monet, a novel and highly accurate motion-guided non-local spatial-temporal network for video crowd counting. Monet first takes people flow (motion information) as guidance to coarsely segment the regions of pixels where a person may be. Given these regions, Monet then uses a non-local spatial-temporal network to extract spatial-temporally both short and long-range contextual information. The whole network is finally trained end-to-end with a fused loss to generate a high-quality density map. Noting the scarcity and low quality (in terms of resolution and scene diversity) of the publicly available video crowd datasets, we have collected and built a large-scale video crowd counting datasets, VidCrowd, to contribute to the community. VidCrowd contains 9,000 frames of high resolution (2560 x 1440), with 1,150,239 head annotations captured in different scenes, crowd density and lighting in two cities. We have conducted extensive experiments on the challenging VideoCrowd and two public video crowd counting datasets: UCSD and Mall. Our approach achieves substantially better performance in terms of MAE and MSE as compared with other state-of-the-art approaches.
Neural Ordinary Differential Equations for Data-Driven Reduced Order Modeling of Environmental Hydrodynamics
Venue: arXiv cs.LG 2021
Year: 2021

Model reduction for fluid flow simulation continues to be of great interest across a number of scientific and engineering fields. Here, we explore the use of Neural Ordinary Differential Equations, a recently introduced family of continuous-depth, differentiable networks (Chen et al 2018), as a way to propagate latent-space dynamics in reduced order models. We compare their behavior with two classical non-intrusive methods based on proper orthogonal decomposition and radial basis function interpolation as well as dynamic mode decomposition. The test problems we consider include incompressible flow around a cylinder as well as real-world applications of shallow water hydrodynamics in riverine and estuarine systems. Our findings indicate that Neural ODEs provide an elegant framework for stable and accurate evolution of latent-space dynamics with a promising potential of extrapolatory predictions. However, in order to facilitate their widespread adoption for large-scale systems, significant effort needs to be directed at accelerating their training times. This will enable a more comprehensive exploration of the hyperparameter space for building generalizable Neural ODE approximations over a wide range of system dynamics.
Joint QoS-aware and Cost-efficient Task Scheduling for Fog-Cloud Resources in a Volunteer Computing System
Venue: arXiv cs.DC 2021
Year: 2021

Volunteer computing is an Internet-based distributed computing system in which volunteers share their extra available resources to manage large-scale tasks. However, computing devices in a Volunteer Computing System (VCS) are highly dynamic and heterogeneous in terms of their processing power, monetary cost, and data transferring latency. To ensure both the high Quality of Service (QoS) and low cost for different requests, all of the available computing resources must be used efficiently. Task scheduling is an NP-hard problem that is considered one of the main critical challenges in a heterogeneous VCS. Due to this, in this paper, we design two task scheduling algorithms for VCSs, named Min-CCV and Min-V. The main goal of the proposed algorithms is jointly minimizing the computation, communication and delay violation cost for the Internet of Things (IoT) requests. Our extensive simulation results show that proposed algorithms are able to allocate tasks to volunteer fog/cloud resources more efficiently than the state-of-the-art. Specifically, our algorithms improve the deadline satisfaction task rates by around 99.5% and decrease the total cost between 15 to 53% in comparison with the genetic-based algorithm.
Filter Distribution Templates in Convolutional Networks for Image Classification Tasks
Venue: arXiv cs.CV 2021
Year: 2021

Neural network designers have reached progressive accuracy by increasing models depth, introducing new layer types and discovering new combinations of layers. A common element in many architectures is the distribution of the number of filters in each layer. Neural network models keep a pattern design of increasing filters in deeper layers such as those in LeNet, VGG, ResNet, MobileNet and even in automatic discovered architectures such as NASNet. It remains unknown if this pyramidal distribution of filters is the best for different tasks and constrains. In this work we present a series of modifications in the distribution of filters in four popular neural network models and their effects in accuracy and resource consumption. Results show that by applying this approach, some models improve up to 8.9% in accuracy showing reductions in parameters up to 54%.
A Temporal Logic for Asynchronous Hyperproperties
Venue: arXiv cs.LO 2021
Year: 2021

Hyperproperties are properties of computational systems that require more than one trace to evaluate, e.g., many information-flow security and concurrency requirements. Where a trace property defines a set of traces, a hyperproperty defines a set of sets of traces. The temporal logics HyperLTL and HyperCTL* have been proposed to express hyperproperties. However, their semantics are synchronous in the sense that all traces proceed at the same speed and are evaluated at the same position. This precludes the use of these logics to analyze systems whose traces can proceed at different speeds and allow that different traces take stuttering steps independently. To solve this problem in this paper, we propose an asynchronous variant of HyperLTL. On the negative side, we show that the model-checking problem for this variant is undecidable. On the positive side, we identify a decidable fragment which covers a rich set of formulas with practical applications. We also propose two model-checking algorithms that reduce our problem to the HyperLTL model-checking problem in the synchronous semantics.
Maneuver-Aware Pooling for Vehicle Trajectory Prediction
Venue: arXiv cs.CV 2021
Year: 2021

Autonomous vehicles should be able to predict the future states of its environment and respond appropriately. Specifically, predicting the behavior of surrounding human drivers is vital for such platforms to share the same road with humans. Behavior of each of the surrounding vehicles is governed by the motion of its neighbor vehicles. This paper focuses on predicting the behavior of the surrounding vehicles of an autonomous vehicle on highways. We are motivated by improving the prediction accuracy when a surrounding vehicle performs lane change and highway merging maneuvers. We propose a novel pooling strategy to capture the inter-dependencies between the neighbor vehicles. Depending solely on Euclidean trajectory representation, the existing pooling strategies do not model the context information of the maneuvers intended by a surrounding vehicle. In contrast, our pooling mechanism employs polar trajectory representation, vehicles orientation and radial velocity. This results in an implicitly maneuver-aware pooling operation. We incorporated the proposed pooling mechanism into a generative encoder-decoder model, and evaluated our method on the public NGSIM dataset. The results of maneuver-based trajectory predictions demonstrate the effectiveness of the proposed method compared with the state-of-the-art approaches. Our "Pooling Toolbox" code is available at https://github.com/m-hasan-n/pooling.
Connecting AI Learning and Blockchain Mining in 6G Systems
Venue: arXiv cs.DC 2021
Year: 2021

The sixth generation (6G) systems are generally recognized to be established on ubiquitous Artificial Intelligence (AI) and distributed ledger such as blockchain. However, the AI training demands tremendous computing resource, which is limited in most 6G devices. Meanwhile, miners in Proof-of-Work (PoW) based blockchains devote massive computing power to block mining, and are widely criticized for the waste of computation. To address this dilemma, we propose an Evolved-Proof-of-Work (E-PoW) consensus that can integrate the matrix computations, which are widely existed in AI training, into the process of brute-force searches in the block mining. Consequently, E-PoW can connect AI learning and block mining via the multiply used common computing resource. Experimental results show that E-PoW can salvage by up to 80 percent computing power from pure block mining for parallel AI training in 6G systems.
Fast Convex Quadratic Optimization Solvers with Adaptive Sketching-based Preconditioners
Venue: arXiv cs.LG 2021
Year: 2021

We consider least-squares problems with quadratic regularization and propose novel sketching-based iterative methods with an adaptive sketch size. The sketch size can be as small as the effective dimension of the data matrix to guarantee linear convergence. However, a major difficulty in choosing the sketch size in terms of the effective dimension lies in the fact that the latter is usually unknown in practice. Current sketching-based solvers for regularized least-squares fall short on addressing this issue. Our main contribution is to propose adaptive versions of standard sketching-based iterative solvers, namely, the iterative Hessian sketch and the preconditioned conjugate gradient method, that do not require a priori estimation of the effective dimension. We propose an adaptive mechanism to control the sketch size according to the progress made in each step of the iterative solver. If enough progress is not made, the sketch size increases to improve the convergence rate. We prove that the adaptive sketch size scales at most in terms of the effective dimension, and that our adaptive methods are guaranteed to converge linearly. Consequently, our adaptive methods improve the state-of-the-art complexity for solving dense, ill-conditioned least-squares problems. Importantly, we illustrate numerically on several synthetic and real datasets that our method is extremely efficient and is often significantly faster than standard least-squares solvers such as a direct factorization based solver, the conjugate gradient method and its preconditioned variants.
LIQA: Lifelong Blind Image Quality Assessment
Venue: arXiv cs.MM 2021
Year: 2021

Existing blind image quality assessment (BIQA) methods are mostly designed in a disposable way and cannot evolve with unseen distortions adaptively, which greatly limits the deployment and application of BIQA models in real-world scenarios. To address this problem, we propose a novel Lifelong blind Image Quality Assessment (LIQA) approach, targeting to achieve the lifelong learning of BIQA. Without accessing to previous training data, our proposed LIQA can not only learn new distortions, but also mitigate the catastrophic forgetting of seen distortions. Specifically, we adopt the Split-and-Merge distillation strategy to train a single-head network that makes task-agnostic predictions. In the split stage, we first employ a distortion-specific generator to obtain the pseudo features of each seen distortion. Then, we use an auxiliary multi-head regression network to generate the predicted quality of each seen distortion. In the merge stage, we replay the pseudo features paired with pseudo labels to distill the knowledge of multiple heads, which can build the final regressed single head. Experimental results demonstrate that the proposed LIQA method can handle the continuous shifts of different distortion types and even datasets. More importantly, our LIQA model can achieve stable performance even if the task sequence is long.
RECKONition: a NLP-based system for Industrial Accidents at Work Prevention
Venue: arXiv cs.CL 2021
Year: 2021

Extracting patterns and useful information from Natural Language datasets is a challenging task, especially when dealing with data written in a language different from English, like Italian. Machine and Deep Learning, together with Natural Language Processing (NLP) techniques have widely spread and improved lately, providing a plethora of useful methods to address both Supervised and Unsupervised problems on textual information. We propose RECKONition, a NLP-based system for Industrial Accidents at Work Prevention. RECKONition, which is meant to provide Natural Language Understanding, Clustering and Inference, is the result of a joint partnership with the Italian National Institute for Insurance against Accidents at Work (INAIL). The obtained results showed the ability to process textual data written in Italian describing industrial accidents dynamics and consequences.
Automated Design Space Exploration of CGRA Processing Element Architectures using Frequent Subgraph Analysis
Venue: arXiv cs.AR 2021
Year: 2021

The architecture of a coarse-grained reconfigurable array (CGRA) processing element (PE) has a significant effect on the performance and energy efficiency of an application running on the CGRA. This paper presents an automated approach for generating specialized PE architectures for an application or an application domain. Frequent subgraphs mined from a set of applications are merged to form a PE architecture specialized to that application domain. For the image processing and machine learning domains, we generate specialized PEs that are up to 10.5x more energy efficient and consume 9.1x less area than a baseline PE.
Practical Channel Estimation and Phase Shift Design for Intelligent Reflecting Surface Empowered MIMO Systems
Venue: arXiv cs.IT 2021
Year: 2021

In this paper, channel estimation techniques and phase shift design for intelligent reflecting surface (IRS)-empowered single-user multiple-input multiple-output (SU-MIMO) systems are proposed. Among four channel estimation techniques developed in the paper, the two novel ones, single-path approximated channel (SPAC) and selective emphasis on rank-one matrices (SEROM), have low training overhead to enable practical IRS-empowered SU-MIMO systems. SPAC is mainly based on parameter estimation by approximating IRS-related channels as dominant single-path channels. SEROM exploits IRS phase shifts as well as training signals for channel estimation and easily adjusts its training overhead. A closed-form solution for IRS phase shift design is also developed to maximize spectral efficiency where the solution only requires basic linear operations. Numerical results show that SPAC and SEROM combined with the proposed IRS phase shift design achieve high spectral efficiency even with low training overhead compared to existing methods.
Spatial Privacy-aware VR streaming
Venue: arXiv cs.MM 2021
Year: 2021

Proactive tile-based virtual reality (VR) video streaming employs the current tracking data of a user to predict future requested tiles, then renders and delivers the predicted tiles before playback. Very recently, privacy protection in proactive VR video streaming starts to raise concerns. However, existing privacy protection may fail even with privacy-preserve federated learning. This is because when the future requested tiles can be predicted accurately, the user-behavior-related data can still be recovered from the predicted tiles. In this paper, we consider how to protect privacy even with accurate predictors and investigate the impact of privacy requirement on the quality of experience (QoE). To this end, we first add extra \textit{camouflaged} tile requests to the real tile requests and model the privacy requirement as the \textit{spatial degree of privacy} (sDoP). By ensuring sDoP, the real tile requests can be hidden and privacy can be protected. Then, we jointly optimize the durations for prediction, computing, and transmitting, aimed at maximizing the privacy-aware QoE given arbitrary predictor and configured resources. From the obtained optimal closed-form solution, we find that the impacts of sDoP on the QoE are two sides of the same coin. On the one side the increase of sDoP improves the capability of communication and computing hence improves QoE. On the other side it degrades the prediction performance hence degrades the QoE. The overall impact depends on which factor dominates the QoE. Simulation with two predictors on a real dataset verifies the analysis and shows that the overall impact of sDoP is to improve the QoE.
Self-Claimed Assumptions in Deep Learning Frameworks: An Exploratory Study
Venue: arXiv cs.SE 2021
Year: 2021

Deep learning (DL) frameworks have been extensively designed, implemented, and used in software projects across many domains. However, due to the lack of knowledge or information, time pressure, complex context, etc., various uncertainties emerge during the development, leading to assumptions made in DL frameworks. Though not all the assumptions are negative to the frameworks, being unaware of certain assumptions can result in critical problems (e.g., system vulnerability and failures, inconsistencies, and increased cost). As the first step of addressing the critical problems, there is a need to explore and understand the assumptions made in DL frameworks. To this end, we conducted an exploratory study to understand self-claimed assumptions (SCAs) about their distribution, classification, and impacts using code comments from nine popular DL framework projects on GitHub. The results are that: (1) 3,084 SCAs are scattered across 1,775 files in the nine DL frameworks, ranging from 1,460 (TensorFlow) to 8 (Keras) SCAs. (2) There are four types of validity of SCAs: Valid SCA, Invalid SCA, Conditional SCA, and Unknown SCA, and four types of SCAs based on their content: Configuration and Context SCA, Design SCA, Tensor and Variable SCA, and Miscellaneous SCA. (3) Both valid and invalid SCAs may have an impact within a specific scope (e.g., in a function) on the DL frameworks. Certain technical debt is induced when making SCAs. There are source code written and decisions made based on SCAs. This is the first study on investigating SCAs in DL frameworks, which helps researchers and practitioners to get a comprehensive understanding on the assumptions made. We also provide the first dataset of SCAs for further research and practice in this area.
Using Fisher's Exact Test to Evaluate Association Measures for N-grams
Venue: arXiv cs.CL 2021
Year: 2021

To determine whether some often-used lexical association measures assign high scores to n-grams that chance could have produced as frequently as observed, we used an extension of Fisher's exact test to sequences longer than two words to analyse a corpus of four million words. The results, based on the precision-recall curve and a new index called chance-corrected average precision, show that, as expected, simple-ll is extremely effective. They also show, however, that MI3 is more efficient than the other hypothesis tests-based measures and even reaches a performance level almost equal to simple-ll for 3-grams. It is additionally observed that some measures are more efficient for 3-grams than for 2-grams, while others stagnate.
Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
Venue: arXiv cs.LG 2021
Year: 2021

The use of deep neural networks (DNNs) in safety-critical applications like mobile health and autonomous driving is challenging due to numerous model-inherent shortcomings. These shortcomings are diverse and range from a lack of generalization over insufficient interpretability to problems with malicious inputs. Cyber-physical systems employing DNNs are therefore likely to suffer from safety concerns. In recent years, a zoo of state-of-the-art techniques aiming to address these safety concerns has emerged. This work provides a structured and broad overview of them. We first identify categories of insufficiencies to then describe research activities aiming at their detection, quantification, or mitigation. Our paper addresses both machine learning experts and safety engineers: The former ones might profit from the broad range of machine learning topics covered and discussions on limitations of recent methods. The latter ones might gain insights into the specifics of modern ML methods. We moreover hope that our contribution fuels discussions on desiderata for ML systems and strategies on how to propel existing approaches accordingly.