OpenRaiser/Intern-Atlas huggingface dataset:
Exploration of Visual Features and their weighted-additive fusion for Video Captioning
Venue: arXiv cs.CV 2021
Year: 2021

Video captioning is a popular task that challenges models to describe events in videos using natural language. In this work, we investigate the ability of various visual feature representations derived from state-of-the-art convolutional neural networks to capture high-level semantic context. We introduce the Weighted Additive Fusion Transformer with Memory Augmented Encoders (WAFTM), a captioning model that incorporates memory in a transformer encoder and uses a novel method, to fuse features, that ensures due importance is given to more significant representations. We illustrate a gain in performance realized by applying Word-Piece Tokenization and a popular REINFORCE algorithm. Finally, we benchmark our model on two datasets and obtain a CIDEr of 92.4 on MSVD and a METEOR of 0.091 on the ActivityNet Captions Dataset.
Continual Learning of Knowledge Graph Embeddings
Venue: arXiv cs.RO 2021
Year: 2021

In recent years, there has been a resurgence in methods that use distributed (neural) representations to represent and reason about semantic knowledge for robotics applications. However, while robots often observe previously unknown concepts, these representations typically assume that all concepts are known a priori, and incorporating new information requires all concepts to be learned afresh. Our work relaxes this limiting assumption of existing representations and tackles the incremental knowledge graph embedding problem by leveraging the principles of a range of continual learning methods. Through an experimental evaluation with several knowledge graphs and embedding representations, we provide insights about trade-offs for practitioners to match a semantics-driven robotics applications to a suitable continual knowledge graph embedding method.
Cocktail Edge Caching: Ride Dynamic Trends of Content Popularity with Ensemble Learning
Venue: arXiv cs.NI 2021
Year: 2021

Edge caching will play a critical role in facilitating the emerging content-rich applications. However, it faces many new challenges, in particular, the highly dynamic content popularity and the heterogeneous caching configurations. In this paper, we propose Cocktail Edge Caching, that tackles the dynamic popularity and heterogeneity through ensemble learning. Instead of trying to find a single dominating caching policy for all the caching scenarios, we employ an ensemble of constituent caching policies and adaptively select the best-performing policy to control the cache. Towards this goal, we first show through formal analysis and experiments that different variations of the LFU and LRU policies have complementary performance in different caching scenarios. We further develop a novel caching algorithm that enhances LFU/LRU with deep recurrent neural network (LSTM) based time-series analysis. Finally, we develop a deep reinforcement learning agent that adaptively combines base caching policies according to their virtual hit ratios on parallel virtual caches. Through extensive experiments driven by real content requests from two large video streaming platforms, we demonstrate that CEC not only consistently outperforms all single policies, but also improves the robustness of them. CEC can be well generalized to different caching scenarios with low computation overheads for deployment.
A Ternary Bi-Directional LSTM Classification for Brain Activation Pattern Recognition Using fNIRS
Venue: arXiv cs.LG 2021
Year: 2021

Functional near-infrared spectroscopy (fNIRS) is a non-invasive, low-cost method used to study the brain's blood flow pattern. Such patterns can enable us to classify performed by a subject. In recent research, most classification systems use traditional machine learning algorithms for the classification of tasks. These methods, which are easier to implement, usually suffer from low accuracy. Further, a complex pre-processing phase is required for data preparation before implementing traditional machine learning methods. The proposed system uses a Bi-Directional LSTM based deep learning architecture for task classification, including mental arithmetic, motor imagery, and idle state using fNIRS data. Further, this system will require less pre-processing than the traditional approach, saving time and computational resources while obtaining an accuracy of 81.48\%, which is considerably higher than the accuracy obtained using conventional machine learning algorithms for the same data set.
Image Enhancement using Fuzzy Intensity Measure and Adaptive Clipping Histogram Equalization
Venue: arXiv cs.CV 2021
Year: 2021

Image enhancement aims at processing an input image so that the visual content of the output image is more pleasing or more useful for certain applications. Although histogram equalization is widely used in image enhancement due to its simplicity and effectiveness, it changes the mean brightness of the enhanced image and introduces a high level of noise and distortion. To address these problems, this paper proposes image enhancement using fuzzy intensity measure and adaptive clipping histogram equalization (FIMHE). FIMHE uses fuzzy intensity measure to first segment the histogram of the original image, and then clip the histogram adaptively in order to prevent excessive image enhancement. Experiments on the Berkeley database and CVF-UGR-Image database show that FIMHE outperforms state-of-the-art histogram equalization based methods.
KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization
Venue: arXiv cs.CL 2021
Year: 2021

Recently, transformer-based language models such as BERT have shown tremendous performance improvement for a range of natural language processing tasks. However, these language models usually are computation expensive and memory intensive during inference. As a result, it is difficult to deploy them on resource-restricted devices. To improve the inference performance, as well as reduce the model size while maintaining the model accuracy, we propose a novel quantization method named KDLSQ-BERT that combines knowledge distillation (KD) with learned step size quantization (LSQ) for language model quantization. The main idea of our method is that the KD technique is leveraged to transfer the knowledge from a "teacher" model to a "student" model when exploiting LSQ to quantize that "student" model during the quantization training process. Extensive experiment results on GLUE benchmark and SQuAD demonstrate that our proposed KDLSQ-BERT not only performs effectively when doing different bit (e.g. 2-bit $\sim$ 8-bit) quantization, but also outperforms the existing BERT quantization methods, and even achieves comparable performance as the full-precision base-line model while obtaining 14.9x compression ratio. Our code will be public available.
DeepWaste: Applying Deep Learning to Waste Classification for a Sustainable Planet
Venue: arXiv cs.LG 2021
Year: 2021

Accurate waste disposal, at the point of disposal, is crucial to fighting climate change. When materials that could be recycled or composted get diverted into landfills, they cause the emission of potent greenhouse gases such as methane. Current attempts to reduce erroneous waste disposal are expensive, inaccurate, and confusing. In this work, we propose DeepWaste, an easy-to-use mobile app, that utilizes highly optimized deep learning techniques to provide users instantaneous waste classification into trash, recycling, and compost. We experiment with several convolution neural network architectures to detect and classify waste items. Our best model, a deep learning residual neural network with 50 layers, achieves an average precision of 0.881 on the test set. We demonstrate the performance and efficiency of our app on a set of real-world images.
Interaction-Aware Behavior Planning for Autonomous Vehicles Validated with Real Traffic Data
Venue: arXiv cs.RO 2021
Year: 2021

Autonomous vehicles (AVs) need to interact with other traffic participants who can be either cooperative or aggressive, attentive or inattentive. Such different characteristics can lead to quite different interactive behaviors. Hence, to achieve safe and efficient autonomous driving, AVs need to be aware of such uncertainties when they plan their own behaviors. In this paper, we formulate such a behavior planning problem as a partially observable Markov Decision Process (POMDP) where the cooperativeness of other traffic participants is treated as an unobservable state. Under different cooperativeness levels, we learn the human behavior models from real traffic data via the principle of maximum likelihood. Based on that, the POMDP problem is solved by Monte-Carlo Tree Search. We verify the proposed algorithm in both simulations and real traffic data on a lane change scenario, and the results show that the proposed algorithm can successfully finish the lane changes without collisions.
Walk in Wild: An Ensemble Approach for Hostility Detection in Hindi Posts
Venue: arXiv cs.CL 2021
Year: 2021

As the reach of the internet increases, pejorative terms started flooding over social media platforms. This leads to the necessity of identifying hostile content on social media platforms. Identification of hostile contents on low-resource languages like Hindi poses different challenges due to its diverse syntactic structure compared to English. In this paper, we develop a simple ensemble based model on pre-trained mBERT and popular classification algorithms like Artificial Neural Network (ANN) and XGBoost for hostility detection in Hindi posts. We formulated this problem as binary classification (hostile and non-hostile class) and multi-label multi-class classification problem (for more fine-grained hostile classes). We received third overall rank in the competition and weighted F1-scores of ~0.969 and ~0.61 on the binary and multi-label multi-class classification tasks respectively.
Motion-Based Handwriting Recognition
Venue: arXiv cs.CV 2021
Year: 2021

We attempt to overcome the restriction of requiring a writing surface for handwriting recognition. In this study, we design a prototype of a stylus equipped with motion sensor, and utilizes gyroscopic and acceleration sensor reading to perform written letter classification using various deep learning techniques such as CNN and RNNs. We also explore various data augmentation techniques and their effects, reaching up to 86% accuracy.
Motion-Based Handwriting Recognition and Word Reconstruction
Venue: arXiv cs.CV 2021
Year: 2021

In this project, we leverage a trained single-letter classifier to predict the written word from a continuously written word sequence, by designing a word reconstruction pipeline consisting of a dynamic-programming algorithm and an auto-correction model. We conduct experiments to optimize models in this pipeline, then employ domain adaptation to explore using this pipeline on unseen data distributions.
Hierarchical Multitask Learning with Dependency Parsing for Japanese Semantic Role Labeling Improves Performance of Argument Identification
Venue: arXiv cs.CL 2021
Year: 2021

With the advent of FrameNet and PropBank, many semantic role labeling (SRL) systems have been proposed in English. Although research on Japanese predicate argument structure analysis (PASA) has been conducted, most studies focused on surface cases. There are only few previous works on Japanese SRL for deep cases, and their models' accuracies are low. Therefore, we propose a hierarchical multitask learning method with dependency parsing (DP) and show that our model achieves state-of-the-art results in Japanese SRL. Also, we conduct experiments with a joint model that performs both argument identification and argument classification simultaneously. The result suggests that multitasking with DP is mainly effective for argument identification.
Video Summarization Using Deep Neural Networks: A Survey
Venue: arXiv cs.CV 2021
Year: 2021

Video summarization technologies aim to create a concise and complete synopsis by selecting the most informative parts of the video content. Several approaches have been developed over the last couple of decades and the current state of the art is represented by methods that rely on modern deep neural network architectures. This work focuses on the recent advances in the area and provides a comprehensive survey of the existing deep-learning-based methods for generic video summarization. After presenting the motivation behind the development of technologies for video summarization, we formulate the video summarization task and discuss the main characteristics of a typical deep-learning-based analysis pipeline. Then, we suggest a taxonomy of the existing algorithms and provide a systematic review of the relevant literature that shows the evolution of the deep-learning-based video summarization technologies and leads to suggestions for future developments. We then report on protocols for the objective evaluation of video summarization algorithms and we compare the performance of several deep-learning-based approaches. Based on the outcomes of these comparisons, as well as some documented considerations about the amount of annotated data and the suitability of evaluation protocols, we indicate potential future research directions.
An Abstract Contract Theory for Programs with Procedures
Venue: arXiv cs.LO 2021
Year: 2021

When developing complex software and systems, contracts provide a means for controlling the complexity by dividing the responsibilities among the components of the system in a hierarchical fashion. In specific application areas, dedicated contract theories formalise the notion of contract and the operations on contracts in a manner that supports best the development of systems in that area. At the other end, contract meta-theories attempt to provide a systematic view on the various contract theories by axiomatising their desired properties. However, there exists a noticeable gap between the most well-known contract meta-theory of Benveniste et al., which focuses on the design of embedded and cyber-physical systems, and the established way of using contracts when developing general software, following Meyer's design-by-contract methodology. At the core of this gap appears to be the notion of procedure: while it is a central unit of composition in software development, the meta-theory does not suggest an obvious way of treating procedures as components. In this paper, we provide a first step towards a contract theory that takes procedures as the basic building block, and is at the same time an instantiation of the meta-theory. To this end, we propose an abstract contract theory for sequential programming languages with procedures, based on denotational semantics. We show that, on the one hand, the specification of contracts of procedures in Hoare logic, and their procedure-modular verification, can be cast naturally in the framework of our abstract contract theory. On the other hand, we also show our contract theory to fulfil the axioms of the meta-theory. In this way, we give further evidence for the utility of the meta-theory, and prepare the ground for combining our instantiation with other, already existing instantiations.
Black-box Adversarial Attacks in Autonomous Vehicle Technology
Venue: arXiv cs.CV 2021
Year: 2021

Despite the high quality performance of the deep neural network in real-world applications, they are susceptible to minor perturbations of adversarial attacks. This is mostly undetectable to human vision. The impact of such attacks has become extremely detrimental in autonomous vehicles with real-time "safety" concerns. The black-box adversarial attacks cause drastic misclassification in critical scene elements such as road signs and traffic lights leading the autonomous vehicle to crash into other vehicles or pedestrians. In this paper, we propose a novel query-based attack method called Modified Simple black-box attack (M-SimBA) to overcome the use of a white-box source in transfer based attack method. Also, the issue of late convergence in a Simple black-box attack (SimBA) is addressed by minimizing the loss of the most confused class which is the incorrect class predicted by the model with the highest probability, instead of trying to maximize the loss of the correct class. We evaluate the performance of the proposed approach to the German Traffic Sign Recognition Benchmark (GTSRB) dataset. We show that the proposed model outperforms the existing models like Transfer-based projected gradient descent (T-PGD), SimBA in terms of convergence time, flattening the distribution of confused class probability, and producing adversarial samples with least confidence on the true class.
Effect of Gameplay Uncertainty, Display Type, and Age on Virtual Reality Exergames
Venue: arXiv cs.HC 2021
Year: 2021

Uncertainty is widely acknowledged as an engaging gameplay element but rarely used in exergames. In this research, we explore the role of uncertainty in exergames and introduce three uncertain elements (false-attacks, misses, and critical hits) to an exergame. We conducted a study under two conditions (uncertain and certain), with two display types (virtual reality and large display) and across young and middle-aged adults to measure their effect on game performance, experience, and exertion. Results show that (1) our designed uncertain elements are instrumental in increasing exertion levels; (2) when playing a motion-based first-person perspective exergame, virtual reality can improve performance, while maintaining the same motion sickness level as a large display; and (3) exergames for middle-aged adults should be designed with age-related declines in mind, similar to designing for elderly adults. We also framed two design guidelines for exergames that have similar features to the game used in this research.
EAGER: Embedding-Assisted Entity Resolution for Knowledge Graphs
Venue: arXiv cs.LG 2021
Year: 2021

Entity Resolution (ER) is a constitutional part for integrating different knowledge graphs in order to identify entities referring to the same real-world object. A promising approach is the use of graph embeddings for ER in order to determine the similarity of entities based on the similarity of their graph neighborhood. The similarity computations for such embeddings translates to calculating the distance between them in the embedding space which is comparatively simple. However, previous work has shown that the use of graph embeddings alone is not sufficient to achieve high ER quality. We therefore propose a more comprehensive ER approach for knowledge graphs called EAGER (Embedding-Assisted Knowledge Graph Entity Resolution) to flexibly utilize both the similarity of graph embeddings and attribute values within a supervised machine learning approach. We evaluate our approach on 23 benchmark datasets with differently sized and structured knowledge graphs and use hypothesis tests to ensure statistical significance of our results. Furthermore we compare our approach with state-of-the-art ER solutions, where our approach yields competitive results for table-oriented ER problems and shallow knowledge graphs but much better results for deeper knowledge graphs.
Learning to Sample from Censored Markov Random Fields
Venue: arXiv cs.LG 2021
Year: 2021

We study learning Censor Markov Random Fields (abbreviated CMRFs). These are Markov Random Fields where some of the nodes are censored (not observed). We present an algorithm for learning high-temperature CMRFs within o(n) transportation distance. Crucially our algorithm makes no assumption about the structure of the graph or the number or location of the observed nodes. We obtain stronger results for high girth high-temperature CMRFs as well as computational lower bounds indicating that our results can not be qualitatively improved.
Node and Edge Nonlinear Eigenvector Centrality for Hypergraphs
Venue: arXiv cs.SI 2021
Year: 2021

Network scientists have shown that there is great value in studying pairwise interactions between components in a system. From a linear algebra point of view, this involves defining and evaluating functions of the associated adjacency matrix. Recent work indicates that there are further benefits from accounting directly for higher order interactions, notably through a hypergraph representation where an edge may involve multiple nodes. Building on these ideas, we motivate, define and analyze a class of spectral centrality measures for identifying important nodes and hyperedges in hypergraphs, generalizing existing network science concepts. By exploiting the latest developments in nonlinear Perron-Frobenius theory, we show how the resulting constrained nonlinear eigenvalue problems have unique solutions that can be computed efficiently via a nonlinear power method iteration. We illustrate the measures on realistic data sets.
APEX-Net: Automatic Plot Extractor Network
Venue: arXiv cs.CV 2021
Year: 2021

Automatic extraction of raw data from 2D line plot images is a problem of great importance having many real-world applications. Several algorithms have been proposed for solving this problem. However, these algorithms involve a significant amount of human intervention. To minimize this intervention, we propose APEX-Net, a deep learning based framework with novel loss functions for solving the plot extraction problem. We introduce APEX-1M, a new large scale dataset which contains both the plot images and the raw data. We demonstrate the performance of APEX-Net on the APEX-1M test set and show that it obtains impressive accuracy. We also show visual results of our network on unseen plot images and demonstrate that it extracts the shape of the plots to a great extent. Finally, we develop a GUI based software for plot extraction that can benefit the community at large. For dataset and more information visit https://sites.google.com/view/apexnetpaper/.
Towards interpreting ML-based automated malware detection models: a survey
Venue: arXiv cs.CR 2021
Year: 2021

Malware is being increasingly threatening and malware detectors based on traditional signature-based analysis are no longer suitable for current malware detection. Recently, the models based on machine learning (ML) are developed for predicting unknown malware variants and saving human strength. However, most of the existing ML models are black-box, which made their pre-diction results undependable, and therefore need further interpretation in order to be effectively deployed in the wild. This paper aims to examine and categorize the existing researches on ML-based malware detector interpretability. We first give a detailed comparison over the previous work on common ML model inter-pretability in groups after introducing the principles, attributes, evaluation indi-cators and taxonomy of common ML interpretability. Then we investigate the interpretation methods towards malware detection, by addressing the importance of interpreting malware detectors, challenges faced by this field, solutions for migitating these challenges, and a new taxonomy for classifying all the state-of-the-art malware detection interpretability work in recent years. The highlight of our survey is providing a new taxonomy towards malware detection interpreta-tion methods based on the common taxonomy summarized by previous re-searches in the common field. In addition, we are the first to evaluate the state-of-the-art approaches by interpretation method attributes to generate the final score so as to give insight to quantifying the interpretability. By concluding the results of the recent researches, we hope our work can provide suggestions for researchers who are interested in the interpretability on ML-based malware de-tection models.