Reviewerly/exHarmony huggingface dataset:
An Attentive Sequence to Sequence Translator for Localizing Video Clips by Natural Language

We propose a novel attentive sequence to sequence translator (ASST) for localizing video clips by natural language descriptions. We make two contributions. First, we propose an attentive mechanism that aligns natural language descriptions and video content. A bi-directional Recurrent Neural Network (RNN) parses natural language descriptions in two directions. Given a video-description pair, ASST generates a vector sequence representation. Each vector represents a video frame, conditioned by the description. The vector sequence representation not only preserves the temporal dependencies between the frames, but also provides an effective way to perform frame-level videolanguage matching. The attentive model then aligns words to each frame, thereby resulting in a more detailed understanding of video content and description semantics. Second, we design a hierarchical architecture for the network to jointly model language descriptions and video content. The hierarchical architecture exploits video content with multiple granularities, ranging from subtle details to global context. The integration of the multiple granularities yields a robust representation for multi-level videolanguage abstraction. We validate the effectiveness of our ASST on two large-scale datasets. Our ASST outperforms the state-ofthe-art by 4.28% in Rank@1 on the DiDeMo dataset. On the Charades-STA dataset, we significantly improve the state-of-theart by 13.41% in Recall@1,IoU = 0.5.
Stochastic Multilevel Composition Optimization Algorithms with Level-Independent Convergence Rates

In this paper, we study smooth stochastic multilevel composition optimization problems, where the objective function is a nested composition of $T$ functions. We assume access to noisy evaluations of the functions and their gradients, through a stochastic first-order oracle. For solving this class of problems, we propose two algorithms using moving-average stochastic estimates, and analyze their convergence to an $\epsilon$-stationary point of the problem. We show that the first algorithm, which is a generalization of [S. Ghadimi, A. Ruszczynski, and M. Wang, SIAM J. Optim., 30 (2020), pp. 960--979] to the $T$ level case, can achieve a sample complexity of $\mathcal{O}_T(1/\epsilon^6)$ by using minibatches of samples in each iteration, where $\mathcal{O}_T$ hides constants that depend on $T$. By modifying this algorithm using linearized stochastic estimates of the function values, we improve the sample complexity to $\mathcal{O}_T(1/\epsilon^4)$. This modification not only removes the requirement of having a minibatch of samples in each iteration, but also makes the algorithm parameter-free and easy to implement. To the best of our knowledge, this is the first time that such an online algorithm designed for the (un)constrained multilevel setting obtains the same sample complexity of the smooth single-level setting, under standard assumptions (unbiasedness and boundedness of the second moments) on the stochastic first-order oracle.
Vulnerability vs. Reliability: Disentangled Adversarial Examples for Cross-Modal Learning

The vulnerability of deep neural networks has gained a great upsurge of research attention, which engages well-designed examples through adding little perturbations to fool a well-performed network. Meanwhile, a progress has been made in leveraging adversarial examples to boost the robustness of deep cross-modal networks. However, for cross-modal learning, both the causes of adversarial examples and their latent advantages in learning cross-modal correlations are under-explored. In this paper, we propose novel Disentangled Adversarial examples for Cross-Modal learning, dubbed DACM. Specifically, we first divide cross-modal data into two aspects, namely modality-related component and modality-unrelated counterpart, and then learn to improve the reliability of network using the modality-related component. To achieve this goal, we apply the generation of adversarial perturbations to strengthen cross-modal correlations, wherein the modality-related component is acquired through gradually detaching the modality-unrelated component. Finally, the proposed DACM is employed to create modality-related examples towards the application of cross-modal hashing retrieval. Extensive experiments carried out on two cross-modal benchmarks show that the adversarial examples learned by DACM are efficient at fooling a target deep cross-modal hashing network. On the other hand, training this target model by merely leveraging our created modality-related examples in turn significantly promotes the robustness of this model itself.
Measuring Model Complexity of Neural Networks with Curve Activation Functions

It is fundamental to measure model complexity of deep neural networks. A good model complexity measure can help to tackle many challenging problems, such as overfitting detection, model selection, and performance improvement. The existing literature on model complexity mainly focuses on neural networks with piecewise linear activation functions. Model complexity of neural networks with general curve activation functions remains an open problem. To tackle the challenge, in this paper, we first propose linear approximation neural network (LANN for short), a piecewise linear framework to approximate a given deep model with curve activation function. LANN constructs individual piecewise linear approximation for the activation function of each neuron, and minimizes the number of linear regions to satisfy a required approximation degree. Then, we analyze the upper bound of the number of linear regions formed by LANNs, and derive the complexity measure based on the upper bound. To examine the usefulness of the complexity measure, we experimentally explore the training process of neural networks and detect overfitting. Our results demonstrate that the occurrence of overfitting is positively correlated with the increase of model complexity during training. We find that the L1 and L2 regularizations suppress the increase of model complexity. Finally, we propose two approaches to prevent overfitting by directly constraining model complexity, namely neuron pruning and customized L1 regularization.
Functional Error Correction for Reliable Neural Networks

When deep neural networks (DNNs) are implemented in hardware, their weights need to be stored in memory devices. As noise accumulates in the stored weights, the DNN's performance will degrade. This paper studies how to use error correcting codes (ECCs) to protect the weights. Different from classic error correction in data storage, the optimization objective is to optimize the DNN's performance after error correction, instead of minimizing the Uncorrectable Bit Error Rate in the protected bits. That is, by seeing the DNN as a function of its input, the error correction scheme is function-oriented. A main challenge is that a DNN often has millions to hundreds of millions of weights, causing a large redundancy overhead for ECCs, and the relationship between the weights and its DNN's performance can be highly complex. To address the challenge, we propose a Selective Protection (SP) scheme, which chooses only a subset of important bits for ECC protection. To find such bits and achieve an optimized tradeoff between ECC's redundancy and DNN's performance, we present an algorithm based on deep reinforcement learning. Experimental results verify that compared to the natural baseline scheme, the proposed algorithm achieves substantially better performance for the functional error correction task.
An Automatic Approach for Generating Rich, Linked Geo-Metadata from Historical Map Images

Historical maps contain detailed geographic information difficult to find elsewhere covering long-periods of time (e.g., 125 years for the historical topographic maps in the US). However, these maps typically exist as scanned images without searchable metadata. Existing approaches making historical maps searchable rely on tedious manual work (including crowd-sourcing) to generate the metadata (e.g., geolocations and keywords). Optical character recognition (OCR) software could alleviate the required manual work, but the recognition results are individual words instead of location phrases (e.g., "Black'' and "Mountain'' vs. "Black Mountain''). This paper presents an end-to-end approach to address the real-world problem of finding and indexing historical map images. This approach automatically processes historical map images to extract their text content and generates a set of metadata that is linked to large external geospatial knowledge bases. The linked metadata in the RDF (Resource Description Framework) format support complex queries for finding and indexing historical maps, such as retrieving all historical maps covering mountain peaks higher than 1,000 meters in California. We have implemented the approach in a system called mapKurator. We have evaluated mapKurator using historical maps from several sources with various map styles, scales, and coverage. Our results show significant improvement over the state-of-the-art methods. The code has been made publicly available as modules of the Kartta Labs project at https://github.com/kartta-labs/Project.
Applying Machine Learning for Integration of Multi-Modal Genomics Data and Imaging Data to Quantify Heterogeneity in Tumour Tissues

With rapid advances in experimental instruments and protocols, imaging and sequencing data are being generated at an unprecedented rate contributing significantly to the current and coming big biomedical data. Meanwhile, unprecedented advances in computational infrastructure and analysis algorithms are realizing image-based digital diagnosis not only in radiology and cardiology but also oncology and other diseases. Machine learning methods, especially deep learning techniques, are already and broadly implemented in diverse technological and industrial sectors, but their applications in healthcare are just starting. Uniquely in biomedical research, a vast potential exists to integrate genomics data with histopathological imaging data. The integration has the potential to extend the pathologist’s limits and boundaries, which may create breakthroughs in diagnosis, treatment, and monitoring at molecular and tissue levels. Moreover, the applications of genomics data are realizing the potential for personalized medicine, making diagnosis, treatment, monitoring, and prognosis more accurate. In this chapter, we discuss machine learning methods readily available for digital pathology applications, new prospects of integrating spatial genomics data on tissues with tissue morphology, and frontier approaches to combining genomics data with pathological imaging data. We present perspectives on how artificial intelligence can be synergized with molecular genomics and imaging to make breakthroughs in biomedical and translational research for computer-aided applications.
Performance Evaluation of Machine Learning Algorithms for Bitcoin Price Prediction

Bitcoin is one of the most valuable cryptocurrency in the world with the prices as high as 19,783 United States Dollar(USD) in December of 2017. It made Bitcoin a very profitable market for investment but Bitcoin saw many ups and down as well. Today’s Bitcoin price is 3913 USD but that doesn’t mean that the prices always keep falling. The price of Bitcoin vary over time and is governed by various factors, like the market it is being traded in, scarcity, supply and demand. What has made Bitcoin valuable is that it can be used as a currency, we can pay a part or a fraction of Bitcoin to a person in exchange for something and the part is easily verifiable by blockchain. The small number of Bitcoin, roughly 16 million Bitcoins for the entire world has made it scare and above that its high utility makes it more prestigious.Trading of Bitcoin has proved to be very profitable to many people but the risk in trading is huge as the market of Bitcoin is very volatile. To decrease the risks, this project has been carried out to predict the price of Bitcoin using Recurrent Neural Network(RNN), Long Short Term Memory (LSTM) and Linear Regression(LR) to predict the price of Bitcoin. Evaluation of these algorithms is carried out to determine which among the two is better for the prediction of Bitcoin prices. The dataset used contains minute by minute prices of Bitcoin of over 5 years and contains almost 30,00,000 entries. Since, the dataset used is a big data, evaluating the performance of algorithms over a large dataset will give accurate results.
MF-CFI: A fused evaluation index for camouflage patterns based on human visual perception

The evaluation index of camouflage patterns is important in the field of military application. It is the goal that researchers have always pursued to make the computable evaluation indicators more in line with the human visual mechanism. In order to make the evaluation method more computationally intelligent, a Multi-Feature Camouflage Fused Index (MF-CFI) is proposed based on the comparison of grayscale, color and texture features between the target and the background. In order to verify the effectiveness of the proposed index, eye movement experiments are conducted to compare the proposed index with existing indexes including Universal Image Quality Index (UIQI), Camouflage Similarity Index (CSI) and Structural Similarity (SSIM). Twenty-four different simulated targets are designed in a grassland background, 28 observers participate in the experiment and record the eye movement data during the observation process. The results show that the highest Pearson correlation coefficient is observed between MF-CFI and the eye movement data, both in the designed digital camouflage patterns and large-spot camouflage patterns. Since MF-CFI is more in line with the detection law of camouflage targets in human visual perception, the proposed index can be used for the comparison and parameter optimization of camouflage design algorithms.
A Multiscale Recognition Method for the Optimization of Traffic Signs Using GMM and Category Quality Focal Loss

Effective traffic sign recognition algorithms can assist drivers or automatic driving systems in detecting and recognizing traffic signs in real-time. This paper proposes a multiscale recognition method for traffic signs based on the Gaussian Mixture Model (GMM) and Category Quality Focal Loss (CQFL) to enhance recognition speed and recognition accuracy. Specifically, GMM is utilized to cluster the prior anchors, which are in favor of reducing the clustering error. Meanwhile, considering the most common issue in supervised learning (i.e., the imbalance of data set categories), the category proportion factor is introduced into Quality Focal Loss, which is referred to as CQFL. Furthermore, a five-scale recognition network with a prior anchor allocation strategy is designed for small target objects i.e., traffic sign recognition. Combining five existing tricks, the best speed and accuracy tradeoff on our data set (40.1% mAP and 15 FPS on a single 1080Ti GPU), can be achieved. The experimental results demonstrate that the proposed method is superior to the existing mainstream algorithms, in terms of recognition accuracy and recognition speed.
Semiring programming: A semantic framework for generalized sum product problems

To solve hard problems, AI relies on a variety of disciplines such as logic, probabilistic reasoning, machine learning and mathematical programming. Although it is widely accepted that solving real-world problems requires an integration amongst these, contemporary representation methodologies offer little support for this. In an attempt to alleviate this situation, we position and motivate a new declarative programming framework in this paper. We focus on the semantical foundations in service of providing abstractions of well-known problems such as SAT, Bayesian inference, generative models, learning and convex optimization. Programs are understood in terms of first-order logic structures with semiring labels, which allows us to freely combine and integrate problems from different AI disciplines and represent non-standard problems over unbounded domains. Thus, the main thrust of this paper is to view such well-known problems through a unified lens in the hope that appropriate solver strategies (exact, approximate, portfolio or hybrid) may emerge that tackle real-world problems in a principled way.
Reputation and Trust Models with Data Quality Metrics for Improving Autonomous Vehicles Traffic Security and Safety

In this paper, we develop, implement, test, and analyze a novel technique that allows to improve the security and safety of intersections crossing by the group of autonomous vehicles. The proposed approach is based on augmenting the trust and reputation models with data quality evaluation and using it for initial trust value assignment. This technique allows increasing accuracy and recall rates in detecting agents that might supply incorrect data and facilitating their removal from the agent group consideration. To evaluate the proposed method, we performed the simulation study of the autonomous vehicle traffic control through an intersection. The conducted experiments showed that the employment of data quality metrics improves detecting autonomous vehicles and other agents that might transmit incorrect data.
Targeted Data-driven Regularization for Out-of-Distribution Generalization

Due to biases introduced by large real-world datasets, deviations of deep learning models from their expected behavior on out-of-distribution test data are worrisome. Especially when data come from imbalanced or heavy-tailed label distributions, or minority groups of a sensitive feature. Classical approaches to address these biases are mostly data- or application-dependent, hence are burdensome to tune. Some meta-learning approaches, on the other hand, aim to learn hyperparameters in the learning process using different objective functions on training and validation data. However, these methods suffer from high computational complexity and are not scalable to large datasets. In this paper, we propose a unified data-driven regularization approach to learn a generalizable model from biased data. The proposed framework, named as targeted data-driven regularization (TDR), is model- and dataset-agnostic, and employs a target dataset that resembles the desired nature of test data in order to guide the learning process in a coupled manner. We cast the problem as a bilevel optimization and propose an efficient stochastic gradient descent based method to solve it. The framework can be utilized to alleviate various types of biases in real-world applications. We empirically show, on both synthetic and real-world datasets, the superior performance of TDR for resolving issues stem from these biases.
Recurrent Halting Chain for Early Multi-label Classification

Early multi-label classification of time series, the assignment of a label set to a time series before the series is entirely observed, is critical for time-sensitive domains such as healthcare. In such cases, waiting too long to classify can render predictions useless, regardless of their accuracy, while predicting prematurely can result in potentially costly erroneous results. When predicting multiple labels (for example, types of infections), dependencies between labels can be learned and leveraged to improve overall accuracy. Together, reliably predicting the correct label set of a time series while observing as few timesteps as possible is challenging because these goals are contradictory in that fewer timesteps often means worse accuracy. To achieve early yet sufficiently accurate predictions, correlations between labels must be accounted for since direct evidence of some labels may only appear late in the series. We design an effective solution to this open problem, the Recurrent Halting Chain (RHC), that for the first time integrates key innovations in both Early and Multi-label Classification into one multi-objective model. RHC uses a recurrent neural network to jointly model raw time series as well as correlations between labels, resulting in a novel order-free classifier chain that tackles this time-sensitive multi-label learning task. Further, RHC employs a reinforcement learning-based halting network to decide at each timestep which, if any, classes should be predicted, learning to build the label set over time. Using two real-world time-sensitive datasets and popular multi-label metrics, we show that RHC outperforms recent alternatives by predicting more-accurate label sets earlier.
Detecting Deepfake Video by Learning Two-Level Features with Two-Stream Convolutional Neural Network

Deepfake techniques has made face swapping in video easy to use. Nowadays, the spread of Deepfake videos over networks is concerned worldwide. This work proposes an approach to more accurate and robust detection of them. Since artifacts left by Deepfake tools can be largely categorized into two classes of different levels, i.e. semantic and noise level, we adopt a two-stream convolutional neural network (CNN) to capture the 2-level features concurrently. Xception network is trained only as the first stream to detect semantic anomalies such as the editing artifacts around face contour, detail missing, and geometric inconsistence in eyes. Meanwhile, the 2nd stream, which contain the constrained convolution filter and median filter, is designed to capture the tampering traces in local noises. By concatenating the 2-level features learned from the both streams, our method obtains very comprehensive knowledge about the existence of face swapping. The experimental results have shown its advantage over the existing methods on both the accuracy and robustness.
The Minimum Upload Cost of Symmetric Private Information Retrieval

For the symmetric private information retrieval problem with K messages and N servers, we show that the minimum (symmetric) upload cost is log <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> (⌈K <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1/N-1</sup> ⌉) bits per server, i.e., the user must upload a q-ary symbol to each server where q is at least ⌈K <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1/N-1</sup> ⌉.
A Reasoning Based Model for Anomaly Detection in the Smart City Domain

Using a visual scene object tracker and a Non-Axiomatic Reasoning System we demonstrate how to predict and detect various anomaly classes. The approach combines an object tracker with a base ontology and the OpenNARS reasoner to learn to classify scene regions based on accumulating evidence from typical entity class (tracked object) behaviours. The system can autonomously satisfy goals related to anomaly detection and respond to user Q&A in real time. The system learns directly from experience with no initial training required (one-shot). The solution is a fusion of neural techniques (object tracker) and a reasoning system (with ontology).