OpenRaiser/Intern-Atlas huggingface dataset:
Mutual Information Based Method for Unsupervised Disentanglement of Video Representation
Venue: arXiv cs.CV 2020
Year: 2020

Video Prediction is an interesting and challenging task of predicting future frames from a given set context frames that belong to a video sequence. Video prediction models have found prospective applications in Maneuver Planning, Health care, Autonomous Navigation and Simulation. One of the major challenges in future frame generation is due to the high dimensional nature of visual data. In this work, we propose Mutual Information Predictive Auto-Encoder (MIPAE) framework, that reduces the task of predicting high dimensional video frames by factorising video representations into content and low dimensional pose latent variables that are easy to predict. A standard LSTM network is used to predict these low dimensional pose representations. Content and the predicted pose representations are decoded to generate future frames. Our approach leverages the temporal structure of the latent generative factors of a video and a novel mutual information loss to learn disentangled video representations. We also propose a metric based on mutual information gap (MIG) to quantitatively access the effectiveness of disentanglement on DSprites and MPI3D-real datasets. MIG scores corroborate with the visual superiority of frames predicted by MIPAE. We also compare our method quantitatively on evaluation metrics LPIPS, SSIM and PSNR.
Theory-guided Auto-Encoder for Surrogate Construction and Inverse Modeling
Venue: arXiv cs.LG 2020
Year: 2020

A Theory-guided Auto-Encoder (TgAE) framework is proposed for surrogate construction and is further used for uncertainty quantification and inverse modeling tasks. The framework is built based on the Auto-Encoder (or Encoder-Decoder) architecture of convolutional neural network (CNN) via a theory-guided training process. In order to achieve the theory-guided training, the governing equations of the studied problems can be discretized and the finite difference scheme of the equations can be embedded into the training of CNN. The residual of the discretized governing equations as well as the data mismatch constitute the loss function of the TgAE. The trained TgAE can be used to construct a surrogate that approximates the relationship between the model parameters and responses with limited labeled data. In order to test the performance of the TgAE, several subsurface flow cases are introduced. The results show the satisfactory accuracy of the TgAE surrogate and efficiency of uncertainty quantification tasks can be improved with the TgAE surrogate. The TgAE also shows good extrapolation ability for cases with different correlation lengths and variances. Furthermore, the parameter inversion task has been implemented with the TgAE surrogate and satisfactory results can be obtained.
Redesigning the classification layer by randomizing the class representation vectors
Venue: arXiv cs.LG 2020
Year: 2020

Neural image classification models typically consist of two components. The first is an image encoder, which is responsible for encoding a given raw image into a representative vector. The second is the classification component, which is often implemented by projecting the representative vector onto target class vectors. The target class vectors, along with the rest of the model parameters, are estimated so as to minimize the loss function. In this paper, we analyze how simple design choices for the classification layer affect the learning dynamics. We show that the standard cross-entropy training implicitly captures visual similarities between different classes, which might deteriorate accuracy or even prevents some models from converging. We propose to draw the class vectors randomly and set them as fixed during training, thus invalidating the visual similarities encoded in these vectors. We analyze the effects of keeping the class vectors fixed and show that it can increase the inter-class separability, intra-class compactness, and the overall model accuracy, while maintaining the robustness to image corruptions and the generalization of the learned concepts.
Towards evaluating and eliciting high-quality documentation for intelligent systems
Venue: arXiv cs.SE 2020
Year: 2020

A vital component of trust and transparency in intelligent systems built on machine learning and artificial intelligence is the development of clear, understandable documentation. However, such systems are notorious for their complexity and opaqueness making quality documentation a non-trivial task. Furthermore, little is known about what makes such documentation "good." In this paper, we propose and evaluate a set of quality dimensions to identify in what ways this type of documentation falls short. Then, using those dimensions, we evaluate three different approaches for eliciting intelligent system documentation. We show how the dimensions identify shortcomings in such documentation and posit how such dimensions can be use to further enable users to provide documentation that is suitable to a given persona or use case.
Automatic Microprocessor Performance Bug Detection
Venue: arXiv cs.AR 2020
Year: 2020

Processor design validation and debug is a difficult and complex task, which consumes the lion's share of the design process. Design bugs that affect processor performance rather than its functionality are especially difficult to catch, particularly in new microarchitectures. This is because, unlike functional bugs, the correct processor performance of new microarchitectures on complex, long-running benchmarks is typically not deterministically known. Thus, when performance benchmarking new microarchitectures, performance teams may assume that the design is correct when the performance of the new microarchitecture exceeds that of the previous generation, despite significant performance regressions existing in the design. In this work, we present a two-stage, machine learning-based methodology that is able to detect the existence of performance bugs in microprocessors. Our results show that our best technique detects 91.5% of microprocessor core performance bugs whose average IPC impact across the studied applications is greater than 1% versus a bug-free design with zero false positives. When evaluated on memory system bugs, our technique achieves 100% detection with zero false positives. Moreover, the detection is automatic, requiring very little performance engineer time.
Improving Calibration in Deep Metric Learning With Cross-Example Softmax
Venue: arXiv cs.LG 2020
Year: 2020

Modern image retrieval systems increasingly rely on the use of deep neural networks to learn embedding spaces in which distance encodes the relevance between a given query and image. In this setting, existing approaches tend to emphasize one of two properties. Triplet-based methods capture top-$k$ relevancy, where all top-$k$ scoring documents are assumed to be relevant to a given query Pairwise contrastive models capture threshold relevancy, where all documents scoring higher than some threshold are assumed to be relevant. In this paper, we propose Cross-Example Softmax which combines the properties of top-$k$ and threshold relevancy. In each iteration, the proposed loss encourages all queries to be closer to their matching images than all queries are to all non-matching images. This leads to a globally more calibrated similarity metric and makes distance more interpretable as an absolute measure of relevance. We further introduce Cross-Example Negative Mining, in which each pair is compared to the hardest negative comparisons across the entire batch. Empirically, we show in a series of experiments on Conceptual Captions and Flickr30k, that the proposed method effectively improves global calibration and also retrieval performance.
Multimodal Prototypical Networks for Few-shot Learning
Venue: arXiv cs.CV 2020
Year: 2020

Although providing exceptional results for many computer vision tasks, state-of-the-art deep learning algorithms catastrophically struggle in low data scenarios. However, if data in additional modalities exist (e.g. text) this can compensate for the lack of data and improve the classification results. To overcome this data scarcity, we design a cross-modal feature generation framework capable of enriching the low populated embedding space in few-shot scenarios, leveraging data from the auxiliary modality. Specifically, we train a generative model that maps text data into the visual feature space to obtain more reliable prototypes. This allows to exploit data from additional modalities (e.g. text) during training while the ultimate task at test time remains classification with exclusively visual data. We show that in such cases nearest neighbor classification is a viable approach and outperform state-of-the-art single-modal and multimodal few-shot learning methods on the CUB-200 and Oxford-102 datasets.
Towards Olfactory Information Extraction from Text: A Case Study on Detecting Smell Experiences in Novels
Venue: arXiv cs.CL 2020
Year: 2020

Environmental factors determine the smells we perceive, but societal factors factors shape the importance, sentiment and biases we give to them. Descriptions of smells in text, or as we call them `smell experiences', offer a window into these factors, but they must first be identified. To the best of our knowledge, no tool exists to extract references to smell experiences from text. In this paper, we present two variations on a semi-supervised approach to identify smell experiences in English literature. The combined set of patterns from both implementations offer significantly better performance than a keyword-based baseline.
Deep Learning Benchmarks and Datasets for Social Media Image Classification for Disaster Response
Venue: arXiv cs.CV 2020
Year: 2020

During a disaster event, images shared on social media helps crisis managers gain situational awareness and assess incurred damages, among other response tasks. Recent advances in computer vision and deep neural networks have enabled the development of models for real-time image classification for a number of tasks, including detecting crisis incidents, filtering irrelevant images, classifying images into specific humanitarian categories, and assessing the severity of damage. Despite several efforts, past works mainly suffer from limited resources (i.e., labeled images) available to train more robust deep learning models. In this study, we propose new datasets for disaster type detection, and informativeness classification, and damage severity assessment. Moreover, we relabel existing publicly available datasets for new tasks. We identify exact- and near-duplicates to form non-overlapping data splits, and finally consolidate them to create larger datasets. In our extensive experiments, we benchmark several state-of-the-art deep learning models and achieve promising results. We release our datasets and models publicly, aiming to provide proper baselines as well as to spur further research in the crisis informatics community.
Secure Location-Aware Authentication and Communication for Intelligent Transportation Systems
Venue: arXiv cs.CR 2020
Year: 2020

Intelligent transportation systems (ITS) are expected to effectively create a stand-alone network for secure communication among autonomous agents. In such a dynamic and fast-changing network with high-speed agents, verifying the authenticity and integrity of messages while taking preventive action (e.g., applying brakes) within tens of milliseconds is one of the main challenges. In such a brief moment after receiving a message, the agent not only must verify the integrity and authenticity of the received message but also needs to perform extra computations to localize the sender of the message for taking appropriate action (e.g., an immediate stop warning from a vehicle in front vs. rear). In this paper, we present an inherently location-aware and lightweight authentication protocol by exploiting in situ visual localization (i.e., SLAM). In this protocol, each agent displays its public key using visual authentication beacons (e.g., QR codes). Thus, receiving agents not only can verify and authenticate the messages but also can easily localize the sender by keeping a shortlist of observed visual beacons within their visual localization system with no additional computation cost. Compared to prior work, our location-aware protocol is scalable, does not depend on any infrastructure, removes the high cost of post-message-delivery localization, and provides trustworthiness guarantees for information that are beyond the reach of each agent sensors.
Stimulating Entrepreneurship in Teaching Human Computer Interaction
Venue: arXiv cs.HC 2020
Year: 2020

Software development requires understanding of users, user needs, user tasks and context in which they are operating. These skills are familiar to entrepreneurs, product managers, and marketing experts. However, our teaching experience suggests that students generally find these topics less attractive as they perceive them to be far too theoretical and thus, not as useful. During the years of teaching the Human Computer Interaction course we have noticed students' preferences for learning technology oriented methods, or what we refer to topics belonging to solution domain. The changes in the modernized HCI course introduced Product Market Fit canvas in order to bridge the gap between 'theoretical' and 'practical' part of the course.
Probabilistic Resource-Aware Session Types
Venue: arXiv cs.PL 2020
Year: 2020

Session types guarantee that message-passing processes adhere to predefined communication protocols. Prior work on session types has focused on deterministic languages but many message-passing systems, such as Markov chains and randomized distributed algorithms, are probabilistic. To model and analyze such systems, this article introduces probabilistic session types and explores their application in automatic expected resource analysis. Probabilistic session types describe probability distributions over messages and are a conservative extension of intuitionistic (binary) session types. To send on a probabilistic channel, processes have to utilize internal randomness from a probabilistic branching expression or external randomness from receiving on a probabilistic channel. The analysis for expected resource bounds is integrated with the type system and is a variant of automatic amortized resource analysis. It can automatically derive symbolic bounds for different cost metrics by reducing type inference to linear constraint solving. The technical contributions include the meta theory that is based on a novel nested multiverse semantics and a type-reconstruction algorithm that allows flexible mixing of different sources of randomness without burdening the programmer with type annotations. The type system has been implemented in the language PRast. Experiments demonstrate that PRast is applicable in different domains such as resource analysis of randomized distributed algorithms, verification of limiting distributions in Markov chains, and analysis of probabilistic digital contracts.
A Hierarchical Multi-Modal Encoder for Moment Localization in Video Corpus
Venue: arXiv cs.CV 2020
Year: 2020

Identifying a short segment in a long video that semantically matches a text query is a challenging task that has important application potentials in language-based video search, browsing, and navigation. Typical retrieval systems respond to a query with either a whole video or a pre-defined video segment, but it is challenging to localize undefined segments in untrimmed and unsegmented videos where exhaustively searching over all possible segments is intractable. The outstanding challenge is that the representation of a video must account for different levels of granularity in the temporal domain. To tackle this problem, we propose the HierArchical Multi-Modal EncodeR (HAMMER) that encodes a video at both the coarse-grained clip level and the fine-grained frame level to extract information at different scales based on multiple subtasks, namely, video retrieval, segment temporal localization, and masked language modeling. We conduct extensive experiments to evaluate our model on moment localization in video corpus on ActivityNet Captions and TVR datasets. Our approach outperforms the previous methods as well as strong baselines, establishing new state-of-the-art for this task.
Do 'altmetric mentions' follow Power Laws? Evidence from social media mention data in Altmetric.com
Venue: arXiv cs.DL 2020
Year: 2020

Power laws are a characteristic distribution that are ubiquitous, in that they are found almost everywhere, in both natural as well as in man-made systems. They tend to emerge in large, connected and self-organizing systems, for example, scholarly publications. Citations to scientific papers have been found to follow a power law, i.e., the number of papers having a certain level of citation x are proportional to x raised to some negative power. The distributional character of altmetrics has not been studied yet as altmetrics are among the newest indicators related to scholarly publications. Here we select a data sample from the altmetrics aggregator Altmetrics.com containing records from the platforms Facebook, Twitter, News, Blogs, etc., and the composite variable Alt-score for the period 2016. The individual and the composite data series of 'mentions' on the various platforms are fit to a power law distribution, and the parameters and goodness of fit determined using least squares regression. The log-log plot of the data, 'mentions' vs. number of papers, falls on an approximately linear line, suggesting the plausibility of a power law distribution. The fit is not very good in all cases due to large fluctuations in the tail. We show that fit to the power law can be improved by truncating the data series to eliminate large fluctuations in the tail. We conclude that altmetric distributions also follow power laws with a fairly good fit over a wide range of values. More rigorous methods of determination may not be necessary at present.
Weighted Entropy Modification for Soft Actor-Critic
Venue: arXiv cs.LG 2020
Year: 2020

We generalize the existing principle of the maximum Shannon entropy in reinforcement learning (RL) to weighted entropy by characterizing the state-action pairs with some qualitative weights, which can be connected with prior knowledge, experience replay, and evolution process of the policy. We propose an algorithm motivated for self-balancing exploration with the introduced weight function, which leads to state-of-the-art performance on Mujoco tasks despite its simplicity in implementation.
An effective method for computing Grothendieck point residue mappings
Venue: arXiv cs.SC 2020
Year: 2020

Grothendieck point residue is considered in the context of computational complex analysis. A new effective method is proposed for computing Grothendieck point residues mappings and residues. Basic ideas of our approach are the use of Grothendieck local duality and a transformation law for local cohomology classes. A new tool is devised for efficiency to solve the extended ideal membership problems in local rings. The resulting algorithms are described with an example to illustrate them. An extension of the proposed method to parametric cases is also discussed as an application.
On the Feasibility and Enhancement of the Tuple Space Explosion Attack against Open vSwitch
Venue: arXiv cs.CR 2020
Year: 2020

Being a crucial part of networked systems, packet classification has to be highly efficient; however, software switches in cloud environments still face performance challenges. The recently proposed Tuple Space Explosion (TSE) attack exploits an algorithmic deficiency in Open vSwitch (OVS). In TSE, legitimate low-rate attack traffic makes the cardinal linear search algorithm in the Tuple Space Search (TSS) algorithm to spend an unaffordable time for classifying each packet resulting in a denial-of-service (DoS) for the rest of the users. In this paper, we investigate the feasibility of TSE from multiple perspectives. Besides showing that TSE is still efficient in the newer version of OVS, we show that when the kernel datapath is compiled from a different source, it can degrade its performance to ~1% of its baseline with less than 1 Mbps attack rate. Finally, we show that TSE is much less effective against OVS-DPDK with userspace datapath due to the enhanced ranking process in its TSS implementation. Therefore, we propose TSE 2.0 to defeat the ranking process and achieve a complete DoS against OVS-DPDK. Furthermore, we present TSE 2.1, which achieves the same goal against OVS-DPDK running on multiple cores without significantly increasing the attack rate.
Dehazing Cost Volume for Deep Multi-view Stereo in Scattering Media with Airlight and Scattering Coefficient Estimation
Venue: arXiv cs.CV 2020
Year: 2020

We propose a learning-based multi-view stereo (MVS) method in scattering media, such as fog or smoke, with a novel cost volume, called the dehazing cost volume. Images captured in scattering media are degraded due to light scattering and attenuation caused by suspended particles. This degradation depends on scene depth; thus, it is difficult for traditional MVS methods to evaluate photometric consistency because the depth is unknown before three-dimensional (3D) reconstruction. The dehazing cost volume can solve this chicken-and-egg problem of depth estimation and image restoration by computing the scattering effect using swept planes in the cost volume. We also propose a method of estimating scattering parameters, such as airlight, and a scattering coefficient, which are required for our dehazing cost volume. The output depth of a network with our dehazing cost volume can be regarded as a function of these parameters; thus, they are geometrically optimized with a sparse 3D point cloud obtained at a structure-from-motion step. Experimental results on synthesized hazy images indicate the effectiveness of our dehazing cost volume against the ordinary cost volume regarding scattering media. We also demonstrated the applicability of our dehazing cost volume to real foggy scenes.
Game Plan: What AI can do for Football, and What Football can do for AI
Venue: arXiv cs.AI 2020
Year: 2020

The rapid progress in artificial intelligence (AI) and machine learning has opened unprecedented analytics possibilities in various team and individual sports, including baseball, basketball, and tennis. More recently, AI techniques have been applied to football, due to a huge increase in data collection by professional teams, increased computational power, and advances in machine learning, with the goal of better addressing new scientific challenges involved in the analysis of both individual players' and coordinated teams' behaviors. The research challenges associated with predictive and prescriptive football analytics require new developments and progress at the intersection of statistical learning, game theory, and computer vision. In this paper, we provide an overarching perspective highlighting how the combination of these fields, in particular, forms a unique microcosm for AI research, while offering mutual benefits for professional teams, spectators, and broadcasters in the years to come. We illustrate that this duality makes football analytics a game changer of tremendous value, in terms of not only changing the game of football itself, but also in terms of what this domain can mean for the field of AI. We review the state-of-the-art and exemplify the types of analysis enabled by combining the aforementioned fields, including illustrative examples of counterfactual analysis using predictive models, and the combination of game-theoretic analysis of penalty kicks with statistical learning of player attributes. We conclude by highlighting envisioned downstream impacts, including possibilities for extensions to other sports (real and virtual).