OpenRaiser/Intern-Atlas huggingface dataset:
Relating Structure and Power: Extended Version
Venue: arXiv cs.LO 2021
Year: 2021

Combinatorial games are widely used in finite model theory, constraint satisfaction, modal logic and concurrency theory to characterize logical equivalences between structures. In particular, Ehrenfeucht-Fraisse games, pebble games, and bisimulation games play a central role. We show how each of these types of games can be described in terms of an indexed family of comonads on the category of relational structures and homomorphisms. The index $k$ is a resource parameter which bounds the degree of access to the underlying structure. The coKleisli categories for these comonads can be used to give syntax-free characterizations of a wide range of important logical equivalences. Moreover, the coalgebras for these indexed comonads can be used to characterize key combinatorial parameters: tree-depth for the Ehrenfeucht-Fraisse comonad, tree-width for the pebbling comonad, and synchronization-tree depth for the modal unfolding comonad. These results pave the way for systematic connections between two major branches of the field of logic in computer science which hitherto have been almost disjoint: categorical semantics, and finite and algorithmic model theory.
Geomechanical simulation of energy storage in salt formations
Venue: arXiv cs.CE 2021
Year: 2021

A promising option for storing large-scale quantities of green gases (e.g., hydrogen) is in subsurface rock salt caverns. The mechanical performance of salt caverns utilized for long-term subsurface energy storage plays a significant role in long-term stability and serviceability. However, rock salt undergoes non-linear creep deformation due to long-term loading caused by subsurface storage. Salt caverns have complex geometries and the geological domain surrounding salt caverns has a vast amount of material heterogeneity. To safely store gases in caverns, a thorough analysis of the geological domain becomes crucial. To date, few studies have attempted to analyze the influence of geometrical and material heterogeneity on the state of stress in salt caverns subjected to long-term loading. In this work, we present a rigorous and systematic modeling study to quantify the impact of heterogeneity on the deformation of salt caverns and quantify the state of stress around the caverns. A 2D finite element simulator was developed to consistently account for the non-linear creep deformation and also to model tertiary creep. The computational scheme was benchmarked with the already existing experimental study. The impact of cyclic loading on the cavern was studied considering maximum and minimum pressure that depends on lithostatic pressure. The influence of geometric heterogeneity such as irregularly-shaped caverns and material heterogeneity, which involves different elastic and creep properties of the different materials in the geological domain, is rigorously studied and quantified. Moreover, multi-cavern simulations are conducted to investigate the influence of a cavern on the adjacent caverns.
Intelligent Reflecting Surface-Assisted Wireless Key Generation for Low-Entropy Environments
Venue: arXiv cs.CR 2021
Year: 2021

Physical layer key generation is a promising candidate for cryptographic key establishment between two wireless communication parties. It offers information-theoretic security and is an attractive alternative to public-key techniques. Here, the inherent randomness of wireless radio channels is used as a shared entropy source to generate cryptographic key material. However, practical implementations often suffer from static channel conditions which exhibit a limited amount of randomness. In the past, considerable research efforts have been made to address this fundamental limitation. However, current solutions are not generic or require dedicated hardware extensions such as reconfigurable antennas. In this paper, we propose a novel wireless key generation architecture based on randomized channel responses from an intelligent reflecting surface (IRS). Due to its passive nature, a cooperative IRS is well-suited to provide randomness for conventional resource-constrained radios. We conduct the first practical studies to successfully demonstrate IRS-based physical-layer key generation with an OFDM system. In a static environment, using a single subcarrier only, our IRS-assisted prototype system achieves a key generation rate (KGR) of 97.39 bps with 6.5% key disagreement rate (KDR) after quantization, while passing standard randomness tests.
Joint Constrained Learning for Event-Event Relation Extraction
Venue: arXiv cs.CL 2021
Year: 2021

Understanding natural language involves recognizing how multiple event mentions structurally and temporally interact with each other. In this process, one can induce event complexes that organize multi-granular events with temporal order and membership relations interweaving among them. Due to the lack of jointly labeled data for these relational phenomena and the restriction on the structures they articulate, we propose a joint constrained learning framework for modeling event-event relations. Specifically, the framework enforces logical constraints within and across multiple temporal and subevent relations by converting these constraints into differentiable learning objectives. We show that our joint constrained learning approach effectively compensates for the lack of jointly labeled data, and outperforms SOTA methods on benchmarks for both temporal relation extraction and event hierarchy construction, replacing a commonly used but more expensive global inference process. We also present a promising case study showing the effectiveness of our approach in inducing event complexes on an external corpus.
Self-Imitation Learning for Robot Tasks with Sparse and Delayed Rewards
Venue: arXiv cs.LG 2021
Year: 2021

The application of reinforcement learning (RL) in robotic control is still limited in the environments with sparse and delayed rewards. In this paper, we propose a practical self-imitation learning method named Self-Imitation Learning with Constant Reward (SILCR). Instead of requiring hand-defined immediate rewards from environments, our method assigns the immediate rewards at each timestep with constant values according to their final episodic rewards. In this way, even if the dense rewards from environments are unavailable, every action taken by the agents would be guided properly. We demonstrate the effectiveness of our method in some challenging continuous robotics control tasks in MuJoCo simulation and the results show that our method significantly outperforms the alternative methods in tasks with sparse and delayed rewards. Even compared with alternatives with dense rewards available, our method achieves competitive performance. The ablation experiments also show the stability and reproducibility of our method.
A Predictive Visual Analytics System for Studying Neurodegenerative Disease based on DTI Fiber Tracts
Venue: arXiv cs.CV 2021
Year: 2021

Diffusion tensor imaging (DTI) has been used to study the effects of neurodegenerative diseases on neural pathways, which may lead to more reliable and early diagnosis of these diseases as well as a better understanding of how they affect the brain. We introduce an intelligent visual analytics system for studying patient groups based on their labeled DTI fiber tract data and corresponding statistics. The system's AI-augmented interface guides the user through an organized and holistic analysis space, including the statistical feature space, the physical space, and the space of patients over different groups. We use a custom machine learning pipeline to help narrow down this large analysis space, and then explore it pragmatically through a range of linked visualizations. We conduct several case studies using real data from the research database of Parkinson's Progression Markers Initiative.
A Systematic Review of Online Exams Solutions in E-learning: Techniques, Tools and Global Adoption
Venue: arXiv cs.CY 2021
Year: 2021

E-learning in higher education is exponentially increased during the past decade due to its inevitable benefits in critical situations like natural disasters, and pandemic. The reliable, fair, and seamless execution of online exams in E-learning is highly significant. Particularly, online exams are conducted on E-learning platforms without the physical presence of students and instructors at the same place. This poses several issues like integrity and security during online exams. To address such issues, researchers frequently proposed different techniques and tools. However, a study summarizing and analyzing latest developments, particularly in the area of online examination, is hard to find in the literature. In this article, an SLR for online examination is performed to select and analyze 53 studies published during the last five years. Subsequently, five leading online exams features targeted in the selected studies are identified and underlying development approaches for the implementation of online exams solutions are explored. Furthermore, 16 important techniques and 11 datasets are presented. In addition, 21 online exams tools proposed in the selected studies are identified. Additionally, 25 leading existing tools used in the selected studies are also presented. Finally, the participation of countries in online exam research is investigated. Key factors for the global adoption of online exams are identified and investigated. This facilitates the selection of right online exam system for a particular country on the basis of existing E-learning infrastructure and overall cost. To conclude, the findings of this article provide a solid platform for the researchers and practitioners of the domain to select appropriate features along with underlying development approaches, tools and techniques for the implementation of a particular online exams solution as per given requirements.
Decoding of Interleaved Alternant Codes
Venue: arXiv cs.IT 2021
Year: 2021

Interleaved Reed-Solomon codes admit efficient decoding algorithms which correct burst errors far beyond half the minimum distance in the random errors regime, e.g., by computing a common solution to the Key Equation for each Reed-Solomon code, as described by Schmidt et al. If this decoder does not succeed, it may either fail to return a codeword or miscorrect to an incorrect codeword, and good upper bounds on the fraction of error matrices for which these events occur are known. The decoding algorithm immediately applies to interleaved alternant codes as well, i.e., the subfield subcodes of interleaved Reed-Solomon codes, but the fraction of decodable error matrices differs, since the error is now restricted to a subfield. In this paper, we present new general lower and upper bounds on the fraction of error matrices decodable by Schmidt et al.'s decoding algorithm, thereby making it the only decoding algorithm for interleaved alternant codes for which such bounds are known.
Geometry matters: Exploring language examples at the decision boundary
Venue: arXiv cs.CL 2021
Year: 2021

A growing body of recent evidence has highlighted the limitations of natural language processing (NLP) datasets and classifiers. These include the presence of annotation artifacts in datasets, classifiers relying on shallow features like a single word (e.g., if a movie review has the word "romantic", the review tends to be positive), or unnecessary words (e.g., learning a proper noun to classify a movie as positive or negative). The presence of such artifacts has subsequently led to the development of challenging datasets to force the model to generalize better. While a variety of heuristic strategies, such as counterfactual examples and contrast sets, have been proposed, the theoretical justification about what makes these examples difficult for the classifier is often lacking or unclear. In this paper, using tools from information geometry, we propose a theoretical way to quantify the difficulty of an example in NLP. Using our approach, we explore difficult examples for several deep learning architectures. We discover that both BERT, CNN and fasttext are susceptible to word substitutions in high difficulty examples. These classifiers tend to perform poorly on the FIM test set. (generated by sampling and perturbing difficult examples, with accuracy dropping below 50%). We replicate our experiments on 5 NLP datasets (YelpReviewPolarity, AGNEWS, SogouNews, YelpReviewFull and Yahoo Answers). On YelpReviewPolarity we observe a correlation coefficient of -0.4 between resilience to perturbations and the difficulty score. Similarly we observe a correlation of 0.35 between the difficulty score and the empirical success probability of random substitutions. Our approach is simple, architecture agnostic and can be used to study the fragilities of text classification models. All the code used will be made publicly available, including a tool to explore the difficult examples for other datasets.
Back to the Future: Cycle Encoding Prediction for Self-supervised Contrastive Video Representation Learning
Venue: arXiv cs.CV 2021
Year: 2021

In this paper we show that learning video feature spaces in which temporal cycles are maximally predictable benefits action classification. In particular, we propose a novel learning approach termed Cycle Encoding Prediction (CEP) that is able to effectively represent high-level spatio-temporal structure of unlabelled video content. CEP builds a latent space wherein the concept of closed forward-backward as well as backward-forward temporal loops is approximately preserved. As a self-supervision signal, CEP leverages the bi-directional temporal coherence of the video stream and applies loss functions that encourage both temporal cycle closure as well as contrastive feature separation. Architecturally, the underpinning network structure utilises a single feature encoder for all video snippets, adding two predictive modules that learn temporal forward and backward transitions. We apply our framework for pretext training of networks for action recognition tasks. We report significantly improved results for the standard datasets UCF101 and HMDB51. Detailed ablation studies support the effectiveness of the proposed components. We publish source code for the CEP components in full with this paper.
An Evasion Attack against Stacked Capsule Autoencoder
Venue: arXiv cs.LG 2021
Year: 2021

Capsule network is a type of neural network that uses the spatial relationship between features to classify images. By capturing the poses and relative positions between features, its ability to recognize affine transformation is improved, and it surpasses traditional convolutional neural networks (CNNs) when handling translation, rotation and scaling. The Stacked Capsule Autoencoder (SCAE) is the state-of-the-art capsule network. The SCAE encodes an image as capsules, each of which contains poses of features and their correlations. The encoded contents are then input into the downstream classifier to predict the categories of the images. Existing research mainly focuses on the security of capsule networks with dynamic routing or EM routing, and little attention has been given to the security and robustness of the SCAE. In this paper, we propose an evasion attack against the SCAE. After a perturbation is generated based on the output of the object capsules in the model, it is added to an image to reduce the contribution of the object capsules related to the original category of the image so that the perturbed image will be misclassified. We evaluate the attack using an image classification experiment, and the experimental results indicate that the attack can achieve high success rates and stealthiness. It confirms that the SCAE has a security vulnerability whereby it is possible to craft adversarial samples without changing the original structure of the image to fool the classifiers. We hope that our work will make the community aware of the threat of this attack and raise the attention given to the SCAE's security.
Reproducing GW150914: the first observation of gravitational waves from a binary black hole merger
Venue: arXiv cs.DC 2021
Year: 2021

In 2016, LIGO and Virgo announced the first observation of gravitational waves from a binary black hole merger, known as GW150914. To establish the confidence of this detection, large-scale scientific workflows were used to measure the event's statistical significance. They used code written by the LIGO/Virgo and were executed on the LIGO Data Grid. The codes are publicly available, but there has not yet been an attempt to directly reproduce the results, although several analyses have replicated the analysis, confirming the detection. We attempt to reproduce the result presented in the GW150914 discovery paper using publicly available code on the Open Science Grid. We show that we can reproduce the main result but we cannot exactly reproduce the LIGO analysis as the original data set used is not public. We discuss the challenges we encountered and make recommendations for scientists who wish to make their work reproducible.
Reducing the Teacher-Student Gap via Spherical Knowledge Disitllation
Venue: arXiv cs.LG 2021
Year: 2021

Knowledge distillation aims at obtaining a compact and effective model by learning the mapping function from a much larger one. Due to the limited capacity of the student, the student would underfit the teacher. Therefore, student performance would unexpectedly drop when distilling from an oversized teacher, termed the capacity gap problem. We investigate this problem by study the gap of confidence between teacher and student. We find that the magnitude of confidence is not necessary for knowledge distillation and could harm the student performance if the student are forced to learn confidence. We propose Spherical Knowledge Distillation to eliminate this gap explicitly, which eases the underfitting problem. We find this novel knowledge representation can improve compact models with much larger teachers and is robust to temperature. We conducted experiments on both CIFAR100 and ImageNet, and achieve significant improvement. Specifically, we train ResNet18 to 73.0 accuracy, which is a substantial improvement over previous SOTA and is on par with resnet34 almost twice the student size. The implementation has been shared at https://github.com/forjiuzhou/Spherical-Knowledge-Distillation.
Revisiting Projection-free Online Learning: the Strongly Convex Case
Venue: arXiv cs.LG 2021
Year: 2021

Projection-free optimization algorithms, which are mostly based on the classical Frank-Wolfe method, have gained significant interest in the machine learning community in recent years due to their ability to handle convex constraints that are popular in many applications, but for which computing projections is often computationally impractical in high-dimensional settings, and hence prohibit the use of most standard projection-based methods. In particular, a significant research effort was put on projection-free methods for online learning. In this paper we revisit the Online Frank-Wolfe (OFW) method suggested by Hazan and Kale \cite{Hazan12} and fill a gap that has been left unnoticed for several years: OFW achieves a faster rate of $O(T^{2/3})$ on strongly convex functions (as opposed to the standard $O(T^{3/4})$ for convex but not strongly convex functions), where $T$ is the sequence length. This is somewhat surprising since it is known that for offline optimization, in general, strong convexity does not lead to faster rates for Frank-Wolfe. We also revisit the bandit setting under strong convexity and prove a similar bound of $\tilde O(T^{2/3})$ (instead of $O(T^{3/4})$ without strong convexity). Hence, in the current state-of-affairs, the best projection-free upper-bounds for the full-information and bandit settings with strongly convex and nonsmooth functions match up to logarithmic factors in $T$.
GMH: A General Multi-hop Reasoning Model for KG Completion
Venue: arXiv cs.AI 2021
Year: 2021

Knowledge graphs are essential for numerous downstream natural language processing applications, but are typically incomplete with many facts missing. This results in research efforts on multi-hop reasoning task, which can be formulated as a search process and current models typically perform short distance reasoning. However, the long-distance reasoning is also vital with the ability to connect the superficially unrelated entities. To the best of our knowledge, there lacks a general framework that approaches multi-hop reasoning in mixed long-short distance reasoning scenarios. We argue that there are two key issues for a general multi-hop reasoning model: i) where to go, and ii) when to stop. Therefore, we propose a general model which resolves the issues with three modules: 1) the local-global knowledge module to estimate the possible paths, 2) the differentiated action dropout module to explore a diverse set of paths, and 3) the adaptive stopping search module to avoid over searching. The comprehensive results on three datasets demonstrate the superiority of our model with significant improvements against baselines in both short and long distance reasoning scenarios.
Inducing Alignment Structure with Gated Graph Attention Networks for Sentence Matching
Venue: arXiv cs.CL 2021
Year: 2021

Sentence matching is a fundamental task of natural language processing with various applications. Most recent approaches adopt attention-based neural models to build word- or phrase-level alignment between two sentences. However, these models usually ignore the inherent structure within the sentences and fail to consider various dependency relationships among text units. To address these issues, this paper proposes a graph-based approach for sentence matching. First, we represent a sentence pair as a graph with several carefully design strategies. We then employ a novel gated graph attention network to encode the constructed graph for sentence matching. Experimental results demonstrate that our method substantially achieves state-of-the-art performance on two datasets across tasks of natural language and paraphrase identification. Further discussions show that our model can learn meaningful graph structure, indicating its superiority on improved interpretability.
Deep Conditional Transformation Models
Venue: arXiv cs.LG 2021
Year: 2021

Learning the cumulative distribution function (CDF) of an outcome variable conditional on a set of features remains challenging, especially in high-dimensional settings. Conditional transformation models provide a semi-parametric approach that allows to model a large class of conditional CDFs without an explicit parametric distribution assumption and with only a few parameters. Existing estimation approaches within this class are, however, either limited in their complexity and applicability to unstructured data sources such as images or text, lack interpretability, or are restricted to certain types of outcomes. We close this gap by introducing the class of deep conditional transformation models which unifies existing approaches and allows to learn both interpretable (non-)linear model terms and more complex neural network predictors in one holistic framework. To this end we propose a novel network architecture, provide details on different model definitions and derive suitable constraints as well as network regularization terms. We demonstrate the efficacy of our approach through numerical experiments and applications.