OpenRaiser/Intern-Atlas huggingface dataset:
End-to-End Video Question-Answer Generation with Generator-Pretester Network
Venue: arXiv cs.MM 2021
Year: 2021

We study a novel task, Video Question-Answer Generation (VQAG), for challenging Video Question Answering (Video QA) task in multimedia. Due to expensive data annotation costs, many widely used, large-scale Video QA datasets such as Video-QA, MSVD-QA and MSRVTT-QA are automatically annotated using Caption Question Generation (CapQG) which inputs captions instead of the video itself. As captions neither fully represent a video, nor are they always practically available, it is crucial to generate question-answer pairs based on a video via Video Question-Answer Generation (VQAG). Existing video-to-text (V2T) approaches, despite taking a video as the input, only generate a question alone. In this work, we propose a novel model Generator-Pretester Network that focuses on two components: (1) The Joint Question-Answer Generator (JQAG) which generates a question with its corresponding answer to allow Video Question "Answering" training. (2) The Pretester (PT) verifies a generated question by trying to answer it and checks the pretested answer with both the model's proposed answer and the ground truth answer. We evaluate our system with the only two available large-scale human-annotated Video QA datasets and achieves state-of-the-art question generation performances. Furthermore, using our generated QA pairs only on the Video QA task, we can surpass some supervised baselines. We apply our generated questions to Video QA applications and surpasses some supervised baselines using generated questions only. As a pre-training strategy, we outperform both CapQG and transfer learning approaches when employing semi-supervised (20%) or fully supervised learning with annotated data. These experimental results suggest the novel perspectives for Video QA training.
A Survey on Advancing the DBMS Query Optimizer: Cardinality Estimation, Cost Model, and Plan Enumeration
Venue: arXiv cs.DB 2021
Year: 2021

Query optimizer is at the heart of the database systems. Cost-based optimizer studied in this paper is adopted in almost all current database systems. A cost-based optimizer introduces a plan enumeration algorithm to find a (sub)plan, and then uses a cost model to obtain the cost of that plan, and selects the plan with the lowest cost. In the cost model, cardinality, the number of tuples through an operator, plays a crucial role. Due to the inaccuracy in cardinality estimation, errors in cost model, and the huge plan space, the optimizer cannot find the optimal execution plan for a complex query in a reasonable time. In this paper, we first deeply study the causes behind the limitations above. Next, we review the techniques used to improve the quality of the three key components in the cost-based optimizer, cardinality estimation, cost model, and plan enumeration. We also provide our insights on the future directions for each of the above aspects.
Modeling Global Semantics for Question Answering over Knowledge Bases
Venue: arXiv cs.AI 2021
Year: 2021

Semantic parsing, as an important approach to question answering over knowledge bases (KBQA), transforms a question into the complete query graph for further generating the correct logical query. Existing semantic parsing approaches mainly focus on relations matching with paying less attention to the underlying internal structure of questions (e.g., the dependencies and relations between all entities in a question) to select the query graph. In this paper, we present a relational graph convolutional network (RGCN)-based model gRGCN for semantic parsing in KBQA. gRGCN extracts the global semantics of questions and their corresponding query graphs, including structure semantics via RGCN and relational semantics (label representation of relations between entities) via a hierarchical relation attention mechanism. Experiments evaluated on benchmarks show that our model outperforms off-the-shelf models.
Janus: Efficient and Accurate Dual-radio Social Contact Detection
Venue: arXiv cs.NI 2021
Year: 2021

Determining when two individuals are within close distance is key to contain a pandemic, e.g., to alert individuals in real-time and trace their social contacts. Common approaches rely on either Bluetooth Low Energy (BLE) or ultra-wideband (UWB) radios, that nonetheless strike opposite tradeoffs for energy efficiency vs. accuracy of distance estimates. Janus reconciles these dimensions with a dual-radio protocol enabling efficient and accurate social contact detection. Measurements show that Janus achieves weeks to months of autonomous operation, depending on the configuration. Several large-scale campaigns in real-world contexts confirm its reliability and practical usefulness in enabling insightful analysis of contact data.
"Brilliant AI Doctor" in Rural China: Tensions and Challenges in AI-Powered CDSS Deployment
Venue: arXiv cs.HC 2021
Year: 2021

Artificial intelligence (AI) technology has been increasingly used in the implementation of advanced Clinical Decision Support Systems (CDSS). Research demonstrated the potential usefulness of AI-powered CDSS (AI-CDSS) in clinical decision making scenarios. However, post-adoption user perception and experience remain understudied, especially in developing countries. Through observations and interviews with 22 clinicians from 6 rural clinics in China, this paper reports the various tensions between the design of an AI-CDSS system ("Brilliant Doctor") and the rural clinical context, such as the misalignment with local context and workflow, the technical limitations and usability barriers, as well as issues related to transparency and trustworthiness of AI-CDSS. Despite these tensions, all participants expressed positive attitudes toward the future of AI-CDSS, especially acting as "a doctor's AI assistant" to realize a Human-AI Collaboration future in clinical settings. Finally we draw on our findings to discuss implications for designing AI-CDSS interventions for rural clinical contexts in developing countries.
AI based Service Management for 6G Green Communications
Venue: arXiv cs.NI 2021
Year: 2021

Green communications have always been a target for the information industry to alleviate energy overhead and reduce fossil fuel usage. In current 5G and future 6G era, there is no doubt that the volume of network infrastructure and the number of connected terminals will keep exponentially increasing, which results in the surging energy cost. It becomes growing important and urgent to drive the development of green communications. However, 6G will inevitably have increasingly stringent and diversified requirements for Quality of Service (QoS), security, flexibility, and even intelligence, all of which challenge the improvement of energy efficiency. Moreover, the dynamic energy harvesting process, which will be adopted widely in 6G, further complicates the power control and network management. To address these challenges and reduce human intervene, Artificial Intelligence (AI) has been widely recognized and acknowledged as the only solution. Academia and industry have conducted extensive research to alleviate energy demand, improve energy efficiency, and manage energy harvesting in various communication scenarios. In this paper, we present the main considerations for green communications and survey the related research on AI-based green communications. We focus on how AI techniques are adopted to manage the network and improve energy harvesting toward the green era. We analyze how state-of-the-art Machine Learning (ML) and Deep Learning (DL) techniques can cooperate with conventional AI methods and mathematical models to reduce the algorithm complexity and optimize the accuracy rate to accelerate the applications in 6G. Finally, we discuss the existing problems and envision the challenges for these emerging techniques in 6G.
Auto-Encoding Molecular Conformations
Venue: arXiv cs.LG 2021
Year: 2021

In this work we introduce an Autoencoder for molecular conformations. Our proposed model converts the discrete spatial arrangements of atoms in a given molecular graph (conformation) into and from a continuous fixed-sized latent representation. We demonstrate that in this latent representation, similar conformations cluster together while distinct conformations split apart. Moreover, by training a probabilistic model on a large dataset of molecular conformations, we demonstrate how our model can be used to generate diverse sets of energetically favorable conformations for a given molecule. Finally, we show that the continuous representation allows us to utilize optimization methods to find molecules that have conformations with favourable spatial properties.
Explainable AI for Robot Failures: Generating Explanations that Improve User Assistance in Fault Recovery
Venue: arXiv cs.AI 2021
Year: 2021

With the growing capabilities of intelligent systems, the integration of robots in our everyday life is increasing. However, when interacting in such complex human environments, the occasional failure of robotic systems is inevitable. The field of explainable AI has sought to make complex-decision making systems more interpretable but most existing techniques target domain experts. On the contrary, in many failure cases, robots will require recovery assistance from non-expert users. In this work, we introduce a new type of explanation, that explains the cause of an unexpected failure during an agent's plan execution to non-experts. In order for error explanations to be meaningful, we investigate what types of information within a set of hand-scripted explanations are most helpful to non-experts for failure and solution identification. Additionally, we investigate how such explanations can be autonomously generated, extending an existing encoder-decoder model, and generalized across environments. We investigate such questions in the context of a robot performing a pick-and-place manipulation task in the home environment. Our results show that explanations capturing the context of a failure and history of past actions, are the most effective for failure and solution identification among non-experts. Furthermore, through a second user evaluation, we verify that our model-generated explanations can generalize to an unseen office environment, and are just as effective as the hand-scripted explanations.
Theory-based Habit Modeling for Enhancing Behavior Prediction
Venue: arXiv cs.AI 2021
Year: 2021

Psychological theories of habit posit that when a strong habit is formed through behavioral repetition, it can trigger behavior automatically in the same environment. Given the reciprocal relationship between habit and behavior, changing lifestyle behaviors (e.g., toothbrushing) is largely a task of breaking old habits and creating new and healthy ones. Thus, representing users' habit strengths can be very useful for behavior change support systems (BCSS), for example, to predict behavior or to decide when an intervention reaches its intended effect. However, habit strength is not directly observable and existing self-report measures are taxing for users. In this paper, built on recent computational models of habit formation, we propose a method to enable intelligent systems to compute habit strength based on observable behavior. The hypothesized advantage of using computed habit strength for behavior prediction was tested using data from two intervention studies, where we trained participants to brush their teeth twice a day for three weeks and monitored their behaviors using accelerometers. Through hierarchical cross-validation, we found that for the task of predicting future brushing behavior, computed habit strength clearly outperformed self-reported habit strength (in both studies) and was also superior to models based on past behavior frequency (in the larger second study). Our findings provide initial support for our theory-based approach of modeling user habits and encourages the use of habit computation to deliver personalized and adaptive interventions.
Dynamic Hybrid Relation Network for Cross-Domain Context-Dependent Semantic Parsing
Venue: arXiv cs.CL 2021
Year: 2021

Semantic parsing has long been a fundamental problem in natural language processing. Recently, cross-domain context-dependent semantic parsing has become a new focus of research. Central to the problem is the challenge of leveraging contextual information of both natural language utterance and database schemas in the interaction history. In this paper, we present a dynamic graph framework that is capable of effectively modelling contextual utterances, tokens, database schemas, and their complicated interaction as the conversation proceeds. The framework employs a dynamic memory decay mechanism that incorporates inductive bias to integrate enriched contextual relation representation, which is further enhanced with a powerful reranking model. At the time of writing, we demonstrate that the proposed framework outperforms all existing models by large margins, achieving new state-of-the-art performance on two large-scale benchmarks, the SParC and CoSQL datasets. Specifically, the model attains a 55.8% question-match and 30.8% interaction-match accuracy on SParC, and a 46.8% question-match and 17.0% interaction-match accuracy on CoSQL.
An XR rapid prototyping framework for interoperability across the reality spectrum
Venue: arXiv cs.GR 2021
Year: 2021

Applications of the Extended Reality (XR) spectrum, a superset of Mixed, Augmented and Virtual Reality, are gaining prominence and can be employed in a variety of areas, such as virtual museums. Examples can be found in the areas of education, cultural heritage, health/treatment, entertainment, marketing, and more. The majority of computer graphics applications nowadays are used to operate only in one of the above realities. The lack of applications across the XR spectrum is a real shortcoming. There are many advantages resulting from this problem's solution. Firstly, releasing an application across the XR spectrum could contribute in discovering its most suitable reality. Moreover, an application could be more immersive within a particular reality, depending on its context. Furthermore, its availability increases to a broader range of users. For instance, if an application is released both in Virtual and Augmented Reality, it is accessible to users that may lack the possession of a VR headset, but not of a mobile AR device. The question that arises at this point, would be "Is it possible for a full s/w application stack to be converted across XR without sacrificing UI/UX in a semi-automatic way?". It may be quite difficult, depending on the architecture and application implementation. Most companies nowadays support only one reality, due to their lack of UI/UX software architecture or resources to support the complete XR spectrum. In this work, we present an "automatic reality transition" in the context of virtual museum applications. We propose a development framework, which will automatically allow this XR transition. This framework transforms any XR project into different realities such as Augmented or Virtual. It also reduces the development time while increasing the XR availability of 3D applications, encouraging developers to release applications across the XR spectrum.
Risk markers by sex for in-hospital mortality in patients with acute coronary syndrome: a machine learning approach
Venue: arXiv cs.LG 2021
Year: 2021

Background: Several studies have highlighted the importance of considering sex differences in the diagnosis and treatment of Acute Coronary Syndrome (ACS). However, the identification of sex-specific risk markers in ACS sub-populations has been scarcely studied. The present study aims to explore machine learning (ML) models to identify in-hospital mortality markers for women and men in ACS sub-populations collected from a public database of electronic health records (EHR). Methods: We extracted 1,299 patients with ST-elevation myocardial infarction (STEMI) and 2,820 patients with non-ST-elevation myocardial infarction (NSTEMI) from the Medical Information Mart for Intensive Care (MIMIC)-III database. We trained and validated mortality prediction models and used an interpretability technique to identify sex-specific markers for each sub-population. Results: The models based on eXtreme Gradient Boosting (XGBoost) achieved the highest performance: area under the curve (AUC) = 0.94 (95\% CI:0.84-0.96) for STEMI and AUC = 0.94 (95\% CI:0.80-0.90) for NSTEMI. For STEMI, the top markers in women are chronic kidney failure, high heart rate, and age over 70 years. For men, the top markers are acute kidney failure, high troponin T levels, and age over 75 years. However, for NSTEMI, the top markers in women are low troponin levels, high urea levels, and age over 80 years. For men, the top markers are high heart rate, creatinine levels, and age over 70 years. Conclusions: Our results show possible significant and coherent sex-specific risk markers of different ACS sub-populations by interpreting ML mortality models trained on EHRs. Differences are observed in the identified risk markers between women and men, highlighting the importance of considering sex-specific markers in implementing more appropriate treatment strategies and better clinical outcomes.
Confluence up to Garbage in Graph Transformation
Venue: arXiv cs.LO 2021
Year: 2021

The transformation of graphs and graph-like structures is ubiquitous in computer science. When a system is described by graph-transformation rules, it is often desirable that the rules are both terminating and confluent so that rule applications in an arbitrary order produce unique resulting graphs. However, there are application scenarios where the rules are not globally confluent but confluent on a subclass of graphs that are of interest. In other words, non-resolvable conflicts can only occur on graphs that are considered as "garbage". In this paper, we introduce the notion of confluence up to garbage and generalise Plump's critical pair lemma for double-pushout graph transformation, providing a sufficient condition for confluence up to garbage by non-garbage critical pair analysis. We apply our results in two case studies about efficient language recognition: we present backtracking-free graph reduction systems which recognise a class of flow diagrams and a class of labelled series-parallel graphs, respectively. Both systems are non-confluent but confluent up to garbage. We also give a critical pair condition for subcommutativity up to garbage which, together with closedness, implies confluence up to garbage even in non-terminating systems.
Deep Learning for Fast and Reliable Initial Access in AI-Driven 6G mmWave Networks
Venue: arXiv cs.NI 2021
Year: 2021

We present DeepIA, a deep neural network (DNN) framework for enabling fast and reliable initial access for AI-driven beyond 5G and 6G millimeter (mmWave) networks. DeepIA reduces the beam sweep time compared to a conventional exhaustive search-based IA process by utilizing only a subset of the available beams. DeepIA maps received signal strengths (RSSs) obtained from a subset of beams to the beam that is best oriented to the receiver. In both line of sight (LoS) and non-line of sight (NLoS) conditions, DeepIA reduces the IA time and outperforms the conventional IA's beam prediction accuracy. We show that the beam prediction accuracy of DeepIA saturates with the number of beams used for IA and depends on the particular selection of the beams. In LoS conditions, the selection of the beams is consequential and improves the accuracy by up to 70%. In NLoS situations, it improves accuracy by up to 35%. We find that, averaging multiple RSS snapshots further reduces the number of beams needed and achieves more than 95% accuracy in both LoS and NLoS conditions. Finally, we evaluate the beam prediction time of DeepIA through embedded hardware implementation and show the improvement over the conventional beam sweeping.
One-shot Policy Elicitation via Semantic Reward Manipulation
Venue: arXiv cs.RO 2021
Year: 2021

Synchronizing expectations and knowledge about the state of the world is an essential capability for effective collaboration. For robots to effectively collaborate with humans and other autonomous agents, it is critical that they be able to generate intelligible explanations to reconcile differences between their understanding of the world and that of their collaborators. In this work we present Single-shot Policy Explanation for Augmenting Rewards (SPEAR), a novel sequential optimization algorithm that uses semantic explanations derived from combinations of planning predicates to augment agents' reward functions, driving their policies to exhibit more optimal behavior. We provide an experimental validation of our algorithm's policy manipulation capabilities in two practically grounded applications and conclude with a performance analysis of SPEAR on domains of increasingly complex state space and predicate counts. We demonstrate that our method makes substantial improvements over the state-of-the-art in terms of runtime and addressable problem size, enabling an agent to leverage its own expertise to communicate actionable information to improve another's performance.
The data synergy effects of time-series deep learning models in hydrology
Venue: arXiv cs.LG 2021
Year: 2021

When fitting statistical models to variables in geoscientific disciplines such as hydrology, it is a customary practice to regionalize - to divide a large spatial domain into multiple regions and study each region separately - instead of fitting a single model on the entire data (also known as unification). Traditional wisdom in these fields suggests that models built for each region separately will have higher performance because of homogeneity within each region. However, by partitioning the training data, each model has access to fewer data points and cannot learn from commonalities between regions. Here, through two hydrologic examples (soil moisture and streamflow), we argue that unification can often significantly outperform regionalization in the era of big data and deep learning (DL). Common DL architectures, even without bespoke customization, can automatically build models that benefit from regional commonality while accurately learning region-specific differences. We highlight an effect we call data synergy, where the results of the DL models improved when data were pooled together from characteristically different regions. In fact, the performance of the DL models benefited from more diverse rather than more homogeneous training data. We hypothesize that DL models automatically adjust their internal representations to identify commonalities while also providing sufficient discriminatory information to the model. The results here advocate for pooling together larger datasets, and suggest the academic community should place greater emphasis on data sharing and compilation.
sGUARD: Towards Fixing Vulnerable Smart Contracts Automatically
Venue: arXiv cs.CR 2021
Year: 2021

Smart contracts are distributed, self-enforcing programs executing on top of blockchain networks. They have the potential to revolutionize many industries such as financial institutes and supply chains. However, smart contracts are subject to code-based vulnerabilities, which casts a shadow on its applications. As smart contracts are unpatchable (due to the immutability of blockchain), it is essential that smart contracts are guaranteed to be free of vulnerabilities. Unfortunately, smart contract languages such as Solidity are Turing-complete, which implies that verifying them statically is infeasible. Thus, alternative approaches must be developed to provide the guarantee. In this work, we develop an approach which automatically transforms smart contracts so that they are provably free of 4 common kinds of vulnerabilities. The key idea is to apply runtime verification in an efficient and provably correct manner. Experiment results with 5000 smart contracts show that our approach incurs minor run-time overhead in terms of time (i.e., 14.79%) and gas (i.e., 0.79%).