Reviewerly/exHarmony huggingface dataset:
Expanding End-to-End Question Answering on Differentiable Knowledge Graphs with Intersection

End-to-end question answering using a differentiable knowledge graph is a promising technique that requires only weak supervision, produces interpretable results, and is fully differentiable.Previous implementations of this technique (Cohen et al., 2020) have focused on single-entity questions using a relation following operation.In this paper, we propose a model that explicitly handles multiple-entity questions by implementing a new intersection operation, which identifies the shared elements between two sets of entities.We find that introducing intersection improves performance over a baseline model on two datasets, WebQuestionsSP (69.6% to 73.3% Hits@1) and ComplexWebQuestions (39.8% to 48.7% Hits@1), and in particular, improves performance on questions with multiple entities by over 14% on WebQuestionsSP and by 19% on ComplexWebQuestions.
Explainable Online Deep Neural Network Selection Using Adaptive Saliency Maps for Time Series Forecasting

Deep neural networks such as Convolutional Neural Networks (CNNs) have been successfully applied to a wide variety of tasks, including time series forecasting. In this paper, we propose a novel approach for online deep CNN selection using saliency maps in the task of time series forecasting. We start with an arbitrarily set of different CNN forecasters with various architectures. Then, we outline a gradient-based technique for generating saliency maps with a coherent design to make it able to specialize the CNN forecasters across different regions in the input time series using a performance-based ranking. In this framework, the selection of the adequate model is performed in an online fashion and the computation of saliency maps responsible for the model selection is achieved adaptively following drift detection in the time series. In addition, the saliency maps can be exploited to provide suitable explanations for the reason behind selecting a specific model at a certain time interval or instant. An extensive empirical study on various real-world datasets demonstrates that our method achieves excellent or on par results in comparison to the state-of-the-art approaches as well as several baselines.
Deep Multiple Instance Hashing for Fast Multi-Object Image Search

Multi-keyword query is widely supported in text search engines. However, an analogue in image retrieval systems, multi-object query, is rarely studied. Meanwhile, traditional object-based image retrieval methods often involve multiple steps separately. In this work, we propose a weakly-supervised Deep Multiple Instance Hashing (DMIH) approach for multi-object image retrieval. Our DMIH approach, which leverages a popular CNN model to build the end-to-end relation between a raw image and the binary hash codes of its multiple objects, can support multi-object queries effectively and integrate object detection with hashing learning seamlessly. We treat object detection as a binary multiple instance learning (MIL) problem and such instances are automatically extracted from multi-scale convolutional feature maps. We also design a conditional random field (CRF) module to capture both the semantic and spatial relations among different class labels. For hashing training, we sample image pairs to learn their semantic relationships in terms of hash codes of the most probable proposals for owned labels as guided by object predictors. The two objectives benefit each other in a multi-task learning scheme. Finally, a two-level inverted index method is proposed to further speed up the retrieval of multi-object queries. Our DMIH approach outperforms state-of-the-arts on public benchmarks for object-based image retrieval and achieves promising results for multi-object queries.
A Novel Multiple-View Adversarial Learning Network for Unsupervised Domain Adaptation Action Recognition

Abstract-domain adaptation action recognition is a hot research topic in machine learning and some effective approaches have been proposed. However, samples in the target domain with label information are often required by these approaches. Moreover, domain-invariant discriminative feature learning, feature fusion, and classifier module learning have not been explored in an end-to-end framework. Thus, in this study, we propose a novel end-to-end multiple-view adversarial learning network (MAN) for unsupervised domain adaptation action recognition in which the fusion of RGB and optical-flow features, domain-invariant discrimination feature learning, and action recognition is conducted in a unified framework. Specifically, a robust spatiotemporal feature extraction network, including a spatial transform network and an adaptive intrachannel weight network, is proposed to improve the scale invariance and robustness of the method. Then, a self-attention mechanism fusion module is designed to adaptively fuse the RGB and optical-flow features. Moreover, a multiview adversarial learning loss is developed to obtain domain-invariant discriminative features. In addition, three benchmark datasets are constructed for unsupervised domain adaptation action recognition, for which all actions and samples are carefully collected from public action datasets, and their action categories are hierarchically augmented, which can guide how to extend existing action datasets. We conduct extensive experiments on four benchmark datasets, and the experimental results demonstrate that our proposed MAN can outperform several state-of-the-art unsupervised domain adaptation action recognition approaches. When the SDAI Action II-6 and SDAI Action II-11 datasets are used, MAN can achieve 3.7% ( H → U ) and 6.1% ( H → U ) improvements over the temporal attentive adversarial adaptation network (published in ICCV 2019) module, respectively. As an added contribution, the SDAI Action II-6, SDAI Action II-11, and SDAI Action II-16 datasets will be released to facilitate future research on domain adaptation action recognition.
The Passenger Experience of Mixed Reality Virtual Display Layouts in Airplane Environments

Augmented / Mixed Reality headsets will in-time see adoption and use in a variety of mobility and transit contexts, allowing users to view and interact with virtual content and displays for productivity and entertainment. However, little is known regarding how multi-display virtual workspaces should be presented in a transit context, nor to what extent the unique affordances of transit environments (e.g. the social presence of others) might influence passenger perception of virtual display layouts. Using a simulated VR passenger airplane environment, we evaluated three different AR-driven virtual display configurations (Horizontal, Vertical, and Focus main display with smaller secondary windows) at two different depths, exploring their usability, user preferences, and the underlying factors that influenced those preferences. We found that the perception of invading other’s personal space significantly influenced preferred layouts in transit contexts. Based on our findings, we reflect on the unique challenges posed by passenger contexts, provide recommendations regarding virtual display layout in the confined airplane environment, and expand on the significant benefits that AR offers over physical displays in said environments.
Evaluation of a machine learning classifier for metamodels

Abstract Modeling is a ubiquitous activity in the process of software development. In recent years, such an activity has reached a high degree of intricacy, guided by the heterogeneity of the components, data sources, and tasks. The democratized use of models has led to the necessity for suitable machinery for mining modeling repositories. Among others, the classification of metamodels into independent categories facilitates personalized searches by boosting the visibility of metamodels. Nevertheless, the manual classification of metamodels is not only a tedious but also an error-prone task. According to our observation, misclassification is the norm which leads to a reduction in reachability as well as reusability of metamodels. Handling such complexity requires suitable tooling to leverage raw data into practical knowledge that can help modelers with their daily tasks. In our previous work, we proposed AURORA as a machine learning classifier for metamodel repositories. In this paper, we present a thorough evaluation of the system by taking into consideration different settings as well as evaluation metrics. More importantly, we improve the original AURORA tool by changing its internal design. Experimental results demonstrate that the proposed amendment is beneficial to the classification of metamodels. We also compared our approach with two baseline algorithms, namely gradient boosted decision tree and support vector machines. Eventually, we see that AURORA outperforms the baselines with respect to various quality metrics.
SPIN: A Blockchain-Based Framework for Sharing COVID-19 Pandemic Information across Nations

The COVID-19 pandemic has caused many countries around the globe to put strict policies and measures in place in an attempt to control the rapid spread of the virus. These measures have affected economic activities and have impacted a broad range of businesses, such as international traveling, restaurants, and shopping malls. As COVID-19 vaccination efforts progress, countries are starting to relax international travel constraints and permit passengers from certain destinations to cross the border. Moreover, travelers from those destinations are likely required to provide certificates of vaccination results or negative COVID-19 tests before crossing the borders. Implementing these travel guidelines requires sharing information between countries, such as the number of COVID-19 cases and vaccination certificates for travelers. In this paper, we introduce SPIN, a framework leveraging a permissioned blockchain for sharing COVID-19 information between countries. This includes public data, such as the number of vaccinated people, and private data, such as vaccination certificates for individuals. Additionally, we employ cancelable fingerprint templates to authenticate private information about travelers. We analyze the framework from scalability, efficiency, security, and privacy perspectives. To validate our framework, we provide a prototype implementation using the Hyperledger Fabric platform.
Optical random phase dropout in a diffractive deep neural network

Unitary learning is a backpropagation (BP) method that serves to update unitary weights in fully connected deep complex-valued neural networks, meeting a prior unitary in an active modulation diffractive deep neural network. However, the square matrix characteristic of unitary weights in each layer results in its learning belonging to a small-sample training, which produces an almost useless network that has a fairly poor generalization capability. To alleviate such a serious over-fitting problem, in this Letter, optical random phase dropout is formulated and designed. The equivalence between unitary forward and diffractive networks deduces a synthetic mask that is seamlessly compounded with a computational modulation and a random sampling comb called dropout. The dropout is filled with random phases in its zero positions that satisfy the Bernoulli distribution, which could slightly deflect parts of transmitted optical rays in each output end to generate statistical inference networks. The enhancement of generalization benefits from the fact that massively parallel full connection with different optical links is involved in the training. The random phase comb introduced into unitary BP is in the form of conjugation, which indicates the significance of optical BP.
Perturbation CheckLists for Evaluating NLG Evaluation Metrics

Natural Language Generation (NLG) evaluation is a multifaceted task requiring assessment of multiple desirable criteria, e.g., fluency, coherency, coverage, relevance, adequacy, overall quality, etc. Across existing datasets for 6 NLG tasks, we observe that the human evaluation scores on these multiple criteria are often not correlated.For example, there is a very low correlation between human scores on fluency and data coverage for the task of structured data to text generation.This suggests that the current recipe of proposing new automatic evaluation metrics for NLG by showing that they correlate well with scores assigned by humans for a single criteria (overall quality) alone is inadequate.Indeed, our extensive study involving 25 automatic evaluation metrics across 6 different tasks and 18 different evaluation criteria shows that there is no single metric which correlates well with human scores on all desirable criteria, for most NLG tasks.Given this situation, we propose CheckLists for better design and evaluation of automatic metrics.We design templates which target a specific criteria (e.g., coverage) and perturb the output such that the quality gets affected only along this specific criteria (e.g., the coverage drops).We show that existing evaluation metrics are not robust against even such simple perturbations and disagree with scores assigned by humans to the perturbed output.The proposed templates thus allow for a fine-grained assessment of automatic evaluation metrics exposing their limitations and will facilitate better design, analysis and evaluation of such metrics. 1 Task Criteria Machine TranslationAdequacy: The generated translation should adequately represent all the information present in the reference. Question GenerationRelevance: Is the question related to the source material they are based upon.Answerability: Is the generated question answerable given the context.Informativeness: The summary should convey the key points of the text.Non-redundancy: The summary should not repeat any points, and ideally have maximal information coverage within the limited text length.Abstractive Summarization Referential clarity: Any intra-sentence or cross-sentence references in the summary should be unambiguous and within the scope of the summary.Focus: The summary needs to have a focus and all the sentences need to contain information related to this focal point.Structure and Coherence: The summary should be a well-organized and coherent body of information Dialogue GenerationMaking sense: Does the bot say things that don't make sense?Engagingness: Is the dialogue agent enjoyable to talk to?Interestingness: Did you find the bot interesting to talk to?Inquisitivenes: Does the bot ask a good amount of questions?Listening: Does the bot pay attention to what you say?Avoiding Repetition: Does the bot repeat itself?(either within or across utterances) Humanness: Is the conversation with a person or a bot? Image CaptioningRelevance: The caption should be specific and related to the image.
Combinatorial Optimization Neural Nets Based on A Hybrid of Lagrange and Transformation Approaches

A novel scheme for combinatorial optimization is suggested via analog constrained optimization. The constraints are separated into two parts. One is relieved by Lagrange approach, and the other by penalty or barrier function. In comparison with the existing Hopfield type networks, the new scheme has several new features. It is applicable to nonquadratic energy functions, and has reduced needs on externally controlling artificial weighting parameters. It also provides new potential for improving convergence and performance. In addition, it supplies a general combinatorial optimization scheme with its special cases being closely related to statistical physics method, EM algorithm, elastic net and robust PCA.
Monocular 3D multi-person pose estimation via predicting factorized correction factors

Despite the great achievement of 3D human pose estimation, recovering the 3D poses of multiple persons in a single image is still a challenging problem. In this paper, we focus on one specific problem in 3D multi-person pose estimation (3D-MPPE): estimating the absolute 3D human poses. We proposed a pipeline consists of human detection, absolute 3D human root localization, and root-relative 3D single-person pose estimation modules. For the absolute 3D human root localization task, we propose a decoupling dual-branch structure to reconstruct the height of the human body, and further output the depth and localization of the 3D human root in the camera coordinate system. Furthermore, a data augmentation strategy is presented to tackle occlusions, such that our model can effectively estimate the root localization with the incomplete bounding boxes. For the 3D human relative pose estimation task, we use the attention mechanism to capture the correlation between human joint coordinates and further improve the accuracy of relative pose estimation. Finally, we merge the absolute depth of human and the relative 3D pose to output the absolute 3D human pose.
Absolute Distance Prediction Based on Deep Learning Object Detection and Monocular Depth Estimation Models

Determining the distance between the objects in a scene and the camera sensor from 2D images is feasible by estimating depth images using stereo cameras or 3D cameras. The outcome of depth estimation is relative distances that can be used to calculate absolute distances to be applicable in reality. However, distance estimation is very challenging using 2D monocular cameras. This paper presents a deep learning framework that consists of two deep networks for depth estimation and object detection using a single image. Firstly, objects in the scene are detected and localized using the You Only Look Once (YOLOv5) network. In parallel, the estimated depth image is computed using a deep autoencoder network to detect the relative distances. The proposed object detection based YOLO was trained using a supervised learning technique, in turn, the network of depth estimation was self-supervised training. The presented distance estimation framework was evaluated on real images of outdoor scenes. The achieved results show that the proposed framework is promising and it yields an accuracy of 96% with RMSE of 0.203 of the correct absolute distance.
KOHTD: Kazakh offline handwritten text dataset

Despite the transition to digital information exchange, many documents, such as invoices, taxes, memos and questionnaires, historical data, and answers to exam questions, still require handwritten inputs. In this regard, there is a need to implement Handwritten Text Recognition (HTR) which is an automatic way to decrypt records using a computer. Handwriting recognition is challenging because of the virtually infinite number of ways a person can write the same message. For this proposal we introduce Kazakh handwritten text recognition research, a comprehensive dataset of Kazakh handwritten texts is necessary. This is particularly true given the lack of a dataset for handwritten Kazakh text. In this paper, we proposed our extensive Kazakh offline Handwritten Text dataset (KOHTD), which has 3000 handwritten exam papers and more than 140335 segmented images and there are approximately 922010 symbols. It can serve researchers in the field of handwriting recognition tasks by using deep and machine learning. We used a variety of popular text recognition methods for word and line recognition in our studies, including CTC-based and attention-based methods. In this paper, we have implemented state-of-the-art deep learning-based methods for Handwriting recognition for KOHTD dataset to create several strong baselines. The findings demonstrate KOHTD’s diversity. Also, we proposed a Genetic Algorithm (GA) for line and word segmentation based on random enumeration of a parameter. The dataset and GA code are available at https://github.com/abdoelsayed2016/KOHTD.
<scp>XAITK</scp>: The explainable AI toolkit

Abstract Recent advances in artificial intelligence (AI), driven mainly by deep neural networks, have yielded remarkable progress in fields, such as computer vision, natural language processing, and reinforcement learning. Despite these successes, the inability to predict how AI systems will behave “in the wild” impacts almost all stages of planning and deployment, including research and development, verification and validation, and user trust and acceptance. The field of explainable artificial intelligence (XAI) seeks to develop techniques enabling AI algorithms to generate explanations of their results; generally these are human‐interpretable representations or visualizations that are meant to “explain” how the system produced its outputs. We introduce the Explainable AI Toolkit (XAITK), a DARPA‐sponsored effort that builds on results from the 4‐year DARPA XAI program. The XAITK has two goals: (a) to consolidate research results from DARPA XAI into a single publicly accessible repository; and (b) to identify operationally relevant capabilities developed on DARPA XAI and assist in their transition to interested partners. We first describe the XAITK website and associated capabilities. These place the research results from DARPA XAI in the wider context of general research in the field of XAI, and include performer contributions of code, data, publications, and reports. We then describe the XAITK analytics and autonomy software frameworks. These are Python‐based frameworks focused on particular XAI domains, and designed to provide a single integration endpoint for multiple algorithm implementations from across DARPA XAI. Each framework generalizes APIs for system‐level data and control while providing a plugin interface for existing and future algorithm implementations. The XAITK project can be followed at: https://xaitk.org .
A Novel Invariant Based Commutative Encryption and Watermarking Algorithm for Vector Maps

Commutative encryption and watermarking (CEW) is an emerging method that combines encryption technology with digital watermarking technology. It has the dual capability of secure transmission and copyright protection. However, the existing CEW methods for vector maps have good robustness in resisting geometric attacks but poor resistance to vertex attacks (e.g., addition, deletion, etc.). To solve this problem, here we propose a novel invariant-based CEW algorithm for vector maps, which consists of permutation-based encryption scheme and coordinates-based watermarking scheme. In the encryption scheme, the encryption key is generated via the Gaussian distribution method combined with the SHA-512 hash method; then, the double random position permutation strategy is applied to the vector map encryption. In watermarking embedding scheme, the original watermark image is scrambled via logistic chaotic encryption before embedding, and the coordinates of all the vertices are normalized. Then, the scrambled watermark image is embedded into the normalized coordinates. Results show that: proposed method is more robust to conventional attacks (e.g., vertex addition and deletion, reordering and data format conversion) and geometric attacks (e.g., scaling and translation). In addition, compared with the existing CEW methods for vector maps, the proposed method has higher security and stronger robustness against vertex attacks.