Reviewerly/exHarmony huggingface dataset:
Interpretable Image Recognition with Hierarchical Prototypes

Vision models are interpretable when they classify objects on the basis of features that a person can directly understand. Recently, methods relying on visual feature prototypes have been developed for this purpose. However, in contrast to how humans categorize objects, these approaches have not yet made use of any taxonomical organization of class labels. With such an approach, for instance, we may see why a chimpanzee is classified as a chimpanzee, but not why it was considered to be a primate or even an animal. In this work we introduce a model that uses hierarchically organized prototypes to classify objects at every level in a predefined taxonomy. Hence, we may find distinct explanations for the prediction an image receives at each level of the taxonomy. The hierarchical prototypes enable the model to perform another important task: interpretably classifying images from previously unseen classes at the level of the taxonomy to which they correctly relate, e.g. classifying a hand gun as a weapon, when the only weapons in the training data are rifles. With a subset of ImageNet, we test our model against its counterpart black-box model on two tasks: 1) classification of data from familiar classes, and 2) classification of data from previously unseen classes at the appropriate level in the taxonomy. We find that our model performs approximately as well as its counterpart black-box model while allowing for each classification to be interpreted.
Supervised Hierarchical Cross-Modal Hashing

Recently, due to the unprecedented growth of multimedia data, cross-modal hashing has gained increasing attention for the efficient cross-media retrieval. Typically, existing methods on cross-modal hashing treat labels of one instance independently but overlook the correlations among labels. Indeed, in many real-world scenarios, like the online fashion domain, instances (items) are labeled with a set of categories correlated by certain hierarchy. In this paper, we propose a new end-to-end solution for supervised cross-modal hashing, named HiCHNet, which explicitly exploits the hierarchical labels of instances. In particular, by the pre-established label hierarchy, we comprehensively characterize each modality of the instance with a set of layer-wise hash representations. In essence, hash codes are encouraged to not only preserve the layer-wise semantic similarities encoded by the label hierarchy, but also retain the hierarchical discriminative capabilities. Due to the lack of benchmark datasets, apart from adapting the existing dataset FashionVC from fashion domain, we create a dataset from the online fashion platform Ssense consisting of 15,696 image-text pairs labeled by 32 hierarchical categories. Extensive experiments on two real-world datasets demonstrate the superiority of our model over the state-of-the-art methods.
ALFA: A dataset for UAV fault and anomaly detection

We present a dataset of several fault types in control surfaces of a fixed-wing Unmanned Aerial Vehicle (UAV) for use in Fault Detection and Isolation (FDI) and Anomaly Detection (AD) research. Currently, the dataset includes processed data for 47 autonomous flights with 23 sudden full engine failure scenarios and 24 scenarios for seven other types of sudden control surface (actuator) faults, with a total of 66 minutes of flight in normal conditions and 13 minutes of post-fault flight time. It additionally includes many hours of raw data of fully-autonomous, autopilot-assisted and manual flights with tens of fault scenarios. The ground truth of the time and type of faults is provided in each scenario to enable evaluation of the methods using the dataset. We have also provided the helper tools in several programming languages to load and work with the data and to help the evaluation of a detection method using the dataset. A set of metrics is proposed to help to compare different methods using the dataset. Most of the current fault detection methods are evaluated in simulation and as far as we know, this dataset is the only one providing the real flight data with faults in such capacity. We hope it will help advance the state-of-the-art in Anomaly Detection or FDI research for Autonomous Aerial Vehicles and mobile robots to enhance the safety of autonomous and remote flight operations further. The dataset and the provided tools can be accessed from https://doi.org/10.1184/R1/12707963.
Contextualized Non-Local Neural Networks for Sequence Learning

Recently, a large number of neural mechanisms and models have been proposed for sequence learning, of which selfattention, as exemplified by the Transformer model, and graph neural networks (GNNs) have attracted much attention. In this paper, we propose an approach that combines and draws on the complementary strengths of these two methods. Specifically, we propose contextualized non-local neural networks (CN3), which can both dynamically construct a task-specific structure of a sentence and leverage rich local dependencies within a particular neighbourhood.Experimental results on ten NLP tasks in text classification, semantic matching, and sequence labelling show that our proposed model outperforms competitive baselines and discovers task-specific dependency structures, thus providing better interpretability to users.
Subword Encoding in Lattice

Jie Yang, Yue Zhang, Shuailong Liang. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019.
Collaboration Based Multi-Label Learning

It is well-known that exploiting label correlations is crucially important to multi-label learning. Most of the existing approaches take label correlations as prior knowledge, which may not correctly characterize the real relationships among labels. Besides, label correlations are normally used to regularize the hypothesis space, while the final predictions are not explicitly correlated. In this paper, we suggest that for each individual label, the final prediction involves the collaboration between its own prediction and the predictions of other labels. Based on this assumption, we first propose a novel method to learn the label correlations via sparse reconstruction in the label space. Then, by seamlessly integrating the learned label correlations into model training, we propose a novel multi-label learning approach that aims to explicitly account for the correlated predictions of labels while training the desired model simultaneously. Extensive experimental results show that our approach outperforms the state-of-the-art counterparts.
PoseRBPF: A Rao-Blackwellized Particle Filter for6D Object Pose Estimation

Tracking 6D poses of objects from videos provides rich information to a robot in performing different tasks such as manipulation and navigation.In this work, we formulate the 6D object pose tracking problem in the Rao-Blackwellized particle filtering framework, where the 3D rotation and the 3D translation of an object are decoupled.This factorization allows our approach, called PoseRBPF to efficiently estimate the 3D translation of an object along with the full distribution over the 3D rotation.This is achieved by discretizing the rotation space in a fine-grained manner, and training an auto-encoder network to construct a codebook of feature embeddings for the discretized rotations.As a result, PoseRBPF can track objects with arbitrary symmetries while still maintaining adequate posterior distributions.Our approach achieves state-of-the-art results on two 6D pose estimation benchmarks.
High-Fidelity Grasping in Virtual Reality using a Glove-based System

This paper presents a design that jointly provides hand pose sensing, hand localization, and haptic feedback to facilitate real-time stable grasps in Virtual Reality (VR). The design is based on an easy-to-replicate glove-based system that can reliably perform (i) a high-fidelity hand pose sensing in real time through a network of 15 IMUs, and (ii) the hand localization using a Vive Tracker. The supported physics-based simulation in VR is capable of detecting collisions and contact points for virtual object manipulation, which drives the collision event to trigger the physical vibration motors on the glove to signal the user, providing a better realism inside virtual environments. A caging-based approach using collision geometry is integrated to determine whether a grasp is stable. In the experiment, we showcase successful grasps of virtual objects with large geometry variations. Comparing to the popular LeapMotion sensor, we demonstrate the proposed glove-based design yields a higher success rate in various tasks in VR. We hope such a glove-based system can simplify the data collection of human manipulations with VR.
Geometrically Invariant Color Medical Image Null-Watermarking Based on Precise Quaternion Polar Harmonic Fourier Moments

The traditional embedded watermarking scheme achieves copyright protection of images by embedding watermarks into the original images. This process will deteriorate the original image, so it is not suitable for medical images. In order to overcome this limitation, this paper proposes a geometrically invariant color medical image null-watermarking scheme based on quaternion polar harmonic Fourier moments (QPHFM), which achieves the copyright protection without changing the original medical image. First, a high-precision QPHFM computation method based on wavelet numerical integration is designed, which can effectively improve the computation precision. Then based on the precise QPHFM (PQPHFM) and chaotic map, a robust null-watermarking scheme for color medical images is proposed to protect the copyright of color medical images. This scheme uses the PQPHFM to construct feature images and its good stability and geometric invariance make the proposed null-watermarking scheme has good robustness and resistance against geometric attacks. Extensive simulation experiments show that the proposed scheme is highly robust in resisting common attacks and geometric attacks such as scaling, rotation, change of length-width ratio, and flipping. The performance of the proposed scheme is superior to that of similar robust null-watermarking schemes and other color medical image watermarking schemes.
Improving Neural Story Generation by Targeted Common Sense Grounding

Stories generated with neural language models have shown promise in grammatical and stylistic consistency.However, the generated stories are still lacking in common sense reasoning, e.g., they often contain sentences deprived of world knowledge.We propose a simple multi-task learning scheme to achieve quantitatively better common sense reasoning in language models by leveraging auxiliary training signals from datasets designed to provide common sense grounding.When combined with our two-stage fine-tuning pipeline, our method achieves improved common sense reasoning and state-of-the-art perplexity on the Writing-Prompts (Fan et al., 2018) story generation dataset.
Entity-Consistent End-to-end Task-Oriented Dialogue System with KB Retriever

Querying the knowledge base (KB) has long been a challenge in the end-to-end taskoriented dialogue system.Previous sequenceto-sequence (Seq2Seq) dialogue generation work treats the KB query as an attention over the entire KB, without the guarantee that the generated entities are consistent with each other.In this paper, we propose a novel framework which queries the KB in two steps to improve the consistency of generated entities.In the first step, inspired by the observation that a response can usually be supported by a single KB row, we introduce a KB retrieval component which explicitly returns the most relevant KB row given a dialogue history.The retrieval result is further used to filter the irrelevant entities in a Seq2Seq response generation model to improve the consistency among the output entities.In the second step, we further perform the attention mechanism to address the most correlated KB column.Two methods are proposed to make the training feasible without labeled retrieval data, which include distant supervision and Gumbel-Softmax technique.Experiments on two publicly available task oriented dialog datasets show the effectiveness of our model by outperforming the baseline systems and producing entity-consistent responses.* * Email corresponding.Address Distance POI type POI Traffic info 638 Amherst St 3 miles grocery store Sigona Farmers Market car collision nearby 269 Alger Dr 1 miles coffee or tea
Discriminative Cross-Modal Transfer Learning and Densely Cross-Level Feedback Fusion for RGB-D Salient Object Detection

This article addresses two key issues in RGB-D salient object detection based on the convolutional neural network (CNN). 1) How to bridge the gap between the "data-hungry" nature of CNNs and the insufficient labeled training data in the depth modality? 2) How to take full advantages of the complementary information among two modalities. To solve the first problem, we model the depth-induced saliency detection as a CNN-based cross-modal transfer learning problem. Instead of directly adopting the RGB CNN as initialization, we additionally train a modality classification network (MCNet) to encourage discriminative modality-specific representations in minimizing the modality classification loss. To solve the second problem, we propose a densely cross-level feedback topology, in which the cross-modal complements are combined in each level and then densely fed back to all shallower layers for sufficient cross-level interactions. Compared to traditional two-stream frameworks, the proposed one can better explore, select, and fuse cross-modal cross-level complements. Experiments show the significant and consistent improvements of the proposed CNN framework over other state-of-the-art methods.
Posing Fair Generalization Tasks for Natural Language Inference

Deep learning models for semantics are generally evaluated using naturalistic corpora.Adversarial methods, in which models are evaluated on new examples with known semantic properties, have begun to reveal that good performance at these naturalistic tasks can hide serious shortcomings.However, we should insist that these evaluations be fair -that the models are given data sufficient to support the requisite kinds of generalization.In this paper, we define and motivate a formal notion of fairness in this sense.We then apply these ideas to natural language inference by constructing very challenging but provably fair artificial datasets and showing that standard neural models fail to generalize in the required ways; only task-specific models that jointly compose the premise and hypothesis are able to achieve high performance, and even these models do not solve the task perfectly.
Table-to-Text Generation with Effective Hierarchical Encoder on Three Dimensions (Row, Column and Time)

Although Seq2Seq models for table-to-text generation have achieved remarkable progress, modeling table representation in one dimension is inadequate.This is because (1) the table consists of multiple rows and columns, which means that encoding a table should not depend only on one dimensional sequence or set of records and (2) most of the tables are time series data (e.g.NBA game data, stock market data), which means that the description of the current table may be affected by its historical data.To address aforementioned problems, not only do we model each table cell considering other records in the same row, we also enrich table's representation by modeling each table cell in context of other cells in the same column or with historical (time dimension) data respectively.In addition, we develop a table cell fusion gate to combine representations from row, column and time dimension into one dense vector according to the saliency of each dimension's representation.We evaluated our methods on ROTOWIRE, a benchmark dataset of NBA basketball games.Both automatic and human evaluation results demonstrate the effectiveness of our model with improvement of 2.66 in BLEU over the strong baseline and outperformance of state-of-the-art model.
An empirical analysis of emoji usage on Twitter

Purpose Emoji has become an essential component of any digital communication and its importance can be attested to by its sustained popularity and widespread use. However, research in Emojis is rarely to be seen due to the lack of data at a greater scale. The purpose of this paper is to systematically analyse and compare the usage of Emojis in a cross-cultural manner. Design/methodology/approach This research conducted an empirical analysis using a large-scale, cross-regional emoji usage data set from Twitter, a platform where the limited 140 characters allowance has made it essential for the inclusion of emojis within tweets. The extremely large textual data set covers a period of only two months, but the 673m tweets authored by more than 2,081,542 unique users is a sufficiently large sample for the authors to yield significant results. Findings This research discovered that the categories and frequencies of Emojis communicated by users can provide a rich source of data to understand cultural differences between Twitter users from a large range of demographics. This research subsequently demonstrated the preferential use of Emojis complies with Hofstede’s Cultural Dimensions Model, in which different representations of demographics and culture within countries present significantly different use of Emojis to communicate emotions. Originality/value This study provides a robust example of how to strategically conduct research using large-scale emoji data to pursue research questions previously difficult. To the best of authors’ knowledge, the present study pioneers the first systematic analysis and comparison of the usage of emojis on Twitter across different cultures; it is the largest, in terms of the scale study of emoji usage to-date.
A Simple but Effective Method to Incorporate Multi-turn Context with BERT for Conversational Machine Comprehension

Conversational machine comprehension (CMC) requires understanding the context of multi-turn dialogue.Using BERT, a pretraining language model, has been successful for single-turn machine comprehension, while modeling multiple turns of question answering with BERT has not been established because BERT has a limit on the number and the length of input sequences.In this paper, we propose a simple but effective method with BERT for CMC.Our method uses BERT to encode a paragraph independently conditioned with each question and each answer in a multi-turn context.Then, the method predicts an answer on the basis of the paragraph representations encoded with BERT.The experiments with representative CMC datasets, QuAC and CoQA, show that our method outperformed recently published methods (+0.8 F1 on QuAC and +2.1 F1 on CoQA).In addition, we conducted a detailed analysis of the effects of the number and types of dialogue history on the accuracy of CMC, and we found that the gold answer history, which may not be given in an actual conversation, contributed to the model performance most on both datasets.
Level set based segmentation using local fitted images and inhomogeneity entropy

Precise image segmentation plays an important role in advancing the understanding of objects of interest, and facilitating the estimation of their morphological changes. In this paper, we proposed a level set method for segmenting desirable objects by introducing an inhomogeneity entropy descriptor and entropy weighted energy functional based on three different local fitted images. The introduced entropy aims to alleviate challenges caused by weak contrast and severe intensity inhomogeneity, while the energy functional is used to assess intensity differences between images to be processed and their local fitted versions. Minimization of the energy can evolve the level set function towards the boundaries of objects depicted on images. Segmentation experiments on synthetic and natural images as well as medical images demonstrated the developed method achieved reasonable segmentation accuracy, as compared to some existing level set methods.
A time-aware dynamic service quality prediction approach for services

Dynamic Quality of Service (QoS) prediction for services is currently a hot topic and a challenge for research in the fields of service recommendation and composition. Our paper addresses the problem with a Time-aWare service Quality Prediction method (named TWQP), a two-phase approach with one phase based on historical time slices and one on the current time slice. In the first phase, if the user had invoked the service in a previous time slice, the QoS value for the user calling the service on the next time slice is predicted on the basis of the historical QoS data; if the user had not invoked the service in a previous time slice, then the Covering Algorithm (CA) is applied to predict the missing values. In the second phase, we predict the missing values for the current time slice according to the results of the previous phase. A large number of experiments on a real-world dataset, WS-Dream, show that, when compared with the classical QoS prediction algorithms, our proposed method greatly improves the prediction accuracy.
Improved Color Attenuation Prior for Single-Image Haze Removal

This paper proposes a single image haze removal algorithm that shows a marked improvement on the color attenuation prior-based method. Through a vast number of experiments on a wide variety of images, it is discovered that there are problems in the color attenuation prior, such as color distortion and background noise, which arise due to the fact that the priors do not hold true in all circumstances. Successful resolution of these problems using the proposed algorithm shows its superior performance to other state-of-the-art methods in terms of both subjective visual quality and quantitative metrics, on both synthetic and natural hazy image datasets. The proposed algorithm also is computationally friendly, due to the use of an efficient quad-decomposition algorithm for atmospheric light estimation and a simple modified hybrid median filter for depth map refinement.
Development and Assessment of a New Global Mammographic Image Feature Analysis Scheme to Predict Likelihood of Malignant Cases

This study aims to develop and evaluate a new computer-aided diagnosis (CADx) scheme based on analysis of global mammographic image features to predict likelihood of cases being malignant. An image dataset involving 1,959 cases was retrospectively assembled. Suspicious lesions were detected and biopsied in each case. Among them, 737 cases are malignant and 1,222 are benign. Each case includes four mammograms of craniocaudal and mediolateral oblique view of left and right breasts. CADx scheme is applied to pre-process mammograms, generate two image maps in frequency domain using discrete cosine transform and fast Fourier transform, compute bilateral image feature differences from left and right breasts, and apply a support vector machine (SVM) to predict likelihood of the case being malignant. Three sub-groups of image features were computed from the original mammograms and two transformation maps. Four SVMs using three sub-groups of image features and fusion of all features were trained and tested using a 10-fold cross-validation method. The computed areas under receiver operating characteristic curves (AUCs) range from 0.85 to 0.91 using image features computed from one of three sub-groups, respectively. By fusion of all image features computed in three sub-groups, the fourth SVM yields a significantly higher performance with AUC = 0.96±0.01 (p<; 0.01). This study demonstrates feasibility of developing a new global image feature analysis based CADx scheme of mammograms with high performance. By avoiding difficulty and possible errors in breast lesion segmentation, this new CADx approach is more efficient in development and potentially more robust in future application.