Reviewerly/exHarmony huggingface dataset:
An End-to-end Model for Entity-level Relation Extraction using Multi-instance Learning

We present a joint model for entity-level relation extraction from documents.In contrast to other approaches -which focus on local intra-sentence mention pairs and thus require annotations on mention level -our model operates on entity level.To do so, a multi-task approach is followed that builds upon coreference resolution and gathers relevant signals via multi-instance learning with multi-level representations combining global entity and local mention information.We achieve state-of-theart relation extraction results on the DocRED dataset and report the first entity-level end-toend relation extraction results for future reference.Finally, our experimental results suggest that a joint approach is on par with taskspecific learning, though more efficient due to shared parameters and training steps.
Quantum Key Distribution Networks: Challenges and Future Research Issues in Security

A quantum key distribution (QKD) network is proposed to allow QKD protocols to be the infrastructure of the Internet for distributing unconditional security keys instead of existing public-key cryptography based on computationally complex mathematical problems. Numerous countries and research institutes have invested enormous resources to execute correlation studies on QKD networks. Thus, in this study, we surveyed existing QKD network studies and practical field experiments to summarize the research results (e.g., type and architecture of QKD networks, key generating rate, maximum communication distance, and routing protocol). Furthermore, we highlight the three challenges and future research issues in the security of QKD networks and then provide some feasible resolution strategies for these challenges.
Radius-SMOTE: A New Oversampling Technique of Minority Samples Based on Radius Distance for Learning From Imbalanced Data

Imbalanced learning problems are a challenge faced by classifiers when data samples have an unbalanced distribution in each class.Furthermore, the synthetic oversampling method (SMOTE) is a preprocessing technique widely used to synthesize new data and balance the different numbers of samples in each class.One of the SMOTE method's expansions is based on the initial selection approach, which determines the best candidates to be oversampled in the data before the process of synthetic example generation starts.However, SMOTE and most of the existing oversampling methods based on initial selection still found overlapping data on the final result.This issue makes it difficult for any classifiers to determine the decision boundary of each class.Therefore, this research proposes a new oversampling technique called Radius-SMOTE, which emphasizes the initial selection approach by creating synthetic data based on a safe radius distance.Furthermore, new synthetic data are prevented from overlapping in the opposite class with the safe radius distance.The Radius-SMOTE was evaluated extensively with thirteen artificial imbalanced datasets from the KEEL repository.The experimental results show that the proposed method is able to achieve the best results on 5 datasets, namely yeast-1-4-5-8_vs_7, ecoli-0-1-3-7_vs_2-6, Umbilical cord, Pima, and Haberman dataset in term of various assessment metrics.Besides that, the computational cost for our proposed method is also relatively low, with an average time of 0.5 to 1 second on the 13 tested datasets.
Anomaly detection in video sequences: A benchmark and computational model

Anomaly detection has attracted considerable search attention. However, existing anomaly detection databases encounter two major problems. Firstly, they are limited in scale. Secondly, training sets contain only video-level labels indicating the existence of an abnormal event during the full video while lacking annotations of precise time durations. To tackle these problems, we contribute a new Large-scale Anomaly Detection (LAD) database as the benchmark for anomaly detection in video sequences, which is featured in two aspects. 1) It contains 2000 video sequences including normal and abnormal video clips with 14 anomaly categories including crash, fire, violence etc. with large scene varieties, making it the largest anomaly analysis database to date. 2) It provides the annotation data, including video-level labels (abnormal/normal video, anomaly type) and frame-level labels (abnormal/normal video frame) to facilitate anomaly detection. Leveraging the above benefits from the LAD database, we further formulate anomaly detection as a fully supervised learning problem and propose a multi-task deep neural network to solve it. We firstly obtain the local spatiotemporal contextual feature by using an Inflated 3D convolutional (I3D) network. Then we construct a recurrent convolutional neural network fed the local spatiotemporal contextual feature to extract the spatiotemporal contextual feature. With the global spatiotemporal contextual feature, the anomaly type and score can be computed simultaneously by a multi-task neural network. Experimental results show that the proposed method outperforms the state-of-the-art anomaly detection methods on our database and other public databases of anomaly detection. Supplementary materials are available at http://sim.jxufe.cn/JDMKL/ymfang/anomaly-detection.html.
DeCEMBERT: Learning from Noisy Instructional Videos via Dense Captions and Entropy Minimization

Leveraging large-scale unlabeled web videos such as instructional videos for pre-training followed by task-specific finetuning has become the de facto approach for many videoand-language tasks.However, these instructional videos are very noisy, the accompanying ASR narrations are often incomplete, and can be irrelevant to or temporally misaligned with the visual content, limiting the performance of the models trained on such data.To address these issues, we propose an improved video-and-language pre-training method that first adds automatically-extracted dense region captions from the video frames as auxiliary text input, to provide informative visual cues for learning better video and language associations.Second, to alleviate the temporal misalignment issue, our method incorporates an entropy minimization-based constrained attention loss, to encourage the model to automatically focus on the correct caption from a pool of candidate ASR captions.Our overall approach is named DECEMBERT (Dense Captions and Entropy Minimization).Comprehensive experiments on three video-and-language tasks (text-to-video retrieval, video captioning, and video question answering) across five datasets demonstrate that our approach outperforms previous state-of-the-art methods.Ablation studies on pre-training and downstream tasks show that adding dense captions and constrained attention loss help improve the model performance.Lastly, we also provide attention visualization to show the effect of applying the proposed constrained attention loss.
Network Pruning Using Adaptive Exemplar Filters

Popular network pruning algorithms reduce redundant information by optimizing hand-crafted models, and may cause suboptimal performance and long time in selecting filters. We innovatively introduce adaptive exemplar filters to simplify the algorithm design, resulting in an automatic and efficient pruning approach called EPruner. Inspired by the face recognition community, we use a message-passing algorithm Affinity Propagation on the weight matrices to obtain an adaptive number of exemplars, which then act as the preserved filters. EPruner breaks the dependence on the training data in determining the “important” filters and allows the CPU implementation in seconds, an order of magnitude faster than GPU-based SOTAs. Moreover, we show that the weights of exemplars provide a better initialization for the fine-tuning. On VGGNet-16, EPruner achieves a 76.34%-FLOPs reduction by removing 88.80% parameters, with 0.06% accuracy improvement on CIFAR-10. In ResNet-152, EPruner achieves a 65.12%-FLOPs reduction by removing 64.18% parameters, with only 0.71% top-5 accuracy loss on ILSVRC-2012. Our code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/lmbxmu/EPruner</uri> .
Cross-Task Instance Representation Interactions and Label Dependencies for Joint Information Extraction with Graph Convolutional Networks

Existing works on information extraction (IE) have mainly solved the four main tasks separately (entity mention recognition, relation extraction, event trigger detection, and argument extraction), thus failing to benefit from inter-dependencies between tasks.This paper presents a novel deep learning model to simultaneously solve the four tasks of IE in a single model (called FourIE).Compared to few prior work on jointly performing four IE tasks, FourIE features two novel contributions to capture inter-dependencies between tasks.First, at the representation level, we introduce an interaction graph between instances of the four tasks that is used to enrich the prediction representation for one instance with those from related instances of other tasks.Second, at the label level, we propose a dependency graph for the information types in the four IE tasks that captures the connections between the types expressed in an input sentence.A new regularization mechanism is introduced to enforce the consistency between the golden and predicted type dependency graphs to improve representation learning.We show that the proposed model achieves the state-of-the-art performance for joint IE on both monolingual and multilingual learning settings with three different languages.
Kernel Matrix Approximation on Class-Imbalanced Data With an Application to Scientific Simulation

Generating low-rank approximations of kernel matrices that arise in nonlinear machine learning techniques holds the potential to significantly alleviate the memory and computational burdens. A compelling approach centers on finding a concise set of exemplars or landmarks to reduce the number of similarity measure evaluations from quadratic to linear concerning the data size. However, a key challenge is to regulate tradeoffs between the quality of landmarks and resource consumption. Despite the volume of research in this area, current understanding is limited regarding the performance of landmark selection techniques in the presence of class-imbalanced data sets that are becoming increasingly prevalent in many applications. Hence, this paper provides a comprehensive empirical investigation using several real-world imbalanced data sets, including scientific data, by evaluating the quality of approximate low-rank decompositions and examining their influence on the accuracy of downstream tasks. Furthermore, we present a new landmark selection technique called Distance-based Importance Sampling and Clustering (DISC), in which the relative importance scores are computed for improving accuracy-efficiency tradeoffs compared to existing works that range from probabilistic sampling to clustering methods. The proposed landmark selection method follows a coarse-to-fine strategy to capture the intrinsic structure of complex data sets, allowing us to substantially reduce the computational complexity and memory footprint with minimal loss in accuracy.
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models

We propose a general framework called Text Modular Networks (TMNs) for building interpretable systems that learn to solve complex tasks by decomposing them into simpler ones solvable by existing models.To ensure solvability of simpler tasks, TMNs learn the textual input-output behavior (i.e., language) of existing models through their datasets.This differs from prior decomposition-based approaches which, besides being designed specifically for each complex task, produce decompositions independent of existing submodels.Specifically, we focus on Question Answering (QA) and show how to train a next-question generator to sequentially produce sub-questions targeting appropriate submodels, without additional human annotation.These sub-questions and answers provide a faithful natural language explanation of the model's reasoning.We use this framework to build MODULARQA, 1 a system that can answer multi-hop reasoning questions by decomposing them into sub-questions answerable by a neural factoid single-span QA model and a symbolic calculator.Our experiments show that MODULARQA is more versatile than existing explainable systems for DROP and Hot-potQA datasets, is more robust than stateof-the-art blackbox (uninterpretable) systems, and generates more understandable and trustworthy explanations compared to prior work.
Development of computational linguistic resources for automated detection of textual cyberbullying threats in Roman Urdu language

Automatic Cyberbullying detection has remained very challenging task since social media content and conversations are usually posted in unstructured free-text form leaving behind the language norms.The major concern and gap in formulating cyberbullying detection strategies is scarcity of available linguistic resources typically for newly evolved languages.Roman Urdu has recently emerged and hence is a resource poor language.Urdu has been widely known as the national language of Pakistan.However, because of socio-cultural and multilingual aspects, Roman Urdu is used widely on the Internet by Asians and more specifically Pakistanis.To fulfil the above stated gap, this research work presents guidelines for data annotation process and developed two linguistic resources: (i) Annotated corpus in Roman Urdu Language for cyberaggression and offensive language detection.The process of data annotation involved bilingual annotators instead of crowdsourcing.It has the benefit of correctly annotating instances that constitute clear cases of cyberbullying without compromising data quality.The developed corpus is highly balanced (with almost negligible skew) unlike most of the existing corpuses even in mature languages.(ii) Processing textual information for NLP tasks involves Stop-word elimination as a sub phase.Stop words carry least semantic information and increase feature space as compared to the other tokens and index terms in corpora.We have developed domain specific stop words for Roman Urdu Language considering all the lexical variants and typically in the context of aggression detection and collected data.The work has been carried out using python programming language and Pycharm IDE.
Crafting Adversarial Examples for Neural Machine Translation

Effective adversary generation for neural machine translation (NMT) is a crucial prerequisite for building robust machine translation systems.In this work, we investigate veritable evaluations of NMT adversarial attacks, and propose a novel method to craft NMT adversarial examples.We first show the current NMT adversarial attacks may be improperly estimated by the commonly used monodirectional translation, and we propose to leverage the round-trip translation technique to build valid metrics for evaluating NMT adversarial attacks.Our intuition is that an effective NMT adversarial example, which imposes minor shifting on the source and degrades the translation dramatically, would naturally lead to a semantic-destroyed round-trip translation result.We then propose a promising black-box attack method called Word Saliency speedup Local Search (WSLS) that could effectively attack the mainstream NMT architectures.Comprehensive experiments demonstrate that the proposed metrics could accurately evaluate the attack effectiveness, and the proposed WSLS could significantly break the state-of-art NMT models with small perturbation.Besides, WSLS exhibits strong transferability on attacking Baidu and Bing online translators.
A Single Image Dehazing Technique Using the Dual Transmission Maps Strategy and Gradient-Domain Guided Image Filtering

In this paper, a single image dehazing technique using dual transmission maps strategy and gradient-domain guided image filtering is presented. A new strategy is adopted to compute the dual transmission maps using the dark channel and atmospheric light. Further, the transmission maps are refined to remove any remaining ill effects using the gradient-domain-guided filter. Finally, using the dark channel, atmospheric light, and refined transmission map, the haze-free image is obtained. The dual transmission maps strategy not only removes halo artifacts and reduces the saturation but also ensures the natural appearance in the recovered images. Furthermore, the proposed scheme is evaluated using a wide range of images and compared with state-of-the-art schemes. The comparison shows the superiority of the proposed technique in terms of recovering haze-free images.
Graph-to-Graph: Towards Accurate and Interpretable Online Handwritten Mathematical Expression Recognition

Recent handwritten mathematical expression recognition (HMER) approaches treat the problem as an image-to-markup generation task where the handwritten formula is translated into a sequence (e.g. LaTeX). The encoder-decoder framework is widely used to solve this image-to-sequence problem. However, (i) for structured mathematical formula, the hierarchical structure neither in the formula nor in the markup has been explored adequately. In addition, (ii) existing image-to-markup methods could not explicitly segment mathematical symbols in the formula corresponding to each target markup token. In this paper, we address the above issues by formulating the HMER as a graph-to-graph (G2G) learning problem. Graph is more flexible and general for structure representation and learning compared with image or sequence. At the core of our method lies the embedding of input formula and output markup into graphs on primitives, with Graph Neural Networks (GNN) to explore the structural information, and a novel sub-graph attention mechanism to match primitives in the input and output graphs. We conduct extensive experiments on CROHME datasets to demonstrate the benefits of the proposed G2G model. Our method yields significant improvements over previous SOTA image-to-markup systems. Moreover, it explicitly resolves the symbol segmentation problem while still being trained end-to-end, making the whole system much more accurate and interpretable.
Relation Classification with Entity Type Restriction

Relation classification aims to predict a relation between two entities in a sentence.The existing methods regard all relations as the candidate relations for the two entities.These methods neglect the restrictions on candidate relations by entity types, which leads to some inappropriate relations being candidate relations.In this paper, we propose a novel paradigm, RElation Classification with ENtity Type restriction (RECENT), which exploits entity types to restrict candidate relations.Specially, the mutual restrictions of relations and entity types are formalized and introduced into relation classification.Besides, the proposed paradigm, RE-CENT, is model-agnostic.Based on two representative models GCN and SpanBERT respectively, RECENT GCN and RECENT SpanBERT are trained in RECENT 1 .Experimental results on a standard dataset indicate that RECENT improves the performance of GCN and Span-BERT by 6.9 and 4.4 F1 points, respectively.Especially, RECENT SpanBERT achieves a new state-of-the-art on TACRED.
Path Planning for Autonomous Ships: A Hybrid Approach Based on Improved APF and Modified VO Methods

In this research, a hybrid approach for path planning of autonomous ships that generates both global and local paths, respectively, is proposed. The global path is obtained via an improved artificial potential field (APF) method, which makes up for the shortcoming that the typical APF method easily falls into a local minimum. A modified velocity obstacle (VO) method that incorporates the closest point of approach (CPA) model and the International Regulations for Preventing Collisions at Sea (COLREGS), based on the typical VO method, can be used to get the local path. The contribution of this research is two-fold: (1) improvement of the typical APF and VO methods, making up for previous shortcomings, and integrated COLREGS rules and good seamanship, making the paths obtained more in line with navigation practice; (2) the research included global and local path planning, considering both the safety and maneuverability of the ship in the process of avoiding collision, and studied the whole process of avoiding collision in a relatively entirely way. A case study was then conducted to test the proposed approach in different situations. The results indicate that the proposed approach can find both global and local paths to avoid the target ship.
Online Learning Behavior Analysis Based on Image Emotion Recognition

There are many limitations of the current online learning platforms in the monitoring of learning behaviors and the evaluation of teaching effects.For example, the platforms cannot sense and correct the changes in the learning states and emotions of the students.To overcome the limitations, this paper tries to analyze online learning behaviors based on image emotion recognition.Firstly, the flow of image emotion recognition was detailed to facilitate the analysis of online learning behaviors, and the key frames were extracted from human face images, using improved local binary pattern (LBP) and wavelet transform.Next, the authors constructed the structure for the system of online learning behavior analysis, proposed a learning emotion recognition method based on facial expressions, and established an image emotion classification model for online learning, based on the attention mechanism.Experimental results show that the proposed algorithm is effective in analyzing online learning behaviors based on image emotion recognition.
A Survey on Spoken Language Understanding: Recent Advances and New Frontiers

Spoken Language Understanding (SLU) aims to extract the semantics frame of user queries, which is a core component in a task-oriented dialog system. With the burst of deep neural networks and the evolution of pre-trained language models, the research of SLU has obtained signiﬁcant breakthroughs. However, there remains a lack of a comprehensive survey summarizing existing approaches and recent trends, which motivated the work presented in this article. In this paper, we survey recent advances and new frontiers in SLU. Specifically, we give a thorough review of this research ﬁeld, covering different aspects including (1) new taxonomy: we provide a new perspective for SLU ﬁled, including single model vs. joint model, implicit joint modeling vs. explicit joint modeling in joint model, non pre-trained paradigm vs. pretrained paradigm; (2) new frontiers: some emerging areas in complex SLU as well as the corresponding challenges; (3) abundant open-source resources: to help the community, we have collected, organized the related papers, baseline projects and leaderboard on a public website where SLU researchers could directly access to the recent progress. We hope that this survey can shed a light on future research in SLU ﬁeld.
Hamiltonian Operator Approximation for Energy Measurement and Ground-State Preparation

A strategy is devised to approximate the Hamiltonian operator in ways that benefit analog quantum simulators, potentially outperforming variational methods in terms of gate depth and a total number of measurements.