Reviewerly/exHarmony huggingface dataset:
Convolutional Transformer with Similarity-based Boundary Prediction for Action Segmentation

Action classification has made great progress, but segmenting and recognizing actions from long videos remains a challenging problem. Recently, Transformer-based models with strong sequence modeling ability have succeeded in many se-quence modeling tasks. However, the lack of inductive bias and the difficulty of handling long video sequences limit the application of the Transformer in the action segmentation task. In order to explore the potential of the Transformer in this task, we replace some specific linear layers in the vanilla Transformer with dilated temporal convolution, and a sparse attention mechanism is utilized to reduce the time and space complexities to process long video sequences. Besides, directly using frame-wise classification loss to train the model will cause that frames at boundaries of actions are treated equally with those in the middle of actions, and the learned features are not sensitive to boundaries. We propose a new local log-context attention module to predict whether each frame is at the beginning, middle, or end of an action. Since boundary frames are similar to their neighboring frames of different classes, our similarity-based boundary prediction helps learn more discriminative features. Extensive experiments on three datasets show the effectiveness of our method.
Graph Structure Guided Transformer for Semantic Segmentation

Segmentation is an essential operation of image processing, and utilizing long-range context information is the key for pixel-wise prediction tasks such as semantic segmentation. Convolutional Neural Networks (CNNs) are good at modeling local relationships through convolutional operations, but they are often inefficient in capturing global relationships between distant regions and require stacking multiple convolutional lay-ers. Utilizing the advantages of transformer in modeling long-range dependency, this paper proposes a novel Graph Structure Guided Transformer (GSGT) to realize semantic segmentation. Different from the previous methods that hard-divide the image in a regular grid manner, our graph projection method maps the two-dimensional feature map into a graph structure according to certain semantic relevance, so as to meet the data structure form required by the transformer. Meanwhile, to fully utilize the graph structure information, we also propose a graph embedding attention module, which utilizes the local topology of the graph structure to complement the global context of transformer. Moreover, GSGT is easy to be incorporated with various CNN backbones and transformer model variants to significantly improve the segmentation accuracy and convergence speed. Experiments on Cityscapes, VOC and ADE20K datasets demonstrate that the proposed method performs well in semantic seamentation task.
A Hybrid Approach for Summarizing User Reviews Based on KL-Divergence and Deep Learning

In the modern day the information age makes available instant access to knowledge that would have been difficult or impossible to find previously. This leads to the problem of exploring too many user reviews on popular products for potential buyers to spend adequate time to read and extract the most salient product details and opinions of previous buyers. Multi-Document Summarization, the process of abbreviating the content of a collection of documents into a short, singular summary, is an important tool for expressing the content of these source while maintaining a level of linguistic quality. We can take advantage of the summarization approach to reduce the time necessary to read a collection of topically-related user reviews to locate their desired information needs by viewing a short summary of these reviews. In this paper, we propose a new summarizer that utilizes both extractive and abstractive techniques to yield such a high-quality summary. Our hybrid summarization approach is novel, since it includes the development of an innovative extractive summarizer that simply employes the KL-divergence measure to quantify sentences that capture the major contents of a set of user reviews on a particular product and retains these sentences in a summary that contains the most salient information of the set of reviews. Hereafter, the extractive summary is passed on to an adapted BERT deep learning model to improve the linguistic quality of the original summary and generate a new shorter text that conveys the most critical information from the original one. The proposed hybrid summarization approach is simple and easy to understand. Our summarizer has been compared against three existing summarizers and shown to outperform each one of them in terms of both expressing relevant content and linguistic features of a set of user reviews.
Blur the Linguistic Boundary: Interpreting Chinese Buddhist Sutra in English via Neural Machine Translation

Buddhism is an influential religion with a long-standing history and profound philosophy. Nowadays, more and more people worldwide aspire to learn the essence of Buddhism, attaching importance to Buddhism dissemination. However, Buddhist scriptures written in classical Chinese are obscure to most people and machine translation applications. For instance, general Chinese-English neural machine translation (NMT) fails in this domain. In this paper, we proposed a novel approach to building a practical NMT model for Buddhist scriptures. The performance of our translation pipeline acquired highly promising results in ablation experiments under three criteria.
Online Improved Vehicle Tracking Accuracy via Unsupervised Route Generation

Vehicle tracking is one of the essential technology for Smart City Systems. Existing vehicle tracking methods have achieved outstanding performance in simple traffic scenarios. However, these methods often fail to track specific traffic scenarios, such as vehicle deformation, lights, and occlusion between vehicles. In this paper, we propose a novel vehicle tracking method to solve the problem of difficult vehicle tracking and slow processing time in complex traffic scenarios. Our proposed framework incorporates strategies for short-term tracking and long-term matching. Short-term tracking is achieved through calculating the offset between consecutive frames, which has the characteristics of short-time consumption and high precision in traffic scenarios. Long-term tracking employs vehicle trajectory prediction and suspicious trajectory (ST) analysis in complex traffic scenarios. The tracking accuracy improved by matching the trajectory points and appearance features of continuous time series. The experimental results demonstrate that our method has superiority in Multi-target tracking (MOT) tasks in complex traffic scenarios.
Local-guided Global Collaborative Learning Transformer for Vehicle Reidentification

Vehicle reidentification(ReID) has attracted much attention and is significant for traffic security surveillance. Due to the variety of views of the same vehicle captured by different camera and the great similarity in the visual appearance of different vehicles, it is necessary to explore how to effectively utilize local detail information to achieve collaborative perception to highlight discriminative appearance features. Different from existing local feature exploration methods that focus on using extra part or keypoint information, we propose a global collaborative learning Transformer guided by local abstract features, named LG-CoT, which aims to highlight the highest-attention regions of vehicle images. We adopt Vision Transformer(ViT) as our backbone to extract global features and obtain all local tokens. To reduce the distribution from the background and drive the network to focus more on details, all attention maps containing low-level texture information and high-level semantic information are multiplied to obtain the local regions with highest-attention. Finally, we design a local-attention-guided pose-optimization feature encoding module, which can help the global features focus on local regions adaptively. Extensive experiments on two popular datasets and a dataset we built in a T-junction traffic scene suggest that our method can achieve comparable performance.
MHTTS: Fast Multi-head Text-to-speech For Spontaneous Speech With Imperfect Transcription

Neural network based end-to-end Text-to-Speech (TTS) has greatly improved the quality of synthesized speech. However, (the efficient usage of massive)[how to efficiently use massive] spontaneous speech without transcription remains an open problem. In this paper, we propose MHTTS, a fast multi-speaker TTS system that is robust to transcription errors of real speech data. Specifically, we introduce a multi-head model and a method that transfer text information from corpora with manual transcription to corpora without annotation. MHTTS has two advantages: 1) Our system synthesizes better quality multi-speaker voice with faster inference speed. 2) Our system is capable of transferring correct text information to data with imperfect transcription, simulated using corruption, or provided by an Automatic Speech Recognizer (ASR).
Order-sensitive Neural Constituency Parsing

We propose a novel algorithm that improves on the previous neural span-based CKY decoder for constituency parsing. In contrast to the traditional span-based decoding, where spans are combined only based on the sum of their scores, we introduce an order-sensitive strategy, where the span combination scores are more carefully derived from an order-sensitive basis. Our decoder can be regarded as a generalization over existing span-based decoder in determining a finer-grain scoring scheme for the combination of lower-level spans into higher-level spans, where we emphasize on the order of the lower-level spans and use order-sensitive span scores as well as order-sensitive combination grammar rule scores to enhance prediction accuracy. We implement the proposed decoding strategy harnessing GPU parallelism and achieve a decoding speed on par with state-of-the-art span-based parsers. Using the previous state-of-the-art model without additional data as our baseline, we outperform it and improve the F1 score on the Penn Treebank Dataset by 0.26% and on the Chinese Treebank Dataset by 0.35%.
A Chunked Local Aggregation Strategy in Federated Learning

Federated Learning (FL) is a distributed machine learning technology that trains models on large-scale distributed devices while keeping training data localized and privatized. However, in settings where data is distributed in a not independent and identically distributed (non-I.I.D.) fashion, the single joint model produced by FL suffers in terms of test set accuracy and communication costs. And a multi-layer topology are widely deployed for FL in real scenarios. Therefore, we propose FedBox, a chunked local aggregation federated learning framework to improve the generalization ability and aggregation efficiency of model in non-I.I.D. data by adapting to the topology of the real network. Moreover, we study the adaptive gradient descent (AGC) to mitigate the feature shift caused by training non-I.I.D. data. In this work, we modified the aggregation strategy of FL by introducing a virtual node layer based on local stochastic gradient methods (SGD), and separate the edge node cluster by the similarity between the local update model and the global update model. We show that FedBox can effectively improve convergence speed and test accuracy, while reducing communication cost. Training results on FederatedEMNIST, Cifar10, Cifar100 and Shakespeare datasets indicate that FedBox allows model training to converge in fewer communication rounds and improves training accuracy by up to 3.1% compared with FedAVG. In addition, we make an empirical analysis of the extended range of virtual nodes.
Network New Word Discovery Framework Based on Sentence Semantic Vector Similarity

New word discovery is a key problem in text information retrieval technology. Methods in new word discovery are often closely related to words. Because their target is words, the findings are obtained by designing methods to analyze words. With the popularity of social networks, individual netizens and online self-media have generated various network texts for the convenience of online life, including network new words that are far from standard Chinese expression. How detect network new words is one of the important goals in the field of new word discovery today. In this paper, we integrate the word embedding model and clustering methods to propose a network new word discovery framework based on sentence semantic similarity (S3-N2WD) to detect network new words effectively from the network texts. This framework constructs sentence semantic vectors through a distributed representation model, uses the similarity of sentence semantic vectors to determine the semantic relationship between sentences, and finally realizes new network word discovery by the meaning of semantic replacement between sentences. The experiment verifies that the framework not only completes the rapid discovery of network new words but also realizes the standard word meaning of the discovery of it, which reflects the effectiveness of our work.
Rotational-XOR Differential Rectangle Cryptanalysis on Simon-Like Ciphers

In this paper, we propose a rectangle-like method called rotational-XOR differential rectangle attack to search for better distinguishers. It is a combination of the rotational-XOR cryptanalysis and differential cryptanalysis in the rectangle-based way. In particular, we put a rotational-XOR characteristic before a differential characteristic to construct a rectangle structure. By choosing some appropriate rotational-XOR and differential characteristics as well as considering multiple differentials, some longer distinguishers that have the probability greater than $$2^{-2n}$$ can be constructed effectively where n is the block size of a block cipher. We apply this new method to some versions of Simon and Simeck block ciphers. As a result, we obtain rotational-XOR differential rectangle distinguishers up to 16, 16, 17, 16 and 21 rounds for Simon32/64, Simon48/72, Simon48/96, Simeck32 and Simeck48, respectively. Our distinguishers for Simon32/64 and Simon48/96 are both longer than the best differential and rotational-XOR distinguishers. Also, our distinguisher for Simeck32 is longer than the best differential distinguisher (14 rounds) and has the full weak key space (i.e., $$2^{64}$$ ) whereas the 16-round rotational-XOR distinguisher has a weak key class of $$2^{36}$$ . In addition, our distinguisher for Simeck48 has a better weak key class ( $$2^{72}$$ weak keys) than the 21-round rotational-XOR distinguisher ( $$2^{60}$$ weak keys). To the best of our knowledge, this is the first time to consider the combinational cryptanalysis based on rotational-XOR and differential cryptanalysis using the rectangle structure.
Event Detection with Cross-Sentence Graph Convolutional Networks

The goal of Event Detection (ED) task is to identify the words that mark the occurrence of events in text, and classify them into a set of event types. To model informative word semantics, some researchers apply Graph Convolutional Network (GCN) to exploit the syntactic graph transformed from the dependency tree, within one sentence. We are motivated to simultaneously leverage syntactic clues and context information across sentences. To this end, we propose a novel ED model with Cross-Sentence Graph Convolutional Networks (CSGCN). The CSGCN contains two main components, including a tree extension module and the syntax-aware graph convolution. Each sentence is parsed to a dependency tree by an automatic toolkit. The first module merges entity-specific subtrees from neighbor sentences into the dependency tree of current sentence, which constructs a cross-sentence dependency tree. On this basis, we transform the tree into an undirected graph. After that, a syntax-aware attention mechanism is employed in the computation of graph convolution. This mechanism dynamically captures syntax-relevant information from neighbor nodes via the graph structure. Finally, we devise an entity aggregation module to aggregate key entity information for trigger candidates. We conduct experiments on the ACE 2005 and KBP 2017 datasets. The results show that our model achieves satisfactory and competitive performance on ACE 2005, and outperforms all State-of-The-Art models on KBP 2017.
Implicit and Explicit Emotion Enhanced Empathetic Dialogue Generation

Empathetic conversation systems identify the users' emotions and give appropriate responses, which is crucial to improve users' experiences. However, existing empathetic dialogue models (especially to the dominant pre-trained language model-based systems) did not focus on modelling the holistic properties of implicit and explicit emotions. In this paper, we propose an Implicit and Explicit Emotion Enhanced (IEEE) empathetic dialogue generation model to handle such challenges. Specifically, we first propose a prompt tuning-based approach to mine emotional words as additional information to obtain the users' explicit emotion. A variational auto-encoder is then introduced to extract the topic words of the input sequence as additional priori knowledge to get the implicit emotion related information. Finally, a pre-trained language model is utilized as the auto-regressive decoder to generate empathetic responses related to the content of the topics and user emotions. To demonstrate the effectiveness of the proposed approach, IEEE has been tested on empathic dialogue dataset. The experimental results show that our method achieves better performance than some competitive models.
A Zero-shot Learning Method with a Multi-Modal Knowledge Graph

Zero-shot learning aims to recognize unseen-classes using some seen-class samples as training set. It is challenging owing to that the feature representations of unseen-class samples are unavailable. Existing methods transfer the mapping from seen-classes to unseen-classes with the correlation as a bridge, in which, the semantic representations are used to discriminate the classes. However, the unavailability of visual representations for unseen-classes and the insufficient discrimination of semantic representations make the zero-shot learning challenging. Therefore, the visual representations are learned as complements to semantic representations to construct a multi-modal knowledge graph (KG), and a zero-shot learning method based on multi-modal KG is proposed in this paper. Specially, a semantic KG is introduced to capture the correlation of classes, and with the correlation, the visual feature representations of all classes are learned. Then, the discriminative visual representations and the semantic representations are used together to construct a multi-modal KG. With the multi-modal KG, the classifier for seen-classes is transferred to unseen classes. Extensive experimental results show the effectiveness of our method.
A game theoretic approach to curriculum reinforcement learning

Current reinforcement learning automated curricu-lum approaches continual learning by updating the environment. The update is often treated as an optimisation problem - with the teacher agent updating the environment to optimise the student's learning. This work proposes an alternative framing of the problem using a game-theoretic formulation. The learning is defined by a leader - follower cooperative game. This formulation provides an approach for multi-agent curriculum learning that improves agent learning and provides more game equilibrium insights. We observed that under this framework, the agents converge faster to perform on the desired outcomes, compared to the reinforcement learning agent baseline.
A Search Space Utility Optimization Based Online POMDP Planning Algorithm

Online planning algorithms based on Monte Carlo tree search have been widely used to solve POMDP problems, but they still have difficulty handling large-scale scenarios. Although POMCPOW adapts to large-scale problems by using weighted particle filter and double progressive widening, it does not differentiate between the contributions of different levels, thus cannot make the best of the fixed number of simulations. A Partially Observable Monte Carlo Planning algorithm based on Search Space Utility Optimization (POMCP-SSUO) is proposed in this paper. POMCP-SSUO takes the level of simulated nodes as a factor of progressive widening, with which the upper bound of branches is negatively correlated. The proposed algorithm improves POMCPOW by exploring more valuable nodes and generating the search space with optimal utility. Experimental results demonstrate that POMCP-SSUO outperforms POMCPOW with varying numbers of simulations on three large benchmark problems.
KPE-GCN: A Keyphrase-Enhanced Graph Convolutional Network for Imbalanced Text Classification

With the rapid development of communication networks, the number of proposals in professional fields has increased explosively. In the era of big data, topic classification and retrieval of proposals has become an important demand for researchers. However, the automatic topic classification of proposals in the professional field has the characteristics of high semantic similarity, that is, fine-grained classification. It is difficult for text classification models in general fields to achieve high-precision classification effects on proposal datasets in professional fields. At the same time, the problem of class imbalance reduces the effective training of the model for minority classes, making the widespread application of the model difficult. In order to solve the proposal classification in professional fields, this paper proposes a keyphrase-enhanced graph convolutional network (KPE-GCN), which uses keyphrase-based data augmentation to effectively alleviate the problem of class imbalance. Meanwhile, we build a two-level heterogeneous graph that combines both word-level and keyphrase-level information, which can add domain features to the proposal structure. It can effectively distinguish the differences between fine-grained categories, and improve the accuracy of fine-grained classification of proposal. We perform extensive experiments on the proposal dataset, and KPE-GCN model can exhibit higher classification performance, raising the F1 value to over 99% for the first time. Our KPE-GCN model provides a unified domain feature extraction scheme, which can be widely used in various professional fields.
Modeling Offline Knowledge Evolution Effect for Online Knowledge Tracing

Knowledge Tracing (KT) focuses on tracing stu-dents' evolving knowledge state in the process of question-answering, which has been widely applied to online learning systems. Existing KT methods leverage students' online question-answering logs to infer their knowledge state and predict their subsequent performance. However, these methods only focus on students' online behaviors without considering the fact that students' knowledge also evolves offline. Specifically, students have plenty of offline learning actions that also make changes in their knowledge state, which is called the offline knowledge evolution effect in this paper. However, these offline actions are not recorded by online learning systems, therefore leading to unreliable knowledge tracing results. To alleviate this problem, we propose a novel model named offline-aware knowledge tracing (OKT) that considers students' offline knowledge evolution for KT. In O KT, we introduce a novel concept named offline trait to represent student's offline learning action preference. Based on the preference, the model evaluates offline knowledge evolution, and apply the evolution to generate a new knowledge state for students' future performance prediction. To the end, we conduct experiments on three real-world benchmark datasets and demonstrate that OKT achieves state-of-the-art performance. Besides, the experimental results indicate that considering offline actions can help interpret the KT results.