Reviewerly/exHarmony huggingface dataset:
Portrait Matting Network with Essential Feature Mining and Fusion

We propose an end-to-end portrait matting algorithm that emphasizes the mining and fusion of critical features to achieve higher accuracy. Previous best-performing portrait matting algorithms still have difficulties in handling difficult parts of portraits, such as hair, and often rely on increasing the network capacity to achieve better accuracy. In this paper, we argue that the effective use of critical features can help the network recover details while adding little to the network capacity and computational effort. Our proposed approach has the following two advantages: 1) it emphasizes the multiplexing of the lowest and highest level feature maps, which allows the model to recover details more effectively while locating the portrait subject more accurately. 2) we introduce a critical-feature-extraction-block to further mine this part of the feature maps, making the model parameters more effective. We evaluate the proposed method on a large portrait dataset P3M-10K, and the experimental results show that our method outperforms other portrait matting models.
Frequency Spectrum with Multi-head Attention for Face Forgery Detection

Incredible realistic fake faces can be easily created using various Generative Adversarial Networks (GANs). For detecting GAN-generated fake images, various popular heavy-weight convolutional neural networks (CNNs) and vision transformers (ViTs) are developed with high accuracy. These methods have high computational costs and require large datasets for training. Their accuracy drops when tested on unseen datasets. To overcome these limitations, we have proposed a light-weight robust model with a significantly lesser number of parameters (approx 0.2 million). We combine the multi-head attention of the transformer with features extracted through frequency and laplacian spectrum of an image. It processes both global and local information of the image for forgery detection. It requires a very small dataset for training and shows better generalization results on unseen datasets. We have performed our experiments on smaller sets of DFFD dataset and tested on larger sets. The proposed model achieves an accuracy of 99%, having significantly fewer parameters in in-domain settings and thus is computationally less expensive. It is also tested on unseen datasets in cross GANs setting with an accuracy that is at par with the existing state-of-the-art, albeit heavy model ResNet-50 and other light-weight models such as MobileNetV3, SqueezeNet, and MobileViT.
An End-to-End Chinese and Japanese Bilingual Speech Recognition Systems with Shared Character Decomposition

The rising number of tourists in most areas in East Asia has increased the requirement for East Asian speech recognition systems (e.g., Chinese and Japanese). However, the large existing character vocabulary limits the performance of Chinese-Japanese bilingual speech recognition systems. In this study, we propose a novel End-to-End ASR system for Chinese-Japanese bilingual circumstances based on a smaller granularity and quantity modeling unit set. We apply the decomposition strategy to both Chinese character and Japanese Kanji, so that two languages can also benefit from the similar representations of sub-character level during the joint training. For Chinese-Japanese bilingual speech, our proposed bilingual model outperformed our monolingual baseline models for both Japanese and Chinese. This proves the feasibility of joint modeling among similar languages, while decreasing the modeling units.
A Fast Stain Normalization Network for Cervical Papanicolaou Images

The domain shift between different styles of stain images greatly challenges the generalization of computer-aided diagnosis (CAD) algorithms. To bridge the gap, color normalization is a prerequisite for most CAD algorithms. The existing algorithms with better normalization effect often require more computational consumption, resisting the fast application in large-size medical stain slide images. This paper designs a fast normalization network (FTNC-Net) for cervical Papanicolaou stain images based on learnable bilateral filtering. In our FTNC-Net, explicit three-attribute estimation and spatially adaptive instance normalization are introduced to guide the model to transfer stain color styles in space accurately, and dynamic blocks are adopted to adapt multiple stain color styles. Our method achieves at least 80 fps over 1024 $$\times $$ 1024 images on our experimental platform, thus it can synchronize with the scanner for image acquisition and processing, and has advantages in visual and quantitative evaluation compared with other methods. Moreover, experiments on our cervical staining image dataset demonstrate that the FTNC-Net improves the precision of abnormal cell detection.
Local-Global Interaction and Progressive Aggregation for Video Salient Object Detection

Video salient object detection (VSOD) aims at locating and segmenting visually distinctive objects in a video sequence. There still exist two problems that are not well handled in VSOD. First, facing unequal and unreliable spatio-temporal information in complex scenes, existing methods only exploit local information from different hierarchies for interaction and neglect the role of global saliency information. Second, they pay little attention to the refinement of the modality-specific features by ignoring fused high-level features. To alleviate the above issues, in this paper, we propose a novel framework named IANet, which contains local-global interaction (LGI) modules and progressive aggregation (PA) modules. LGI locally captures complementary representation to enhance RGB and OF (optical flow) features mutually, and meanwhile globally learns confidence weights of the corresponding saliency branch for elaborate interaction. In addition, PA evolves and aggregates RGB features, OF features and up-sampled features from the higher level, and can refine saliency-related features progressively. The sophisticated designs of interaction and aggregation phases effectively boost the performance. Experimental results on six benchmark datasets demonstrate the superiority of our IANet over nine cutting-edge VSOD models.
Graph Attention Mixup Transformer for Graph Classification

Transformers have been successfully applied to graph representation learning due to the powerful expressive ability. Yet, existing Transformer-based graph learning models have the challenge of overfitting because of the huge number of parameters compared to graph neural networks (GNNs). To address this issue, we propose an end-to-end regularized training scheme based on Mixup for graph Transformer models called Graph Attention Mixup Transformer (GAMT). We first apply a GNN-based learnable graph coarsening method to align the number of nodes and preserve local neighborhood information. Then, we mix the pairwise graph attention weights and node embeddings as additional synthetic training samples to train a graph Transformer model with smoother decision boundaries. Finally, we introduce contrastive learning to further regularize the graph Transformer model. Extensive experiments on real-world graph classification benchmark datasets show that GAMT improves the performance and generalization ability of graph Transformer models without additional computing resource consumption.
Neural Distinguishers on $$\texttt {TinyJAMBU-128}$$ and $$\texttt {GIFT-64}$$

In CRYPTO 2019, Gohr first introduced a pioneering attempt, and successfully applied neural differential distinguisher ( $$\mathcal {NDD}$$ ) based differential cryptanalysis against Speck32/64, achieving higher accuracy than the pure differential distinguishers and reducing the data complexity of chosen plaintexts. Inspired by Gohr’s work, we attempt to use neural network to analyze the cipher $$\texttt {TinyJAMBU-128}$$ which is one of ten NIST’s lightweight cryptography standardization process finalists. Based on MLP, we construct a Neural Single Differential Distinguisher ( $$\mathcal {NSDD}$$ ), on which we get an accuracy of $$99.58\%$$ with 32-bit associated data(AD). The experiment results show that $$\texttt {TinyJAMBU-128}$$ with 32-bit AD is vulnerable to differential attacks. In this article, we also explore $$\texttt {GIFT-64}$$ . Based on Long Short-Term Memory (LSTM), we construct $$\mathcal {NSDD}$$ and Neural Polytopic Differential Distinguisher( $$\mathcal {NPDD}$$ ). For 4-,5-,6-round $$\texttt {GIFT-64}$$ , we get an accuracy of $$99.73\%, 85.08\%$$ , $$57.54\%$$ with $$\mathcal {NPDD}$$ and obtain an accuracy of $$97.97\%, 75.11\%, 57.25\%$$ with $$\mathcal {NSDD}$$ respectively. Compared with Yadav’s research in which MLP is used, we get a higher acccuracy with only $$\frac{1}{4}$$ train dataset. It shows that our model is better than Yadav’s.
Learning from Hindsight Demonstrations

Learning from demonstrations (LfD) is an important technique to help reinforcement learning (RL) boost the training process, especially in the case of sparse rewards. But a major obstacle is the acquisition of expert demonstrations, which is difficult or expensive to obtain in many cases. In this paper, we propose a unique method called Learning from Hindsight Demonstrations (LfHD) to automatically produce hindsight demonstrations, on which LfD can be performed and the cost of acquiring expert demonstrations is avoided. The produced demonstrations are comparable to those of experts at certain success rate. We also improve the LfD method to make better use of the produced demonstrations. Experiments show that our method can greatly improve the training efficiency compared to existing algorithms.
Safety Issues Investigation in Deep Learning Based Chatbots Answers to Medical Advice Requests

Recently, most of open-domain dialogue systems or chatbots have been trained using the deep learning technique on large human conversations from the internet. They can generate more natural and diverse responses than task-oriented or retrieval-based ones. However, their response generation is difficult to control, and they can learn and produce unsuitable and even unsafe responses. In this paper, we investigate the ability of deep learning based chatbots to produce unsafe medical advice when they receive requests for medical advice from the end users. We introduce a new benchmark for training medical context detector in a chatbot message. Then we conduct experiments to assess the safety of two well-known chatbots answers to medical advice requests and discuss the limitations of the proposed method. Our study demonstrates that popular neural network based chatbot models have a significant propensity to produce unsafe medical advice.
ND-NER: A Named Entity Recognition Dataset for OSINT Towards the National Defense Domain

The public data on the Internet contains a large amount of high-value open source intelligence (OSINT) for the national defense. As the fundamental information extraction task, Named Entity Recognition (NER) plays a key role in question answering systems, knowledge graphs and reasoning. However, NER for the national defense domain achieves little progress due to unavailable datasets. Most previous methods mainly work on general-purpose datasets which lack insight into the particularity of the national defense. In this paper, we propose a Chinese NER dataset, ND-NER, for the national defense based on the data crawled from Sina Weibo. This is the first public human-annotation NER dataset for OSINT towards the national defense domain with 19 entity types and 418,227 tokens. We construct two baseline tasks and implement a series of popular models on our dataset. The empirical results show that ND-NER is a challenging dataset concerning the long entities with the nest structure, domain specialization, ambiguous entity boundaries, informality and colloquialism issues of social media. We believe that the published ND-NER at https://github.com/XinyanLi2016/ND-NER will encourage further exploring for OSINT towards the national defense domain.
Extractive Question Answering Using Transformer-Based LM

Many institutions, organizations, and government bodies deal with a large number of financial documents (which can be structured or unstructured). To avoid the labor-intensive, manual tasks, we propose a Question Answering System in the finance domain to create profitable and competitive advantages for various organizations by making it easier for financial advisors to make decisions. Various pre-trained language models have proven highly effective at extractive question answering. Yet, generalizability stays a challenge for most of these pre-trained language models. In our work, we trained and fine-tuned RoBERTa model on other questions answering datasets of varying difficulty levels to decide which models are competent for generalizing the most thoroughly across varying datasets. Further, we proposed a new methodology to handle long-form answers by modifying the BERT and RoBERTa architecture. We have added the dynamic masking (instead of using static masking) and performed stride-shift (similar to kernel shift in computer vision) in BERT and RoBERTa architecture and compared it with different pre-trained LM to decide if adding dynamic masking and shifting the strides can improve model performance. We have used MRR (Mean Reciprocal Rank), NDCG (Normalized Discounted Cumulative Gain), and Precision@1 to check the performance of our model on FiQA datasets. Moreover, we have used F1-score and Exact Match as performance metrics to set the benchmark for review-based SubjQA datasets. We found out that combining RoBERTa with dynamic masking and stride shift and using Dense Passage Retriever for extracting relevant passages performs the best on both the datasets SubjQA and Financial Question Answer (FiQA) [1, 2], and it outperforms the baseline BERT model. The results show an improvement in each metric as measured against the various other models.
Towards Human Keypoint Detection in Infrared Images

Human keypoint detection is not applicable in low-light and nighttime conditions. In this work, we innovatively use infrared images for multi-person keypoint detection, which makes some computer vision tasks, such as action recognition and behavior analysis, applicable in complex illumination environments. By fully considering the physical characteristics of infrared imaging, we design a top-down solution that first uses a single-stage target detection network, YOLO, to predict the bounding box of the human body, then feed the detected human body into a following human keypoint detection network. We chose SimpleBaseline, well-known in human keypoint detection using visible images, as the base network. Since the infrared image is blur imaging and low resolution, we use targeted feature fusion, channel attention, and spatial attention to capture the feature of the infrared image. In addition, we use depth-separable convolution to reduce the number of parameters in the network. In the literature, there is no benchmark infrared image dataset for multi-person keypoint detection. We construct an infrared image dataset containing 1500 annotated images carefully selected from several public infrared pedestrian datasets. Compared with the SimpleBaseline, extensive experimental results show that our method achieves nearly the same performance on the visible COCO dataset, but has about 8% higher AP on the self-built infrared dataset.
Hindsight Balanced Reward Shaping

Sparse rewards is a tricky problem in reinforcement learning and reward shaping is commonly used to solve the problem of sparse rewards in specific tasks, but it often requires priori knowledge and manually designing rewards, which are costly in many cases. Hindsight experience replay (HER) solves the problem of sparse rewards in multi-goal scenarios by replacing the goal of a failed trajectory with a virtual goal. Our method integrates the ideas of reward shaping and HER, which has two advantages: First, it can automatically perform reward shaping without manually-designed reward functions; Second, it can solve the problem arising from the use of virtual goals in HER. Experiment results show our method can significantly improve the performance in both Bit-Flipping environment and Mujoco environment.
How the Presence of Cognitive Biases in Phishing Emails Affects Human Decision-Making?

The rate of phishing attacks is increasing over time. Although hackers design emails with cognitive biases for their phishing attacks to succeed, little is known about how effectively these biases fool people via phishing emails. Also, little is known how machine learning algorithms can predict human tendency to get phished via phishing emails in the presence of human attributes. In this paper, the main objective is to investigate how the presence of two cognitive biases, authority bias (the tendency of humans to get influenced by the emails sent by authority) and hyperbolic discounting bias (the inclination of humans towards immediate rewards), influence human decision making via a phishing email detection simulation. In an experiment, 210 participants judged emails to be genuine or phishing. The next part of this research predicted the human responses to phishing emails captured in the experiment via machine learning models such as logistic regression (LR), multinomial Naive Bayes (MNB), decision tree (DT), and Random Forest (RF). The results from the study conducted on humans revealed that the authority bias was more effective compared to hyperbolic discounting in phishing humans. Furthermore, the LR classifier effectively predicted human responses in the presence of cognitive biases and human attributes with training and test accuracy of around 90.77% and 82.70%, respectively. We discuss the implications of this work for real-world phishing attacks.
HPC Based Scalable Logarithmic Kernelized Fuzzy Clustering Algorithms for Handling Big Data

With the advent of big data in the new millennium, the previous scalable clustering methods were no longer able to match the accuracy and efficiency requirements of big clustering. In light of this, we propose a fuzzy-based scalable incremental kernelized clustering algorithm for Big Data. First, we present the details of scalable kernelized fuzzy clustering techniques for Big Data that are based on the Radial Basis Function (RBF). Next, we define the membership degree and the cluster center for the logarithmic kernel function. For the purpose of managing Big Data, the Logarithmic Kernelized Scalable Random Sampling with Iterative Optimization Fuzzy c-Means (LKSRSIO-FCM) clustering algorithm has been developed on Apache Spark. These kernel functions translate the input data space non-linearly onto a high-dimensional feature space, so, these kernelized clustering approaches have developed in order to deal with non-linearly separable problems. Hence, our aim is to design and implement the logarithmic kernelized incremental fuzzy clustering algorithms on Apache Spark, which, as a result of its in-memory cluster computing methodology, is able to effectively perform the clustering of Big Data. Extensive experiments on a variety of datasets derived from the real world demonstrate that the proposed LKSRSIO-FCM has superior performance to the scalable kernelized fuzzy clustering algorithms based on the RBF kernel in terms of Normalized Mutual Information (NMI), Adjusted Rand Index (ARI), and F-score, respectively. The results were obtained by comparing the LKSRSIO-FCM to the scalable kernelized fuzzy clustering algorithms.
Binary Orthogonal Non-negative Matrix Factorization

We propose a method for computing binary orthogonal non-negative matrix factorization (BONMF) for clustering and classification. The method is tested on several representative real-world data sets. The numerical results confirm that the method has improved accuracy compared to the related techniques. The proposed method is fast for training and classification and space efficient.
Multi-Grained Fusion Graph Neural Networks for Sequential Recommendation

The target of sequential recommendation is to predict the next item that users will interact with according to their historical interaction sequences. The next item depends largely on several items that the user has just accessed. However, sequential recommendation systems face some challenges due to substantial increase of users and items: (1) the hardness of integrating the multi-grained interests based on multiple aspects from sparse implicit feedback; (2) the difficulty of fusing long-term and short-term interests. In this paper, we design a new method called Multi-Grained Fusion Graph Neural Networks (MGF-GNN) to address the above challenges. In particular, we utilize a hierarchical graph neural networks to model user short-term interests. In addition, we capture coarse-grained and fine-grained interests by attention mechanism and then fuse them as a multi-grained interest representation. Empirical studies on three real-world datasets demonstrate the effectiveness of our proposed method.
MulEA: Multi-type Entity Alignment of Heterogeneous Medical Knowledge Graphs

The large-scale application of medical knowledge graphs has greatly raised the intelligence level of modern medicine. Considering that entity references between multiple medical knowledge graphs can lead to redundancy, knowledge graph alignment tasks are required to identify entity pairs or subgraphs of heterogeneous knowledge graphs pointing to the same elements in the real world. Existing medical knowledge graph alignment methods do not consider the multi-type features of medical entities. To tackle above challenge, we propose a Multi-type Entity Alignment model of medical knowledge graphs based on attention mechanism named MulEA. Firstly, MulEA integrates the multi-type information features of medical entities to align various types of entities jointly through constructing an entity multi-type information embedding matrix for medical knowledge graphs. Secondly, MulEA designs a relationship collective aggregation module to fully utilize the features of different relationships to improve alignment accuracy. Finally, we evaluate the performance of MulEA on a real medical knowledge graph alignment dataset MED-OWN-15K, and it achieves the state-of-the-art performance on several metrics, showing the effectiveness of our method.
MixMBR: Contrastive Learning for Multi-behavior Recommendation

Contrastive learning has emerged as a dominant technique for unsupervised representation learning. Recent studies reveal that contrastive learning can effectively alleviate the limited supervision signals in Multi-behavior Recommendation(MBR). However, we argue that contrastive learning is not directly applicable to MBR due to (i) embedding different behaviors of the anchor close to positive samples violates the diversity of multiple behaviors; (ii) neglecting the effect of hard negative samples leads to sub-optimal performance. Inspired by mixup operation as a form of data argumentation in the field of computer vision, we propose MixMBR, a simple learning principle to alleviate these issues. Specifically, for each behavior, we execute the LightGCN to learn the user and item embeddings for a supervised task with dual type-specific feature transformation matrices. Meanwhile, we interpolate these learned transformation matrices to generate mixed views for a contrastive learning task. By doing so, parameter interpolation yields a parameter sharing contrastive learning, resulting in mining hard negative samples and preserving commonalities hidden in different behaviors. Extensive experiments on two real-world datasets indicate that our method outperforms state-of-the-art recommendation methods.