Reviewerly/exHarmony huggingface dataset:
Multi-Modal Representation via Contrastive Learning with Attention Bottleneck Fusion and Attentive Statistics Features

The integration of information from multiple modalities is a highly active area of research. Previous techniques have predominantly focused on fusing shallow features or high-level representations generated by deep unimodal networks, which only capture a subset of the hierarchical relationships across modalities. However, previous methods are often limited to exploiting the fine-grained statistical features inherent in multimodal data. This paper proposes an approach that densely integrates representations by computing image features' means and standard deviations. The global statistics of features afford a holistic perspective, capturing the overarching distribution and trends inherent in the data, thereby facilitating enhanced comprehension and characterization of multimodal data. We also leverage a Transformer-based fusion encoder to effectively capture global variations in multimodal features. To further enhance the learning process, we incorporate a contrastive loss function that encourages the discovery of shared information across different modalities. To validate the effectiveness of our approach, we conduct experiments on three widely used multimodal sentiment analysis datasets. The results demonstrate the efficacy of our proposed method, achieving significant performance improvements compared to existing approaches.
A new location verification protocol and blockchain-based drone rental mechanism in smart farming

In smart farming, information and communication technologies such as IoT are driving the agricultural industry towards improved efficiency and productivity while reducing human intervention. As agricultural systems become increasingly connected to the Internet, they provide many new opportunities but also introduce new challenges. Connecting such systems to a network exposes them to cyberattacks and vulnerabilities; therefore, security and privacy are important challenges that should be addressed. Specifically, because of the distinct characteristics of smart farming such as environmental and farm conditions, security solutions suitable for smart farming are indispensable. In this paper, we propose a location verification (LV) protocol as a security solution for autonomous and intelligent moving devices such as drones and robots deployed in smart farming. Moreover, as an application of the proposed LV protocol, we propose a blockchain-based secure drone rental mechanism for smart farming. We present a security analysis in which the proposed LV protocol and rental mechanism are secure against man-in-the-middle, impersonation, modification, replay, and relay attacks, with the rental mechanism providing mutual authentication. Through a performance evaluation of the LV protocol in terms of the communication overhead, computation cost, and storage cost and a performance evaluation of the blockchain for drone rental with regard to throughput and latency, we confirm the feasibility and efficiency of the LV protocol and drone rental mechanism in the smart farming.
The FineMotion entry to the GENEA Challenge 2023: DeepPhase for conversational gestures generation

This paper describes FineMotion’s entry to the GENEA Challenge 2023. We explore the potential of DeepPhase embeddings by adapting neural motion controllers to conversational gesture generation. This is achieved by introducing a recurrent encoder for control features. We additionally use VQ-VAE codebook encoding of gestures to support dyadic setup. The resulting system generates stable realistic motion controllable by audio, text and interlocutor’s motion.
FEIN-Z: Autoregressive Behavior Cloning for Speech-Driven Gesture Generation

Human communication relies on multiple modalities such as verbal expressions, facial cues, and bodily gestures. Developing computational approaches to process and generate these multimodal signals is critical for seamless human-agent interaction. A particular challenge is the generation of co-speech gestures due to the large variability and number of gestures that can accompany a verbal utterance, leading to a one-to-many mapping problem. This paper presents an approach based on a Feature Extraction Infusion Network (FEIN-Z) that adopts insights from robot imitation learning and applies them to co-speech gesture generation. Building on the BC-Z architecture, our framework combines transformer architectures and Wasserstein generative adversarial networks. We describe the FEIN-Z methodology and evaluation results obtained within the GENEA Challenge 2023, demonstrating good results and significant improvements in human-likeness over the GENEA baseline. We discuss potential areas for improvement, such as refining input segmentation, employing more fine-grained control networks, and exploring alternative inference methods.
Text length effects on the reliability of syntactic complexity indices

Automated tools are widely used to assess syntactic complexity in second language (L2) writing studies; however, the effects of text length on syntactic complexity indices remain unclear. This can pose a challenge when studying underrepresented populations (e.g., young learners, adults with limited literacy skills), as their lower proficiency may result in less text production. To address this issue, we investigated the minimum text length threshold at which automated measures of syntactic complexity become the most reliable while considering L2 proficiency and prompt topic. Essays from the ICNALE corpus, a dataset of 5,200 essays with four proficiency levels, were used to create a dataset of texts of varying lengths (50, 100, 150, and 200 words). Mixed-effects regression models showed that seven out of 14 indices were not affected by text length regardless of learner proficiency and prompt topic. The other seven differed only between the 50- and 200-word texts within intermediate levels. We suggest a minimum of 100 words as a conservative threshold for the reliability of syntactic complexity indices. Finally, we emphasize the importance of transparent reporting practice regarding text length information.
Multimodal Dialogue Understanding via Holistic Modeling and Sequence Labeling

This paper introduces the experimental schemes of Team HLT-base for the NLPCC-2023-Shared-Task-10 Learn to Watch TV: Multimodal Dialogue Understanding and Response Prediction (MDUG) competition. In this paper, we focus on two subtasks of multimodal dialogue understanding: the dialogue scene identification task and the dialogue session identification task. To solve these subtasks, we propose a simple and efficient multimodal framework, where two points are taken into account: i.e., modeling the interaction of different utterances and effectively fusing the information of different modalities. For the former, we concatenate all utterances into a single sentence and feed it into the pre-trained model; for the latter, we use a transformer layer to fuse the multimodal features. Extensive experiments show that our proposed framework achieves state-of-the-art (SOTA) performance compared with other competitive methods, and ranks 1st in both subtasks (i.e., track1: dialogue scene identification and track2: dialogue session identification) in the MDUG competition.
Towards Robust Chinese Spelling Check Systems: Multi-round Error Correction with Ensemble Enhancement

Chinese Spelling Check requires a system to automatically correct spelling errors in a sentence. There are diverse methods proposed to solve this task. A few methods improve the robustness of the model through data augmentation, but they have some weaknesses. Errors inserted randomly might disturb the real distribution of data. Moreover, different models may produce different results when predicting the same error sentence. Based on these intuitions, we develop a multi-round error correction method with ensemble enhancement, which is robust in solving Chinese Spelling Check challenges. Specifically, multi-round error correction follows an iterative correction pipeline, where a single error is corrected at each round, and the subsequent correction is conducted based on the previous results. Furthermore, we proposed two strategies of ensemble enhancement. For each predicted correction, results of multiple models are mutually authenticated by weighted voting and dominate voting. Experiments have proved the effectiveness of our system. It achieves the best performance on NLPCC 2023 CSC shared tasks. More analyses verify that both multi-round error correction and ensemble enhancement contribute to its good results. Our code is publicly available on GitHub.
A Comprehensive Survey of Meta-heuristic Algorithms

In the last years, the application of standard optimization algorithms, presented results typically inefficient in the solution of some problems, especially in non-linear (NP-hard) problems of large scale. This disadvantage motivated researchers to study different types of optimization. The introduction and use of metaheuristic optimization algorithms are to find optimal solutions with a logical computing time, especially for NP-hard problems. Most of the presented algorithms are based on different phenomena in nature. Some other works are found on the processes of human social behaviors, sports, physics, etc. It is noteworthy that there are always new meta-heuristic algorithms that are introduced every day for optimization. In this chapter, some selected different algorithms including PSO, ABC, ICA, IWO, QIWO, FA, and WCO, as the representative of the meta-heuristic algorithms is analyzed and studied, where the results could extend for all of the meta-heuristics.
Solving Math Word Problem with Problem Type Classification

Math word problems (MWPs) require analyzing text descriptions and generating mathematical equations to derive solutions. Existing works focus on solving MWPs with two types of solvers: tree-based solver and large language model (LLM) solver. However, these approaches always solve MWPs by a single solver, which will bring the following problems: (1) Single type of solver is hard to solve all types of MWPs well. (2) A single solver will result in poor performance due to over-fitting. To address these challenges, this paper utilizes multiple ensemble approaches to improve MWP-solving ability. Firstly, We propose a problem type classifier that combines the strengths of the tree-based solver and the LLM solver. This ensemble approach leverages their respective advantages and broadens the range of MWPs that can be solved. Furthermore, we also apply ensemble techniques to both tree-based solver and LLM solver to improve their performance. For the tree-based solver, we propose an ensemble learning framework based on ten-fold cross-validation and voting mechanism. In the LLM solver, we adopt self-consistency (SC) method to improve answer selection. Experimental results demonstrate the effectiveness of these ensemble approaches in enhancing MWP-solving ability. The comprehensive evaluation showcases improved performance, validating the advantages of our proposed approach. Our code is available at this url: https://github.com/zhouzihao501/NLPCC2023-Shared-Task3-ChineseMWP.
A Study on the Classification of Chinese Medicine Records Using BERT, Chest Impediment as an Example

Traditional Chinese Medicine (TCM) is the treasure of Chinese civilization and plays an indispensable role in China's medical system, but the diagnosis of TCM relies heavily on doctors' experience, which can affect the accuracy of diagnosis in practice. With the development of natural language processing technology, its mechanism can learn from a large amount of unstructured text to obtain a comprehensive and unified classification model. In this paper, we take chest impediment disease ( i.e. coronary heart disease in Western medicine) as an example and build a pre-training diagnostic model based on the BERT model for TCM texts to accomplish the text classification task for different types of chest impediment medical records. Its overall F1 value reached 0.851, which improved 0.096 compared with the model without TCM pre-training; it also explored the problem of long text truncation and stopwords removing of TCM cases, which improved 0.087 compared with no TCM stopwords removing. This paper introduces natural language processing into the TCM auxiliary diagnosis problem, in order to improve the informationization, standardization and intelligence of TCM in the new era.
A novel low cybersickness dynamic rotation gain enhancer based on spatial position and orientation in virtual environments

Abstract Hardware and software resources nowadays make possible new Virtual Reality (VR) interaction methods. Numerous challenges have been involved over the years, and one of the most interesting is locomotion in virtual environments. In particular, Real Walking (RW) is one of the most effective Virtual Locomotion Techniques (VLT). It causes only reduced cybersickness, but it also requires proportional real walkable space to the virtual one, often requiring wide areas. In this context, optimization techniques have been proposed in the literature, e.g., reorientation or relocation. In this work, a novel method for improving reorientation in a virtual environment, exploiting a dynamic Rotation Gain Multiplication Factor (RGMF) based on the competence of the user in VR systems usage is proposed. The results highlight the effectiveness of the system and show the specific target of users that mainly appreciated it.
IMTM: Invisible Multi-trigger Multimodal Backdoor Attack

Computer Vision and Natural Language Processing have made significant advancements, leading to the emergence of multimodal models that seamlessly integrate diverse input modalities. While these models hold great potential for various applications, their complex nature with multiple inputs also makes them more susceptible to attacks. Unfortunately, minimal attention has been devoted to backdoor attacks targeting multimodal models. In this work, we propose a stealthy backdoor attack method called “Invisible Multi-trigger Multimodal Backdoor” for multimodal deep learning models. Inspired by steganography, our method leverages the features of multiple input sources to clandestinely embed multi-triggers that are undetectable to humans, enabling a covert backdoor attack on multimodal models. This study represents a pioneering exploration of steganography application within the context of multimodal models. To the best of our knowledge, no prior research has employed steganography techniques specifically for multimodal models. Experimental results demonstrate the effectiveness and scalability of our approach, as it successfully executes the attack without compromising the normal model’s performance. With a 1% poisoning rate, our attack achieves a 98.9% attack success rate, while maintaining high model accuracy. To foster research on defense mechanisms against multi-modal backdoors, we have made available a COCO toxic dataset (TrojCOCO). This dataset has been meticulously curated to specifically address the study of vulnerabilities associated with these types of attacks.
Overview of NLPCC Shared Task 2: Multi-perspective Scientific Machine Reading Comprehension

In this report, we give an overview of the shared task about multi-perspective scientific machine reading comprehension at the 12th CCF Conference on Natural Language Processing and Chinese Computing (NLPCC 2023). Scientific machine reading comprehension (SMRC) aims to understand scientific texts through interactions with humans by given questions. In this task, questions about scientific texts include perspectives from beginners, students and experts. It requires different levels of understanding of scientific texts. We describe the task, the corpus, the participating teams and their results.
A knowledge-augmented neural network model for sarcasm detection

Automatic sarcasm detection from text is one important research task in text mining and natural language processing and has attracted extensive attention from researchers. Most approaches focus on designing various models and features according to the original text, without making use of the knowledge and information from external knowledge source such as Wikipedia, which is publicly available. In this paper, we investigate a knowledge-augmented neural network model that leverages the contextual information of the original text from external knowledge source, for sarcasm detection. We first extract the context from external knowledge source for the original text. Then, the original text and its context are fed sequentially into the embedding layer and encoding layer, to automatically learn high-level semantic representation. Finally, we use the softmax layer to output the classification probability. Based on the Semeval-2018 Task 3 dataset, results show that our proposed model gives 82.79% F1 score, outperforming the existing models and strong neural baselines with significant margins. Experimental analysis also indicates that the contextual information is highly important for sarcasm detection.
Geometric properties of quantum entanglement and machine learning

Objectives . Fast data analysis based on hidden patterns is one of the main issues for adaptive artificial intelligence systems development. This paper aims to propose and verify a method of such analysis based on the representation of data in the form of a quantum state, or, alternatively, in the form of a geometric object in a space allowing online machine learning. Methods. This paper uses Feynman formalism to represent quantum states and operations on them, the representation of quantum computing in the form of quantum circuits, geometric transformations, topological classification, as well as methods of classical and quantum machine learning. The Python programming language is used as a development tool. Optimization tools for machine learning are taken from the SciPy module. The datasets for analysis are taken from open sources. Data preprocessing was performed by the method of mapping features into numerical vectors, then the method of bringing the data to the desired dimension was applied. The data was then displayed in a quantum state. A proprietary quantum computing emulator is used (it is in the public domain). Results . The results of computational experiments revealed the ability of very simple quantum circuits to classify data without optimization. Comparative indicators of classification quality are obtained without the use of optimization, as well as with its use. Experiments were carried out with different datasets and for different values of the dimension of feature spaces. The efficiency of the models and methods of machine learning proposed in the work, as well as methods of combining them into network structures, is practically confirmed. Conclusions . The proposed method of machine learning and the model of quantum neural networks can be used to create adaptive artificial intelligence systems as part of an online learning module. Free online optimization learning process allows it to be applied in data streaming, that is, adapting to changes in the environment. The developed software does not require quantum computers and can be used in the development of artificial intelligence systems in Python as imported modules.
Secure IoT-enabled sharing of digital medical records: An integrated approach with reversible data hiding, symmetric cryptosystem, and IPFS

In the digitizing medical field, secure sharing of patients' medical records over the Internet while preserving privacy is critical. Medical image encryption safeguards healthcare, IoMT, and cloud services from increasing digital threats. Incorporating vital diagnostic data like blood pressure, haemoglobin levels, sugar levels, body temperature, and medical images such as X-rays, CT, and MRI scans, comprehensive medical records benefit from Internet of Things (IoT) technology. This research paper presents a secure solution for sharing medical records among medical experts. The system employs the tamper-proof InterPlanetary File System (IPFS) for secure storage and distribution. It establishes a cryptographic association between medical reports and images, enabling secure outsourcing and retrieval by authorized users. This is achieved through the use of the reversible data hiding technique based on the Parametric Binary Tree Labeling (PBTL) scheme, efficiently linking Medical Reports and Images. To ensure robust security, the Medical Report in PDF format is encrypted using the Advanced Encryption Standard (AES-256) for symmetric encryption, and the encrypted file is securely uploaded to IPFS, generating a hash value serving as a reliable reference for future use. Furthermore, IPFS treats the hash value as confidential information and embeds it into the medical image using a novel Algorithm to embed secrets. Subsequently, the embedded image undergoes encryption before being uploaded to IPFS. The extensive evaluation and comparison with existing systems affirm the effectiveness of the proposed method in ensuring security and privacy preservation. Various analyses, including image quality, sensitivity, security, correlation, quality, and attack robustness, demonstrate the efficiency and strength of the proposed approach.
A Three-Player GAN for Super-Resolution in Magnetic Resonance Imaging

Learning based single image super resolution (SISR) is well investigated in 2D images. However, SISR for 3D magnetic resonance images (MRI) is more challenging compared to 2D, mainly due to the increased number of neural network parameters, the larger memory requirement, and the limited amount of available training data. Current SISR methods for 3D volumetric images are based on generative adversarial networks (GANs), especially Wasserstein GANs. Other common architectures in the 2D domain, e.g. transformer models, require large amounts of training data and are therefore not suitable for the limited 3D data. As a popular GAN variant, Wasserstein GANs sometimes produce blurry results as they may not converge to a global (or local) optimum. In this paper, we propose a new method for 3D SISR using a GAN framework. Our approach incorporates instance noise to improve the training stability of the GAN, as well as a relativistic GAN loss function and an updating feature extractor as the third player during the training process. Our experiments demonstrate that our method produces highly accurate results using very few training samples. Specifically, we show that we need less than 30 training samples, which is a significant reduction compared to the thousands typically required in previous studies. Moreover, our method improves out-of-sample results, demonstrating its effectiveness. The code is available in https://github.com/wqlevi/threeplayerGANSR .