Reviewerly/exHarmony huggingface dataset:
Are Neural Open-Domain Dialog Systems Robust to Speech Recognition Errors in the Dialog History? An Empirical Study

Large end-to-end neural open-domain chatbots are becoming increasingly popular. However, research on building such chatbots has typically assumed that the user input is written in nature and it is not clear whether these chatbots would seamlessly integrate with automatic speech recognition (ASR) models to serve the speech modality. We aim to bring attention to this important question by empirically studying the effects of various types of synthetic and actual ASR hypotheses in the dialog history on TransferTransfo, a state-of-the-art Generative Pre-trained Transformer (GPT) based neural open-domain dialog system from the NeurIPS ConvAI2 challenge. We observe that TransferTransfo trained on written data is very sensitive to such hypotheses introduced to the dialog history during inference time. As a baseline mitigation strategy, we introduce synthetic ASR hypotheses to the dialog history during training and observe marginal improvements, demonstrating the need for further research into techniques to make end-to-end open-domain chatbots fully speech-robust. To the best of our knowledge, this is the first study to evaluate the effects of synthetic and actual ASR hypotheses on a state-of-the-art neural open-domain dialog system and we hope it promotes speech-robustness as an evaluation criterion in open-domain dialog.
RethinkCWS: Is Chinese Word Segmentation a Solved Task?

The performance of the Chinese Word Segmentation (CWS) systems has gradually reached a plateau with the rapid development of deep neural networks, especially the successful use of large pre-trained models.In this paper, we take stock of what we have achieved and rethink what's left in the CWS task.Methodologically, we propose a finegrained evaluation for existing CWS systems, which not only allows us to diagnose the strengths and weaknesses of existing models (under the in-dataset setting), but enables us to quantify the discrepancy between different criterion and alleviate the negative transfer problem when doing multi-criteria learning.Strategically, despite not aiming to propose a novel model in this paper, our comprehensive experiments on eight models and seven datasets, as well as thorough analysis, could search for some promising direction for future research.We make all codes publicly available and release an interface that can quickly evaluate and diagnose user's models: https://github.com/neulab/InterpretEval.
Explanation-Based Weakly-Supervised Learning of Visual Relations with Graph Networks

Visual relationship detection is fundamental for holistic image understanding. However, the localization and classification of (subject, predicate, object) triplets remain challenging tasks, due to the combinatorial explosion of possible relationships, their long-tailed distribution in natural images, and an expensive annotation process. This paper introduces a novel weakly-supervised method for visual relationship detection that relies on minimal image-level predicate labels. A graph neural network is trained to classify predicates in images from a graph representation of detected objects, implicitly encoding an inductive bias for pairwise relations. We then frame relationship detection as the explanation of such a predicate classifier, i.e. we obtain a complete relation by recovering the subject and object of a predicted predicate. We present results comparable to recent fully- and weakly-supervised methods on three diverse and challenging datasets: HICO-DET for human-object interaction, Visual Relationship Detection for generic object-to-object relations, and UnRel for unusual triplets; demonstrating robustness to non-comprehensive annotations and good few-shot generalization.
Dragonblood is Still Leaking: Practical Cache-based Side-Channel in the Wild

Recently, the Dragonblood attacks have attracted new interests on the security of WPA-3 implementation and in particular on the Dragonfly code deployed on many open-source libraries. One attack concerns the protection of users passwords during authentication. In the Password Authentication Key Exchange (PAKE) protocol called Dragonfly, the secret, namely the password, is mapped to an elliptic curve point. This operation is sensitive, as it involves the secret password, and therefore its resistance against side-channel attacks is of utmost importance. Following the initial disclosure of Dragonblood, we notice that this particular attack has been partially patched by only a few implementations.
PTN: A Poisson Transfer Network for Semi-supervised Few-shot Learning

The predicament in semi-supervised few-shot learning (SSFSL) is to maximize the value of the extra unlabeled data to boost the few-shot learner. In this paper, we propose a Poisson Transfer Network (PTN) to mine the unlabeled information for SSFSL from two aspects. First, the Poisson Merriman–Bence–Osher (MBO) model builds a bridge for the communications between labeled and unlabeled examples. This model serves as a more stable and informative classifier than traditional graph-based SSFSL methods in the message-passing process of the labels. Second, the extra unlabeled samples are employed to transfer the knowledge from base classes to novel classes through contrastive learning. Specifically, we force the augmented positive pairs close while push the negative ones distant. Our contrastive transfer scheme implicitly learns the novel-class embeddings to alleviate the over-fitting problem on the few labeled data. Thus, we can mitigate the degeneration of embedding generality in novel classes. Extensive experiments indicate that PTN outperforms the state-of-the-art few-shot and SSFSL models on miniImageNet and tieredImageNet benchmark datasets.
Mask Selection and Propagation for Unsupervised Video Object Segmentation

In this work we present a novel approach for Unsupervised Video Object Segmentation, that is automatically generating instance level segmentation masks for salient objects and tracking them in a video. We efficiently handle problems present in existing methods such as drift while temporal propagation, tracking and addition of new objects. To this end, we propose a novel idea of improving masks in an online manner using ensemble of criteria whose task is to inspect the quality of masks. We introduce a novel idea of assessing mask quality using a neural network called Selector Net. The proposed network is trained is such way that it is generalizes across various datasets. Our proposed method is able to limit the noise accumulated along the video, giving state of the art result on Davis 2019 Unsupervised challenge dataset with $\mathcal{J}\& {\mathcal{F}}$ mean 61.6%. We also tested on datasets such as FBMS and SegTrack V2 and performed better or on par compared to the other methods.
Evaluating Fidelity of Explainable Methods for Predictive Process Analytics

Predictive process analytics focuses on predicting the future states of running instances of a business process. While advanced machine learning techniques have been used to increase the accuracy of predictions, the resulting predictive models lack transparency. Explainable machine learning methods can be used to interpret black-box models. However, it is unclear how fit for purpose these methods are in explaining process predictive models. In this paper, we aim to investigate the capabilities of two explainable methods, LIME and SHAP, in reproducing the decision-making processes of black-box process predictive models. We focus on fidelity metrics and propose a method to evaluate the faithfulness of LIME and SHAP when explaining process predictive models built on a Gradient Boosting Machine classifier. We conduct the evaluation using three real-life event logs and analyze the fidelity evaluation results to derive insights. The research contributes to evaluating the trustworthiness of explainable methods for predictive process analytics as a fundamental and key step towards human user-oriented evaluation.
Ideography Leads Us to the Field of Cognition: A Radical-Guided Associative Model for Chinese Text Classification

Cognitive psychology research shows that humans have the instinct for abstract thinking, where association plays an essential role in language comprehension. Especially for Chinese, its ideographic writing system allows radicals to trigger semantic association without the need of phonetics. In fact, subconsciously using the associative information guided by radicals is a key for readers to ensure the robustness of semantic understanding. Fortunately, many basic and extended concepts related to radicals are systematically included in Chinese language dictionaries, which leaves a handy but unexplored way for improving Chinese text representation and classification. To this end, we draw inspirations from cognitive principles between ideography and human associative behavior to propose a novel Radical-guided Associative Model (RAM) for Chinese text classification. RAM comprises two coupled spaces, namely Literal Space and Associative Space, which imitates the real process in people's mind when understanding a Chinese text. To be specific, we first devise a serialized modeling structure in Literal Space to thoroughly capture the sequential information of Chinese text. Then, based on the authoritative information provided by Chinese language dictionaries, we design an association module and put forward a strategy called Radical-Word Association to use ideographic radicals as the medium to associate prior concept words in Associative Space. Afterwards, we design an attention module to imitate people's matching and decision between Literal Space and Associative Space, which can balance the importance of each associative words under specific contexts. Finally, extensive experiments on two real-world datasets prove the effectiveness and rationality of RAM, with good cognitive insights for future language modeling.
Development of a Low-Cost Wearable Data Glove for Capturing Finger Joint Angles

Capturing finger joint angle information has important applications in human–computer interaction and hand function evaluation. In this paper, a novel wearable data glove is proposed for capturing finger joint angles. A sensing unit based on a grating strip and an optical detector is specially designed for finger joint angle measurement. To measure the angles of finger joints, 14 sensing units are arranged on the back of the glove. There is a sensing unit on the back of each of the middle phalange, proximal phalange, and metacarpal of each finger, except for the thumb. For the thumb, two sensing units are distributed on the back of the proximal phalange and metacarpal, respectively. Sensing unit response tests and calibration experiments are conducted to evaluate the feasibility of using the designed sensing unit for finger joint measurement. Experimental results of calibration show that the comprehensive precision of measuring the joint angle of a wooden finger model is 1.67%. Grasping tests and static digital gesture recognition experiments are conducted to evaluate the performance of the designed glove. We achieve a recognition accuracy of 99% by using the designed glove and a generalized regression neural network (GRNN). These preliminary experimental results indicate that the designed data glove is effective in capturing finger joint angles.
An Enhanced Discrete Symbiotic Organism Search Algorithm for Optimal Task Scheduling in the Cloud

Recently, cloud computing has begun to experience tremendous growth because government agencies and private organisations are migrating to the cloud environment. Hence, having a task scheduling strategy that is efficient is paramount for effectively improving the prospects of cloud computing. Typically, a certain number of tasks are scheduled to use diverse resources (virtual machines) to minimise the makespan and achieve the optimum utilisation of the system by reducing the response time within the cloud environment. The task scheduling problem is NP-complete; as such, obtaining a precise solution is difficult, particularly for large-scale tasks. Therefore, in this paper, we propose a metaheuristic enhanced discrete symbiotic organism search (eDSOS) algorithm for optimal task scheduling in the cloud computing setting. Our proposed algorithm is an extension of the standard symbiotic organism search (SOS), a nature-inspired algorithm that has been implemented to solve various numerical optimisation problems. This algorithm imitates the symbiotic associations (mutualism, commensalism, and parasitism stages) displayed by organisms in an ecosystem. Despite the improvements made with the discrete symbiotic organism search (DSOS) algorithm, it still becomes trapped in local optima due to the large size of the values of the makespan and response time. The local search space of the DSOS is diversified by substituting the best value with any candidate in the population at the mutualism phase of the DSOS algorithm, which makes it worthy for use in task scheduling problems in the cloud. Thus, the eDSOS strategy converges faster when the search space is larger or more prominent due to diversification. The CloudSim simulator was used to conduct the experiment, and the simulation results show that the proposed eDSOS was able to produce a solution with a good quality when compared with that of the DSOS. Lastly, we analysed the proposed strategy by using a two-sample t-test, which revealed that the performance of eDSOS was of significance compared to the benchmark strategy (DSOS), particularly for large search spaces. The percentage improvements were 26.23% for the makespan and 63.34% for the response time.
Blockchain Based Spectrum Sharing over 6G Hybrid Cloud

This work investigates the dynamic spectrum sharing in the 6G enhanced ultra-reliable and low latency communications (eURLLC) services, as more and more Internet of Things (IoT) devices need to communicate between each other in the limited spectrum. We combine blockchain technology with 6G hybrid cloud to realize the spectrum sharing between IoT devices. Due to blockchain's distributed storage and credibility, blockchain based dynamic spectrum sharing can achieve flexible, feasible and much more efficient spectrum allocation. In this paper, we analyze the process of spectrum sharing, and a radio spectrum resource sharing structure in eURLLC based on reinforcement learning (RL) is proposed, including resource classification, resource scheduling, overall model, and neural network optimization. In this design, we integrate blockchain with hybrid cloud to build a public management platform to register and manage the information of IoT devices using the unified coding and identification standards.
Cross-modal Retrieval and Synthesis (X-MRS): Closing the Modality Gap in Shared Subspace Learning

Computational food analysis (CFA) naturally requires multi-modal evidence of a particular food, e.g., images, recipe text, etc. A key to making CFA possible is multi-modal shared representation learning, which aims to create a joint representation of the multiple views (text and image) of the data. In this work we propose a method for food domain cross-modal shared representation learning that preserves the vast semantic richness present in the food data. Our proposed method employs an effective transformer-based multilingual recipe encoder coupled with a traditional image embedding architecture. Here, we propose the use of imperfect multilingual translations to effectively regularize the model while at the same time adding functionality across multiple languages and alphabets. Experimental analysis on the public Recipe1M dataset shows that the representation learned via the proposed method significantly outperforms the current state-of-the-arts (SOTA) on retrieval tasks. Furthermore, the representational power of the learned representation is demonstrated through a generative food image synthesis model conditioned on recipe embeddings. Synthesized images can effectively reproduce the visual appearance of paired samples, indicating that the learned representation captures the joint semantics of both the textual recipe and its visual content, thus narrowing the modality gap.
Exploring content moderation in the decentralised web

Decentralising the Web is a desirable but challenging goal. One particular challenge is achieving decentralised content moderation in the face of various adversaries (e.g. trolls). To overcome this challenge, many Decentralised Web (DW) implementations rely on federation policies. Administrators use these policies to create rules that ban or modify content that matches specific rules. This, however, can have unintended consequences for many users. In this paper, we present the first study of federation policies on the DW, their in-the-wild usage, and their impact on users. We identify how these policies may negatively impact "innocent" users and outline possible solutions to avoid this problem in the future.
Towards a More Realistic and Detailed Deep-Learning-Based Radar Echo Extrapolation Method

Deep-learning-based radar echo extrapolation methods have achieved remarkable progress in the precipitation nowcasting field. However, they suffer from a common notorious problem—they tend to produce blurry predictions. Although some efforts have been made in recent years, the blurring problem is still under-addressed. In this work, we propose three effective strategies to assist deep-learning-based radar echo extrapolation methods to achieve more realistic and detailed prediction. Specifically, we propose a spatial generative adversarial network (GAN) and a spectrum GAN to improve image fidelity. The spatial and spectrum GANs aim at penalizing the distribution discrepancy between generated and real images from the spatial domain and spectral domain, respectively. In addition, a masked style loss is devised to further enhance the details by transferring the detailed texture of ground truth radar sequences to extrapolated ones. We apply a foreground mask to prevent the background noise from transferring to the outputs. Moreover, we also design a new metric termed the power spectral density score (PSDS) to quantify the perceptual quality from a frequency perspective. The PSDS metric can be applied as a complement to other visual evaluation metrics (e.g., LPIPS) to achieve a comprehensive measurement of image sharpness. We test our approaches with both ConvLSTM baseline and U-Net baseline, and comprehensive ablation experiments on the SEVIR dataset show that the proposed approaches are able to produce much more realistic radar images than baselines. Most notably, our methods can be readily applied to any deep-learning-based spatiotemporal forecasting models to acquire more detailed results.
Robot-Assisted Language Learning: Integrating Artificial Intelligence and Virtual Reality into English Tour Guide Practice

This action research created an application system using robots as a tool for training English-language tour guides. It combined artificial intelligence (AI) and virtual reality (VR) technologies to develop content for tours and a 3D VR environment using the AI Unity plug-in for programming. Students learned to orally interact with the robot and act as a guide to various destinations. The qualitative methods included observation, interviews, and self-reporting of learning outcomes. Two students voluntarily participated in the study. The intervention lasted for ten weeks. The results indicated the teaching effectiveness of robot-assisted language learning (RALL). The students acknowledged the value of RALL and had positive attitudes toward it. The contextualized VR learning environment increased their motivation and engagement in learning, and students perceived that RALL could help develop autonomy, enhance interaction, and provide an active learning experience. The implications of the study are that RALL has potential and that it provides an alternative learning opportunity for students.
ALONE: A Dataset for Toxic Behavior among Adolescents on Twitter

The convenience of social media has also enabled its misuse, potentially resulting in toxic behavior. Nearly 66% of internet users have observed online harassment, and 41% claim personal experience, with 18% facing severe forms of online harassment. This toxic communication has a significant impact on the well-being of young individuals, affecting mental health and, in some cases, resulting in suicide. These communications exhibit complex linguistic and contextual characteristics, making recognition of such narratives challenging. In this paper, we provide a multimodal dataset of toxic social media interactions between confirmed high school students, called ALONE (AdoLescents ON twittEr), along with descriptive explanation. Each instance of interaction includes tweets, images, emoji and related metadata. Our observations show that individual tweets do not provide sufficient evidence for toxic behavior, and meaningful use of context in interactions can enable highlighting or exonerating tweets with purported toxicity.
Unification of quantum resources in distributed scenarios

Quantum resources, such as coherence, discord, and entanglement, play a key role for demonstrating an advantage in many computation and communication tasks. In order to find the nature behind these resources, tremendous efforts have been made to explore the connections between them. In this work, we extend the single party coherence resource framework to the distributed scenario and relate it to basis-dependent discord. We show the operational meaning of basis-dependent discord in quantum key distribution and propose its resource measures. With any basis-dependent discord measure, we show how to extend it to measure coherence, discord, and entanglement by considering the relation of the resource free states. Our results lead to a unification of measures of quantum resources.
Improved graph cut model with features of superpixels and neighborhood patches for myocardium segmentation from ultrasound image

Ultrasound (US) imaging has the technical advantages for the functional evaluation of myocardium compared with other imaging modalities. However, it is a challenge of extracting the myocardial tissues from the background due to low quality of US imaging. To better extract the myocardial tissues, this study proposes a semi-supervised segmentation method of fast Superpixels and Neighborhood Patches based Continuous Min-Cut (<i>fSP-CMC</i>). The US image is represented by a graph, which is constructed depending on the features of superpixels and neighborhood patches. A novel similarity measure is defined to capture and enhance the features correlation using Pearson correlation coefficient and Pearson distance. Interactive labels provided by user play a subsidiary role in the semi-supervised segmentation. The continuous graph cut model is solved via a fast minimization algorithm based on augmented Lagrangian and operator splitting. Additionally, Non-Uniform Rational B-Spline (NURBS) curve fitting is used as post-processing to solve the low resolution problem caused by the graph-based method. 200 B-mode US images of left ventricle of the rats were collected in this study. The myocardial tissues were segmented using the proposed <i>fSP-CMC</i> method compared with the method of fast Neighborhood Patches based Continuous Min-Cut (<i>fP-CMC</i>). The results show that the <i>fSP-CMC</i> segmented the myocardial tissues with a higher agreement with the ground truth (<i>GT</i>) provided by medical experts. The mean absolute distance (MAD) and Hausdorff distance (HD) were significantly lower than those values of <i>fP-CMC</i> (<i>p</i> &lt; 0.05), while the Dice was significantly higher (<i>p</i> &lt; 0.05). In conclusion, the proposed <i>fSP-CMC</i> method accurately and effectively segments the myocardiumn in US images. This method has potentials to be a reliable segmentation method and useful for the functional evaluation of myocardium in the future study.