Reviewerly/exHarmony huggingface dataset:
Multi-hop Relational Graph Attention Network for Text-to-SQL Parsing

Text-to-SQL aims to parse natural language problems into SQL queries, which can provide a simple interface to access large databases enabling SQL novices a quicker entry into databases. As the Text-to-SQL field is intensively studied, more and more models use GNNs to encode heterogeneous graph information in this task, and how to better obtain path information between nodes in database schema heterogeneous graphs and question-database schema heterogeneous graphs will greatly affect the effectiveness of the model parsing. Our work intends to explore the problem of solving the encoding of heterogeneous graph meta-paths in the Text-to-SQL task. Previous approaches usually use multi-layer GNNs to aggregate topological structure information between nodes. However, they ignored the structural information embedded at the edges and also failed to obtain nodes that are not directly connected but can provide contextual information through meta-paths. To solve the above problem, we propose Multi-Hop Relational Graph Attention Network based Text-to-SQL Parsing Model (MHRGATSQL) for learning topological information between nodes while obtaining semantic information embedded in the edge topology. We use multi-hop attention to modify the relational graph attention network to diffuse the attention scores throughout the network, thus increasing the “receptive field” of each layer of RGAT. Experimental results on the large-scale cross-domain Text-to-SQL dataset Spider show that our model obtains an absolute improvement of 1.7% compared to the baseline and alleviates the over-smoothing problem in the deep network model.
Hard Negative Sampling via Regularized Optimal Transport for Contrastive Representation Learning

We study the problem of designing hard negative sampling distributions for unsupervised contrastive representation learning. We propose and analyze a novel min-max framework that seeks a representation which minimizes the maximum (worst-case) generalized contrastive learning loss over all couplings (joint distributions between positive and negative samples subject to marginal constraints) and prove that the resulting min-max optimum representation will be degenerate. This provides the first theoretical justification for incorporating additional regularization constraints on the couplings. We re-interpret the min-max problem through the lens of Optimal Transport (OT) theory and utilize regularized transport couplings to control the degree of hardness of negative examples. Through experiments we demonstrate that the negative samples generated from our designed negative distribution are more similar to the anchor than those generated from the baseline negative distribution. We also demonstrate that entropic regularization yields negative sampling distributions with parametric form similar to that in a recent state-of-the-art negative sampling design and has similar performance in multiple datasets. Utilizing the uncovered connection with OT, we propose a new ground cost for designing the negative distribution and show improved performance of the learned representation on downstream tasks compared to the representation learned when using squared Euclidean cost. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Our code is publicly available at https://github.com/rjiang03/HCL-OT.
Disease Diagnosis with Cost-Sensitive Grouped Features Based on Deep Reinforcement Learning

Traditional machine learning algorithms rely on the assumption that all features of a given dataset are available for free. However, in disease diagnosis, acquiring features requires a certain cost because there are many concerns, such as monetary data collection costs, patient discomfort in medical procedures, and privacy impacts of data collection that require careful consideration. Moreover, some medical examination items are grouped in the real world, such as lab tests that return multiple measurements, resulting in naturally grouped features. In this work, to consider features that come in groups, we propose a novel feature selection method for disease diagnosis with cost-sensitive grouped features based on deep reinforcement learning (RL). We optimize the objective which is a trade-off between the cost of grouped features and the diagnostic accuracy, by training deep RL models. In addition, the approach is flexible, as it can be improved with new deep reinforcement learning algorithms. We evaluate the effectiveness of the proposed method on four publicly available datasets. Compared with the existing competitive feature selection methods, the proposed method can achieve higher learning accuracy with lower feature costs. This paper takes disease diagnosis as an example, and the proposed method is universal for problems with grouped features of different domains. (The source code is available at https://anonymous.4open.science/r/DRL-CSGF- Y821).
Video-based Red Panda Individual Identification by Adaptively Aggregating Discriminative Features

Existing animal individual identification methods mostly use only single images and cannot effectively leverage complementary features in video frames. To further improve the robustness and accuracy of individual identification of animals like red pandas that have complex body deformation or pose variations, we propose in this paper a deep network to learn hybrid feature representation of red pandas that adaptively aggregates local and global features for red panda identification. The local feature representation is obtained by adaptively finding discriminative local patches of the red panda in each frame and aggregating the local features across frames via a hypergraph neural network. The global feature representation is obtained by aggregating the features of different frames via average pooling. Red panda individuals are finally identified based on the concatenated local and global feature representations. Evaluation experiments have been done on a self-collected dataset of red panda videos. The results prove the utility of video data in animal individual identification as well as the superiority of our proposed method in exploiting discriminative features in video frames for identifying individual animals.
Adversarial Attacks &amp; Detection on a Deep Learning-Based Digital Pathology Model

Medical imaging modalities, like magnetic resonance imaging (MRI), have enabled efficient diagnosis of various conditions, including cancer, lung disease, and brain tumors. With the advancements in machine learning, AI-based medical image segmentation and classification systems have emerged, potentially replacing human diagnosis. However, the security and robustness of these systems are crucial, as they are vulnerable to adversarial attacks, as demonstrated in previous studies. In this respect, the current work explores the one-pixel attack’s impact on the reliable VGG16 model, the effectiveness of combining the one-pixel attack with the FGSM attack, the potential of using the squeezing color bits detector to counter the one-pixel attack, and the possibility of using a combination of the squeezing color bits and PCA whitening detectors to protect against the aforementioned attacks.
A Multi-Agent Framework for Recommendation with Heterogeneous Sources

With the ever prospering of the web technologies, there is a common need to make recommendations from heterogeneous sources, such as recommending products and advertisements together on the e-commerce websites. People usually solve such recommendation problem by a two-stage paradigm, where the first stage is generating candidates from each source, and the second one is aggregating and ranking the generated heterogeneous candidates to produce the final results. While existing models have achieved many successes, they mostly optimize the above two stages separately, where the user preferences can only be used to supervise the second stage, while for the first one, there is no signal to tell whether the generated candidates are accurate enough to cover the user preference. To solve the above problem, in this paper, we design a multi-agent framework to jointly optimize the above two stages. In specific, suppose there are N sources in our problem, then we deploy N+1 agents, where the first N agents correspond one-to-one with the sources, aiming to select the sources-specific candidates, and the last agent is designed to aggregate the candidates from different sources for the final recommendation. All the agents play a cooperative game, aiming to maximize the rewards revealing user preferences. We implement our idea based on the Deep Q-network, where we design a decomposable reward to enhance the training efficiency. We adapt our model to a real-world recommendation problem abstracted from a famous short video platform-Kuaishou.com. We conduct extensive experiments to demonstrate the effectiveness of our model.
Hybrid Data Augmentation for Citation Function Classification

The citation function generally signifies the purpose or reason underlying a citation within a scholarly paper or a research article. Automatic citation function classification is, therefore, a task in computational linguistics and information science that can facilitate further applications in reference research, citation recommendation, and evaluation of research activities. By taking into account the state of the art, we identify two major constraints pertinent to the data of the citation function classification task, i.e., data imbalance and data sparsity. On the one hand, the natural distribution of different types of citations in one scientific literature is uneven leading to data imbalance in the real scenario. On the other hand, the citation function data is generally labeled by an expert which takes huge human effort resulting in a limited data scale. To this end, in this paper, we propose HybridDA, a two-stage model based on GPT-2 data argumentation and data retrieval to synthesize more high-quality annotated citation function data in a bid to solve both data imbalance and data sparsity problems. We conduct experiments on imbalance setting and low resource setting with our proposed approach. The experimental results on both of these settings demonstrate that our proposed model can achieve competitive performance in contrast to the other baseline models.
A Health Profiling Framework for Children Leveraging Multimodal Learning Based on Ambient Sensor Signals

Traditional methods for health profiling are usually expensive and require specialized expertise. The growing prevalence and development of wearable devices have made it feasible to collect ambient sensor signals, providing us with new opportunities to profile children’s health in a cost-effective and comprehensive manner. Inspired by recent works in multimodal learning, we propose a health profiling framework for children. First, we extract context and motion patterns from their personal and family characteristics and acceleration signals. Then, context and motion embeddings are generated by two encoders and input into a lightweight neural network to profile children’s health from the perspectives of physical activity intensity, physical functioning, health confidence, psychosocial functioning, resilience, and connectedness. We evaluate the proposed method on real-world datasets, and the results show its outstanding performance. Specifically, the context pattern is effective in profiling children’s health, while the motion pattern is significantly effective in assessing children’s physical activity intensity.
Efficient Utilization of Large Pre-Trained Models for Low Resource ASR

Unsupervised representation learning has recently helped automatic speech recognition (ASR) to tackle tasks with limited labeled data. Following this, hardware limitations and applications give rise to the question how to take advantage of large pre-trained models efficiently and reduce their complexity. In this work, we study a challenging low resource conversational telephony speech corpus from the medical domain in Vietnamese and German. We show the benefits of using unsupervised techniques beyond simple fine-tuning of large pre-trained models, discuss how to adapt them to a practical telephony task including bandwidth transfer and investigate different data conditions for pre-training and fine-tuning. We outperform the project baselines by 22% relative using pre-training techniques. Further gains of 29% can be achieved by refinements of architecture and training and 6% by adding 0.8 h of in-domain adaptation data.
CluCDD: Contrastive Dialogue Disentanglement Via Clustering

A huge number of multi-participant dialogues happen online every day, which leads to difficulty in understanding the nature of dialogue dynamics for both humans and machines. Dialogue disentanglement aims at separating an entangled dialogue into detached sessions, thus increasing the readability of long disordered dialogue. Previous studies mainly focus on message-pair classification and clustering in two-step methods, which cannot guarantee the whole clustering performance in a dialogue. To address this challenge, we propose a simple yet effective model named CluCDD, which aggregates utterances by contrastive learning. More specifically, our model pulls utterances in the same session together and pushes away utterances in different ones. Then a clustering method is adopted to generate predicted clustering labels. Comprehensive experiments conducted on the Movie Dialogue dataset and IRC dataset demonstrate that our model achieves a new state-of-the-art result <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> . <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Code is available at https://github.com/gaojingsheng/CluCDD
Gloss Alignment using Word Embeddings

Capturing and annotating Sign language datasets is a time consuming and costly process. Current datasets are orders of magnitude too small to successfully train unconstrained Sign Language Translation (SLT) models. As a result, research has turned to TV broadcast content as a source of large-scale training data, consisting of both the sign language interpreter and the associated audio subtitle. However, lack of sign language annotation limits the usability of this data and has led to the development of automatic annotation techniques such as sign spotting. These spottings are aligned to the video rather than the subtitle, which often results in a misalignment between the subtitle and spotted signs. In this paper we propose a method for aligning spottings with their corresponding subtitles using large spoken language models. Using a single modality means our method is computationally inexpensive and can be utilized in conjunction with existing alignment techniques. We quantitatively demonstrate the effectiveness of our method on the Meine DGS-Annotated (MeineDGS) and BBC-Oxford British Sign Language (BOBSL) datasets, recovering up to a 33.22 BLEU-1 score in word alignment.
Generating Artistic Images Via Few-Shot Style Transfer

Generating images from a predefined style with heterogeneous and limited data is a challenging task for generative models. This work focuses on the conditional generation of artistic images, aiming to learn from a small set of paintings with high variability how to convert real-world photos into impressionistic paintings with the same given style. We design a few-shot style transfer model using a mixture of diverse one-shot style transfer generative models based on the SinGAN model. The proposed few-shot model coineEnSinGAN utilizes an ensemble of different SinGAN realizations to style transfer realistic photos to their closest painting style, by incorporating a novel aggregation mechanism based on the minimum cosine distance in the latent space of the feature vectors. EnSinGAN generates convincing impressionistic landscape images, and was awarded the first place in the Kaggle competition “I’m something of a painter myself” by being the closest in distribution to the test images.
Automatic Alignment Between Sign Language Videos And Motion Capture Data: A Motion Energy-Based Approach

In this paper, we propose a method for the automatic alignment of sign language videos and their corresponding motion capture data, useful for the preparation of multi-modal sign language corpora. First, we extract an estimate of the motion energy from both the video and the motion capture data. Second, we align the two curves to minimize their distance. Our tests show that it is possible to achieve a mean absolute error as low as 1.11 frames using optical flow for video energy extraction and a set of 22 bones for skeletal energy extraction.
Incremental Image Labeling Via Iterative Refinement

Data quality is critical for multimedia tasks, while various types of systematic flaws are found in image benchmark datasets, as discussed in recent work. In particular, the existence of the semantic gap problem leads to a many-to-many mapping between the information extracted from an image and its linguistic description. This unavoidable bias further leads to poor performance on current computer vision tasks. To address this issue, we introduce a Knowledge Representation (KR)-based methodology to provide guidelines driving the labeling process, thereby indirectly introducing intended semantics in ML models. Specifically, an iterative refinement-based annotation method is proposed to optimize data labeling by organizing objects in a classification hierarchy according to their visual properties, ensuring that they are aligned with their linguistic descriptions. Preliminary results verify the effectiveness of the proposed method.
A Study on the Impact of Virtual Reality on User Attention

In this contribution, an analysis of the effect of the adoption of virtual reality on human attention is presented. Specifically, we analyzed whether the use of virtual reality leads to a higher level of attention with respect to the use of the same application on a 2D display system. For this purpose, a subjective experiment based on the Stroop test has been implemented. The analysis performed on the collected data suggests that the immersive virtual reality experience helps the user to focus, thus increasing user performance and reducing the impact of external distractions. The heart rate of the observers was also measured during the experiment. Tests show that the heart rate tends to be higher and more regular when using a virtual reality headset than when using a 2D screen. This may indicate greater user engagement.
Texture Quality Criteria Comparison

Visual scene recognition or modeling predominantly uses visual textures representing an object’s material properties. However, the single material texture varies in scale and illumination angles due to mapping an object’s shape. We present a comparative study of thirteen possible texture quality criteria and show the superior performance of two multispectral measures derived from the Markovian descriptive model.
SL-REDU GSL: A Large Greek Sign Language Recognition Corpus

We present a large multi-signer video corpus for the Greek Sign Language (GSL), suitable for the development and evaluation of GSL recognition algorithms. The database has been collected as part of the “SL-ReDu” project that focuses on the education use-case of systematic teaching of GSL as a second language (L2). The project aims to assist this process by allowing self-monitoring and objective assessment of GSL learners’ productions through the use of recognition technology, thus requiring suitable data resources relevant to the aforementioned use-case. To this end, we present the SL-ReDu GSL corpus, an extensive RGB+D video collection of 21 informants with a duration of 36 hours, recorded under studio conditions, consisting of: (i) isolated signs; (ii) continuous signing (annotated at the sentence level); and (iii) fingerspelling of words. We provide a detailed description of the design and acquisition methods used to develop it, along with corpus statistics and a comparison to existing sign language datasets. The SL-ReDu GSL corpus, as well as proposed frameworks for recognition experiments on it, are publicly available at https://www.sl-redu.e-ce.uth.gr/corpus.
Towards Incorporating 3D Space-Awareness Into an Augmented Reality Sign Language Interpreter

This paper describes the concept and the software architecture of a fully integrated system supporting a dialog between a deaf person and a hearing person through a virtual sign language interpreter (aka avatar) projected in the real space by an Augmented Reality device. In addition, a Visual Simultaneous Localization and Mapping system provides information about the 3D location of the objects recognized in the surrounding environment, allowing the avatar to orient, look and point towards the real location of discourse entities during the translation. The goal being to provide a modular architecture to test single software components in a fully integrated framework and move virtual sign language interpreters beyond the standard “front-facing” interaction paradigm.
Phone and Speaker Spatial Organization in Self-Supervised Speech Representations

Self-supervised representations of speech are currently being widely used for a large number of applications. Recently, some efforts have been made in trying to analyze the type of information present in each of these representations. Most such work uses downstream models to test whether the representations can be successfully used for a specific task. The downstream models, though, typically perform nonlinear operations on the representation extracting information that may not have been readily available in the original representation. In this work, we analyze the spatial organization of phone and speaker information in several state-of-the-art speech representations using methods that do not require a downstream model. We measure how different layers encode basic acoustic parameters such as formants and pitch using representation similarity analysis. Further, we study the extent to which each representation clusters the speech samples by phone or speaker classes using non-parametric statistical testing. Our results indicate that models represent these speech attributes differently depending on the target task used during pretraining.
Distractor-Based Evaluation of Sign Spotting

Sign spotting is a subtask of sign language processing in which we determine when a given target sign occurs in a given sign sequence. This paper proposes a method for evaluating sign spotting systems, which we argue to be more reflective of the degree to which a system would satisfy the user’s requirements in practice than previously proposed evaluation methods. To deal with an incomplete ground truth, we introduce the concept of distractors: signs which are similar to the target sign according to a given distance measure. We assume that the performance of a sign spotting model when distinguishing a given target sign from the associated distractors will reflect the performance of the model on the complete ground truth. We develop a sign spotting model to demonstrate our evaluation method.