Reviewerly/exHarmony huggingface dataset:
Learning Data Representations with Joint Diffusion Models

Joint machine learning models that allow synthesizing and classifying data often offer uneven performance between those tasks or are unstable to train. In this work, we depart from a set of empirical observations that indicate the usefulness of internal representations built by contemporary deep diffusion-based generative models not only for generating but also predicting. We then propose to extend the vanilla diffusion model with a classifier that allows for stable joint end-to-end training with shared parameterization between those objectives. The resulting joint diffusion model outperforms recent state-of-the-art hybrid methods in terms of both classification and generation quality on all evaluated benchmarks. On top of our joint training approach, we present how we can directly benefit from shared generative and discriminative representations by introducing a method for visual counterfactual explanations.
Efficient Fine-Tuning Large Language Models for Knowledge-Aware Response Planning

Large Language Models (LLMs) have shown impressive emergent language capabilities, especially in applications with high ambiguity, such as language reasoning and knowledge consolidation. However, previous work explores the use of LLMs for acquiring information using either parametric or external knowledge, which might lead to serious issues such as hallucination. Toward solving these issues, we present a novel approach of knowledge-aware response planning (KARP) and propose a novel framework that employs (i) a knowledge retriever to obtain relevant information from web documents or databases for a given user query, and (ii) a robust fine-tuning strategy for LLMs to exploit the retrieved external knowledge for planning a final response. Experimental results show that our proposed framework can provide natural, concise answers for open-domain questions with high accuracy.
Triplets Oversampling for Class Imbalanced Federated Datasets

Class imbalance is a pervasive problem in machine learning, leading to poor performance in the minority class that is inadequately represented. Federated learning, which trains a shared model collaboratively among multiple clients with their data locally for privacy protection, is also susceptible to class imbalance. The distributed structure and privacy rules in federated learning introduce extra complexities to the challenge of isolated, small, and highly skewed datasets. While sampling and ensemble learning are state-of-the-art techniques for mitigating class imbalance from the data and algorithm perspectives, they face limitations in the context of federated learning. To address this challenge, we propose a novel oversampling algorithm called "Triplets" that generates synthetic samples for both minority and majority classes based on their shared classification boundary. The proposed algorithm captures new minority samples by leveraging three triplets around the boundary, where two come from the majority class and one from the minority class. This approach offers several advantages over existing oversampling techniques on federated datasets. We evaluate the effectiveness of our proposed algorithm through extensive experiments using various real-world datasets and different models in both centralized and federated learning environments. Our results demonstrate the effectiveness of our proposed algorithm, which outperforms existing oversampling techniques. In conclusion, our proposed algorithm offers a promising solution to the class imbalance problem in federated learning. The source code is released at github.com/Xiao-Chenguang/Triplets-Oversampling .
Make a Long Image Short: Adaptive Token Length for Vision Transformers

The vision transformer is a model that breaks down each image into a sequence of tokens with a fixed length and processes them similarly to words in natural language processing. Although increasing the number of tokens typically results in better performance, it also leads to a considerable increase in computational cost. Motivated by the saying “A picture is worth a thousand words,” we propose an innovative approach to accelerate the ViT model by shortening long images. Specifically, we introduce a method for adaptively assigning token length for each image at test time to accelerate inference speed. First, we train a Resizable-ViT (ReViT) model capable of processing input with diverse token lengths. Next, we extract token-length labels from ReViT that indicate the minimum number of tokens required to achieve accurate predictions. We then use these labels to train a lightweight Token-Length Assigner (TLA) that allocates the optimal token length for each image during inference. The TLA enables ReViT to process images with the minimum sufficient number of tokens, reducing token numbers in the ViT model and improving inference speed. Our approach is general and compatible with modern vision transformer architectures, significantly reducing computational costs. We verified the effectiveness of our methods on multiple representative ViT models on image classification and action recognition.
Python software libraries for computing with words (CWW) methodologies

Computing with Words (CWW) is a remarkable approach that enables human beings to interact with computing machines using linguistic information. Recently, there have been various contributions in the realm of CWW methodologies (both theoretical and application oriented). However, very less work exists on the provision of software libraries of these methodologies to the interested users/ researchers, and where available are mostly on proprietary platforms like MATLAB, to the best of our knowledge. As, python has recently emerged as a versatile programming language with its application in various domains, hence, we have also developed the python based implementations for numerous CWW methodologies. Through this work, we present their details for the interested users, draw a comparisons and improvements as well as provide links to access the python-based software of the popular CWW methodologies, from our github repository: https://github.com/pguptak/CWW.
marl-jax: Multi-agent Reinforcement Leaning Framework for Social Generalization

Recent advances in Reinforcement Learning (RL) have led to many exciting applications. These advancements have been driven by improvements in both algorithms and engineering, which have resulted in faster training of RL agents. We present marl-jax, a multi-agent reinforcement learning software package for training and evaluating social generalization of the agents. The package is designed for training a population of agents in multi-agent environments and evaluating their ability to generalize to diverse background agents. It is built on top of DeepMind’s JAX ecosystem [2] and leverages the RL ecosystem developed by DeepMind. Our framework marl-jax is capable of working in cooperative and competitive, simultaneous-acting environments with multiple agents. The package offers an intuitive and user-friendly command-line interface for training a population and evaluating its generalization capabilities. In conclusion, marl-jax provides a valuable resource for researchers interested in exploring social generalization in the context of MARL. The open-source code for marl-jax is available at: https://github.com/kinalmehta/marl-jax .
Using Multiple RDF Knowledge Graphs for Enriching ChatGPT Responses

There is a recent trend for using the novel Artificial Intelligence ChatGPT chatbox, which provides detailed responses and articulate answers across many domains of knowledge. However, in many cases it returns plausible-sounding but incorrect or inaccurate responses, whereas it does not provide evidence. Therefore, any user has to further search for checking the accuracy of the answer or/and for finding more information about the entities of the response. At the same time there is a high proliferation of RDF Knowledge Graphs (KGs) over any real domain, that offer high quality structured data. For enabling the combination of ChatGPT and RDF KGs, we present a research prototype, called $$\texttt{GPT}{\bullet }\texttt{LODS} $$ , which is able to enrich any ChatGPT response with more information from hundreds of RDF KGs. In particular, it identifies and annotates each entity of the response with statistics and hyperlinks to LODsyndesis KG (which contains integrated data from 400 RDF KGs and over 412 million entities). In this way, it is feasible to enrich the content of entities and to perform fact checking and validation for the facts of the response at real time. URL: https://demos.isl.ics.forth.gr/GPToLODS/Annot_Enrichment Demo Video: https://youtu.be/H30bSv9NfUw
Multivariate Time-Series Anomaly Detection with Temporal Self-supervision and Graphs: Application to Vehicle Failure Prediction

Failure prediction is key to ensuring the reliable operation of vehicles, especially for organizations that depend on a fleet of vehicles. However, traditional approaches often rely on rule-based or heuristic methods that may not be effective in detecting subtle anomalies, rare events, or in more modern vehicles containing a complex sensory network. This paper presents a novel approach to vehicle failure prediction, called mVSG-VFP, which employs self-supervised learning and graph-based techniques. The proposed method realizes the failure prediction task by exploring information hidden in the time-series data recorded through the sensors embedded in the vehicle. mVSG-VFP includes two main components: a graph-based autoencoder that learns representations of normal data while considering the relationship between different sensors and a self-supervised component that maps temporally-adjacent data to similar representations. We propose a novel approach to define the notion of adjacency in vehicle temporal data. To evaluate mVSG-VFP, we apply it to a dataset comprised of vehicle sensor recordings to identify the abnormal data samples that signal a potential future failure. We performed a flurry of experiments to verify the accuracy of our model and demonstrate it outperforms state-of-the-art models in this task. Overall, the method is robust and intuitive, making it a useful tool for real-world applications.
Object Anchoring for Autonomous Robots Using the Spatio-Temporal-Semantic Environment Representation SEEREP

For single-plant specific weed regulation, robotic systems and agricultural machinery in general have to collect a large amount of temporal and spatial high-resolution sensor data. SEEREP, the Spatio-Temporal-Semantic Environment Representation, can be used to structure and manage such data more efficiently. SEEREP deals with the spatial, temporal and semantic modalities of data simultaneously and provides an efficient query interface for all three modalities that can be combined for high-level analyses. It supports popular robotic sensor data such as images and point clouds, as well as sensor and robot coordinate frames changing over time. This query interface enables high-level reasoning systems as well as other data analysis methods to handle partially unstructured environments that change over time, as for example agricultural environments. But the current methodology of SEEREP cannot store the result of the analysis methods regarding specific objects instances in the world. Especially the results of the anchoring problem which searches for a connection between symbolic and sub-symbolic data cannot be represented nor queried. Thus, we propose a further development of the SEEREP methodology in this paper: For a given object, we link the existing semantic labels in different datasets to a unique common instance, thereby enabling queries for datasets showing this object instance and with this enabling the efficient provision of datasets for object-centric analysis algorithms. Additionally, the results of those algorithms can be stored linked to the instance either by adding facts in a triple-store like manner or by adding further data linked to the instance, like a point, representing the position of the instance. We show the benefits of our anchoring approach in an agricultural setting with the use-case of single-plant specific weed regulation.
Planning Landmark Based Goal Recognition Revisited: Does Using Initial State Landmarks Make Sense?

Goal recognition is an important problem in many application domains (e.g., pervasive computing, intrusion detection, computer games, etc.). In many application scenarios, it is important that goal recognition algorithms can recognize goals of an observed agent as fast as possible. However, many early approaches in the area of Plan Recognition As Planning, require quite large amounts of computation time to calculate a solution. Mainly to address this issue, recently, Pereira et al. developed an approach that is based on planning landmarks and is much more computationally efficient than previous approaches. However, the approach, as proposed by Pereira et al., considers trivial landmarks (i.e., facts that are part of the initial state and goal description are landmarks by definition) for goal recognition. In this paper, we show that it does not provide any benefit to use landmarks that are part of the initial state in a planning landmark based goal recognition approach. The empirical results show that omitting initial state landmarks for goal recognition improves goal recognition performance.
Click-Aware Structure Transfer with Sample Weight Assignment for Post-Click Conversion Rate Estimation

Post-click Conversion Rate (CVR) prediction task plays an essential role in industrial applications, such as recommendation and advertising. Conventional CVR methods typically suffer from the data sparsity problem as they rely only on samples where the user has clicked. To address this problem, researchers have introduced the method of multi-task learning, which utilizes non-clicked samples and shares feature representations of the Click-Through Rate (CTR) task with the CVR task. However, it should be noted that the CVR and CTR tasks are fundamentally different and may even be contradictory. Therefore, introducing a large amount of CTR information without distinction may drown out valuable information related to CVR. This phenomenon is called the curse of knowledge problem in this paper. To tackle this issue, we argue that a trade-off should be achieved between the introduction of large amounts of auxiliary information and the protection of valuable information related to CVR. Hence, we propose a Click-aware Structure Transfer model with sample Weight Assignment, abbreviated as CSTWA. It pays more attention to the latent structure information, which could refine the input information related to CVR, instead of directly sharing feature representations. Meanwhile, to capture the representation conflict between CTR and CVR, we calibrate the representation layer and reweight the discriminant layer to excavate the click bias information from the CTR tower. Moreover, it incorporates a sample weight assignment algorithm biased towards CVR modeling, to make the knowledge from CTR would not mislead the CVR. Extensive experiments on industrial and public datasets have demonstrated that CSTWA significantly outperforms widely used and competitive models.
Corpus-Based Relation Extraction by Identifying and Refining Relation Patterns

Automated relation extraction without extensive human-annotated data is a crucial yet challenging task in text mining. Existing studies typically use lexical patterns to label a small set of high-precision relation triples and then employ distributional methods to enhance detection recall. This precision-first approach works well for common relation types but struggles with unconventional and infrequent ones. In this work, we propose a recall-first approach that first leverages high-recall patterns (e.g., a per:siblings relation normally requires both the head and tail entities in the person type) to provide initial candidate relation triples with weak labels and then clusters these candidate relation triples in a latent spherical space to extract high-quality weak supervisions. Specifically, we present a novel framework, RClus, where each relation triple is represented by its head/tail entity type and the shortest dependency path between the entity mentions. RClus first applies high-recall patterns to narrow down each relation type's candidate space. Then, it embeds candidate relation triples in a latent space and conducts spherical clustering to further filter out noisy candidates and identify high-quality weakly-labeled triples. Finally, RClus leverages the above-obtained triples to prompt-tune a pre-trained language model and utilizes it for improved extraction coverage. We conduct extensive experiments on three public datasets and demonstrate that RClus outperforms the weakly-supervised baselines by a large margin and achieves generally better performance than fully-supervised methods in low-resource settings.
Learning Disentangled Discrete Representations

Recent successes in image generation, model-based reinforcement learning, and text-to-image generation have demonstrated the empirical advantages of discrete latent representations, although the reasons behind their benefits remain unclear. We explore the relationship between discrete latent spaces and disentangled representations by replacing the standard Gaussian variational autoencoder (VAE) with a tailored categorical variational autoencoder. We show that the underlying grid structure of categorical distributions mitigates the problem of rotational invariance associated with multivariate Gaussian distributions, acting as an efficient inductive prior for disentangled representations. We provide both analytical and empirical findings that demonstrate the advantages of discrete VAEs for learning disentangled representations. Furthermore, we introduce the first unsupervised model selection strategy that favors disentangled representations.
Deep Explainable Relational Reinforcement Learning: A Neuro-Symbolic Approach

Despite its successes, Deep Reinforcement Learning (DRL) yields non-interpretable policies. Moreover, since DRL does not exploit symbolic relational representations, it has difficulties in coping with structural changes in its environment (such as increasing the number of objects). Meanwhile, Relational Reinforcement Learning inherits the relational representations from symbolic planning to learn reusable policies. However, it has so far been unable to scale up and exploit the power of deep neural networks. We propose Deep Explainable Relational Reinforcement Learning (DERRL), a framework that exploits the best of both – neural and symbolic worlds. By resorting to a neuro-symbolic approach, DERRL combines relational representations and constraints from symbolic planning with deep learning to extract interpretable policies. These policies are in the form of logical rules that explain why each decision (or action) is arrived at. Through several experiments, in setups like the Countdown Game, Blocks World, Gridworld, Traffic, and Mingrid, we show that the policies learned by DERRL are adaptable to varying configurations and environmental changes.
Hypernetworks Build Implicit Neural Representations of Sounds

Implicit Neural Representations (INRs) are nowadays used to represent multimedia signals across various real-life applications, including image super-resolution, image compression, or 3D rendering. Existing methods that leverage INRs are predominantly focused on visual data, as their application to other modalities, such as audio, is nontrivial due to the inductive biases present in architectural attributes of image-based INR models. To address this limitation, we introduce HyperSound, the first meta-learning approach to produce INRs for audio samples that leverages hypernetworks to generalize beyond samples observed in training. Our approach reconstructs audio samples with quality comparable to other state-of-the-art models and provides a viable alternative to contemporary sound representations used in deep neural networks for audio processing, such as spectrograms. Our code is publicly available at https://github.com/WUT-AI/hypersound .
Detecting Evasion Attacks in Deployed Tree Ensembles

Tree ensembles are powerful models that are widely used. However, they are susceptible to evasion attacks where an adversary purposely constructs an adversarial example in order to elicit a misprediction from the model. This can degrade performance and erode a user's trust in the model. Typically, approaches try to alleviate this problem by verifying how robust a learned ensemble is or robustifying the learning process. We take an alternative approach and attempt to detect adversarial examples in a post-deployment setting. We present a novel method for this task that works by analyzing an unseen example's output configuration, which is the set of leaves activated by the example in the ensemble's constituent trees. Our approach works with any additive tree ensemble and does not require training a separate model. We evaluate our approach on three different tree ensemble learners. We empirically show that our method is currently the best adversarial detection method for tree ensembles.
Neural layered min-sum decoders for cyclic codes

This paper proposes a low-complexity neural network decoder based on the layered min-sum algorithm to decode cyclic codes. By generalizing the layered min-sum algorithm to its neural network counterpart, the number of network weights decreases while retaining a good error correction performance. The Bose–Chaudhuri–Hocquenghem (BCH) codes, quadratic residue (QR) codes, and punctured Reed–Muller (RM) codes are selected as three exemplary binary cyclic codes. Simulation results show that the proposed neural decoder achieves superior performance with less computational complexity compared with the state-of-the-art neural network decoder. Further, a neural decoder incorporating the modified random redundant decoding (mRRD) algorithm is investigated to approach the performance of maximum-likelihood decoding for some short codes.
MIFINN: A novel multi-information fusion and interaction neural network for aspect-based sentiment analysis

Aspect-based Sentiment Analysis (ABSA) is used to detect corresponding sentiment polarities toward different aspect terms. In recent years, graph convolutional network-based methods achieve great success for ABSA. However, previous studies ignore syntactic dependency types and aspect-related syntactic distances. Besides, effectively fusing semantic and syntactic information remains a challenging problem. Previous studies also neglect interactions between aspect terms and context. To alleviate the above problems, we develop a Multi-Information Fusion and Interaction Neural Network (MIFINN), which consists of an Aspect and Syntax-Aware Multi-Information Fusion Graph Convolutional Network (ASAMIFGCN) and a Multi-Information Interaction and Gating Network (MIIGN). The ASAMIFGCN model can integrate and fuse contextual semantics, syntactic dependency types, and aspect-related syntactic distances. Specifically, two different attentions are designed to learn global and aspect-related semantic and syntactic information, respectively. A syntactic distance gating module is proposed to acquire more precise score matrices. Besides, we propose a semantic and syntactic association module to fuse multi-information simultaneously. The MIIGN model can effectively interact aspect terms and context. Specifically, two interactive attentions are developed to learn different weights toward aspect terms and context via interactions between each other, respectively. Moreover, we propose a gating information module for controlling the information flow of aspect terms and context. We conduct experiments on multiple datasets and MIFINN achieves state-of-the-art performance. The results also demonstrate that effectively fusing and interacting multi-information can improve model performance for ABSA significantly.