bluuebunny/crossref_metadata_2025 huggingface dataset:
TSFool: Crafting Highly-Imperceptible Adversarial Time Series Through Multi-Objective Attack
Authors: Wang Yanyun , Du Dehui , Hu Haibo , Liang Zi , Liu Yuanhao
Published: October 2024

Recent years have witnessed the success of recurrent neural network (RNN) models in time series classification (TSC). However, neural networks (NNs) are vulnerable to adversarial samples, which cause real-life adversarial attacks that undermine the robustness of AI models. To date, most existing attacks target at feed-forward NNs and image recognition tasks, but they cannot perform well on RNN-based TSC. This is due to the cyclical computation of RNN, which prevents direct model differentiation. In addition, the high visual sensitivity of time series to perturbations also poses challenges to local objective optimization of adversarial samples. In this paper, we propose an efficient method called TSFool to craft highly-imperceptible adversarial time series for RNN-based TSC. The core idea is a new global optimization objective known as “Camouflage Coefficient” that captures the imperceptibility of adversarial samples from the class distribution. Based on this, we reduce the adversarial attack problem to a multi-objective optimization problem that enhances the perturbation quality. Furthermore, to speed up the optimization process, we propose to use a representation model for RNN to capture deeply embedded vulnerable samples whose features deviate from the latent manifold. Experiments on 11 UCR and UEA datasets showcase that TSFool significantly outperforms six white-box and three black-box benchmark attacks in terms of effectiveness, efficiency and imperceptibility from various perspectives including standard measure, human study and real-world defense.
Efficient Model-Stealing Attacks Against Inductive Graph Neural Networks
Authors: Podhajski Marcin , Dubiński Jan , Boenisch Franziska , Dziedzic Adam , Pregowska Agnieszka , Michalak Tomasz P.
Published: October 2024

Graph Neural Networks (GNNs) are recognized as potent tools for processing real-world data organized in graph structures. Especially inductive GNNs, which allow for the processing of graph-structured data without relying on predefined graph structures, are becoming increasingly important in a wide range of applications. As such these networks become attractive targets for model-stealing attacks where an adversary seeks to replicate the functionality of the targeted network. Significant efforts have been devoted to developing model-stealing attacks that extract models trained on images and texts. However, little attention has been given to stealing GNNs trained on graph data. This paper identifies a new method of performing unsupervised model-stealing attacks against inductive GNNs, utilizing graph contrastive learning and spectral graph augmentations to efficiently extract information from the targeted model. The new type of attack is thoroughly evaluated on six datasets and the results show that our approach outperforms the current state-of-the-art by Shen et al. (2021). In particular, our attack surpasses the baseline across all benchmarks, attaining superior fidelity and downstream accuracy of the stolen model while necessitating fewer queries directed toward the target model.
Extending Context Window of Attention Based Knowledge Tracing Models via Length Extrapolation
Authors: Li Xueyi , Bai Youheng , Guo Teng , Zheng Ying , Hou Mingliang , Zhan Bojun , Huang Yaying , Liu Zitao , Gao Boyu , Luo Weiqi
Published: October 2024

Knowledge tracing (KT) is a prediction task that aims to predict students’ future performance based on their past learning data. The rapid progress in attention mechanisms has led to the emergence of various high-performing attention based KT models. However, in online or personalized education settings, students’ varying learning paths result in different lengths of student interaction sequences, which poses a significant challenge for attention based KT models as their context window sizes are fixed during both training and prediction stages. We refer to this as the length extrapolation of KT model. In this paper, we propose extraKT to facilitate better extrapolation that learn from student interactions with a short context window and continue to perform well across various longer context window sizes at prediction stage. Specifically, we negatively bias attention scores with linearly decreasing penalties that are proportional to query-key distance, which efficiently represents short-term forgetting characteristics of student knowledge states. We conduct comprehensive and rigorous experiments on three real-world educational datasets. The results show that our extraKT model exhibits robust length extrapolation capability and outperforms state-of-the-art baseline models in terms of AUC and accuracy. To encourage reproducible research, we merge our data and code to the publicly available pyKT benchmark at https://github.com/pykt-team/pykt-toolkit.
HAIformer: Human-AI Collaboration Framework for Disease Diagnosis via Doctor-Enhanced Transformer
Authors: Zhao Xuehan , Liu Jiaqi , Zhang Yao , Yu Zhiwen , Guo Bin
Published: October 2024

Online disease diagnosis, gathering the patients’ symptoms and making diagnoses through online dialogue, grows rapidly worldwide. Manual-based approach, e.g., Haodaifu, employs real-world doctors, providing high-quality but high-cost medical services. In contrast, machine-based approach, e.g., 01bot, that utilizes machine learning models can make automatic diagnosis but lacks reliable accuracy. While some work has enabled human-AI collaboration in disease diagnosis, their collaboration pattern is simple and needs to be further improved. Therefore, we aim to introduce a doctor-enhanced and low-cost human-AI collaboration pattern. There are two key challenges. 1) How to utilize expert knowledge in doctor feedback to enhance AI’s capability? 2) How to design a collaboration workflow to achieve a low-cost doctor workload while ensuring accuracy? To address the above challenges, we propose the Human-AI collaboration framework for disease diagnosis via doctor-enhanced transformer, called HAIformer. Specifically, to enhance AI’s capability, we propose a machine module that leverages doctors’ medical knowledge through doctor-enhanced attention, using a graph attention-based matrix; to reduce doctor workload, we propose an activation module that uses two units in a cascading manner for human-AI allocation. Experiments on four real-world datasets show that HAIformer can achieve up to 91.2% accuracy with only 18.9% human effort and one-third of dialogue turns. Further real-world clinic study highlights its advantages in practical applications.
Interactive and Iterative Peer Assessment
Authors: Dery Lihi
Published: October 2024

Iterative peer grading activities may keep students engaged during in-class project presentations. Effective methods for collecting and aggregating peer assessment data are essential. Students tend to grade projects favorably. So, while asking students for numeric grades is a common approach, it often leads to inflated grades across all projects, resulting in numerous ties for the top grades. Additionally, students may strategically assign lower grades to others’ projects so that their projects will shine. Alternatively, requesting students to rank all projects from best to worst presents challenges due to limitations in human cognitive capacity. To address these issues, we propose a novel peer grading model consisting of (a) an algorithm designed to elicit student evaluations and (b) a median-based voting protocol for aggregating grades to a single ranked order that reduces ties. An application based on our model was deployed and tested in a university course, demonstrating fewer ties between alternatives and a significant decrease in students’ cognitive and communication burdens.
Improving Calibration by Relating Focal Loss, Temperature Scaling, and Properness
Authors: Komisarenko Viacheslav , Kull Meelis
Published: October 2024

Proper losses such as cross-entropy incentivize classifiers to produce class probabilities that are well-calibrated on the training data. Due to the generalization gap, these classifiers tend to become overconfident on the test data, mandating calibration methods such as temperature scaling. The focal loss is not proper, but training with it has been shown to often result in classifiers that are better calibrated on test data. Our first contribution is a simple explanation about why focal loss training often leads to better calibration than cross-entropy training. For this, we prove that focal loss can be decomposed into a confidence-raising transformation and a proper loss. This is why focal loss pushes the model to provide under-confident predictions on the training data, resulting in being better calibrated on the test data, due to the generalization gap. Secondly, we reveal a strong connection between temperature scaling and focal loss through its confidence-raising transformation, which we refer to as the focal calibration map. Thirdly, we propose focal temperature scaling - a new post-hoc calibration method combining focal calibration and temperature scaling. Our experiments on three image classification datasets demonstrate that focal temperature scaling outperforms standard temperature scaling.
WPN: An Unlearning Method Based on N-pair Contrastive Learning in Language Models
Authors: Chen Guitao , Wang Yunshen , Sun Hongye , Chen Guang
Published: October 2024

Generative language models (LMs) offer numerous advantages but may produce inappropriate or harmful outputs due to the harmful knowledge acquired during pre-training. This knowledge often manifests as undesirable correspondences, such as “harmful prompts” leading to “harmful outputs,” which our research aims to mitigate through unlearning techniques.However, existing unlearning methods based on gradient ascent can significantly impair the performance of LMs. To address this issue, we propose a novel approach called Weighted Positional N-pair (WPN) Learning, which leverages position-weighted mean pooling within an n-pair contrastive learning framework. WPN is designed to modify the output distribution of LMs by eliminating specific harmful outputs (e.g., replacing toxic responses with neutral ones), thereby transforming the model’s behavior from “harmful prompt-harmful output” to “harmful prompt-harmless response”. Experiments on OPT and GPT-NEO LMs show that WPN effectively reduces the proportion of harmful responses, achieving a harmless rate of up to 95.8% while maintaining stable performance on nine common benchmarks (with less than 2% degradation on average). Moreover, we provide empirical evidence to demonstrate WPN’s ability to weaken the harmful correspondences in terms of generalizability and robustness, as evaluated on out-of-distribution test sets and under adversarial attacks.
Molecular Topological Profile (MOLTOP) - Simple and Strong Baseline for Molecular Graph Classification
Authors: Adamczyk Jakub , Czech Wojciech
Published: October 2024

We revisit the effectiveness of topological descriptors for molecular graph classification and design a simple, yet strong baseline. We demonstrate that a simple approach to feature engineering - employing histogram aggregation of edge descriptors and one-hot encoding for atomic numbers and bond types - when combined with a Random Forest classifier, can establish a strong baseline for Graph Neural Networks (GNNs). The novel algorithm, Molecular Topological Profile (MOLTOP), integrates Edge Betweenness Centrality, Adjusted Rand Index and SCAN Structural Similarity score. This approach proves to be remarkably competitive when compared to modern GNNs, while also being simple, fast, low-variance and hyperparameter-free. Our approach is rigorously tested on MoleculeNet datasets using fair evaluation protocol provided by Open Graph Benchmark. We additionally show out-of-domain generation capabilities on peptide classification task from Long Range Graph Benchmark. The evaluations across eleven benchmark datasets reveal MOLTOP’s strong discriminative capabilities, surpassing the 1-WL test and even 3-WL test for some classes of graphs. Our conclusion is that descriptor-based baselines, such as the one we propose, are still crucial for accurately assessing advancements in the GNN domain.
VANER: Leveraging Large Language Model for Versatile and Adaptive Biomedical Named Entity Recognition
Authors: Bian Junyi , Zhai Weiqi , Huang Xiaodi , Zheng Jiaxuan , Zhu Shanfeng
Published: October 2024

The prevalent solution for BioNER involves using representation learning techniques combined with sequence labeling. However, such methods are inherently task-specific, demonstrate poor generalizability, and often require a dedicated model for each dataset. To leverage the versatile capabilities of recent large language models (LLMs), several approaches have explored generative techniques for entity extraction. Yet, these approaches often fall short compared to previous sequence labeling approaches. In this paper, we utilize the open-sourced LLM LLaMA2 as the backbone model, and design specific instructions to distinguish between different types of entities and datasets. By combining the LLM’s understanding of instructions with sequence labeling techniques, we train a model using a mix of datasets capable of extracting various types of entities. Given that the backbone LLMs lacks specialized medical knowledge, we also integrate external entity knowledge bases and employ instruction tuning to enable the model to densely recognize curated entities. Our parameter-efficient training model, VANER, significantly outperforms previous LLMs-based models. For the first time, as an LLM-based model, VANER surpasses the majority of conventional state-of-the-art BioNER systems, achieving the highest F1 scores across three datasets.
General Lipschitz: Certified Robustness Against Resolvable Semantic Transformations via Transformation-Dependent Randomized Smoothing
Authors: Korzh Dmitrii , Pautov Mikhail , Tsymboi Olga , Oseledets Ivan
Published: October 2024

Randomized smoothing is the state-of-the-art approach to constructing image classifiers that are provably robust against additive adversarial perturbations of bounded magnitude. However, it is more complicated to compute reasonable certificates against semantic transformations (e.g., image blurring, translation, gamma correction) and their compositions. In this work, we propose General Lipschitz (GL), a new flexible framework to certify neural networks against resolvable semantic transformations. Within the framework, we analyze transformation-dependent Lipschitz-continuity of smoothed classifiers w.r.t. transformation parameters and derive corresponding robustness certificates. To assess the effectiveness of the proposed approach, we evaluate it on different image classification datasets against several state-of-the-art certification methods.
Fully Hyperbolic Rotation for Knowledge Graph Embedding
Authors: Liang Qiuyu , Wang Weihua , Bao Feilong , Gao Guanglai
Published: October 2024

Hyperbolic rotation is commonly used to effectively model knowledge graphs and their inherent hierarchies. However, existing hyperbolic rotation models rely on logarithmic and exponential mappings for feature transformation. These models only project data features into hyperbolic space for rotation, limiting their ability to fully exploit the hyperbolic space. To address this problem, we propose a novel fully hyperbolic model designed for knowledge graph embedding. Instead of feature mappings, we define the model directly in hyperbolic space with the Lorentz model. Our model considers each relation in knowledge graphs as a Lorentz rotation from the head entity to the tail entity. We adopt the Lorentzian version distance as the scoring function for measuring the plausibility of triplets. Extensive results on standard knowledge graph completion benchmarks demonstrated that our model achieves competitive results with fewer parameters. In addition, our model get the state-of-the-art performance on datasets of CoDEx-s and CoDEx-m, which are more diverse and challenging than before. Our code is available at https://github.com/llqy123/FHRE.
DrDiff: Drug Response Prediction Through Controllable Diffusion-GE and Graph Attention Network
Authors: Choi Seungyeon , Seo Samgmin , Choi Jonghwan , Park Chihyun , Park Sanghyun
Published: October 2024

The accurate prediction of drug responses based on the genomic profile of a patient is essential to progress in the field of precision medicine. The advent of various deep-learning algorithms based on publicly available large-scale omics datasets is the driving force behind research in this field. The characteristics of biological datasets, characterized by high dimensions and low sample sizes, pose challenges of overfitting and limited generalization in prediction models. Additionally, constructing prediction models using biological data such as gene expression is further complicated by the need to account for the complex relationships among genes, which exacerbates the aforementioned challenges. To address these challenges, we propose a drug response prediction framework (DrDiff) that integrates a denoising diffusion probabilistic model (DDPM) based data augmentation module with a graph attention network based drug response prediction module. The proposed model showed a 10% higher AUC than the state-of-the-art models for drug response prediction for the six drugs considered in the study, suggesting the superior generalization performance of DrDiff over other baseline models. Furthermore, we demonstrated the feasibility of generative models, which form one of the modules of the proposed framework, in overcoming the fundamental limitations of omics datasets. Further experiments bear out the feasibility of generative models, which form one of the modules of the proposed framework, in augmenting gene expression data.
On the Discovery of Conceptual Clustering Models Through Pattern Mining
Authors: Ben Hassine Motaz , Jabbour Saïd , Kmimech Mourad , Raddaoui Badran , Graiet Mohamed
Published: October 2024

Conceptual clustering is a well-studied research area in the field of unsupervised machine learning. It aims to identify disjoint clusters, where each cluster represents a collection of similar transactions described by a common pattern. The first phase of earlier conceptual clustering methods relies on the enumeration of closed patterns. Nevertheless, the extraction of such patterns can be challenging, primarily due to their rigorous nature. Indeed, closed patterns can be not frequent or fail to cover all the transactions within a cluster. To overcome this issue, this paper presents a novel approach based on the relaxation of frequent patterns called k-relaxed frequent patterns. Then, we introduce a propositional satisfiability method for enumerating such patterns. Afterwards, we employ an integer linear programming approach to compute the set of disjoint clusters. Finally, we demonstrate the efficiency of our approach through an extensive experiments conducted on several popular real-life datasets.
Hyperbolic Contrastive Learning for Document Representations – A Multi-View Approach with Paragraph-level Similarities
Authors: Nam Jaeeun , Chalkidis Ilias , Rezaei Mina
Published: October 2024

Self-supervised learning (SSL) has gained prominence due to the increasing availability of unlabeled data and advances in computational efficiency, leading to revolutionized natural language processing with pre-trained language models like BERT and GPT. Representation learning, a core concept in SSL, aims to reduce data dimensionality while preserving meaningful aspects. Conventional SSL methods typically embed data in Euclidean space. However, recent research has revealed that alternative geometries can hold even richer representations, unlocking more meaningful insights from the data. Motivated by this, we propose two novel methods for integrating Hilbert geometry into self-supervised learning for efficient document embedding. First, we present a method directly incorporating Hilbert geometry into the standard Euclidean contrastive learning framework. Additionally, we propose a multi-view hyperbolic contrastive learning framework contrasting both documents and paragraphs. Our findings demonstrate that contrasting only paragraphs, rather than entire documents, can lead to superior efficiency and effectiveness.
TabMedBERT: A Tabular Knowledge Enhanced Biomedical Pretrained Language Model
Authors: Yan Xu , Geng Lei , Cao Ziqiang , Li Juntao , Li Wenjie , Li Sujian , Zhou Xinjie , Yang Yang , Zhang Jun
Published: October 2024

Most existing biomedical language models are trained on plain text with general learning goals such as random word infilling, failing to capture the knowledge in the biomedical corpus sufficiently. Since biomedical articles usually contain many tables summarising the main entities and their relations, in the paper, we propose a Tabular knowledge enhanced bioMedical pretrained language model, called TabMedBERT. Specifically, we align entities between table cells, and article text spans with pre-defined rules. Then we add two table-related self-supervised tasks to integrate tabular knowledge into the language model: Entity Infilling (EI) and Table Cloze Test (TCT). While EI masks tokens within aligned entities in the article, TCT converts aligned entities in the table layout into a cloze text by erasing one entity and prompts the model to extract the appropriate span to fill in the blank. Experimental results demonstrate that TabMedBERT surpasses all competing language models without adding additional parameters, establishing a new state-of-the-art performance of 85.59% (+1.29%) on the BLURB biomedical NLP benchmark and 7 additional information extraction datasets. Moreover, the model architecture for TCT provides a straightforward solution to revise information extraction with paired entities.
JOSAL: Joint Learning Framework for Open-Set Active Learning
Authors: Xie Jun , Song Xiaohui , Cao Yangjie , Liu Zhi , Wang Weiping , Xu Hongli
Published: October 2024

Previous research in active learning has primarily focused on selecting examples from closed-set data, which consists solely of unlabeled examples from the target classes. However, this approach overlooks the more prevalent scenario of open-set data in real-world applications. Open-set data encompasses examples from both target classes and non-target classes. To fill this gap, we propose a novel framework called JOSAL, which enhances the accuracy of the classifier by precisely selecting the target class examples from open-set data. The JOSAL framework introduces the concept of joint learning, where the Sampler and Classifier components perform sampling and classification tasks, respectively, by sharing example features extracted from a pre-trained Encoder. To maximize the classification accuracy of the Classifier, the framework adopts a novel joint learning strategy. This strategy initially prioritizes optimizing the Sampler and gradually shifts the optimization attention to the Classifier. The experimental results demonstrate that, compared to baselines, our approach exhibits stronger sampling precision and achieves higher classification accuracy. To the best of our knowledge, this is the first work to address the open-set active learning problem using the joint learning paradigm.
Graph Continual Learning with Debiased Lossless Memory Replay
Authors: Niu Chaoxi , Pang Guansong , Chen Ling
Published: October 2024

Real-life graph data often expands continually, rendering the learning of graph neural networks (GNNs) on static graph data impractical. Graph continual learning (GCL) tackles this problem by continually adapting GNNs to the expanded graph of the current task while maintaining the performance over the graph of previous tasks. Memory replay-based methods, which aim to replay data of previous tasks when learning new tasks, have been explored as one principled approach to mitigate the forgetting of the knowledge learned from the previous tasks. In this paper we extend this methodology with a novel framework, called Debiased Lossless Memory replay (DeLoMe). Unlike existing methods that sample nodes/edges of previous graphs to construct the memory, DeLoMe learns lossless prototypical node representations as the memory. The learned memory can not only preserve the graph data privacy but also capture the holistic graph information, both of which the sampling-based methods fail to achieve. Further, prior methods suffer from bias toward the current task due to the data imbalance between the classes in the memory data and the current data. A debiased GCL loss function is devised in DeLoMe to effectively alleviate this bias. Extensive experiments on four graph datasets show the effectiveness of DeLoMe under both class- and task-incremental learning settings. Code is available at https://github.com/mala-lab/DeLoMe.
Mind the Model, Not the Agent: The Primacy Bias in Model-Based RL
Authors: Qiao Zhongjian , Lyu Jiafei , Li Xiu
Published: October 2024

The primacy bias in model-free reinforcement learning (MFRL), which refers to the agent’s tendency to overfit early data and lose the ability to learn from new data, can significantly decrease the performance of MFRL algorithms. Previous studies have shown that employing simple techniques, such as resetting the agent’s parameters, can substantially alleviate the primacy bias in MFRL. However, the primacy bias in model-based reinforcement learning (MBRL) remains unexplored. In this work, we focus on investigating the primacy bias in MBRL. We begin by observing that resetting the agent’s parameters harms its performance in the context of MBRL. We further find that the primacy bias in MBRL is more closely related to the primacy bias of the world model instead of the primacy bias of the agent. Based on this finding, we propose world model resetting, a simple yet effective technique to alleviate the primacy bias in MBRL. We apply our method to two different MBRL algorithms, MBPO and DreamerV2. We validate the effectiveness of our method on multiple continuous control tasks on MuJoCo and DeepMind Control Suite, as well as discrete control tasks on Atari 100k benchmark. The experimental results show that world model resetting can significantly alleviate the primacy bias in the model-based setting and improve the algorithm’s performance. We also give a guide on how to perform world model resetting effectively.
Defending Our Privacy with Backdoors
Authors: Hintersdorf Dominik , Struppek Lukas , Neider Daniel , Kersting Kristian
Published: October 2024

The proliferation of large AI models trained on uncurated, often sensitive web-scraped data has raised significant privacy concerns. One of the concerns is that adversaries can extract information about the training data using privacy attacks. Unfortunately, the task of removing specific information from the models without sacrificing performance is not straightforward and has proven to be challenging. We propose a rather easy yet effective defense based on backdoor attacks to remove private information, such as names and faces of individuals, from vision-language models by fine-tuning them for only a few minutes instead of re-training them from scratch. Specifically, by strategically inserting backdoors into text encoders, we align the embeddings of sensitive phrases with those of neutral terms–“a person” instead of the person’s actual name. For image encoders, we map individuals’ embeddings to be removed from the model to a universal, anonymous embedding. The results of our extensive experimental evaluation demonstrate the effectiveness of our backdoor-based defense on CLIP by assessing its performance using a specialized privacy attack for zero-shot classifiers. Our approach provides a new “dual-use” perspective on backdoor attacks and presents a promising avenue to enhance the privacy of individuals within models trained on uncurated web-scraped data.
Enhancing Reinforcement Learning Through Guided Search
Authors: Arjonilla Jérôme , Saffidine Abdallah , Cazenave Tristan
Published: October 2024

With the aim of improving performance in Markov Decision Problem in an Off-Policy setting, we suggest taking inspiration from what is done in Offline Reinforcement Learning (RL). In Offline RL, it is a common practice during policy learning to maintain proximity to a reference policy to mitigate uncertainty, reduce potential policy errors, and help improve performance. We find ourselves in a different setting, yet it raises questions about whether a similar concept can be applied to enhance performance i.e., whether it is possible to find a guiding policy capable of contributing to performance improvement, and how to incorporate it into our RL agent. Our attention is particularly focused on algorithms based on Monte Carlo Tree Search (MCTS) as a guide. MCTS renowned for its state-of-the-art capabilities across various domains, catches our interest due to its ability to converge to equilibrium in single-player and two-player contexts. By harnessing the power of MCTS as a guide for our RL agent, we observed a significant performance improvement, surpassing the outcomes achieved by utilizing each method in isolation. Our experiments were carried out on the Atari 100k benchmark.
Domain Adaptational Steganographic Text Detection Using Few-Shot Adversary-Refinement Framework
Authors: Ziwei Zhang , Juan Wen , Yinghan Zhou , Liting Gao , Yiming Xue
Published: October 2024

Text steganography involves discreetly concealing sensitive messages within natural text, while text steganalysis serves as its counterpart by aiming to detect suspicious text that may contain embedded secret information. Detecting steganographic text has become increasingly difficult because evolving steganographic algorithms produce ever-changing text distributions. Consequently, few-shot text steganalysis, which identifies steganographic text with scarce examples regardless of its distribution has become a research hotspot. The state-of-the-art few-shot text steganalysis relies on the inter-class variance between classes, i.e., they behave satisfactorily in detecting large-variance classes while being incompetent in distinguishing confusable samples from similar steganographic settings. In this paper, we propose an Adversary-Refinement Framework for Text Steganalysis, namely ARTS, which employs a task-invariant extractor and a task-relevant projector to implement an “attract and repel” process. Specifically, in the “attract” stage, we align task-invariant features through adversarial training to shorten the intra-class distance. Afterward, the refined prototypes are projected to a new space in the “repel” stage, and then a refined penalty item is applied to enlarge the inter-class distance. Extensive experiments conducted in six datasets with different inter-class variances demonstrate the superiority of the proposed model over the SOTA models.
Relation Time-Aware Heterogeneous Dynamic Graph Neural Networks
Authors: Wang Yili , Chen Jiamin , Li Qiutong , He Changlong , Gao Jianliang
Published: October 2024

Heterogeneous dynamic graph neural networks (HDGNNs) are effective methods for processing heterogeneous temporal graphs (HTGs), which serve as ubiquitous data structures in real-world scenarios. The previous HDGNN paradigm obtains representations of future target nodes by mining the spatial heterogeneity and temporal dependence of node attributes, ignoring the learning of relation temporal dependence. However, through experience, we find that the learning of relation temporal dependence, which describes the evolving trends in the importance of neighbors under a certain relation, is beneficial for representation learning of HTGs. To bridge this gap, we propose a novel end-to-end heterogeneous temporal graph learning paradigm called Relation Time-aware Heterogeneous Dynamic Graph Neural Networks (ReTag). Compared to previous HDGNNs, ReTag extracts the temporal dependence of relations from historical relation information and the evolving node attributes to drive subsequent spatio-temporal representation learning. As far as we know, ReTag is the first attempt to perform learning of the temporal dependence of relation, which can generate a more effective representation for different downstream tasks of HTGs. The experimental results of different downstream tasks of HTGs based on multiple benchmark datasets show that ReTag can obtain obvious performance advantages compared with the sota baseline method.
Memory Adaptive and Spatially Specialized Model Ensembles for Industrial Anomaly Detection
Authors: Wagenstetter Marco , Landerer Niklas , Thyroff Johannes , Aicher Thomas , Hellmich Arvid , Ihlenfeldt Steffen
Published: October 2024

Industrial anomaly detection models proved to be a viable solution for automating the task of visual inspection. Despite their effectiveness, the deployment of current deep learning models for anomaly detection for high-resolution images involves large memory requirements, which are unattainable in industrial deployments on edge devices. In this paper we introduce a tiling approach for a memory bank based feature embedding model to reduce the memory requirements during training and inference. Additionally, we analyze the correlations between image size, dataset size and required memory. To evaluate the detection performance of different tiling schemes, we create and apply a real industrial training and validation dataset collected in the electric motor housing production of the BMW Group. Our experiments show that depending on the tiling scheme the overall required memory can be decreased by a factor of 12 while leaving the detection performance unchanged and even increase the detection performance when the memory requirement is only halved. This venture underscores the viability of our tiling approach in real-world industrial applications but also marks a significant leap towards resolving the issue of resolution limitations in deploying advanced anomaly detection models.
On the Improvement of Generalization and Stability of Forward-Only Learning via Neural Polarization
Authors: Terres-Escudero Erik B. , Del Ser Javier , Garcia-Bringas Pablo
Published: October 2024

Forward-only learning algorithms have recently gained attention as alternatives to gradient backpropagation, replacing the backward step of this latter solver with an additional contrastive forward pass. Among these approaches, the so-called Forward-Forward Algorithm (FFA) has been shown to achieve competitive levels of performance in terms of generalization and complexity. Networks trained using FFA learn to contrastively maximize a layer-wise defined goodness score when presented with real data (denoted as positive samples) and to minimize it when processing synthetic data (corr. negative samples). However, this algorithm still faces weaknesses that negatively affect the model accuracy and training stability, primarily due to a gradient imbalance between positive and negative samples. To overcome this issue, in this work we propose a novel implementation of the FFA algorithm, denoted as Polar-FFA, which extends the original formulation by introducing a neural division (polarization) between positive and negative instances. Neurons in each of these groups aim to maximize their goodness when presented with their respective data type, thereby creating a symmetric gradient behavior. To empirically gauge the improved learning capabilities of our proposed Polar-FFA, we perform several systematic experiments using different activation and goodness functions over image classification datasets. Our results demonstrate that Polar-FFA outperforms FFA in terms of accuracy and convergence speed. Furthermore, its lower reliance on hyperparameters reduces the need for hyperparameter tuning to guarantee optimal generalization capabilities, thereby allowing for a broader range of neural network configurations.
Take a Step and Reconsider: Sequence Decoding for Self-Improved Neural Combinatorial Optimization
Authors: Pirnay Jonathan , Grimm Dominik G.
Published: October 2024

The constructive approach within Neural Combinatorial Optimization (NCO) treats a combinatorial optimization problem as a finite Markov decision process, where solutions are built incrementally through a sequence of decisions guided by a neural policy network. To train the policy, recent research is shifting toward a ‘self-improved’ learning methodology that addresses the limitations of reinforcement learning and supervised approaches. Here, the policy is iteratively trained in a supervised manner, with solutions derived from the current policy serving as pseudo-labels. The way these solutions are obtained from the policy determines the quality of the pseudo-labels. In this paper, we present a simple and problem-independent sequence decoding method for self-improved learning based on sampling sequences without replacement. We incrementally follow the best solution found and repeat the sampling process from intermediate partial solutions. By modifying the policy to ignore previously sampled sequences, we force it to consider only unseen alternatives, thereby increasing solution diversity. Experimental results for the Traveling Salesman and Capacitated Vehicle Routing Problem demonstrate its strong performance. Furthermore, our method outperforms previous NCO approaches on the Job Shop Scheduling Problem.
Learning Order Forest for Qualitative-Attribute Data Clustering
Authors: Zhao Mingjie , Feng Sen , Zhang Yiqun , Li Mengke , Lu Yang , Cheung Yiu-Ming
Published: October 2024

Clustering is a fundamental approach to understanding data patterns, wherein the intuitive Euclidean distance space is commonly adopted. However, this is not the case for implicit cluster distributions reflected by qualitative attribute values, e.g., the nominal values of attributes like symptoms, marital status, etc. This paper, therefore, discovered a tree-like distance structure to flexibly represent the local order relationship among intra-attribute qualitative values. That is, treating a value as the vertex of the tree allows to capture rich order relationships among the vertex value and the others. To obtain the trees in a clustering-friendly form, a joint learning mechanism is proposed to iteratively obtain more appropriate tree structures and clusters. It turns out that the latent distance space of the whole dataset can be well-represented by a forest consisting of the learned trees. Extensive experiments demonstrate that the joint learning adapts the forest to the clustering task to yield accurate results. Comparisons of 10 counterparts on 12 real benchmark datasets with significance tests verify the superiority of the proposed method. Source code of the proposed method is available at [39].
Open-Set Multivariate Time-Series Anomaly Detection
Authors: Lai Thomas , Ho Thi Kieu Khanh , Armanfard Narges
Published: October 2024

Numerous methods for time-series anomaly detection (TSAD) have emerged in recent years, most of which are unsupervised and assume that only normal samples are available during the training phase, due to the challenge of obtaining abnormal data in real-world scenarios. Still, limited samples of abnormal data are often available, albeit they are far from representative of all possible anomalies. Supervised methods can be utilized to classify normal and seen anomalies, but they tend to overfit to the seen anomalies present during training, hence, they fail to generalize to unseen anomalies. We propose the first algorithm to address the open-set TSAD problem, called Multivariate Open-Set time-series Anomaly Detector (MOSAD), that leverages only a few shots of labeled anomalies during the training phase in order to achieve superior anomaly detection performance compared to both supervised and unsupervised TSAD algorithms. MOSAD is a novel multi-head TSAD framework with a shared representation space and specialized heads, including the Generative head, the Discriminative head, and the Anomaly-Aware Contrastive head. The latter produces a superior representation space for anomaly detection compared to conventional supervised contrastive learning. Extensive experiments on three real-world datasets establish MOSAD as a new state-of-the-art in the TSAD field.
Learning After Learning: Positive Backward Transfer in Continual Learning
Authors: Wong Wern Sen , Koh Yun Sing , Dobbie Gillian
Published: October 2024

Continual Learning (CL) methods enable models to learn new tasks without forgetting previously learned ones. Catastrophic Forgetting (CF) occurs when the parameters of a neural network are updated for a new task, causing the model to lose performance on tasks it has previously learned. To mitigate CF, parameter isolation methods use a “task mask” to allocate a subset of weights to each task; these weights are typically frozen to preserve task performance. However, frozen weights can limit positive backward transfer, which is the beneficial reuse of knowledge from new tasks to improve the accuracy of previously learned tasks. To address this gap, we introduce LEarning AFter learning (LEAF), a novel CL method that enables positive backward transfer by dynamically updating frozen task masks based on gradient updates that signal sufficient backward knowledge transfer. This mechanism allows for selective integration of new knowledge without sacrificing previously acquired knowledge. Our experiments show that LEAF surpasses existing state-of-the-art methods in terms of accuracy while maintaining comparable memory and runtime efficiencies. Moreover, it outperforms other backward transfer techniques in improving the accuracy of a prioritized task. Our code is available at https://github.com/wernse/LEAF.
The Role of Depth, Width, and Tree Size in Expressiveness of Deep Forest
Authors: Lyu Shen-Huan , Wu Jin-Hui , Zheng Qin-Cheng , Ye Baoliu
Published: October 2024

Random forests are classical ensemble algorithms that construct multiple randomized decision trees and aggregate their predictions using naive averaging. Zhou and Feng [51] further propose a deep forest algorithm with multi-layer forests, which outperforms random forests in various tasks. The performance of deep forests is related to three hyperparameters in practice: depth, width, and tree size, but little has been known about its theoretical explanation. This work provides the first upper and lower bounds on the approximation complexity of deep forests concerning the three hyperparameters. Our results confirm the distinctive role of depth, which can exponentially enhance the expressiveness of deep forests compared with width and tree size. Experiments validate these theoretical findings. The detailed proof and code are available in the full version [31].
FedReMa: Improving Personalized Federated Learning via Leveraging the Most Relevant Clients
Authors: Liang Han , Zhan Ziwei , Liu Weijie , Zhang Xiaoxi , Tan Chee Wei , Chen Xu
Published: October 2024

Federated Learning (FL) is a distributed machine learning paradigm that achieves a globally robust model through decentralized computation and periodic model synthesis, primarily focusing on the global model’s accuracy over aggregated datasets of all participating clients. Personalized Federated Learning (PFL) instead tailors exclusive models for each client, aiming to enhance the accuracy of clients’ individual models on specific local data distributions. Despite of their wide adoption, existing FL and PFL works have yet to comprehensively address the class-imbalance issue, one of the most critical challenges within the realm of data heterogeneity in PFL and FL research. In this paper, we propose FedReMa, an efficient PFL algorithm that can tackle class-imbalance by 1) utilizing an adaptive inter-client co-learning approach to identify and harness different clients’ expertise on different data classes throughout various phases of the training process, and 2) employing distinct aggregation methods for clients’ feature extractors and classifiers, with the choices informed by the different roles and implications of these model components. Specifically, driven by our experimental findings on inter-client similarity dynamics, we develop critical co-learning period (CCP), wherein we introduce a module named maximum difference segmentation (MDS) to assess and manage task relevance by analyzing the similarities between clients’ logits of their classifiers. Outside the CCP, we employ an additional scheme for model aggregation that utilizes historical records of each client’s most relevant peers to further enhance the personalization stability. We demonstrate the superiority of our FedReMa in extensive experiments. The code is available at https://github.com/liangh68/FedReMa.
From Document to Program Embeddings: Can Distributional Hypothesis Really Be Used on Programming Languages?
Authors: Martinet Thibaut , Cleuziou Guillaume , Exbrayat Matthieu , Flouvat Frédéric
Published: October 2024

Programming language processing is a field of increasing interest, as more and more models become available, either to address specific tasks or to acquire general knowledge which can then be fine-tuned on downstream tasks. All these models are based on architectures that come from the field of natural language processing, most of them being built on the distributional hypothesis from linguistics. Although this transition from one field to another appears to have occurred naturally, it is not so obvious to claim that this hypothesis will be appropriate for extracting semantics from programs. In this paper, we investigate to which extent a distributional hypothesis can be applied to code embedding. To this end, we first formulate various hypotheses adapted to the specific information contained in programming languages. We then provide a framework to evaluate the effectiveness of these hypotheses through the quality of the resulting embedding spaces. This framework is based on the doc2vec model as a generic language model, as its implementation of the original distributional hypothesis is easy to understand and to adapt to any new ones. Among other tools, we propose a new evaluation method based on program analogies, which measures how well the models capture the underlying structure and meaning of the code. We apply the proposed framework to a set of (distributional) hypotheses and show that we can rule out certain hypotheses in favor of others. Specifically, our study indicates that instruction-based hypotheses capture less semantic information than token-based ones. Furthermore, we observe that distributional hypotheses on tokens are effective in both source code, execution traces, and abstract syntax trees. Additionally, we find that the semantics captured on programs by these three hypotheses are of comparable levels and natures.
Identifying the Best Arm in the Presence of Global Environment Shifts
Authors: Srisawad Phurinut , Branke Juergen , Tran-Thanh Long
Published: October 2024

This paper formulates a new Best-Arm Identification problem in the non-stationary stochastic bandits setting, where the means of all arms are shifted in the same way due to a global influence of the environment. The aim is to identify the unique best arm across environmental change given a fixed total budget. While this setting can be regarded as a special case of Adversarial Bandits or Corrupted Bandits, we demonstrate that existing solutions tailored to those settings do not fully utilise the nature of this global influence, and thus, do not work well in practice (despite their theoretical guarantees). To overcome this issue, in this paper we develop a novel selection policy that is consistent and robust in dealing with global environmental shifts. We then propose an allocation policy, LinLUCB, which exploits information about global shifts across all arms in each environment. Empirical tests depict a significant improvement in our policies against other existing methods.