bluuebunny/crossref_metadata_2025 huggingface dataset:
Hyperparameter Importance Analysis for Multi-Objective AutoML
Authors: Theodorakopoulos Daphne , Stahl Frederic , Lindauer Marius
Published: October 2024

Hyperparameter optimization plays a pivotal role in enhancing the predictive performance and generalization capabilities of ML models. However, in many applications, we do not only care about predictive performance but also about additional objectives such as inference time, memory, or energy consumption. In such multi-objective scenarios, determining the importance of hyperparameters poses a significant challenge due to the complex interplay between the conflicting objectives. In this paper, we propose the first method for assessing the importance of hyperparameters in multi-objective hyperparameter optimization. Our approach leverages surrogate-based hyperparameter importance measures, i.e., fANOVA and ablation paths, to provide insights into the impact of hyperparameters on the optimization objectives. Specifically, we compute the a-priori scalarization of the objectives and determine the importance of the hyperparameters for different objective tradeoffs. Through extensive empirical evaluations on diverse benchmark datasets with three different objective pairs, each combined with accuracy, namely time, demographic parity loss, and energy consumption, we demonstrate the effectiveness and robustness of our proposed method. Our findings not only offer valuable guidance for hyperparameter tuning in multi-objective optimization tasks but also contribute to advancing the understanding of hyperparameter importance in complex optimization scenarios.
IndMask: Inductive Explanation for Multivariate Time Series Black-Box Models
Authors: Nasr Seham , Sikdar Sandipan
Published: October 2024

In this paper, we introduce IndMask, a framework for explaining decisions of black-box time series models. While there exists a plethora of methods for providing explanations of machine learning models, time series data requires additional considerations. One needs to consider the time aspect in the explanations as well as deal with a large number of input features. Recent work has proposed explaining a time series prediction by generating a mask over the input time series. Each entry in the mask corresponds to an importance score for each feature at each time step. However, these methods only generate instancewise explanations, which means a mask needs to be computed for each input individually, thereby making them unsuited for inductive settings, where explanations need to be generated for numerous inputs, and instancewise explanation generation is severely prohibitive. Additionally, these methods have mostly been evaluated on simple recurrent neural networks and are often only applicable to a specific downstream task. Our proposed framework IndMask addresses these issues by utilizing a parameterized model for mask generation. We also go beyond recurrent neural networks and deploy IndMask to transformer architectures, thereby genuinely demonstrating its model-agnostic nature. The effectiveness of IndMask is further demonstrated through experiments over real-world datasets and time series classification and forecasting tasks. It is also computationally efficient and can be deployed in conjunction with any time series model.
Temporal Fairness in Decision Making Problems
Authors: Torres Manuel R. , Zehtabi Parisa , Cashmore Michael , Magazzeni Daniele , Veloso Manuela
Published: October 2024

In this work we consider a new interpretation of fairness in decision making problems. Building upon existing fairness formulations, we focus on how to reason over fairness from a temporal perspective, taking into account the fairness of a history of past decisions. After introducing the concept of temporal fairness, we propose three approaches that incorporate temporal fairness in decision making problems formulated as optimization problems. We present a qualitative evaluation of our approach in four different domains and compare the solutions against a baseline approach that does not consider the temporal aspect of fairness.
Empowering Biomedical Named Entity Recognition through Multi-Tagger Collaboration
Authors: Zhao Jin , Xie Jian , Zhu Tinghui , Guo Qian , Li Zhixu , Xiao Yanghua
Published: October 2024

Biomedical Named Entity Recognition (BioNER) plays a crucial role in automatically identifying specific categories of entities from biomedical texts. Currently, region-based methods have shown promising performance in BioNER. However, existing paradigms in the region-based methods suffer from inherent limitations, including the generation of negative samples, and the ignorance of token dependencies. To overcome these limitations, we propose a new paradigm, implemented as Token Cascade Tagger (TCT), which combines span identification and category classification. The TCT utilizes category information to enhance the correlation between the heads and tails of entities, effectively reducing the generation of negative samples. Additionally, we introduce a Token Dependency Tagger (TDT) that captures token dependencies within entity spans by identifying the longest span in a sentence. The TDT filters out incorrect spans and further improves the accuracy of span detection obtained from the TCT. Furthermore, we employ a multi-task learning framework to optimize both the TCT and TDT, leading to superior performance in BioNER. Extensive experiments on publicly available biomedical datasets demonstrate our method outperforms the previous state-of-the-art methods, achieving 92.44%, 92.54%, and 81.26% on NCBI-Disease, BC5CDR, and GENIA, respectively, in terms of F1 score.
Detecting Patterns of Attacks to Network Security in Urban Air Mobility with Answer Set Programming
Authors: Sterlicchio Gioacchino , Lisi Francesca Alessandra
Published: October 2024

The growth of unmanned aerial vehicles (UAVs) will make the sky more crowded and pose several challenges as regards safety and security. Enabling high-rate, low-latency and ultra-reliable wireless communication between UAVs and ground base is crucial to realize their large-scale usage in the future, especially in the field of Urban Air Mobility. Recently, cellular-connected UAVs have drawn significant attention as a promising technology for Automatic Dependence Surveillance Broadcast (ADS-B) Like communication, which leverages other types of communication such as 4G LTE. In this work, we address the current lack of ADS-B security features and propose to use Answer Set Programming (ASP) for finding contrast sequential patterns that characterize different attacks on the 4G LTE network. The experiments show that a declarative approach is feasible in this context, and that the implementation of span and gap constraints make the search for patterns more efficient and effective.
Learning Compositional, Time-Varying Neural Barrier Contracts
Authors: Low Matthew , Wang Timothy E. , Nuzzo Pierluigi
Published: October 2024

Certificate functions can be used to efficiently capture and prove various properties of a system or controller. Control barrier functions (CBFs) are certificate functions that define a region of forward-invariance, which makes them a natural choice to enforce a notion of “safety” for a system. However, synthesizing CBFs, especially in the case of complex systems with learning-enabled components, remains a challenge. Recent work achieves promising results by leveraging neural networks as function approximators to learn CBFs. However, a single CBF may not exist or be difficult to obtain for complex hybrid systems. To overcome this difficulty, this paper presents a framework to simultaneously learn simpler, time-varying control barrier functions (TV-CBFs) that are composable. We embed these neural certificates in a compositional framework based on assume-guarantee contracts. The resulting neural barrier contracts can then be combined by leveraging the rigorous contract algebra. Learning multiple, composable CBFs empowers the verification process (1) by simplifying the verification of complex systems via decomposition and (2) by broadening the expressivity in capturing complex systems and controllers. We illustrate the effectiveness of our approach on the verification of an aircraft’s automatic landing system and a quadrotor navigating an indoor environment.
Breaking the Weak Semantics Bottleneck of Transformers in Time Series Forecasting
Authors: Yang Ziang , Zhou Biyu , Tang Xuehai , Li Ruixuan , Hu Songlin
Published: October 2024

Transformer with self-attention was initially crafted to model language sequences, where discrete tokens (i.e., words) showcase high semantic density. However, when applied to time series token inputs (i.e., datapoints) with weak-density semantics and temporal redundancy, it faces challenges as these time-domain tokens impede its ability to capture the intricate latent properties of time series dynamics. While time-frequency transformation presents a viable solution by bringing forth a new space with heightened expressive power, existing approaches fall short of fully exploiting its potential. In response to these limitations, we propose a general-purpose transformer-based model, named Scattering Transformer, for multivariate time series forecasting and self-supervised representation learning. It is based on two innovative components: i) scattering self-attention mechanism incorporating wavelet key/value and standard query to unify the learning of cross-domain relationships between the time and wavelet domains; and ii) stochastic scaling positional encoding scheme that relies solely on order information, emulating longer sequence positions to generalize up to ultra-long horizon case. Extensive experiments on eight real-world benchmarks show the potential of our Scattering Transformer as a robust and versatile solution, showcasing its quadruple efficacy of non-stationary forecasting, ultra-long horizons forecasting, representation learning, and reduction in time and space complexity.
Deep Learning for in vivo Bronchial Carinae Detection in Flexible Bronchoscopy
Authors: Ghyselinck Robin , Delchevalerie Valentin , Poitier Pierre , Frénay Benoît , Dumas Bruno
Published: October 2024

Early lung cancer detection strongly increases survival rate. During a navigational bronchoscopy, pulmonologists perform tissue sampling for biopsies based on preoperative medical images. The bronchial carina is an airway structure that appears at each bronchus bifurcation. It is an important landmark to detect during navigations as it indicates the need to choose between multiple paths and keep track of the position in the lungs. In this paper, we assessed various deep learning pipelines including the use of semi-supervised, segmentation and recurrent methods under different setups to perform bronchial carinae detection. In contrast to most previous works that focus on phantoms, cadavers or virtual images, we exploit a large corpus of proprietary in vivo data captured during real endoscopic procedures using a mini probe. To the best of our knowledge, it is the first work that deals with this quantity of real and challenging data. After performing a comparison study, we conclude that the best performance to detect bronchial carinae is achieved by a semi-supervised pipeline that leverages the ability of nnU-Net to solve segmentation tasks, coupled with Gated Recurrent Units that extracts temporal contexts from image sequences.
Multi-Agent Reinforcement Learning for Alternating-Time Logic
Authors: Hahn Ernst Moritz , Perez Mateo , Schewe Sven , Somenzi Fabio , Trivedi Ashutosh , Wojtczak Dominik
Published: October 2024

Alternating-time temporal logic (ATL) extends branching time logic by enabling quantification over paths that result from the strategic choices made by multiple agents in various coalitions within the system. While classical temporal logics express properties of “closed” systems, ATL can express properties of “open” systems resulting from interactions among several agents. Reinforcement learning (RL) is a sampling-based approach to decision-making where learning agents, guided by a scalar reward function, discover optimal policies through repeated interactions with the environment. The challenge of translating high-level objectives into scalar rewards for RL has garnered increased interest, particularly following the success of model-free RL algorithms. This paper presents an approach for deploying model-free RL to verify multi-agent systems against ATL specifications. The key contribution of this paper is a verification procedure for model-free RL of quantitative and non-nested classic ATL properties, based on Q-learning, demonstrated on a natural subclass of non-nested ATL formulas.
Modelling Brain Connectomes Networks: Solv is a Worthy Competitor to Hyperbolic Geometry!
Authors: Celińska-Kopczyńska Dorota , Kopczyński Eryk
Published: October 2024

Finding suitable embeddings for connectomes (spatially embedded complex networks that map neural connections in the brain) is crucial for analyzing and understanding cognitive processes. Recent studies have found two-dimensional hyperbolic embeddings superior to Euclidean embeddings in modeling connectomes across species, especially human connectomes. However, those studies had limitations: geometries other than Euclidean, hyperbolic, or spherical were not considered. Following William Thurston’s suggestion that the networks of neurons in the brain could be successfully represented in Solv geometry, we study the goodness-of-fit of the embeddings for 21 connectome networks (8 species). To this end, we suggest an embedding algorithm based on Simulating Annealing that allows us to embed connectomes to Euclidean, Spherical, Hyperbolic, Solv, Nil, and product geometries. Our algorithm tends to find better embeddings than the state-of-the-art, even in the hyperbolic case. Our findings suggest that while three-dimensional hyperbolic embeddings yield the best results in many cases, Solv embeddings perform reasonably well.
Capacitated Online Clustering Algorithm
Authors: Gupta Shivam , Jain Shweta , Krishnan Narayanan C , Ghalme Ganesh , Hemachandra Nandyala
Published: October 2024

Clustering is a widely used unsupervised learning tool with applications in numerous real-world problems. Traditional clustering methods can result in highly skewed clusters where one cluster is notably larger than others, rendering them unsuitable for scenarios such as logistics and routing. In response, capacitated clustering approaches have emerged over the past decade. These approaches limit the number of data points each cluster can accommodate, thus resulting in more uniform cluster formations. In an online version of capacitated clustering, the algorithm must make an irrevocable decision for each incoming data point, determining whether to establish it as a new center or allocate it to existing centers. The goal is to minimize the count of opened centers while adhering to capacity constraints and achieving a satisfactory approximation of the clustering cost compared to the optimal solution. Although exploring online capacitated clustering remains uncharted, we are the first to propose a probabilistic Capacitated Online Clustering Algorithm (called COCA) for h-dimensional euclidean spaces. We theoretically bound the number of centers opened and provide constant cost approximation guarantees. Additionally, we conduct rigorous experiments to validate the computational efficacy of the proposed approaches.
Cross-Stage Transfer in Multi-Stage Cascade Ranking and Filtering Systems
Authors: Pan Yifan , Luo Guibo , Zhu Yuesheng
Published: October 2024

Unsupervised Domain Adaptation (UDA) aims to transfer a model from a labeled source domain to an unlabeled target domain, addressing challenges of distinct data distributions, termed domain shift. Existing UDA research primarily focuses on classification-like tasks, but neglects ranking and filtering tasks essential for applications like medical diagnosis and search engines. This paper is the first to notice and identify a new real-world transfer problem: cross-stage transfer in multi-stage cascade ranking and filtering systems, a common issue in diverse applications, including information retrieval systems, medical diagnosis, and other real-world ranking/filtering systems. In this problem, we emphasize the crucial assumption of order-invariance and address the key issue named Cross-stage Class Concept Conflict (C4), highlighting potential inconsistencies in class concepts for the same sample at different stages. To tackle these challenges, we propose a novel method, Unsupervised Rank Adaptation (URA), comprising two key components: order-conditional distribution alignment, characterizing the order-conditional distribution intra-stage and aligning them across stages; and principal projection alignment, aligning the principal component’s projection matrix with classifier parameters to ensure order-invariance without guessing pseudo-labels, mitigating the influence of C4. Experimental results show that our approach reaches state-of-the-art performance in various cross-stage transfer tasks.
Adversarially Robust Neural Lyapunov Control
Authors: Wei Li , Jiang Yuankun , Li Chenglin , Dai Wenrui , Zou Junni , Xiong Hongkai
Published: October 2024

State-of-the-art learning-based stability control methods for nonlinear robotic systems suffer from the issue of reality gap, which stems from discrepancy of the system dynamics between training and target (test) environments. To mitigate this gap, we propose an adversarially robust neural Lyapunov control (ARNLC) method to improve the robustness and generalization capabilities for Lyapunov theory-based stability control. Specifically, inspired by adversarial learning, we introduce an adversary to simulate the dynamics discrepancy, which is learned through deep reinforcement learning to generate the worst-case perturbations during the controller’s training. By alternatively updating the controller to minimize the perturbed Lyapunov risk and the adversary to deviate the controller from its objective, the learned control policy enjoys a theoretical guarantee of stability. Empirical evaluations on five stability control tasks with the uniform and worst-case perturbations demonstrate that ARNLC not only accelerates the convergence to asymptotic stability, but can generalize better in the entire perturbation space.
Cluster Exploration Using Informative Manifold Projections
Authors: Gerolymatos Stavros , Evangelopoulos Xenophon , Gusev Vladimir V. , Goulermas John Y.
Published: October 2024

Dimensionality reduction (DR) is one of the key tools for the visual exploration of high-dimensional data and uncovering its cluster structure in two- or three-dimensional spaces. The vast majority of DR methods in the literature do not take into account any prior knowledge a practitioner may have regarding the dataset under consideration. We propose a novel method to generate informative embeddings which not only factor out the structure associated with different kinds of prior knowledge but also aim to reveal any remaining underlying structure. To achieve this, we employ a linear combination of two objectives: firstly, contrastive PCA that discounts the structure associated with the prior information, and secondly, kurtosis projection pursuit which ensures meaningful data separation in the obtained embeddings. We formulate this task as a manifold optimization problem and validate it empirically across a variety of datasets considering three distinct types of prior knowledge. Lastly, we provide an automated framework to perform iterative visual exploration of high-dimensional data.
Perturb-and-Compare Approach for Detecting Out-of-Distribution Samples in Constrained Access Environments
Authors: Lee Heeyoung , Byun Hoyoon , Oh Changdae , Bak JinYeong , Song Kyungwoo
Published: October 2024

Accessing machine learning models through remote APIs has been gaining prevalence following the recent trend of scaling up model parameters for increased performance. Even though these models exhibit remarkable ability, detecting out-of-distribution (OOD) samples remains a crucial safety concern for end users as these samples may induce unreliable outputs from the model. In this work, we propose an OOD detection framework, MixDiff, that is applicable even when the model’s parameters or its activations are not accessible to the end user. To bypass the access restriction, MixDiff applies an identical input-level perturbation to a given target sample and a similar in-distribution (ID) sample, then compares the relative difference in the model outputs of these two samples. MixDiff is model-agnostic and compatible with existing output-based OOD detection methods. We provide theoretical analysis to illustrate MixDiff’s effectiveness in discerning OOD samples that induce overconfident outputs from the model and empirically demonstrate that MixDiff consistently enhances the OOD detection performance on various datasets in vision and text domains.
Advancing ConvNet Architectures: A Novel XGB-based Pruning Algorithm for Transfer Learning Efficiency
Authors: Ratajczyk Igor , Horzyk Adrian
Published: October 2024

This paper introduces a novel pruning method designed for transfer-learning models in computer vision, leveraging XGBoost to enhance model efficiency through simultaneous depth and width pruning. Contemporary computer vision tasks often rely on transfer learning due to its efficiency after short training periods. However, these pre-trained architectures, while robust, are typically overparameterized, leading to inefficiencies when fine-tuning for simpler problems. Our approach utilizes Parallel Tree Boosting as a surrogate neural classifier to evaluate and prune unnecessary convolutional filters in these architectures effectively. This method addresses the challenge of interference that is often present in transfer-learning models and enables a significant reduction in the size of the model. We demonstrate that it is feasible to achieve both depth and width pruning concurrently, allowing for substantial reductions in model complexity without compromising performance. Experimental results show that our proposed XGB-based Pruner not only reduces the model size by nearly 50% but also improves the accuracy by up to 9% in test data compared to conventional models. Moreover, it can achieve a dramatic reduction in model size by over a hundred times with minimal accuracy loss, proving its effectiveness across various architectures and datasets. This approach marks a significant advancement in optimizing transfer-learning models, promising enhanced performance in real-world applications.
Theoretical Proportion Label Perturbation for Learning from Label Proportions in Large Bags
Authors: Kubo Shunsuke , Matsuo Shinnosuke , Suehiro Daiki , Terada Kazuhiro , Ito Hiroaki , Yoshizawa Akihiko , Bise Ryoma
Published: October 2024

Learning from label proportions (LLP) is a kind of weakly supervised learning that trains an instance-level classifier from label proportions of bags, which consist of sets of instances without using instance labels. A challenge in LLP arises when the number of instances in a bag (bag size) is numerous, making the traditional LLP methods difficult due to GPU memory limitations. This study aims to develop an LLP method capable of learning from bags with large sizes. In our method, smaller bags (mini-bags) are generated by sampling instances from large-sized bags (original bags), and these mini-bags are used in place of the original bags. However, the proportion of a mini-bag is unknown and differs from that of the original bag, leading to overfitting. To address this issue, we propose a perturbation method for the proportion labels of sampled mini-bags to mitigate overfitting to noisy label proportions. This perturbation is added based on the multivariate hypergeometric distribution, which is statistically modeled. Additionally, loss weighting is implemented to reduce the negative impact of proportions sampled from the tail of the distribution. Experimental results demonstrate that the proportion label perturbation and loss weighting achieve classification accuracy comparable to that obtained without sampling. Our codes are available at https://github.com/stainlessnight/LLP-LargeBags.
TabCGOK: Intra-Class Groups Retrieval and Inter-Class Ordinal Knowledge Augmented Network for Ordinal Tabular Data Prediction
Authors: Luo Zhengdong , Atawulla Abibulla , Yang Fengyi , Zhu Yongqing , Ren Yixiao , Han Yunfei , Zhou Xi
Published: October 2024

Ordinal tabular data, with advantages of structured knowledge representation in tabular data and the characteristic of inter-class ranks, has drawn increasing attention. However, existing retrieval-based tabular deep learning methods designed primarily for classical tabular data pay less attention to ordinal tabular data. Ordinal knowledge of ordinal tabular data provides a more explicit objective for tabular ordinal classification by considering both classification and regression properties. Furthermore, these approaches overlook the significance of intra-class group features which can balance the retrieved probability of various sample size groups and capture shared knowledge among multiple samples within same group. In this work, we propose the Intra-Class Groups Retrieval and Inter-Class Ordinal Knowledge Augmented Network (TabCGOK) model for ordinal tabular data prediction, equipped with Intra-Class Groups Retrieval (CG) module and Inter-Class Ordinal Knowledge Augmented (OK) module. The CG module provides intra-class group features candidate set for subsequent retrieval operation. It divides each class into several groups, then extracts the representation of each group as intra-class group features. And the intra-class group features candidate set consists of all intra-class group features from each class. The OK module is designed to capture inter-class ordinal knowledge. It estimates the ordinal distances by calculating inter-class feature distances, which could correspond to the inter-class non-isometric nature of ordinal knowledge, and then aggregates the previous ordinal distances to clarify the containment relationship of ordinal knowledge. OK module utilizes the attention mechanism for fusing the captured ordinal knowledge to retrieved intra-class group features. Finally, TabCGOK integrates fused intra-class group features with sample level features for ordinal tabular data prediction. Extensive experiments on several ordinal tabular datasets demonstrate the effectiveness of our method. The source code is available at https://github.com/luozhengdong/TabCGOK.
Combining Active Learning and Learning to Reject for Anomaly Detection
Authors: Stradiotti Luca , Perini Lorenzo , Davis Jesse
Published: October 2024

Anomaly detection attempts to identify instances in the data that do not conform to the expected behavior. Because it is often difficult to label instances, the problem is tackled in an unsupervised way by employing data-driven heuristics to identify anomalies. However, the heuristics are imperfect which can degrade a detector’s performance. One way to mitigate this problem is using Active Learning to collect labels that help correct cases where the employed heuristics are incorrect. Alternatively, one can allow the detector to abstain (i.e., say “I do not know”) whenever it is likely to make mispredictions at test time, which is called Learning to Reject (LtR). However, while both have been studied in the context of anomaly detection, they have not been considered in conjunction. Although they both need labels to accomplish their task, integrating these two ideas is challenging for two reasons. First, their label selection strategies are intertwined but they acquire different types of labels. Second, it is unclear how to best divide the limited budget between labeling instances that help AL and those that help LtR. In this paper, we introduce SADAL, the first semi-supervised detector that allocates the label budget between AL and LtR by relying on a reward-based selection function. Experimentally on 25 datasets, we show that our approach outperforms several baselines by achieving a better performance.
Adversarial Attack for Explanation Robustness of Rationalization Models
Authors: Zhang Yuankai , Kong Lingxiao , Wang Haozhao , Li Ruixuan , Wang Jun , Li Yuhua , Liu Wei
Published: October 2024

Rationalization models, which select a subset of input text as rationale—crucial for humans to understand and trust predictions—have recently emerged as a prominent research area in eXplainable Artificial Intelligence (XAI). However, most of previous studies mainly focus on improving the quality of the rationale, ignoring its robustness to malicious attack. Specifically, whether the rationalization models can still generate high-quality rationale under the adversarial attack remains unknown. To explore this, this paper proposes UAT2E, which aims to undermine the explainability of rationalization models without altering their predictions, thereby eliciting distrust in these models from human users. UAT2E employs the gradient-based search on triggers and then inserts them into the original input to conduct both the non-target and target attack. Experimental results on five datasets reveal the vulnerability of rationalization models in terms of explanation, where they tend to select more meaningless tokens under attacks. Based on this, we make a series of recommendations for improving rationalization models in terms of explanation.
Winning the 2023 CityLearn Challenge: A Community-Based Hierarchical Energy Systems Coordination Algorithm
Authors: I. Garmendia Andoni , Morri Francesco , Cappart Quentin , Le Cadre Hélène
Published: October 2024

The effective management and control of building energy systems are crucial for reducing the energy consumption peak loads, CO2 emissions, and ensuring the stability of the power grid, while maintaining optimal comfort levels within buildings. The difficulty to accommodate this trade-off is amplified by dynamic environmental conditions and the need for scalable solutions that can adapt across various building types and geographic locations. Acknowledging the importance of this problem, NeurIPS conference hosted since 2020 the CityLearn control challenge to foster the design of innovative solutions in building energy management. Participants were tasked with developing strategies that not only enhance energy efficiency but also prioritize sustainability and occupant comfort. This paper introduces the Community-based Hierarchical Energy Systems Coordination Algorithm (CHESCA), the winning approach of the 2023 edition. We rely on a hierarchical approach adaptable to an arbitrary number of buildings, first optimizing building-level metrics individually, and later refining these through a central community-level controller to improve grid-related metrics. Compared to the other high-ranked competitors, our approach demonstrated fast inference capabilities like learning-based methods, while offering a better interpretability and a superior generalization capabilities with minimal data requirements. This paper details our approach, supported by comprehensive experimental results and ablation studies.
Representation Matters for Mastering Chess: Improved Feature Representation in AlphaZero Outperforms Switching to Transformers
Authors: Czech Johannes , Blüml Jannis , Kersting Kristian , Steingrimsson Hedinn
Published: October 2024

While transformers have gained recognition as a versatile tool for artificial intelligence (AI), an unexplored challenge arises in the context of chess — a classical AI benchmark. Here, incorporating Vision Transformers (ViTs) into AlphaZero is insufficient for chess mastery, mainly due to ViTs’ computational limitations. The attempt to optimize their efficiency by combining MobileNet and NextViT outperformed AlphaZero by about 30 Elo. However, we propose a practical improvement that involves a simple change in the input representation and value loss functions. As a result, we achieve a significant performance boost of up to 180 Elo points beyond what is currently achievable with AlphaZero in chess. In addition to these improvements, our experimental results using the Integrated Gradient technique confirm the effectiveness of the newly introduced features.
Global Optimisation of Black-Box Functions with Generative Models in the Wasserstein Space
Authors: Ramazyan Tigran , Hushchyn Mikhail , Derkach Denis
Published: October 2024

We propose a new uncertainty estimator for gradient-free optimisation of black-box simulators using deep generative surrogate models. Optimisation of these simulators is especially challenging for stochastic simulators and higher dimensions. To address these issues, we utilise a deep generative surrogate approach to model the black box response for the entire parameter space. We then leverage this knowledge to estimate the proposed uncertainty based on the Wasserstein distance - the Wasserstein uncertainty. This approach is employed in a posterior agnostic gradient-free optimisation algorithm that minimises regret over the entire parameter space. A series of tests were conducted to demonstrate that our method is more robust to the shape of both the black box function and the stochastic response of the black box than state-of-the-art methods, such as efficient global optimisation with a deep Gaussian process surrogate.
Learning Backdoors for Mixed Integer Linear Programs with Contrastive Learning
Authors: Cai Junyang , Huang Taoan , Dilkina Bistra
Published: October 2024

Many real-world problems can be efficiently modeled as Mixed Integer Linear Programs (MILPs) and solved with the Branch-and-Bound method. Prior work has shown the existence of MILP backdoors, small sets of variables such that prioritizing branching on them when possible leads to faster running times. However, finding high-quality backdoors that improve running times remains an open question. Previous work learns to estimate the relative solver speed of randomly sampled backdoors through ranking and learns to decide whether to use the highest-ranked backdoor candidate. In this paper, we utilize the Monte-Carlo tree search method to collect backdoors for training, rather than relying on random sampling, and adapt a contrastive learning framework to train a Graph Attention Network model to predict backdoors. Our method, evaluated on several common MILP problem domains, demonstrates performance improvements over both Gurobi and previous models.
Grouped Logit Distillation Enhanced with Superclass Awareness for Efficient Knowledge Transfer
Authors: Zhang Shuoxi , Liu Hanpeng , Wang Yuyi , He Kun
Published: October 2024

Knowledge distillation (KD) facilitates student training by transferring information beyond plain labels, specifically through the categorical relationships from the teacher. However, this class relationship knowledge is, by nature, easily dominated by a few classes. This phenomenon prevents knowledge distillation from fully extracting the knowledge of the teacher model, thereby impeding the transfer of knowledge. To this end, we introduce a grouping strategy to the knowledge distillation paradigm, termed Grouped Logit Distillation (GLD). This strategy involves distilling knowledge within each group and across all groups, potentially transferring relationships in a comprehensive manner. Furthermore, we delve deeper into the grouping mechanism and attempt to incorporate a superclass mechanism using information derived from features of the teacher model. Our enhanced version, GLD++, performs knowledge distillation more meticulously by organizing information based on superclasses. We evaluate the effectiveness of our approaches through extensive experiments across standard benchmark datasets, obtaining state-of-the-art performance.
One-Shot Collaborative Data Distillation
Authors: Holland William , Thapa Chandra , Shao Wei , Camtepe Seyit , Ali Siddiqui Sarah
Published: October 2024

Large machine-learning training datasets can be distilled into small collections of informative synthetic data samples. These synthetic sets support efficient model learning and reduce the communication cost of data sharing. Thus, high-fidelity distilled data can support the efficient deployment of machine learning applications in distributed network environments. A naive way to construct a synthetic set in a distributed environment is to allow each client to perform local data distillation and to merge local distillations at a central server. However, the quality of the resulting set is impaired by heterogeneity in the distributions of the local data held by clients. To overcome this challenge, we introduce the first collaborative data distillation technique, called CollabDM, which captures the global distribution of the data and requires only a single round of communication between client and server. Our method outperforms the state-of-the-art one-shot learning method on skewed data in distributed learning environments. We also show the promising practical benefits of our method when applied to attack detection in 5G networks.
Learning Compact Neural Networks via Generalized Structured Sparsity
Authors: Bian Ke , Sun Lu , Zhao Dengji
Published: October 2024

Deep neural networks have shown excellent performance in various domains, but the large number of parameters and computational inefficiency pose significant challenges in practice. Existing sparse learning methods, such as pruning and regularization, play a crucial role in reducing model size and improving generalization. However, they are limited to the single-level grouping structure and ignore the correlation between consecutive layers, leading to insufficient sparsity and performance degradation. To address these challenges, we propose a novel sparsity regularizer that promotes structured sparsity based on the multi-level grouping structure. It encourages inter-group cooperation and intra-group competition at the first-level, and promotes inter-group competition and intra-group cooperation at the second-level. The multi-level grouping nature can flexibly model the correlation between consecutive layers or convolutional kernels by carefully defining the groups based on specific neural architectures. Moreover, we introduce a more general form that unifies a family of convex and non-covnex sparse regularizers and prove its equivalence to multiplicative weight decomposition, which helps us develop a simple but efficient optimization algorithm. Extensive experiments on real-world datasets show that the proposed method can generate more compact and efficient models compared to cutting-edge methods.
Lightweight Transformer for sEMG Gesture Recognition with Feature Distilled Variational Information Bottleneck
Authors: Wang Zefeng , Hu Bingbing , Yao Junfeng , Su Jinsong
Published: October 2024

Gesture recognition based on surface electromyography (sEMG) has seen considerable improvements in performance across various tasks and metrics with the rapid development of deep learning. However, challenges still exist in current deep neural networks for sEMG recognition. For instance, convolutional neural networks exhibit poor capturing of global features, recurrent neural networks have limited parallel processing capabilities, and their hybrids are usually more complex. Additionally, recent networks based on Transformers rarely consider the locality of attention and noise resistance. To fully explore the essence of sEMG sequences, and to make the model more lightweight and robust while ensuring feature learning performance, in this paper, we propose the feature distilled variational information bottleneck (FDVIB). Specifically, this method leverages knowledge distillation to learn from a high-precision teacher model at levels of feature and prediction, significantly reducing parameters and computations, and simplifying the structure. It also uses VIB to enhance the model’s robustness. We construct a Transformer model using the proposed method and conduct a series of evaluations. Experimental results show that our classification accuracy is competitive with state-of-the-art and also demonstrate the effectiveness of our method in enhancing model lightness and robustness.
Unveiling the Power of Sparse Neural Networks for Feature Selection
Authors: Atashgahi Zahra , Liu Tennison , Pechenizkiy Mykola , Veldhuis Raymond , Mocanu Decebal Constantin , van der Schaar Mihaela
Published: October 2024

Sparse Neural Networks (SNNs) have emerged as powerful tools for efficient feature selection. Leveraging the dynamic sparse training (DST) algorithms within SNNs has demonstrated promising feature selection capabilities while drastically reducing computational overheads. Despite these advancements, several critical aspects remain insufficiently explored for feature selection. Questions persist regarding the choice of the DST algorithm for network training, the choice of metric for ranking features/neurons, and the comparative performance of these methods across diverse datasets when compared to dense networks. This paper addresses these gaps by presenting a comprehensive systematic analysis of feature selection with sparse neural networks. Moreover, we introduce a novel metric considering sparse neural network characteristics, which is designed to quantify feature importance within the context of SNNs. Our findings show that feature selection with SNNs trained with DST algorithms can achieve, on average, more than 50% memory and 55% FLOPs reduction compared to the dense networks, while outperforming them in terms of the quality of the selected features. Our code and the supplementary material are available on GitHub (https://github.com/zahraatashgahi/Neuron-Attribution).
Scalable Variational Causal Discovery Unconstrained by Acyclicity
Authors: Hoang Nu , Duong Bao , Nguyen Thin
Published: October 2024

Bayesian causal discovery offers the power to quantify epistemic uncertainties among a broad range of structurally diverse causal theories potentially explaining the data, represented in forms of directed acyclic graphs (DAGs). However, existing methods struggle with efficient DAG sampling due to the complex acyclicity constraint. In this study, we propose a scalable Bayesian approach to effectively learn the posterior distribution over causal graphs given observational data thanks to the ability to generate DAGs without explicitly enforcing acyclicity. Specifically, we introduce a novel differentiable DAG sampling method that can generate a valid acyclic causal graph by mapping an unconstrained distribution of implicit topological orders to a distribution over DAGs. Given this efficient DAG sampling scheme, we are able to model the posterior distribution over causal graphs using a simple variational distribution over a continuous domain, which can be learned via the variational inference framework. Extensive empirical experiments on both simulated and real datasets demonstrate the superior performance of the proposed model compared to several state-of-the-art baselines.
The Propensity for Density in Feed-Forward Models
Authors: Schoots Nandi , Jackson Alex , Kholmovia Ali , McBurney Peter , Shanahan Murray
Published: October 2024

Does the process of training a neural network to solve a task tend to use all of the available weights even when the task could be solved with fewer weights? To address this question we study the effects of pruning fully connected, convolutional and residual models while varying their widths. We find that the proportion of weights that can be pruned without degrading performance is largely invariant to model size. Increasing the width of a model has little effect on the density of the pruned model relative to the increase in absolute size of the pruned network. In particular, we find substantial prunability across a large range of model sizes, where our biggest model is 50 times as wide as our smallest model. We explore three hypotheses that could explain these findings. Source code: [29].
FedStale: leveraging Stale Updates in Federated Learning
Authors: Rodio Angelo , Neglia Giovanni
Published: October 2024

Federated learning algorithms, such as FedAvg, are negatively affected by data heterogeneity and partial client participation. To mitigate the latter problem, global variance reduction methods, like FedVARP, leverage stale model updates for non-participating clients. These methods are effective under homogeneous client participation. Yet, this paper shows that, when some clients participate much less than others, aggregating updates with different levels of staleness can detrimentally affect the training process. Motivated by this observation, we introduce FedStale, a novel algorithm that updates the global model in each round through a convex combination of “fresh” updates from participating clients and “stale” updates from non-participating ones. By adjusting the weight in the convex combination, FedStale interpolates between FedAvg, which only uses fresh updates, and FedVARP, which treats fresh and stale updates equally. Our analysis of FedStale convergence yields novel findings: i) it integrates and extends previous FedAvg and FedVARP analyses to heterogeneous client participation; ii) it underscores how the least participating client influences convergence error; iii) it provides practical guidelines to best exploit stale updates, showing that their usefulness diminishes as data heterogeneity decreases and participation heterogeneity increases. Extensive experiments featuring diverse levels of client data and participation heterogeneity not only confirm these findings but also show that FedStale outperforms both FedAvg and FedVARP in many settings.1 1 A preprint version of this paper, including supplementary material, is available.
Deep Generative Models for Subgraph Prediction
Authors: Mahmoudzadeh Erfaneh , Naddaf Parmis , Zahirnia Kiarash , Schulte Oliver
Published: October 2024

Graph Neural Networks (GNNs) are important across different domains, such as social network analysis and recommendation systems, due to their ability to model complex relational data. This paper introduces subgraph queries as a new task for deep graph learning. Unlike traditional graph prediction tasks that focus on individual components like link prediction or node classification, subgraph queries jointly predict the components of a target subgraph based on evidence that is represented by an observed subgraph. For instance, a subgraph query can predict a set of target links and/or node labels. To answer subgraph queries, we utilize a probabilistic deep Graph Generative Model. Specifically, we inductively train a Variational Graph Auto-Encoder (VGAE) model, augmented to represent a joint distribution over links, node features and labels. Bayesian optimization is used to tune a weighting for the relative importance of links, node features and labels in a specific domain. We describe a deterministic and a sampling-based inference method for estimating subgraph probabilities from the VGAE generative graph distribution, without retraining, in zero-shot fashion. For evaluation, we apply the inference methods on a range of subgraph queries on six benchmark datasets. We find that inference from a model achieves superior predictive performance, surpassing independent prediction baselines with improvements in AUC scores ranging from 0.06 to 0.2 points, depending on the dataset.