OpenRaiser/Intern-Atlas huggingface dataset:
Remembering Transformer for Continual Learning
Venue: arXiv cs.LG 2024
Year: 2024

Neural networks encounter the challenge of Catastrophic Forgetting (CF) in continual learning, where new task learning interferes with previously learned knowledge. Existing data fine-tuning and regularization methods necessitate task identity information during inference and cannot eliminate interference among different tasks, while soft parameter sharing approaches encounter the problem of an increasing model parameter size. To tackle these challenges, we propose the Remembering Transformer, inspired by the brain's Complementary Learning Systems (CLS). Remembering Transformer employs a mixture-of-adapters architecture and a generative model-based novelty detection mechanism in a pretrained Transformer to alleviate CF. Remembering Transformer dynamically routes task data to the most relevant adapter with enhanced parameter efficiency based on knowledge distillation. We conducted extensive experiments, including ablation studies on the novelty detection mechanism and model capacity of the mixture-of-adapters, in a broad range of class-incremental split tasks and permutation tasks. Our approach demonstrated SOTA performance surpassing the second-best method by 15.90% in the split tasks, reducing the memory footprint from 11.18M to 0.22M in the five splits CIFAR10 task.
Enhancing Policy Gradient with the Polyak Step-Size Adaption
Venue: arXiv cs.LG 2024
Year: 2024

Policy gradient is a widely utilized and foundational algorithm in the field of reinforcement learning (RL). Renowned for its convergence guarantees and stability compared to other RL algorithms, its practical application is often hindered by sensitivity to hyper-parameters, particularly the step-size. In this paper, we introduce the integration of the Polyak step-size in RL, which automatically adjusts the step-size without prior knowledge. To adapt this method to RL settings, we address several issues, including unknown f* in the Polyak step-size. Additionally, we showcase the performance of the Polyak step-size in RL through experiments, demonstrating faster convergence and the attainment of more stable policies.
Bayesian Federated Model Compression for Communication and Computation Efficiency
Venue: arXiv cs.LG 2024
Year: 2024

In this paper, we investigate Bayesian model compression in federated learning (FL) to construct sparse models that can achieve both communication and computation efficiencies. We propose a decentralized Turbo variational Bayesian inference (D-Turbo-VBI) FL framework where we firstly propose a hierarchical sparse prior to promote a clustered sparse structure in the weight matrix. Then, by carefully integrating message passing and VBI with a decentralized turbo framework, we propose the D-Turbo-VBI algorithm which can (i) reduce both upstream and downstream communication overhead during federated training, and (ii) reduce the computational complexity during local inference. Additionally, we establish the convergence property for thr proposed D-Turbo-VBI algorithm. Simulation results show the significant gain of our proposed algorithm over the baselines in reducing communication overhead during federated training and computational complexity of final model.
Impact of Training Instance Selection on Automated Algorithm Selection Models for Numerical Black-box Optimization
Venue: arXiv cs.NE 2024
Year: 2024

The recently proposed MA-BBOB function generator provides a way to create numerical black-box benchmark problems based on the well-established BBOB suite. Initial studies on this generator highlighted its ability to smoothly transition between the component functions, both from a low-level landscape feature perspective, as well as with regard to algorithm performance. This suggests that MA-BBOB-generated functions can be an ideal testbed for automated machine learning methods, such as automated algorithm selection (AAS). In this paper, we generate 11800 functions in dimensions $d=2$ and $d=5$, respectively, and analyze the potential gains from AAS by studying performance complementarity within a set of eight algorithms. We combine this performance data with exploratory landscape features to create an AAS pipeline that we use to investigate how to efficiently select training sets within this space. We show that simply using the BBOB component functions for training yields poor test performance, while the ranking between uniformly chosen and diversity-based training sets strongly depends on the distribution of the test set.
Multi-rater Prompting for Ambiguous Medical Image Segmentation
Venue: arXiv cs.CV 2024
Year: 2024

Multi-rater annotations commonly occur when medical images are independently annotated by multiple experts (raters). In this paper, we tackle two challenges arisen in multi-rater annotations for medical image segmentation (called ambiguous medical image segmentation): (1) How to train a deep learning model when a group of raters produces a set of diverse but plausible annotations, and (2) how to fine-tune the model efficiently when computation resources are not available for re-training the entire model on a different dataset domain. We propose a multi-rater prompt-based approach to address these two challenges altogether. Specifically, we introduce a series of rater-aware prompts that can be plugged into the U-Net model for uncertainty estimation to handle multi-annotation cases. During the prompt-based fine-tuning process, only 0.3% of learnable parameters are required to be updated comparing to training the entire model. Further, in order to integrate expert consensus and disagreement, we explore different multi-rater incorporation strategies and design a mix-training strategy for comprehensive insight learning. Extensive experiments verify the effectiveness of our new approach for ambiguous medical image segmentation on two public datasets while alleviating the heavy burden of model re-training.
Measuring Geographic Diversity of Foundation Models with a Natural Language--based Geo-guessing Experiment on GPT-4
Venue: arXiv cs.CY 2024
Year: 2024

Generative AI based on foundation models provides a first glimpse into the world represented by machines trained on vast amounts of multimodal data ingested by these models during training. If we consider the resulting models as knowledge bases in their own right, this may open up new avenues for understanding places through the lens of machines. In this work, we adopt this thinking and select GPT-4, a state-of-the-art representative in the family of multimodal large language models, to study its geographic diversity regarding how well geographic features are represented. Using DBpedia abstracts as a ground-truth corpus for probing, our natural language--based geo-guessing experiment shows that GPT-4 may currently encode insufficient knowledge about several geographic feature types on a global level. On a local level, we observe not only this insufficiency but also inter-regional disparities in GPT-4's geo-guessing performance on UNESCO World Heritage Sites that carry significance to both local and global populations, and the inter-regional disparities may become smaller as the geographic scale increases. Morever, whether assessing the geo-guessing performance on a global or local level, we find inter-model disparities in GPT-4's geo-guessing performance when comparing its unimodal and multimodal variants. We hope this work can initiate a discussion on geographic diversity as an ethical principle within the GIScience community in the face of global socio-technical challenges.
Audio Dialogues: Dialogues dataset for audio and music understanding
Venue: arXiv cs.CL 2024
Year: 2024

Existing datasets for audio understanding primarily focus on single-turn interactions (i.e. audio captioning, audio question answering) for describing audio in natural language, thus limiting understanding audio via interactive dialogue. To address this gap, we introduce Audio Dialogues: a multi-turn dialogue dataset containing 163.8k samples for general audio sounds and music. In addition to dialogues, Audio Dialogues also has question-answer pairs to understand and compare multiple input audios together. Audio Dialogues leverages a prompting-based approach and caption annotations from existing datasets to generate multi-turn dialogues using a Large Language Model (LLM). We evaluate existing audio-augmented large language models on our proposed dataset to demonstrate the complexity and applicability of Audio Dialogues. Our code for generating the dataset will be made publicly available. Detailed prompts and generated dialogues can be found on the demo website https://audiodialogues.github.io/.
Simba: Mamba augmented U-ShiftGCN for Skeletal Action Recognition in Videos
Venue: arXiv cs.CV 2024
Year: 2024

Skeleton Action Recognition (SAR) involves identifying human actions using skeletal joint coordinates and their interconnections. While plain Transformers have been attempted for this task, they still fall short compared to the current leading methods, which are rooted in Graph Convolutional Networks (GCNs) due to the absence of structural priors. Recently, a novel selective state space model, Mamba, has surfaced as a compelling alternative to the attention mechanism in Transformers, offering efficient modeling of long sequences. In this work, to the utmost extent of our awareness, we present the first SAR framework incorporating Mamba. Each fundamental block of our model adopts a novel U-ShiftGCN architecture with Mamba as its core component. The encoder segment of the U-ShiftGCN is devised to extract spatial features from the skeletal data using downsampling vanilla Shift S-GCN blocks. These spatial features then undergo intermediate temporal modeling facilitated by the Mamba block before progressing to the encoder section, which comprises vanilla upsampling Shift S-GCN blocks. Additionally, a Shift T-GCN (ShiftTCN) temporal modeling unit is employed before the exit of each fundamental block to refine temporal representations. This particular integration of downsampling spatial, intermediate temporal, upsampling spatial, and ultimate temporal subunits yields promising results for skeleton action recognition. We dub the resulting model \textbf{Simba}, which attains state-of-the-art performance across three well-known benchmark skeleton action recognition datasets: NTU RGB+D, NTU RGB+D 120, and Northwestern-UCLA. Interestingly, U-ShiftGCN (Simba without Intermediate Mamba Block) by itself is capable of performing reasonably well and surpasses our baseline.
On the role of ethics and sustainability in business innovation
Venue: arXiv cs.CY 2024
Year: 2024

For organizations to survive and flourish in the long term, innovation and novelty must be continually introduced, which is particularly true in today's rapidly changing world. This raises a variety of ethical and sustainability considerations that seldom receive the attention they deserve. Existing innovation adoption frameworks often focus on technological, organizational, environmental, and social factors impacting adoption. In this chapter, we explore the ethical and sustainability angles, particularly as they relate to emerging technologies, artificial intelligence (AI) being a prominent example. We consider how to facilitate the development and cultivation of innovation cultures in organizations, including budding startups as well as established enterprises, through approaches such as systems thinking.
Correctness of Flow Migration Across Network Function Instances
Venue: arXiv cs.NI 2024
Year: 2024

Network Functions (NFs) improve the safety and efficiency of networks. Flows traversing NFs may need to be migrated to balance load, conserve energy, etc. When NFs are stateful, the information stored on the NF per flow must be migrated before the flows are migrated, to avoid problems of consistency. We examine what it means to correctly migrate flows from a stateful NF instance. We define the property of Weak-O, where only the state information required for packets to be correctly forwarded is migrated first, while the remaining states are eventually migrated. Weak-O can be preserved without buffering or dropping packets, unlike existing algorithms. We propose an algorithm that preserves Weak-O and prove its correctness. Even though this may cause packet re-ordering, we experimentally demonstrate that the goodputs with and without migration are comparable when the old and new paths have the same delays and bandwidths, or when the new path has larger bandwidth or at most 5 times longer delays, thus making this practical, contrary to what was thought before. We also prove that no criterion stronger than Weak-O can be preserved in a flow migration system that requires no buffering or dropping of packets and eventually synchronizes its states.
Learning Hamiltonian Dynamics with Reproducing Kernel Hilbert Spaces and Random Features
Venue: arXiv cs.LG 2024
Year: 2024

A method for learning Hamiltonian dynamics from a limited and noisy dataset is proposed. The method learns a Hamiltonian vector field on a reproducing kernel Hilbert space (RKHS) of inherently Hamiltonian vector fields, and in particular, odd Hamiltonian vector fields. This is done with a symplectic kernel, and it is shown how the kernel can be modified to an odd symplectic kernel to impose the odd symmetry. A random feature approximation is developed for the proposed odd kernel to reduce the problem size. The performance of the method is validated in simulations for three Hamiltonian systems. It is demonstrated that the use of an odd symplectic kernel improves prediction accuracy and data efficiency, and that the learned vector fields are Hamiltonian and exhibit the imposed odd symmetry characteristics.
Point cloud obstacle detection with the map filtration
Venue: arXiv cs.RO 2024
Year: 2024

Obstacle detection is one of the basic tasks of a robot movement in an unknown environment. The use of a LiDAR (Light Detection And Ranging) sensor allows one to obtain a point cloud in the vicinity of the sensor. After processing this data, obstacles can be found and recorded on a map. For this task, I present a pipeline capable of detecting obstacles even on a computationally limited device. The pipeline was also tested on a real robot and qualitatively evaluated on a dataset, which was collected in Brno University of Technology lab. Time consumption was recorded and compared with 3D object detectors.
Monte Carlo Tree Search with Boltzmann Exploration
Venue: arXiv cs.AI 2024
Year: 2024

Monte-Carlo Tree Search (MCTS) methods, such as Upper Confidence Bound applied to Trees (UCT), are instrumental to automated planning techniques. However, UCT can be slow to explore an optimal action when it initially appears inferior to other actions. Maximum ENtropy Tree-Search (MENTS) incorporates the maximum entropy principle into an MCTS approach, utilising Boltzmann policies to sample actions, naturally encouraging more exploration. In this paper, we highlight a major limitation of MENTS: optimal actions for the maximum entropy objective do not necessarily correspond to optimal actions for the original objective. We introduce two algorithms, Boltzmann Tree Search (BTS) and Decaying ENtropy Tree-Search (DENTS), that address these limitations and preserve the benefits of Boltzmann policies, such as allowing actions to be sampled faster by using the Alias method. Our empirical analysis shows that our algorithms show consistent high performance across several benchmark domains, including the game of Go.
Mitigating Vulnerable Road Users Occlusion Risk Via Collective Perception: An Empirical Analysis
Venue: arXiv cs.NI 2024
Year: 2024

Recent reports from the World Health Organization highlight that Vulnerable Road Users (VRUs) have been involved in over half of the road fatalities in recent years, with occlusion risk - a scenario where VRUs are hidden from drivers' view by obstacles like parked vehicles - being a critical contributing factor. To address this, we present a novel algorithm that quantifies occlusion risk based on the dynamics of both vehicles and VRUs. This algorithm has undergone testing and evaluation using a real-world dataset from German intersections. Additionally, we introduce the concept of Maximum Tracking Loss (MTL), which measures the longest consecutive duration a VRU remains untracked by any vehicle in a given scenario. Our study extends to examining the role of the Collective Perception Service (CPS) in VRU safety. CPS enhances safety by enabling vehicles to share sensor information, thereby potentially reducing occlusion risks. Our analysis reveals that a 25% market penetration of CPS-equipped vehicles can substantially diminish occlusion risks and significantly curtail MTL. These findings demonstrate how various scenarios pose different levels of risk to VRUs and how the deployment of Collective Perception can markedly improve their safety. Furthermore, they underline the efficacy of our proposed metrics to capture occlusion risk as a safety factor.
Joint Conditional Diffusion Model for Image Restoration with Mixed Degradations
Venue: arXiv cs.CV 2024
Year: 2024

Image restoration is rather challenging in adverse weather conditions, especially when multiple degradations occur simultaneously. Blind image decomposition was proposed to tackle this issue, however, its effectiveness heavily relies on the accurate estimation of each component. Although diffusion-based models exhibit strong generative abilities in image restoration tasks, they may generate irrelevant contents when the degraded images are severely corrupted. To address these issues, we leverage physical constraints to guide the whole restoration process, where a mixed degradation model based on atmosphere scattering model is constructed. Then we formulate our Joint Conditional Diffusion Model (JCDM) by incorporating the degraded image and degradation mask to provide precise guidance. To achieve better color and detail recovery results, we further integrate a refinement network to reconstruct the restored image, where Uncertainty Estimation Block (UEB) is employed to enhance the features. Extensive experiments performed on both multi-weather and weather-specific datasets demonstrate the superiority of our method over state-of-the-art competing methods.
From the Lab to the Theater: An Unconventional Field Robotics Journey
Venue: arXiv cs.RO 2024
Year: 2024

Artistic performances involving robotic systems present unique technical challenges akin to those encountered in other field deployments. In this paper, we delve into the orchestration of robotic artistic performances, focusing on the complexities inherent in communication protocols and localization methods. Through our case studies and experimental insights, we demonstrate the breadth of technical requirements for this type of deployment, and, most importantly, the significant contributions of working closely with non-experts.
The Effectiveness of a Simplified Model Structure for Crowd Counting
Venue: arXiv cs.CV 2024
Year: 2024

In the field of crowd counting research, many recent deep learning based methods have demonstrated robust capabilities for accurately estimating crowd sizes. However, the enhancement in their performance often arises from an increase in the complexity of the model structure. This paper discusses how to construct high-performance crowd counting models using only simple structures. We proposes the Fuss-Free Network (FFNet) that is characterized by its simple and efficieny structure, consisting of only a backbone network and a multi-scale feature fusion structure. The multi-scale feature fusion structure is a simple structure consisting of three branches, each only equipped with a focus transition module, and combines the features from these branches through the concatenation operation. Our proposed crowd counting model is trained and evaluated on four widely used public datasets, and it achieves accuracy that is comparable to that of existing complex models. Furthermore, we conduct a comprehensive evaluation by replacing the existing backbones of various models such as FFNet and CCTrans with different networks, including MobileNet-v3, ConvNeXt-Tiny, and Swin-Transformer-Small. The experimental results further indicate that excellent crowd counting performance can be achieved with the simplied structure proposed by us.
Resolve Domain Conflicts for Generalizable Remote Physiological Measurement
Venue: arXiv cs.CV 2024
Year: 2024

Remote photoplethysmography (rPPG) technology has become increasingly popular due to its non-invasive monitoring of various physiological indicators, making it widely applicable in multimedia interaction, healthcare, and emotion analysis. Existing rPPG methods utilize multiple datasets for training to enhance the generalizability of models. However, they often overlook the underlying conflict issues across different datasets, such as (1) label conflict resulting from different phase delays between physiological signal labels and face videos at the instance level, and (2) attribute conflict stemming from distribution shifts caused by head movements, illumination changes, skin types, etc. To address this, we introduce the DOmain-HArmonious framework (DOHA). Specifically, we first propose a harmonious phase strategy to eliminate uncertain phase delays and preserve the temporal variation of physiological signals. Next, we design a harmonious hyperplane optimization that reduces irrelevant attribute shifts and encourages the model's optimization towards a global solution that fits more valid scenarios. Our experiments demonstrate that DOHA significantly improves the performance of existing methods under multiple protocols. Our code is available at https://github.com/SWY666/rPPG-DOHA.
Konnektor: Connection Protocol for Ensuring Peer Uniqueness in Decentralized P2P Networks
Venue: arXiv cs.NI 2024
Year: 2024

Konnektor is a connection protocol designed to solve the challenge of managing unique peers within distributed peer-to-peer networks. By prioritizing network integrity and efficiency, Konnektor offers a comprehensive solution that safeguards against the spread of duplicate peers while optimizing resource utilization. This paper provides a detailed explanation of the protocol's key components, including peer addressing, connection initialization, detecting peer duplications and mitigation strategies against potential security threats.
Auditing health-related recommendations in social media: A Case Study of Abortion on YouTube
Venue: arXiv cs.SI 2024
Year: 2024

Recommendation algorithms (RS) used by social media, like YouTube, significantly shape our information consumption across various domains, especially in healthcare. Hence, algorithmic auditing becomes crucial to uncover their potential bias and misinformation, particularly in the context of controversial topics like abortion. We introduce a simple yet effective sock puppet auditing approach to investigate how YouTube recommends abortion-related videos to individuals with different backgrounds. Our framework allows for efficient auditing of RS, regardless of the complexity of the underlying algorithms