GenAI4ELab/papercli-papers huggingface dataset:
Finding good policies in average-reward Markov Decision Processes without prior knowledge
Authors: Adrienne Tuynman; Rémy Degenne; Emilie Kaufmann
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: We revisit the identification of an $\varepsilon$-optimal policy in average-reward Markov Decision Processes (MDP). In such MDPs, two measures of complexity have appeared in the literature: the diameter, $D$, and the optimal bias span, $H$, which satisfy $H\leq D$.
  Prior work have studied the complexity of $\varepsilon$-optimal policy identification only when a generative model is available. In this case, it is known that there exists an MDP with $D \simeq H$ for which the sample complexity to output an $\varepsilon$-optimal policy is $\Omega(SAD/\varepsilon^2)$ where $S$ and $A$ are the sizes of the state and action spaces. Recently, an algorithm with a sample complexity of order $SAH/\varepsilon^2$ has been proposed, but it requires the knowledge of $H$. We first show that the sample complexity required to estimate $H$ is not bounded by any function of $S,A$ and $H$, ruling out the possibility to easily make the previous algorithm agnostic to $H$. By relying instead on a diameter estimation procedure, we propose the first algorithm for $(\varepsilon,\delta)$-PAC policy identification that does not need any form of prior knowledge on the MDP. Its sample complexity scales in $SAD/\varepsilon^2$ in the regime of small $\varepsilon$, which is near-optimal. In the online setting, our first contribution is a lower bound which implies that a sample complexity polynomial in $H$ cannot be achieved in this setting. Then, we propose an online algorithm with a sample complexity in $SAD^2/\varepsilon^2$, as well as a novel approach based on a data-dependent stopping rule that we believe is promising to further reduce this bound.
Optimistic Verifiable Training by Controlling Hardware Nondeterminism
Authors: Megha Srivastava; Simran Arora; Dan Boneh
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: The increasing compute demands of AI systems has led to the emergence of services that train models on behalf of clients lacking necessary resources. However, ensuring correctness of training and guarding against potential training-time attacks, such as data poisoning and backdoors, poses challenges. Existing works on verifiable training largely fall into two classes: proof-based systems, which can be difficult to scale, and ``optimistic'' methods that consider a trusted third-party auditor who replicates the training process. A key challenge with the latter is that hardware nondeterminism between GPU types during training prevents an auditor from replicating the training process exactly, and such schemes are therefore non-robust. We propose a method that combines training in a higher precision than the target model, rounding after intermediate computation steps, and storing rounding decisions based on an adaptive thresholding procedure, to successfully control for  nondeterminism. Across three different NVIDIA GPUs (A40, Titan XP, RTX 2080 Ti), we achieve exact training replication at FP32 precision for both full-training and fine-tuning of ResNet-50 (23M) and GPT-2 (117M) models. Our  verifiable training scheme significantly decreases the storage and time costs compared to proof-based systems.
Mimicking To Dominate: Imitation Learning Strategies for Success in Multiagent Games
Authors: The Viet Bui; Tien Anh Mai; Thanh Hong Nguyen
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: Training agents in multi-agent games presents significant challenges due to their intricate nature. These challenges are exacerbated by dynamics influenced not only by the environment but also by strategies of opponents. Existing methods often struggle with slow convergence and instability.
To address these challenges, we harness the potential of imitation learning (IL) to comprehend and anticipate actions of the opponents, aiming to mitigate uncertainties with respect to the game dynamics.
Our key contributions include:
(i) a new multi-agent IL model for predicting next moves of the opponents - our model works with hidden actions of opponents and local observations;
(ii) a new multi-agent reinforcement learning (MARL) algorithm that combines our IL model and policy training into one single training process;
and (iii) extensive experiments in three challenging game environments, including an advanced version of the Star-Craft multi-agent challenge (i.e., SMACv2).
Experimental results show that our approach achieves superior performance compared to state-of-the-art MARL algorithms.
ReplaceAnything3D: Text-Guided Object Replacement in 3D Scenes with Compositional Scene Representations
Authors: Edward Bartrum; Thu Nguyen-Phuoc; Chris Xie; Zhengqin Li; Numair Khan; Armen Avetisyan; Douglas Lanman; Lei Xiao
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: We introduce ReplaceAnything3D model RAM3D, a novel method for 3D object replacement in 3D scenes based on users' text description. Given multi-view images of a scene, a text prompt describing the object to replace, and another describing the new object, our Erase-and-Replace approach can effectively swap objects in 3D scenes with newly generated content while maintaining 3D consistency across multiple viewpoints. We demonstrate the versatility of RAM3D by applying it to various realistic 3D scene types, showcasing results of modified objects that blend in seamlessly with the scene without impacting its overall integrity.
Neural Flow Diffusion Models: Learnable Forward Process for Improved Diffusion Modelling
Authors: Grigory Bartosh; Dmitry Vetrov; Christian A. Naesseth
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: Conventional diffusion models typically relies on a fixed forward process, which implicitly defines complex marginal distributions over latent variables. This can often complicate the reverse process’ task in learning generative trajectories, and results in costly inference for diffusion models. To address these limitations, we introduce Neural Flow Diffusion Models (NFDM), a novel framework that enhances diffusion models by supporting a broader range of forward processes beyond the standard Gaussian. We also propose a novel parameterization technique for learning the forward process. Our framework provides an end-to-end, simulation-free optimization objective, effectively minimizing a variational upper bound on the negative log-likelihood. Experimental results demonstrate NFDM’s strong performance, evidenced by state-of-the-art likelihood estimation. Furthermore, we investigate NFDM’s capacity for learning generative dynamics with specific characteristics, such as deterministic straight lines trajectories, and demonstrate how the framework may be adopted for learning bridges between two distributions. The results underscores NFDM’s versatility and its potential for a wide range of applications.
Resource-Aware Federated Self-Supervised Learning with Global Class Representations
Authors: Mingyi Li; Xiao Zhang; Qi Wang; Tengfei LIU; Ruofan Wu; Weiqiang Wang; Fuzhen Zhuang; Hui Xiong; Dongxiao Yu
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: Due to the heterogeneous architectures and class skew, the global representation models training in resource-adaptive federated self-supervised learning face with tricky challenges: $\textit{deviated representation abilities}$ and $\textit{inconsistent representation spaces}$. 
In this work, we are the first to propose a multi-teacher knowledge distillation framework, namely $\textit{FedMKD}$, to learn global representations with whole class knowledge from heterogeneous clients even under extreme class skew. Firstly, the adaptive knowledge integration mechanism is designed to learn better representations from all heterogeneous models with deviated representation abilities. Then the weighted combination of the self-supervised loss and the distillation loss can support the global model to encode all classes from clients into a unified space. Besides, the global knowledge anchored alignment module can make the local representation spaces close to the global spaces, which further improves the representation abilities of local ones. Finally, extensive experiments conducted on two datasets demonstrate the effectiveness of $\textit{FedMKD}$ which outperforms state-of-the-art baselines 4.78\% under linear evaluation on average.
Consensus Learning with Deep Sets for Essential Matrix Estimation
Authors: Dror Moran; Yuval Margalit; Guy Trostianetsky; Fadi Khatib; Meirav Galun; Ronen Basri
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: Robust estimation of the essential matrix, which encodes the relative position and orientation of two cameras, is a fundamental step in structure from motion pipelines. Recent deep-based methods achieved accurate estimation by using complex network architectures that involve graphs, attention layers, and hard pruning steps. Here, we propose a simpler network architecture based on Deep Sets. Given a collection of point matches extracted from two images, our method identifies outlier point matches and models the displacement noise in inlier matches. A weighted DLT module uses these predictions to regress the essential matrix. Our network achieves accurate recovery that is superior to existing networks with significantly more complex architectures.
Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs
Authors: Abhimanyu Hans; John Kirchenbauer; Yuxin Wen; Neel Jain; Hamid Kazemi; Prajwal Singhania; Siddharth Singh; Gowthami Somepalli; Jonas Geiping; Abhinav Bhatele; Tom Goldstein
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: Large language models can memorize and repeat their training data, causing privacy and copyright risks. To mitigate memorization, we introduce a subtle modification to the next-token training objective that we call the goldfish loss. During training, a randomly sampled subsets of tokens are excluded from the loss computation. These dropped tokens are not memorized by the model, which prevents verbatim reproduction of a complete chain of tokens from the training set. We run extensive experiments training billion-scale LLaMA-2 models, both pre-trained and trained from scratch, and demonstrate significant reductions in extractable memorization with little to no impact on downstream benchmarks.

_Code and checkpoints: https://github.com/ahans30/goldfish-loss_
Self-Retrieval: End-to-End Information Retrieval with One Large Language Model
Authors: Qiaoyu Tang; Jiawei Chen; Zhuoqun Li; Bowen Yu; Yaojie Lu; ChengFu; Haiyang Yu; Hongyu Lin; Fei Huang; Ben He; Xianpei Han; Le Sun; Yongbin Li
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: The rise of large language models (LLMs) has significantly transformed both the construction and application of information retrieval (IR) systems. 
However, current interactions between IR systems and LLMs remain limited, with LLMs merely serving as part of components within IR systems, and IR systems being constructed independently of LLMs. This separated architecture restricts knowledge sharing and deep collaboration between them.
In this paper, we introduce Self-Retrieval, a novel end-to-end LLM-driven information retrieval architecture.
Self-Retrieval unifies all essential IR functions within a single LLM, leveraging the inherent capabilities of LLMs throughout the IR process.
Specifically, Self-Retrieval internalizes the retrieval corpus through self-supervised learning,  transforms the retrieval process into sequential passage generation, and performs relevance assessment for reranking.
Experimental results demonstrate that Self-Retrieval not only outperforms existing retrieval approaches by a significant margin, but also substantially enhances the performance of LLM-driven downstream applications like retrieval-augmented generation.
Adam on Local Time: Addressing Nonstationarity in RL with Relative Adam Timesteps
Authors: Benjamin Ellis; Matthew Thomas Jackson; Andrei Lupu; Alexander D. Goldie; Mattie Fellows; Shimon Whiteson; Jakob Nicolaus Foerster
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: In reinforcement learning (RL), it is common to apply techniques used broadly in 
machine learning such as neural network function approximators and momentum-based optimizers. However, such tools were largely developed for supervised learning rather than nonstationary RL, leading practitioners to adopt target networks, clipped policy updates, and other RL-specific implementation tricks to combat this mismatch, rather than directly adapting this toolchain for use in RL. 
In this paper, we take a different approach and instead address the effect of nonstationarity by adapting the widely used Adam optimiser. 
We first analyse the impact of nonstationary gradient magnitude --- such as that caused by a change in target network --- on Adam's update size, demonstrating that such a change can lead to large updates and hence sub-optimal performance.
To address this, we introduce Adam-Rel.
Rather than using the global timestep in the Adam update, Adam-Rel uses the *local* timestep within an epoch, essentially resetting Adam's timestep to 0 after target changes.
We demonstrate that this avoids large updates and reduces to learning rate annealing in the absence of such increases in gradient magnitude. Evaluating Adam-Rel in both on-policy and off-policy RL, we demonstrate improved performance in both Atari and Craftax.
We then show that increases in gradient norm occur in RL in practice, and examine the differences between our 
theoretical model and the observed data.
Inverse Factorized Soft Q-Learning for Cooperative Multi-agent Imitation Learning
Authors: The Viet Bui; Tien Anh Mai; Thanh Hong Nguyen
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: This paper concerns imitation learning (IL) in cooperative multi-agent systems.
The learning problem under consideration poses several challenges, characterized by high-dimensional state and action spaces and intricate inter-agent dependencies. In a single-agent setting, IL was shown to be done efficiently via an inverse soft-Q learning process. However, extending this framework to a multi-agent context introduces the need to simultaneously learn both local value functions to capture local observations and individual actions, and a joint value function for exploiting centralized learning.
In this work, we introduce a new multi-agent IL algorithm designed to address these challenges. Our approach enables the
centralized learning by leveraging mixing networks to aggregate  decentralized Q functions.
We further establish conditions for the mixing networks under which the multi-agent IL objective function exhibits convexity within the Q function space.
We present  extensive experiments conducted on some challenging multi-agent game environments, including an advanced version of the Star-Craft multi-agent challenge (SMACv2), which demonstrates the effectiveness of our algorithm.
Bridging semantics and pragmatics in information-theoretic emergent communication
Authors: Eleonora Gualdoni; Mycal Tucker; Roger P. Levy; Noga Zaslavsky
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: Human languages support both semantic categorization and local pragmatic interactions that require context-sensitive reasoning about meaning. While semantics and pragmatics are two fundamental aspects of language, they are typically studied independently and their co-evolution is largely under-explored. Here, we aim to bridge this gap by studying how a shared lexicon may emerge from local pragmatic interactions. To this end, we extend a recent information-theoretic framework for emergent communication in artificial agents, which integrates utility maximization, associated with pragmatics, with general communicative constraints that are believed to shape human semantic systems. Specifically, we show how to adapt this framework to train agents via unsupervised pragmatic interactions, and then evaluate their emergent lexical semantics. We test this approach in a rich visual domain of naturalistic images, and find that key human-like properties of the lexicon emerge when agents are guided by both context-specific utility and general communicative pressures, suggesting that both aspects are crucial for understanding how language may evolve in humans and in artificial agents.
Frequency-aware Generative Models for Multivariate Time Series Imputation
Authors: Xinyu Yang; Yu Sun; Xiaojie Yuan; Xinyang Chen
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: Missing data in multivariate time series are common issues that can affect the analysis and downstream applications.
Although multivariate time series data generally consist of the trend, seasonal and residual terms, existing works mainly focus on optimizing the modeling for the first two items. However, we find that the residual term is more crucial for getting accurate fillings, since it is more related to the diverse changes of data and the biggest component of imputation errors.
Therefore, in this study, we introduce frequency-domain information and design Frequency-aware Generative Models for Multivariate Time Series Imputation (FGTI). Specifically, FGTI employs a high-frequency filter to boost the residual term imputation, supplemented by a dominant-frequency filter for the trend and seasonal imputation. Cross-domain representation learning module then fuses frequency-domain insights with deep representations.
Experiments over various datasets with real-world missing values show that FGTI achieves superiority in both data imputation and downstream applications.
Graph Neural Networks and Arithmetic Circuits
Authors: Timon Barlag; Vivian Holzapfel; Laura Strieker; Jonni Virtema; Heribert Vollmer
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: We characterize the computational power of neural networks that follow the graph neural network (GNN) architecture, not restricted to aggregate-combine GNNs or other particular types. We establish an exact correspondence between the expressivity of GNNs using diverse activation functions and arithmetic circuits over real numbers. In our results the activation function of the network becomes a gate type in the circuit. Our result holds for families of constant depth circuits and networks, both uniformly and non-uniformly, for all common activation functions.
Dynamic Neural Regeneration: Enhancing Deep Learning Generalization on Small Datasets
Authors: Vijaya Raghavan T Ramkumar; Elahe Arani; Bahram Zonooz
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: The efficacy of deep learning techniques is contingent upon access to large volumes of data (labeled or unlabeled). However, in practical domains such as medical applications, data availability is often limited. This presents a significant challenge: How can we effectively train deep neural networks on relatively small datasets while improving generalization? Recent works have explored evolutionary or iterative training paradigms, which reinitialize a subset of parameters to enhance generalization performance for small datasets. However, these methods typically rely on randomly selected parameter subsets and maintain fixed masks throughout training, potentially leading to suboptimal outcomes. Inspired by neurogenesis in the brain, we propose a novel iterative training framework, Dynamic Neural Regeneration (DNR), that employs a data-aware dynamic masking scheme to eliminate redundant connections by estimating their significance. This approach increases the model's capacity for further learning through random weight reinitialization. Experimental results demonstrate that our approach outperforms existing methods in accuracy and robustness, highlighting its potential for real-world applications where data collection is challenging.
ROIDICE: Offline Return on Investment Maximization for Efficient Decision Making
Authors: Woosung Kim; Hayeong Lee; Jongmin Lee; Byung-Jun Lee
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: In this paper, we propose a novel policy optimization framework that maximizes Return on Investment (ROI) of a policy using a fixed dataset within a Markov Decision Process (MDP) equipped with a cost function. ROI, defined as the ratio between the return and the accumulated cost of a policy, serves as a measure of efficiency of the policy. Despite the importance of maximizing ROI in various applications, it remains a challenging problem due to its nature as a ratio of two long-term values: return and accumulated cost. To address this, we formulate the ROI maximizing reinforcement learning problem as a linear fractional programming. We then incorporate the stationary distribution correction (DICE) framework to develop a practical offline ROI maximization algorithm.
Our proposed algorithm, ROIDICE, yields an efficient policy that offers a superior trade-off between return and accumulated cost compared to policies trained using existing frameworks.
Prediction with Action: Visual Policy Learning via Joint Denoising Process
Authors: Yanjiang Guo; Yucheng Hu; Jianke Zhang; Yen-Jen Wang; Xiaoyu Chen; Chaochao Lu; Jianyu Chen
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: Diffusion models have demonstrated remarkable capabilities in image generation tasks, including image editing and video creation, representing a good understanding of the physical world. On the other line, diffusion models have also shown promise in robotic control tasks by denoising actions, known as diffusion policy. Although the diffusion generative model and diffusion policy exhibit distinct capabilities—image prediction and robotic action, respectively—they technically follow similar denoising process. In robotic tasks, the ability to predict future images and generate actions is highly correlated since they share the same underlying dynamics of the physical world. Building on this insight, we introduce \textbf{PAD}, a novel visual policy learning framework that unifies image \textbf{P}rediction and robot \textbf{A}ction within a joint \textbf{D}enoising process.  Specifically, PAD utilizes Diffusion Transformers (DiT) to seamlessly integrate images and robot states, enabling the simultaneous prediction of future images and robot actions. Additionally, PAD supports co-training on both robotic demonstrations and large-scale video datasets and can be easily extended to other robotic modalities, such as depth images. 
PAD outperforms previous methods, achieving a significant 38.9\% relative improvement on the full Metaworld benchmark, by utilizing a single text-conditioned visual policy within a data-efficient imitation learning setting. Furthermore, PAD demonstrates superior generalization to unseen tasks in real-world robot manipulation settings with 28.0\% success rate increase compared to the strongest baseline. 
Videos of PAD can be found at https://sites.google.com/view/pad-paper
Erasing Undesirable Concepts in Diffusion Models with Adversarial Preservation
Authors: Anh Tuan Bui; Long Tung Vuong; Khanh Doan; Trung Le; Paul Montague; Tamas Abraham; Dinh Phung
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: Diffusion models excel at generating visually striking content from text but can inadvertently produce undesirable or harmful content when trained on unfiltered internet data. A practical solution is to selectively removing target concepts from the model, but this may impact the remaining concepts. Prior approaches have tried to balance this by introducing a loss term to preserve neutral content or a regularization term to minimize changes in the model parameters, yet resolving this trade-off remains challenging. In this work, we propose to identify and preserving concepts most affected by parameter changes, termed as *adversarial concepts*. This approach ensures stable erasure with minimal impact on the other concepts. We demonstrate the effectiveness of our method using the Stable Diffusion model, showing that it outperforms state-of-the-art erasure methods in eliminating unwanted content while maintaining the integrity of other unrelated elements. Our code is available at \url{https://github.com/tuananhbui89/Erasing-Adversarial-Preservation}.