GenAI4ELab/papercli-papers huggingface dataset:
Animal-Bench: Benchmarking Multimodal Video Models for Animal-centric Video Understanding
Authors: Yinuo Jing; Ruxu Zhang; Kongming Liang; Yongxiang Li; Zhongjiang He; Zhanyu Ma; Jun Guo
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: With the emergence of large pre-trained multimodal video models, multiple benchmarks have been proposed to evaluate model capabilities. However, most of the benchmarks are human-centric, with evaluation data and tasks centered around human applications. Animals are an integral part of the natural world, and animal-centric video understanding is crucial for animal welfare and conservation efforts. Yet, existing benchmarks overlook evaluations focused on animals, limiting the application of the models. To address this limitation, our work established an animal-centric benchmark, namely Animal-Bench, to allow for a comprehensive evaluation of model capabilities in real-world contexts, overcoming agent-bias in previous benchmarks. Animal-Bench includes 13 tasks encompassing both common tasks shared with humans and special tasks relevant to animal conservation, spanning 7 major animal categories and 819 species, comprising a total of 41,839 data entries. To generate this benchmark, we defined a task system centered on animals and proposed an automated pipeline for animal-centric data processing. To further validate the robustness of models against real-world challenges, we utilized a video editing approach to simulate realistic scenarios like weather changes and shooting parameters due to animal movements. We evaluated 8 current multimodal video models on our benchmark and found considerable room for improvement. We hope our work provides insights for the community and opens up new avenues for research in multimodal video models. Our data and code will be released at https://github.com/PRIS-CV/Animal-Bench.
Achieving Linear Convergence with Parameter-Free Algorithms in Decentralized Optimization
Authors: Ilya Kuruzov; Gesualdo Scutari; Alexander Gasnikov
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: This paper addresses the minimization of the sum of strongly convex, smooth
functions over a network of agents without a centralized server. Existing decentralized algorithms require knowledge of functions and network parameters, such as the Lipschitz constant of the global gradient and/or network connectivity, for
hyperparameter tuning. Agents usually cannot access this information, leading
to conservative selections and slow convergence or divergence. This paper introduces a decentralized algorithm that eliminates the need for specific parameter
tuning. Our approach employs an operator splitting technique with a novel variable
metric, enabling a local backtracking line-search to adaptively select the stepsize
without global information or extensive communications. This results in favorable
convergence guarantees and dependence on optimization and network parameters
compared to existing nonadaptive methods. Notably, our method is the first adaptive decentralized algorithm that achieves linear convergence for strongly convex,
smooth objectives. Preliminary numerical experiments support our theoretical
findings, demonstrating superior performance in convergence speed and scalability.
HORSE: Hierarchical Representation for Large-Scale Neural Subset Selection
Authors: Binghui Xie; Yixuan Wang; Yongqiang Chen; Kaiwen Zhou; Yu Li; Wei Meng; James Cheng
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: Subset selection tasks, such as anomaly detection and compound selection in AI-assisted drug discovery, are crucial for a wide range of applications. Learning subset-valued functions with neural networks has achieved great success by incorporating permutation invariance symmetry into the architecture. However, existing neural set architectures often struggle to either capture comprehensive information from the superset or address complex interactions within the input. Additionally, they often fail to perform in scenarios where superset sizes surpass available memory capacity. To address these challenges, we introduce the novel concept of the Identity Property, which requires models to integrate information from the originating set, resulting in the development of neural networks that excel at performing effective subset selection from large supersets. Moreover, we present the Hierarchical Representation of Neural Subset Selection (HORSE), an attention-based method that learns complex interactions and retains information from both the input set and the optimal subset supervision signal. Specifically, HORSE enables the partitioning of the input ground set into manageable chunks that can be processed independently and then aggregated, ensuring consistent outcomes across different partitions. Through extensive experimentation, we demonstrate that HORSE significantly enhances neural subset selection performance by capturing more complex information and surpasses state-of-the-art methods in handling large-scale inputs by a margin of up to 20%.
Customized Subgraph Selection and Encoding for Drug-drug Interaction Prediction
Authors: Haotong Du; Quanming Yao; Juzheng Zhang; Yang Liu; Zhen Wang
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: Subgraph-based methods have proven to be effective and interpretable in predicting drug-drug interactions (DDIs),
which are essential for medical practice and drug development. 
Subgraph selection and encoding are critical stages in these methods, 
yet customizing these components remains underexplored due to the high cost of manual adjustments. 
In this study, 
inspired by the success of neural architecture search (NAS), 
we propose a method to search for data-specific components within subgraph-based frameworks. 
Specifically, 
we introduce extensive subgraph selection and encoding spaces that account for the diverse contexts of drug interactions in DDI prediction. 
To address the challenge of large search spaces and high sampling costs, 
we design a relaxation mechanism that uses an approximation strategy to efficiently explore optimal subgraph configurations. This approach allows for robust exploration of the search space. 
Extensive experiments demonstrate the effectiveness and superiority of the proposed method, 
with the discovered subgraphs and encoding functions highlighting the model’s adaptability.
On conditional diffusion models for PDE simulations
Authors: Aliaksandra Shysheya; Cristiana Diaconu; Federico Bergamin; Paris Perdikaris; José Miguel Hernández-Lobato; Richard E. Turner; Emile Mathieu
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: Modelling partial differential equations (PDEs) is of crucial importance in science and engineering, and it includes tasks ranging from forecasting to inverse problems, such as data assimilation. However, most previous numerical and machine learning approaches that target forecasting cannot be applied out-of-the-box for data assimilation. Recently, diffusion models have emerged as a powerful tool for conditional generation, being able to flexibly incorporate observations without retraining. In this work, we perform a comparative study of score-based diffusion models for forecasting and assimilation of sparse observations.  In particular, we focus on diffusion models that are either trained in a conditional manner, or conditioned after unconditional training. We address the shortcomings of existing models by proposing 1) an autoregressive sampling approach, that significantly improves performance in forecasting, 2) a new training strategy for conditional score-based models that achieves stable performance over a range of history lengths, and 3) a hybrid model which employs flexible pre-training conditioning on initial conditions and flexible post-training conditioning to handle data assimilation. We empirically show that these modifications are crucial for successfully tackling the combination of forecasting and data assimilation, a task commonly encountered in real-world scenarios.
Dual Defense: Enhancing Privacy and Mitigating Poisoning Attacks in Federated Learning
Authors: Runhua Xu; Shiqi Gao; Chao Li; James Joshi; Jianxin Li
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: Federated learning (FL) is inherently susceptible to privacy breaches and poisoning attacks. To tackle these challenges, researchers have separately devised secure aggregation mechanisms to protect data privacy and robust aggregation methods that withstand poisoning attacks. However, simultaneously addressing both concerns is challenging; secure aggregation facilitates poisoning attacks as most anomaly detection techniques require access to unencrypted local model updates, which are obscured by secure aggregation. Few recent efforts to simultaneously tackle both challenges offen depend on impractical assumption of non-colluding two-server setups that disrupt FL's topology, or three-party computation which introduces scalability issues, complicating deployment and application. To overcome this dilemma, this paper introduce a Dual Defense Federated learning (DDFed) framework. DDFed simultaneously boosts privacy protection and mitigates poisoning attacks, without introducing new participant roles or disrupting the existing FL topology. DDFed initially leverages cutting-edge fully homomorphic encryption (FHE) to securely aggregate model updates, without the impractical requirement for non-colluding two-server setups and ensures strong privacy protection. Additionally, we proposes a unique two-phase anomaly detection mechanism for encrypted model updates, featuring secure similarity computation and feedback-driven collaborative selection, with additional measures to prevent potential privacy breaches from Byzantine clients incorporated into the detection process. We conducted extensive experiments on various model poisoning attacks and FL scenarios, including both cross-device and cross-silo FL. Experiments on publicly available datasets demonstrate that DDFed successfully protects model privacy and effectively defends against model poisoning threats.
Scaling laws for learning with real and surrogate data
Authors: Ayush Jain; Andrea Montanari; Eren Sasoglu
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: Collecting large quantities of high-quality data can be  prohibitively expensive or impractical, and a bottleneck in machine learning. One may instead augment a small set of $n$ data points from the target distribution with data from more accessible sources, e.g. data collected under different circumstances or synthesized by generative models. We refer to such data  as `surrogate data'. We study a weighted empirical risk minimization (ERM) approach for integrating surrogate data into training. We analyze mathematically this method under several classical statistical models, and validate our findings empirically on datasets from different domains. Our main findings are: $(i)$ Integrating surrogate data can significantly reduce the test error on the original distribution. Surprisingly, this can happen even when the surrogate data is unrelated to the original ones. We trace back this behavior to the classical Stein's paradox. $(ii)$ In order to reap the benefit of surrogate data, it is crucial to use optimally weighted ERM. $(iii)$ The test error of models trained on mixtures of real and surrogate data is approximately described by a scaling law. This scaling law can be used to predict the optimal weighting scheme, and to choose the amount of surrogate data to add.
Class Distribution Shifts in Zero-Shot Learning: Learning Robust Representations
Authors: Yuli Slavutsky; Yuval Benjamini
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: Zero-shot learning methods typically assume that the new, unseen classes encountered during deployment come from the same distribution as the the classes in the training set. However, real-world scenarios often involve class distribution shifts (e.g., in age or gender for person identification), posing challenges for zero-shot classifiers that rely on learned representations from training classes. In this work, we propose and analyze a model that assumes that the attribute responsible for the shift is unknown in advance. We show that in this setting, standard training may lead to non-robust representations. To mitigate this, we develop an algorithm for learning robust representations in which (a) synthetic data environments are constructed via hierarchical sampling, and (b) environment balancing penalization, inspired by out-of-distribution problems, is applied. We show that our algorithm improves generalization to diverse class distributions in both simulations and experiments on real-world datasets.
A Near-optimal Algorithm for Learning Margin Halfspaces with Massart Noise
Authors: Ilias Diakonikolas; Nikos Zarifis
Venue: NeurIPS 2024 (NeurIPS 2024 spotlight)

Abstract: We study the problem of PAC learning $\gamma$-margin halfspaces in the presence of Massart noise. 
Without computational considerations, the sample complexity of this learning problem is known to be 
$\widetilde{\Theta}(1/(\gamma^2 \epsilon))$. 
Prior computationally efficient algorithms for the problem incur sample complexity 
$\tilde{O}(1/(\gamma^4 \epsilon^3))$ and achieve 0-1 error of $\eta+\epsilon$, 
where $\eta<1/2$ is the upper bound on the noise rate.
Recent work gave evidence of an information-computation tradeoff, 
suggesting that a quadratic dependence on $1/\epsilon$ is required 
for computationally efficient algorithms. 
Our main result is a computationally efficient learner with sample complexity 
$\widetilde{\Theta}(1/(\gamma^2 \epsilon^2))$, nearly matching this lower bound. 
In addition, our algorithm is simple and practical, 
relying on online SGD on a carefully selected sequence of convex losses.
MoEUT: Mixture-of-Experts Universal Transformers
Authors: Róbert Csordás; Kazuki Irie; Jürgen Schmidhuber; Christopher Potts; Christopher D Manning
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: Previous work on Universal Transformers (UTs) has demonstrated the importance of parameter sharing across layers. By allowing recurrence in depth, UTs have advantages over standard Transformers in learning compositional generalizations, but layer-sharing comes with a practical limitation of parameter-compute ratio: it drastically reduces the parameter count compared to the non-shared model with the same dimensionality. Naively scaling up the layer size to compensate for the loss of parameters makes its computational resource requirements prohibitive. In practice, no previous work has succeeded in proposing a shared-layer Transformer design that is competitive in parameter count-dominated tasks such as language modeling. Here we propose MoEUT (pronounced "moot"), an effective mixture-of-experts (MoE)-based shared-layer Transformer architecture, which combines several recent advances in MoEs for both feedforward and attention layers of standard Transformers together with novel layer-normalization and grouping schemes that are specific and crucial to UTs. The resulting UT model, for the first time, slightly outperforms standard Transformers on language modeling tasks such as BLiMP and PIQA, while using significantly less compute and memory.
On the Benefits of Public Representations for Private Transfer Learning under Distribution Shift
Authors: Pratiksha Thaker; Amrith Setlur; Steven Wu; Virginia Smith
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: Public pretraining is a promising approach to improve differentially private model training. However, recent work has noted that many positive research results studying this paradigm only consider in-distribution tasks, and may not apply to settings where there is distribution shift between the pretraining and finetuning data---a scenario that is likely when finetuning private tasks due to the sensitive nature of the data. In this work, we show empirically across three tasks that even in settings with large distribution shift, where both zero-shot performance from public data and training from scratch with private data give unusably weak results, public features can in fact improve private training accuracy by up to 67\% over private training from scratch. We provide a theoretical explanation for this phenomenon, showing that if the public and private data share a low-dimensional representation, public representations can improve the sample complexity of private training even if it is \emph{impossible} to learn the private task from the public data alone. Altogether, our results provide evidence that public data can indeed make private training practical in realistic settings of extreme distribution shift.
Temporal-Difference Learning Using Distributed Error Signals
Authors: Jonas Guan; Shon Eduard Verch; Claas A Voelcker; Ethan C Jackson; Nicolas Papernot; William A Cunningham
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: A computational problem in biological reward-based learning is how credit assignment is performed in the nucleus accumbens (NAc). Much research suggests that NAc dopamine encodes temporal-difference (TD) errors for learning value predictions. However, dopamine is synchronously distributed in regionally homogeneous concentrations, which does not support explicit credit assignment (like used by backpropagation). It is unclear whether distributed errors alone are sufficient for synapses to make coordinated updates to learn complex, nonlinear reward-based learning tasks. We design a new deep Q-learning algorithm, Artificial Dopamine, to computationally demonstrate that synchronously distributed, per-layer TD errors may be sufficient to learn surprisingly complex RL tasks. We empirically evaluate our algorithm on MinAtar, the DeepMind Control Suite, and classic control tasks, and show it often achieves comparable performance to deep RL algorithms that use backpropagation.
The Price of Implicit Bias in Adversarially Robust Generalization
Authors: Nikolaos Tsilivis; Natalie Frank; Nathan Srebro; Julia Kempe
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: We study the implicit bias of optimization in robust empirical risk minimization (robust ERM) and its connection with robust generalization.  In classification settings under adversarial perturbations with linear models, we study what type of regularization should ideally be applied for a given perturbation set to improve (robust) generalization. We then show that the implicit bias of optimization in robust ERM can significantly affect the robustness of the model and identify two ways this can happen; either through the optimization algorithm or the architecture. We verify our predictions in simulations with synthetic data and experimentally study the importance of implicit bias in robust ERM with deep neural networks.
Feature-Level Adversarial Attacks and Ranking Disruption for Visible-Infrared Person Re-identification
Authors: Xi Yang; Huanling Liu; De Cheng; Nannan Wang; Xinbo Gao
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: Visible-infrared person re-identification (VIReID) is widely used in fields such as video surveillance and intelligent transportation, imposing higher demands on model security. In practice, the adversarial attacks based on VIReID aim to disrupt output ranking and quantify the security risks of models. Although numerous studies have been emerged on adversarial attacks and defenses in fields such as face recognition, person re-identification, and pedestrian detection, there is currently a lack of research on the security of VIReID systems. To this end, we propose to explore the vulnerabilities of VIReID systems and prevent potential serious losses due to insecurity. Compared to research on single-modality ReID, adversarial feature alignment and modality differences need to be particularly emphasized. Thus, we advocate for feature-level adversarial attacks to disrupt the output rankings of VIReID systems. To obtain adversarial features, we introduce \textit{Universal Adversarial Perturbations} (UAP) to simulate common disturbances in real-world environments. Additionally, we employ a \textit{Frequency-Spatial Attention Module} (FSAM), integrating frequency information extraction and spatial focusing mechanisms, and further emphasize important regional features from different domains on the shared features. This ensures that adversarial features maintain consistency within the feature space. Finally, we employ an \textit{Auxiliary Quadruple Adversarial Loss} to amplify the differences between modalities, thereby improving the distinction and recognition of features between visible and infrared images, which causes the system to output incorrect rankings. Extensive experiments on two VIReID benchmarks (i.e., SYSU-MM01, RegDB) and different systems validate the effectiveness of our method.
Constrained Synthesis with Projected Diffusion Models
Authors: Jacob K Christopher; Stephen Baek; Ferdinando Fioretto
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: This paper introduces an approach to endow generative diffusion processes the ability to satisfy and certify compliance with constraints and physical principles. The proposed method recast the traditional sampling process of generative diffusion models as a constrained optimization problem, steering the generated data distribution to remain within a specified region to ensure adherence to the given constraints.
These capabilities are validated on applications featuring both convex and challenging, non-convex, constraints as well as ordinary differential equations, in domains spanning from synthesizing new materials with precise morphometric properties, generating physics-informed motion, optimizing paths in planning scenarios, and human motion synthesis.
Block Transformer: Global-to-Local Language Modeling for Fast Inference
Authors: Namgyu Ho; Sangmin Bae; Taehyeon Kim; hyunjik.jo; Yireun Kim; Tal Schuster; Adam Fisch; James Thorne; Se-Young Yun
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: We introduce the Block Transformer which adopts hierarchical global-to-local modeling to autoregressive transformers to mitigate the inference bottlenecks associated with self-attention. Self-attention requires the key-value (KV) cache of all previous sequences to be retrieved from memory at every decoding step to retrieve context information, leading to two primary bottlenecks during batch inference. First, there is a significant delay in obtaining the first token, as the information of the entire prompt must first be processed to prefill the KV cache. Second, computation of subsequent tokens is bottlenecked by the high memory I/O demand of fetching the entire KV cache, which grows linearly with sequence length, incurring quadratic memory reads overall. We design the Block Transformer to strategically mitigate these costs, by incorporating coarsity and locality into an integrated global-to-local architecture. At the lower layers, we aggregate tokens into fixed size blocks to apply attention across the entire sequence at coarse-grained detail, to capture the global context while minimizing KV cache overhead. At upper layers, we apply attention within each block to decode individual tokens, to model fine-grained details with a lightweight local KV cache. We pretrain vanilla and Block Transformers from scratch and demonstrate that Block Transformers reach 10--20x inference throughput compared to vanilla transformers with equivalent perplexity and zero-shot task performance.