GenAI4ELab/papercli-papers huggingface dataset:
A Fast and Accurate Estimator for Large Scale Linear Model via Data Averaging
Authors: Rui Wang; Yanyan Ouyang; Panpan Yu; Wangli Xu
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: This work is concerned with the estimation problem of linear model when the
sample size is extremely large and the data dimension can vary with the sample
size. In this setting, the least square estimator based on the full data is not feasible
with limited computational resources. Many existing methods for this problem are
based on the sketching technique which uses the sketched data to perform least
square estimation. We derive fine-grained lower bounds of the conditional mean
squared error for sketching methods. For sampling methods, our lower bound
provides an attainable optimal convergence rate. Our result implies that when the
dimension is large, there is hardly a sampling method can have a faster convergence
rate than the uniform sampling method. To achieve a better statistical performance,
we propose a new sketching method based on data averaging. The proposed
method reduces the original data to a few averaged observations. These averaged
observations still satisfy the linear model and are used to estimate the regression
coefficients. The asymptotic behavior of the proposed estimation procedure is
studied. Our theoretical results show that the proposed method can achieve a
faster convergence rate than the optimal convergence rate for sampling methods.
Theoretical and numerical results show that the proposed estimator has good
statistical performance as well as low computational cost.
Implicit Regularization in Over-Parameterized Support Vector Machine
Authors: Yang Sui; Xin HE; Yang Bai
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: In this paper, we design a regularization-free algorithm for high-dimensional support vector machines (SVMs) by integrating over-parameterization with Nesterov's smoothing method, and provide theoretical guarantees for the induced implicit regularization phenomenon. In particular, we construct an over-parameterized hinge loss function and estimate the true parameters by leveraging regularization-free gradient descent on this loss function. The utilization of Nesterov's method enhances the computational efficiency of our algorithm, especially in terms of determining the stopping criterion and reducing computational complexity. With appropriate choices of initialization, step size, and smoothness parameter, we demonstrate that unregularized gradient descent achieves a near-oracle statistical convergence rate. Additionally, we verify our theoretical findings through a variety of numerical experiments and compare the proposed method with explicit regularization. Our results illustrate the advantages of employing implicit regularization via gradient descent in conjunction with over-parameterization in sparse SVMs.
Auxiliary Losses for Learning Generalizable Concept-based Models
Authors: Ivaxi Sheth; Samira Ebrahimi Kahou
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: The increasing use of neural networks in various applications has lead to increasing apprehensions, underscoring the necessity to understand their operations beyond mere final predictions. As a solution to enhance model transparency, Concept Bottleneck Models (CBMs) have gained popularity since their introduction. CBMs essentially limit the latent space of a model to human-understandable high-level concepts.  While beneficial, CBMs have been reported to often learn irrelevant concept representations that consecutively damage model performance. To overcome the performance trade-off, we propose a cooperative-Concept Bottleneck Model (coop-CBM). The concept representation of our model is particularly meaningful when fine-grained concept labels are absent. Furthermore, we introduce the concept orthogonal loss (COL) to encourage the separation between the concept representations and to reduce the intra-concept distance. This paper presents extensive experiments on real-world datasets for image classification tasks, namely CUB, AwA2, CelebA and TIL. We also study the performance of coop-CBM models under various distributional shift settings. We show that our proposed method achieves higher accuracy in all distributional shift settings even compared to the black-box models with the highest concept accuracy.
EvoPrompting: Language Models for Code-Level Neural Architecture Search
Authors: Angelica Chen; David Dohan; David So
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: Given the recent impressive accomplishments of language models (LMs) for code generation, we explore the use of LMs as general adaptive mutation and crossover operators for an evolutionary neural architecture search (NAS) algorithm.
While NAS still proves too difficult a task for LMs to succeed at solely through prompting, we find that the combination of evolutionary prompt engineering with soft prompt-tuning, a method we term EvoPrompting, consistently finds diverse and high performing models. We first demonstrate that EvoPrompting is effective on the computationally efficient MNIST-1D dataset, where EvoPrompting produces convolutional architecture variants that outperform both those designed by human experts and naive few-shot prompting in terms of accuracy and model size. We then apply our method to searching for graph neural networks on the CLRS Algorithmic Reasoning Benchmark, where EvoPrompting is able to design *novel* architectures that outperform current state-of-the-art models on 21 out of 30 algorithmic reasoning tasks while maintaining similar model size. EvoPrompting is successful at designing accurate and efficient neural network architectures across a variety of machine learning tasks, while also being general enough for easy adaptation to other tasks beyond neural network design.
GAIA: Delving into Gradient-based Attribution Abnormality for Out-of-distribution Detection
Authors: Jinggang Chen; Junjie Li; Xiaoyang Qu; Jianzong Wang; Jiguang Wan; Jing Xiao
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: Detecting out-of-distribution (OOD) examples is crucial to guarantee the reliability and safety of deep neural networks in real-world settings. In this paper, we offer an innovative perspective on quantifying the disparities between in-distribution (ID) and OOD data---analyzing the uncertainty that arises when models attempt to explain their predictive decisions. This perspective is motivated by our observation that gradient-based attribution methods encounter challenges in assigning feature importance to OOD data, thereby yielding divergent explanation patterns. Consequently, we investigate how attribution gradients lead to uncertain explanation outcomes and introduce two forms of abnormalities for OOD detection: the zero-deflation abnormality and the channel-wise average abnormality. We then propose GAIA, a simple and effective approach that incorporates Gradient Abnormality Inspection and Aggregation.  The effectiveness of GAIA is validated on both commonly utilized (CIFAR) and large-scale (ImageNet-1k) benchmarks. Specifically, GAIA reduces the average FPR95 by 23.10% on CIFAR10 and by 45.41% on CIFAR100 compared to advanced post-hoc methods.
Prompt-augmented Temporal Point Process for Streaming Event Sequence
Authors: Siqiao Xue; Yan Wang; Zhixuan Chu; Xiaoming Shi; Caigao JIANG; Hongyan Hao; Gangwei Jiang; Xiaoyun Feng; James Y. Zhang; JUN ZHOU
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: Neural Temporal Point Processes (TPPs)  are the prevalent paradigm for modeling continuous-time event sequences, such as user activities on the web and financial transactions. In real world applications, the event data typically comes in a streaming manner, where the distribution of the patterns may shift over time. Under the privacy and memory constraints commonly seen in real scenarios, how to continuously monitor a TPP to learn the streaming event sequence is an important yet under-investigated problem. In this work, we approach this problem by adopting Continual Learning (CL), which aims to enable a model to continuously learn a sequence of tasks without catastrophic forgetting. While CL for event sequence is less well studied, we present a simple yet effective framework, PromptTPP, by integrating the base TPP with a continuous-time retrieval prompt pool. In our proposed framework, prompts are small learnable parameters, maintained in a memory space and jointly optimized with the base TPP so that the model is properly instructed to learn event streams arriving sequentially without buffering past examples or task-specific attributes. We formalize a novel and realistic experimental setup for modeling event streams, where PromptTPP consistently sets state-of-the-art performance across two real user behavior datasets.
Grassmann Manifold Flows for Stable Shape Generation
Authors: Ryoma Yataka; Kazuki Hirashima; Masashi Shiraishi
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: Recently, studies on machine learning have focused on methods that use symmetry implicit in a specific manifold as an inductive bias.
Grassmann manifolds provide the ability to handle fundamental shapes represented as shape spaces, enabling stable shape analysis. 
In this paper, we present a novel approach in which we establish the theoretical foundations for learning distributions on the Grassmann manifold via continuous normalization flows, with the explicit goal of generating stable shapes.
Our approach facilitates more robust generation by effectively eliminating the influence of extraneous transformations, such as rotations and inversions, through learning and generating within a Grassmann manifold designed to accommodate the essential shape information of the object.
The experimental results indicated that the proposed method could generate high-quality samples by capturing the data structure.
Furthermore, the proposed method significantly outperformed state-of-the-art methods in terms of the log-likelihood or evidence lower bound.
The results obtained are expected to stimulate further research in this field, leading to advances for stable shape generation and analysis.
Bayesian Optimisation of Functions on Graphs
Authors: Xingchen Wan; Pierre Osselin; Henry Kenlay; Binxin Ru; Michael A Osborne; Xiaowen Dong
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: The increasing availability of graph-structured data motivates the task of optimising over functions defined on the node set of graphs. Traditional graph search algorithms can be applied in this case, but they may be sample-inefficient and do not make use of information about the function values; on the other hand, Bayesian optimisation is a class of promising black-box solvers with superior sample efficiency, but it has scarcely been applied to such novel setups. To fill this gap, we propose a novel Bayesian optimisation framework that optimises over functions defined on generic, large-scale and potentially unknown graphs. Through the learning of suitable kernels on graphs, our framework has the advantage of adapting to the behaviour of the target function. The local modelling approach further guarantees the efficiency of our method. Extensive experiments on both synthetic and real-world graphs demonstrate the effectiveness of the proposed optimisation framework.
Learning Probabilistic Symmetrization for Architecture Agnostic Equivariance
Authors: Jinwoo Kim; Dat Tien Nguyen; Ayhan Suleymanzade; Hyeokjun An; Seunghoon Hong
Venue: NeurIPS 2023 (NeurIPS 2023 spotlight)

Abstract: We present a novel framework to overcome the limitations of equivariant architectures in learning functions with group symmetries. In contrary to equivariant architectures, we use an arbitrary base model such as an MLP or a transformer and symmetrize it to be equivariant to the given group by employing a small equivariant network that parameterizes the probabilistic distribution underlying the symmetrization. The distribution is end-to-end trained with the base model which can maximize performance while reducing sample complexity of symmetrization. We show that this approach ensures not only equivariance to given group but also universal approximation capability in expectation. We implement our method on various base models, including patch-based transformers that can be initialized from pretrained vision transformers, and test them for a wide range of symmetry groups including permutation and Euclidean groups and their combinations. Empirical tests show competitive results against tailored equivariant architectures, suggesting the potential for learning equivariant functions for diverse groups using a non-equivariant universal base architecture. We further show evidence of enhanced learning in symmetric modalities, like graphs, when pretrained from non-symmetric modalities, like vision. Code is available at https://github.com/jw9730/lps.
Structured State Space Models for In-Context Reinforcement Learning
Authors: Chris Lu; Yannick Schroecker; Albert Gu; Emilio Parisotto; Jakob Nicolaus Foerster; Satinder Singh; Feryal Behbahani
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: Structured state space sequence (S4) models have recently achieved state-of-the-art performance on long-range sequence modeling tasks. These models also have fast inference speeds and parallelisable training, making them potentially useful in many reinforcement learning settings. We propose a  modification to a variant of S4 that enables us to initialise and reset the hidden state in parallel, allowing us to tackle reinforcement learning tasks. We show that our modified architecture runs asymptotically faster than Transformers in sequence length and performs better than RNN's on a simple memory-based task. We evaluate our modified architecture on a set of partially-observable environments and find that, in practice, our model outperforms RNN's while also running over five times faster. Then, by leveraging the model’s ability to handle long-range sequences, we achieve strong performance on a challenging meta-learning task in which the agent is given a randomly-sampled continuous control environment, combined with a randomly-sampled linear projection of the environment's observations and actions. Furthermore, we show the resulting model can adapt to out-of-distribution held-out tasks. Overall, the results presented in this paper show that structured state space models are fast and performant for in-context reinforcement learning tasks. We provide code at https://github.com/luchris429/s5rl.
Normalizing flow neural networks by JKO scheme
Authors: Chen Xu; Xiuyuan Cheng; Yao Xie
Venue: NeurIPS 2023 (NeurIPS 2023 spotlight)

Abstract: Normalizing flow is a class of deep generative models for efficient sampling and likelihood estimation, which achieves attractive performance, particularly in high dimensions. The flow is often implemented using a sequence of invertible residual blocks. Existing works adopt special network architectures and regularization of flow trajectories. In this paper, we develop a neural ODE flow network called JKO-iFlow, inspired by the Jordan-Kinderleherer-Otto (JKO) scheme, which unfolds the discrete-time dynamic of the Wasserstein gradient flow. The proposed method stacks residual blocks one after another, allowing efficient block-wise training of the residual blocks, avoiding sampling SDE trajectories and score matching or variational learning, thus reducing the memory load and difficulty in end-to-end training. We also develop adaptive time reparameterization of the flow network with a progressive refinement of the induced trajectory in probability space to improve the model accuracy further. Experiments with synthetic and real data show that the proposed JKO-iFlow network achieves competitive performance compared with existing flow and diffusion models at a significantly reduced computational and memory cost.
On-the-Fly Adapting Code Summarization on Trainable Cost-Effective Language Models
Authors: Yufan Cai; Yun Lin; Chenyan Liu; Jinglian Wu; Yifan Zhang; Yiming Liu; Yeyun Gong; Jin Song Dong
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: Deep learning models are emerging to summarize source code to comment,
  facilitating tasks of code documentation and program comprehension.
  Scaled-up large language models trained on large open corpus have achieved good performance in such tasks.
  However, in practice, the subject code in one certain project can be specific,
  which may not align with the overall training corpus.
  Some code samples from other projects may be contradictory and introduce inconsistencies when the models try to fit all the samples.

  In this work, we introduce a novel approach, Adacom, to improve the performance of comment generators by on-the-fly model adaptation.
  This research is motivated by the observation that deep comment generators
  often need to strike a balance as they need to fit all the training samples.
  Specifically, for one certain target code $c$,
  some training samples $S_p$ could have made more contributions while other samples $S_o$ could have counter effects.
  However, the traditional fine-tuned models need to fit both $S_p$ and $S_o$ from a global perspective, 
  leading to compromised performance for one certain target code $c$.
  In this context, we design Adacom to
  (1) detect whether the model might have a compromised performance on a target code $c$ and
  (2) retrieve a few helpful training samples $S_p$ that have contradictory samples in the training dataset and,
  (3) adapt the model on the fly by re-training the $S_p$ to strengthen the helpful samples and unlearn the harmful samples.
  Our extensive experiments on 7 comment generators and 4 public datasets show that
  (1) can significantly boost the performance of comment generation (BLEU4 score by on average 14.9\%, METEOR by 12.2\%, and ROUGE-L by 7.4\%),
  (2) the adaptation on one code sample is cost-effective and acceptable as an on-the-fly solution, and
  (3) can adapt well on out-of-distribution code samples.
Sample based Explanations via Generalized Representers
Authors: Che-Ping Tsai; Chih-Kuan Yeh; Pradeep Kumar Ravikumar
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: We propose a general class of sample based explanations of machine learning models, which we term generalized representers. To measure the effect of a training sample on a model's test prediction, generalized representers use two components: a global sample importance that quantifies the importance of the training point to the model and is invariant to test samples, and a local sample importance that measures similarity between the training sample and the test point with a kernel. A key contribution of the paper is to show that generalized representers are the only class of sample based explanations satisfying a natural set of axiomatic properties. We discuss approaches to extract global importances given a kernel, and also natural choices of kernels given modern non-linear models. As we show, many popular existing sample based explanations could be cast as generalized representers with particular choices of kernels and approaches to extract global importances. Additionally, we conduct empirical comparisons of different generalized representers on two image classification datasets.
Sample Efficient Reinforcement Learning in Mixed Systems through Augmented Samples and Its Applications to Queueing Networks
Authors: Honghao Wei; Xin Liu; Weina Wang; Lei Ying
Venue: NeurIPS 2023 (NeurIPS 2023 spotlight)

Abstract: This paper considers a class of reinforcement learning problems, which involve systems with two types of states: stochastic and pseudo-stochastic. In such systems, stochastic states follow a stochastic transition kernel while the transitions of pseudo-stochastic states are deterministic {\em given} the stochastic states/transitions. We refer to such systems as mixed systems, which are widely used in various applications, including Manufacturing systems, communication networks, and queueing networks. We propose a sample-efficient RL method that accelerates learning by generating augmented data samples. The proposed algorithm is data-driven (model-free), but it learns the policy from data samples from both real and augmented samples. This method significantly improves learning by reducing the sample complexity such that the dataset only needs to have sufficient coverage of the stochastic states. We analyze the sample complexity of the proposed method under Fitted Q Iteration (FQI) and demonstrate that the optimality gap decreases as  $O\left(\sqrt{\frac{1}{n}}+\sqrt{\frac{1}{m}}\right),$ where $n$ represents the number of real samples, and $m$ is the number of augmented samples per real sample. It is important to note that without augmented samples, the optimality gap is $O(1)$ due to the insufficient data coverage of the pseudo-stochastic states. Our experimental results on multiple queueing network applications confirm that the proposed method indeed significantly accelerates both deep Q-learning and deep policy gradient.
Feature Likelihood Divergence: Evaluating the Generalization of Generative Models Using Samples
Authors: Marco Jiralerspong; Joey Bose; Ian Gemp; Chongli Qin; Yoram Bachrach; Gauthier Gidel
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: The past few years have seen impressive progress in the development of deep generative models capable of producing high-dimensional, complex, and photo-realistic data. However, current methods for evaluating such models remain incomplete: standard likelihood-based metrics do not always apply and rarely correlate with perceptual fidelity, while sample-based metrics, such as FID, are insensitive to overfitting, i.e., inability to generalize beyond the training set. To address these limitations, we propose a new metric called the Feature Likelihood Divergence (FLD), a parametric sample-based score that uses density estimation to provide a comprehensive trichotomic evaluation accounting for novelty (i.e., different from the training samples), fidelity, and diversity of generated samples.  We empirically demonstrate the ability of FLD to identify specific overfitting problem cases, where previously proposed metrics fail. We also extensively evaluate FLD on various image datasets and model classes, demonstrating its ability to match intuitions of previous metrics like FID while offering a more comprehensive evaluation of generative models.
Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers
Authors: Sotiris Anagnostidis; Dario Pavllo; Luca Biggio; Lorenzo Noci; Aurelien Lucchi; Thomas Hofmann
Venue: NeurIPS 2023 (NeurIPS 2023 spotlight)

Abstract: Autoregressive Transformers adopted in Large Language Models (LLMs) are hard to scale to long sequences. Despite several works trying to reduce their computational cost, most of LLMs still adopt attention layers between all pairs of tokens in the sequence, thus incurring a quadratic cost. In this study, we present a novel approach that dynamically prunes contextual information while preserving the model's expressiveness, resulting in reduced memory and computational requirements during inference. Our method employs a learnable mechanism that determines which uninformative tokens can be dropped from the context at any point across the generation process. By doing so, our approach not only addresses performance concerns but also enhances interpretability, providing valuable insight into the model's decision-making process. Our technique can be applied to existing pre-trained models through a straightforward fine-tuning process, and the pruning strength can be specified by a sparsity parameter. Notably, our empirical findings demonstrate that we can effectively prune up to 80\% of the context without significant performance degradation on downstream tasks, offering a valuable tool for mitigating inference costs. Our reference implementation achieves up to $2\times$ increase in inference throughput and even greater memory savings.