GenAI4ELab/papercli-papers huggingface dataset:
Importance-aware Co-teaching for Offline Model-based Optimization
Authors: Ye Yuan; Can Chen; Zixuan Liu; Willie Neiswanger; Xue Liu
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: Offline model-based optimization aims to find a design that maximizes a property of interest using only an offline dataset, with applications in robot, protein, and molecule design, among others. A prevalent approach is gradient ascent, where a proxy model is trained on the offline dataset and then used to optimize the design. This method suffers from an out-of-distribution issue, where the proxy is not accurate for unseen designs. To mitigate this issue, we explore using a pseudo-labeler to generate valuable data for fine-tuning the proxy. 
Specifically, we propose $\textit{\textbf{I}mportance-aware \textbf{C}o-\textbf{T}eaching for Offline Model-based Optimization}~(\textbf{ICT})$. This method maintains three symmetric proxies with their mean ensemble as the final proxy, and comprises two steps. The first step is $\textit{pseudo-label-driven co-teaching}$. In this step, one proxy is iteratively selected as the pseudo-labeler for designs near the current optimization point, generating pseudo-labeled data.  Subsequently, a co-teaching process identifies small-loss samples as valuable data and exchanges them between the other two proxies for fine-tuning, promoting knowledge transfer.  This procedure is repeated three times, with a different proxy chosen as the pseudo-labeler each time, ultimately enhancing the ensemble performance.
To further improve accuracy of pseudo-labels, we perform a secondary step of $\textit{meta-learning-based sample reweighting}$,
which assigns importance weights to samples in the pseudo-labeled dataset and updates them via meta-learning. ICT achieves state-of-the-art results across multiple design-bench tasks, achieving the best mean rank $3.1$ and median rank $2$ among $15$ methods.
Our source code can be accessed here.
Parallel-mentoring for Offline Model-based Optimization
Authors: Can Chen; Christopher Beckham; Zixuan Liu; Xue Liu; Christopher Pal
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: We study offline model-based optimization to maximize a black-box objective function with a static dataset of designs and scores. These designs encompass a variety of domains, including materials, robots, DNA sequences, and proteins. A common approach trains a proxy on the static dataset and performs gradient ascent to obtain new designs. However, this often results in poor designs due to the proxy inaccuracies for out-of-distribution designs. Recent studies indicate that (a) gradient ascent with a mean ensemble of proxies generally outperforms simple gradient ascent, and (b) a trained proxy provides weak ranking supervision signals for design selection. Motivated by (a) and (b), we propose $\textit{parallel-mentoring}$ as an effective and novel method that facilitates mentoring among proxies, creating a more robust ensemble to mitigate the out-of-distribution issue. We focus on the three-proxy case in the main paper and our method consists of two modules. The first module, $\textit{voting-based pairwise supervision}$, operates on three parallel proxies and captures their ranking supervision signals as pairwise comparison labels. These labels are combined through majority voting to generate consensus labels, which incorporates ranking supervision signals from all proxies and enables mutual mentoring. Yet, label noise arises due to possible incorrect consensus. To alleviate this, we introduce an $\textit{adaptive soft-labeling}$ module with soft-labels initialized as consensus labels. Based on bi-level optimization, this module fine-tunes proxies in the inner level and learns more accurate labels in the outer level to adaptively mentor proxies, resulting in a more robust ensemble. Experiments validate the effectiveness of our method. Our code is available here.
Predicting a Protein's Stability under a Million Mutations
Authors: Jeffrey Ouyang-Zhang; Daniel Jesus Diaz; Adam Klivans; Philipp Kraehenbuehl
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: Stabilizing proteins is a foundational step in protein engineering. However, the evolutionary pressure of all extant proteins makes identifying the scarce number of mutations that will improve thermodynamic stability challenging. 
Deep learning has recently emerged as a powerful tool for identifying promising mutations.
Existing approaches, however, are computationally expensive, as the number of model inferences scales with the number of mutations queried. 
Our main contribution is a simple, parallel decoding algorithm.
Mutate Everything is capable of predicting the effect of all single and double mutations in one forward pass. 
It is even versatile enough to predict higher-order mutations with minimal computational overhead.
We build Mutate Everything on top of ESM2 and AlphaFold, neither of which were trained to predict thermodynamic stability.
We trained on the Mega-Scale cDNA proteolysis dataset and achieved state-of-the-art performance on single and higher-order mutations on S669, ProTherm, and ProteinGym datasets.
Our code is available at https://github.com/jozhang97/MutateEverything.
LayoutGPT: Compositional Visual Planning and Generation with Large Language Models
Authors: Weixi Feng; Wanrong Zhu; Tsu-Jui Fu; Varun Jampani; Arjun Reddy Akula; Xuehai He; S Basu; Xin Eric Wang; William Yang Wang
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: Attaining a high degree of user controllability in visual generation often requires intricate, fine-grained inputs like layouts. However, such inputs impose a substantial burden on users when compared to simple text inputs. To address the issue, we study how Large Language Models (LLMs) can serve as visual planners by generating layouts from text conditions, and thus collaborate with visual generative models. We propose LayoutGPT, a method to compose in-context visual demonstrations in style sheet language to enhance visual planning skills of LLMs. We show that LayoutGPT can generate plausible layouts in multiple domains, ranging from 2D images to 3D indoor scenes. LayoutGPT also shows superior performance in converting challenging language concepts like numerical and spatial relations to layout arrangements for faithful text-to-image generation. When combined with a downstream image generation model, LayoutGPT outperforms text-to-image models/systems by 20-40\% and achieves comparable performance as human users in designing visual layouts for numerical and spatial correctness. Lastly, LayoutGPT achieves comparable performance to supervised methods in 3D indoor scene synthesis, demonstrating its effectiveness and potential in multiple visual domains.
ARTIC3D: Learning Robust Articulated 3D Shapes from Noisy Web Image Collections
Authors: Chun-Han Yao; Amit Raj; Wei-Chih Hung; Michael Rubinstein; Yuanzhen Li; Ming-Hsuan Yang; Varun Jampani
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: Estimating 3D articulated shapes like animal bodies from monocular images is inherently challenging due to the ambiguities of camera viewpoint, pose, texture, lighting, etc. We propose ARTIC3D, a self-supervised framework to reconstruct per-instance 3D shapes from a sparse image collection in-the-wild. Specifically, ARTIC3D is built upon a skeleton-based surface representation and is further guided by 2D diffusion priors from Stable Diffusion. First, we enhance the input images with occlusions/truncation via 2D diffusion to obtain cleaner mask estimates and semantic features. Second, we perform diffusion-guided 3D optimization to estimate shape and texture that are of high-fidelity and faithful to input images. We also propose a novel technique to calculate more stable image-level gradients via diffusion models compared to existing alternatives. Finally, we produce realistic animations by fine-tuning the rendered shape and texture under rigid part transformations. Extensive evaluations on multiple existing datasets as well as newly introduced noisy web image collections with occlusions and truncation demonstrate that ARTIC3D outputs are more robust to noisy images, higher quality in terms of shape and texture details, and more realistic when animated.
Symbol-LLM: Leverage Language Models for Symbolic System in Visual Human Activity Reasoning
Authors: Xiaoqian Wu; Yong-Lu Li; Jianhua Sun; Cewu Lu
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: Human reasoning can be understood as a cooperation between the intuitive, associative "System-1'' and the deliberative, logical "System-2''. For existing System-1-like methods in visual activity understanding, it is crucial to integrate System-2 processing to improve explainability, generalization, and data efficiency. One possible path of activity reasoning is building a symbolic system composed of symbols and rules, where one rule connects multiple symbols, implying human knowledge and reasoning abilities.
Previous methods have made progress, but are defective with limited symbols from handcraft and limited rules from visual-based annotations, failing to cover the complex patterns of activities and lacking compositional generalization. 
To overcome the defects, we propose a new symbolic system with two ideal important properties: broad-coverage symbols and rational rules. Collecting massive human knowledge via manual annotations is expensive to instantiate this symbolic system. Instead, we leverage the recent advancement of LLMs (Large Language Models) as an approximation of the two ideal properties, i.e., Symbols from Large Language Models (Symbol-LLM). 
Then, given an image, visual contents from the images are extracted and
checked as symbols and activity semantics are reasoned out based on rules via fuzzy logic calculation.
Our method shows superiority in extensive activity understanding tasks. Code and data are available at https://mvig-rhos.com/symbol_llm.
Faster Differentially Private Convex Optimization via Second-Order Methods
Authors: Arun Ganesh; Mahdi Haghifam; Thomas Steinke; Abhradeep Guha Thakurta
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: Differentially private (stochastic) gradient descent is the workhorse of DP private machine learning in both the convex and non-convex settings. Without privacy constraints, second-order methods, like Newton's method, converge faster than first-order methods like gradient descent. In this work, we investigate the prospect of using the second-order information from the loss function to accelerate DP convex optimization. We first develop a private variant of the regularized cubic Newton method of Nesterov and Polyak, and show that for the class of strongly convex loss functions, our algorithm has quadratic convergence and achieves the optimal excess loss. We then design a practical second-order DP algorithm for the unconstrained logistic regression problem. We theoretically and empirically study the performance of our algorithm. Empirical results show our algorithm consistently achieves the best excess loss compared to other baselines and is 10-40x faster than DP-GD/DP-SGD for challenging datasets.
Counting Distinct Elements Under Person-Level Differential Privacy
Authors: Thomas Steinke; Alexander Knop
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: We study the problem of counting the number of distinct elements in a dataset subject to the constraint of differential privacy. 
We consider the challenging setting of person-level DP (a.k.a. user-level DP) where each person may contribute an unbounded number of items and hence the sensitivity is unbounded.

Our approach is to compute a bounded-sensitivity version of this query, which reduces to solving a max-flow problem. 
The sensitivity bound is optimized to balance the noise we must add to privatize the answer against the error of the approximation of the bounded-sensitivity query to the true number of unique elements.
Privacy Auditing with One (1) Training Run
Authors: Thomas Steinke; Milad Nasr; Matthew Jagielski
Venue: NeurIPS 2023 (NeurIPS 2023 oral)

Abstract: We propose a scheme for auditing differentially private machine learning systems with a single training run. This exploits the parallelism of being able to add or remove multiple training examples independently. We analyze this using the connection between differential privacy and statistical generalization, which avoids the cost of group privacy. Our auditing scheme requires minimal assumptions about the algorithm and can be applied in the black-box or white-box setting. We demonstrate the effectiveness of our framework by applying it to DP-SGD, where we can achieve meaningful empirical privacy lower bounds by training only one model. In contrast, standard methods would require training hundreds of models.
Faster Discrete Convex Function Minimization with Predictions: The M-Convex Case
Authors: Taihei Oki; Shinsaku Sakaue
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: Recent years have seen a growing interest in accelerating optimization algorithms with machine-learned predictions. Sakaue and Oki (NeurIPS 2022) have developed a general framework that warm-starts the *L-convex function minimization* method with predictions, revealing the idea's usefulness for various discrete optimization problems. In this paper, we present a framework for using predictions to accelerate *M-convex function minimization*, thus complementing previous research and extending the range of discrete optimization algorithms that can benefit from predictions. Our framework is particularly effective for an important subclass called *laminar convex minimization*, which appears in many operations research applications. Our methods can improve time complexity bounds upon the best worst-case results by using predictions and even have potential to go beyond a lower-bound result.
Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examples
Authors: Hao Sun; Alihan Hüyük; Daniel Jarrett; Mihaela van der Schaar
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: Learning controllers with offline data in decision-making systems is an essential area of research due to its potential to reduce the risk of applications in real-world systems. However, in responsibility-sensitive settings such as healthcare, decision accountability is of paramount importance, yet has not been adequately addressed by the literature.
This paper introduces the Accountable Offline Controller (AOC) that employs the offline dataset as the Decision Corpus and performs accountable control based on a tailored selection of examples, referred to as the Corpus Subset. AOC operates effectively in low-data scenarios, can be extended to the strictly offline imitation setting, and displays qualities of both conservation and adaptability.
We assess AOC's performance in both simulated and real-world healthcare scenarios, emphasizing its capability to manage offline control tasks with high levels of performance while maintaining accountability.
KD-Zero: Evolving Knowledge Distiller for Any Teacher-Student Pairs
Authors: Lujun Li; Peijie Dong; Anggeng Li; Zimian Wei; Yang Ya
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: Knowledge distillation (KD) has emerged as an effective technique for compressing models that can enhance the lightweight model.  Conventional KD methods propose various designs to allow student model to imitate the teacher better.  However, these handcrafted KD designs heavily rely on expert knowledge and may be sub-optimal for various teacher-student pairs.  In this paper, we present a novel framework, KD-Zero, which utilizes evolutionary search to automatically discover promising distiller  from scratch for any teacher-student architectures.  Specifically, we first decompose the generalized distiller into knowledge transformations, distance functions, and loss weights.  Then,  we construct our distiller search space by selecting advanced operations for these three components.  With sharpness and represent gap as fitting objectives, we evolve candidate populations and generate better distillers by crossover and mutation.  To ensure efficient searching, we employ the loss-rejection protocol, search space shrinkage, and proxy settings during the search process.  In this manner, the discovered distiller can address the capacity gap and cross-architecture challenges for any teacher-student pairs in the final distillation stage.  Comprehensive experiments reveal that KD-Zero consistently outperforms other state-of-the-art methods across diverse architectures on classification, detection, and segmentation tasks.  Noticeably, we provide some practical insights in designing the distiller by analyzing the distiller discovered.  Codes are available in supplementary materials.
No Regrets: Investigating and Improving Regret Approximations for Curriculum Discovery
Authors: Alexander Rutherford; Michael Beukman; Timon Willi; Bruno Lacerda; Nick Hawes; Jakob Nicolaus Foerster
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: What data or environments to use for training to improve downstream performance is a longstanding and very topical question in reinforcement learning. 
In particular, Unsupervised Environment Design (UED) methods have gained recent attention as their adaptive curricula promise to enable agents to be robust to in- and out-of-distribution tasks.
This work investigates how existing UED methods select training environments, focusing on task prioritisation metrics.
Surprisingly, despite methods aiming to maximise regret in theory, the practical approximations do not correlate with regret but with success rate.
As a result, a significant portion of an agent's experience comes from environments it has already mastered, offering little to no contribution toward enhancing its abilities. Put differently, current methods fail to predict intuitive measures of *learnability*. Specifically, they are unable to consistently identify those scenarios that the agent can sometimes solve, but not always.
Based on our analysis, we develop a method that directly trains on scenarios with high learnability. This simple and intuitive approach outperforms existing UED methods in several binary-outcome environments, including the standard domain of Minigrid and a novel setting closely inspired by a real-world robotics problem. 
We further introduce a new adversarial evaluation procedure for directly measuring robustness, closely mirroring the conditional value at risk (CVaR).
We open-source all our code and present visualisations of final policies here: https://github.com/amacrutherford/sampling-for-learnability.
Autoregressive Image Diffusion: Generation of Image Sequence and Application in MRI
Authors: Guanxiong Luo; Shoujin Huang; Martin Uecker
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: Magnetic resonance imaging (MRI) is a widely used non-invasive imaging modality. However, a persistent challenge  lies in balancing image quality with imaging speed. This trade-off is primarily constrained by k-space measurements, which traverse specific trajectories in the spatial Fourier domain (k-space). These measurements are often undersampled to shorten acquisition times, resulting in image artifacts and compromised quality. Generative models learn image distributions and can be used to reconstruct high-quality images from undersampled k-space data. In this work, we present the autoregressive image diffusion (AID) model for image sequences and use it to sample the posterior for accelerated MRI reconstruction. The algorithm incorporates both undersampled k-space and pre-existing information. Models trained with fastMRI dataset are evaluated comprehensively. The results show that the AID model can robustly generate sequentially coherent image sequences. In MRI applications, the AID can outperform the standard diffusion model and reduce hallucinations, due to the learned inter-image dependencies. The project code is available at https://github.com/mrirecon/aid.
The Implicit Bias of Gradient Descent on Separable Multiclass Data
Authors: Hrithik Ravi; Clayton Scott; Daniel Soudry; Yutong Wang
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: Implicit bias describes the phenomenon where optimization-based training algorithms, without explicit regularization, show a preference for simple estimators even when more complex estimators have equal objective values. Multiple works have developed the theory of implicit bias for binary classification under the assumption that the loss satisfies an *exponential tail property*. However, there is a noticeable gap in analysis for multiclass classification, with only a handful of results which themselves are restricted to the cross-entropy loss. In this work, we employ the framework of Permutation Equivariant and Relative Margin-based (PERM) losses [Wang and Scott, 2024] to introduce a multiclass extension of the exponential tail property. This class of losses includes not only cross-entropy but also other losses. Using this framework, we extend the implicit bias result of Soudry et al. [2018] to multiclass classification. Furthermore, our proof techniques closely mirror those of the binary case, thus illustrating the power of the PERM framework for bridging the binary-multiclass gap.
How many classifiers do we need?
Authors: Hyunsuk Kim; Liam Hodgkinson; Ryan Theisen; Michael W. Mahoney
Venue: NeurIPS 2024 (NeurIPS 2024 poster)

Abstract: As performance gains through scaling data and/or model size experience diminishing returns, it is becoming increasingly popular to turn to ensembling, where the predictions of multiple models are combined to improve accuracy. 
In this paper, we provide a detailed analysis of how the disagreement and the polarization (a notion we introduce and define in this paper) among classifiers relate to the performance gain achieved by aggregating individual classifiers, for majority vote strategies in classification tasks.
We address these questions in the following ways. 
(1) An upper bound for polarization is derived, and we propose what we call a neural polarization law: most interpolating neural network models are 4/3-polarized. Our empirical results not only support this conjecture but also show that polarization is nearly constant for a dataset, regardless of hyperparameters or architectures of classifiers. 
(2) The error rate of the majority vote classifier is considered under restricted entropy conditions, and we present a tight upper bound that indicates that the disagreement is linearly correlated with the error rate, and that the slope is linear in the polarization.
(3) We prove results for the asymptotic behavior of the disagreement in terms of the number of classifiers, which we show can help in predicting the performance for a larger number of classifiers from that of a smaller number. 
Our theoretical findings are supported by empirical results on several image classification tasks with various types of neural networks.