GenAI4ELab/papercli-papers huggingface dataset:
Credible, Sealed-bid, Optimal Repeated Auctions With Differentiable Economics
Authors: Davidson Cheng; Yang Hong; Ian Miers; John P Dickerson; Michael Curry
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: Online advertisement auctions happen billions of times per day.  Bidders in auctions strategize to improve their own utility, subject to published auctions' rules.  Yet, bidders may not know that an auction has been run as promised.  A credible auction is one in which bidders can trust the auctioneer to run its allocation and pricing mechanisms as promised.  It is known that, assuming no communication between bidders, no credible, sealed-bid, and incentive compatible (aka ``truth-telling'' or otherwise truthful-participation-incentivizing) mechanism can exist. In reality, bidders can certainly communicate, so what happens if we relax this (typically unrealistic) constraint? 

In this work, we propose a framework incorporating cryptography to allow computationally-efficient, credible, revenue-maximizing (aka ``optimal'') auctions in a repeated auction setting. Our contribution is two-fold: first, we introduce a protocol for running repeated auctions with a verification scheme, and we show such a protocol can eliminate the auctioneer's incentive to deviate while costing negligible additional computation. Secondly, we provide a method for training optimal auctions under uncertain bidder participation profiles, which generalizes our protocol to a much wider class of auctions in the online ad market. Our empirical results show strong support for both the theory and competency of the proposed method.
What Do Self-Supervised Vision Transformers Learn?
Authors: Namuk Park; Wonjae Kim; Byeongho Heo; Taekyung Kim; Sangdoo Yun
Venue: ICLR 2023 (ICLR 2023 poster)

Abstract: We present a comparative study on how and why contrastive learning (CL) and masked image modeling (MIM) differ in their representations and in their performance of downstream tasks. In particular, we demonstrate that self-supervised Vision Transformers (ViTs) have the following properties: (1) CL trains self-attentions to capture longer-range global patterns than MIM, such as the shape of an object, especially in the later layers of the ViT architecture. This CL property helps ViTs linearly separate images in their representation spaces. However, it also makes the self-attentions collapse into homogeneity for all query tokens and heads. Such homogeneity of self-attention reduces the diversity of representations, worsening scalability and dense prediction performance. (2) CL utilizes the low-frequency signals of the representations, but MIM utilizes high-frequencies. Since low- and high-frequency information respectively represent shapes and textures, CL is more shape-oriented and MIM more texture-oriented. (3) CL plays a crucial role in the later layers, while MIM mainly focuses on the early layers. Upon these analyses, we find that CL and MIM can complement each other and observe that even the simplest harmonization can help leverage the advantages of both methods. The code is available at https://github.com/naver-ai/cl-vs-mim.
Effective Offline Reinforcement Learning via Conservative State Value Estimation
Authors: Liting Chen; Jie Yan; Lu Wang; Qingwei Lin; Dongmei Zhang; Saravan Rajmohan; Thomas Moscibroda
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: Offline RL promises to learn effective policies from static experience datasets without further interaction, which expect to perform well in the online environment. However, it faces up to a major challenge of value over-estimation introduced by the distributional drift between the dataset and the current learned policy, which leads to learning failure in practice. The common approach is to add a penalty term to reward or value estimation in the Bellman iterations, which has given rise to a number of successful algorithms such as CQL. Meanwhile, to avoid extrapolation on unseen states, existing methods focus on conservative Q-function estimation. In this paper, we propose CSVE, a new approach that directly imposes penalty on out-of-distribution states. We prove that for the evaluated policy, our conservative state value estimation satisfies: (1) over the state distribution that samples penalizing states, it lower bounds the true values in expectation, and (2) over the marginal state distribution of data, it is no more than the true values in expectation plus a constant decided by sampling error. Further, we develop a practical actor-critic algorithm in which the critic does the conservative value estimation by additionally sampling and penalizing the states 'around' the dataset, while the actor applies advantage weighted updates to improve the policy. We evaluate in classic continual control tasks of D4RL, showing that our method performs better than the conservative Q-function learning methods (e.g., CQL) and is strongly competitive among recent SOTA methods.
Do We Really Need Complicated Model Architectures For Temporal Networks?
Authors: Weilin Cong; Si Zhang; Jian Kang; Baichuan Yuan; Hao Wu; Xin Zhou; Hanghang Tong; Mehrdad Mahdavi
Venue: ICLR 2023 (ICLR 2023 notable top 5%)

Abstract: Recurrent neural network (RNN) and self-attention mechanism (SAM) are the de facto methods to extract spatial-temporal information for temporal graph learning. Interestingly, we found that although both RNN and SAM could lead to a good performance, in practice neither of them is always necessary. In this paper, we propose GraphMixer, a conceptually and technically simple architecture that consists of three components: (1) a link-encoder that is only based on multi-layer perceptrons (MLP) to summarize the information from temporal links, (2) a node-encoder that is only based on neighbor mean-pooling to summarize node information, and (3) an MLP-based link classifier that performs link prediction based on the outputs of the encoders. Despite its simplicity, GraphMixer attains an outstanding performance on temporal link prediction benchmarks with faster convergence and better generalization performance. These results motivate us to rethink the importance of simpler model architecture.
Reprogramming Large Pretrained Language Models for Antibody Sequence Infilling
Authors: Igor Melnyk; Vijil Chenthamarakshan; Pin-Yu Chen; Payel Das; Amit Dhurandhar; Inkit Padhi; Devleena Das
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: Antibodies comprise the most versatile class of binding molecules, with numerous applications in biomedicine. Therapeutic antibody development requires designing novel and diverse  sequences with improved properties, while maintaining the structural consistency. Computational design of antibodies involves unusual challenges relative to designing  other classes of proteins, as antibodies comprise multiple long, variable, and unstructured loops at the complementarity-determining region (CDR) that determine the antigen binding affinity and specificity of an antibody. Recently, deep language models  and graph neural networks have shown impressive success in antibody sequence generation. Since only a limited number of antibody structures are known, training a model using this limited data can lead to degraded performance, particularly lacking diversity in the generated samples.  To address such issues, we leverage the method of Model Reprogramming (MR) here,  which focuses on repurposing pretrained machine learning models for target domain tasks with scarce data, where it may be difficult to train a high-performing model from scratch. Prior works in MR have primarily focused on classification-based tasks. We extend the capabilities of reprogramming beyond classification tasks, and towards a more complex problem of antibody sequence generation. Specifically, we introduce Reprogramming for Protein Sequence Infilling, a framework in which pretrained natural language models are repurposed for  protein sequence infilling via reprogramming, to infill protein sequence templates as a method of novel protein generation. For variable CDR sequence design, we formulate the task as  text infilling that uses the constant region of an antibody as the sequence template. Results on  antibody design benchmarks show that our reprogrammed model on low resourced antibody sequence dataset provides highly diverse CDR sequences, up to more than a two-fold increase of diversity over the baselines, without losing  structural integrity and naturalness. The performance benefit of the reprogrammed model learning only from antibody sequences is more evident for longer CDR design or for multiple loop infilling at once, compared to existing graph-based models that require additional structural information.  The generated sequences also demonstrate  enhanced antigen binding specificity or virus neutralization ability.
Image as Set of Points
Authors: Xu Ma; Yuqian Zhou; Huan Wang; Can Qin; Bin Sun; Chang Liu; Yun Fu
Venue: ICLR 2023 (ICLR 2023 notable top 5%)

Abstract: 
What is an image, and how to extract latent features? 
Convolutional Networks (ConvNets) consider an image as organized pixels in a rectangular shape and extract features via convolutional operation in a local region; Vision Transformers (ViTs) treat an image as a sequence of patches and extract features via attention mechanism in a global range. In this work, we introduce a straightforward and promising paradigm for visual representation, which is called Context Clusters. Context clusters (CoCs) view an image as a set of unorganized points and extract features via a simplified clustering algorithm. In detail, each point includes the raw feature (e.g., color) and positional information (e.g., coordinates), and a simplified clustering algorithm is employed to group and extract deep features hierarchically. Our CoCs are convolution- and attention-free, only relying on clustering algorithm for spatial interaction. Owing to the simple design, we show CoCs endow gratifying interpretability via the visualization of the clustering process.  
Our CoCs aim at providing a new perspective on image and visual representation, which may enjoy broad applications in different domains and exhibit profound insights. Even though we are not targeting SOTA performance, COCs still achieve comparable or even better performance than ConvNets or ViTs on several benchmarks.
Information-Theoretic Underpinnings of Generalization and Translation in Emergent Communication
Authors: Mycal Tucker; Julie Shah; Roger P. Levy; Noga Zaslavsky
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: Traditional emergent communication (EC) methods often fail to generalize to novel settings or align with representations of natural language. While these limitations may at first appear unrelated, in this work, we show how controlling the Information Bottleneck (IB) tradeoff between complexity and informativeness (a principle thought to guide human languages) helps to address both of these problems in EC. Specifically, we build on VQ-VIB, a recently proposed method for training EC agents while controlling the IB tradeoff, in addition to maximizing agents' utility. We find that increasing informativeness, which is a task-agnostic measure of how well a listener can reconstruct a speaker's meaning, allows EC agents to better generalize to novel settings and more challenging tasks. At the same time, in translation experiments between EC and English, we find that increasing EC informativeness only improves team performance up to a certain threshold, corresponding to the English informativeness-complexity tradeoff. Jointly, our results indicate the importance of training EC systems while controlling the informativeness-complexity tradeoff to simultaneously support improved self-play performance and human-agent interaction.
MBrain: A Multi-channel Self-Supervised Learning Framework for Brain Signals
Authors: Donghong Cai; Junru Chen; Yang Yang; Teng Liu; Yafeng Li
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: Brain signals are important quantitative data for understanding physiological activities and diseases of human brain. Meanwhile, rapidly developing deep learning methods offer a wide range of opportunities for better modeling brain signals, which has attracted considerable research efforts recently. Most existing studies pay attention to supervised learning methods, which, however, require high-cost clinical labels. In addition, the huge difference in the clinical patterns of brain signals measured by invasive (e.g., SEEG) and non-invasive (e.g., EEG) methods leads to the lack of a unified method. To handle the above issues, in this paper, we propose to study the self-supervised learning (SSL) framework for brain signals that can be applied to pre-train either SEEG or EEG data. Intuitively, brain signals, generated by the firing of neurons, are transmitted among different connecting structures in human brain. Inspired by this, we propose to learn implicit spatial and temporal correlations between different channels (i.e., contacts of the electrode, corresponding to different brain areas) as the cornerstone for uniformly modeling different types of brain signals. Specifically, we capture the temporal correlation by designing the delayed-time-shift prediction task; we represent the spatial correlation by a graph structure, which is built with the goal to maximize the mutual information of each channel and its correlated ones. We further theoretically prove that our design can lead to a better predictive representation. Extensive experiments of seizure detection on both EEG and SEEG large-scale real- world datasets demonstrate our model outperforms several state-of-the-art time series SSL and unsupervised models.
Understanding Why Generalized Reweighting Does Not Improve Over ERM
Authors: Runtian Zhai; Chen Dan; J Zico Kolter; Pradeep Kumar Ravikumar
Venue: ICLR 2023 (ICLR 2023 poster)

Abstract: Empirical risk minimization (ERM) is known to be non-robust in practice to distributional shift where the training and the test distributions are different. A suite of approaches, such as importance weighting, and variants of distributionally robust optimization (DRO), have been proposed to solve this problem. But a line of recent work has empirically shown that these approaches do not significantly improve over ERM in real applications with distribution shift. The goal of this work is to obtain a comprehensive theoretical understanding of this intriguing phenomenon. We first posit the class of Generalized Reweighting (GRW) algorithms, as a broad category of approaches that iteratively update model parameters based on iterative reweighting of the training samples. We show that when overparameterized models are trained under GRW, the resulting models are close to that obtained by ERM. We also show that adding small regularization which does not greatly affect the empirical training accuracy does not help. Together, our results show that a broad category of what we term GRW approaches are not able to achieve distributionally robust generalization. Our work thus has the following sobering takeaway: to make progress towards distributionally robust generalization, we either have to develop non-GRW approaches, or perhaps devise novel classification/regression loss functions that are adapted to GRW approaches.
Attribute Alignment and Enhancement for Generalized Zero-Shot Learning
Authors: Nannan Lu; MingKai Qiu
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: Generalized zero-shot learning (GZSL) aims to recognize both seen and unseen classes, which challenges the generalization ability of a model. In this paper, we propose a novel approach to fully utilize attributes information, referred to as attribute alignment and enhancement (A3E) network. It contains two modules. First, attribute localization (AL) module utilizes the supervision of class attribute vectors to guide visual localization for attributes through the implicit localization capability within the feature extractor, and the visual features corresponding to the attributes (attribute-visual features) are obtained. Second, enhanced attribute scoring (EAS) module employs the supervision of the attribute word vectors (attribute semantics) to project input attribute visual features to attribute semantic space using Graph Attention Network (GAT). Based on the constructed attribute relation graph (ARG), EAS module generates enhanced representation of attributes. Experiments on standard datasets demonstrate that the enhanced attribute representation greatly improves the classification performance, which helps A3E to achieve state-of-the-art performances in both ZSL and GZSL tasks.
Sampling-based inference for large linear models, with application to linearised Laplace
Authors: Javier Antoran; Shreyas Padhy; Riccardo Barbano; Eric Nalisnick; David Janz; José Miguel Hernández-Lobato
Venue: ICLR 2023 (ICLR 2023 poster)

Abstract: Large-scale linear models are ubiquitous throughout machine learning, with contemporary application as surrogate models for neural network uncertainty quantification; that is, the linearised Laplace method. Alas, the computational cost associated with Bayesian linear models constrains this method's application to small networks, small output spaces and small datasets. We address this limitation by introducing a scalable sample-based Bayesian inference method for conjugate Gaussian multi-output linear models, together with a matching method for hyperparameter (regularisation) selection. Furthermore, we use a classic feature normalisation method (the g-prior) to resolve a previously highlighted pathology of the linearised Laplace method. Together, these contributions allow us to perform linearised neural network inference with ResNet-18 on CIFAR100 (11M parameters, 100 output dimensions × 50k datapoints) and with a U-Net on a high-resolution tomographic reconstruction task (2M parameters, 251k output dimensions).
Joint Gaussian Mixture Model for Versatile Deep Visual Model Explanation
Authors: Zhouyang Xie; Duanbing Chen
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: Post-hoc explanations of deep neural networks improve human understanding on the learned representations, decision-making process and uncertainty of the model with faithfulness. Explaining deep convolutional neural networks(DCNN) is especially challenging, due to the high dimensionality of deep features and the complexity of model inference. Most post-hoc explaining methods serve a single form of explanation, restricting the diversity and consistency of the explanation. This paper proposes joint Gaussian mixture model(JGMM), a probabilistic model jointly models inter-layer deep features and produces faithful and consistent post-hoc explanations. JGMM explains deep features by Gaussian mixture model and inter-layer deep feature relations by posterior distribution on the latent component variables. JGMM enables a versatile explaining framework that unifies interpretable proxy model, global or local explanatory example generation or mining. Experiments are performed on various DCNN image classifiers in comparison with other explaining methods. It shows that JGMM can efficiently produce versatile, consistent, faithful and understandable explanations.
Learning by Distilling Context
Authors: Charlie Victor Snell; Dan Klein; Ruiqi Zhong
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: Language models significantly benefit from context tokens, such as prompts or scratchpads. They perform better when prompted with concrete training examples and abstract statements about the target task (instructions), and they acquire new capabilities to perform complex tasks by generating step-by-step reasoning (scratch-pad) before predicting the final answers. However, they do not internalize these performance gains, which disappear when the context tokens are gone. Consequently, we always need to pay extra computation for this gain, and it is unclear how to transfer the capabilities acquired by context tokens to other tasks, or how to leverage the context tokens when their length exceeds the context window size. Our work proposes to apply context distillation so that a language model can internalize these gains. Concretely, given an input for the target task, we let the model use all relevant context tokens to generate the output, using ``[instructions] + [task-input]'' to predict ``[scratch-pad] + [final answer]''; then we fine-tune the same model to predict the above ``[final answer]'' conditioned on the ``[task-input]'', without seeing the ``[instructions]'' or using the ``[scratch-pad]''. This incentivizes the model to behave as if the context were present, hence updating the parameters to internalize the context information. We show that context distillation can be used as a general method for learning. In particular, we demonstrate that context distillation can effectively internalize 3 types of contexts: 1) abstract task instructions and natural language explanations of why an output is correct or incorrect on Natural-Instructions-V2; 2) step-by-step reasoning on 8-digit addition questions, where we show the model can apply this newly acquired capability to downstream question answering tasks; and 3) concrete training examples on the SPIDER Text-to-SQL dataset, where context distillation outperforms directly learning with gradient descent by 7%.
Towards Semi-Supervised Learning with Non-Random Missing Labels
Authors: Yue Duan; Zhen Zhao; Lei Qi; Luping Zhou; Lei Wang; Yinghuan Shi
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: Semi-supervised learning (SSL) tackles the label missing problem by enabling the effective usage of unlabeled data. While existing SSL methods focus on the traditional setting, a practical and challenging scenario called label Missing Not At Random (MNAR) is usually ignored. In MNAR, the labeled and unlabeled data fall into different class distributions resulting in biased label imputation, which deteriorates the performance of SSL models. In this work, class transition tracking based Pseudo-Rectifying Guidance (PRG) is devised for MNAR. We explore the class-level guidance information obtained by the Markov random walk, which is modeled on a dynamically created graph built over the class tracking matrix. PRG unifies the history information of each class transition caused by the pseudo-rectifying procedure to activate the model's enthusiasm for neglected classes, so as the quality of pseudo-labels on both popular classes and rare classes in MNAR could be improved. We show the superior performance of PRG across a variety of the MNAR scenarios and the conventional SSL setting, outperforming the latest SSL solutions by a large margin. Checkpoints and evaluation code are available at the anonymous link https://anonymous.4open.science/r/PRG4SSL-MNAR-8DE2 while the source code will be available upon paper acceptance.
Oscillation Neural Ordinary Differential Equations
Authors: Muhao Guo; Yang Weng
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: Neural ordinary differential equations (NODEs) have received a lot of attention in recent years due to their memory efficiency. Different from traditional deep learning, it defines a continuous deep learning architecture based on the theory of ordinary differential equations (ODEs), which also improves the interpretability of deep learning. However, it has several obvious limitations, such as a NODE is not a universal approximator, it requires a large number of function evaluations (NFEs), and it has a slow convergence rate. We address these drawbacks by modeling and adding an oscillator to the framework of the NODEs. The oscillator enables the trajectories of our model to cross each other. We prove that our model is a universal approximator, even in the original input space. Due to the presence of oscillators, the flows learned by the model will be simpler, thus our model needs fewer NFEs and has a faster convergence speed. We apply our model to various tasks including classification and time series extrapolation, then compare several metrics including accuracy, NFEs, and convergence speed. The experiments show that our model can achieve better results compared to the existing baselines.