GenAI4ELab/papercli-papers huggingface dataset:
H2O-SDF: Two-phase Learning for 3D Indoor Reconstruction using Object Surface Fields
Authors: Minyoung Park; Mirae Do; Yeon Jae Shin; Jaeseok Yoo; Jongkwang Hong; Joongrock Kim; Chul Lee
Venue: ICLR 2024 (ICLR 2024 spotlight)

Abstract: Advanced techniques using Neural Radiance Fields (NeRF), Signed Distance Fields (SDF), and Occupancy Fields have recently emerged as solutions for 3D indoor scene reconstruction. We introduce a novel two-phase learning approach, H2O-SDF,  that discriminates between object and non-object regions within indoor environments. This method achieves a nuanced balance, carefully preserving the geometric integrity of room layouts while also capturing intricate surface details of specific objects. A cornerstone of our two-phase learning framework is the introduction of the Object Surface Field (OSF), a novel concept designed to mitigate the persistent vanishing gradient problem that has previously hindered the capture of high-frequency details in other methods. Our proposed approach is validated through several experiments that include ablation studies.
RDesign: Hierarchical Data-efficient Representation Learning for Tertiary Structure-based RNA Design
Authors: Cheng Tan; Yijie Zhang; Zhangyang Gao; Bozhen Hu; Siyuan Li; Zicheng Liu; Stan Z. Li
Venue: ICLR 2024 (ICLR 2024 poster)

Abstract: While artificial intelligence has made remarkable strides in revealing the relationship between biological macromolecules' primary sequence and tertiary structure, designing RNA sequences based on specified tertiary structures remains challenging. Though existing approaches in protein design have thoroughly explored structure-to-sequence dependencies in proteins, RNA design still confronts difficulties due to structural complexity and data scarcity. Moreover, direct transplantation of protein design methodologies into RNA design fails to achieve satisfactory outcomes although sharing similar structural components. In this study, we aim to systematically construct a data-driven RNA design pipeline. We crafted a large, well-curated benchmark dataset and designed a comprehensive structural modeling approach to represent the complex RNA tertiary structure. More importantly, we proposed a hierarchical data-efficient representation learning framework that learns structural representations through contrastive learning at both cluster-level and sample-level to fully leverage the limited data. By constraining data representations within a limited hyperspherical space, the intrinsic relationships between data points could be explicitly imposed. Moreover, we incorporated extracted secondary structures with base pairs as prior knowledge to facilitate the RNA design process. Extensive experiments demonstrate the effectiveness of our proposed method, providing a reliable baseline for future RNA design tasks. The source code and benchmark dataset are available at https://github.com/A4Bio/RDesign.
Memory-Assisted Sub-Prototype Mining for Universal Domain Adaptation
Authors: Yuxiang Lai; Yi Zhou; Xinghong Liu; Tao Zhou
Venue: ICLR 2024 (ICLR 2024 poster)

Abstract: Universal domain adaptation aims to align the classes and reduce the feature gap between the same category of the source and target domains. The target private category is set as the unknown class during the adaptation process, as it is not included in the source domain. However, most existing methods overlook the intra-class structure within a category, especially in cases where there exists significant concept shift between the samples belonging to the same category. When samples with large concept shift are forced to be pushed together, it may negatively affect the adaptation performance. Moreover, from the interpretability aspect, it is unreasonable to align visual features with significant differences, such as fighter jets and civil aircraft, into the same category. Unfortunately, due to such semantic ambiguity and annotation cost, categories are not always classified in detail, making it difficult for the model to perform precise adaptation. To address these issues, we propose a novel Memory-Assisted Sub-Prototype Mining (MemSPM) method that can learn the differences between samples belonging to the same category and mine sub-classes when there exists significant concept shift between them. By doing so, our model learns a more reasonable feature space that enhances the transferability and reflects the inherent differences among samples annotated as the same category. We evaluate the effectiveness of our MemSPM method over multiple scenarios, including UniDA, OSDA, and PDA. Our method achieves state-of-the-art performance on four benchmarks in most cases.
Large Language Models as Tool Makers
Authors: Tianle Cai; Xuezhi Wang; Tengyu Ma; Xinyun Chen; Denny Zhou
Venue: ICLR 2024 (ICLR 2024 poster)

Abstract: Recent research has highlighted the potential of large language models (LLMs)
to improve their problem-solving capabilities with the aid of suitable external
tools. In our work, we further advance this concept by introducing a closed-
loop framework, referred to as LLMs A s Tool Makers (LATM), where LLMs
create their own reusable tools for problem-solving. Our approach consists of two
phases: 1) tool making: an LLM acts as the tool maker that crafts tools for a set
of tasks, where a tool is implemented as a Python utility function. 2) tool using:
another LLM acts as the tool user, which applies the tool built by the tool maker
for problem-solving. The tool user can be either the same or a different LLM
from the tool maker. On the problem-solving server side, tool-making enables
continual tool generation and caching as new requests emerge. This framework
enables subsequent requests to access cached tools via their corresponding APIs,
enhancing the efficiency of task resolution. Beyond enabling LLMs to create their
own tools, our framework also uncovers intriguing opportunities to optimize the
serving cost of LLMs: Recognizing that tool-making requires more sophisticated
capabilities, we assign this task to a powerful, albeit resource-intensive, model.
Conversely, the simpler tool-using phase is delegated to a lightweight model. This
strategic division of labor allows the once-off cost of tool-making to be spread
over multiple instances of tool-using, significantly reducing average costs while
maintaining strong performance. Furthermore, our method offers a functional
cache through the caching and reuse of tools, which stores the functionality of
a class of requests instead of the natural language responses from LLMs, thus
extending the applicability of the conventional cache mechanism. We evaluate
our approach across various complex reasoning tasks, including Big-Bench tasks.
With GPT-4 as the tool maker and GPT-3.5 as the tool user, LATM demonstrates
performance equivalent to using GPT-4 for both roles, but with a significantly
reduced inference cost.
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
Authors: Zhibin Gou; Zhihong Shao; Yeyun Gong; yelong shen; Yujiu Yang; Nan Duan; Weizhu Chen
Venue: ICLR 2024 (ICLR 2024 poster)

Abstract: Recent developments in large language models (LLMs) have been impressive. However, these models sometimes show inconsistencies and problematic behavior, such as hallucinating facts, generating flawed code, or creating offensive and toxic content. Unlike these models, humans typically utilize external tools to cross-check and refine their initial content, like using a search engine for fact-checking, or a code interpreter for debugging. Inspired by this observation, we introduce a framework called CRITIC that allows LLMs, which are essentially “black boxes” to validate and progressively amend their own outputs in a manner similar to human interaction with tools. More specifically, starting with an initial output, CRITIC interacts with appropriate tools to evaluate certain aspects of the text, and then revises the output based on the feedback obtained during this validation process. Comprehensive evaluations involving free-form question answering, mathematical program synthesis, and toxicity reduction demonstrate that CRITIC consistently enhances the performance of LLMs. Meanwhile, our research highlights the crucial importance of external feedback in promoting the ongoing self-improvement of LLMs.
Sample-Efficient Quality-Diversity by Cooperative Coevolution
Authors: Ke Xue; Ren-Jian Wang; Pengyi Li; Dong Li; Jianye HAO; Chao Qian
Venue: ICLR 2024 (ICLR 2024 spotlight)

Abstract: Quality-Diversity (QD) algorithms, as a subset of evolutionary algorithms, have emerged as a powerful optimization paradigm with the aim of generating a set of high-quality and diverse solutions. Although QD has demonstrated competitive performance in reinforcement learning, its low sample efficiency remains a significant impediment for real-world applications. Recent research has primarily focused on augmenting sample efficiency by refining selection and variation operators of QD. However, one of the less considered yet crucial factors is the inherently large-scale issue of the QD optimization problem. In this paper, we propose a novel Cooperative Coevolution QD (CCQD) framework, which decomposes a policy network naturally into two types of layers, corresponding to representation and decision respectively, and thus simplifies the problem significantly. The resulting two (representation and decision) subpopulations are coevolved cooperatively. CCQD can be implemented with different selection and variation operators. Experiments on several popular tasks within the QDAX suite demonstrate that an instantiation of CCQD achieves approximately a 200% improvement in sample efficiency.
Geometrically Aligned Transfer Encoder for Inductive Transfer in Regression Tasks
Authors: Sung Moon Ko; Sumin Lee; Dae-Woong Jeong; Woohyung Lim; Sehui Han
Venue: ICLR 2024 (ICLR 2024 poster)

Abstract: Transfer learning is a crucial technique for handling a small amount of data that is potentially related to other abundant data. However, most of the existing methods are focused on classification tasks using images and language datasets. Therefore, in order to expand the transfer learning scheme to regression tasks, we propose a novel transfer technique based on differential geometry, namely the Geometrically Aligned Transfer Encoder (${\it GATE}$). In this method, we interpret the latent vectors from the model to exist on a Riemannian curved manifold. We find a proper diffeomorphism between pairs of tasks to ensure that every arbitrary point maps to a locally flat coordinate in the overlapping region, allowing the transfer of knowledge from the source to the target data. This also serves as an effective regularizer for the model to behave in extrapolation regions. In this article, we demonstrate that ${\it GATE}$ outperforms conventional methods and exhibits stable behavior in both the latent space and extrapolation regions for various molecular graph datasets.
Zero and Few-shot Semantic Parsing with Ambiguous Inputs
Authors: Elias Stengel-Eskin; Kyle Rawlins; Benjamin Van Durme
Venue: ICLR 2024 (ICLR 2024 poster)

Abstract: Despite the frequent challenges posed by ambiguity when representing meaning via natural language, it is often ignored or deliberately removed in tasks mapping language to formally-designed representations, which generally assume a one-to-one mapping between linguistic and formal representations. 
We attempt to address this shortcoming by introducing AmP, a framework, dataset, and challenge for translating ambiguous natural language to formal representations like logic and code. 
We define templates and generate data for five well-documented linguistic ambiguities.
Using AmP, we investigate how several few-shot text-to-code systems handle ambiguity, introducing three new metrics.
We find that large pre-trained models perform poorly at capturing the distribution of possible meanings without deliberate instruction.
However, models are able to capture the distribution well when ambiguity is attested in their inputs. 
These results motivate a call for including ambiguity explicitly in datasets and promote considering the distribution of possible outputs when evaluating systems. We release our data and code.
Graph Generation with  $K^2$-trees
Authors: Yunhui Jang; Dongwoo Kim; Sungsoo Ahn
Venue: ICLR 2024 (ICLR 2024 poster)

Abstract: Generating graphs from a target distribution is a significant challenge across many domains, including drug discovery and social network analysis. In this work, we introduce a novel graph generation method leveraging $K^2$ representation, originally designed for lossless graph compression. The $K^2$ representation enables compact generation while concurrently capturing an inherent hierarchical structure of a graph. In addition, we make contributions by (1) presenting a sequential $K^2$ representation that incorporates pruning, flattening, and tokenization processes and (2) introducing a Transformer-based architecture designed to generate the sequence by incorporating a specialized tree positional encoding scheme. Finally, we extensively evaluate our algorithm on four general and two molecular graph datasets to confirm its superiority for graph generation.
Manifold Preserving Guided Diffusion
Authors: Yutong He; Naoki Murata; Chieh-Hsin Lai; Yuhta Takida; Toshimitsu Uesaka; Dongjun Kim; Wei-Hsiang Liao; Yuki Mitsufuji; J Zico Kolter; Ruslan Salakhutdinov; Stefano Ermon
Venue: ICLR 2024 (ICLR 2024 poster)

Abstract: Despite the recent advancements, conditional image generation still faces challenges of cost, generalizability, and the need for task-specific training. In this paper, we propose Manifold Preserving Guided Diffusion (MPGD), a training-free conditional generation framework that leverages pretrained diffusion models and off-the-shelf neural networks with minimal additional inference cost for a broad range of tasks. Specifically, we leverage the manifold hypothesis to refine the guided diffusion steps and introduce a shortcut algorithm in the process. We then propose two methods for on-manifold training-free guidance using pre-trained autoencoders and demonstrate that our shortcut inherently preserves the manifolds when applied to latent diffusion models. Our experiments show that MPGD is efficient and effective for solving a variety of conditional generation applications in low-compute settings, and can consistently offer up to 3.8× speed-ups with the same number of diffusion steps while maintaining high sample quality compared to the baselines.
Neural Common Neighbor with Completion for Link Prediction
Authors: Xiyuan Wang; Haotong Yang; Muhan Zhang
Venue: ICLR 2024 (ICLR 2024 poster)

Abstract: In this work, we propose a novel link prediction model and further boost it by studying graph incompleteness. First, We introduce MPNN-then-SF, an innovative architecture leveraging structural feature (SF) to guide MPNN's representation pooling, with its implementation, namely Neural Common Neighbor (NCN). NCN exhibits superior expressiveness and scalability compared with existing models, which can be classified into two categories: SF-then-MPNN, augmenting MPNN's input with SF, and SF-and-MPNN, decoupling SF and MPNN. Second, we investigate the impact of graph incompleteness---the phenomenon that some links are unobserved in the input graph---on SF, like the common neighbor. Through dataset visualization, we observe that incompleteness reduces common neighbors and induces distribution shifts, significantly affecting model performance. To address this issue, we propose to use a link prediction model to complete the common neighbor structure. Combining this method with NCN, we propose Neural Common Neighbor with Completion (NCNC). NCN and NCNC outperform recent strong baselines by large margins, and NCNC further surpasses state-of-the-art models in standard link prediction benchmarks.
Latent Intuitive Physics: Learning to Transfer Hidden Physics from A 3D Video
Authors: Xiangming Zhu; Huayu Deng; Haochen Yuan; Yunbo Wang; Xiaokang Yang
Venue: ICLR 2024 (ICLR 2024 poster)

Abstract: We introduce latent intuitive physics, a transfer learning framework for physics simulation that can infer hidden properties of fluids from a single 3D video and simulate the observed fluid in novel scenes. Our key insight is to use latent features drawn from a learnable prior distribution conditioned on the underlying particle states to capture the invisible and complex physical properties. To achieve this, we train a parametrized prior learner given visual observations to approximate the visual posterior of inverse graphics, and both the particle states and the visual posterior are obtained from a learned neural renderer. The converged prior learner is embedded in our probabilistic physics engine, allowing us to perform novel simulations on unseen geometries, boundaries, and dynamics without knowledge of the true physical parameters. We validate our model in three ways: (i) novel scene simulation with the learned visual-world physics, (ii) future prediction of the observed fluid dynamics, and (iii) supervised particle simulation. Our model demonstrates strong performance in all three tasks.
Privacy-Preserving In-Context Learning for Large Language Models
Authors: Tong Wu; Ashwinee Panda; Jiachen T. Wang; Prateek Mittal
Venue: ICLR 2024 (ICLR 2024 poster)

Abstract: In-context learning (ICL) is an important capability of Large Language Models (LLMs), enabling these models to dynamically adapt based on specific, in-context exemplars, thereby improving accuracy and relevance.
However, LLM's responses may leak the sensitive private information contained in in-context exemplars. 
To address this challenge, we propose Differentially Private In-context Learning (DP-ICL), a general paradigm for privatizing ICL tasks. 
The key idea for DP-ICL paradigm is generating differentially private responses through a noisy consensus among an ensemble of LLM's responses based on disjoint exemplar sets. 
Based on the general paradigm of DP-ICL, we instantiate several techniques showing how to privatize ICL for text classification and language generation. 
We experiment on four text classification benchmarks and two language generation tasks, and our empirical findings suggest that our DP-ICL achieves a strong utility-privacy tradeoff.
Masked Distillation Advances Self-Supervised Transformer Architecture Search
Authors: Caixia Yan; Xiaojun Chang; Zhihui Li; Lina Yao; Minnan Luo; Qinghua Zheng
Venue: ICLR 2024 (ICLR 2024 poster)

Abstract: Transformer architecture search (TAS) has achieved remarkable progress in automating the neural architecture design process of vision transformers. Recent TAS advancements have discovered outstanding transformer architectures while saving tremendous labor from human experts. However, it is still cumbersome to deploy these methods in real-world applications due to the expensive costs of data labeling under the supervised learning paradigm. To this end, this paper proposes a masked image modelling (MIM) based self-supervised neural architecture search method specifically designed for vision transformers, termed as MaskTAS, which completely avoids the expensive costs of data labeling inherited from supervised learning. Based on the one-shot NAS framework, MaskTAS requires to train various weight-sharing subnets, which can easily diverged without strong supervision in MIM-based self-supervised learning. For this issue, we design the search space of MaskTAS as a siamesed teacher-student architecture to distill knowledge from pre-trained networks, allowing for efficient training of the transformer supernet. To achieve self-supervised transformer architecture search, we further design a novel unsupervised evaluation metric for the evolutionary search algorithm, where each candidate of the student branch is rated by measuring its consistency with the larger teacher network. Extensive experiments demonstrate that the searched architectures can achieve state-of-the-art accuracy on CIFAR-10, CIFAR-100, and ImageNet datasets even without using manual labels. Moreover, the proposed MaskTAS can generalize well to various data domains and tasks by searching specialized transformer architectures in self-supervised manner.
Adaptive Self-training Framework for Fine-grained Scene Graph Generation
Authors: Kibum Kim; Kanghoon Yoon; Yeonjun In; Jinyoung Moon; Donghyun Kim; Chanyoung Park
Venue: ICLR 2024 (ICLR 2024 poster)

Abstract: Scene graph generation (SGG) models have suffered from inherent problems regarding the benchmark datasets such as the long-tailed predicate distribution and missing annotation problems. In this work, we aim to alleviate the long-tailed problem of SGG by utilizing unannotated triplets. To this end, we introduce a **S**elf-**T**raining framework for **SGG** **(ST-SGG)** that assigns pseudo-labels for unannotated triplets based on which the SGG models are trained. While there has been significant progress in self-training for image recognition, designing a self-training framework for the SGG task is more challenging due to its inherent nature such as the semantic ambiguity and the long-tailed distribution of predicate classes. Hence, we propose a novel pseudo-labeling technique for SGG, called **C**lass-specific **A**daptive **T**hresholding with **M**omentum **(CATM)**, which is a model-agnostic framework that can be applied to any existing SGG models. Furthermore, we devise a graph structure learner (GSL) that is beneficial when adopting our proposed self-training framework to the state-of-the-art message-passing neural network (MPNN)-based SGG models. Our extensive experiments verify the effectiveness of ST-SGG on various SGG models, particularly in enhancing the performance on fine-grained predicate classes.
Exploring Effective Stimulus Encoding via Vision System Modeling for Visual Prostheses
Authors: Chuanqing Wang; Di Wu; Chaoming Fang; Jie Yang; Mohamad Sawan
Venue: ICLR 2024 (ICLR 2024 poster)

Abstract: Visual prostheses are potential devices to restore vision for blind people, which highly depends on the quality of stimulation patterns of the implanted electrode array. However, existing processing frameworks prioritize the generation of stimulation while disregarding the potential impact of restoration effects and fail to assess the quality of the generated stimulation properly. In this paper, we propose for the first time an end-to-end visual prosthesis framework (StimuSEE) that generates stimulation patterns with proper quality verification using V1 neuron spike patterns as supervision. StimuSEE consists of a retinal network to predict the stimulation pattern, a phosphene model, and a primary vision system network (PVS-net) to simulate the signal processing from the retina to the visual cortex and predict the firing rate of V1 neurons. Experimental results show that the predicted stimulation shares similar patterns to the original scenes, whose different stimulus amplitudes contribute to a similar firing rate with normal cells. Numerically, the predicted firing rate and the recorded response of normal neurons achieve a Pearson correlation coefficient of 0.78.