Orionnnn/hub-stats huggingface dataset:
Large Language Models Orchestrating Structured Reasoning Achieve Kaggle
  Grandmaster Level
Authors: name: Antoine Grosnit, name: Alexandre Maraval, name: James Doran, name: Giuseppe Paolo, name: Albert Thomas, name: Refinath Shahul Hameed Nabeezath Beevi, name: Jonas Gonzalez, name: Khyati Khandelwal, name: Ignacio Iacobacci, name: Abdelhakim Benechehab, name: Hamza Cherkaoui, name: Youssef Attia El-Hili, name: Kun Shao, name: Jianye Hao, name: Jun Yao, name: Balazs Kegl, name: Haitham Bou-Ammar, name: Jun Wang
Published: 2024-11-05 23:55:23

We introduce Agent K v1.0, an end-to-end autonomous data science agent
designed to automate, optimise, and generalise across diverse data science
tasks. Fully automated, Agent K v1.0 manages the entire data science life cycle
by learning from experience. It leverages a highly flexible structured
reasoning framework to enable it to dynamically process memory in a nested
structure, effectively learning from accumulated experience stored to handle
complex reasoning tasks. It optimises long- and short-term memory by
selectively storing and retrieving key information, guiding future decisions
based on environmental rewards. This iterative approach allows it to refine
decisions without fine-tuning or backpropagation, achieving continuous
improvement through experiential learning. We evaluate our agent's apabilities
using Kaggle competitions as a case study. Following a fully automated
protocol, Agent K v1.0 systematically addresses complex and multimodal data
science tasks, employing Bayesian optimisation for hyperparameter tuning and
feature engineering. Our new evaluation framework rigorously assesses Agent K
v1.0's end-to-end capabilities to generate and send submissions starting from a
Kaggle competition URL. Results demonstrate that Agent K v1.0 achieves a 92.5\%
success rate across tasks, spanning tabular, computer vision, NLP, and
multimodal domains. When benchmarking against 5,856 human Kaggle competitors by
calculating Elo-MMR scores for each, Agent K v1.0 ranks in the top 38\%,
demonstrating an overall skill level comparable to Expert-level users. Notably,
its Elo-MMR score falls between the first and third quartiles of scores
achieved by human Grandmasters. Furthermore, our results indicate that Agent K
v1.0 has reached a performance level equivalent to Kaggle Grandmaster, with a
record of 6 gold, 3 silver, and 7 bronze medals, as defined by Kaggle's
progression system.
Both Text and Images Leaked! A Systematic Analysis of Multimodal LLM
  Data Contamination
Authors: name: Dingjie Song, name: Sicheng Lai, name: Shunian Chen, name: Lichao Sun, name: Benyou Wang
Published: 2024-11-06 10:44:15

The rapid progression of multimodal large language models (MLLMs) has
demonstrated superior performance on various multimodal benchmarks. However,
the issue of data contamination during training creates challenges in
performance evaluation and comparison. While numerous methods exist for
detecting dataset contamination in large language models (LLMs), they are less
effective for MLLMs due to their various modalities and multiple training
phases. In this study, we introduce a multimodal data contamination detection
framework, MM-Detect, designed for MLLMs. Our experimental results indicate
that MM-Detect is sensitive to varying degrees of contamination and can
highlight significant performance improvements due to leakage of the training
set of multimodal benchmarks. Furthermore, We also explore the possibility of
contamination originating from the pre-training phase of LLMs used by MLLMs and
the fine-tuning phase of MLLMs, offering new insights into the stages at which
contamination may be introduced.
Adaptive Length Image Tokenization via Recurrent Allocation
Authors: name: Shivam Duggal, name: Phillip Isola, name: Antonio Torralba, name: William T. Freeman
Published: 2024-11-04 18:58:01

Current vision systems typically assign fixed-length representations to
images, regardless of the information content. This contrasts with human
intelligence - and even large language models - which allocate varying
representational capacities based on entropy, context and familiarity. Inspired
by this, we propose an approach to learn variable-length token representations
for 2D images. Our encoder-decoder architecture recursively processes 2D image
tokens, distilling them into 1D latent tokens over multiple iterations of
recurrent rollouts. Each iteration refines the 2D tokens, updates the existing
1D latent tokens, and adaptively increases representational capacity by adding
new tokens. This enables compression of images into a variable number of
tokens, ranging from 32 to 256. We validate our tokenizer using reconstruction
loss and FID metrics, demonstrating that token count aligns with image entropy,
familiarity and downstream task requirements. Recurrent token processing with
increasing representational capacity in each iteration shows signs of token
specialization, revealing potential for object / part discovery.
Inference Optimal VLMs Need Only One Visual Token but Larger Models
Authors: name: Kevin Y. Li, name: Sachin Goyal, name: Joao D. Semedo, name: J. Zico Kolter
Published: 2024-11-05 18:54:21

Vision Language Models (VLMs) have demonstrated strong capabilities across
various visual understanding and reasoning tasks. However, their real-world
deployment is often constrained by high latency during inference due to
substantial compute required to process the large number of input tokens
(predominantly from the image) by the LLM. To reduce inference costs, one can
either downsize the LLM or reduce the number of input image-tokens, the latter
of which has been the focus of many recent works around token compression.
However, it is unclear what the optimal trade-off is, as both the factors
directly affect the VLM performance. We first characterize this optimal
trade-off between the number of visual tokens and LLM parameters by
establishing scaling laws that capture variations in performance with these two
factors. Our results reveal a surprising trend: for visual reasoning tasks, the
inference-optimal behavior in VLMs, i.e., minimum downstream error at any given
fixed inference compute, is achieved when using the largest LLM that fits
within the inference budget while minimizing visual token count - often to a
single token. While the token reduction literature has mainly focused on
maintaining base model performance by modestly reducing the token count (e.g.,
5-10times), our results indicate that the compute-optimal inference regime
requires operating under even higher token compression ratios. Based on these
insights, we take some initial steps towards building approaches tailored for
high token compression settings. Code is available at
https://github.com/locuslab/llava-token-compression.
GarVerseLOD: High-Fidelity 3D Garment Reconstruction from a Single
  In-the-Wild Image using a Dataset with Levels of Details
Authors: name: Zhongjin Luo, name: Haolin Liu, name: Chenghong Li, name: Wanghao Du, name: Zirong Jin, name: Wanhu Sun, name: Yinyu Nie, name: Weikai Chen, name: Xiaoguang Han
Published: 2024-11-05 12:30:07

Neural implicit functions have brought impressive advances to the
state-of-the-art of clothed human digitization from multiple or even single
images. However, despite the progress, current arts still have difficulty
generalizing to unseen images with complex cloth deformation and body poses. In
this work, we present GarVerseLOD, a new dataset and framework that paves the
way to achieving unprecedented robustness in high-fidelity 3D garment
reconstruction from a single unconstrained image. Inspired by the recent
success of large generative models, we believe that one key to addressing the
generalization challenge lies in the quantity and quality of 3D garment data.
Towards this end, GarVerseLOD collects 6,000 high-quality cloth models with
fine-grained geometry details manually created by professional artists. In
addition to the scale of training data, we observe that having disentangled
granularities of geometry can play an important role in boosting the
generalization capability and inference accuracy of the learned model. We hence
craft GarVerseLOD as a hierarchical dataset with levels of details (LOD),
spanning from detail-free stylized shape to pose-blended garment with
pixel-aligned details. This allows us to make this highly under-constrained
problem tractable by factorizing the inference into easier tasks, each narrowed
down with smaller searching space. To ensure GarVerseLOD can generalize well to
in-the-wild images, we propose a novel labeling paradigm based on conditional
diffusion models to generate extensive paired images for each garment model
with high photorealism. We evaluate our method on a massive amount of
in-the-wild images. Experimental results demonstrate that GarVerseLOD can
generate standalone garment pieces with significantly better quality than prior
approaches. Project page: https://garverselod.github.io/
Correlation of Object Detection Performance with Visual Saliency and
  Depth Estimation
Authors: name: Matthias Bartolo, name: Dylan Seychell
Published: 2024-11-05 06:34:19

As object detection techniques continue to evolve, understanding their
relationships with complementary visual tasks becomes crucial for optimising
model architectures and computational resources. This paper investigates the
correlations between object detection accuracy and two fundamental visual
tasks: depth prediction and visual saliency prediction. Through comprehensive
experiments using state-of-the-art models (DeepGaze IIE, Depth Anything,
DPT-Large, and Itti's model) on COCO and Pascal VOC datasets, we find that
visual saliency shows consistently stronger correlations with object detection
accuracy (mArho up to 0.459 on Pascal VOC) compared to depth prediction
(mArho up to 0.283). Our analysis reveals significant variations in these
correlations across object categories, with larger objects showing correlation
values up to three times higher than smaller objects. These findings suggest
incorporating visual saliency features into object detection architectures
could be more beneficial than depth information, particularly for specific
object categories. The observed category-specific variations also provide
insights for targeted feature engineering and dataset design improvements,
potentially leading to more efficient and accurate object detection systems.
Controlling Language and Diffusion Models by Transporting Activations
Authors: name: Pau Rodriguez, name: Arno Blaas, name: Michal Klein, name: Luca Zappella, name: Nicholas Apostoloff, name: Marco Cuturi, name: Xavier Suau
Published: 2024-10-30 14:21:33

The increasing capabilities of large generative models and their ever more
widespread deployment have raised concerns about their reliability, safety, and
potential misuse. To address these issues, recent works have proposed to
control model generation by steering model activations in order to effectively
induce or prevent the emergence of concepts or behaviors in the generated
output. In this paper we introduce Activation Transport (AcT), a general
framework to steer activations guided by optimal transport theory that
generalizes many previous activation-steering works. AcT is modality-agnostic
and provides fine-grained control over the model behavior with negligible
computational overhead, while minimally impacting model abilities. We
experimentally show the effectiveness and versatility of our approach by
addressing key challenges in large language models (LLMs) and text-to-image
diffusion models (T2Is). For LLMs, we show that AcT can effectively mitigate
toxicity, induce arbitrary concepts, and increase their truthfulness. In T2Is,
we show how AcT enables fine-grained style control and concept negation.
Sample-Efficient Alignment for LLMs
Authors: name: Zichen Liu, name: Changyu Chen, name: Chao Du, name: Wee Sun Lee, name: Min Lin
Published: 2024-11-03 09:18:28

We study methods for efficiently aligning large language models (LLMs) with
human preferences given budgeted online feedback. We first formulate the LLM
alignment problem in the frame of contextual dueling bandits. This formulation,
subsuming recent paradigms such as online RLHF and online DPO, inherently
quests for sample-efficient algorithms that incorporate online active
exploration. Leveraging insights from bandit theory, we introduce a unified
algorithm based on Thompson sampling and highlight its applications in two
distinct LLM alignment scenarios. The practical agent that efficiently
implements this algorithm, named SEA (Sample-Efficient Alignment), is
empirically validated through extensive experiments across three model scales
(1B, 2.8B, 6.9B) and three preference learning algorithms (DPO, IPO, SLiC). The
results demonstrate that SEA achieves highly sample-efficient alignment with
oracle's preferences, outperforming recent active exploration methods for LLMs.
Additionally, we release the implementation of SEA together with an efficient
codebase designed for online alignment of LLMs, aiming to accelerate future
research in this field.
DreamPolish: Domain Score Distillation With Progressive Geometry
  Generation
Authors: name: Yean Cheng, name: Ziqi Cai, name: Ming Ding, name: Wendi Zheng, name: Shiyu Huang, name: Yuxiao Dong, name: Jie Tang, name: Boxin Shi
Published: 2024-11-03 15:15:01

We introduce DreamPolish, a text-to-3D generation model that excels in
producing refined geometry and high-quality textures. In the geometry
construction phase, our approach leverages multiple neural representations to
enhance the stability of the synthesis process. Instead of relying solely on a
view-conditioned diffusion prior in the novel sampled views, which often leads
to undesired artifacts in the geometric surface, we incorporate an additional
normal estimator to polish the geometry details, conditioned on viewpoints with
varying field-of-views. We propose to add a surface polishing stage with only a
few training steps, which can effectively refine the artifacts attributed to
limited guidance from previous stages and produce 3D objects with more
desirable geometry. The key topic of texture generation using pretrained
text-to-image models is to find a suitable domain in the vast latent
distribution of these models that contains photorealistic and consistent
renderings. In the texture generation phase, we introduce a novel score
distillation objective, namely domain score distillation (DSD), to guide neural
representations toward such a domain. We draw inspiration from the
classifier-free guidance (CFG) in textconditioned image generation tasks and
show that CFG and variational distribution guidance represent distinct aspects
in gradient guidance and are both imperative domains for the enhancement of
texture quality. Extensive experiments show our proposed model can produce 3D
assets with polished surfaces and photorealistic textures, outperforming
existing state-of-the-art methods.
Zebra-Llama: A Context-Aware Large Language Model for Democratizing Rare
  Disease Knowledge
Authors: name: Karthik Soman, name: Andrew Langdon, name: Catalina Villouta, name: Chinmay Agrawal, name: Lashaw Salta, name: Braian Peetoom, name: Gianmarco Bellucci, name: Orion J Buske
Published: 2024-11-04 22:45:52

Rare diseases present unique challenges in healthcare, often suffering from
delayed diagnosis and fragmented information landscapes. The scarcity of
reliable knowledge in these conditions poses a distinct challenge for Large
Language Models (LLMs) in supporting clinical management and delivering precise
patient information underscoring the need for focused training on these 'zebra'
cases. We present Zebra-Llama, a specialized context-aware language model with
high precision Retrieval Augmented Generation (RAG) capability, focusing on
Ehlers-Danlos Syndrome (EDS) as our case study. EDS, affecting 1 in 5,000
individuals, exemplifies the complexities of rare diseases with its diverse
symptoms, multiple subtypes, and evolving diagnostic criteria. By implementing
a novel context-aware fine-tuning methodology trained on questions derived from
medical literature, patient experiences, and clinical resources, along with
expertly curated responses, Zebra-Llama demonstrates unprecedented capabilities
in handling EDS-related queries. On a test set of real-world questions
collected from EDS patients and clinicians, medical experts evaluated the
responses generated by both models, revealing Zebra-Llama's substantial
improvements over base model (Llama 3.1-8B-Instruct) in thoroughness (77.5% vs.
70.1%), accuracy (83.0% vs. 78.8%), clarity (74.7% vs. 72.0%) and citation
reliability (70.6% vs. 52.3%). Released as an open-source resource, Zebra-Llama
not only provides more accessible and reliable EDS information but also
establishes a framework for developing specialized AI solutions for other rare
conditions. This work represents a crucial step towards democratizing
expert-level knowledge in rare disease management, potentially transforming how
healthcare providers and patients navigate the complex landscape of rare
diseases.
LLaMo: Large Language Model-based Molecular Graph Assistant
Authors: name: Jinyoung Park, name: Minseong Bae, name: Dohwan Ko, name: Hyunwoo J. Kim
Published: 2024-10-31 03:56:05

Large Language Models (LLMs) have demonstrated remarkable generalization and
instruction-following capabilities with instruction tuning. The advancements in
LLMs and instruction tuning have led to the development of Large
Vision-Language Models (LVLMs). However, the competency of the LLMs and
instruction tuning have been less explored in the molecular domain. Thus, we
propose LLaMo: Large Language Model-based Molecular graph assistant, which is
an end-to-end trained large molecular graph-language model. To bridge the
discrepancy between the language and graph modalities, we present the
multi-level graph projector that transforms graph representations into graph
tokens by abstracting the output representations of each GNN layer and motif
representations with the cross-attention mechanism. We also introduce
machine-generated molecular graph instruction data to instruction-tune the
large molecular graph-language model for general-purpose molecule and language
understanding. Our extensive experiments demonstrate that LLaMo shows the best
performance on diverse tasks, such as molecular description generation,
property prediction, and IUPAC name prediction. The code of LLaMo is available
at https://github.com/mlvlab/LLaMo.
DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for
  Efficient Robot Execution
Authors: name: Yang Yue, name: Yulin Wang, name: Bingyi Kang, name: Yizeng Han, name: Shenzhi Wang, name: Shiji Song, name: Jiashi Feng, name: Gao Huang
Published: 2024-11-04 18:26:08

MLLMs have demonstrated remarkable comprehension and reasoning capabilities
with complex language and visual data. These advances have spurred the vision
of establishing a generalist robotic MLLM proficient in understanding complex
human instructions and accomplishing various embodied tasks. However,
developing MLLMs for real-world robots is challenging due to the typically
limited computation and memory capacities available on robotic platforms. In
contrast, the inference of MLLMs involves storing billions of parameters and
performing tremendous computation, imposing significant hardware demands. In
our paper, we propose a Dynamic Early-Exit Framework for Robotic
Vision-Language-Action Model (DeeR-VLA, or simply DeeR) that automatically
adjusts the size of the activated MLLM based on each situation at hand. The
approach leverages a multi-exit architecture in MLLMs, which allows the model
to terminate processing once a proper size of the model has been activated for
a specific situation, thus avoiding further redundant computation.
Additionally, we develop novel algorithms that establish early-termination
criteria for DeeR, conditioned on predefined demands such as average
computational cost (i.e., power consumption), as well as peak computational
consumption (i.e., latency) and GPU memory usage. These enhancements ensure
that DeeR operates efficiently under varying resource constraints while
maintaining competitive performance. On the CALVIN robot manipulation
benchmark, DeeR demonstrates significant reductions in computational costs of
LLM by 5.2-6.5x and GPU memory of LLM by 2-6x without compromising performance.
Code and checkpoints are available at https://github.com/yueyang130/DeeR-VLA.
HtmlRAG: HTML is Better Than Plain Text for Modeling Retrieved Knowledge
  in RAG Systems
Authors: name: Jiejun Tan, name: Zhicheng Dou, name: Wen Wang, name: Mang Wang, name: Weipeng Chen, name: Ji-Rong Wen
Published: 2024-11-05 09:58:36

Retrieval-Augmented Generation (RAG) has been shown to improve knowledge
capabilities and alleviate the hallucination problem of LLMs. The Web is a
major source of external knowledge used in RAG systems, and many commercial
systems such as ChatGPT and Perplexity have used Web search engines as their
major retrieval systems. Typically, such RAG systems retrieve search results,
download HTML sources of the results, and then extract plain texts from the
HTML sources. Plain text documents or chunks are fed into the LLMs to augment
the generation. However, much of the structural and semantic information
inherent in HTML, such as headings and table structures, is lost during this
plain-text-based RAG process. To alleviate this problem, we propose HtmlRAG,
which uses HTML instead of plain text as the format of retrieved knowledge in
RAG. We believe HTML is better than plain text in modeling knowledge in
external documents, and most LLMs possess robust capacities to understand HTML.
However, utilizing HTML presents new challenges. HTML contains additional
content such as tags, JavaScript, and CSS specifications, which bring extra
input tokens and noise to the RAG system. To address this issue, we propose
HTML cleaning, compression, and pruning strategies, to shorten the HTML while
minimizing the loss of information. Specifically, we design a two-step
block-tree-based pruning method that prunes useless HTML blocks and keeps only
the relevant part of the HTML. Experiments on six QA datasets confirm the
superiority of using HTML in RAG systems.
SALSA: Soup-based Alignment Learning for Stronger Adaptation in RLHF
Authors: name: Atoosa Chegini, name: Hamid Kazemi, name: Iman Mirzadeh, name: Dong Yin, name: Maxwell Horton, name: Moin Nabi, name: Mehrdad Farajtabar, name: Keivan Alizadeh
Published: 2024-11-04 04:53:43

In Large Language Model (LLM) development, Reinforcement Learning from Human
Feedback (RLHF) is crucial for aligning models with human values and
preferences. RLHF traditionally relies on the Kullback-Leibler (KL) divergence
between the current policy and a frozen initial policy as a reference, which is
added as a penalty in policy optimization algorithms like Proximal Policy
Optimization (PPO). While this constraint prevents models from deviating too
far from the initial checkpoint, it limits exploration of the reward landscape,
reducing the model's ability to discover higher-quality solutions. As a result,
policy optimization is often trapped in a narrow region of the parameter space,
leading to suboptimal alignment and performance. This paper presents SALSA
(Soup-based Alignment Learning for Stronger Adaptation), a novel approach
designed to overcome these limitations by creating a more flexible and better
located reference model through weight-space averaging of two independent
supervised fine-tuned (SFT) models. This model soup allows for larger deviation
in KL divergence and exploring a promising region of the solution space without
sacrificing stability. By leveraging this more robust reference model, SALSA
fosters better exploration, achieving higher rewards and improving model
robustness, out-of-distribution generalization, and performance. We validate
the effectiveness of SALSA through extensive experiments on popular open models
(Llama2-7B, Mistral-7B, and Gemma-2B) across various benchmarks (MT-Bench,
Arena-Hard, UltraFeedback), where it consistently surpasses PPO by fostering
deeper exploration and achieving superior alignment in LLMs.