Orionnnn/hub-stats huggingface dataset:
LLaRA: Supercharging Robot Learning Data for Vision-Language Policy
Authors: name: Xiang Li, name: Cristina Mata, name: Jongwoo Park, name: Kumara Kahatapitiya, name: Yoo Sung Jang, name: Jinghuan Shang, name: Kanchana Ranasinghe, name: Ryan Burgert, name: Mu Cai, name: Yong Jae Lee, name: Michael S. Ryoo
Published: 2024-06-28 17:59:12

Large Language Models (LLMs) equipped with extensive world knowledge and
strong reasoning skills can tackle diverse tasks across domains, often by
posing them as conversation-style instruction-response pairs. In this paper, we
propose LLaRA: Large Language and Robotics Assistant, a framework which
formulates robot action policy as conversations, and provides improved
responses when trained with auxiliary data that complements policy learning.
LLMs with visual inputs, i.e., Vision Language Models (VLMs), have the capacity
to process state information as visual-textual prompts and generate optimal
policy decisions in text. To train such action policy VLMs, we first introduce
an automated pipeline to generate diverse high-quality robotics instruction
data from existing behavior cloning data. A VLM finetuned with the resulting
collection of datasets based on a conversation-style formulation tailored for
robotics tasks, can generate meaningful robot action policy decisions. Our
experiments across multiple simulated and real-world environments demonstrate
the state-of-the-art performance of the proposed LLaRA framework. The code,
datasets, and pretrained models are available at
https://github.com/LostXine/LLaRA.
Efficient World Models with Context-Aware Tokenization
Authors: name: Vincent Micheli, name: Eloi Alonso, name: François Fleuret
Published: 2024-06-27 16:54:12

Scaling up deep Reinforcement Learning (RL) methods presents a significant
challenge. Following developments in generative modelling, model-based RL
positions itself as a strong contender. Recent advances in sequence modelling
have led to effective transformer-based world models, albeit at the price of
heavy computations due to the long sequences of tokens required to accurately
simulate environments. In this work, we propose Delta-IRIS, a new agent with
a world model architecture composed of a discrete autoencoder that encodes
stochastic deltas between time steps and an autoregressive transformer that
predicts future deltas by summarizing the current state of the world with
continuous tokens. In the Crafter benchmark, Delta-IRIS sets a new state of
the art at multiple frame budgets, while being an order of magnitude faster to
train than previous attention-based approaches. We release our code and models
at https://github.com/vmicheli/delta-iris.
GaussianDreamerPro: Text to Manipulable 3D Gaussians with Highly
  Enhanced Quality
Authors: name: Taoran Yi, name: Jiemin Fang, name: Zanwei Zhou, name: Junjie Wang, name: Guanjun Wu, name: Lingxi Xie, name: Xiaopeng Zhang, name: Wenyu Liu, name: Xinggang Wang, name: Qi Tian
Published: 2024-06-26 16:12:09

Recently, 3D Gaussian splatting (3D-GS) has achieved great success in
reconstructing and rendering real-world scenes. To transfer the high rendering
quality to generation tasks, a series of research works attempt to generate
3D-Gaussian assets from text. However, the generated assets have not achieved
the same quality as those in reconstruction tasks. We observe that Gaussians
tend to grow without control as the generation process may cause indeterminacy.
Aiming at highly enhancing the generation quality, we propose a novel framework
named GaussianDreamerPro. The main idea is to bind Gaussians to reasonable
geometry, which evolves over the whole generation process. Along different
stages of our framework, both the geometry and appearance can be enriched
progressively. The final output asset is constructed with 3D Gaussians bound to
mesh, which shows significantly enhanced details and quality compared with
previous methods. Notably, the generated asset can also be seamlessly
integrated into downstream manipulation pipelines, e.g. animation, composition,
and simulation etc., greatly promoting its potential in wide applications.
Demos are available at https://taoranyi.com/gaussiandreamerpro/.
EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything
  Model
Authors: name: Yuxuan Zhang, name: Tianheng Cheng, name: Rui Hu, name: ei Liu, name: Heng Liu, name: Longjin Ran, name: Xiaoxin Chen, name: Wenyu Liu, name: Xinggang Wang
Published: 2024-06-28 17:38:18

Segment Anything Model (SAM) has attracted widespread attention for its
superior interactive segmentation capabilities with visual prompts while
lacking further exploration of text prompts. In this paper, we empirically
investigate what text prompt encoders (e.g., CLIP or LLM) are good for adapting
SAM for referring expression segmentation and introduce the Early
Vision-language Fusion-based SAM (EVF-SAM). EVF-SAM is a simple yet effective
referring segmentation method which exploits multimodal prompts (i.e., image
and text) and comprises a pre-trained vision-language model to generate
referring prompts and a SAM model for segmentation. Surprisingly, we observe
that: (1) multimodal prompts and (2) vision-language models with early fusion
(e.g., BEIT-3) are beneficial for prompting SAM for accurate referring
segmentation. Our experiments show that the proposed EVF-SAM based on BEIT-3
can obtain state-of-the-art performance on RefCOCO/+/g for referring expression
segmentation and demonstrate the superiority of prompting SAM with early
vision-language fusion. In addition, the proposed EVF-SAM with 1.32B parameters
achieves remarkably higher performance while reducing nearly 82% of parameters
compared to previous SAM methods based on large multimodal models.
Direct Preference Knowledge Distillation for Large Language Models
Authors: name: Yixing Li, name: Yuxian Gu, name: Li Dong, name: Dequan Wang, name: Yu Cheng, name: Furu Wei
Published: 2024-06-28 09:23:40

In the field of large language models (LLMs), Knowledge Distillation (KD) is
a critical technique for transferring capabilities from teacher models to
student models. However, existing KD methods face limitations and challenges in
distillation of LLMs, including efficiency and insufficient measurement
capabilities of traditional KL divergence. It is shown that LLMs can serve as
an implicit reward function, which we define as a supplement to KL divergence.
In this work, we propose Direct Preference Knowledge Distillation (DPKD) for
LLMs. DPKD utilizes distribution divergence to represent the preference loss
and implicit reward function. We re-formulate KD of LLMs into two stages: first
optimizing and objective consisting of implicit reward and reverse KL
divergence and then improving the preference probability of teacher outputs
over student outputs. We conducted experiments and analysis on various datasets
with LLM parameters ranging from 120M to 13B and demonstrate the broad
applicability and effectiveness of our DPKD approach. Meanwhile, we prove the
value and effectiveness of the introduced implicit reward and output preference
in KD through experiments and theoretical analysis. The DPKD method outperforms
the baseline method in both output response precision and exact match
percentage. Code and data are available at https://aka.ms/dpkd.
AutoRAG-HP: Automatic Online Hyper-Parameter Tuning for
  Retrieval-Augmented Generation
Authors: name: Jia Fu, name: Xiaoting Qin, name: Fangkai Yang, name: Lu Wang, name: Jue Zhang, name: Qingwei Lin, name: Yubo Chen, name: Dongmei Zhang, name: Saravan Rajmohan, name: Qi Zhang
Published: 2024-06-27 15:18:21

Recent advancements in Large Language Models have transformed ML/AI
development, necessitating a reevaluation of AutoML principles for the
Retrieval-Augmented Generation (RAG) systems. To address the challenges of
hyper-parameter optimization and online adaptation in RAG, we propose the
AutoRAG-HP framework, which formulates the hyper-parameter tuning as an online
multi-armed bandit (MAB) problem and introduces a novel two-level Hierarchical
MAB (Hier-MAB) method for efficient exploration of large search spaces. We
conduct extensive experiments on tuning hyper-parameters, such as top-k
retrieved documents, prompt compression ratio, and embedding methods, using the
ALCE-ASQA and Natural Questions datasets. Our evaluation from jointly
optimization all three hyper-parameters demonstrate that MAB-based online
learning methods can achieve Recall@5 approx 0.8 for scenarios with
prominent gradients in search space, using only sim20% of the LLM API calls
required by the Grid Search approach. Additionally, the proposed Hier-MAB
approach outperforms other baselines in more challenging optimization
scenarios. The code will be made available at https://aka.ms/autorag.
HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into
  Multimodal LLMs at Scale
Authors: name: Junying Chen, name: Ruyi Ouyang, name: Anningzhe Gao, name: Shunian Chen, name: Guiming Hardy Chen, name: Xidong Wang, name: Ruifei Zhang, name: Zhenyang Cai, name: Ke Ji, name: Guangjun Yu, name: Xiang Wan, name: Benyou Wang
Published: 2024-06-27 15:50:41

The rapid development of multimodal large language models (MLLMs), such as
GPT-4V, has led to significant advancements. However, these models still face
challenges in medical multimodal capabilities due to limitations in the
quantity and quality of medical vision-text data, stemming from data privacy
concerns and high annotation costs. While pioneering approaches utilize
PubMed's large-scale, de-identified medical image-text pairs to address these
limitations, they still fall short due to inherent data noise. To tackle this,
we refined medical image-text pairs from PubMed and employed MLLMs (GPT-4V) in
an 'unblinded' capacity to denoise and reformat the data, resulting in the
creation of the PubMedVision dataset with 1.3 million medical VQA samples. Our
validation demonstrates that: (1) PubMedVision can significantly enhance the
medical multimodal capabilities of current MLLMs, showing significant
improvement in benchmarks including the MMMU Health & Medicine track; (2)
manual checks by medical experts and empirical results validate the superior
data quality of our dataset compared to other data construction methods. Using
PubMedVision, we train a 34B medical MLLM HuatuoGPT-Vision, which shows
superior performance in medical multimodal scenarios among open-source MLLMs.
Scaling Synthetic Data Creation with 1,000,000,000 Personas
Authors: name: Xin Chan, name: Xiaoyang Wang, name: Dian Yu, name: Haitao Mi, name: Dong Yu
Published: 2024-06-28 17:59:01

We propose a novel persona-driven data synthesis methodology that leverages
various perspectives within a large language model (LLM) to create diverse
synthetic data. To fully exploit this methodology at scale, we introduce
Persona Hub -- a collection of 1 billion diverse personas automatically curated
from web data. These 1 billion personas (~13% of the world's total population),
acting as distributed carriers of world knowledge, can tap into almost every
perspective encapsulated within the LLM, thereby facilitating the creation of
diverse synthetic data at scale for various scenarios. By showcasing Persona
Hub's use cases in synthesizing high-quality mathematical and logical reasoning
problems, instructions (i.e., user prompts), knowledge-rich texts, game NPCs
and tools (functions) at scale, we demonstrate persona-driven data synthesis is
versatile, scalable, flexible, and easy to use, potentially driving a paradigm
shift in synthetic data creation and applications in practice, which may have a
profound impact on LLM research and development.
AUTOHALLUSION: Automatic Generation of Hallucination Benchmarks for
  Vision-Language Models
Authors: name: Xiyang Wu, name: Tianrui Guan, name: Dianqi Li, name: Shuaiyi Huang, name: Xiaoyu Liu, name: Xijun Wang, name: Ruiqi Xian, name: Abhinav Shrivastava, name: Furong Huang, name: Jordan Lee Boyd-Graber, name: Tianyi Zhou, name: Dinesh Manocha
Published: 2024-06-16 11:44:43

Large vision-language models (LVLMs) hallucinate: certain context cues in an
image may trigger the language module's overconfident and incorrect reasoning
on abnormal or hypothetical objects. Though a few benchmarks have been
developed to investigate LVLM hallucinations, they mainly rely on hand-crafted
corner cases whose fail patterns may hardly generalize, and finetuning on them
could undermine their validity. These motivate us to develop the first
automatic benchmark generation approach, AUTOHALLUSION, that harnesses a few
principal strategies to create diverse hallucination examples. It probes the
language modules in LVLMs for context cues and uses them to synthesize images
by: (1) adding objects abnormal to the context cues; (2) for two co-occurring
objects, keeping one and excluding the other; or (3) removing objects closely
tied to the context cues. It then generates image-based questions whose
ground-truth answers contradict the language module's prior. A model has to
overcome contextual biases and distractions to reach correct answers, while
incorrect or inconsistent answers indicate hallucinations. AUTOHALLUSION
enables us to create new benchmarks at the minimum cost and thus overcomes the
fragility of hand-crafted benchmarks. It also reveals common failure patterns
and reasons, providing key insights to detect, avoid, or control
hallucinations. Comprehensive evaluations of top-tier LVLMs, e.g.,
GPT-4V(ision), Gemini Pro Vision, Claude 3, and LLaVA-1.5, show a 97.7% and
98.7% success rate of hallucination induction on synthetic and real-world
datasets of AUTOHALLUSION, paving the way for a long battle against
hallucinations.
T-FREE: Tokenizer-Free Generative LLMs via Sparse Representations for
  Memory-Efficient Embeddings
Authors: name: Björn Deiseroth, name: Manuel Brack, name: Patrick Schramowski, name: Kristian Kersting, name: Samuel Weinbach
Published: 2024-06-27 14:49:08

Tokenizers are crucial for encoding information in Large Language Models, but
their development has recently stagnated, and they contain inherent weaknesses.
Major limitations include computational overhead, ineffective vocabulary use,
and unnecessarily large embedding and head layers. Additionally, their
performance is biased towards a reference corpus, leading to reduced
effectiveness for underrepresented languages.
  To remedy these issues, we propose T-FREE, which directly embeds words
through sparse activation patterns over character triplets, and does not
require a reference corpus. T-FREE inherently exploits morphological
similarities and allows for strong compression of embedding layers. In our
exhaustive experimental evaluation, we achieve competitive downstream
performance with a parameter reduction of more than 85% on these layers.
Further, T-FREE shows significant improvements in cross-lingual transfer
learning.
Understand What LLM Needs: Dual Preference Alignment for
  Retrieval-Augmented Generation
Authors: name: Guanting Dong, name: Yutao Zhu, name: Chenghao Zhang, name: Zechen Wang, name: Zhicheng Dou, name: Ji-Rong Wen
Published: 2024-06-26 18:26:53

Retrieval-augmented generation (RAG) has demonstrated effectiveness in
mitigating the hallucination problem of large language models (LLMs). However,
the difficulty of aligning the retriever with the diverse LLMs' knowledge
preferences inevitably poses an inevitable challenge in developing a reliable
RAG system. To address this issue, we propose DPA-RAG, a universal framework
designed to align diverse knowledge preferences within RAG systems.
Specifically, we initially introduce a preference knowledge construction
pipline and incorporate five novel query augmentation strategies to alleviate
preference data scarcity. Based on preference data, DPA-RAG accomplishes both
external and internal preference alignment: 1) It jointly integrate pair-wise,
point-wise, and contrastive preference alignment abilities into the reranker,
achieving external preference alignment among RAG components. 2) It further
introduces a pre-aligned stage before vanilla Supervised Fine-tuning (SFT),
enabling LLMs to implicitly capture knowledge aligned with their reasoning
preferences, achieving LLMs' internal alignment. Experimental results across
four knowledge-intensive QA datasets demonstrate that DPA-RAG outperforms all
baselines and seamlessly integrates both black-box and open-sourced LLM
readers. Further qualitative analysis and discussions also provide empirical
guidance for achieving reliable RAG systems. Our code is publicly available at
https://github.com/dongguanting/DPA-RAG.
MUMU: Bootstrapping Multimodal Image Generation from Text-to-Image Data
Authors: name: William Berman, name: Alexander Peysakhovich
Published: 2024-06-26 23:21:42

We train a model to generate images from multimodal prompts of interleaved
text and images such as "a <picture of a man> man and his <picture of a dog>
dog in an <picture of a cartoon> animated style." We bootstrap a multimodal
dataset by extracting semantically meaningful image crops corresponding to
words in the image captions of synthetically generated and publicly available
text-image data. Our model, MUMU, is composed of a vision-language model
encoder with a diffusion decoder and is trained on a single 8xH100 GPU node.
Despite being only trained on crops from the same image, MUMU learns to compose
inputs from different images into a coherent output. For example, an input of a
realistic person and a cartoon will output the same person in the cartoon
style, and an input of a standing subject and a scooter will output the subject
riding the scooter. As a result, our model generalizes to tasks such as style
transfer and character consistency. Our results show the promise of using
multimodal models as general purpose controllers for image generation.
Benchmarking Mental State Representations in Language Models
Authors: name: Matteo Bortoletto, name: Constantin Ruhdorfer, name: Lei Shi, name: Andreas Bulling
Published: 2024-06-25 12:51:06

While numerous works have assessed the generative performance of language
models (LMs) on tasks requiring Theory of Mind reasoning, research into the
models' internal representation of mental states remains limited. Recent work
has used probing to demonstrate that LMs can represent beliefs of themselves
and others. However, these claims are accompanied by limited evaluation, making
it difficult to assess how mental state representations are affected by model
design and training choices. We report an extensive benchmark with various LM
types with different model sizes, fine-tuning approaches, and prompt designs to
study the robustness of mental state representations and memorisation issues
within the probes. Our results show that the quality of models' internal
representations of the beliefs of others increases with model size and, more
crucially, with fine-tuning. We are the first to study how prompt variations
impact probing performance on theory of mind tasks. We demonstrate that models'
representations are sensitive to prompt variations, even when such variations
should be beneficial. Finally, we complement previous activation editing
experiments on Theory of Mind tasks and show that it is possible to improve
models' reasoning performance by steering their activations without the need to
train any probe.
Dataset Size Recovery from LoRA Weights
Authors: name: Mohammad Salama, name: Jonathan Kahana, name: Eliahu Horwitz, name: Yedid Hoshen
Published: 2024-06-27 17:59:53

Model inversion and membership inference attacks aim to reconstruct and
verify the data which a model was trained on. However, they are not guaranteed
to find all training samples as they do not know the size of the training set.
In this paper, we introduce a new task: dataset size recovery, that aims to
determine the number of samples used to train a model, directly from its
weights. We then propose DSiRe, a method for recovering the number of images
used to fine-tune a model, in the common case where fine-tuning uses LoRA. We
discover that both the norm and the spectrum of the LoRA matrices are closely
linked to the fine-tuning dataset size; we leverage this finding to propose a
simple yet effective prediction algorithm. To evaluate dataset size recovery of
LoRA weights, we develop and release a new benchmark, LoRA-WiSE, consisting of
over 25000 weight snapshots from more than 2000 diverse LoRA fine-tuned models.
Our best classifier can predict the number of fine-tuning images with a mean
absolute error of 0.36 images, establishing the feasibility of this attack.
ResumeAtlas: Revisiting Resume Classification with Large-Scale Datasets
  and Large Language Models
Authors: name: Ahmed Heakl, name: Youssef Mohamed, name: Noran Mohamed, name: Ali Sharkaway, name: Ahmed Zaky
Published: 2024-06-26 07:25:18

The increasing reliance on online recruitment platforms coupled with the
adoption of AI technologies has highlighted the critical need for efficient
resume classification methods. However, challenges such as small datasets, lack
of standardized resume templates, and privacy concerns hinder the accuracy and
effectiveness of existing classification models. In this work, we address these
challenges by presenting a comprehensive approach to resume classification. We
curated a large-scale dataset of 13,389 resumes from diverse sources and
employed Large Language Models (LLMs) such as BERT and Gemma1.1 2B for
classification. Our results demonstrate significant improvements over
traditional machine learning approaches, with our best model achieving a top-1
accuracy of 92\% and a top-5 accuracy of 97.5\%. These findings underscore the
importance of dataset quality and advanced model architectures in enhancing the
accuracy and robustness of resume classification systems, thus advancing the
field of online recruitment practices.
LiveBench: A Challenging, Contamination-Free LLM Benchmark
Authors: name: Colin White, name: Samuel Dooley, name: Manley Roberts, name: Arka Pal, name: Ben Feuer, name: Siddhartha Jain, name: Ravid Shwartz-Ziv, name: Neel Jain, name: Khalid Saifullah, name: Siddartha Naidu, name: Chinmay Hegde, name: Yann LeCun, name: Tom Goldstein, name: Willie Neiswanger, name: Micah Goldblum
Published: 2024-06-27 16:47:42

Test set contamination, wherein test data from a benchmark ends up in a newer
model's training set, is a well-documented obstacle for fair LLM evaluation and
can quickly render benchmarks obsolete. To mitigate this, many recent
benchmarks crowdsource new prompts and evaluations from human or LLM judges;
however, these can introduce significant biases, and break down when scoring
hard questions. In this work, we introduce a new benchmark for LLMs designed to
be immune to both test set contamination and the pitfalls of LLM judging and
human crowdsourcing. We release LiveBench, the first benchmark that (1)
contains frequently-updated questions from recent information sources, (2)
scores answers automatically according to objective ground-truth values, and
(3) contains a wide variety of challenging tasks, spanning math, coding,
reasoning, language, instruction following, and data analysis. To achieve this,
LiveBench contains questions that are based on recently-released math
competitions, arXiv papers, news articles, and datasets, and it contains
harder, contamination-free versions of tasks from previous benchmarks such as
Big-Bench Hard, AMPS, and IFEval. We evaluate many prominent closed-source
models, as well as dozens of open-source models ranging from 0.5B to 110B in
size. LiveBench is difficult, with top models achieving below 65% accuracy. We
release all questions, code, and model answers. Questions will be added and
updated on a monthly basis, and we will release new tasks and harder versions
of tasks over time so that LiveBench can distinguish between the capabilities
of LLMs as they improve in the future. We welcome community engagement and
collaboration for expanding the benchmark tasks and models.