Orionnnn/hub-stats huggingface dataset:
Nash Learning from Human Feedback
Authors: name: Rémi Munos, name: Michal Valko, name: Daniele Calandriello, name: Mohammad Gheshlaghi Azar, name: Mark Rowland, name: Daniel Guo, name: Yunhao Tang, name: Matthieu Geist, name: Thomas Mésnard, name: Andrea Michi, name: Marco Selvi, name: Sertan Girgin, name: Nikola Momchev, name: Olivier Bachem, name: Daniel J. Mankowitz, name: Doina Precup, name: Bilal Piot
Published: 2023-12-01 19:26:23

Reinforcement learning from human feedback (RLHF) has emerged as the main
paradigm for aligning large language models (LLMs) with human preferences.
Typically, RLHF involves the initial step of learning a reward model from human
feedback, often expressed as preferences between pairs of text generations
produced by a pre-trained LLM. Subsequently, the LLM's policy is fine-tuned by
optimizing it to maximize the reward model through a reinforcement learning
algorithm. However, an inherent limitation of current reward models is their
inability to fully represent the richness of human preferences and their
dependency on the sampling distribution.
  In this study, we introduce an alternative pipeline for the fine-tuning of
LLMs using pairwise human feedback. Our approach entails the initial learning
of a preference model, which is conditioned on two inputs given a prompt,
followed by the pursuit of a policy that consistently generates responses
preferred over those generated by any competing policy, thus defining the Nash
equilibrium of this preference model. We term this approach Nash learning from
human feedback (NLHF).
  In the context of a tabular policy representation, we present a novel
algorithmic solution, Nash-MD, founded on the principles of mirror descent.
This algorithm produces a sequence of policies, with the last iteration
converging to the regularized Nash equilibrium. Additionally, we explore
parametric representations of policies and introduce gradient descent
algorithms for deep-learning architectures. To demonstrate the effectiveness of
our approach, we present experimental results involving the fine-tuning of a
LLM for a text summarization task. We believe NLHF offers a compelling avenue
for preference learning and policy optimization with the potential of advancing
the field of aligning LLMs with human preferences.
Object Recognition as Next Token Prediction
Authors: name: Kaiyu Yue, name: Bor-Chun Chen, name: Jonas Geiping, name: Hengduo Li, name: Tom Goldstein, name: Ser-Nam Lim
Published: 2023-12-04 18:58:40

We present an approach to pose object recognition as next token prediction.
The idea is to apply a language decoder that auto-regressively predicts the
text tokens from image embeddings to form labels. To ground this prediction
process in auto-regression, we customize a non-causal attention mask for the
decoder, incorporating two key features: modeling tokens from different labels
to be independent, and treating image tokens as a prefix. This masking
mechanism inspires an efficient method - one-shot sampling - to simultaneously
sample tokens of multiple labels in parallel and rank generated labels by their
probabilities during inference. To further enhance the efficiency, we propose a
simple strategy to construct a compact decoder by simply discarding the
intermediate blocks of a pretrained language model. This approach yields a
decoder that matches the full model's performance while being notably more
efficient. The code is available at https://github.com/kaiyuyue/nxtp
The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context
  Learning
Authors: name: Bill Yuchen Lin, name: Abhilasha Ravichander, name: Ximing Lu, name: Nouha Dziri, name: Melanie Sclar, name: Khyathi Chandu, name: Chandra Bhagavatula, name: Yejin Choi
Published: 2023-12-04 00:46:11

The alignment tuning process of large language models (LLMs) typically
involves instruction learning through supervised fine-tuning (SFT) and
preference tuning via reinforcement learning from human feedback (RLHF). A
recent study, LIMA (Zhou et al. 2023), shows that using merely 1K examples for
SFT can achieve significant alignment performance as well, suggesting that the
effect of alignment tuning might be "superficial." This raises questions about
how exactly the alignment tuning transforms a base LLM.
  We analyze the effect of alignment tuning by examining the token distribution
shift between base LLMs and their aligned counterpart. Our findings reveal that
base LLMs and their alignment-tuned versions perform nearly identically in
decoding on the majority of token positions. Most distribution shifts occur
with stylistic tokens. These direct evidence strongly supports the Superficial
Alignment Hypothesis suggested by LIMA.
  Based on these findings, we rethink the alignment of LLMs by posing the
research question: how effectively can we align base LLMs without SFT or RLHF?
To address this, we introduce a simple, tuning-free alignment method, URIAL.
URIAL achieves effective alignment purely through in-context learning (ICL)
with base LLMs, requiring as few as three constant stylistic examples and a
system prompt. We conduct a fine-grained and interpretable evaluation on a
diverse set of examples, named JUST-EVAL-INSTRUCT. Results demonstrate that
base LLMs with URIAL can match or even surpass the performance of LLMs aligned
with SFT or SFT+RLHF. We show that the gap between tuning-free and tuning-based
alignment methods can be significantly reduced through strategic prompting and
ICL. Our findings on the superficial nature of alignment tuning and results
with URIAL suggest that deeper analysis and theoretical understanding of
alignment is crucial to future LLM research.
Using Large Language Models to Accelerate Communication for Users with
  Severe Motor Impairments
Authors: name: Shanqing Cai, name: Subhashini Venugopalan, name: Katie Seaver, name: Xiang Xiao, name: Katrin Tomanek, name: Sri Jalasutram, name: Meredith Ringel Morris, name: Shaun Kane, name: Ajit Narayanan, name: Robert L. MacDonald, name: Emily Kornman, name: Daniel Vance, name: Blair Casey, name: Steve M. Gleason, name: Philip Q. Nelson, name: Michael P. Brenner
Published: 2023-12-03 23:12:49

Finding ways to accelerate text input for individuals with profound motor
impairments has been a long-standing area of research. Closing the speed gap
for augmentative and alternative communication (AAC) devices such as
eye-tracking keyboards is important for improving the quality of life for such
individuals. Recent advances in neural networks of natural language pose new
opportunities for re-thinking strategies and user interfaces for enhanced
text-entry for AAC users. In this paper, we present SpeakFaster, consisting of
large language models (LLMs) and a co-designed user interface for text entry in
a highly-abbreviated form, allowing saving 57% more motor actions than
traditional predictive keyboards in offline simulation. A pilot study with 19
non-AAC participants typing on a mobile device by hand demonstrated gains in
motor savings in line with the offline simulation, while introducing relatively
small effects on overall typing speed. Lab and field testing on two eye-gaze
typing users with amyotrophic lateral sclerosis (ALS) demonstrated text-entry
rates 29-60% faster than traditional baselines, due to significant saving of
expensive keystrokes achieved through phrase and word predictions from
context-aware LLMs. These findings provide a strong foundation for further
exploration of substantially-accelerated text communication for motor-impaired
users and demonstrate a direction for applying LLMs to text-based user
interfaces.
TextGenSHAP: Scalable Post-hoc Explanations in Text Generation with Long
  Documents
Authors: name: James Enouen, name: Hootan Nakhost, name: Sayna Ebrahimi, name: Sercan O Arik, name: Yan Liu, name: Tomas Pfister
Published: 2023-12-03 04:35:04

Large language models (LLMs) have attracted huge interest in practical
applications given their increasingly accurate responses and coherent reasoning
abilities. Given their nature as black-boxes using complex reasoning processes
on their inputs, it is inevitable that the demand for scalable and faithful
explanations for LLMs' generated content will continue to grow. There have been
major developments in the explainability of neural network models over the past
decade. Among them, post-hoc explainability methods, especially Shapley values,
have proven effective for interpreting deep learning models. However, there are
major challenges in scaling up Shapley values for LLMs, particularly when
dealing with long input contexts containing thousands of tokens and
autoregressively generated output sequences. Furthermore, it is often unclear
how to effectively utilize generated explanations to improve the performance of
LLMs. In this paper, we introduce TextGenSHAP, an efficient post-hoc
explanation method incorporating LM-specific techniques. We demonstrate that
this leads to significant increases in speed compared to conventional Shapley
value computations, reducing processing times from hours to minutes for
token-level explanations, and to just seconds for document-level explanations.
In addition, we demonstrate how real-time Shapley values can be utilized in two
important scenarios, providing better understanding of long-document question
answering by localizing important words and sentences; and improving existing
document retrieval systems through enhancing the accuracy of selected passages
and ultimately the final responses.
Segment and Caption Anything
Authors: name: Xiaoke Huang, name: Jianfeng Wang, name: Yansong Tang, name: Zheng Zhang, name: Han Hu, name: Jiwen Lu, name: Lijuan Wang, name: Zicheng Liu
Published: 2023-12-01 19:00:17

We propose a method to efficiently equip the Segment Anything Model (SAM)
with the ability to generate regional captions. SAM presents strong
generalizability to segment anything while is short for semantic understanding.
By introducing a lightweight query-based feature mixer, we align the
region-specific features with the embedding space of language models for later
caption generation. As the number of trainable parameters is small (typically
in the order of tens of millions), it costs less computation, less memory
usage, and less communication bandwidth, resulting in both fast and scalable
training. To address the scarcity problem of regional caption data, we propose
to first pre-train our model on objection detection and segmentation tasks. We
call this step weak supervision pretraining since the pre-training data only
contains category names instead of full-sentence descriptions. The weak
supervision pretraining allows us to leverage many publicly available object
detection and segmentation datasets. We conduct extensive experiments to
demonstrate the superiority of our method and validate each design choice. This
work serves as a stepping stone towards scaling up regional captioning data and
sheds light on exploring efficient ways to augment SAM with regional semantics.
The project page, along with the associated code, can be accessed via the
following https://xk-huang.github.io/segment-caption-anything/.
VMC: Video Motion Customization using Temporal Attention Adaption for
  Text-to-Video Diffusion Models
Authors: name: Hyeonho Jeong, name: Geon Yeong Park, name: Jong Chul Ye
Published: 2023-12-01 06:50:11

Text-to-video diffusion models have advanced video generation significantly.
However, customizing these models to generate videos with tailored motions
presents a substantial challenge. In specific, they encounter hurdles in (a)
accurately reproducing motion from a target video, and (b) creating diverse
visual variations. For example, straightforward extensions of static image
customization methods to video often lead to intricate entanglements of
appearance and motion data. To tackle this, here we present the Video Motion
Customization (VMC) framework, a novel one-shot tuning approach crafted to
adapt temporal attention layers within video diffusion models. Our approach
introduces a novel motion distillation objective using residual vectors between
consecutive frames as a motion reference. The diffusion process then preserves
low-frequency motion trajectories while mitigating high-frequency
motion-unrelated noise in image space. We validate our method against
state-of-the-art video generative models across diverse real-world motions and
contexts. Our codes, data and the project demo can be found at
https://video-motion-customization.github.io
DiffiT: Diffusion Vision Transformers for Image Generation
Authors: name: Ali Hatamizadeh, name: Jiaming Song, name: Guilin Liu, name: Jan Kautz, name: Arash Vahdat
Published: 2023-12-04 18:57:01

Diffusion models with their powerful expressivity and high sample quality
have enabled many new applications and use-cases in various domains. For sample
generation, these models rely on a denoising neural network that generates
images by iterative denoising. Yet, the role of denoising network architecture
is not well-studied with most efforts relying on convolutional residual U-Nets.
In this paper, we study the effectiveness of vision transformers in
diffusion-based generative learning. Specifically, we propose a new model,
denoted as Diffusion Vision Transformers (DiffiT), which consists of a hybrid
hierarchical architecture with a U-shaped encoder and decoder. We introduce a
novel time-dependent self-attention module that allows attention layers to
adapt their behavior at different stages of the denoising process in an
efficient manner. We also introduce latent DiffiT which consists of transformer
model with the proposed self-attention layers, for high-resolution image
generation. Our results show that DiffiT is surprisingly effective in
generating high-fidelity images, and it achieves state-of-the-art (SOTA)
benchmarks on a variety of class-conditional and unconditional synthesis tasks.
In the latent space, DiffiT achieves a new SOTA FID score of 1.73 on
ImageNet-256 dataset. Repository: https://github.com/NVlabs/DiffiT
Style Aligned Image Generation via Shared Attention
Authors: name: Amir Hertz, name: Andrey Voynov, name: Shlomi Fruchter, name: Daniel Cohen-Or
Published: 2023-12-04 18:55:35

Large-scale Text-to-Image (T2I) models have rapidly gained prominence across
creative fields, generating visually compelling outputs from textual prompts.
However, controlling these models to ensure consistent style remains
challenging, with existing methods necessitating fine-tuning and manual
intervention to disentangle content and style. In this paper, we introduce
StyleAligned, a novel technique designed to establish style alignment among a
series of generated images. By employing minimal `attention sharing' during the
diffusion process, our method maintains style consistency across images within
T2I models. This approach allows for the creation of style-consistent images
using a reference style through a straightforward inversion operation. Our
method's evaluation across diverse styles and text prompts demonstrates
high-quality synthesis and fidelity, underscoring its efficacy in achieving
consistent style across various inputs.
GIVT: Generative Infinite-Vocabulary Transformers
Authors: name: Michael Tschannen, name: Cian Eastwood, name: Fabian Mentzer
Published: 2023-12-04 18:48:02

We introduce generative infinite-vocabulary transformers (GIVT) which
generate vector sequences with real-valued entries, instead of discrete tokens
from a finite vocabulary. To this end, we propose two surprisingly simple
modifications to decoder-only transformers: 1) at the input, we replace the
finite-vocabulary lookup table with a linear projection of the input vectors;
and 2) at the output, we replace the logits prediction (usually mapped to a
categorical distribution) with the parameters of a multivariate Gaussian
mixture model. Inspired by the image-generation paradigm of VQ-GAN and MaskGIT,
where transformers are used to model the discrete latent sequences of a VQ-VAE,
we use GIVT to model the unquantized real-valued latent sequences of a VAE.
When applying GIVT to class-conditional image generation with iterative masked
modeling, we show competitive results with MaskGIT, while our approach
outperforms both VQ-GAN and MaskGIT when using it for causal modeling. Finally,
we obtain competitive results outside of image generation when applying our
approach to panoptic segmentation and depth estimation with a VAE-based variant
of the UViM framework.
Generative Rendering: Controllable 4D-Guided Video Generation with 2D
  Diffusion Models
Authors: name: Shengqu Cai, name: Duygu Ceylan, name: Matheus Gadelha, name: Chun-Hao Paul Huang, name: Tuanfeng Yang Wang, name: Gordon Wetzstein
Published: 2023-12-03 14:17:11

Traditional 3D content creation tools empower users to bring their
imagination to life by giving them direct control over a scene's geometry,
appearance, motion, and camera path. Creating computer-generated videos,
however, is a tedious manual process, which can be automated by emerging
text-to-video diffusion models. Despite great promise, video diffusion models
are difficult to control, hindering a user to apply their own creativity rather
than amplifying it. To address this challenge, we present a novel approach that
combines the controllability of dynamic 3D meshes with the expressivity and
editability of emerging diffusion models. For this purpose, our approach takes
an animated, low-fidelity rendered mesh as input and injects the ground truth
correspondence information obtained from the dynamic mesh into various stages
of a pre-trained text-to-image generation model to output high-quality and
temporally consistent frames. We demonstrate our approach on various examples
where motion can be obtained by animating rigged assets or changing the camera
path.
Magicoder: Source Code Is All You Need
Authors: name: Yuxiang Wei, name: Zhe Wang, name: Jiawei Liu, name: Yifeng Ding, name: Lingming Zhang
Published: 2023-12-04 18:50:35

We introduce Magicoder, a series of fully open-source (code, weights, and
data) Large Language Models (LLMs) for code that significantly closes the gap
with top code models while having no more than 7B parameters. Magicoder models
are trained on 75K synthetic instruction data using OSS-Instruct, a novel
approach to enlightening LLMs with open-source code snippets to generate
high-quality instruction data for code. Our main motivation is to mitigate the
inherent bias of the synthetic data generated by LLMs by empowering them with a
wealth of open-source references for the production of more diverse, realistic,
and controllable data. The orthogonality of OSS-Instruct and other data
generation methods like Evol-Instruct further enables us to build an enhanced
MagicoderS. Both Magicoder and MagicoderS substantially outperform
state-of-the-art code models with similar or even larger sizes on a wide range
of coding benchmarks, including Python text-to-code generation, multilingual
coding, and data-science program completion. Notably, MagicoderS-CL-7B based on
CodeLlama even surpasses the prominent ChatGPT on HumanEval+ (66.5 vs. 65.9 in
pass@1). Overall, OSS-Instruct opens a new direction for low-bias and
high-quality instruction tuning using abundant open-source references.
RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from
  Fine-grained Correctional Human Feedback
Authors: name: Tianyu Yu, name: Yuan Yao, name: Haoye Zhang, name: Taiwen He, name: Yifeng Han, name: Ganqu Cui, name: Jinyi Hu, name: Zhiyuan Liu, name: Hai-Tao Zheng, name: Maosong Sun, name: Tat-Seng Chua
Published: 2023-12-01 11:36:08

Multimodal Large Language Models (MLLMs) have recently demonstrated
impressive capabilities in multimodal understanding, reasoning, and
interaction. However, existing MLLMs prevalently suffer from serious
hallucination problems, generating text that is not factually grounded in
associated images. The problem makes existing MLLMs untrustworthy and thus
impractical in real-world (especially high-stakes) applications. To address the
challenge, we present RLHF-V, which enhances MLLM trustworthiness via behavior
alignment from fine-grained correctional human feedback. Specifically, RLHF-V
collects human preference in the form of segment-level corrections on
hallucinations, and performs dense direct preference optimization over the
human feedback. Comprehensive experiments on five benchmarks in both automatic
and human evaluation show that, RLHF-V can enable substantially more
trustworthy MLLM behaviors with promising data and computation efficiency.
Remarkably, using 1.4k annotated data samples, RLHF-V significantly reduces the
hallucination rate of the base MLLM by 34.8%, outperforming the concurrent
LLaVA-RLHF trained on 10k annotated data. The final model achieves
state-of-the-art performance in trustworthiness among open-source MLLMs, and
shows better robustness than GPT-4V in preventing hallucinations aroused from
over-generalization. We open-source our code, model, and data at
https://github.com/RLHF-V/RLHF-V.
FSGS: Real-Time Few-shot View Synthesis using Gaussian Splatting
Authors: name: Zehao Zhu, name: Zhiwen Fan, name: Yifan Jiang, name: Zhangyang Wang
Published: 2023-12-01 09:30:02

Novel view synthesis from limited observations remains an important and
persistent task. However, high efficiency in existing NeRF-based few-shot view
synthesis is often compromised to obtain an accurate 3D representation. To
address this challenge, we propose a few-shot view synthesis framework based on
3D Gaussian Splatting that enables real-time and photo-realistic view synthesis
with as few as three training views. The proposed method, dubbed FSGS, handles
the extremely sparse initialized SfM points with a thoughtfully designed
Gaussian Unpooling process. Our method iteratively distributes new Gaussians
around the most representative locations, subsequently infilling local details
in vacant areas. We also integrate a large-scale pre-trained monocular depth
estimator within the Gaussians optimization process, leveraging online
augmented views to guide the geometric optimization towards an optimal
solution. Starting from sparse points observed from limited input viewpoints,
our FSGS can accurately grow into unseen regions, comprehensively covering the
scene and boosting the rendering quality of novel views. Overall, FSGS achieves
state-of-the-art performance in both accuracy and rendering efficiency across
diverse datasets, including LLFF, Mip-NeRF360, and Blender. Project website:
https://zehaozhu.github.io/FSGS/.
Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering
Authors: name: Tao Lu, name: Mulin Yu, name: Linning Xu, name: Yuanbo Xiangli, name: Limin Wang, name: Dahua Lin, name: Bo Dai
Published: 2023-11-30 17:58:57

Neural rendering methods have significantly advanced photo-realistic 3D scene
rendering in various academic and industrial applications. The recent 3D
Gaussian Splatting method has achieved the state-of-the-art rendering quality
and speed combining the benefits of both primitive-based representations and
volumetric representations. However, it often leads to heavily redundant
Gaussians that try to fit every training view, neglecting the underlying scene
geometry. Consequently, the resulting model becomes less robust to significant
view changes, texture-less area and lighting effects. We introduce Scaffold-GS,
which uses anchor points to distribute local 3D Gaussians, and predicts their
attributes on-the-fly based on viewing direction and distance within the view
frustum. Anchor growing and pruning strategies are developed based on the
importance of neural Gaussians to reliably improve the scene coverage. We show
that our method effectively reduces redundant Gaussians while delivering
high-quality rendering. We also demonstrates an enhanced capability to
accommodate scenes with varying levels-of-detail and view-dependent
observations, without sacrificing the rendering speed.