Orionnnn/hub-stats huggingface dataset:
Vript: A Video Is Worth Thousands of Words
Authors: name: Dongjie Yang, name: Suyuan Huang, name: Chengqiang Lu, name: Xiaodong Han, name: Haoxin Zhang, name: Yan Gao, name: Yao Hu, name: Hai Zhao
Published: 2024-06-10 06:17:55

Advancements in multimodal learning, particularly in video understanding and
generation, require high-quality video-text datasets for improved model
performance. Vript addresses this issue with a meticulously annotated corpus of
12K high-resolution videos, offering detailed, dense, and script-like captions
for over 420K clips. Each clip has a caption of ~145 words, which is over 10x
longer than most video-text datasets. Unlike captions only documenting static
content in previous datasets, we enhance video captioning to video scripting by
documenting not just the content, but also the camera operations, which include
the shot types (medium shot, close-up, etc) and camera movements (panning,
tilting, etc). By utilizing the Vript, we explore three training paradigms of
aligning more text with the video modality rather than clip-caption pairs. This
results in Vriptor, a top-performing video captioning model among open-source
models, comparable to GPT-4V in performance. Vriptor is also a powerful model
capable of end-to-end generation of dense and detailed captions for long
videos. Moreover, we introduce Vript-Hard, a benchmark consisting of three
video understanding tasks that are more challenging than existing benchmarks:
Vript-HAL is the first benchmark evaluating action and object hallucinations in
video LLMs, Vript-RR combines reasoning with retrieval resolving question
ambiguity in long-video QAs, and Vript-ERO is a new task to evaluate the
temporal understanding of events in long videos rather than actions in short
videos in previous works. All code, models, and datasets are available in
https://github.com/mutonix/Vript.
IllumiNeRF: 3D Relighting without Inverse Rendering
Authors: name: Xiaoming Zhao, name: Pratul P. Srinivasan, name: Dor Verbin, name: Keunhong Park, name: Ricardo Martin Brualla, name: Philipp Henzler
Published: 2024-06-10 17:59:59

Existing methods for relightable view synthesis -- using a set of images of
an object under unknown lighting to recover a 3D representation that can be
rendered from novel viewpoints under a target illumination -- are based on
inverse rendering, and attempt to disentangle the object geometry, materials,
and lighting that explain the input images. Furthermore, this typically
involves optimization through differentiable Monte Carlo rendering, which is
brittle and computationally-expensive. In this work, we propose a simpler
approach: we first relight each input image using an image diffusion model
conditioned on lighting and then reconstruct a Neural Radiance Field (NeRF)
with these relit images, from which we render novel views under the target
lighting. We demonstrate that this strategy is surprisingly competitive and
achieves state-of-the-art results on multiple relighting benchmarks. Please see
our project page at https://illuminerf.github.io/.
Autoregressive Model Beats Diffusion: Llama for Scalable Image
  Generation
Authors: name: Peize Sun, name: Yi Jiang, name: Shoufa Chen, name: Shilong Zhang, name: Bingyue Peng, name: Ping Luo, name: Zehuan Yuan
Published: 2024-06-10 17:59:52

We introduce LlamaGen, a new family of image generation models that apply
original ``next-token prediction'' paradigm of large language models to visual
generation domain. It is an affirmative answer to whether vanilla
autoregressive models, e.g., Llama, without inductive biases on visual signals
can achieve state-of-the-art image generation performance if scaling properly.
We reexamine design spaces of image tokenizers, scalability properties of image
generation models, and their training data quality. The outcome of this
exploration consists of: (1) An image tokenizer with downsample ratio of 16,
reconstruction quality of 0.94 rFID and codebook usage of 97% on ImageNet
benchmark. (2) A series of class-conditional image generation models ranging
from 111M to 3.1B parameters, achieving 2.18 FID on ImageNet 256x256
benchmarks, outperforming the popular diffusion models such as LDM, DiT. (3) A
text-conditional image generation model with 775M parameters, from two-stage
training on LAION-COCO and high aesthetics quality images, demonstrating
competitive performance of visual quality and text alignment. (4) We verify the
effectiveness of LLM serving frameworks in optimizing the inference speed of
image generation models and achieve 326% - 414% speedup. We release all models
and codes to facilitate open-source community of visual generation and
multimodal foundation models.
Towards a Personal Health Large Language Model
Authors: name: Justin Cosentino, name: Anastasiya Belyaeva, name: Xin Liu, name: Nicholas A. Furlotte, name: Zhun Yang, name: Chace Lee, name: Erik Schenck, name: Yojan Patel, name: Jian Cui, name: Logan Douglas Schneider, name: Robby Bryant, name: Ryan G. Gomes, name: Allen Jiang, name: Roy Lee, name: Yun Liu, name: Javier Perez, name: Jameson K. Rogers, name: Cathy Speed, name: Shyam Tailor, name: Megan Walker, name: Jeffrey Yu, name: Tim Althoff, name: Conor Heneghan, name: John Hernandez, name: Mark Malhotra, name: Leor Stern, name: Yossi Matias, name: Greg S. Corrado, name: Shwetak Patel, name: Shravya Shetty, name: Jiening Zhan, name: Shruthi Prabhakara, name: Daniel McDuff, name: Cory Y. McLean
Published: 2024-06-10 17:16:49

In health, most large language model (LLM) research has focused on clinical
tasks. However, mobile and wearable devices, which are rarely integrated into
such tasks, provide rich, longitudinal data for personal health monitoring.
Here we present Personal Health Large Language Model (PH-LLM), fine-tuned from
Gemini for understanding and reasoning over numerical time-series personal
health data. We created and curated three datasets that test 1) production of
personalized insights and recommendations from sleep patterns, physical
activity, and physiological responses, 2) expert domain knowledge, and 3)
prediction of self-reported sleep outcomes. For the first task we designed 857
case studies in collaboration with domain experts to assess real-world
scenarios in sleep and fitness. Through comprehensive evaluation of
domain-specific rubrics, we observed that Gemini Ultra 1.0 and PH-LLM are not
statistically different from expert performance in fitness and, while experts
remain superior for sleep, fine-tuning PH-LLM provided significant improvements
in using relevant domain knowledge and personalizing information for sleep
insights. We evaluated PH-LLM domain knowledge using multiple choice sleep
medicine and fitness examinations. PH-LLM achieved 79% on sleep and 88% on
fitness, exceeding average scores from a sample of human experts. Finally, we
trained PH-LLM to predict self-reported sleep quality outcomes from textual and
multimodal encoding representations of wearable data, and demonstrate that
multimodal encoding is required to match performance of specialized
discriminative models. Although further development and evaluation are
necessary in the safety-critical personal health domain, these results
demonstrate both the broad knowledge and capabilities of Gemini models and the
benefit of contextualizing physiological data for personal health applications
as done with PH-LLM.
Unified Text-to-Image Generation and Retrieval
Authors: name: Leigang Qu, name: Haochuan Li, name: Tan Wang, name: Wenjie Wang, name: Yongqi Li, name: Liqiang Nie, name: Tat-Seng Chua
Published: 2024-06-09 15:00:28

How humans can efficiently and effectively acquire images has always been a
perennial question. A typical solution is text-to-image retrieval from an
existing database given the text query; however, the limited database typically
lacks creativity. By contrast, recent breakthroughs in text-to-image generation
have made it possible to produce fancy and diverse visual content, but it faces
challenges in synthesizing knowledge-intensive images. In this work, we rethink
the relationship between text-to-image generation and retrieval and propose a
unified framework in the context of Multimodal Large Language Models (MLLMs).
Specifically, we first explore the intrinsic discriminative abilities of MLLMs
and introduce a generative retrieval method to perform retrieval in a
training-free manner. Subsequently, we unify generation and retrieval in an
autoregressive generation way and propose an autonomous decision module to
choose the best-matched one between generated and retrieved images as the
response to the text query. Additionally, we construct a benchmark called
TIGeR-Bench, including creative and knowledge-intensive domains, to standardize
the evaluation of unified text-to-image generation and retrieval. Extensive
experimental results on TIGeR-Bench and two retrieval benchmarks, i.e.,
Flickr30K and MS-COCO, demonstrate the superiority and effectiveness of our
proposed method.
VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text
  to Speech Synthesizers
Authors: name: Sanyuan Chen, name: Shujie Liu, name: Long Zhou, name: Yanqing Liu, name: Xu Tan, name: Jinyu Li, name: Sheng Zhao, name: Yao Qian, name: Furu Wei
Published: 2024-06-08 06:31:03

This paper introduces VALL-E 2, the latest advancement in neural codec
language models that marks a milestone in zero-shot text-to-speech synthesis
(TTS), achieving human parity for the first time. Based on its predecessor,
VALL-E, the new iteration introduces two significant enhancements: Repetition
Aware Sampling refines the original nucleus sampling process by accounting for
token repetition in the decoding history. It not only stabilizes the decoding
but also circumvents the infinite loop issue. Grouped Code Modeling organizes
codec codes into groups to effectively shorten the sequence length, which not
only boosts inference speed but also addresses the challenges of long sequence
modeling. Our experiments on the LibriSpeech and VCTK datasets show that VALL-E
2 surpasses previous systems in speech robustness, naturalness, and speaker
similarity. It is the first of its kind to reach human parity on these
benchmarks. Moreover, VALL-E 2 consistently synthesizes high-quality speech,
even for sentences that are traditionally challenging due to their complexity
or repetitive phrases. The advantages of this work could contribute to valuable
endeavors, such as generating speech for individuals with aphasia or people
with amyotrophic lateral sclerosis. Demos of VALL-E 2 will be posted to
https://aka.ms/valle2.
MLCM: Multistep Consistency Distillation of Latent Diffusion Model
Authors: name: Qingsong Xie, name: Zhenyi Liao, name: Zhijie Deng, name: Chen chen, name: Shixiang Tang, name: Haonan Lu
Published: 2024-06-09 12:55:50

Distilling large latent diffusion models (LDMs) into ones that are fast to
sample from is attracting growing research interest. However, the majority of
existing methods face a dilemma where they either (i) depend on multiple
individual distilled models for different sampling budgets, or (ii) sacrifice
generation quality with limited (e.g., 2-4) and/or moderate (e.g., 5-8)
sampling steps. To address these, we extend the recent multistep consistency
distillation (MCD) strategy to representative LDMs, establishing the Multistep
Latent Consistency Models (MLCMs) approach for low-cost high-quality image
synthesis. MLCM serves as a unified model for various sampling steps due to the
promise of MCD. We further augment MCD with a progressive training strategy to
strengthen inter-segment consistency to boost the quality of few-step
generations. We take the states from the sampling trajectories of the teacher
model as training data for MLCMs to lift the requirements for high-quality
training datasets and to bridge the gap between the training and inference of
the distilled model. MLCM is compatible with preference learning strategies for
further improvement of visual quality and aesthetic appeal. Empirically, MLCM
can generate high-quality, delightful images with only 2-8 sampling steps. On
the MSCOCO-2017 5K benchmark, MLCM distilled from SDXL gets a CLIP Score of
33.30, Aesthetic Score of 6.19, and Image Reward of 1.20 with only 4 steps,
substantially surpassing 4-step LCM [23], 8-step SDXL-Lightning [17], and
8-step HyperSD [33]. We also demonstrate the versatility of MLCMs in
applications including controllable generation, image style transfer, and
Chinese-to-image generation.
Large Language Model Confidence Estimation via Black-Box Access
Authors: name: Tejaswini Pedapati, name: Amit Dhurandhar, name: Soumya Ghosh, name: Soham Dan, name: Prasanna Sattigeri
Published: 2024-06-01 02:08:44

Estimating uncertainty or confidence in the responses of a model can be
significant in evaluating trust not only in the responses, but also in the
model as a whole. In this paper, we explore the problem of estimating
confidence for responses of large language models (LLMs) with simply black-box
or query access to them. We propose a simple and extensible framework where, we
engineer novel features and train a (interpretable) model (viz. logistic
regression) on these features to estimate the confidence. We empirically
demonstrate that our simple framework is effective in estimating confidence of
flan-ul2, llama-13b and mistral-7b with it consistently outperforming existing
black-box confidence estimation approaches on benchmark datasets such as
TriviaQA, SQuAD, CoQA and Natural Questions by even over 10% (on AUROC) in
some cases. Additionally, our interpretable approach provides insight into
features that are predictive of confidence, leading to the interesting and
useful discovery that our confidence models built for one LLM generalize
zero-shot across others on a given dataset.
Proofread: Fixes All Errors with One Tap
Authors: name: Renjie Liu, name: Yanxiang Zhang, name: Yun Zhu, name: Haicheng Sun, name: Yuanbo Zhang, name: Michael Xuelin Huang, name: Shanqing Cai, name: Lei Meng, name: Shumin Zhai
Published: 2024-06-06 21:38:08

The impressive capabilities in Large Language Models (LLMs) provide a
powerful approach to reimagine users' typing experience. This paper
demonstrates Proofread, a novel Gboard feature powered by a server-side LLM in
Gboard, enabling seamless sentence-level and paragraph-level corrections with a
single tap. We describe the complete system in this paper, from data
generation, metrics design to model tuning and deployment. To obtain models
with sufficient quality, we implement a careful data synthetic pipeline
tailored to online use cases, design multifaceted metrics, employ a two-stage
tuning approach to acquire the dedicated LLM for the feature: the Supervised
Fine Tuning (SFT) for foundational quality, followed by the Reinforcement
Learning (RL) tuning approach for targeted refinement. Specifically, we find
sequential tuning on Rewrite and proofread tasks yields the best quality in SFT
stage, and propose global and direct rewards in the RL tuning stage to seek
further improvement. Extensive experiments on a human-labeled golden set showed
our tuned PaLM2-XS model achieved 85.56\% good ratio. We launched the feature
to Pixel 8 devices by serving the model on TPU v5 in Google Cloud, with
thousands of daily active users. Serving latency was significantly reduced by
quantization, bucket inference, text segmentation, and speculative decoding.
Our demo could be seen in https://youtu.be/4ZdcuiwFU7I{Youtube}.
Boosting Large-scale Parallel Training Efficiency with C4: A
  Communication-Driven Approach
Authors: name: Jianbo Dong, name: Bin Luo, name: Jun Zhang, name: Pengcheng Zhang, name: Fei Feng, name: Yikai Zhu, name: Ang Liu, name: Zian Chen, name: Yi Shi, name: Hairong Jiao, name: Gang Lu, name: Yu Guan, name: Ennan Zhai, name: Wencong Xiao, name: Hanyu Zhao, name: Man Yuan, name: Siran Yang, name: Xiang Li, name: Jiamang Wang, name: Rui Men, name: Jianwei Zhang, name: Huang Zhong, name: Dennis Cai, name: Yuan Xie, name: Binzhang Fu
Published: 2024-06-07 02:58:35

The emergence of Large Language Models (LLMs) has necessitated the adoption
of parallel training techniques, involving the deployment of thousands of GPUs
to train a single model. Unfortunately, we have found that the efficiency of
current parallel training is often suboptimal, largely due to the following two
main issues. Firstly, hardware failures are inevitable, leading to
interruptions in the training tasks. The inability to quickly identify the
faulty components results in a substantial waste of GPU resources. Secondly,
since GPUs must wait for parameter synchronization to complete before
proceeding to the next round of computation, network congestions can greatly
increase the waiting time for GPUs. To address these challenges, this paper
introduces a communication-driven solution, namely the C4. The key insights of
C4 are two folds. First, in parallel training, collective communication
exhibits periodic and homogeneous characteristics, so any anomalies are
certainly due to some form of hardware malfunction. By leveraging this feature,
C4 can rapidly identify the faulty components, swiftly isolate the anomaly, and
restart the task, thereby avoiding resource wastage caused by delays in anomaly
detection. Second, the predictable communication model of collective
communication, involving few large flows, allows C4 to efficiently execute
traffic planning, substantially reducing network congestion. C4 has been
extensively implemented across our production systems, cutting error-induced
overhead by roughly 30% and enhancing runtime performance by about 15% for
certain applications with moderate communication costs.
CRAG -- Comprehensive RAG Benchmark
Authors: name: Xiao Yang, name: Kai Sun, name: Hao Xin, name: Yushi Sun, name: Nikita Bhalla, name: Xiangsen Chen, name: Sajal Choudhary, name: Rongze Daniel Gui, name: Ziran Will Jiang, name: Ziyu Jiang, name: Lingkun Kong, name: Brian Moran, name: Jiaqi Wang, name: Yifan Ethan Xu, name: An Yan, name: Chenyu Yang, name: Eting Yuan, name: Hanwen Zha, name: Nan Tang, name: Lei Chen, name: Nicolas Scheffer, name: Yue Liu, name: Nirav Shah, name: Rakesh Wanga, name: Anuj Kumar, name: Wen-tau Yih, name: Xin Luna Dong
Published: 2024-06-07 08:43:07

Retrieval-Augmented Generation (RAG) has recently emerged as a promising
solution to alleviate Large Language Model (LLM)'s deficiency in lack of
knowledge. Existing RAG datasets, however, do not adequately represent the
diverse and dynamic nature of real-world Question Answering (QA) tasks. To
bridge this gap, we introduce the Comprehensive RAG Benchmark (CRAG), a factual
question answering benchmark of 4,409 question-answer pairs and mock APIs to
simulate web and Knowledge Graph (KG) search. CRAG is designed to encapsulate a
diverse array of questions across five domains and eight question categories,
reflecting varied entity popularity from popular to long-tail, and temporal
dynamisms ranging from years to seconds. Our evaluation on this benchmark
highlights the gap to fully trustworthy QA. Whereas most advanced LLMs achieve
<=34% accuracy on CRAG, adding RAG in a straightforward manner improves the
accuracy only to 44%. State-of-the-art industry RAG solutions only answer 63%
questions without any hallucination. CRAG also reveals much lower accuracy in
answering questions regarding facts with higher dynamism, lower popularity, or
higher complexity, suggesting future research directions. The CRAG benchmark
laid the groundwork for a KDD Cup 2024 challenge, attracting thousands of
participants and submissions within the first 50 days of the competition. We
commit to maintaining CRAG to serve research communities in advancing RAG
solutions and general QA solutions.
Why Has Predicting Downstream Capabilities of Frontier AI Models with
  Scale Remained Elusive?
Authors: name: Rylan Schaeffer, name: Hailey Schoelkopf, name: Brando Miranda, name: Gabriel Mukobi, name: Varun Madan, name: Adam Ibrahim, name: Herbie Bradley, name: Stella Biderman, name: Sanmi Koyejo
Published: 2024-06-06 17:46:56

Predictable behavior from scaling advanced AI systems is an extremely
desirable property. Although a well-established literature exists on how
pretraining performance scales, the literature on how particular downstream
capabilities scale is significantly muddier. In this work, we take a step back
and ask: why has predicting specific downstream capabilities with scale
remained elusive? While many factors are certainly responsible, we identify a
new factor that makes modeling scaling behavior on widely used multiple-choice
question-answering benchmarks challenging. Using five model families and twelve
well-established multiple-choice benchmarks, we show that downstream
performance is computed from negative log likelihoods via a sequence of
transformations that progressively degrade the statistical relationship between
performance and scale. We then reveal the mechanism causing this degradation:
downstream metrics require comparing the correct choice against a small number
of specific incorrect choices, meaning accurately predicting downstream
capabilities requires predicting not just how probability mass concentrates on
the correct choice with scale, but also how probability mass fluctuates on
specific incorrect choices with scale. We empirically study how probability
mass on the correct choice co-varies with probability mass on incorrect choices
with increasing compute, suggesting that scaling laws for incorrect choices
might be achievable. Our work also explains why pretraining scaling laws are
commonly regarded as more predictable than downstream capabilities and
contributes towards establishing scaling-predictable evaluations of frontier AI
models.
WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in
  the Wild
Authors: name: Bill Yuchen Lin, name: Yuntian Deng, name: Khyathi Chandu, name: Faeze Brahman, name: Abhilasha Ravichander, name: Valentina Pyatkin, name: Nouha Dziri, name: Ronan Le Bras, name: Yejin Choi
Published: 2024-06-07 09:15:44

We introduce WildBench, an automated evaluation framework designed to
benchmark large language models (LLMs) using challenging, real-world user
queries. WildBench consists of 1,024 tasks carefully selected from over one
million human-chatbot conversation logs. For automated evaluation with
WildBench, we have developed two metrics, WB-Reward and WB-Score, which are
computable using advanced LLMs such as GPT-4-turbo. WildBench evaluation uses
task-specific checklists to evaluate model outputs systematically and provides
structured explanations that justify the scores and comparisons, resulting in
more reliable and interpretable automatic judgments. WB-Reward employs
fine-grained pairwise comparisons between model responses, generating five
potential outcomes: much better, slightly better, slightly worse, much worse,
or a tie. Unlike previous evaluations that employed a single baseline model, we
selected three baseline models at varying performance levels to ensure a
comprehensive pairwise evaluation. Additionally, we propose a simple method to
mitigate length bias, by converting outcomes of ``slightly better/worse'' to
``tie'' if the winner response exceeds the loser one by more than K
characters. WB-Score evaluates the quality of model outputs individually,
making it a fast and cost-efficient evaluation metric. WildBench results
demonstrate a strong correlation with the human-voted Elo ratings from Chatbot
Arena on hard tasks. Specifically, WB-Reward achieves a Pearson correlation of
0.98 with top-ranking models. Additionally, WB-Score reaches 0.95, surpassing
both ArenaHard's 0.91 and AlpacaEval2.0's 0.89 for length-controlled win rates,
as well as the 0.87 for regular win rates.