Orionnnn/hub-stats huggingface dataset:
MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation
  in Videos
Authors: name: Xuehai He, name: Weixi Feng, name: Kaizhi Zheng, name: Yujie Lu, name: Wanrong Zhu, name: Jiachen Li, name: Yue Fan, name: Jianfeng Wang, name: Linjie Li, name: Zhengyuan Yang, name: Kevin Lin, name: William Yang Wang, name: Lijuan Wang, name: Xin Eric Wang
Published: 2024-06-12 16:54:54

Multimodal Language Language Models (MLLMs) demonstrate the emerging
abilities of "world models" -- interpreting and reasoning about complex
real-world dynamics. To assess these abilities, we posit videos are the ideal
medium, as they encapsulate rich representations of real-world dynamics and
causalities. To this end, we introduce MMWorld, a new benchmark for
multi-discipline, multi-faceted multimodal video understanding. MMWorld
distinguishes itself from previous video understanding benchmarks with two
unique advantages: (1) multi-discipline, covering various disciplines that
often require domain expertise for comprehensive understanding; (2)
multi-faceted reasoning, including explanation, counterfactual thinking, future
prediction, etc. MMWorld consists of a human-annotated dataset to evaluate
MLLMs with questions about the whole videos and a synthetic dataset to analyze
MLLMs within a single modality of perception. Together, MMWorld encompasses
1,910 videos across seven broad disciplines and 69 subdisciplines, complete
with 6,627 question-answer pairs and associated captions. The evaluation
includes 2 proprietary and 10 open-source MLLMs, which struggle on MMWorld
(e.g., GPT-4V performs the best with only 52.3\% accuracy), showing large room
for improvement. Further ablation studies reveal other interesting findings
such as models' different skill sets from humans. We hope MMWorld can serve as
an essential step towards world model evaluation in videos.
What If We Recaption Billions of Web Images with LLaMA-3?
Authors: name: Xianhang Li, name: Haoqin Tu, name: Mude Hui, name: Zeyu Wang, name: Bingchen Zhao, name: Junfei Xiao, name: Sucheng Ren, name: Jieru Mei, name: Qing Liu, name: Huangjie Zheng, name: Yuyin Zhou, name: Cihang Xie
Published: 2024-06-12 17:59:07

Web-crawled image-text pairs are inherently noisy. Prior studies demonstrate
that semantically aligning and enriching textual descriptions of these pairs
can significantly enhance model training across various vision-language tasks,
particularly text-to-image generation. However, large-scale investigations in
this area remain predominantly closed-source. Our paper aims to bridge this
community effort, leveraging the powerful and open-sourced LLaMA-3, a
GPT-4 level LLM. Our recaptioning pipeline is simple: first, we fine-tune a
LLaMA-3-8B powered LLaVA-1.5 and then employ it to recaption 1.3 billion images
from the DataComp-1B dataset. Our empirical results confirm that this enhanced
dataset, Recap-DataComp-1B, offers substantial benefits in training advanced
vision-language models. For discriminative models like CLIP, we observe
enhanced zero-shot performance in cross-modal retrieval tasks. For generative
models like text-to-image Diffusion Transformers, the generated images exhibit
a significant improvement in alignment with users' text instructions,
especially in following complex queries. Our project page is
https://www.haqtu.me/Recap-Datacomp-1B/
Hierarchical Patch Diffusion Models for High-Resolution Video Generation
Authors: name: Ivan Skorokhodov, name: Willi Menapace, name: Aliaksandr Siarohin, name: Sergey Tulyakov
Published: 2024-06-12 01:12:53

Diffusion models have demonstrated remarkable performance in image and video
synthesis. However, scaling them to high-resolution inputs is challenging and
requires restructuring the diffusion pipeline into multiple independent
components, limiting scalability and complicating downstream applications. This
makes it very efficient during training and unlocks end-to-end optimization on
high-resolution videos. We improve PDMs in two principled ways. First, to
enforce consistency between patches, we develop deep context fusion -- an
architectural technique that propagates the context information from low-scale
to high-scale patches in a hierarchical manner. Second, to accelerate training
and inference, we propose adaptive computation, which allocates more network
capacity and computation towards coarse image details. The resulting model sets
a new state-of-the-art FVD score of 66.32 and Inception Score of 87.68 in
class-conditional video generation on UCF-101 256^2, surpassing recent
methods by more than 100%. Then, we show that it can be rapidly fine-tuned from
a base 36times 64 low-resolution generator for high-resolution 64 times
288 times 512 text-to-video synthesis. To the best of our knowledge, our
model is the first diffusion-based architecture which is trained on such high
resolutions entirely end-to-end. Project webpage:
https://snap-research.github.io/hpdm.
AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and
  Video Generation
Authors: name: Kai Wang, name: Shijian Deng, name: Jing Shi, name: Dimitrios Hatzinakos, name: Yapeng Tian
Published: 2024-06-11 20:05:58

Recent Diffusion Transformers (DiTs) have shown impressive capabilities in
generating high-quality single-modality content, including images, videos, and
audio. However, it is still under-explored whether the transformer-based
diffuser can efficiently denoise the Gaussian noises towards superb multimodal
content creation. To bridge this gap, we introduce AV-DiT, a novel and
efficient audio-visual diffusion transformer designed to generate high-quality,
realistic videos with both visual and audio tracks. To minimize model
complexity and computational costs, AV-DiT utilizes a shared DiT backbone
pre-trained on image-only data, with only lightweight, newly inserted adapters
being trainable. This shared backbone facilitates both audio and video
generation. Specifically, the video branch incorporates a trainable temporal
attention layer into a frozen pre-trained DiT block for temporal consistency.
Additionally, a small number of trainable parameters adapt the image-based DiT
block for audio generation. An extra shared DiT block, equipped with
lightweight parameters, facilitates feature interaction between audio and
visual modalities, ensuring alignment. Extensive experiments on the AIST++ and
Landscape datasets demonstrate that AV-DiT achieves state-of-the-art
performance in joint audio-visual generation with significantly fewer tunable
parameters. Furthermore, our results highlight that a single shared image
generative backbone with modality-specific adaptations is sufficient for
constructing a joint audio-video generator. Our source code and pre-trained
models will be released.
Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs
  with Nothing
Authors: name: Zhangchen Xu, name: Fengqing Jiang, name: Luyao Niu, name: Yuntian Deng, name: Radha Poovendran, name: Yejin Choi, name: Bill Yuchen Lin
Published: 2024-06-12 17:52:30

High-quality instruction data is critical for aligning large language models
(LLMs). Although some models, such as Llama-3-Instruct, have open weights,
their alignment data remain private, which hinders the democratization of AI.
High human labor costs and a limited, predefined scope for prompting prevent
existing open-source data creation methods from scaling effectively,
potentially limiting the diversity and quality of public alignment datasets. Is
it possible to synthesize high-quality instruction data at scale by extracting
it directly from an aligned LLM? We present a self-synthesis method for
generating large-scale alignment data named Magpie. Our key observation is that
aligned LLMs like Llama-3-Instruct can generate a user query when we input only
the left-side templates up to the position reserved for user messages, thanks
to their auto-regressive nature. We use this method to prompt Llama-3-Instruct
and generate 4 million instructions along with their corresponding responses.
We perform a comprehensive analysis of the extracted data and select 300K
high-quality instances. To compare Magpie data with other public instruction
datasets, we fine-tune Llama-3-8B-Base with each dataset and evaluate the
performance of the fine-tuned models. Our results indicate that in some tasks,
models fine-tuned with Magpie perform comparably to the official
Llama-3-8B-Instruct, despite the latter being enhanced with 10 million data
points through supervised fine-tuning (SFT) and subsequent feedback learning.
We also show that using Magpie solely for SFT can surpass the performance of
previous public datasets utilized for both SFT and preference optimization,
such as direct preference optimization with UltraFeedback. This advantage is
evident on alignment benchmarks such as AlpacaEval, ArenaHard, and WildBench.
Merging Improves Self-Critique Against Jailbreak Attacks
Authors: name: Victor Gallego
Published: 2024-06-11 12:01:09

The robustness of large language models (LLMs) against adversarial
manipulations, such as jailbreak attacks, remains a significant challenge. In
this work, we propose an approach that enhances the self-critique capability of
the LLM and further fine-tunes it over sanitized synthetic data. This is done
with the addition of an external critic model that can be merged with the
original, thus bolstering self-critique capabilities and improving the
robustness of the LLMs response to adversarial prompts. Our results demonstrate
that the combination of merging and self-critique can reduce the attack success
rate of adversaries significantly, thus offering a promising defense mechanism
against jailbreak attacks. Code, data and models released at
https://github.com/vicgalle/merging-self-critique-jailbreaks .
Neural Gaffer: Relighting Any Object via Diffusion
Authors: name: Haian Jin, name: Yuan Li, name: Fujun Luan, name: Yuanbo Xiangli, name: Sai Bi, name: Kai Zhang, name: Zexiang Xu, name: Jin Sun, name: Noah Snavely
Published: 2024-06-11 17:50:15

Single-image relighting is a challenging task that involves reasoning about
the complex interplay between geometry, materials, and lighting. Many prior
methods either support only specific categories of images, such as portraits,
or require special capture conditions, like using a flashlight. Alternatively,
some methods explicitly decompose a scene into intrinsic components, such as
normals and BRDFs, which can be inaccurate or under-expressive. In this work,
we propose a novel end-to-end 2D relighting diffusion model, called Neural
Gaffer, that takes a single image of any object and can synthesize an accurate,
high-quality relit image under any novel environmental lighting condition,
simply by conditioning an image generator on a target environment map, without
an explicit scene decomposition. Our method builds on a pre-trained diffusion
model, and fine-tunes it on a synthetic relighting dataset, revealing and
harnessing the inherent understanding of lighting present in the diffusion
model. We evaluate our model on both synthetic and in-the-wild Internet imagery
and demonstrate its advantages in terms of generalization and accuracy.
Moreover, by combining with other generative methods, our model enables many
downstream 2D tasks, such as text-based relighting and object insertion. Our
model can also operate as a strong relighting prior for 3D tasks, such as
relighting a radiance field.
McEval: Massively Multilingual Code Evaluation
Authors: name: Linzheng Chai, name: Shukai Liu, name: Jian Yang, name: Yuwei Yin, name: Ke Jin, name: Jiaheng Liu, name: Tao Sun, name: Ge Zhang, name: Changyu Ren, name: Hongcheng Guo, name: Zekun Wang, name: Boyang Wang, name: Xianjie Wu, name: Bing Wang, name: Tongliang Li, name: Liqun Yang, name: Sufeng Duan, name: Zhoujun Li
Published: 2024-06-11 16:45:17

Code large language models (LLMs) have shown remarkable advances in code
understanding, completion, and generation tasks. Programming benchmarks,
comprised of a selection of code challenges and corresponding test cases, serve
as a standard to evaluate the capability of different LLMs in such tasks.
However, most existing benchmarks primarily focus on Python and are still
restricted to a limited number of languages, where other languages are
translated from the Python samples (e.g. MultiPL-E) degrading the data
diversity. To further facilitate the research of code LLMs, we propose a
massively multilingual code benchmark covering 40 programming languages
(McEval) with 16K test samples, which substantially pushes the limits of code
LLMs in multilingual scenarios. The benchmark contains challenging code
completion, understanding, and generation evaluation tasks with finely curated
massively multilingual instruction corpora McEval-Instruct. In addition, we
introduce an effective multilingual coder mCoder trained on McEval-Instruct to
support multilingual programming language generation. Extensive experimental
results on McEval show that there is still a difficult journey between
open-source models and closed-source LLMs (e.g. GPT-series models) in numerous
languages. The instruction corpora, evaluation benchmark, and leaderboard are
available at https://mceval.github.io/.
Separating the "Chirp" from the "Chat": Self-supervised Visual Grounding
  of Sound and Language
Authors: name: Mark Hamilton, name: Andrew Zisserman, name: John R. Hershey, name: William T. Freeman
Published: 2024-06-09 03:38:21

We present DenseAV, a novel dual encoder grounding architecture that learns
high-resolution, semantically meaningful, and audio-visually aligned features
solely through watching videos. We show that DenseAV can discover the
``meaning'' of words and the ``location'' of sounds without explicit
localization supervision. Furthermore, it automatically discovers and
distinguishes between these two types of associations without supervision. We
show that DenseAV's localization abilities arise from a new multi-head feature
aggregation operator that directly compares dense image and audio
representations for contrastive learning. In contrast, many other systems that
learn ``global'' audio and video representations cannot localize words and
sound. Finally, we contribute two new datasets to improve the evaluation of AV
representations through speech and sound prompted semantic segmentation. On
these and other datasets we show DenseAV dramatically outperforms the prior art
on speech and sound prompted semantic segmentation. DenseAV outperforms the
previous state-of-the-art, ImageBind, on cross-modal retrieval using fewer than
half of the parameters. Project Page:
https://aka.ms/denseav{https://aka.ms/denseav}
SEE-2-SOUND: Zero-Shot Spatial Environment-to-Spatial Sound
Authors: name: Rishit Dagli, name: Shivesh Prakash, name: Robert Wu, name: Houman Khosravani
Published: 2024-06-06 22:55:01

Generating combined visual and auditory sensory experiences is critical for
the consumption of immersive content. Recent advances in neural generative
models have enabled the creation of high-resolution content across multiple
modalities such as images, text, speech, and videos. Despite these successes,
there remains a significant gap in the generation of high-quality spatial audio
that complements generated visual content. Furthermore, current audio
generation models excel in either generating natural audio or speech or music
but fall short in integrating spatial audio cues necessary for immersive
experiences. In this work, we introduce SEE-2-SOUND, a zero-shot approach that
decomposes the task into (1) identifying visual regions of interest; (2)
locating these elements in 3D space; (3) generating mono-audio for each; and
(4) integrating them into spatial audio. Using our framework, we demonstrate
compelling results for generating spatial audio for high-quality videos,
images, and dynamic images from the internet, as well as media generated by
learned approaches.
MedFuzz: Exploring the Robustness of Large Language Models in Medical
  Question Answering
Authors: name: Robert Osazuwa Ness, name: Katie Matton, name: Hayden Helm, name: Sheng Zhang, name: Junaid Bajwa, name: Carey E. Priebe, name: Eric Horvitz
Published: 2024-06-03 18:15:56

Large language models (LLM) have achieved impressive performance on medical
question-answering benchmarks. However, high benchmark accuracy does not imply
that the performance generalizes to real-world clinical settings. Medical
question-answering benchmarks rely on assumptions consistent with quantifying
LLM performance but that may not hold in the open world of the clinic. Yet LLMs
learn broad knowledge that can help the LLM generalize to practical conditions
regardless of unrealistic assumptions in celebrated benchmarks. We seek to
quantify how well LLM medical question-answering benchmark performance
generalizes when benchmark assumptions are violated. Specifically, we present
an adversarial method that we call MedFuzz (for medical fuzzing). MedFuzz
attempts to modify benchmark questions in ways aimed at confounding the LLM. We
demonstrate the approach by targeting strong assumptions about patient
characteristics presented in the MedQA benchmark. Successful "attacks" modify a
benchmark item in ways that would be unlikely to fool a medical expert but
nonetheless "trick" the LLM into changing from a correct to an incorrect
answer. Further, we present a permutation test technique that can ensure a
successful attack is statistically significant. We show how to use performance
on a "MedFuzzed" benchmark, as well as individual successful attacks. The
methods show promise at providing insights into the ability of an LLM to
operate robustly in more realistic settings.
Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts
  Language Models
Authors: name: Tianwen Wei, name: Bo Zhu, name: Liang Zhao, name: Cheng Cheng, name: Biye Li, name: Weiwei Lü, name: Peng Cheng, name: Jianhao Zhang, name: Xiaoyu Zhang, name: Liang Zeng, name: Xiaokun Wang, name: Yutuan Ma, name: Rui Hu, name: Shuicheng Yan, name: Han Fang, name: Yahui Zhou
Published: 2024-06-03 03:58:41

In this technical report, we introduce the training methodologies implemented
in the development of Skywork-MoE, a high-performance mixture-of-experts (MoE)
large language model (LLM) with 146 billion parameters and 16 experts. It is
initialized from the pre-existing dense checkpoints of our Skywork-13B model.
We explore the comparative effectiveness of upcycling versus training from
scratch initializations. Our findings suggest that the choice between these two
approaches should consider both the performance of the existing dense
checkpoints and the MoE training budget. We highlight two innovative
techniques: gating logit normalization, which improves expert diversification,
and adaptive auxiliary loss coefficients, allowing for layer-specific
adjustment of auxiliary loss coefficients. Our experimental results validate
the effectiveness of these methods. Leveraging these techniques and insights,
we trained our upcycled Skywork-MoE on a condensed subset of our SkyPile
corpus. The evaluation results demonstrate that our model delivers strong
performance across a wide range of benchmarks.
The Prompt Report: A Systematic Survey of Prompting Techniques
Authors: name: Sander Schulhoff, name: Michael Ilie, name: Nishant Balepur, name: Konstantine Kahadze, name: Amanda Liu, name: Chenglei Si, name: Yinheng Li, name: Aayush Gupta, name: HyoJung Han, name: Sevien Schulhoff, name: Pranav Sandeep Dulepet, name: Saurav Vidyadhara, name: Dayeon Ki, name: Sweta Agrawal, name: Chau Pham, name: Gerson Kroiz, name: Feileen Li, name: Hudson Tao, name: Ashay Srivastava, name: Hevander Da Costa, name: Saloni Gupta, name: Megan L. Rogers, name: Inna Goncearenco, name: Giuseppe Sarli, name: Igor Galynker, name: Denis Peskoff, name: Marine Carpuat, name: Jules White, name: Shyamal Anadkat, name: Alexander Hoyle, name: Philip Resnik
Published: 2024-06-06 18:10:11

Generative Artificial Intelligence (GenAI) systems are being increasingly
deployed across all parts of industry and research settings. Developers and end
users interact with these systems through the use of prompting or prompt
engineering. While prompting is a widespread and highly researched concept,
there exists conflicting terminology and a poor ontological understanding of
what constitutes a prompt due to the area's nascency. This paper establishes a
structured understanding of prompts, by assembling a taxonomy of prompting
techniques and analyzing their use. We present a comprehensive vocabulary of 33
vocabulary terms, a taxonomy of 58 text-only prompting techniques, and 40
techniques for other modalities. We further present a meta-analysis of the
entire literature on natural language prefix-prompting.
An Image is Worth 32 Tokens for Reconstruction and Generation
Authors: name: Qihang Yu, name: Mark Weber, name: Xueqing Deng, name: Xiaohui Shen, name: Daniel Cremers, name: Liang-Chieh Chen
Published: 2024-06-11 17:59:56

Recent advancements in generative models have highlighted the crucial role of
image tokenization in the efficient synthesis of high-resolution images.
Tokenization, which transforms images into latent representations, reduces
computational demands compared to directly processing pixels and enhances the
effectiveness and efficiency of the generation process. Prior methods, such as
VQGAN, typically utilize 2D latent grids with fixed downsampling factors.
However, these 2D tokenizations face challenges in managing the inherent
redundancies present in images, where adjacent regions frequently display
similarities. To overcome this issue, we introduce Transformer-based
1-Dimensional Tokenizer (TiTok), an innovative approach that tokenizes images
into 1D latent sequences. TiTok provides a more compact latent representation,
yielding substantially more efficient and effective representations than
conventional techniques. For example, a 256 x 256 x 3 image can be reduced to
just 32 discrete tokens, a significant reduction from the 256 or 1024 tokens
obtained by prior methods. Despite its compact nature, TiTok achieves
competitive performance to state-of-the-art approaches. Specifically, using the
same generator framework, TiTok attains 1.97 gFID, outperforming MaskGIT
baseline significantly by 4.21 at ImageNet 256 x 256 benchmark. The advantages
of TiTok become even more significant when it comes to higher resolution. At
ImageNet 512 x 512 benchmark, TiTok not only outperforms state-of-the-art
diffusion model DiT-XL/2 (gFID 2.74 vs. 3.04), but also reduces the image
tokens by 64x, leading to 410x faster generation process. Our best-performing
variant can significantly surpasses DiT-XL/2 (gFID 2.13 vs. 3.04) while still
generating high-quality samples 74x faster.
Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo
  Tree Self-refine with LLaMa-3 8B
Authors: name: Di Zhang, name: Jiatong Li, name: Xiaoshui Huang, name: Dongzhan Zhou, name: Yuqiang Li, name: Wanli Ouyang
Published: 2024-06-11 16:01:07

This paper introduces the MCT Self-Refine (MCTSr) algorithm, an innovative
integration of Large Language Models (LLMs) with Monte Carlo Tree Search
(MCTS), designed to enhance performance in complex mathematical reasoning
tasks. Addressing the challenges of accuracy and reliability in LLMs,
particularly in strategic and mathematical reasoning, MCTSr leverages
systematic exploration and heuristic self-refine mechanisms to improve
decision-making frameworks within LLMs. The algorithm constructs a Monte Carlo
search tree through iterative processes of Selection, self-refine,
self-evaluation, and Backpropagation, utilizing an improved Upper Confidence
Bound (UCB) formula to optimize the exploration-exploitation balance. Extensive
experiments demonstrate MCTSr's efficacy in solving Olympiad-level mathematical
problems, significantly improving success rates across multiple datasets,
including GSM8K, GSM Hard, MATH, and Olympiad-level benchmarks, including Math
Odyssey, AIME, and OlympiadBench. The study advances the application of LLMs in
complex reasoning tasks and sets a foundation for future AI integration,
enhancing decision-making accuracy and reliability in LLM-driven applications.