Orionnnn/hub-stats huggingface dataset:
Vid3D: Synthesis of Dynamic 3D Scenes using 2D Video Diffusion
Authors: name: Rishab Parthasarathy, name: Zack Ankner, name: Aaron Gokaslan
Published: 2024-06-17 04:09:04

A recent frontier in computer vision has been the task of 3D video
generation, which consists of generating a time-varying 3D representation of a
scene. To generate dynamic 3D scenes, current methods explicitly model 3D
temporal dynamics by jointly optimizing for consistency across both time and
views of the scene. In this paper, we instead investigate whether it is
necessary to explicitly enforce multiview consistency over time, as current
approaches do, or if it is sufficient for a model to generate 3D
representations of each timestep independently. We hence propose a model,
Vid3D, that leverages 2D video diffusion to generate 3D videos by first
generating a 2D "seed" of the video's temporal dynamics and then independently
generating a 3D representation for each timestep in the seed video. We evaluate
Vid3D against two state-of-the-art 3D video generation methods and find that
Vid3D is achieves comparable results despite not explicitly modeling 3D
temporal dynamics. We further ablate how the quality of Vid3D depends on the
number of views generated per frame. While we observe some degradation with
fewer views, performance degradation remains minor. Our results thus suggest
that 3D temporal knowledge may not be necessary to generate high-quality
dynamic 3D scenes, potentially enabling simpler generative algorithms for this
task.
mDPO: Conditional Preference Optimization for Multimodal Large Language
  Models
Authors: name: Fei Wang, name: Wenxuan Zhou, name: James Y. Huang, name: Nan Xu, name: Sheng Zhang, name: Hoifung Poon, name: Muhao Chen
Published: 2024-06-17 17:59:58

Direct preference optimization (DPO) has shown to be an effective method for
large language model (LLM) alignment. Recent works have attempted to apply DPO
to multimodal scenarios but have found it challenging to achieve consistent
improvement. Through a comparative experiment, we identify the unconditional
preference problem in multimodal preference optimization, where the model
overlooks the image condition. To address this problem, we propose mDPO, a
multimodal DPO objective that prevents the over-prioritization of language-only
preferences by also optimizing image preference. Moreover, we introduce a
reward anchor that forces the reward to be positive for chosen responses,
thereby avoiding the decrease in their likelihood -- an intrinsic problem of
relative preference optimization. Experiments on two multimodal LLMs of
different sizes and three widely used benchmarks demonstrate that mDPO
effectively addresses the unconditional preference problem in multimodal
preference optimization and significantly improves model performance,
particularly in reducing hallucination.
L4GM: Large 4D Gaussian Reconstruction Model
Authors: name: Jiawei Ren, name: Kevin Xie, name: Ashkan Mirzaei, name: Hanxue Liang, name: Xiaohui Zeng, name: Karsten Kreis, name: Ziwei Liu, name: Antonio Torralba, name: Sanja Fidler, name: Seung Wook Kim, name: Huan Ling
Published: 2024-06-14 17:51:18

We present L4GM, the first 4D Large Reconstruction Model that produces
animated objects from a single-view video input -- in a single feed-forward
pass that takes only a second. Key to our success is a novel dataset of
multiview videos containing curated, rendered animated objects from Objaverse.
This dataset depicts 44K diverse objects with 110K animations rendered in 48
viewpoints, resulting in 12M videos with a total of 300M frames. We keep our
L4GM simple for scalability and build directly on top of LGM, a pretrained 3D
Large Reconstruction Model that outputs 3D Gaussian ellipsoids from multiview
image input. L4GM outputs a per-frame 3D Gaussian Splatting representation from
video frames sampled at a low fps and then upsamples the representation to a
higher fps to achieve temporal smoothness. We add temporal self-attention
layers to the base LGM to help it learn consistency across time, and utilize a
per-timestep multiview rendering loss to train the model. The representation is
upsampled to a higher framerate by training an interpolation model which
produces intermediate 3D Gaussian representations. We showcase that L4GM that
is only trained on synthetic data generalizes extremely well on in-the-wild
videos, producing high quality animated 3D assets.
DataComp-LM: In search of the next generation of training sets for
  language models
Authors: name: Jeffrey Li, name: Alex Fang, name: Georgios Smyrnis, name: Maor Ivgi, name: Matt Jordan, name: Samir Gadre, name: Hritik Bansal, name: Etash Guha, name: Sedrick Keh, name: Kushal Arora, name: Saurabh Garg, name: Rui Xin, name: Niklas Muennighoff, name: Reinhard Heckel, name: Jean Mercat, name: Mayee Chen, name: Suchin Gururangan, name: Mitchell Wortsman, name: Alon Albalak, name: Yonatan Bitton, name: Marianna Nezhurina, name: Amro Abbas, name: Cheng-Yu Hsieh, name: Dhruba Ghosh, name: Josh Gardner, name: Maciej Kilian, name: Hanlin Zhang, name: Rulin Shao, name: Sarah Pratt, name: Sunny Sanyal, name: Gabriel Ilharco, name: Giannis Daras, name: Kalyani Marathe, name: Aaron Gokaslan, name: Jieyu Zhang, name: Khyathi Chandu, name: Thao Nguyen, name: Igor Vasiljevic, name: Sham Kakade, name: Shuran Song, name: Sujay Sanghavi, name: Fartash Faghri, name: Sewoong Oh, name: Luke Zettlemoyer, name: Kyle Lo, name: Alaaeldin El-Nouby, name: Hadi Pouransari, name: Alexander Toshev, name: Stephanie Wang, name: Dirk Groeneveld, name: Luca Soldani, name: Pang Wei Koh, name: Jenia Jitsev, name: Thomas Kollar, name: Alexandros G. Dimakis, name: Yair Carmon, name: Achal Dave, name: Ludwig Schmidt, name: Vaishaal Shankar
Published: 2024-06-17 17:42:57

We introduce DataComp for Language Models (DCLM), a testbed for controlled
dataset experiments with the goal of improving language models. As part of
DCLM, we provide a standardized corpus of 240T tokens extracted from Common
Crawl, effective pretraining recipes based on the OpenLM framework, and a broad
suite of 53 downstream evaluations. Participants in the DCLM benchmark can
experiment with data curation strategies such as deduplication, filtering, and
data mixing at model scales ranging from 412M to 7B parameters. As a baseline
for DCLM, we conduct extensive experiments and find that model-based filtering
is key to assembling a high-quality training set. The resulting dataset,
DCLM-Baseline enables training a 7B parameter language model from scratch to
64% 5-shot accuracy on MMLU with 2.6T training tokens. Compared to MAP-Neo, the
previous state-of-the-art in open-data language models, DCLM-Baseline
represents a 6.6 percentage point improvement on MMLU while being trained with
40% less compute. Our baseline model is also comparable to Mistral-7B-v0.3 and
Llama 3 8B on MMLU (63% & 66%), and performs similarly on an average of 53
natural language understanding tasks while being trained with 6.6x less compute
than Llama 3 8B. Our results highlight the importance of dataset design for
training language models and offer a starting point for further research on
data curation.
GAMA: A Large Audio-Language Model with Advanced Audio Understanding and
  Complex Reasoning Abilities
Authors: name: Sreyan Ghosh, name: Sonal Kumar, name: Ashish Seth, name: Chandra Kiran Reddy Evuru, name: Utkarsh Tyagi, name: S Sakshi, name: Oriol Nieto, name: Ramani Duraiswami, name: Dinesh Manocha
Published: 2024-06-17 17:31:01

Perceiving and understanding non-speech sounds and non-verbal speech is
essential to making decisions that help us interact with our surroundings. In
this paper, we propose GAMA, a novel General-purpose Large Audio-Language Model
(LALM) with Advanced Audio Understanding and Complex Reasoning Abilities. We
build GAMA by integrating an LLM with multiple types of audio representations,
including features from a custom Audio Q-Former, a multi-layer aggregator that
aggregates features from multiple layers of an audio encoder. We fine-tune GAMA
on a large-scale audio-language dataset, which augments it with audio
understanding capabilities. Next, we propose CompA-R (Instruction-Tuning for
Complex Audio Reasoning), a synthetically generated instruction-tuning (IT)
dataset with instructions that require the model to perform complex reasoning
on the input audio. We instruction-tune GAMA with CompA-R to endow it with
complex reasoning abilities, where we further add a soft prompt as input with
high-level semantic evidence by leveraging event tags of the input audio.
Finally, we also propose CompA-R-test, a human-labeled evaluation dataset for
evaluating the capabilities of LALMs on open-ended audio question-answering
that requires complex reasoning. Through automated and expert human
evaluations, we show that GAMA outperforms all other LALMs in literature on
diverse audio understanding tasks by margins of 1%-84%. Further, GAMA IT-ed on
CompA-R proves to be superior in its complex reasoning and instruction
following capabilities.
Be like a Goldfish, Don't Memorize! Mitigating Memorization in
  Generative LLMs
Authors: name: Abhimanyu Hans, name: Yuxin Wen, name: Neel Jain, name: John Kirchenbauer, name: Hamid Kazemi, name: Prajwal Singhania, name: Siddharth Singh, name: Gowthami Somepalli, name: Jonas Geiping, name: Abhinav Bhatele, name: Tom Goldstein
Published: 2024-06-14 17:44:22

Large language models can memorize and repeat their training data, causing
privacy and copyright risks. To mitigate memorization, we introduce a subtle
modification to the next-token training objective that we call the goldfish
loss. During training, a randomly sampled subset of tokens are excluded from
the loss computation. These dropped tokens are not memorized by the model,
which prevents verbatim reproduction of a complete chain of tokens from the
training set. We run extensive experiments training billion-scale Llama-2
models, both pre-trained and trained from scratch, and demonstrate significant
reductions in extractable memorization with little to no impact on downstream
benchmarks.
RVT-2: Learning Precise Manipulation from Few Demonstrations
Authors: name: Ankit Goyal, name: Valts Blukis, name: Jie Xu, name: Yijie Guo, name: Yu-Wei Chao, name: Dieter Fox
Published: 2024-06-12 18:00:01

In this work, we study how to build a robotic system that can solve multiple
3D manipulation tasks given language instructions. To be useful in industrial
and household domains, such a system should be capable of learning new tasks
with few demonstrations and solving them precisely. Prior works, like PerAct
and RVT, have studied this problem, however, they often struggle with tasks
requiring high precision. We study how to make them more effective, precise,
and fast. Using a combination of architectural and system-level improvements,
we propose RVT-2, a multitask 3D manipulation model that is 6X faster in
training and 2X faster in inference than its predecessor RVT. RVT-2 achieves a
new state-of-the-art on RLBench, improving the success rate from 65% to 82%.
RVT-2 is also effective in the real world, where it can learn tasks requiring
high precision, like picking up and inserting plugs, with just 10
demonstrations. Visual results, code, and trained model are provided at:
https://robotic-view-transformer-2.github.io/.
Vivid-ZOO: Multi-View Video Generation with Diffusion Model
Authors: name: Bing Li, name: Cheng Zheng, name: Wenxuan Zhu, name: Jinjie Mai, name: Biao Zhang, name: Peter Wonka, name: Bernard Ghanem
Published: 2024-06-12 21:44:04

While diffusion models have shown impressive performance in 2D image/video
generation, diffusion-based Text-to-Multi-view-Video (T2MVid) generation
remains underexplored. The new challenges posed by T2MVid generation lie in the
lack of massive captioned multi-view videos and the complexity of modeling such
multi-dimensional distribution. To this end, we propose a novel diffusion-based
pipeline that generates high-quality multi-view videos centered around a
dynamic 3D object from text. Specifically, we factor the T2MVid problem into
viewpoint-space and time components. Such factorization allows us to combine
and reuse layers of advanced pre-trained multi-view image and 2D video
diffusion models to ensure multi-view consistency as well as temporal coherence
for the generated multi-view videos, largely reducing the training cost. We
further introduce alignment modules to align the latent spaces of layers from
the pre-trained multi-view and the 2D video diffusion models, addressing the
reused layers' incompatibility that arises from the domain gap between 2D and
multi-view data. In support of this and future research, we further contribute
a captioned multi-view video dataset. Experimental results demonstrate that our
method generates high-quality multi-view videos, exhibiting vivid motions,
temporal coherence, and multi-view consistency, given a variety of text
prompts.
AV-GS: Learning Material and Geometry Aware Priors for Novel View
  Acoustic Synthesis
Authors: name: Swapnil Bhosale, name: Haosen Yang, name: Diptesh Kanojia, name: Jiankang Deng, name: Xiatian Zhu
Published: 2024-06-13 08:34:12

Novel view acoustic synthesis (NVAS) aims to render binaural audio at any
target viewpoint, given a mono audio emitted by a sound source at a 3D scene.
Existing methods have proposed NeRF-based implicit models to exploit visual
cues as a condition for synthesizing binaural audio. However, in addition to
low efficiency originating from heavy NeRF rendering, these methods all have a
limited ability of characterizing the entire scene environment such as room
geometry, material properties, and the spatial relation between the listener
and sound source. To address these issues, we propose a novel Audio-Visual
Gaussian Splatting (AV-GS) model. To obtain a material-aware and geometry-aware
condition for audio synthesis, we learn an explicit point-based scene
representation with an audio-guidance parameter on locally initialized Gaussian
points, taking into account the space relation from the listener and sound
source. To make the visual scene model audio adaptive, we propose a point
densification and pruning strategy to optimally distribute the Gaussian points,
with the per-point contribution in sound propagation (e.g., more points needed
for texture-less wall surfaces as they affect sound path diversion). Extensive
experiments validate the superiority of our AV-GS over existing alternatives on
the real-world RWAS and simulation-based SoundSpaces datasets.
VideoGUI: A Benchmark for GUI Automation from Instructional Videos
Authors: name: Kevin Qinghong Lin, name: Linjie Li, name: Difei Gao, name: Qinchen WU, name: Mingyi Yan, name: Zhengyuan Yang, name: Lijuan Wang, name: Mike Zheng Shou
Published: 2024-06-14 17:59:08

Graphical User Interface (GUI) automation holds significant promise for
enhancing human productivity by assisting with computer tasks. Existing task
formulations primarily focus on simple tasks that can be specified by a single,
language-only instruction, such as "Insert a new slide." In this work, we
introduce VideoGUI, a novel multi-modal benchmark designed to evaluate GUI
assistants on visual-centric GUI tasks. Sourced from high-quality web
instructional videos, our benchmark focuses on tasks involving professional and
novel software (e.g., Adobe Photoshop or Stable Diffusion WebUI) and complex
activities (e.g., video editing). VideoGUI evaluates GUI assistants through a
hierarchical process, allowing for identification of the specific levels at
which they may fail: (i) high-level planning: reconstruct procedural subtasks
from visual conditions without language descriptions; (ii) middle-level
planning: generate sequences of precise action narrations based on visual state
(i.e., screenshot) and goals; (iii) atomic action execution: perform specific
actions such as accurately clicking designated elements. For each level, we
design evaluation metrics across individual dimensions to provide clear
signals, such as individual performance in clicking, dragging, typing, and
scrolling for atomic action execution. Our evaluation on VideoGUI reveals that
even the SoTA large multimodal model GPT4o performs poorly on visual-centric
GUI tasks, especially for high-level planning.
SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for
  Southeast Asian Languages
Authors: name: Holy Lovenia, name: Rahmad Mahendra, name: Salsabil Maulana Akbar, name: Lester James V. Miranda, name: Jennifer Santoso, name: Elyanah Aco, name: Akhdan Fadhilah, name: Jonibek Mansurov, name: Joseph Marvin Imperial, name: Onno P. Kampman, name: Joel Ruben Antony Moniz, name: Muhammad Ravi Shulthan Habibi, name: Frederikus Hudi, name: Railey Montalan, name: Ryan Ignatius, name: Joanito Agili Lopo, name: William Nixon, name: Börje F. Karlsson, name: James Jaya, name: Ryandito Diandaru, name: Yuze Gao, name: Patrick Amadeus, name: Bin Wang, name: Jan Christian Blaise Cruz, name: Chenxi Whitehouse, name: Ivan Halim Parmonangan, name: Maria Khelli, name: Wenyu Zhang, name: Lucky Susanto, name: Reynard Adha Ryanda, name: Sonny Lazuardi Hermawan, name: Dan John Velasco, name: Muhammad Dehan Al Kautsar, name: Willy Fitra Hendria, name: Yasmin Moslem, name: Noah Flynn, name: Muhammad Farid Adilazuarda, name: Haochen Li, name: Johanes Lee, name: R. Damanhuri, name: Shuo Sun, name: Muhammad Reza Qorib, name: Amirbek Djanibekov, name: Wei Qi Leong, name: Quyet V. Do, name: Niklas Muennighoff, name: Tanrada Pansuwan, name: Ilham Firdausi Putra, name: Yan Xu, name: Ngee Chia Tai, name: Ayu Purwarianti, name: Sebastian Ruder, name: William Tjhi, name: Peerat Limkonchotiwat, name: Alham Fikri Aji, name: Sedrick Keh, name: Genta Indra Winata, name: Ruochen Zhang, name: Fajri Koto, name: Zheng-Xin Yong, name: Samuel Cahyawijaya
Published: 2024-06-14 15:23:39

Southeast Asia (SEA) is a region rich in linguistic diversity and cultural
variety, with over 1,300 indigenous languages and a population of 671 million
people. However, prevailing AI models suffer from a significant lack of
representation of texts, images, and audio datasets from SEA, compromising the
quality of AI models for SEA languages. Evaluating models for SEA languages is
challenging due to the scarcity of high-quality datasets, compounded by the
dominance of English training data, raising concerns about potential cultural
misrepresentation. To address these challenges, we introduce SEACrowd, a
collaborative initiative that consolidates a comprehensive resource hub that
fills the resource gap by providing standardized corpora in nearly 1,000 SEA
languages across three modalities. Through our SEACrowd benchmarks, we assess
the quality of AI models on 36 indigenous languages across 13 tasks, offering
valuable insights into the current AI landscape in SEA. Furthermore, we propose
strategies to facilitate greater AI advancements, maximizing potential utility
and resource equity for the future of AI in SEA.
Training-free Camera Control for Video Generation
Authors: name: Chen Hou, name: Guoqiang Wei, name: Yan Zeng, name: Zhibo Chen
Published: 2024-06-14 15:33:00

We propose a training-free and robust solution to offer camera movement
control for off-the-shelf video diffusion models. Unlike previous work, our
method does not require any supervised finetuning on camera-annotated datasets
or self-supervised training via data augmentation. Instead, it can be plugged
and played with most pretrained video diffusion models and generate camera
controllable videos with a single image or text prompt as input. The
inspiration of our work comes from the layout prior that intermediate latents
hold towards generated results, thus rearranging noisy pixels in them will make
output content reallocated as well. As camera move could also be seen as a kind
of pixel rearrangement caused by perspective change, videos could be
reorganized following specific camera motion if their noisy latents change
accordingly. Established on this, we propose our method CamTrol, which enables
robust camera control for video diffusion models. It is achieved by a two-stage
process. First, we model image layout rearrangement through explicit camera
movement in 3D point cloud space. Second, we generate videos with camera motion
using layout prior of noisy latents formed by a series of rearranged images.
Extensive experiments have demonstrated the robustness our method holds in
controlling camera motion of generated videos. Furthermore, we show that our
method can produce impressive results in generating 3D rotation videos with
dynamic content. Project page at https://lifedecoder.github.io/CamTrol/.
Make It Count: Text-to-Image Generation with an Accurate Number of
  Objects
Authors: name: Lital Binyamin, name: Yoad Tewel, name: Hilit Segev, name: Eran Hirsch, name: Royi Rassin, name: Gal Chechik
Published: 2024-06-14 17:46:08

Despite the unprecedented success of text-to-image diffusion models,
controlling the number of depicted objects using text is surprisingly hard.
This is important for various applications from technical documents, to
children's books to illustrating cooking recipes. Generating object-correct
counts is fundamentally challenging because the generative model needs to keep
a sense of separate identity for every instance of the object, even if several
objects look identical or overlap, and then carry out a global computation
implicitly during generation. It is still unknown if such representations
exist. To address count-correct generation, we first identify features within
the diffusion model that can carry the object identity information. We then use
them to separate and count instances of objects during the denoising process
and detect over-generation and under-generation. We fix the latter by training
a model that predicts both the shape and location of a missing object, based on
the layout of existing ones, and show how it can be used to guide denoising
with correct object count. Our approach, CountGen, does not depend on external
source to determine object layout, but rather uses the prior from the diffusion
model itself, creating prompt-dependent and seed-dependent layouts. Evaluated
on two benchmark datasets, we find that CountGen strongly outperforms the
count-accuracy of existing baselines.
BABILong: Testing the Limits of LLMs with Long Context
  Reasoning-in-a-Haystack
Authors: name: Yuri Kuratov, name: Aydar Bulatov, name: Petr Anokhin, name: Ivan Rodkin, name: Dmitry Sorokin, name: Artyom Sorokin, name: Mikhail Burtsev
Published: 2024-06-14 16:00:29

In recent years, the input context sizes of large language models (LLMs) have
increased dramatically. However, existing evaluation methods have not kept
pace, failing to comprehensively assess the efficiency of models in handling
long contexts. To bridge this gap, we introduce the BABILong benchmark,
designed to test language models' ability to reason across facts distributed in
extremely long documents. BABILong includes a diverse set of 20 reasoning
tasks, including fact chaining, simple induction, deduction, counting, and
handling lists/sets. These tasks are challenging on their own, and even more
demanding when the required facts are scattered across long natural text. Our
evaluations show that popular LLMs effectively utilize only 10-20\% of the
context and their performance declines sharply with increased reasoning
complexity. Among alternatives to in-context reasoning, Retrieval-Augmented
Generation methods achieve a modest 60\% accuracy on single-fact question
answering, independent of context length. Among context extension methods, the
highest performance is demonstrated by recurrent memory transformers, enabling
the processing of lengths up to 11 million tokens. The BABILong benchmark is
extendable to any length to support the evaluation of new upcoming models with
increased capabilities, and we provide splits up to 1 million token lengths.