Orionnnn/hub-stats huggingface dataset:
Click-Gaussian: Interactive Segmentation to Any 3D Gaussians
Authors: name: Seokhun Choi, name: Hyeonseop Song, name: Jaechul Kim, name: Taehyeong Kim, name: Hoseok Do
Published: 2024-07-16 14:49:27

Interactive segmentation of 3D Gaussians opens a great opportunity for
real-time manipulation of 3D scenes thanks to the real-time rendering
capability of 3D Gaussian Splatting. However, the current methods suffer from
time-consuming post-processing to deal with noisy segmentation output. Also,
they struggle to provide detailed segmentation, which is important for
fine-grained manipulation of 3D scenes. In this study, we propose
Click-Gaussian, which learns distinguishable feature fields of two-level
granularity, facilitating segmentation without time-consuming post-processing.
We delve into challenges stemming from inconsistently learned feature fields
resulting from 2D segmentation obtained independently from a 3D scene. 3D
segmentation accuracy deteriorates when 2D segmentation results across the
views, primary cues for 3D segmentation, are in conflict. To overcome these
issues, we propose Global Feature-guided Learning (GFL). GFL constructs the
clusters of global feature candidates from noisy 2D segments across the views,
which smooths out noises when training the features of 3D Gaussians. Our method
runs in 10 ms per click, 15 to 130 times as fast as the previous methods, while
also significantly improving segmentation accuracy. Our project page is
available at https://seokhunchoi.github.io/Click-Gaussian
Qwen2-Audio Technical Report
Authors: name: Yunfei Chu, name: Jin Xu, name: Qian Yang, name: Haojie Wei, name: Xipin Wei, name: Zhifang Guo, name: Yichong Leng, name: Yuanjun Lv, name: Jinzheng He, name: Junyang Lin, name: Chang Zhou, name: Jingren Zhou
Published: 2024-07-15 14:38:09

We introduce the latest progress of Qwen-Audio, a large-scale audio-language
model called Qwen2-Audio, which is capable of accepting various audio signal
inputs and performing audio analysis or direct textual responses with regard to
speech instructions. In contrast to complex hierarchical tags, we have
simplified the pre-training process by utilizing natural language prompts for
different data and tasks, and have further expanded the data volume. We have
boosted the instruction-following capability of Qwen2-Audio and implemented two
distinct audio interaction modes for voice chat and audio analysis. In the
voice chat mode, users can freely engage in voice interactions with Qwen2-Audio
without text input. In the audio analysis mode, users could provide audio and
text instructions for analysis during the interaction. Note that we do not use
any system prompts to switch between voice chat and audio analysis modes.
Qwen2-Audio is capable of intelligently comprehending the content within audio
and following voice commands to respond appropriately. For instance, in an
audio segment that simultaneously contains sounds, multi-speaker conversations,
and a voice command, Qwen2-Audio can directly understand the command and
provide an interpretation and response to the audio. Additionally, DPO has
optimized the model's performance in terms of factuality and adherence to
desired behavior. According to the evaluation results from AIR-Bench,
Qwen2-Audio outperformed previous SOTAs, such as Gemini-1.5-pro, in tests
focused on audio-centric instruction-following capabilities. Qwen2-Audio is
open-sourced with the aim of fostering the advancement of the multi-modal
language community.
Scaling Diffusion Transformers to 16 Billion Parameters
Authors: name: Zhengcong Fei, name: Mingyuan Fan, name: Changqian Yu, name: Debang Li, name: Junshi Huang
Published: 2024-07-16 11:55:23

In this paper, we present DiT-MoE, a sparse version of the diffusion
Transformer, that is scalable and competitive with dense networks while
exhibiting highly optimized inference. The DiT-MoE includes two simple designs:
shared expert routing and expert-level balance loss, thereby capturing common
knowledge and reducing redundancy among the different routed experts. When
applied to conditional image generation, a deep analysis of experts
specialization gains some interesting observations: (i) Expert selection shows
preference with spatial position and denoising time step, while insensitive
with different class-conditional information; (ii) As the MoE layers go deeper,
the selection of experts gradually shifts from specific spacial position to
dispersion and balance. (iii) Expert specialization tends to be more
concentrated at the early time step and then gradually uniform after half. We
attribute it to the diffusion process that first models the low-frequency
spatial information and then high-frequency complex information. Based on the
above guidance, a series of DiT-MoE experimentally achieves performance on par
with dense networks yet requires much less computational load during inference.
More encouragingly, we demonstrate the potential of DiT-MoE with synthesized
image data, scaling diffusion model at a 16.5B parameter that attains a new
SoTA FID-50K score of 1.80 in 512times512 resolution settings. The project
page: https://github.com/feizc/DiT-MoE.
Animate3D: Animating Any 3D Model with Multi-view Video Diffusion
Authors: name: Yanqin Jiang, name: Chaohui Yu, name: Chenjie Cao, name: Fan Wang, name: Weiming Hu, name: Jin Gao
Published: 2024-07-16 05:35:57

Recent advances in 4D generation mainly focus on generating 4D content by
distilling pre-trained text or single-view image-conditioned models. It is
inconvenient for them to take advantage of various off-the-shelf 3D assets with
multi-view attributes, and their results suffer from spatiotemporal
inconsistency owing to the inherent ambiguity in the supervision signals. In
this work, we present Animate3D, a novel framework for animating any static 3D
model. The core idea is two-fold: 1) We propose a novel multi-view video
diffusion model (MV-VDM) conditioned on multi-view renderings of the static 3D
object, which is trained on our presented large-scale multi-view video dataset
(MV-Video). 2) Based on MV-VDM, we introduce a framework combining
reconstruction and 4D Score Distillation Sampling (4D-SDS) to leverage the
multi-view video diffusion priors for animating 3D objects. Specifically, for
MV-VDM, we design a new spatiotemporal attention module to enhance spatial and
temporal consistency by integrating 3D and video diffusion models.
Additionally, we leverage the static 3D model's multi-view renderings as
conditions to preserve its identity. For animating 3D models, an effective
two-stage pipeline is proposed: we first reconstruct motions directly from
generated multi-view videos, followed by the introduced 4D-SDS to refine both
appearance and motion. Qualitative and quantitative experiments demonstrate
that Animate3D significantly outperforms previous approaches. Data, code, and
models will be open-released.
DreamCatalyst: Fast and High-Quality 3D Editing via Controlling
  Editability and Identity Preservation
Authors: name: Jiwook Kim, name: Seonho Lee, name: Jaeyo Shin, name: Jiho Choi, name: Hyunjung Shim
Published: 2024-07-16 05:26:14

Score distillation sampling (SDS) has emerged as an effective framework in
text-driven 3D editing tasks due to its inherent 3D consistency. However,
existing SDS-based 3D editing methods suffer from extensive training time and
lead to low-quality results, primarily because these methods deviate from the
sampling dynamics of diffusion models. In this paper, we propose DreamCatalyst,
a novel framework that interprets SDS-based editing as a diffusion reverse
process. Our objective function considers the sampling dynamics, thereby making
the optimization process of DreamCatalyst an approximation of the diffusion
reverse process in editing tasks. DreamCatalyst aims to reduce training time
and improve editing quality. DreamCatalyst presents two modes: (1) a faster
mode, which edits the NeRF scene in only about 25 minutes, and (2) a
high-quality mode, which produces superior results in less than 70 minutes.
Specifically, our high-quality mode outperforms current state-of-the-art NeRF
editing methods both in terms of speed and quality. See more extensive results
on our project page: https://dream-catalyst.github.io.
Grasping Diverse Objects with Simulated Humanoids
Authors: name: Zhengyi Luo, name: Jinkun Cao, name: Sammy Christen, name: Alexander Winkler, name: Kris Kitani, name: Weipeng Xu
Published: 2024-07-16 05:05:02

We present a method for controlling a simulated humanoid to grasp an object
and move it to follow an object trajectory. Due to the challenges in
controlling a humanoid with dexterous hands, prior methods often use a
disembodied hand and only consider vertical lifts or short trajectories. This
limited scope hampers their applicability for object manipulation required for
animation and simulation. To close this gap, we learn a controller that can
pick up a large number (>1200) of objects and carry them to follow randomly
generated trajectories. Our key insight is to leverage a humanoid motion
representation that provides human-like motor skills and significantly speeds
up training. Using only simplistic reward, state, and object representations,
our method shows favorable scalability on diverse object and trajectories. For
training, we do not need dataset of paired full-body motion and object
trajectories. At test time, we only require the object mesh and desired
trajectories for grasping and transporting. To demonstrate the capabilities of
our method, we show state-of-the-art success rates in following object
trajectories and generalizing to unseen objects. Code and models will be
released.
From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from
  Low-Rank Gradients
Authors: name: Ajay Jaiswal, name: Lu Yin, name: Zhenyu Zhang, name: Shiwei Liu, name: Jiawei Zhao, name: Yuandong Tian, name: Zhangyang Wang
Published: 2024-07-15 21:05:20

Modern Large Language Models (LLMs) are composed of matrices with billions of
elements, making their storage and processing quite demanding in terms of
computational resources and memory usage. Being significantly large, such
matrices can often be expressed in low-rank format with potential to relax
resource requirements. Unlike prior works which focus on developing novel
matrix decomposition algorithms, in this work we first study the emergence of
low-rank structures across matrices within different layers of LLMs and
establish a consequential relationship between the gradient dynamics and
emerging low-rank expressiveness of matrices. Our findings reveal that
different layers exhibit varying levels of converged low-rank structure,
necessitating a non-uniform rank reduction across them to minimize performance
drop due to compression. In view of that, we present Weight Low-Rank Projection
(WeLore) that unifies weight compression and memory-efficient fine-tuning as
ONE, in a data-agnostic and one-shot way. WeLore capitalizes the heavy-tail
distribution of singular values to identify a suitable rank reduction ratio for
matrices within LLMs. Going beyond only as a compression technique, WeLore
categorizes weight matrices into Low-rank Components (LRCs) and Non-Low-rank
Components (N-LRCs) based on their ability to express themselves as low-rank.
Our gradient perspective and extensive experiments illustrate that LRCs tend to
have better finetuning capabilities and can closely mimic (sometimes
outperform) the training loss trajectory and performance of full-finetuning
with notable memory and compute footprint reduction. For example, finetuning a
50\% compressed LLaMa-2 7B model using only a fraction of parameters in LRCs
(WeLore) can outperform its full finetuning with ~3x better throughput and
~0.6x GPU requirement. Our codes are available at
https://github.com/VITA-Group/welore
YouTube-SL-25: A Large-Scale, Open-Domain Multilingual Sign Language
  Parallel Corpus
Authors: name: Garrett Tanzer, name: Biao Zhang
Published: 2024-07-15 18:08:34

Even for better-studied sign languages like American Sign Language (ASL),
data is the bottleneck for machine learning research. The situation is worse
yet for the many other sign languages used by Deaf/Hard of Hearing communities
around the world. In this paper, we present YouTube-SL-25, a large-scale,
open-domain multilingual corpus of sign language videos with seemingly
well-aligned captions drawn from YouTube. With >3000 hours of videos across >25
sign languages, YouTube-SL-25 is a) >3x the size of YouTube-ASL, b) the largest
parallel sign language dataset to date, and c) the first or largest parallel
dataset for many of its component languages. We provide baselines for
sign-to-text tasks using a unified multilingual multitask model based on T5 and
report scores on benchmarks across 4 sign languages. The results demonstrate
that multilingual transfer benefits both higher- and lower-resource sign
languages within YouTube-SL-25.
Foundational Autoraters: Taming Large Language Models for Better
  Automatic Evaluation
Authors: name: Tu Vu, name: Kalpesh Krishna, name: Salaheddin Alzubi, name: Chris Tar, name: Manaal Faruqui, name: Yun-Hsuan Sung
Published: 2024-07-15 15:33:45

As large language models (LLMs) advance, it becomes more challenging to
reliably evaluate their output due to the high costs of human evaluation. To
make progress towards better LLM autoraters, we introduce FLAMe, a family of
Foundational Large Autorater Models. FLAMe is trained on our large and diverse
collection of 100+ quality assessment tasks comprising 5M+ human judgments,
curated and standardized using publicly released human evaluations from
previous research. FLAMe significantly improves generalization to a wide
variety of held-out tasks, outperforming LLMs trained on proprietary data like
GPT-4 and Claude-3 on many tasks. We show that FLAMe can also serve as a
powerful starting point for further downstream fine-tuning, using reward
modeling evaluation as a case study (FLAMe-RM). Notably, on RewardBench, our
FLAMe-RM-24B model (with an accuracy of 87.8%) is the top-performing generative
model trained exclusively on permissively licensed data, outperforming both
GPT-4-0125 (85.9%) and GPT-4o (84.7%). Additionally, we explore a more
computationally efficient approach using a novel tail-patch fine-tuning
strategy to optimize our FLAMe multitask mixture for reward modeling evaluation
(FLAMe-Opt-RM), offering competitive RewardBench performance while requiring
approximately 25x less training datapoints. Overall, our FLAMe variants
outperform all popular proprietary LLM-as-a-Judge models we consider across 8
out of 12 autorater evaluation benchmarks, encompassing 53 quality assessment
tasks, including RewardBench and LLM-AggreFact. Finally, our analysis reveals
that FLAMe is significantly less biased than these LLM-as-a-Judge models on the
CoBBLEr autorater bias benchmark, while effectively identifying high-quality
responses for code generation.
Spider2-V: How Far Are Multimodal Agents From Automating Data Science
  and Engineering Workflows?
Authors: name: Ruisheng Cao, name: Fangyu Lei, name: Haoyuan Wu, name: Jixuan Chen, name: Yeqiao Fu, name: Hongcheng Gao, name: Xinzhuang Xiong, name: Hanchong Zhang, name: Yuchen Mao, name: Wenjing Hu, name: Tianbao Xie, name: Hongshen Xu, name: Danyang Zhang, name: Sida Wang, name: Ruoxi Sun, name: Pengcheng Yin, name: Caiming Xiong, name: Ansong Ni, name: Qian Liu, name: Victor Zhong, name: Lu Chen, name: Kai Yu, name: Tao Yu
Published: 2024-07-15 17:54:37

Data science and engineering workflows often span multiple stages, from
warehousing to orchestration, using tools like BigQuery, dbt, and Airbyte. As
vision language models (VLMs) advance in multimodal understanding and code
generation, VLM-based agents could potentially automate these workflows by
generating SQL queries, Python code, and GUI operations. This automation can
improve the productivity of experts while democratizing access to large-scale
data analysis. In this paper, we introduce Spider2-V, the first multimodal
agent benchmark focusing on professional data science and engineering
workflows, featuring 494 real-world tasks in authentic computer environments
and incorporating 20 enterprise-level professional applications. These tasks,
derived from real-world use cases, evaluate the ability of a multimodal agent
to perform data-related tasks by writing code and managing the GUI in
enterprise data software systems. To balance realistic simulation with
evaluation simplicity, we devote significant effort to developing automatic
configurations for task setup and carefully crafting evaluation metrics for
each task. Furthermore, we supplement multimodal agents with comprehensive
documents of these enterprise data software systems. Our empirical evaluation
reveals that existing state-of-the-art LLM/VLM-based agents do not reliably
automate full data workflows (14.0% success). Even with step-by-step guidance,
these agents still underperform in tasks that require fine-grained,
knowledge-intensive GUI actions (16.2%) and involve remote cloud-hosted
workspaces (10.6%). We hope that Spider2-V paves the way for autonomous
multimodal agents to transform the automation of data science and engineering
workflow. Our code and data are available at https://spider2-v.github.io.
LLM Circuit Analyses Are Consistent Across Training and Scale
Authors: name: Curt Tigges, name: Michael Hanna, name: Qinan Yu, name: Stella Biderman
Published: 2024-07-15 15:38:51

Most currently deployed large language models (LLMs) undergo continuous
training or additional finetuning. By contrast, most research into LLMs'
internal mechanisms focuses on models at one snapshot in time (the end of
pre-training), raising the question of whether their results generalize to
real-world settings. Existing studies of mechanisms over time focus on
encoder-only or toy models, which differ significantly from most deployed
models. In this study, we track how model mechanisms, operationalized as
circuits, emerge and evolve across 300 billion tokens of training in
decoder-only LLMs, in models ranging from 70 million to 2.8 billion parameters.
We find that task abilities and the functional components that support them
emerge consistently at similar token counts across scale. Moreover, although
such components may be implemented by different attention heads over time, the
overarching algorithm that they implement remains. Surprisingly, both these
algorithms and the types of components involved therein can replicate across
model scale. These results suggest that circuit analyses conducted on small
models at the end of pre-training can provide insights that still apply after
additional pre-training and over model scale.
Q-Sparse: All Large Language Models can be Fully Sparsely-Activated
Authors: name: Hongyu Wang, name: Shuming Ma, name: Ruiping Wang, name: Furu Wei
Published: 2024-07-15 17:59:29

We introduce, Q-Sparse, a simple yet effective approach to training
sparsely-activated large language models (LLMs). Q-Sparse enables full sparsity
of activations in LLMs which can bring significant efficiency gains in
inference. This is achieved by applying top-K sparsification to the activations
and the straight-through-estimator to the training. The key results from this
work are, (1) Q-Sparse can achieve results comparable to those of baseline LLMs
while being much more efficient at inference time; (2) We present an
inference-optimal scaling law for sparsely-activated LLMs; (3) Q-Sparse is
effective in different settings, including training-from-scratch,
continue-training of off-the-shelf LLMs, and finetuning; (4) Q-Sparse works for
both full-precision and 1-bit LLMs (e.g., BitNet b1.58). Particularly, the
synergy of BitNet b1.58 and Q-Sparse (can be equipped with MoE) provides the
cornerstone and a clear path to revolutionize the efficiency, including cost
and energy consumption, of future LLMs.
DataDream: Few-shot Guided Dataset Generation
Authors: name: Jae Myung Kim, name: Jessica Bader, name: Stephan Alaniz, name: Cordelia Schmid, name: Zeynep Akata
Published: 2024-07-15 17:10:31

While text-to-image diffusion models have been shown to achieve
state-of-the-art results in image synthesis, they have yet to prove their
effectiveness in downstream applications. Previous work has proposed to
generate data for image classifier training given limited real data access.
However, these methods struggle to generate in-distribution images or depict
fine-grained features, thereby hindering the generalization of classification
models trained on synthetic datasets. We propose DataDream, a framework for
synthesizing classification datasets that more faithfully represents the real
data distribution when guided by few-shot examples of the target classes.
DataDream fine-tunes LoRA weights for the image generation model on the few
real images before generating the training data using the adapted model. We
then fine-tune LoRA weights for CLIP using the synthetic data to improve
downstream image classification over previous approaches on a large variety of
datasets. We demonstrate the efficacy of DataDream through extensive
experiments, surpassing state-of-the-art classification accuracy with few-shot
data across 7 out of 10 datasets, while being competitive on the other 3.
Additionally, we provide insights into the impact of various factors, such as
the number of real-shot and generated images as well as the fine-tuning compute
on model performance. The code is available at
https://github.com/ExplainableML/DataDream.
MMM: Multilingual Mutual Reinforcement Effect Mix Datasets & Test with
  Open-domain Information Extraction Large Language Models
Authors: name: Chengguang Gan, name: Qingyu Yin, name: Xinyang He, name: Hanjun Wei, name: Yunhao Liang, name: Younghun Lim, name: Shijian Wang, name: Hexiang Huang, name: Qinghao Zhang, name: Shiwen Ni, name: Tatsunori Mori
Published: 2024-07-15 17:50:43

The Mutual Reinforcement Effect (MRE) represents a promising avenue in
information extraction and multitasking research. Nevertheless, its
applicability has been constrained due to the exclusive availability of MRE mix
datasets in Japanese, thereby limiting comprehensive exploration by the global
research community. To address this limitation, we introduce a Multilingual MRE
mix dataset (MMM) that encompasses 21 sub-datasets in English, Japanese, and
Chinese. In this paper, we also propose a method for dataset translation
assisted by Large Language Models (LLMs), which significantly reduces the
manual annotation time required for dataset construction by leveraging LLMs to
translate the original Japanese datasets. Additionally, we have enriched the
dataset by incorporating open-domain Named Entity Recognition (NER) and
sentence classification tasks. Utilizing this expanded dataset, we developed a
unified input-output framework to train an Open-domain Information Extraction
Large Language Model (OIELLM). The OIELLM model demonstrates the capability to
effectively process novel MMM datasets, exhibiting significant improvements in
performance.