Orionnnn/hub-stats huggingface dataset:
WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio
  Language Modeling
Authors: name: Shengpeng Ji, name: Ziyue Jiang, name: Xize Cheng, name: Yifu Chen, name: Minghui Fang, name: Jialong Zuo, name: Qian Yang, name: Ruiqi Li, name: Ziang Zhang, name: Xiaoda Yang, name: Rongjie Huang, name: Yidi Jiang, name: Qian Chen, name: Siqi Zheng, name: Wen Wang, name: Zhou Zhao
Published: 2024-08-29 13:43:36

Language models have been effectively applied to modeling natural signals,
such as images, video, speech, and audio. A crucial component of these models
is the codec tokenizer, which compresses high-dimensional natural signals into
lower-dimensional discrete tokens. In this paper, we introduce WavTokenizer,
which offers several advantages over previous SOTA acoustic codec models in the
audio domain: 1)extreme compression. By compressing the layers of quantizers
and the temporal dimension of the discrete codec, one-second audio of 24kHz
sampling rate requires only a single quantizer with 40 or 75 tokens. 2)improved
subjective quality. Despite the reduced number of tokens, WavTokenizer achieves
state-of-the-art reconstruction quality with outstanding UTMOS scores and
inherently contains richer semantic information. Specifically, we achieve these
results by designing a broader VQ space, extended contextual windows, and
improved attention networks, as well as introducing a powerful multi-scale
discriminator and an inverse Fourier transform structure. We conducted
extensive reconstruction experiments in the domains of speech, audio, and
music. WavTokenizer exhibited strong performance across various objective and
subjective metrics compared to state-of-the-art models. We also tested semantic
information, VQ utilization, and adaptability to generative models.
Comprehensive ablation studies confirm the necessity of each module in
WavTokenizer. The related code, demos, and pre-trained models are available at
https://github.com/jishengpeng/WavTokenizer.
SAM2Point: Segment Any 3D as Videos in Zero-shot and Promptable Manners
Authors: name: Ziyu Guo, name: Renrui Zhang, name: Xiangyang Zhu, name: Chengzhuo Tong, name: Peng Gao, name: Chunyuan Li, name: Pheng-Ann Heng
Published: 2024-08-29 17:59:45

We introduce SAM2Point, a preliminary exploration adapting Segment Anything
Model 2 (SAM 2) for zero-shot and promptable 3D segmentation. SAM2Point
interprets any 3D data as a series of multi-directional videos, and leverages
SAM 2 for 3D-space segmentation, without further training or 2D-3D projection.
Our framework supports various prompt types, including 3D points, boxes, and
masks, and can generalize across diverse scenarios, such as 3D objects, indoor
scenes, outdoor environments, and raw sparse LiDAR. Demonstrations on multiple
3D datasets, e.g., Objaverse, S3DIS, ScanNet, Semantic3D, and KITTI, highlight
the robust generalization capabilities of SAM2Point. To our best knowledge, we
present the most faithful implementation of SAM in 3D, which may serve as a
starting point for future research in promptable 3D segmentation. Online Demo:
https://huggingface.co/spaces/ZiyuG/SAM2Point . Code:
https://github.com/ZiyuGuo99/SAM2Point .
ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion
  Model
Authors: name: Fangfu Liu, name: Wenqiang Sun, name: Hanyang Wang, name: Yikai Wang, name: Haowen Sun, name: Junliang Ye, name: Jun Zhang, name: Yueqi Duan
Published: 2024-08-29 17:59:40

Advancements in 3D scene reconstruction have transformed 2D images from the
real world into 3D models, producing realistic 3D results from hundreds of
input photos. Despite great success in dense-view reconstruction scenarios,
rendering a detailed scene from insufficient captured views is still an
ill-posed optimization problem, often resulting in artifacts and distortions in
unseen areas. In this paper, we propose ReconX, a novel 3D scene reconstruction
paradigm that reframes the ambiguous reconstruction challenge as a temporal
generation task. The key insight is to unleash the strong generative prior of
large pre-trained video diffusion models for sparse-view reconstruction.
However, 3D view consistency struggles to be accurately preserved in directly
generated video frames from pre-trained models. To address this, given limited
input views, the proposed ReconX first constructs a global point cloud and
encodes it into a contextual space as the 3D structure condition. Guided by the
condition, the video diffusion model then synthesizes video frames that are
both detail-preserved and exhibit a high degree of 3D consistency, ensuring the
coherence of the scene from various perspectives. Finally, we recover the 3D
scene from the generated video through a confidence-aware 3D Gaussian Splatting
optimization scheme. Extensive experiments on various real-world datasets show
the superiority of our ReconX over state-of-the-art methods in terms of quality
and generalizability.
CSGO: Content-Style Composition in Text-to-Image Generation
Authors: name: Peng Xing, name: Haofan Wang, name: Yanpeng Sun, name: Qixun Wang, name: Xu Bai, name: Hao Ai, name: Renyuan Huang, name: Zechao Li
Published: 2024-08-29 17:59:30

The diffusion model has shown exceptional capabilities in controlled image
generation, which has further fueled interest in image style transfer. Existing
works mainly focus on training free-based methods (e.g., image inversion) due
to the scarcity of specific data. In this study, we present a data construction
pipeline for content-style-stylized image triplets that generates and
automatically cleanses stylized data triplets. Based on this pipeline, we
construct a dataset IMAGStyle, the first large-scale style transfer dataset
containing 210k image triplets, available for the community to explore and
research. Equipped with IMAGStyle, we propose CSGO, a style transfer model
based on end-to-end training, which explicitly decouples content and style
features employing independent feature injection. The unified CSGO implements
image-driven style transfer, text-driven stylized synthesis, and text
editing-driven stylized synthesis. Extensive experiments demonstrate the
effectiveness of our approach in enhancing style control capabilities in image
generation. Additional visualization and access to the source code can be
located on the project page: https://csgo-gen.github.io/.
CogVLM2: Visual Language Models for Image and Video Understanding
Authors: name: Wenyi Hong, name: Weihan Wang, name: Ming Ding, name: Wenmeng Yu, name: Qingsong Lv, name: Yan Wang, name: Yean Cheng, name: Shiyu Huang, name: Junhui Ji, name: Zhao Xue, name: Lei Zhao, name: Zhuoyi Yang, name: Xiaotao Gu, name: Xiaohan Zhang, name: Guanyu Feng, name: Da Yin, name: Zihan Wang, name: Ji Qi, name: Xixuan Song, name: Peng Zhang, name: Debing Liu, name: Bin Xu, name: Juanzi Li, name: Yuxiao Dong, name: Jie Tang
Published: 2024-08-29 12:59:12

Beginning with VisualGLM and CogVLM, we are continuously exploring VLMs in
pursuit of enhanced vision-language fusion, efficient higher-resolution
architecture, and broader modalities and applications. Here we propose the
CogVLM2 family, a new generation of visual language models for image and video
understanding including CogVLM2, CogVLM2-Video and GLM-4V. As an image
understanding model, CogVLM2 inherits the visual expert architecture with
improved training recipes in both pre-training and post-training stages,
supporting input resolution up to 1344 times 1344 pixels. As a video
understanding model, CogVLM2-Video integrates multi-frame input with timestamps
and proposes automated temporal grounding data construction. Notably, CogVLM2
family has achieved state-of-the-art results on benchmarks like MMBench,
MM-Vet, TextVQA, MVBench and VCGBench. All models are open-sourced in
https://github.com/THUDM/CogVLM2 and https://github.com/THUDM/GLM-4,
contributing to the advancement of the field.
Law of Vision Representation in MLLMs
Authors: name: Shijia Yang, name: Bohan Zhai, name: Quanzeng You, name: Jianbo Yuan, name: Hongxia Yang, name: Chenfeng Xu
Published: 2024-08-29 08:56:48

We present the "Law of Vision Representation" in multimodal large language
models (MLLMs). It reveals a strong correlation between the combination of
cross-modal alignment, correspondence in vision representation, and MLLM
performance. We quantify the two factors using the cross-modal Alignment and
Correspondence score (AC score). Through extensive experiments involving
thirteen different vision representation settings and evaluations across eight
benchmarks, we find that the AC score is linearly correlated to model
performance. By leveraging this relationship, we are able to identify and train
the optimal vision representation only, which does not require finetuning the
language model every time, resulting in a 99.7% reduction in computational
cost.
3D Reconstruction with Spatial Memory
Authors: name: Hengyi Wang, name: Lourdes Agapito
Published: 2024-08-28 18:01:00

We present Spann3R, a novel approach for dense 3D reconstruction from ordered
or unordered image collections. Built on the DUSt3R paradigm, Spann3R uses a
transformer-based architecture to directly regress pointmaps from images
without any prior knowledge of the scene or camera parameters. Unlike DUSt3R,
which predicts per image-pair pointmaps each expressed in its local coordinate
frame, Spann3R can predict per-image pointmaps expressed in a global coordinate
system, thus eliminating the need for optimization-based global alignment. The
key idea of Spann3R is to manage an external spatial memory that learns to keep
track of all previous relevant 3D information. Spann3R then queries this
spatial memory to predict the 3D structure of the next frame in a global
coordinate system. Taking advantage of DUSt3R's pre-trained weights, and
further fine-tuning on a subset of datasets, Spann3R shows competitive
performance and generalization ability on various unseen datasets and can
process ordered image collections in real time. Project page:
https://hengyiwang.github.io/projects/spanner
Scaling Up Diffusion and Flow-based XGBoost Models
Authors: name: Jesse C. Cresswell, name: Taewoo Kim
Published: 2024-08-28 18:00:00

Novel machine learning methods for tabular data generation are often
developed on small datasets which do not match the scale required for
scientific applications. We investigate a recent proposal to use XGBoost as the
function approximator in diffusion and flow-matching models on tabular data,
which proved to be extremely memory intensive, even on tiny datasets. In this
work, we conduct a critical analysis of the existing implementation from an
engineering perspective, and show that these limitations are not fundamental to
the method; with better implementation it can be scaled to datasets 370x larger
than previously used. Our efficient implementation also unlocks scaling models
to much larger sizes which we show directly leads to improved performance on
benchmark tasks. We also propose algorithmic improvements that can further
benefit resource usage and model performance, including multi-output trees
which are well-suited to generative modeling. Finally, we present results on
large-scale scientific datasets derived from experimental particle physics as
part of the Fast Calorimeter Simulation Challenge. Code is available at
https://github.com/layer6ai-labs/calo-forest.
Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
Authors: name: Lean Wang, name: Huazuo Gao, name: Chenggang Zhao, name: Xu Sun, name: Damai Dai
Published: 2024-08-28 09:31:09

For Mixture-of-Experts (MoE) models, an unbalanced expert load will lead to
routing collapse or increased computational overhead. Existing methods commonly
employ an auxiliary loss to encourage load balance, but a large auxiliary loss
will introduce non-negligible interference gradients into training and thus
impair the model performance. In order to control load balance while not
producing undesired gradients during training, we propose Loss-Free Balancing,
featured by an auxiliary-loss-free load balancing strategy. To be specific,
before the top-K routing decision, Loss-Free Balancing will first apply an
expert-wise bias to the routing scores of each expert. By dynamically updating
the bias of each expert according to its recent load, Loss-Free Balancing can
consistently maintain a balanced distribution of expert load. In addition,
since Loss-Free Balancing does not produce any interference gradients, it also
elevates the upper bound of model performance gained from MoE training. We
validate the performance of Loss-Free Balancing on MoE models with up to 3B
parameters trained on up to 200B tokens. Experimental results show that
Loss-Free Balancing achieves both better performance and better load balance
compared with traditional auxiliary-loss-controlled load balancing strategies.
Efficient LLM Scheduling by Learning to Rank
Authors: name: Yichao Fu, name: Siqi Zhu, name: Runlong Su, name: Aurick Qiao, name: Ion Stoica, name: Hao Zhang
Published: 2024-08-28 13:35:54

In Large Language Model (LLM) inference, the output length of an LLM request
is typically regarded as not known a priori. Consequently, most LLM serving
systems employ a simple First-come-first-serve (FCFS) scheduling strategy,
leading to Head-Of-Line (HOL) blocking and reduced throughput and service
quality. In this paper, we reexamine this assumption -- we show that, although
predicting the exact generation length of each request is infeasible, it is
possible to predict the relative ranks of output lengths in a batch of
requests, using learning to rank. The ranking information offers valuable
guidance for scheduling requests. Building on this insight, we develop a novel
scheduler for LLM inference and serving that can approximate the
shortest-job-first (SJF) schedule better than existing approaches. We integrate
this scheduler with the state-of-the-art LLM serving system and show
significant performance improvement in several important applications: 2.8x
lower latency in chatbot serving and 6.5x higher throughput in synthetic data
generation. Our code is available at https://github.com/hao-ai-lab/vllm-ltr.git
Distribution Backtracking Builds A Faster Convergence Trajectory for
  One-step Diffusion Distillation
Authors: name: Shengyuan Zhang, name: Ling Yang, name: Zejian Li, name: An Zhao, name: Chenye Meng, name: Changyuan Yang, name: Guang Yang, name: Zhiyuan Yang, name: Lingyun Sun
Published: 2024-08-28 17:58:17

Accelerating the sampling speed of diffusion models remains a significant
challenge. Recent score distillation methods distill a heavy teacher model into
an one-step student generator, which is optimized by calculating the difference
between the two score functions on the samples generated by the student model.
However, there is a score mismatch issue in the early stage of the distillation
process, because existing methods mainly focus on using the endpoint of
pre-trained diffusion models as teacher models, overlooking the importance of
the convergence trajectory between the student generator and the teacher model.
To address this issue, we extend the score distillation process by introducing
the entire convergence trajectory of teacher models and propose Distribution
Backtracking Distillation (DisBack) for distilling student generators. DisBask
is composed of two stages: Degradation Recording and Distribution Backtracking.
Degradation Recording is designed to obtain the convergence trajectory of
teacher models, which records the degradation path from the trained teacher
model to the untrained initial student generator. The degradation path
implicitly represents the intermediate distributions of teacher models. Then
Distribution Backtracking trains a student generator to backtrack the
intermediate distributions for approximating the convergence trajectory of
teacher models. Extensive experiments show that DisBack achieves faster and
better convergence than the existing distillation method and accomplishes
comparable generation performance. Notably, DisBack is easy to implement and
can be generalized to existing distillation methods to boost performance. Our
code is publicly available on https://github.com/SYZhang0805/DisBack.
Knowledge Navigator: LLM-guided Browsing Framework for Exploratory
  Search in Scientific Literature
Authors: name: Uri Katz, name: Mosh Levy, name: Yoav Goldberg
Published: 2024-08-28 14:48:37

The exponential growth of scientific literature necessitates advanced tools
for effective knowledge exploration. We present Knowledge Navigator, a system
designed to enhance exploratory search abilities by organizing and structuring
the retrieved documents from broad topical queries into a navigable, two-level
hierarchy of named and descriptive scientific topics and subtopics. This
structured organization provides an overall view of the research themes in a
domain, while also enabling iterative search and deeper knowledge discovery
within specific subtopics by allowing users to refine their focus and retrieve
additional relevant documents. Knowledge Navigator combines LLM capabilities
with cluster-based methods to enable an effective browsing method. We
demonstrate our approach's effectiveness through automatic and manual
evaluations on two novel benchmarks, CLUSTREC-COVID and SCITOC. Our code,
prompts, and benchmarks are made publicly available.
BaichuanSEED: Sharing the Potential of ExtensivE Data Collection and
  Deduplication by Introducing a Competitive Large Language Model Baseline
Authors: name: Guosheng Dong, name: Da Pan, name: Yiding Sun, name: Shusen Zhang, name: Zheng Liang, name: Xin Wu, name: Yanjun Shen, name: Fan Yang, name: Haoze Sun, name: Tianpeng Li, name: Mingan Lin, name: Jianhua Xu, name: Yufan Zhang, name: Xiaonan Nie, name: Lei Su, name: Bingning Wang, name: Wentao Zhang, name: Jiaxin Mao, name: Zenan Zhou, name: Weipeng Chen
Published: 2024-08-27 14:08:23

The general capabilities of Large Language Models (LLM) highly rely on the
composition and selection on extensive pretraining datasets, treated as
commercial secrets by several institutions. To mitigate this issue, we
open-source the details of a universally applicable data processing pipeline
and validate its effectiveness and potential by introducing a competitive LLM
baseline. Specifically, the data processing pipeline consists of broad
collection to scale up and reweighting to improve quality. We then pretrain a
7B model BaichuanSEED with 3T tokens processed by our pipeline without any
deliberate downstream task-related optimization, followed by an easy but
effective supervised fine-tuning stage. BaichuanSEED demonstrates consistency
and predictability throughout training and achieves comparable performance on
comprehensive benchmarks with several commercial advanced large language
models, such as Qwen1.5 and Llama3. We also conduct several heuristic
experiments to discuss the potential for further optimization of downstream
tasks, such as mathematics and coding.
In-Context Imitation Learning via Next-Token Prediction
Authors: name: Letian Fu, name: Huang Huang, name: Gaurav Datta, name: Lawrence Yunliang Chen, name: William Chung-Ho Panitch, name: Fangchen Liu, name: Hui Li, name: Ken Goldberg
Published: 2024-08-28 17:50:19

We explore how to enhance next-token prediction models to perform in-context
imitation learning on a real robot, where the robot executes new tasks by
interpreting contextual information provided during the input phase, without
updating its underlying policy parameters. We propose In-Context Robot
Transformer (ICRT), a causal transformer that performs autoregressive
prediction on sensorimotor trajectories without relying on any linguistic data
or reward function. This formulation enables flexible and training-free
execution of new tasks at test time, achieved by prompting the model with
sensorimotor trajectories of the new task composing of image observations,
actions and states tuples, collected through human teleoperation. Experiments
with a Franka Emika robot demonstrate that the ICRT can adapt to new tasks
specified by prompts, even in environment configurations that differ from both
the prompt and the training data. In a multitask environment setup, ICRT
significantly outperforms current state-of-the-art next-token prediction models
in robotics on generalizing to unseen tasks. Code, checkpoints and data are
available on https://icrt.dev/
ReMamba: Equip Mamba with Effective Long-Sequence Modeling
Authors: name: Danlong Yuan, name: Jiahao Liu, name: Bei Li, name: Huishuai Zhang, name: Jingang Wang, name: Xunliang Cai, name: Dongyan Zhao
Published: 2024-08-28 02:47:27

While the Mamba architecture demonstrates superior inference efficiency and
competitive performance on short-context natural language processing (NLP)
tasks, empirical evidence suggests its capacity to comprehend long contexts is
limited compared to transformer-based models. In this study, we investigate the
long-context efficiency issues of the Mamba models and propose ReMamba, which
enhances Mamba's ability to comprehend long contexts. ReMamba incorporates
selective compression and adaptation techniques within a two-stage re-forward
process, incurring minimal additional inference costs overhead. Experimental
results on the LongBench and L-Eval benchmarks demonstrate ReMamba's efficacy,
improving over the baselines by 3.2 and 1.6 points, respectively, and attaining
performance almost on par with same-size transformer models.
Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of
  Encoders
Authors: name: Min Shi, name: Fuxiao Liu, name: Shihao Wang, name: Shijia Liao, name: Subhashree Radhakrishnan, name: De-An Huang, name: Hongxu Yin, name: Karan Sapra, name: Yaser Yacoob, name: Humphrey Shi, name: Bryan Catanzaro, name: Andrew Tao, name: Jan Kautz, name: Zhiding Yu, name: Guilin Liu
Published: 2024-08-28 17:59:31

The ability to accurately interpret complex visual information is a crucial
topic of multimodal large language models (MLLMs). Recent work indicates that
enhanced visual perception significantly reduces hallucinations and improves
performance on resolution-sensitive tasks, such as optical character
recognition and document analysis. A number of recent MLLMs achieve this goal
using a mixture of vision encoders. Despite their success, there is a lack of
systematic comparisons and detailed ablation studies addressing critical
aspects, such as expert selection and the integration of multiple vision
experts. This study provides an extensive exploration of the design space for
MLLMs using a mixture of vision encoders and resolutions. Our findings reveal
several underlying principles common to various existing strategies, leading to
a streamlined yet effective design approach. We discover that simply
concatenating visual tokens from a set of complementary vision encoders is as
effective as more complex mixing architectures or strategies. We additionally
introduce Pre-Alignment to bridge the gap between vision-focused encoders and
language tokens, enhancing model coherence. The resulting family of MLLMs,
Eagle, surpasses other leading open-source models on major MLLM benchmarks.
Models and code: https://github.com/NVlabs/Eagle