Orionnnn/hub-stats huggingface dataset:
Emma-X: An Embodied Multimodal Action Model with Grounded Chain of
  Thought and Look-ahead Spatial Reasoning
Authors: name: Qi Sun, name: Pengfei Hong, name: Tej Deep Pala, name: Vernon Toh, name: U-Xuan Tan, name: Deepanway Ghosal, name: Soujanya Poria
Published: 2024-12-16 16:58:28

Traditional reinforcement learning-based robotic control methods are often
task-specific and fail to generalize across diverse environments or unseen
objects and instructions. Visual Language Models (VLMs) demonstrate strong
scene understanding and planning capabilities but lack the ability to generate
actionable policies tailored to specific robotic embodiments. To address this,
Visual-Language-Action (VLA) models have emerged, yet they face challenges in
long-horizon spatial reasoning and grounded task planning. In this work, we
propose the Embodied Multimodal Action Model with Grounded Chain of Thought and
Look-ahead Spatial Reasoning, Emma-X. Emma-X leverages our constructed
hierarchical embodiment dataset based on BridgeV2, containing 60,000 robot
manipulation trajectories auto-annotated with grounded task reasoning and
spatial guidance. Additionally, we introduce a trajectory segmentation strategy
based on gripper states and motion trajectories, which can help mitigate
hallucination in grounding subtask reasoning generation. Experimental results
demonstrate that Emma-X achieves superior performance over competitive
baselines, particularly in real-world robotic tasks requiring spatial
reasoning.
DynamicScaler: Seamless and Scalable Video Generation for Panoramic
  Scenes
Authors: name: Jinxiu Liu, name: Shaoheng Lin, name: Yinxiao Li, name: Ming-Hsuan Yang
Published: 2024-12-15 07:42:26

The increasing demand for immersive AR/VR applications and spatial
intelligence has heightened the need to generate high-quality scene-level and
360{\deg} panoramic video. However, most video diffusion models are constrained
by limited resolution and aspect ratio, which restricts their applicability to
scene-level dynamic content synthesis. In this work, we propose the
DynamicScaler, addressing these challenges by enabling spatially scalable and
panoramic dynamic scene synthesis that preserves coherence across panoramic
scenes of arbitrary size. Specifically, we introduce a Offset Shifting
Denoiser, facilitating efficient, synchronous, and coherent denoising panoramic
dynamic scenes via a diffusion model with fixed resolution through a seamless
rotating Window, which ensures seamless boundary transitions and consistency
across the entire panoramic space, accommodating varying resolutions and aspect
ratios. Additionally, we employ a Global Motion Guidance mechanism to ensure
both local detail fidelity and global motion continuity. Extensive experiments
demonstrate our method achieves superior content and motion quality in
panoramic scene-level video generation, offering a training-free, efficient,
and scalable solution for immersive dynamic scene creation with constant VRAM
consumption regardless of the output video resolution. Our project page is
available at https://dynamic-scaler.pages.dev/.
The Open Source Advantage in Large Language Models (LLMs)
Authors: name: Jiya Manchanda, name: Laura Boettcher, name: Matheus Westphalen, name: Jasser Jasser
Published: 2024-12-16 17:32:11

Large language models (LLMs) mark a key shift in natural language processing
(NLP), having advanced text generation, translation, and domain-specific
reasoning. Closed-source models like GPT-4, powered by proprietary datasets and
extensive computational resources, lead with state-of-the-art performance
today. However, they face criticism for their "black box" nature and for
limiting accessibility in a manner that hinders reproducibility and equitable
AI development. By contrast, open-source initiatives like LLaMA and BLOOM
prioritize democratization through community-driven development and
computational efficiency. These models have significantly reduced performance
gaps, particularly in linguistic diversity and domain-specific applications,
while providing accessible tools for global researchers and developers.
Notably, both paradigms rely on foundational architectural innovations, such as
the Transformer framework by Vaswani et al. (2017). Closed-source models excel
by scaling effectively, while open-source models adapt to real-world
applications in underrepresented languages and domains. Techniques like
Low-Rank Adaptation (LoRA) and instruction-tuning datasets enable open-source
models to achieve competitive results despite limited resources. To be sure,
the tension between closed-source and open-source approaches underscores a
broader debate on transparency versus proprietary control in AI. Ethical
considerations further highlight this divide. Closed-source systems restrict
external scrutiny, while open-source models promote reproducibility and
collaboration but lack standardized auditing documentation frameworks to
mitigate biases. Hybrid approaches that leverage the strengths of both
paradigms are likely to shape the future of LLM innovation, ensuring
accessibility, competitive technical performance, and ethical deployment.
Reliable, Reproducible, and Really Fast Leaderboards with Evalica
Authors: name: Dmitry Ustalov
Published: 2024-12-15 21:22:46

The rapid advancement of natural language processing (NLP) technologies, such
as instruction-tuned large language models (LLMs), urges the development of
modern evaluation protocols with human and machine feedback. We introduce
Evalica, an open-source toolkit that facilitates the creation of reliable and
reproducible model leaderboards. This paper presents its design, evaluates its
performance, and demonstrates its usability through its Web interface,
command-line interface, and Python API.
SepLLM: Accelerate Large Language Models by Compressing One Segment into
  One Separator
Authors: name: Guoxuan Chen, name: Han Shi, name: Jiawei Li, name: Yihang Gao, name: Xiaozhe Ren, name: Yimeng Chen, name: Xin Jiang, name: Zhenguo Li, name: Weiyang Liu, name: Chao Huang
Published: 2024-12-16 18:58:57

Large Language Models (LLMs) have exhibited exceptional performance across a
spectrum of natural language processing tasks. However, their substantial sizes
pose considerable challenges, particularly in computational demands and
inference speed, due to their quadratic complexity. In this work, we have
identified a key pattern: certain seemingly meaningless special tokens (i.e.,
separators) contribute disproportionately to attention scores compared to
semantically meaningful tokens. This observation suggests that information of
the segments between these separator tokens can be effectively condensed into
the separator tokens themselves without significant information loss. Guided by
this insight, we introduce SepLLM, a plug-and-play framework that accelerates
inference by compressing these segments and eliminating redundant tokens.
Additionally, we implement efficient kernels for training acceleration.
Experimental results across training-free, training-from-scratch, and
post-training settings demonstrate SepLLM's effectiveness. Notably, using the
Llama-3-8B backbone, SepLLM achieves over 50% reduction in KV cache on the
GSM8K-CoT benchmark while maintaining comparable performance. Furthermore, in
streaming settings, SepLLM effectively processes sequences of up to 4 million
tokens or more while maintaining consistent language modeling capabilities.
Whisper-GPT: A Hybrid Representation Audio Large Language Model
Authors: name: Prateek Verma
Published: 2024-12-16 05:03:48

We propose WHISPER-GPT: A generative large language model (LLM) for speech
and music that allows us to work with continuous audio representations and
discrete tokens simultaneously as part of a single architecture. There has been
a huge surge in generative audio, speech, and music models that utilize
discrete audio tokens derived from neural compression algorithms, e.g. ENCODEC.
However, one of the major drawbacks of this approach is handling the context
length. It blows up for high-fidelity generative architecture if one has to
account for all the audio contents at various frequencies for the next token
prediction. By combining continuous audio representation like the spectrogram
and discrete acoustic tokens, we retain the best of both worlds: Have all the
information needed from the audio at a specific time instance in a single
token, yet allow LLM to predict the future token to allow for sampling and
other benefits discrete space provides. We show how our architecture improves
the perplexity and negative log-likelihood scores for the next token prediction
compared to a token-based LLM for speech and music.
TidyBot++: An Open-Source Holonomic Mobile Manipulator for Robot
  Learning
Authors: name: Jimmy Wu, name: William Chong, name: Robert Holmberg, name: Aaditya Prasad, name: Yihuai Gao, name: Oussama Khatib, name: Shuran Song, name: Szymon Rusinkiewicz, name: Jeannette Bohg
Published: 2024-12-11 18:54:22

Exploiting the promise of recent advances in imitation learning for mobile
manipulation will require the collection of large numbers of human-guided
demonstrations. This paper proposes an open-source design for an inexpensive,
robust, and flexible mobile manipulator that can support arbitrary arms,
enabling a wide range of real-world household mobile manipulation tasks.
Crucially, our design uses powered casters to enable the mobile base to be
fully holonomic, able to control all planar degrees of freedom independently
and simultaneously. This feature makes the base more maneuverable and
simplifies many mobile manipulation tasks, eliminating the kinematic
constraints that create complex and time-consuming motions in nonholonomic
bases. We equip our robot with an intuitive mobile phone teleoperation
interface to enable easy data acquisition for imitation learning. In our
experiments, we use this interface to collect data and show that the resulting
learned policies can successfully perform a variety of common household mobile
manipulation tasks.
Just a Simple Transformation is Enough for Data Protection in Vertical
  Federated Learning
Authors: name: Andrei Semenov, name: Philip Zmushko, name: Alexander Pichugin, name: Aleksandr Beznosikov
Published: 2024-12-16 12:02:12

Vertical Federated Learning (VFL) aims to enable collaborative training of
deep learning models while maintaining privacy protection. However, the VFL
procedure still has components that are vulnerable to attacks by malicious
parties. In our work, we consider feature reconstruction attacks, a common risk
targeting input data compromise. We theoretically claim that feature
reconstruction attacks cannot succeed without knowledge of the prior
distribution on data. Consequently, we demonstrate that even simple model
architecture transformations can significantly impact the protection of input
data during VFL. Confirming these findings with experimental results, we show
that MLP-based models are resistant to state-of-the-art feature reconstruction
attacks.
Byte Latent Transformer: Patches Scale Better Than Tokens
Authors: name: Artidoro Pagnoni, name: Ram Pasunuru, name: Pedro Rodriguez, name: John Nguyen, name: Benjamin Muller, name: Margaret Li, name: Chunting Zhou, name: Lili Yu, name: Jason Weston, name: Luke Zettlemoyer, name: Gargi Ghosh, name: Mike Lewis, name: Ari Holtzman, name: Srinivasan Iyer
Published: 2024-12-13 05:33:32

We introduce the Byte Latent Transformer (BLT), a new byte-level LLM
architecture that, for the first time, matches tokenization-based LLM
performance at scale with significant improvements in inference efficiency and
robustness. BLT encodes bytes into dynamically sized patches, which serve as
the primary units of computation. Patches are segmented based on the entropy of
the next byte, allocating more compute and model capacity where increased data
complexity demands it. We present the first FLOP controlled scaling study of
byte-level models up to 8B parameters and 4T training bytes. Our results
demonstrate the feasibility of scaling models trained on raw bytes without a
fixed vocabulary. Both training and inference efficiency improve due to
dynamically selecting long patches when data is predictable, along with
qualitative improvements on reasoning and long tail generalization. Overall,
for fixed inference costs, BLT shows significantly better scaling than
tokenization-based models, by simultaneously growing both patch and model size.
BrushEdit: All-In-One Image Inpainting and Editing
Authors: name: Yaowei Li, name: Yuxuan Bian, name: Xuan Ju, name: Zhaoyang Zhang, name: Ying Shan, name: Qiang Xu
Published: 2024-12-13 17:58:06

Image editing has advanced significantly with the development of diffusion
models using both inversion-based and instruction-based methods. However,
current inversion-based approaches struggle with big modifications (e.g.,
adding or removing objects) due to the structured nature of inversion noise,
which hinders substantial changes. Meanwhile, instruction-based methods often
constrain users to black-box operations, limiting direct interaction for
specifying editing regions and intensity. To address these limitations, we
propose BrushEdit, a novel inpainting-based instruction-guided image editing
paradigm, which leverages multimodal large language models (MLLMs) and image
inpainting models to enable autonomous, user-friendly, and interactive
free-form instruction editing. Specifically, we devise a system enabling
free-form instruction editing by integrating MLLMs and a dual-branch image
inpainting model in an agent-cooperative framework to perform editing category
classification, main object identification, mask acquisition, and editing area
inpainting. Extensive experiments show that our framework effectively combines
MLLMs and inpainting models, achieving superior performance across seven
metrics including mask region preservation and editing effect coherence.
Evaluation Agent: Efficient and Promptable Evaluation Framework for
  Visual Generative Models
Authors: name: Fan Zhang, name: Shulin Tian, name: Ziqi Huang, name: Yu Qiao, name: Ziwei Liu
Published: 2024-12-10 18:52:39

Recent advancements in visual generative models have enabled high-quality
image and video generation, opening diverse applications. However, evaluating
these models often demands sampling hundreds or thousands of images or videos,
making the process computationally expensive, especially for diffusion-based
models with inherently slow sampling. Moreover, existing evaluation methods
rely on rigid pipelines that overlook specific user needs and provide numerical
results without clear explanations. In contrast, humans can quickly form
impressions of a model's capabilities by observing only a few samples. To mimic
this, we propose the Evaluation Agent framework, which employs human-like
strategies for efficient, dynamic, multi-round evaluations using only a few
samples per round, while offering detailed, user-tailored analyses. It offers
four key advantages: 1) efficiency, 2) promptable evaluation tailored to
diverse user needs, 3) explainability beyond single numerical scores, and 4)
scalability across various models and tools. Experiments show that Evaluation
Agent reduces evaluation time to 10% of traditional methods while delivering
comparable results. The Evaluation Agent framework is fully open-sourced to
advance research in visual generative models and their efficient evaluation.
Wonderland: Navigating 3D Scenes from a Single Image
Authors: name: Hanwen Liang, name: Junli Cao, name: Vidit Goel, name: Guocheng Qian, name: Sergei Korolev, name: Demetri Terzopoulos, name: Konstantinos N. Plataniotis, name: Sergey Tulyakov, name: Jian Ren
Published: 2024-12-16 18:58:17

This paper addresses a challenging question: How can we efficiently create
high-quality, wide-scope 3D scenes from a single arbitrary image? Existing
methods face several constraints, such as requiring multi-view data,
time-consuming per-scene optimization, low visual quality in backgrounds, and
distorted reconstructions in unseen areas. We propose a novel pipeline to
overcome these limitations. Specifically, we introduce a large-scale
reconstruction model that uses latents from a video diffusion model to predict
3D Gaussian Splattings for the scenes in a feed-forward manner. The video
diffusion model is designed to create videos precisely following specified
camera trajectories, allowing it to generate compressed video latents that
contain multi-view information while maintaining 3D consistency. We train the
3D reconstruction model to operate on the video latent space with a progressive
training strategy, enabling the efficient generation of high-quality,
wide-scope, and generic 3D scenes. Extensive evaluations across various
datasets demonstrate that our model significantly outperforms existing methods
for single-view 3D scene generation, particularly with out-of-domain images.
For the first time, we demonstrate that a 3D reconstruction model can be
effectively built upon the latent space of a diffusion model to realize
efficient 3D scene generation.
ColorFlow: Retrieval-Augmented Image Sequence Colorization
Authors: name: Junhao Zhuang, name: Xuan Ju, name: Zhaoyang Zhang, name: Yong Liu, name: Shiyi Zhang, name: Chun Yuan, name: Ying Shan
Published: 2024-12-16 14:32:49

Automatic black-and-white image sequence colorization while preserving
character and object identity (ID) is a complex task with significant market
demand, such as in cartoon or comic series colorization. Despite advancements
in visual colorization using large-scale generative models like diffusion
models, challenges with controllability and identity consistency persist,
making current solutions unsuitable for industrial application.To address this,
we propose ColorFlow, a three-stage diffusion-based framework tailored for
image sequence colorization in industrial applications. Unlike existing methods
that require per-ID finetuning or explicit ID embedding extraction, we propose
a novel robust and generalizable Retrieval Augmented Colorization pipeline for
colorizing images with relevant color references. Our pipeline also features a
dual-branch design: one branch for color identity extraction and the other for
colorization, leveraging the strengths of diffusion models. We utilize the
self-attention mechanism in diffusion models for strong in-context learning and
color identity matching. To evaluate our model, we introduce ColorFlow-Bench, a
comprehensive benchmark for reference-based colorization. Results show that
ColorFlow outperforms existing models across multiple metrics, setting a new
standard in sequential image colorization and potentially benefiting the art
industry. We release our codes and models on our project page:
https://zhuang2002.github.io/ColorFlow/.
Causal Diffusion Transformers for Generative Modeling
Authors: name: Chaorui Deng, name: Deyao Zh, name: Kunchang Li, name: Shi Guan, name: Haoqi Fan
Published: 2024-12-16 18:59:29

We introduce Causal Diffusion as the autoregressive (AR) counterpart of
Diffusion models. It is a next-token(s) forecasting framework that is friendly
to both discrete and continuous modalities and compatible with existing
next-token prediction models like LLaMA and GPT. While recent works attempt to
combine diffusion with AR models, we show that introducing sequential
factorization to a diffusion model can substantially improve its performance
and enables a smooth transition between AR and diffusion generation modes.
Hence, we propose CausalFusion - a decoder-only transformer that
dual-factorizes data across sequential tokens and diffusion noise levels,
leading to state-of-the-art results on the ImageNet generation benchmark while
also enjoying the AR advantage of generating an arbitrary number of tokens for
in-context reasoning. We further demonstrate CausalFusion's multimodal
capabilities through a joint image generation and captioning model, and
showcase CausalFusion's ability for zero-shot in-context image manipulations.
We hope that this work could provide the community with a fresh perspective on
training multimodal models over discrete and continuous data.
Wonderful Matrices: Combining for a More Efficient and Effective
  Foundation Model Architecture
Authors: name: Jingze Shi, name: Bingheng Wu
Published: 2024-12-16 14:56:28

In order to make the foundation model more efficient and effective, our idea
is combining sequence transformation and state transformation. First, we prove
the availability of rotary position embedding in the state space duality
algorithm, which reduces the perplexity of the hybrid quadratic causal
self-attention and state space duality by more than 4%, to ensure that the
combining sequence transformation unifies position encoding. Second, we propose
dynamic mask attention, which maintains 100% accuracy in the more challenging
multi-query associative recall task, improving by more than 150% compared to
quadratic causal self-attention and state space duality, to ensure that the
combining sequence transformation selectively filters relevant information.
Third, we design cross domain mixture of experts, which makes the computational
speed of expert retrieval with more than 1024 experts 8 to 10 times faster than
the mixture of experts, to ensure that the combining state transformation
quickly retrieval mixture. Finally, we summarize these matrix algorithms that
can form the foundation model: Wonderful Matrices, which can be a competitor to
popular model architectures.
SPaR: Self-Play with Tree-Search Refinement to Improve
  Instruction-Following in Large Language Models
Authors: name: Jiale Cheng, name: Xiao Liu, name: Cunxiang Wang, name: Xiaotao Gu, name: Yida Lu, name: Dan Zhang, name: Yuxiao Dong, name: Jie Tang, name: Hongning Wang, name: Minlie Huang
Published: 2024-12-16 09:47:43

Instruction-following is a fundamental capability of language models,
requiring the model to recognize even the most subtle requirements in the
instructions and accurately reflect them in its output. Such an ability is
well-suited for and often optimized by preference learning. However, existing
methods often directly sample multiple independent responses from the model
when creating preference pairs. Such practice can introduce content variations
irrelevant to whether the instruction is precisely followed (e.g., different
expressions about the same semantic), interfering with the goal of teaching
models to recognize the key differences that lead to improved instruction
following. In light of this, we introduce SPaR, a self-play framework
integrating tree-search self-refinement to yield valid and comparable
preference pairs free from distractions. By playing against itself, an LLM
employs a tree-search strategy to refine its previous responses with respect to
the instruction while minimizing unnecessary variations. Our experiments show
that a LLaMA3-8B model, trained over three iterations guided by SPaR, surpasses
GPT-4-Turbo on the IFEval benchmark without losing general capabilities.
Furthermore, SPaR demonstrates promising scalability and transferability,
greatly enhancing models like GLM-4-9B and LLaMA3-70B. We also identify how
inference scaling in tree search would impact model performance. Our code and
data are publicly available at https://github.com/thu-coai/SPaR.