Orionnnn/hub-stats huggingface dataset:
VBench++: Comprehensive and Versatile Benchmark Suite for Video
  Generative Models
Authors: name: Ziqi Huang, name: Fan Zhang, name: Xiaojie Xu, name: Yinan He, name: Jiashuo Yu, name: Ziyue Dong, name: Qianli Ma, name: Nattapol Chanpaisit, name: Chenyang Si, name: Yuming Jiang, name: Yaohui Wang, name: Xinyuan Chen, name: Ying-Cong Chen, name: Limin Wang, name: Dahua Lin, name: Yu Qiao, name: Ziwei Liu
Published: 2024-11-20 17:54:41

Video generation has witnessed significant advancements, yet evaluating these
models remains a challenge. A comprehensive evaluation benchmark for video
generation is indispensable for two reasons: 1) Existing metrics do not fully
align with human perceptions; 2) An ideal evaluation system should provide
insights to inform future developments of video generation. To this end, we
present VBench, a comprehensive benchmark suite that dissects "video generation
quality" into specific, hierarchical, and disentangled dimensions, each with
tailored prompts and evaluation methods. VBench has several appealing
properties: 1) Comprehensive Dimensions: VBench comprises 16 dimensions in
video generation (e.g., subject identity inconsistency, motion smoothness,
temporal flickering, and spatial relationship, etc). The evaluation metrics
with fine-grained levels reveal individual models' strengths and weaknesses. 2)
Human Alignment: We also provide a dataset of human preference annotations to
validate our benchmarks' alignment with human perception, for each evaluation
dimension respectively. 3) Valuable Insights: We look into current models'
ability across various evaluation dimensions, and various content types. We
also investigate the gaps between video and image generation models. 4)
Versatile Benchmarking: VBench++ supports evaluating text-to-video and
image-to-video. We introduce a high-quality Image Suite with an adaptive aspect
ratio to enable fair evaluations across different image-to-video generation
settings. Beyond assessing technical quality, VBench++ evaluates the
trustworthiness of video generative models, providing a more holistic view of
model performance. 5) Full Open-Sourcing: We fully open-source VBench++ and
continually add new video generation models to our leaderboard to drive forward
the field of video generation.
Is Your LLM Secretly a World Model of the Internet? Model-Based Planning
  for Web Agents
Authors: name: Yu Gu, name: Boyuan Zheng, name: Boyu Gou, name: Kai Zhang, name: Cheng Chang, name: Sanjari Srivastava, name: Yanan Xie, name: Peng Qi, name: Huan Sun, name: Yu Su
Published: 2024-11-10 18:50:51

Language agents have demonstrated promising capabilities in automating
web-based tasks, though their current reactive approaches still underperform
largely compared to humans. While incorporating advanced planning algorithms,
particularly tree search methods, could enhance these agents' performance,
implementing tree search directly on live websites poses significant safety
risks and practical constraints due to irreversible actions such as confirming
a purchase. In this paper, we introduce a novel paradigm that augments language
agents with model-based planning, pioneering the innovative use of large
language models (LLMs) as world models in complex web environments. Our method,
WebDreamer, builds on the key insight that LLMs inherently encode comprehensive
knowledge about website structures and functionalities. Specifically,
WebDreamer uses LLMs to simulate outcomes for each candidate action (e.g.,
"what would happen if I click this button?") using natural language
descriptions, and then evaluates these imagined outcomes to determine the
optimal action at each step. Empirical results on two representative web agent
benchmarks with online interaction -- VisualWebArena and Mind2Web-live --
demonstrate that WebDreamer achieves substantial improvements over reactive
baselines. By establishing the viability of LLMs as world models in web
environments, this work lays the groundwork for a paradigm shift in automated
web interaction. More broadly, our findings open exciting new avenues for
future research into 1) optimizing LLMs specifically for world modeling in
complex, dynamic environments, and 2) model-based speculative planning for
language agents.
SageAttention2 Technical Report: Accurate 4 Bit Attention for
  Plug-and-play Inference Acceleration
Authors: name: Jintao Zhang, name: Haofeng Huang, name: Pengle Zhang, name: Jia Wei, name: Jun Zhu, name: Jianfei Chen
Published: 2024-11-17 04:35:49

Although quantization for linear layers has been widely used, its application
to accelerate the attention process remains limited. SageAttention utilizes
8-bit matrix multiplication, 16-bit matrix multiplication with 16-bit
accumulator, and precision-enhancing methods, implementing an accurate and 2x
speedup kernel compared to FlashAttention2. To further enhance the efficiency
of attention computation while maintaining precision, we propose
SageAttention2, which utilizes significantly faster 4-bit matrix multiplication
(Matmul) alongside additional precision-enhancing techniques. First, we propose
to quantize matrixes (Q, K) to INT4 in a warp-level granularity and quantize
matrixes (widetilde P, V) to FP8. Second, we propose a method to smooth Q
and V, enhancing the accuracy of attention with INT4 QK and FP8 PV.
Third, we analyze the quantization accuracy across timesteps and layers, then
propose an adaptive quantization method to ensure the end-to-end metrics over
various models. The operations per second (OPS) of SageAttention2 surpass
FlashAttention2 and xformers by about 3x and 5x on RTX4090, respectively.
Comprehensive experiments confirm that our approach incurs negligible
end-to-end metrics loss across diverse models, including those for large
language processing, image generation, and video generation. The codes are
available at https://github.com/thu-ml/SageAttention.
SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking
  with Motion-Aware Memory
Authors: name: Cheng-Yen Yang, name: Hsiang-Wei Huang, name: Wenhao Chai, name: Zhongyu Jiang, name: Jenq-Neng Hwang
Published: 2024-11-18 05:59:03

The Segment Anything Model 2 (SAM 2) has demonstrated strong performance in
object segmentation tasks but faces challenges in visual object tracking,
particularly when managing crowded scenes with fast-moving or self-occluding
objects. Furthermore, the fixed-window memory approach in the original model
does not consider the quality of memories selected to condition the image
features for the next frame, leading to error propagation in videos. This paper
introduces SAMURAI, an enhanced adaptation of SAM 2 specifically designed for
visual object tracking. By incorporating temporal motion cues with the proposed
motion-aware memory selection mechanism, SAMURAI effectively predicts object
motion and refines mask selection, achieving robust, accurate tracking without
the need for retraining or fine-tuning. SAMURAI operates in real-time and
demonstrates strong zero-shot performance across diverse benchmark datasets,
showcasing its ability to generalize without fine-tuning. In evaluations,
SAMURAI achieves significant improvements in success rate and precision over
existing trackers, with a 7.1% AUC gain on LaSOT_{ext} and a 3.5% AO
gain on GOT-10k. Moreover, it achieves competitive results compared to fully
supervised methods on LaSOT, underscoring its robustness in complex tracking
scenarios and its potential for real-world applications in dynamic
environments. Code and results are available at
https://github.com/yangchris11/samurai.
SymDPO: Boosting In-Context Learning of Large Multimodal Models with
  Symbol Demonstration Direct Preference Optimization
Authors: name: Hongrui Jia, name: Chaoya Jiang, name: Haiyang Xu, name: Wei Ye, name: Mengfan Dong, name: Ming Yan, name: Ji Zhang, name: Fei Huang, name: Shikun Zhang
Published: 2024-11-17 08:29:14

As language models continue to scale, Large Language Models (LLMs) have
exhibited emerging capabilities in In-Context Learning (ICL), enabling them to
solve language tasks by prefixing a few in-context demonstrations (ICDs) as
context. Inspired by these advancements, researchers have extended these
techniques to develop Large Multimodal Models (LMMs) with ICL capabilities.
However, existing LMMs face a critical issue: they often fail to effectively
leverage the visual context in multimodal demonstrations and instead simply
follow textual patterns. This indicates that LMMs do not achieve effective
alignment between multimodal demonstrations and model outputs. To address this
problem, we propose Symbol Demonstration Direct Preference Optimization
(SymDPO). Specifically, SymDPO aims to break the traditional paradigm of
constructing multimodal demonstrations by using random symbols to replace text
answers within instances. This forces the model to carefully understand the
demonstration images and establish a relationship between the images and the
symbols to answer questions correctly. We validate the effectiveness of this
method on multiple benchmarks, demonstrating that with SymDPO, LMMs can more
effectively understand the multimodal context within examples and utilize this
knowledge to answer questions better.
FlipSketch: Flipping Static Drawings to Text-Guided Sketch Animations
Authors: name: Hmrishav Bandyopadhyay, name: Yi-Zhe Song
Published: 2024-11-16 14:53:03

Sketch animations offer a powerful medium for visual storytelling, from
simple flip-book doodles to professional studio productions. While traditional
animation requires teams of skilled artists to draw key frames and in-between
frames, existing automation attempts still demand significant artistic effort
through precise motion paths or keyframe specification. We present FlipSketch,
a system that brings back the magic of flip-book animation -- just draw your
idea and describe how you want it to move! Our approach harnesses motion priors
from text-to-video diffusion models, adapting them to generate sketch
animations through three key innovations: (i) fine-tuning for sketch-style
frame generation, (ii) a reference frame mechanism that preserves visual
integrity of input sketch through noise refinement, and (iii) a dual-attention
composition that enables fluid motion without losing visual consistency. Unlike
constrained vector animations, our raster frames support dynamic sketch
transformations, capturing the expressive freedom of traditional animation. The
result is an intuitive system that makes sketch animation as simple as doodling
and describing, while maintaining the artistic essence of hand-drawn animation.
SEAGULL: No-reference Image Quality Assessment for Regions of Interest
  via Vision-Language Instruction Tuning
Authors: name: Zewen Chen, name: Juan Wang, name: Wen Wang, name: Sunhan Xu, name: Hang Xiong, name: Yun Zeng, name: Jian Guo, name: Shuxun Wang, name: Chunfeng Yuan, name: Bing Li, name: Weiming Hu
Published: 2024-11-15 13:07:22

Existing Image Quality Assessment (IQA) methods achieve remarkable success in
analyzing quality for overall image, but few works explore quality analysis for
Regions of Interest (ROIs). The quality analysis of ROIs can provide
fine-grained guidance for image quality improvement and is crucial for
scenarios focusing on region-level quality. This paper proposes a novel
network, SEAGULL, which can SEe and Assess ROIs quality with GUidance from a
Large vision-Language model. SEAGULL incorporates a vision-language model
(VLM), masks generated by Segment Anything Model (SAM) to specify ROIs, and a
meticulously designed Mask-based Feature Extractor (MFE) to extract global and
local tokens for specified ROIs, enabling accurate fine-grained IQA for ROIs.
Moreover, this paper constructs two ROI-based IQA datasets, SEAGULL-100w and
SEAGULL-3k, for training and evaluating ROI-based IQA. SEAGULL-100w comprises
about 100w synthetic distortion images with 33 million ROIs for pre-training to
improve the model's ability of regional quality perception, and SEAGULL-3k
contains about 3k authentic distortion ROIs to enhance the model's ability to
perceive real world distortions. After pre-training on SEAGULL-100w and
fine-tuning on SEAGULL-3k, SEAGULL shows remarkable performance on fine-grained
ROI quality assessment. Code and datasets are publicly available at the
https://github.com/chencn2020/Seagull.
ITACLIP: Boosting Training-Free Semantic Segmentation with Image, Text,
  and Architectural Enhancements
Authors: name: M. Arda Aydın, name: Efe Mert Çırpar, name: Elvin Abdinli, name: Gozde Unal, name: Yusuf H. Sahin
Published: 2024-11-18 20:31:38

Recent advances in foundational Vision Language Models (VLMs) have reshaped
the evaluation paradigm in computer vision tasks. These foundational models,
especially CLIP, have accelerated research in open-vocabulary computer vision
tasks, including Open-Vocabulary Semantic Segmentation (OVSS). Although the
initial results are promising, the dense prediction capabilities of VLMs still
require further improvement. In this study, we enhance the semantic
segmentation performance of CLIP by introducing new modules and modifications:
1) architectural changes in the last layer of ViT and the incorporation of
attention maps from the middle layers with the last layer, 2) Image
Engineering: applying data augmentations to enrich input image representations,
and 3) using Large Language Models (LLMs) to generate definitions and synonyms
for each class name to leverage CLIP's open-vocabulary capabilities. Our
training-free method, ITACLIP, outperforms current state-of-the-art approaches
on segmentation benchmarks such as COCO-Stuff, COCO-Object, Pascal Context, and
Pascal VOC. Our code is available at https://github.com/m-arda-aydn/ITACLIP.
RedPajama: an Open Dataset for Training Large Language Models
Authors: name: Maurice Weber, name: Daniel Fu, name: Quentin Anthony, name: Yonatan Oren, name: Shane Adams, name: Anton Alexandrov, name: Xiaozhong Lyu, name: Huu Nguyen, name: Xiaozhe Yao, name: Virginia Adams, name: Ben Athiwaratkun, name: Rahul Chalamala, name: Kezhen Chen, name: Max Ryabinin, name: Tri Dao, name: Percy Liang, name: Christopher Ré, name: Irina Rish, name: Ce Zhang
Published: 2024-11-19 09:35:28

Large language models are increasingly becoming a cornerstone technology in
artificial intelligence, the sciences, and society as a whole, yet the optimal
strategies for dataset composition and filtering remain largely elusive. Many
of the top-performing models lack transparency in their dataset curation and
model development processes, posing an obstacle to the development of fully
open language models. In this paper, we identify three core data-related
challenges that must be addressed to advance open-source language models. These
include (1) transparency in model development, including the data curation
process, (2) access to large quantities of high-quality data, and (3)
availability of artifacts and metadata for dataset curation and analysis. To
address these challenges, we release RedPajama-V1, an open reproduction of the
LLaMA training dataset. In addition, we release RedPajama-V2, a massive
web-only dataset consisting of raw, unfiltered text data together with quality
signals and metadata. Together, the RedPajama datasets comprise over 100
trillion tokens spanning multiple domains and with their quality signals
facilitate the filtering of data, aiming to inspire the development of numerous
new datasets. To date, these datasets have already been used in the training of
strong language models used in production, such as Snowflake Arctic,
Salesforce's XGen and AI2's OLMo. To provide insight into the quality of
RedPajama, we present a series of analyses and ablation studies with
decoder-only language models with up to 1.6B parameters. Our findings
demonstrate how quality signals for web data can be effectively leveraged to
curate high-quality subsets of the dataset, underscoring the potential of
RedPajama to advance the development of transparent and high-performing
language models at scale.
Building Trust: Foundations of Security, Safety and Transparency in AI
Authors: name: Huzaifa Sidhpurwala, name: Garth Mollett, name: Emily Fox, name: Mark Bestavros, name: Huamin Chen
Published: 2024-11-19 06:55:57

This paper explores the rapidly evolving ecosystem of publicly available AI
models, and their potential implications on the security and safety landscape.
As AI models become increasingly prevalent, understanding their potential risks
and vulnerabilities is crucial. We review the current security and safety
scenarios while highlighting challenges such as tracking issues, remediation,
and the apparent absence of AI model lifecycle and ownership processes.
Comprehensive strategies to enhance security and safety for both model
developers and end-users are proposed. This paper aims to provide some of the
foundational pieces for more standardized security, safety, and transparency in
the development and operation of AI models and the larger open ecosystems and
communities forming around them.
Soft Robotic Dynamic In-Hand Pen Spinning
Authors: name: Yunchao Yao, name: Uksang Yoo, name: Jean Oh, name: Christopher G. Atkeson, name: Jeffrey Ichnowski
Published: 2024-11-19 18:57:41

Dynamic in-hand manipulation remains a challenging task for soft robotic
systems that have demonstrated advantages in safe compliant interactions but
struggle with high-speed dynamic tasks. In this work, we present SWIFT, a
system for learning dynamic tasks using a soft and compliant robotic hand.
Unlike previous works that rely on simulation, quasi-static actions and precise
object models, the proposed system learns to spin a pen through trial-and-error
using only real-world data without requiring explicit prior knowledge of the
pen's physical attributes. With self-labeled trials sampled from the real
world, the system discovers the set of pen grasping and spinning primitive
parameters that enables a soft hand to spin a pen robustly and reliably. After
130 sampled actions per object, SWIFT achieves 100% success rate across three
pens with different weights and weight distributions, demonstrating the
system's generalizability and robustness to changes in object properties. The
results highlight the potential for soft robotic end-effectors to perform
dynamic tasks including rapid in-hand manipulation. We also demonstrate that
SWIFT generalizes to spinning items with different shapes and weights such as a
brush and a screwdriver which we spin with 10/10 and 5/10 success rates
respectively. Videos, data, and code are available at
https://soft-spin.github.io.
Evaluating Tokenizer Performance of Large Language Models Across
  Official Indian Languages
Authors: name: S. Tamang, name: D. J. Bora
Published: 2024-11-19 05:37:17

Large Language Models (LLMs) based on transformer architectures have
revolutionized a variety of domains, with tokenization playing a pivotal role
in their pre-processing and fine-tuning stages. In multilingual models,
particularly those tailored for Indic languages, effective tokenization is
crucial for optimizing performance. This paper presents a comprehensive
evaluation of tokenizers used by 12 LLMs across all 22 official languages of
India, with a focus on comparing the efficiency of their tokenization
processes. We employed the Normalized Sequence Length (NSL) as a key metric in
our analysis. Our findings reveal that the SUTRA tokenizer outperforms all
other models, including several Indic-specific models, excelling in 14
languages. Notable insights include the SUTRA tokenizer's superior handling of
Indic languages, GPT-4o's advancement over its predecessor GPT-4 in processing
Indian languages, and the limited performance of Project Indus in certain
languages. This study underscores the critical importance of developing
targeted tokenization strategies for multilingual and Indic-centric models,
laying the groundwork for future improvements in tokenizer design to enhance
linguistic coverage and model efficiency.
Continuous Speculative Decoding for Autoregressive Image Generation
Authors: name: Zili Wang, name: Robert Zhang, name: Kun Ding, name: Qi Yang, name: Fei Li, name: Shiming Xiang
Published: 2024-11-18 09:19:15

Continuous-valued Autoregressive (AR) image generation models have
demonstrated notable superiority over their discrete-token counterparts,
showcasing considerable reconstruction quality and higher generation fidelity.
However, the computational demands of the autoregressive framework result in
significant inference overhead. While speculative decoding has proven effective
in accelerating Large Language Models (LLMs), their adaptation to
continuous-valued visual autoregressive models remains unexplored. This work
generalizes the speculative decoding algorithm from discrete tokens to
continuous space. By analyzing the intrinsic properties of output distribution,
we establish a tailored acceptance criterion for the diffusion distributions
prevalent in such models. To overcome the inconsistency that occurred in
speculative decoding output distributions, we introduce denoising trajectory
alignment and token pre-filling methods. Additionally, we identify the
hard-to-sample distribution in the rejection phase. To mitigate this issue, we
propose a meticulous acceptance-rejection sampling method with a proper upper
bound, thereby circumventing complex integration. Experimental results show
that our continuous speculative decoding achieves a remarkable 2.33times
speed-up on off-the-shelf models while maintaining the output distribution.
Codes will be available at https://github.com/MarkXCloud/CSpD
VeGaS: Video Gaussian Splatting
Authors: name: Weronika Smolak-Dyżewska, name: Dawid Malarz, name: Kornel Howil, name: Jan Kaczmarczyk, name: Marcin Mazur, name: Przemysław Spurek
Published: 2024-11-17 10:02:36

Implicit Neural Representations (INRs) employ neural networks to approximate
discrete data as continuous functions. In the context of video data, such
models can be utilized to transform the coordinates of pixel locations along
with frame occurrence times (or indices) into RGB color values. Although INRs
facilitate effective compression, they are unsuitable for editing purposes. One
potential solution is to use a 3D Gaussian Splatting (3DGS) based model, such
as the Video Gaussian Representation (VGR), which is capable of encoding video
as a multitude of 3D Gaussians and is applicable for numerous video processing
operations, including editing. Nevertheless, in this case, the capacity for
modification is constrained to a limited set of basic transformations. To
address this issue, we introduce the Video Gaussian Splatting (VeGaS) model,
which enables realistic modifications of video data. To construct VeGaS, we
propose a novel family of Folded-Gaussian distributions designed to capture
nonlinear dynamics in a video stream and model consecutive frames by 2D
Gaussians obtained as respective conditional distributions. Our experiments
demonstrate that VeGaS outperforms state-of-the-art solutions in frame
reconstruction tasks and allows realistic modifications of video data. The code
is available at: https://github.com/gmum/VeGaS.
Evaluating the role of `Constitutions' for learning from AI feedback
Authors: name: Saskia Redgate, name: Andrew M. Bean, name: Adam Mahdi
Published: 2024-11-15 13:16:11

The growing capabilities of large language models (LLMs) have led to their
use as substitutes for human feedback for training and assessing other LLMs.
These methods often rely on `constitutions', written guidelines which a critic
model uses to provide feedback and improve generations. We investigate how the
choice of constitution affects feedback quality by using four different
constitutions to improve patient-centered communication in medical interviews.
In pairwise comparisons conducted by 215 human raters, we found that detailed
constitutions led to better results regarding emotive qualities. However, none
of the constitutions outperformed the baseline in learning more
practically-oriented skills related to information gathering and provision. Our
findings indicate that while detailed constitutions should be prioritised,
there are possible limitations to the effectiveness of AI feedback as a reward
signal in certain areas.