Orionnnn/hub-stats huggingface dataset:
WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum
  Reinforcement Learning
Authors: name: Zehan Qi, name: Xiao Liu, name: Iat Long Iong, name: Hanyu Lai, name: Xueqiao Sun, name: Xinyue Yang, name: Jiadai Sun, name: Yu Yang, name: Shuntian Yao, name: Tianjie Zhang, name: Wei Xu, name: Jie Tang, name: Yuxiao Dong
Published: 2024-11-04 17:59:58

Large language models (LLMs) have shown remarkable potential as autonomous
agents, particularly in web-based tasks. However, existing LLM web agents
heavily rely on expensive proprietary LLM APIs, while open LLMs lack the
necessary decision-making capabilities. This paper introduces WebRL, a
self-evolving online curriculum reinforcement learning framework designed to
train high-performance web agents using open LLMs. WebRL addresses three key
challenges in building LLM web agents, including the scarcity of training
tasks, sparse feedback signals, and policy distribution drift in online
learning. Specifically, WebRL incorporates 1) a self-evolving curriculum that
generates new tasks from unsuccessful attempts, 2) a robust outcome-supervised
reward model (ORM), and 3) adaptive reinforcement learning strategies to ensure
consistent improvements. We apply WebRL to transform open Llama-3.1 and GLM-4
models into proficient web agents. On WebArena-Lite, WebRL improves the success
rate of Llama-3.1-8B from 4.8% to 42.4%, and from 6.1% to 43% for GLM-4-9B.
These open models significantly surpass the performance of GPT-4-Turbo (17.6%)
and GPT-4o (13.9%) and outperform previous state-of-the-art web agents trained
on open LLMs (AutoWebGLM, 18.2%). Our findings demonstrate WebRL's
effectiveness in bridging the gap between open and proprietary LLM-based web
agents, paving the way for more accessible and powerful autonomous web
interaction systems.
Adaptive Caching for Faster Video Generation with Diffusion Transformers
Authors: name: Kumara Kahatapitiya, name: Haozhe Liu, name: Sen He, name: Ding Liu, name: Menglin Jia, name: Michael S. Ryoo, name: Tian Xie
Published: 2024-11-04 18:59:44

Generating temporally-consistent high-fidelity videos can be computationally
expensive, especially over longer temporal spans. More-recent Diffusion
Transformers (DiTs) -- despite making significant headway in this context --
have only heightened such challenges as they rely on larger models and heavier
attention mechanisms, resulting in slower inference speeds. In this paper, we
introduce a training-free method to accelerate video DiTs, termed Adaptive
Caching (AdaCache), which is motivated by the fact that "not all videos are
created equal": meaning, some videos require fewer denoising steps to attain a
reasonable quality than others. Building on this, we not only cache
computations through the diffusion process, but also devise a caching schedule
tailored to each video generation, maximizing the quality-latency trade-off. We
further introduce a Motion Regularization (MoReg) scheme to utilize video
information within AdaCache, essentially controlling the compute allocation
based on motion content. Altogether, our plug-and-play contributions grant
significant inference speedups (e.g. up to 4.7x on Open-Sora 720p - 2s video
generation) without sacrificing the generation quality, across multiple video
DiT baselines.
Decoding Dark Matter: Specialized Sparse Autoencoders for Interpreting
  Rare Concepts in Foundation Models
Authors: name: Aashiq Muhamed, name: Mona Diab, name: Virginia Smith
Published: 2024-11-01 17:09:34

Understanding and mitigating the potential risks associated with foundation
models (FMs) hinges on developing effective interpretability methods. Sparse
Autoencoders (SAEs) have emerged as a promising tool for disentangling FM
representations, but they struggle to capture rare, yet crucial concepts in the
data. We introduce Specialized Sparse Autoencoders (SSAEs), designed to
illuminate these elusive dark matter features by focusing on specific
subdomains. We present a practical recipe for training SSAEs, demonstrating the
efficacy of dense retrieval for data selection and the benefits of Tilted
Empirical Risk Minimization as a training objective to improve concept recall.
Our evaluation of SSAEs on standard metrics, such as downstream perplexity and
L_0 sparsity, show that they effectively capture subdomain tail concepts,
exceeding the capabilities of general-purpose SAEs. We showcase the practical
utility of SSAEs in a case study on the Bias in Bios dataset, where SSAEs
achieve a 12.5\% increase in worst-group classification accuracy when applied
to remove spurious gender information. SSAEs provide a powerful new lens for
peering into the inner workings of FMs in subdomains.
How Far is Video Generation from World Model: A Physical Law Perspective
Authors: name: Bingyi Kang, name: Yang Yue, name: Rui Lu, name: Zhijie Lin, name: Yang Zhao, name: Kaixin Wang, name: Gao Huang, name: Jiashi Feng
Published: 2024-11-04 18:53:05

OpenAI's Sora highlights the potential of video generation for developing
world models that adhere to fundamental physical laws. However, the ability of
video generation models to discover such laws purely from visual data without
human priors can be questioned. A world model learning the true law should give
predictions robust to nuances and correctly extrapolate on unseen scenarios. In
this work, we evaluate across three key scenarios: in-distribution,
out-of-distribution, and combinatorial generalization. We developed a 2D
simulation testbed for object movement and collisions to generate videos
deterministically governed by one or more classical mechanics laws. This
provides an unlimited supply of data for large-scale experimentation and
enables quantitative evaluation of whether the generated videos adhere to
physical laws. We trained diffusion-based video generation models to predict
object movements based on initial frames. Our scaling experiments show perfect
generalization within the distribution, measurable scaling behavior for
combinatorial generalization, but failure in out-of-distribution scenarios.
Further experiments reveal two key insights about the generalization mechanisms
of these models: (1) the models fail to abstract general physical rules and
instead exhibit "case-based" generalization behavior, i.e., mimicking the
closest training example; (2) when generalizing to new cases, models are
observed to prioritize different factors when referencing training data: color
> size > velocity > shape. Our study suggests that scaling alone is
insufficient for video generation models to uncover fundamental physical laws,
despite its role in Sora's broader success. See our project page at
https://phyworld.github.io
LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in
  Large Language Models
Authors: name: Nam V. Nguyen, name: Thong T. Doan, name: Luong Tran, name: Van Nguyen, name: Quang Pham
Published: 2024-11-01 14:04:36

Mixture of Experts (MoEs) plays an important role in the development of more
efficient and effective large language models (LLMs). Due to the enormous
resource requirements, studying large scale MoE algorithms remain in-accessible
to many researchers. This work develops LibMoE, a comprehensive and
modular framework to streamline the research, training, and evaluation of MoE
algorithms. Built upon three core principles: (i) modular design, (ii)
efficient training; (iii) comprehensive evaluation, LibMoE brings MoE in LLMs
more accessible to a wide range of researchers by standardizing the training
and evaluation pipelines. Using LibMoE, we extensively benchmarked five
state-of-the-art MoE algorithms over three different LLMs and 11 datasets under
the zero-shot setting. The results show that despite the unique
characteristics, all MoE algorithms perform roughly similar when averaged
across a wide range of tasks. With the modular design and extensive evaluation,
we believe LibMoE will be invaluable for researchers to make meaningful
progress towards the next generation of MoE and LLMs. Project page:
https://fsoft-aic.github.io/fsoft-LibMoE.github.io.
DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical
  Reasoning Robustness of Vision Language Models
Authors: name: Chengke Zou, name: Xingang Guo, name: Rui Yang, name: Junyu Zhang, name: Bin Hu, name: Huan Zhang
Published: 2024-10-29 17:29:19

The rapid advancements in Vision-Language Models (VLMs) have shown great
potential in tackling mathematical reasoning tasks that involve visual context.
Unlike humans who can reliably apply solution steps to similar problems with
minor modifications, we found that SOTA VLMs like GPT-4o can consistently fail
in these scenarios, revealing limitations in their mathematical reasoning
capabilities. In this paper, we investigate the mathematical reasoning
robustness in VLMs and evaluate how well these models perform under different
variants of the same question, such as changes in visual numerical values or
function graphs. While several vision-based math benchmarks have been developed
to assess VLMs' problem-solving capabilities, these benchmarks contain only
static sets of problems and cannot easily evaluate mathematical reasoning
robustness. To fill this gap, we introduce DynaMath, a dynamic visual math
benchmark designed for in-depth assessment of VLMs. DynaMath includes 501
high-quality, multi-topic seed questions, each represented as a Python program.
Those programs are carefully designed and annotated to enable the automatic
generation of a much larger set of concrete questions, including many different
types of visual and textual variations. DynaMath allows us to evaluate the
generalization ability of VLMs, by assessing their performance under varying
input conditions of a seed question. We evaluated 14 SOTA VLMs with 5,010
generated concrete questions. Our results show that the worst-case model
accuracy, defined as the percentage of correctly answered seed questions in all
10 variants, is significantly lower than the average-case accuracy. Our
analysis emphasizes the need to study the robustness of VLMs' reasoning
abilities, and DynaMath provides valuable insights to guide the development of
more reliable models for mathematical reasoning.
Fashion-VDM: Video Diffusion Model for Virtual Try-On
Authors: name: Johanna Karras, name: Yingwei Li, name: Nan Liu, name: Luyang Zhu, name: Innfarn Yoo, name: Andreas Lugmayr, name: Chris Lee, name: Ira Kemelmacher-Shlizerman
Published: 2024-10-31 21:52:33

We present Fashion-VDM, a video diffusion model (VDM) for generating virtual
try-on videos. Given an input garment image and person video, our method aims
to generate a high-quality try-on video of the person wearing the given
garment, while preserving the person's identity and motion. Image-based virtual
try-on has shown impressive results; however, existing video virtual try-on
(VVT) methods are still lacking garment details and temporal consistency. To
address these issues, we propose a diffusion-based architecture for video
virtual try-on, split classifier-free guidance for increased control over the
conditioning inputs, and a progressive temporal training strategy for
single-pass 64-frame, 512px video generation. We also demonstrate the
effectiveness of joint image-video training for video try-on, especially when
video data is limited. Our qualitative and quantitative experiments show that
our approach sets the new state-of-the-art for video virtual try-on. For
additional results, visit our project page:
https://johannakarras.github.io/Fashion-VDM.
Physics in Next-token Prediction
Authors: name: Hongjun An, name: Yiliang Song, name: Xuelong Li
Published: 2024-11-01 15:26:15

We discovered the underlying physics in Next-token Prediction (NTP). We
identified the law of information conservation within NTP and proposed the
First Law of Information Capacity (IC-1), demonstrating that the essence of
intelligence emergence in auto-regressive models is fundamentally a process of
information transfer. We also introduced Landauer's Principle into NTP,
formulating the Second Law of Information Capacity (IC-2), which establishes
the relationship between auto-regressive model training and energy consumption.
Additionally, we presented several corollaries, which hold practical
significance for production practices. Finally, we validated the compatibility
and complementarity of our findings with existing theories.
TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal
  Foundation Models
Authors: name: Ziyao Shangguan, name: Chuhan Li, name: Yuxuan Ding, name: Yanan Zheng, name: Yilun Zhao, name: Tesca Fitzgerald, name: Arman Cohan
Published: 2024-10-30 17:50:23

Existing benchmarks often highlight the remarkable performance achieved by
state-of-the-art Multimodal Foundation Models (MFMs) in leveraging temporal
context for video understanding. However, how well do the models truly perform
visual temporal reasoning? Our study of existing benchmarks shows that this
capability of MFMs is likely overestimated as many questions can be solved by
using a single, few, or out-of-order frames. To systematically examine current
visual temporal reasoning tasks, we propose three principles with corresponding
metrics: (1) Multi-Frame Gain, (2) Frame Order Sensitivity, and (3) Frame
Information Disparity. Following these principles, we introduce TOMATO,
Temporal Reasoning Multimodal Evaluation, a novel benchmark crafted to
rigorously assess MFMs' temporal reasoning capabilities in video understanding.
TOMATO comprises 1,484 carefully curated, human-annotated questions spanning
six tasks (i.e., action count, direction, rotation, shape & trend, velocity &
frequency, and visual cues), applied to 1,417 videos, including 805
self-recorded and -generated videos, that encompass human-centric, real-world,
and simulated scenarios. Our comprehensive evaluation reveals a human-model
performance gap of 57.3% with the best-performing model. Moreover, our in-depth
analysis uncovers more fundamental limitations beyond this gap in current MFMs.
While they can accurately recognize events in isolated frames, they fail to
interpret these frames as a continuous sequence. We believe TOMATO will serve
as a crucial testbed for evaluating the next-generation MFMs and as a call to
the community to develop AI systems capable of comprehending human world
dynamics through the video modality.
CityGaussianV2: Efficient and Geometrically Accurate Reconstruction for
  Large-Scale Scenes
Authors: name: Yang Liu, name: Chuanchen Luo, name: Zhongkai Mao, name: Junran Peng, name: Zhaoxiang Zhang
Published: 2024-11-01 17:59:31

Recently, 3D Gaussian Splatting (3DGS) has revolutionized radiance field
reconstruction, manifesting efficient and high-fidelity novel view synthesis.
However, accurately representing surfaces, especially in large and complex
scenarios, remains a significant challenge due to the unstructured nature of
3DGS. In this paper, we present CityGaussianV2, a novel approach for
large-scale scene reconstruction that addresses critical challenges related to
geometric accuracy and efficiency. Building on the favorable generalization
capabilities of 2D Gaussian Splatting (2DGS), we address its convergence and
scalability issues. Specifically, we implement a decomposed-gradient-based
densification and depth regression technique to eliminate blurry artifacts and
accelerate convergence. To scale up, we introduce an elongation filter that
mitigates Gaussian count explosion caused by 2DGS degeneration. Furthermore, we
optimize the CityGaussian pipeline for parallel training, achieving up to
10times compression, at least 25% savings in training time, and a 50%
decrease in memory usage. We also established standard geometry benchmarks
under large-scale scenes. Experimental results demonstrate that our method
strikes a promising balance between visual quality, geometric accuracy, as well
as storage and training costs. The project page is available at
https://dekuliutesla.github.io/CityGaussianV2/.
Face Anonymization Made Simple
Authors: name: Han-Wei Kung, name: Tuomas Varanka, name: Sanjay Saha, name: Terence Sim, name: Nicu Sebe
Published: 2024-11-01 17:45:21

Current face anonymization techniques often depend on identity loss
calculated by face recognition models, which can be inaccurate and unreliable.
Additionally, many methods require supplementary data such as facial landmarks
and masks to guide the synthesis process. In contrast, our approach uses
diffusion models with only a reconstruction loss, eliminating the need for
facial landmarks or masks while still producing images with intricate,
fine-grained details. We validated our results on two public benchmarks through
both quantitative and qualitative evaluations. Our model achieves
state-of-the-art performance in three key areas: identity anonymization, facial
attribute preservation, and image quality. Beyond its primary function of
anonymization, our model can also perform face swapping tasks by incorporating
an additional facial image as input, demonstrating its versatility and
potential for diverse applications. Our code and models are available at
https://github.com/hanweikung/face_anon_simple .
GPT or BERT: why not both?
Authors: name: Lucas Georges Gabriel Charpentier, name: David Samuel
Published: 2024-10-31 17:18:11

We present a simple way to merge masked language modeling with causal
language modeling. This hybrid training objective results in a model that
combines the strengths of both modeling paradigms within a single transformer
stack: GPT-BERT can be transparently used like any standard causal or masked
language model. We test the pretraining process that enables this flexible
behavior on the BabyLM Challenge 2024. The results show that the hybrid
pretraining outperforms masked-only or causal-only models. We openly release
the models, training corpora and code.
Zipfian Whitening
Authors: name: Sho Yokoi, name: Han Bao, name: Hiroto Kurita, name: Hidetoshi Shimodaira
Published: 2024-11-01 15:40:19

The word embedding space in neural models is skewed, and correcting this can
improve task performance. We point out that most approaches for modeling,
correcting, and measuring the symmetry of an embedding space implicitly assume
that the word frequencies are uniform; in reality, word frequencies follow a
highly non-uniform distribution, known as Zipf's law. Surprisingly, simply
performing PCA whitening weighted by the empirical word frequency that follows
Zipf's law significantly improves task performance, surpassing established
baselines. From a theoretical perspective, both our approach and existing
methods can be clearly categorized: word representations are distributed
according to an exponential family with either uniform or Zipfian base
measures. By adopting the latter approach, we can naturally emphasize
informative low-frequency words in terms of their vector norm, which becomes
evident from the information-geometric perspective, and in terms of the loss
functions for imbalanced classification. Additionally, our theory corroborates
that popular natural language processing methods, such as skip-gram negative
sampling, WhiteningBERT, and headless language models, work well just because
their word embeddings encode the empirical word frequency into the underlying
probabilistic model.
Constant Acceleration Flow
Authors: name: Dogyun Park, name: Sojin Lee, name: Sihyeon Kim, name: Taehoon Lee, name: Youngjoon Hong, name: Hyunwoo J. Kim
Published: 2024-11-01 02:43:56

Rectified flow and reflow procedures have significantly advanced fast
generation by progressively straightening ordinary differential equation (ODE)
flows. They operate under the assumption that image and noise pairs, known as
couplings, can be approximated by straight trajectories with constant velocity.
However, we observe that modeling with constant velocity and using reflow
procedures have limitations in accurately learning straight trajectories
between pairs, resulting in suboptimal performance in few-step generation. To
address these limitations, we introduce Constant Acceleration Flow (CAF), a
novel framework based on a simple constant acceleration equation. CAF
introduces acceleration as an additional learnable variable, allowing for more
expressive and accurate estimation of the ODE flow. Moreover, we propose two
techniques to further improve estimation accuracy: initial velocity
conditioning for the acceleration model and a reflow process for the initial
velocity. Our comprehensive studies on toy datasets, CIFAR-10, and ImageNet
64x64 demonstrate that CAF outperforms state-of-the-art baselines for one-step
generation. We also show that CAF dramatically improves few-step coupling
preservation and inversion over Rectified flow. Code is available at
https://github.com/mlvlab/CAF{https://github.com/mlvlab/CAF}.
WikiNER-fr-gold: A Gold-Standard NER Corpus
Authors: name: Danrun Cao, name: Nicolas Béchet, name: Pierre-François Marteau
Published: 2024-10-29 08:00:16

We address in this article the the quality of the WikiNER corpus, a
multilingual Named Entity Recognition corpus, and provide a consolidated
version of it. The annotation of WikiNER was produced in a semi-supervised
manner i.e. no manual verification has been carried out a posteriori. Such
corpus is called silver-standard. In this paper we propose WikiNER-fr-gold
which is a revised version of the French proportion of WikiNER. Our corpus
consists of randomly sampled 20% of the original French sub-corpus (26,818
sentences with 700k tokens). We start by summarizing the entity types included
in each category in order to define an annotation guideline, and then we
proceed to revise the corpus. Finally we present an analysis of errors and
inconsistency observed in the WikiNER-fr corpus, and we discuss potential
future work directions.
OS-ATLAS: A Foundation Action Model for Generalist GUI Agents
Authors: name: Zhiyong Wu, name: Zhenyu Wu, name: Fangzhi Xu, name: Yian Wang, name: Qiushi Sun, name: Chengyou Jia, name: Kanzhi Cheng, name: Zichen Ding, name: Liheng Chen, name: Paul Pu Liang, name: Yu Qiao
Published: 2024-10-30 17:10:19

Existing efforts in building GUI agents heavily rely on the availability of
robust commercial Vision-Language Models (VLMs) such as GPT-4o and
GeminiProVision. Practitioners are often reluctant to use open-source VLMs due
to their significant performance lag compared to their closed-source
counterparts, particularly in GUI grounding and Out-Of-Distribution (OOD)
scenarios. To facilitate future research in this area, we developed OS-Atlas -
a foundational GUI action model that excels at GUI grounding and OOD agentic
tasks through innovations in both data and modeling. We have invested
significant engineering effort in developing an open-source toolkit for
synthesizing GUI grounding data across multiple platforms, including Windows,
Linux, MacOS, Android, and the web. Leveraging this toolkit, we are releasing
the largest open-source cross-platform GUI grounding corpus to date, which
contains over 13 million GUI elements. This dataset, combined with innovations
in model training, provides a solid foundation for OS-Atlas to understand GUI
screenshots and generalize to unseen interfaces. Through extensive evaluation
across six benchmarks spanning three different platforms (mobile, desktop, and
web), OS-Atlas demonstrates significant performance improvements over previous
state-of-the-art models. Our evaluation also uncovers valuable insights into
continuously improving and scaling the agentic capabilities of open-source
VLMs.