Orionnnn/hub-stats huggingface dataset:
Number it: Temporal Grounding Videos like Flipping Manga
Authors: name: Yongliang Wu, name: Xinting Hu, name: Yuyang Sun, name: Yizhou Zhou, name: Wenbo Zhu, name: Fengyun Rao, name: Bernt Schiele, name: Xu Yang
Published: 2024-11-15 16:32:34

Video Large Language Models (Vid-LLMs) have made remarkable advancements in
comprehending video content for QA dialogue. However, they struggle to extend
this visual understanding to tasks requiring precise temporal localization,
known as Video Temporal Grounding (VTG). To address this gap, we introduce
Number-Prompt (NumPro), a novel method that empowers Vid-LLMs to bridge visual
comprehension with temporal grounding by adding unique numerical identifiers to
each video frame. Treating a video as a sequence of numbered frame images,
NumPro transforms VTG into an intuitive process: flipping through manga panels
in sequence. This allows Vid-LLMs to "read" event timelines, accurately linking
visual content with corresponding temporal information. Our experiments
demonstrate that NumPro significantly boosts VTG performance of top-tier
Vid-LLMs without additional computational cost. Furthermore, fine-tuning on a
NumPro-enhanced dataset defines a new state-of-the-art for VTG, surpassing
previous top-performing methods by up to 6.9\% in mIoU for moment retrieval and
8.5\% in mAP for highlight detection. The code will be available at
https://github.com/yongliang-wu/NumPro.
The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer
  Use
Authors: name: Siyuan Hu, name: Mingyu Ouyang, name: Difei Gao, name: Mike Zheng Shou
Published: 2024-11-15 16:23:52

The recently released model, Claude 3.5 Computer Use, stands out as the first
frontier AI model to offer computer use in public beta as a graphical user
interface (GUI) agent. As an early beta, its capability in the real-world
complex environment remains unknown. In this case study to explore Claude 3.5
Computer Use, we curate and organize a collection of carefully designed tasks
spanning a variety of domains and software. Observations from these cases
demonstrate Claude 3.5 Computer Use's unprecedented ability in end-to-end
language to desktop actions. Along with this study, we provide an
out-of-the-box agent framework for deploying API-based GUI automation models
with easy implementation. Our case studies aim to showcase a groundwork of
capabilities and limitations of Claude 3.5 Computer Use with detailed analyses
and bring to the fore questions about planning, action, and critic, which must
be considered for future improvement. We hope this preliminary exploration will
inspire future research into the GUI agent community. All the test cases in the
paper can be tried through the project:
https://github.com/showlab/computer_use_ootb.
GaussianAnything: Interactive Point Cloud Latent Diffusion for 3D
  Generation
Authors: name: Yushi Lan, name: Shangchen Zhou, name: Zhaoyang Lyu, name: Fangzhou Hong, name: Shuai Yang, name: Bo Dai, name: Xingang Pan, name: Chen Change Loy
Published: 2024-11-12 18:59:32

While 3D content generation has advanced significantly, existing methods
still face challenges with input formats, latent space design, and output
representations. This paper introduces a novel 3D generation framework that
addresses these challenges, offering scalable, high-quality 3D generation with
an interactive Point Cloud-structured Latent space. Our framework employs a
Variational Autoencoder (VAE) with multi-view posed RGB-D(epth)-N(ormal)
renderings as input, using a unique latent space design that preserves 3D shape
information, and incorporates a cascaded latent diffusion model for improved
shape-texture disentanglement. The proposed method, GaussianAnything, supports
multi-modal conditional 3D generation, allowing for point cloud, caption, and
single/multi-view image inputs. Notably, the newly proposed latent space
naturally enables geometry-texture disentanglement, thus allowing 3D-aware
editing. Experimental results demonstrate the effectiveness of our approach on
multiple datasets, outperforming existing methods in both text- and
image-conditioned 3D generation.
LLaVA-o1: Let Vision Language Models Reason Step-by-Step
Authors: name: Guowei Xu, name: Peng Jin, name: Li Hao, name: Yibing Song, name: Lichao Sun, name: Li Yuan
Published: 2024-11-15 18:58:31

Large language models have demonstrated substantial advancements in reasoning
capabilities, particularly through inference-time scaling, as illustrated by
models such as OpenAI's o1. However, current Vision-Language Models (VLMs)
often struggle to perform systematic and structured reasoning, especially when
handling complex visual question-answering tasks. In this work, we introduce
LLaVA-o1, a novel VLM designed to conduct autonomous multistage reasoning.
Unlike chain-of-thought prompting, LLaVA-o1 independently engages in sequential
stages of summarization, visual interpretation, logical reasoning, and
conclusion generation. This structured approach enables LLaVA-o1 to achieve
marked improvements in precision on reasoning-intensive tasks. To accomplish
this, we compile the LLaVA-o1-100k dataset, integrating samples from various
visual question answering sources and providing structured reasoning
annotations. Besides, we propose an inference-time stage-level beam search
method, which enables effective inference-time scaling. Remarkably, with only
100k training samples and a simple yet effective inference time scaling method,
LLaVA-o1 not only outperforms its base model by 8.9% on a wide range of
multimodal reasoning benchmarks, but also surpasses the performance of larger
and even closed-source models, such as Gemini-1.5-pro, GPT-4o-mini, and
Llama-3.2-90B-Vision-Instruct.
Inconsistencies In Consistency Models: Better ODE Solving Does Not Imply
  Better Samples
Authors: name: Noël Vouitsis, name: Rasa Hosseinzadeh, name: Brendan Leigh Ross, name: Valentin Villecroze, name: Satya Krishna Gorti, name: Jesse C. Cresswell, name: Gabriel Loaiza-Ganem
Published: 2024-11-13 19:00:02

Although diffusion models can generate remarkably high-quality samples, they
are intrinsically bottlenecked by their expensive iterative sampling procedure.
Consistency models (CMs) have recently emerged as a promising diffusion model
distillation method, reducing the cost of sampling by generating high-fidelity
samples in just a few iterations. Consistency model distillation aims to solve
the probability flow ordinary differential equation (ODE) defined by an
existing diffusion model. CMs are not directly trained to minimize error
against an ODE solver, rather they use a more computationally tractable
objective. As a way to study how effectively CMs solve the probability flow
ODE, and the effect that any induced error has on the quality of generated
samples, we introduce Direct CMs, which directly minimize this error.
Intriguingly, we find that Direct CMs reduce the ODE solving error compared to
CMs but also result in significantly worse sample quality, calling into
question why exactly CMs work well in the first place. Full code is available
at: https://github.com/layer6ai-labs/direct-cms.
Hermes: A Large Language Model Framework on the Journey to Autonomous
  Networks
Authors: name: Fadhel Ayed, name: Ali Maatouk, name: Nicola Piovesan, name: Antonio De Domenico, name: Merouane Debbah, name: Zhi-Quan Luo
Published: 2024-11-10 15:12:12

The drive toward automating cellular network operations has grown with the
increasing complexity of these systems. Despite advancements, full autonomy
currently remains out of reach due to reliance on human intervention for
modeling network behaviors and defining policies to meet target requirements.
Network Digital Twins (NDTs) have shown promise in enhancing network
intelligence, but the successful implementation of this technology is
constrained by use case-specific architectures, limiting its role in advancing
network autonomy. A more capable network intelligence, or "telecommunications
brain", is needed to enable seamless, autonomous management of cellular
network. Large Language Models (LLMs) have emerged as potential enablers for
this vision but face challenges in network modeling, especially in reasoning
and handling diverse data types. To address these gaps, we introduce Hermes, a
chain of LLM agents that uses "blueprints" for constructing NDT instances
through structured and explainable logical steps. Hermes allows automatic,
reliable, and accurate network modeling of diverse use cases and
configurations, thus marking progress toward fully autonomous network
operations.
Sharingan: Extract User Action Sequence from Desktop Recordings
Authors: name: Yanting Chen, name: Yi Ren, name: Xiaoting Qin, name: Jue Zhang, name: Kehong Yuan, name: Lu Han, name: Qingwei Lin, name: Dongmei Zhang, name: Saravan Rajmohan, name: Qi Zhang
Published: 2024-11-13 16:53:29

Video recordings of user activities, particularly desktop recordings, offer a
rich source of data for understanding user behaviors and automating processes.
However, despite advancements in Vision-Language Models (VLMs) and their
increasing use in video analysis, extracting user actions from desktop
recordings remains an underexplored area. This paper addresses this gap by
proposing two novel VLM-based methods for user action extraction: the Direct
Frame-Based Approach (DF), which inputs sampled frames directly into VLMs, and
the Differential Frame-Based Approach (DiffF), which incorporates explicit
frame differences detected via computer vision techniques. We evaluate these
methods using a basic self-curated dataset and an advanced benchmark adapted
from prior work. Our results show that the DF approach achieves an accuracy of
70% to 80% in identifying user actions, with the extracted action sequences
being re-playable though Robotic Process Automation. We find that while VLMs
show potential, incorporating explicit UI changes can degrade performance,
making the DF approach more reliable. This work represents the first
application of VLMs for extracting user action sequences from desktop
recordings, contributing new methods, benchmarks, and insights for future
research.
Cut Your Losses in Large-Vocabulary Language Models
Authors: name: Erik Wijmans, name: Brody Huval, name: Alexander Hertzberg, name: Vladlen Koltun, name: Philipp Krähenbühl
Published: 2024-11-13 20:30:15

As language models grow ever larger, so do their vocabularies. This has
shifted the memory footprint of LLMs during training disproportionately to one
single layer: the cross-entropy in the loss computation. Cross-entropy builds
up a logit matrix with entries for each pair of input tokens and vocabulary
items and, for small models, consumes an order of magnitude more memory than
the rest of the LLM combined. We propose Cut Cross-Entropy (CCE), a method that
computes the cross-entropy loss without materializing the logits for all tokens
into global memory. Rather, CCE only computes the logit for the correct token
and evaluates the log-sum-exp over all logits on the fly. We implement a custom
kernel that performs the matrix multiplications and the log-sum-exp reduction
over the vocabulary in flash memory, making global memory consumption for the
cross-entropy computation negligible. This has a dramatic effect. Taking the
Gemma 2 (2B) model as an example, CCE reduces the memory footprint of the loss
computation from 24 GB to 1 MB, and the total training-time memory consumption
of the classifier head from 28 GB to 1 GB. To improve the throughput of CCE, we
leverage the inherent sparsity of softmax and propose to skip elements of the
gradient computation that have a negligible (i.e., below numerical precision)
contribution to the gradient. Experiments demonstrate that the dramatic
reduction in memory consumption is accomplished without sacrificing training
speed or convergence.
LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models
Authors: name: Zhengyi Wang, name: Jonathan Lorraine, name: Yikai Wang, name: Hang Su, name: Jun Zhu, name: Sanja Fidler, name: Xiaohui Zeng
Published: 2024-11-14 17:08:23

This work explores expanding the capabilities of large language models (LLMs)
pretrained on text to generate 3D meshes within a unified model. This offers
key advantages of (1) leveraging spatial knowledge already embedded in LLMs,
derived from textual sources like 3D tutorials, and (2) enabling conversational
3D generation and mesh understanding. A primary challenge is effectively
tokenizing 3D mesh data into discrete tokens that LLMs can process seamlessly.
To address this, we introduce LLaMA-Mesh, a novel approach that represents the
vertex coordinates and face definitions of 3D meshes as plain text, allowing
direct integration with LLMs without expanding the vocabulary. We construct a
supervised fine-tuning (SFT) dataset enabling pretrained LLMs to (1) generate
3D meshes from text prompts, (2) produce interleaved text and 3D mesh outputs
as required, and (3) understand and interpret 3D meshes. Our work is the first
to demonstrate that LLMs can be fine-tuned to acquire complex spatial knowledge
for 3D mesh generation in a text-based format, effectively unifying the 3D and
text modalities. LLaMA-Mesh achieves mesh generation quality on par with models
trained from scratch while maintaining strong text generation performance.
ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical
  Prediction?
Authors: name: Canyu Chen, name: Jian Yu, name: Shan Chen, name: Che Liu, name: Zhongwei Wan, name: Danielle Bitterman, name: Fei Wang, name: Kai Shu
Published: 2024-11-10 14:07:43

Large Language Models (LLMs) hold great promise to revolutionize current
clinical systems for their superior capacities on medical text processing tasks
and medical licensing exams. Meanwhile, traditional ML models such as SVM and
XGBoost have still been mainly adopted in clinical prediction tasks. An
emerging question is Can LLMs beat traditional ML models in clinical
prediction? Thus, we build a new benchmark ClinicalBench to comprehensively
study the clinical predictive modeling capacities of both general-purpose and
medical LLMs, and compare them with traditional ML models. ClinicalBench
embraces three common clinical prediction tasks, two databases, 14
general-purpose LLMs, 8 medical LLMs, and 11 traditional ML models. Through
extensive empirical investigation, we discover that both general-purpose and
medical LLMs, even with different model scales, diverse prompting or
fine-tuning strategies, still cannot beat traditional ML models in clinical
prediction yet, shedding light on their potential deficiency in clinical
reasoning and decision-making. We call for caution when practitioners adopt
LLMs in clinical applications. ClinicalBench can be utilized to bridge the gap
between LLMs' development for healthcare and real-world clinical practice.
MagicQuill: An Intelligent Interactive Image Editing System
Authors: name: Zichen Liu, name: Yue Yu, name: Hao Ouyang, name: Qiuyu Wang, name: Ka Leong Cheng, name: Wen Wang, name: Zhiheng Liu, name: Qifeng Chen, name: Yujun Shen
Published: 2024-11-14 18:59:57

Image editing involves a variety of complex tasks and requires efficient and
precise manipulation techniques. In this paper, we present MagicQuill, an
integrated image editing system that enables swift actualization of creative
ideas. Our system features a streamlined yet functionally robust interface,
allowing for the articulation of editing operations (e.g., inserting elements,
erasing objects, altering color) with minimal input. These interactions are
monitored by a multimodal large language model (MLLM) to anticipate editing
intentions in real time, bypassing the need for explicit prompt entry. Finally,
we apply a powerful diffusion prior, enhanced by a carefully learned two-branch
plug-in module, to process editing requests with precise control. Experimental
results demonstrate the effectiveness of MagicQuill in achieving high-quality
image edits. Please visit https://magic-quill.github.io to try out our system.
PerceiverS: A Multi-Scale Perceiver with Effective Segmentation for
  Long-Term Expressive Symbolic Music Generation
Authors: name: Yungang Yi, name: Weihua Li, name: Matthew Kuo, name: Quan Bai
Published: 2024-11-13 03:14:10

Music generation has progressed significantly, especially in the domain of
audio generation. However, generating symbolic music that is both
long-structured and expressive remains a significant challenge. In this paper,
we propose PerceiverS (Segmentation and Scale), a novel architecture designed
to address this issue by leveraging both Effective Segmentation and Multi-Scale
attention mechanisms. Our approach enhances symbolic music generation by
simultaneously learning long-term structural dependencies and short-term
expressive details. By combining cross-attention and self-attention in a
Multi-Scale setting, PerceiverS captures long-range musical structure while
preserving performance nuances. The proposed model, evaluated on datasets like
Maestro, demonstrates improvements in generating coherent and diverse music
with both structural consistency and expressive variation. The project demos
and the generated music samples can be accessed through the link:
https://perceivers.github.io.
Motion Control for Enhanced Complex Action Video Generation
Authors: name: Qiang Zhou, name: Shaofeng Zhang, name: Nianzu Yang, name: Ye Qian, name: Hao Li
Published: 2024-11-13 04:20:45

Existing text-to-video (T2V) models often struggle with generating videos
with sufficiently pronounced or complex actions. A key limitation lies in the
text prompt's inability to precisely convey intricate motion details. To
address this, we propose a novel framework, MVideo, designed to produce
long-duration videos with precise, fluid actions. MVideo overcomes the
limitations of text prompts by incorporating mask sequences as an additional
motion condition input, providing a clearer, more accurate representation of
intended actions. Leveraging foundational vision models such as GroundingDINO
and SAM2, MVideo automatically generates mask sequences, enhancing both
efficiency and robustness. Our results demonstrate that, after training, MVideo
effectively aligns text prompts with motion conditions to produce videos that
simultaneously meet both criteria. This dual control mechanism allows for more
dynamic video generation by enabling alterations to either the text prompt or
motion condition independently, or both in tandem. Furthermore, MVideo supports
motion condition editing and composition, facilitating the generation of videos
with more complex actions. MVideo thus advances T2V motion generation, setting
a strong benchmark for improved action depiction in current video diffusion
models. Our project page is available at https://mvideo-v1.github.io/.
Can sparse autoencoders be used to decompose and interpret steering
  vectors?
Authors: name: Harry Mayne, name: Yushi Yang, name: Adam Mahdi
Published: 2024-11-13 17:16:48

Steering vectors are a promising approach to control the behaviour of large
language models. However, their underlying mechanisms remain poorly understood.
While sparse autoencoders (SAEs) may offer a potential method to interpret
steering vectors, recent findings show that SAE-reconstructed vectors often
lack the steering properties of the original vectors. This paper investigates
why directly applying SAEs to steering vectors yields misleading
decompositions, identifying two reasons: (1) steering vectors fall outside the
input distribution for which SAEs are designed, and (2) steering vectors can
have meaningful negative projections in feature directions, which SAEs are not
designed to accommodate. These limitations hinder the direct use of SAEs for
interpreting steering vectors.
CamemBERT 2.0: A Smarter French Language Model Aged to Perfection
Authors: name: Wissam Antoun, name: Francis Kulumba, name: Rian Touchent, name: Éric de la Clergerie, name: Benoît Sagot, name: Djamé Seddah
Published: 2024-11-13 18:49:35

French language models, such as CamemBERT, have been widely adopted across
industries for natural language processing (NLP) tasks, with models like
CamemBERT seeing over 4 million downloads per month. However, these models face
challenges due to temporal concept drift, where outdated training data leads to
a decline in performance, especially when encountering new topics and
terminology. This issue emphasizes the need for updated models that reflect
current linguistic trends. In this paper, we introduce two new versions of the
CamemBERT base model-CamemBERTav2 and CamemBERTv2-designed to address these
challenges. CamemBERTav2 is based on the DeBERTaV3 architecture and makes use
of the Replaced Token Detection (RTD) objective for better contextual
understanding, while CamemBERTv2 is built on RoBERTa, which uses the Masked
Language Modeling (MLM) objective. Both models are trained on a significantly
larger and more recent dataset with longer context length and an updated
tokenizer that enhances tokenization performance for French. We evaluate the
performance of these models on both general-domain NLP tasks and
domain-specific applications, such as medical field tasks, demonstrating their
versatility and effectiveness across a range of use cases. Our results show
that these updated models vastly outperform their predecessors, making them
valuable tools for modern NLP systems. All our new models, as well as
intermediate checkpoints, are made openly available on Huggingface.
Large Language Models Can Self-Improve in Long-context Reasoning
Authors: name: Siheng Li, name: Cheng Yang, name: Zesen Cheng, name: Lemao Liu, name: Mo Yu, name: Yujiu Yang, name: Wai Lam
Published: 2024-11-12 19:53:00

Large language models (LLMs) have achieved substantial progress in processing
long contexts but still struggle with long-context reasoning. Existing
approaches typically involve fine-tuning LLMs with synthetic data, which
depends on annotations from human experts or advanced models like GPT-4, thus
restricting further advancements. To address this issue, we investigate the
potential for LLMs to self-improve in long-context reasoning and propose \ours,
an approach specifically designed for this purpose. This approach is
straightforward: we sample multiple outputs for each question, score them with
Minimum Bayes Risk, and then apply supervised fine-tuning or preference
optimization based on these outputs. Extensive experiments on several leading
LLMs demonstrate the effectiveness of \ours, with an absolute improvement of
4.2 points for Llama-3.1-8B-Instruct. Furthermore, \ours achieves superior
performance compared to prior approaches that depend on data produced by human
experts or advanced models. We anticipate that this work will open new avenues
for self-improvement techniques in long-context scenarios, which are essential
for the continual advancement of LLMs.