Orionnnn/hub-stats huggingface dataset:
Large Language Model-Brained GUI Agents: A Survey
Authors: name: Chaoyun Zhang, name: Shilin He, name: Jiaxu Qian, name: Bowen Li, name: Liqun Li, name: Si Qin, name: Yu Kang, name: Minghua Ma, name: Qingwei Lin, name: Saravan Rajmohan, name: Dongmei Zhang, name: Qi Zhang
Published: 2024-11-27 12:13:39

GUIs have long been central to human-computer interaction, providing an
intuitive and visually-driven way to access and interact with digital systems.
The advent of LLMs, particularly multimodal models, has ushered in a new era of
GUI automation. They have demonstrated exceptional capabilities in natural
language understanding, code generation, and visual processing. This has paved
the way for a new generation of LLM-brained GUI agents capable of interpreting
complex GUI elements and autonomously executing actions based on natural
language instructions. These agents represent a paradigm shift, enabling users
to perform intricate, multi-step tasks through simple conversational commands.
Their applications span across web navigation, mobile app interactions, and
desktop automation, offering a transformative user experience that
revolutionizes how individuals interact with software. This emerging field is
rapidly advancing, with significant progress in both research and industry.
  To provide a structured understanding of this trend, this paper presents a
comprehensive survey of LLM-brained GUI agents, exploring their historical
evolution, core components, and advanced techniques. We address research
questions such as existing GUI agent frameworks, the collection and utilization
of data for training specialized GUI agents, the development of large action
models tailored for GUI tasks, and the evaluation metrics and benchmarks
necessary to assess their effectiveness. Additionally, we examine emerging
applications powered by these agents. Through a detailed analysis, this survey
identifies key research gaps and outlines a roadmap for future advancements in
the field. By consolidating foundational knowledge and state-of-the-art
developments, this work aims to guide both researchers and practitioners in
overcoming challenges and unlocking the full potential of LLM-brained GUI
agents.
UniPose: A Unified Multimodal Framework for Human Pose Comprehension,
  Generation and Editing
Authors: name: Yiheng Li, name: Ruibing Hou, name: Hong Chang, name: Shiguang Shan, name: Xilin Chen
Published: 2024-11-25 08:06:30

Human pose plays a crucial role in the digital age. While recent works have
achieved impressive progress in understanding and generating human poses, they
often support only a single modality of control signals and operate in
isolation, limiting their application in real-world scenarios. This paper
presents UniPose, a framework employing Large Language Models (LLMs) to
comprehend, generate, and edit human poses across various modalities, including
images, text, and 3D SMPL poses. Specifically, we apply a pose tokenizer to
convert 3D poses into discrete pose tokens, enabling seamless integration into
the LLM within a unified vocabulary. To further enhance the fine-grained pose
perception capabilities, we facilitate UniPose with a mixture of visual
encoders, among them a pose-specific visual encoder. Benefiting from a unified
learning strategy, UniPose effectively transfers knowledge across different
pose-relevant tasks, adapts to unseen tasks, and exhibits extended
capabilities. This work serves as the first attempt at building a
general-purpose framework for pose comprehension, generation, and editing.
Extensive experiments highlight UniPose's competitive and even superior
performance across various pose-relevant tasks.
Optimizing Brain Tumor Segmentation with MedNeXt: BraTS 2024 SSA and
  Pediatrics
Authors: name: Sarim Hashmi, name: Juan Lugo, name: Abdelrahman Elsayed, name: Dinesh Saggurthi, name: Mohammed Elseiagy, name: Alikhan Nurkamal, name: Jaskaran Walia, name: Fadillah Adamsyah Maani, name: Mohammad Yaqub
Published: 2024-11-24 15:19:19

Identifying key pathological features in brain MRIs is crucial for the
long-term survival of glioma patients. However, manual segmentation is
time-consuming, requiring expert intervention and is susceptible to human
error. Therefore, significant research has been devoted to developing machine
learning methods that can accurately segment tumors in 3D multimodal brain MRI
scans. Despite their progress, state-of-the-art models are often limited by the
data they are trained on, raising concerns about their reliability when applied
to diverse populations that may introduce distribution shifts. Such shifts can
stem from lower quality MRI technology (e.g., in sub-Saharan Africa) or
variations in patient demographics (e.g., children). The BraTS-2024 challenge
provides a platform to address these issues. This study presents our
methodology for segmenting tumors in the BraTS-2024 SSA and Pediatric Tumors
tasks using MedNeXt, comprehensive model ensembling, and thorough
postprocessing. Our approach demonstrated strong performance on the unseen
validation set, achieving an average Dice Similarity Coefficient (DSC) of 0.896
on the BraTS-2024 SSA dataset and an average DSC of 0.830 on the BraTS
Pediatric Tumor dataset. Additionally, our method achieved an average Hausdorff
Distance (HD95) of 14.682 on the BraTS-2024 SSA dataset and an average HD95 of
37.508 on the BraTS Pediatric dataset. Our GitHub repository can be accessed
here: Project Repository :
https://github.com/python-arch/BioMbz-Optimizing-Brain-Tumor-Segmentation-with-MedNeXt-BraTS-2024-SSA-and-Pediatrics
Training and Evaluating Language Models with Template-based Data
  Generation
Authors: name: Yifan Zhang
Published: 2024-11-27 07:32:56

The rapid advancement of large language models (LLMs) such as GPT-3, PaLM,
and Llama has significantly transformed natural language processing, showcasing
remarkable capabilities in understanding and generating language. However,
these models often struggle with tasks requiring complex reasoning,
particularly in mathematical problem-solving, due in part to the scarcity of
large-scale, high-quality, domain-specific datasets necessary for training
sophisticated reasoning abilities. To address this limitation, we introduce
Template-based Data Generation (TDG), a novel approach that leverages LLMs
(GPT-4) to automatically generate parameterized meta-templates, which are then
used to synthesize a vast array of high-quality problems and solutions.
Leveraging TDG, we create TemplateMath Part I: TemplateGSM, a dataset
comprising over 7 million synthetically generated grade school math
problems--each accompanied by code-based and natural language solutions--with
the potential to generate an effectively unlimited number more. This dataset
alleviates the scarcity of large-scale mathematical datasets and serves as a
valuable resource for pre-training, fine-tuning, and evaluating LLMs in
mathematical reasoning. Our method not only enables the generation of virtually
infinite data but also elevates data augmentation to a new level by using GPT-4
for meta-template generation, ensuring diverse and high-quality problem
structures. The TemplateMath Part I: TemplateGSM dataset is publicly available
at https://huggingface.co/datasets/math-ai/TemplateGSM. The code is available
at https://github.com/iiis-ai/TemplateMath.
CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models
Authors: name: Rundi Wu, name: Ruiqi Gao, name: Ben Poole, name: Alex Trevithick, name: Changxi Zheng, name: Jonathan T. Barron, name: Aleksander Holynski
Published: 2024-11-27 18:57:16

We present CAT4D, a method for creating 4D (dynamic 3D) scenes from monocular
video. CAT4D leverages a multi-view video diffusion model trained on a diverse
combination of datasets to enable novel view synthesis at any specified camera
poses and timestamps. Combined with a novel sampling approach, this model can
transform a single monocular video into a multi-view video, enabling robust 4D
reconstruction via optimization of a deformable 3D Gaussian representation. We
demonstrate competitive performance on novel view synthesis and dynamic scene
reconstruction benchmarks, and highlight the creative capabilities for 4D scene
generation from real or generated videos. See our project page for results and
interactive demos: cat-4d.github.io.
DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous
  Driving
Authors: name: Bencheng Liao, name: Shaoyu Chen, name: Haoran Yin, name: Bo Jiang, name: Cheng Wang, name: Sixu Yan, name: Xinbang Zhang, name: Xiangyu Li, name: Ying Zhang, name: Qian Zhang, name: Xinggang Wang
Published: 2024-11-22 18:59:47

Recently, the diffusion model has emerged as a powerful generative technique
for robotic policy learning, capable of modeling multi-mode action
distributions. Leveraging its capability for end-to-end autonomous driving is a
promising direction. However, the numerous denoising steps in the robotic
diffusion policy and the more dynamic, open-world nature of traffic scenes pose
substantial challenges for generating diverse driving actions at a real-time
speed. To address these challenges, we propose a novel truncated diffusion
policy that incorporates prior multi-mode anchors and truncates the diffusion
schedule, enabling the model to learn denoising from anchored Gaussian
distribution to the multi-mode driving action distribution. Additionally, we
design an efficient cascade diffusion decoder for enhanced interaction with
conditional scene context. The proposed model, DiffusionDrive, demonstrates
10times reduction in denoising steps compared to vanilla diffusion policy,
delivering superior diversity and quality in just 2 steps. On the
planning-oriented NAVSIM dataset, with the aligned ResNet-34 backbone,
DiffusionDrive achieves 88.1 PDMS without bells and whistles, setting a new
record, while running at a real-time speed of 45 FPS on an NVIDIA 4090.
Qualitative results on challenging scenarios further confirm that
DiffusionDrive can robustly generate diverse plausible driving actions. Code
and model will be available at https://github.com/hustvl/DiffusionDrive.
Draft Model Knows When to Stop: A Self-Verification Length Policy for
  Speculative Decoding
Authors: name: Ziyin Zhang, name: Jiahao Xu, name: Tian Liang, name: Xingyu Chen, name: Zhiwei He, name: Rui Wang, name: Zhaopeng Tu
Published: 2024-11-27 15:53:17

Speculative Decoding (SD) has become an important technique in accelerating
the inference speed of large language models. Conventional SD methods employ a
fixed draft length, which ignores the token generation difficulty across tasks.
Consequently, in this paper, we address such an issue and introduce SVIP - a
difficulty-aware dynamic draft length policy for speculative decoding systems.
Based on a theoretical lower bound of draft token acceptance rate and its
inference-time approximation, SVIP adaptively determines the lengths of draft
sequences based on the entropy of each draft token distribution. Experimental
results on mainstream SD benchmarks and frameworks demonstrate the superior
performance of SVIP, achieving up to 20\% walltime speedup on SpecBench over
baseline SD methods and 60\% speedup on MT-Bench for long-form generation of up
to 8K tokens. Moreover, SVIP is totally training-free and compatible with any
existing SD methods that generate draft tokens autoregressively. Experimental
results also show that SVIP yields consistent walltime improvement on top of
GliDe & CaPE and EAGLE-2.
Collaborative Decoding Makes Visual Auto-Regressive Modeling Efficient
Authors: name: Zigeng Chen, name: Xinyin Ma, name: Gongfan Fang, name: Xinchao Wang
Published: 2024-11-26 15:13:15

In the rapidly advancing field of image generation, Visual Auto-Regressive
(VAR) modeling has garnered considerable attention for its innovative
next-scale prediction approach. This paradigm offers substantial improvements
in efficiency, scalability, and zero-shot generalization. Yet, the inherently
coarse-to-fine nature of VAR introduces a prolonged token sequence, leading to
prohibitive memory consumption and computational redundancies. To address these
bottlenecks, we propose Collaborative Decoding (CoDe), a novel efficient
decoding strategy tailored for the VAR framework. CoDe capitalizes on two
critical observations: the substantially reduced parameter demands at larger
scales and the exclusive generation patterns across different scales. Based on
these insights, we partition the multi-scale inference process into a seamless
collaboration between a large model and a small model. The large model serves
as the 'drafter', specializing in generating low-frequency content at smaller
scales, while the smaller model serves as the 'refiner', solely focusing on
predicting high-frequency details at larger scales. This collaboration yields
remarkable efficiency with minimal impact on quality: CoDe achieves a 1.7x
speedup, slashes memory usage by around 50%, and preserves image quality with
only a negligible FID increase from 1.95 to 1.98. When drafting steps are
further decreased, CoDe can achieve an impressive 2.9x acceleration ratio,
reaching 41 images/s at 256x256 resolution on a single NVIDIA 4090 GPU, while
preserving a commendable FID of 2.27. The code is available at
https://github.com/czg1225/CoDe
Interleaved Scene Graph for Interleaved Text-and-Image Generation
  Assessment
Authors: name: Dongping Chen, name: Ruoxi Chen, name: Shu Pu, name: Zhaoyi Liu, name: Yanru Wu, name: Caixi Chen, name: Benlin Liu, name: Yue Huang, name: Yao Wan, name: Pan Zhou, name: Ranjay Krishna
Published: 2024-11-26 07:55:57

Many real-world user queries (e.g. "How do to make egg fried rice?") could
benefit from systems capable of generating responses with both textual steps
with accompanying images, similar to a cookbook. Models designed to generate
interleaved text and images face challenges in ensuring consistency within and
across these modalities. To address these challenges, we present ISG, a
comprehensive evaluation framework for interleaved text-and-image generation.
ISG leverages a scene graph structure to capture relationships between text and
image blocks, evaluating responses on four levels of granularity: holistic,
structural, block-level, and image-specific. This multi-tiered evaluation
allows for a nuanced assessment of consistency, coherence, and accuracy, and
provides interpretable question-answer feedback. In conjunction with ISG, we
introduce a benchmark, ISG-Bench, encompassing 1,150 samples across 8
categories and 21 subcategories. This benchmark dataset includes complex
language-vision dependencies and golden answers to evaluate models effectively
on vision-centric tasks such as style transfer, a challenging area for current
models. Using ISG-Bench, we demonstrate that recent unified vision-language
models perform poorly on generating interleaved content. While compositional
approaches that combine separate language and image models show a 111%
improvement over unified models at the holistic level, their performance
remains suboptimal at both block and image levels. To facilitate future work,
we develop ISG-Agent, a baseline agent employing a "plan-execute-refine"
pipeline to invoke tools, achieving a 122% performance improvement.
ROICtrl: Boosting Instance Control for Visual Generation
Authors: name: Yuchao Gu, name: Yipin Zhou, name: Yunfan Ye, name: Yixin Nie, name: Licheng Yu, name: Pingchuan Ma, name: Kevin Qinghong Lin, name: Mike Zheng Shou
Published: 2024-11-27 00:10:27

Natural language often struggles to accurately associate positional and
attribute information with multiple instances, which limits current text-based
visual generation models to simpler compositions featuring only a few dominant
instances. To address this limitation, this work enhances diffusion models by
introducing regional instance control, where each instance is governed by a
bounding box paired with a free-form caption. Previous methods in this area
typically rely on implicit position encoding or explicit attention masks to
separate regions of interest (ROIs), resulting in either inaccurate coordinate
injection or large computational overhead. Inspired by ROI-Align in object
detection, we introduce a complementary operation called ROI-Unpool. Together,
ROI-Align and ROI-Unpool enable explicit, efficient, and accurate ROI
manipulation on high-resolution feature maps for visual generation. Building on
ROI-Unpool, we propose ROICtrl, an adapter for pretrained diffusion models that
enables precise regional instance control. ROICtrl is compatible with
community-finetuned diffusion models, as well as with existing spatial-based
add-ons (\eg, ControlNet, T2I-Adapter) and embedding-based add-ons (\eg,
IP-Adapter, ED-LoRA), extending their applications to multi-instance
generation. Experiments show that ROICtrl achieves superior performance in
regional instance control while significantly reducing computational costs.
Identity-Preserving Text-to-Video Generation by Frequency Decomposition
Authors: name: Shenghai Yuan, name: Jinfa Huang, name: Xianyi He, name: Yunyuan Ge, name: Yujun Shi, name: Liuhan Chen, name: Jiebo Luo, name: Li Yuan
Published: 2024-11-26 13:58:24

Identity-preserving text-to-video (IPT2V) generation aims to create
high-fidelity videos with consistent human identity. It is an important task in
video generation but remains an open problem for generative models. This paper
pushes the technical frontier of IPT2V in two directions that have not been
resolved in literature: (1) A tuning-free pipeline without tedious case-by-case
finetuning, and (2) A frequency-aware heuristic identity-preserving DiT-based
control scheme. We propose ConsisID, a tuning-free DiT-based controllable IPT2V
model to keep human identity consistent in the generated video. Inspired by
prior findings in frequency analysis of diffusion transformers, it employs
identity-control signals in the frequency domain, where facial features can be
decomposed into low-frequency global features and high-frequency intrinsic
features. First, from a low-frequency perspective, we introduce a global facial
extractor, which encodes reference images and facial key points into a latent
space, generating features enriched with low-frequency information. These
features are then integrated into shallow layers of the network to alleviate
training challenges associated with DiT. Second, from a high-frequency
perspective, we design a local facial extractor to capture high-frequency
details and inject them into transformer blocks, enhancing the model's ability
to preserve fine-grained features. We propose a hierarchical training strategy
to leverage frequency information for identity preservation, transforming a
vanilla pre-trained video generation model into an IPT2V model. Extensive
experiments demonstrate that our frequency-aware heuristic scheme provides an
optimal control solution for DiT-based models. Thanks to this scheme, our
ConsisID generates high-quality, identity-preserving videos, making strides
towards more effective IPT2V.
MARVEL-40M+: Multi-Level Visual Elaboration for High-Fidelity Text-to-3D
  Content Creation
Authors: name: Sankalp Sinha, name: Mohammad Sadil Khan, name: Muhammad Usama, name: Shino Sam, name: Didier Stricker, name: Sk Aziz Ali, name: Muhammad Zeshan Afzal
Published: 2024-11-26 23:39:43

Generating high-fidelity 3D content from text prompts remains a significant
challenge in computer vision due to the limited size, diversity, and annotation
depth of the existing datasets. To address this, we introduce MARVEL-40M+, an
extensive dataset with 40 million text annotations for over 8.9 million 3D
assets aggregated from seven major 3D datasets. Our contribution is a novel
multi-stage annotation pipeline that integrates open-source pretrained
multi-view VLMs and LLMs to automatically produce multi-level descriptions,
ranging from detailed (150-200 words) to concise semantic tags (10-20 words).
This structure supports both fine-grained 3D reconstruction and rapid
prototyping. Furthermore, we incorporate human metadata from source datasets
into our annotation pipeline to add domain-specific information in our
annotation and reduce VLM hallucinations. Additionally, we develop MARVEL-FX3D,
a two-stage text-to-3D pipeline. We fine-tune Stable Diffusion with our
annotations and use a pretrained image-to-3D network to generate 3D textured
meshes within 15s. Extensive evaluations show that MARVEL-40M+ significantly
outperforms existing datasets in annotation quality and linguistic diversity,
achieving win rates of 72.41% by GPT-4 and 73.40% by human evaluators.
EfficientViM: Efficient Vision Mamba with Hidden State Mixer based State
  Space Duality
Authors: name: Sanghyeok Lee, name: Joonmyung Choi, name: Hyunwoo J. Kim
Published: 2024-11-22 02:02:06

For the deployment of neural networks in resource-constrained environments,
prior works have built lightweight architectures with convolution and attention
for capturing local and global dependencies, respectively. Recently, the state
space model has emerged as an effective global token interaction with its
favorable linear computational cost in the number of tokens. Yet, efficient
vision backbones built with SSM have been explored less. In this paper, we
introduce Efficient Vision Mamba (EfficientViM), a novel architecture built on
hidden state mixer-based state space duality (HSM-SSD) that efficiently
captures global dependencies with further reduced computational cost. In the
HSM-SSD layer, we redesign the previous SSD layer to enable the channel mixing
operation within hidden states. Additionally, we propose multi-stage hidden
state fusion to further reinforce the representation power of hidden states,
and provide the design alleviating the bottleneck caused by the memory-bound
operations. As a result, the EfficientViM family achieves a new
state-of-the-art speed-accuracy trade-off on ImageNet-1k, offering up to a 0.7%
performance improvement over the second-best model SHViT with faster speed.
Further, we observe significant improvements in throughput and accuracy
compared to prior works, when scaling images or employing distillation
training. Code is available at https://github.com/mlvlab/EfficientViM.
Controllable Human Image Generation with Personalized Multi-Garments
Authors: name: Yisol Choi, name: Sangkyung Kwak, name: Sihyun Yu, name: Hyungwon Choi, name: Jinwoo Shin
Published: 2024-11-25 12:37:13

We present BootComp, a novel framework based on text-to-image diffusion
models for controllable human image generation with multiple reference
garments. Here, the main bottleneck is data acquisition for training:
collecting a large-scale dataset of high-quality reference garment images per
human subject is quite challenging, i.e., ideally, one needs to manually gather
every single garment photograph worn by each human. To address this, we propose
a data generation pipeline to construct a large synthetic dataset, consisting
of human and multiple-garment pairs, by introducing a model to extract any
reference garment images from each human image. To ensure data quality, we also
propose a filtering strategy to remove undesirable generated data based on
measuring perceptual similarities between the garment presented in human image
and extracted garment. Finally, by utilizing the constructed synthetic dataset,
we train a diffusion model having two parallel denoising paths that use
multiple garment images as conditions to generate human images while preserving
their fine-grained details. We further show the wide-applicability of our
framework by adapting it to different types of reference-based generation in
the fashion domain, including virtual try-on, and controllable human image
generation with other conditions, e.g., pose, face, etc.