Orionnnn/hub-stats huggingface dataset:
LongKey: Keyphrase Extraction for Long Documents
Authors: name: Jeovane Honorio Alves, name: Radu State, name: Cinthia Obladen de Almendra Freitas, name: Jean Paul Barddal
Published: 2024-11-26 20:26:47

In an era of information overload, manually annotating the vast and growing
corpus of documents and scholarly papers is increasingly impractical. Automated
keyphrase extraction addresses this challenge by identifying representative
terms within texts. However, most existing methods focus on short documents (up
to 512 tokens), leaving a gap in processing long-context documents. In this
paper, we introduce LongKey, a novel framework for extracting keyphrases from
lengthy documents, which uses an encoder-based language model to capture
extended text intricacies. LongKey uses a max-pooling embedder to enhance
keyphrase candidate representation. Validated on the comprehensive LDKP
datasets and six diverse, unseen datasets, LongKey consistently outperforms
existing unsupervised and language model-based keyphrase extraction methods.
Our findings demonstrate LongKey's versatility and superior performance,
marking an advancement in keyphrase extraction for varied text lengths and
domains.
TryOffDiff: Virtual-Try-Off via High-Fidelity Garment Reconstruction
  using Diffusion Models
Authors: name: Riza Velioglu, name: Petra Bevandic, name: Robin Chan, name: Barbara Hammer
Published: 2024-11-27 13:53:09

This paper introduces Virtual Try-Off (VTOFF), a novel task focused on
generating standardized garment images from single photos of clothed
individuals. Unlike traditional Virtual Try-On (VTON), which digitally dresses
models, VTOFF aims to extract a canonical garment image, posing unique
challenges in capturing garment shape, texture, and intricate patterns. This
well-defined target makes VTOFF particularly effective for evaluating
reconstruction fidelity in generative models. We present TryOffDiff, a model
that adapts Stable Diffusion with SigLIP-based visual conditioning to ensure
high fidelity and detail retention. Experiments on a modified VITON-HD dataset
show that our approach outperforms baseline methods based on pose transfer and
virtual try-on with fewer pre- and post-processing steps. Our analysis reveals
that traditional image generation metrics inadequately assess reconstruction
quality, prompting us to rely on DISTS for more accurate evaluation. Our
results highlight the potential of VTOFF to enhance product imagery in
e-commerce applications, advance generative model evaluation, and inspire
future work on high-fidelity reconstruction. Demo, code, and models are
available at: https://rizavelioglu.github.io/tryoffdiff/
ChatGen: Automatic Text-to-Image Generation From FreeStyle Chatting
Authors: name: Chengyou Jia, name: Changliang Xia, name: Zhuohang Dang, name: Weijia Wu, name: Hangwei Qian, name: Minnan Luo
Published: 2024-11-26 07:31:12

Despite the significant advancements in text-to-image (T2I) generative
models, users often face a trial-and-error challenge in practical scenarios.
This challenge arises from the complexity and uncertainty of tedious steps such
as crafting suitable prompts, selecting appropriate models, and configuring
specific arguments, making users resort to labor-intensive attempts for desired
images. This paper proposes Automatic T2I generation, which aims to automate
these tedious steps, allowing users to simply describe their needs in a
freestyle chatting way. To systematically study this problem, we first
introduce ChatGenBench, a novel benchmark designed for Automatic T2I. It
features high-quality paired data with diverse freestyle inputs, enabling
comprehensive evaluation of automatic T2I models across all steps.
Additionally, recognizing Automatic T2I as a complex multi-step reasoning task,
we propose ChatGen-Evo, a multi-stage evolution strategy that progressively
equips models with essential automation skills. Through extensive evaluation
across step-wise accuracy and image quality, ChatGen-Evo significantly enhances
performance over various baselines. Our evaluation also uncovers valuable
insights for advancing automatic T2I. All our data, code, and models will be
available in https://chengyou-jia.github.io/ChatGen-Home
Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoning
Authors: name: Di Zhang, name: Jingdi Lei, name: Junxian Li, name: Xunzhi Wang, name: Yujie Liu, name: Zonglin Yang, name: Jiatong Li, name: Weida Wang, name: Suorong Yang, name: Jianbo Wu, name: Peng Ye, name: Wanli Ouyang, name: Dongzhan Zhou
Published: 2024-11-27 10:28:57

Vision-language models~(VLMs) have shown remarkable advancements in
multimodal reasoning tasks. However, they still often generate inaccurate or
irrelevant responses due to issues like hallucinated image understandings or
unrefined reasoning paths. To address these challenges, we introduce Critic-V,
a novel framework inspired by the Actor-Critic paradigm to boost the reasoning
capability of VLMs. This framework decouples the reasoning process and critic
process by integrating two independent components: the Reasoner, which
generates reasoning paths based on visual and textual inputs, and the Critic,
which provides constructive critique to refine these paths. In this approach,
the Reasoner generates reasoning responses according to text prompts, which can
evolve iteratively as a policy based on feedback from the Critic. This
interaction process was theoretically driven by a reinforcement learning
framework where the Critic offers natural language critiques instead of scalar
rewards, enabling more nuanced feedback to boost the Reasoner's capability on
complex reasoning tasks. The Critic model is trained using Direct Preference
Optimization (DPO), leveraging a preference dataset of critiques ranked by
Rule-based Reward(RBR) to enhance its critic capabilities. Evaluation results
show that the Critic-V framework significantly outperforms existing methods,
including GPT-4V, on 5 out of 8 benchmarks, especially regarding reasoning
accuracy and efficiency. Combining a dynamic text-based policy for the Reasoner
and constructive feedback from the preference-optimized Critic enables a more
reliable and context-sensitive multimodal reasoning process. Our approach
provides a promising solution to enhance the reliability of VLMs, improving
their performance in real-world reasoning-heavy multimodal applications such as
autonomous driving and embodied intelligence.
Morph: A Motion-free Physics Optimization Framework for Human Motion
  Generation
Authors: name: Zhuo Li, name: Mingshuang Luo, name: Ruibing Hou, name: Xin Zhao, name: Hao Liu, name: Hong Chang, name: Zimo Liu, name: Chen Li
Published: 2024-11-22 14:09:56

Human motion generation plays a vital role in applications such as digital
humans and humanoid robot control. However, most existing approaches disregard
physics constraints, leading to the frequent production of physically
implausible motions with pronounced artifacts such as floating and foot
sliding. In this paper, we propose Morph, a
Motion-free physics optimization framework,
comprising a Motion Generator and a Motion Physics Refinement module, for
enhancing physical plausibility without relying on costly real-world motion
data. Specifically, the Motion Generator is responsible for providing
large-scale synthetic motion data, while the Motion Physics Refinement Module
utilizes these synthetic data to train a motion imitator within a physics
simulator, enforcing physical constraints to project the noisy motions into a
physically-plausible space. These physically refined motions, in turn, are used
to fine-tune the Motion Generator, further enhancing its capability.
Experiments on both text-to-motion and music-to-dance generation tasks
demonstrate that our framework achieves state-of-the-art motion generation
quality while improving physical plausibility drastically.
Diffusion Self-Distillation for Zero-Shot Customized Image Generation
Authors: name: Shengqu Cai, name: Eric Chan, name: Yunzhi Zhang, name: Leonidas Guibas, name: Jiajun Wu, name: Gordon Wetzstein
Published: 2024-11-27 18:58:52

Text-to-image diffusion models produce impressive results but are frustrating
tools for artists who desire fine-grained control. For example, a common use
case is to create images of a specific instance in novel contexts, i.e.,
"identity-preserving generation". This setting, along with many other tasks
(e.g., relighting), is a natural fit for image+text-conditional generative
models. However, there is insufficient high-quality paired data to train such a
model directly. We propose Diffusion Self-Distillation, a method for using a
pre-trained text-to-image model to generate its own dataset for
text-conditioned image-to-image tasks. We first leverage a text-to-image
diffusion model's in-context generation ability to create grids of images and
curate a large paired dataset with the help of a Visual-Language Model. We then
fine-tune the text-to-image model into a text+image-to-image model using the
curated paired dataset. We demonstrate that Diffusion Self-Distillation
outperforms existing zero-shot methods and is competitive with per-instance
tuning techniques on a wide range of identity-preservation generation tasks,
without requiring test-time optimization.
Edit Away and My Face Will not Stay: Personal Biometric Defense against
  Malicious Generative Editing
Authors: name: Hanhui Wang, name: Yihua Zhang, name: Ruizheng Bai, name: Yue Zhao, name: Sijia Liu, name: Zhengzhong Tu
Published: 2024-11-25 18:59:03

Recent advancements in diffusion models have made generative image editing
more accessible, enabling creative edits but raising ethical concerns,
particularly regarding malicious edits to human portraits that threaten privacy
and identity security. Existing protection methods primarily rely on
adversarial perturbations to nullify edits but often fail against diverse
editing requests. We propose FaceLock, a novel approach to portrait protection
that optimizes adversarial perturbations to destroy or significantly alter
biometric information, rendering edited outputs biometrically unrecognizable.
FaceLock integrates facial recognition and visual perception into perturbation
optimization to provide robust protection against various editing attempts. We
also highlight flaws in commonly used evaluation metrics and reveal how they
can be manipulated, emphasizing the need for reliable assessments of
protection. Experiments show FaceLock outperforms baselines in defending
against malicious edits and is robust against purification techniques. Ablation
studies confirm its stability and broad applicability across diffusion-based
editing algorithms. Our work advances biometric defense and sets the foundation
for privacy-preserving practices in image editing. The code is available at:
https://github.com/taco-group/FaceLock.
Video-Guided Foley Sound Generation with Multimodal Controls
Authors: name: Ziyang Chen, name: Prem Seetharaman, name: Bryan Russell, name: Oriol Nieto, name: David Bourgin, name: Andrew Owens, name: Justin Salamon
Published: 2024-11-26 18:59:58

Generating sound effects for videos often requires creating artistic sound
effects that diverge significantly from real-life sources and flexible control
in the sound design. To address this problem, we introduce MultiFoley, a model
designed for video-guided sound generation that supports multimodal
conditioning through text, audio, and video. Given a silent video and a text
prompt, MultiFoley allows users to create clean sounds (e.g., skateboard wheels
spinning without wind noise) or more whimsical sounds (e.g., making a lion's
roar sound like a cat's meow). MultiFoley also allows users to choose reference
audio from sound effects (SFX) libraries or partial videos for conditioning. A
key novelty of our model lies in its joint training on both internet video
datasets with low-quality audio and professional SFX recordings, enabling
high-quality, full-bandwidth (48kHz) audio generation. Through automated
evaluations and human studies, we demonstrate that MultiFoley successfully
generates synchronized high-quality sounds across varied conditional inputs and
outperforms existing methods. Please see our project page for video results:
https://ificl.github.io/MultiFoley/
3D Convex Splatting: Radiance Field Rendering with 3D Smooth Convexes
Authors: name: Jan Held, name: Renaud Vandeghen, name: Abdullah Hamdi, name: Adrien Deliege, name: Anthony Cioppa, name: Silvio Giancola, name: Andrea Vedaldi, name: Bernard Ghanem, name: Marc Van Droogenbroeck
Published: 2024-11-22 14:31:39

Recent advances in radiance field reconstruction, such as 3D Gaussian
Splatting (3DGS), have achieved high-quality novel view synthesis and fast
rendering by representing scenes with compositions of Gaussian primitives.
However, 3D Gaussians present several limitations for scene reconstruction.
Accurately capturing hard edges is challenging without significantly increasing
the number of Gaussians, creating a large memory footprint. Moreover, they
struggle to represent flat surfaces, as they are diffused in space. Without
hand-crafted regularizers, they tend to disperse irregularly around the actual
surface. To circumvent these issues, we introduce a novel method, named 3D
Convex Splatting (3DCS), which leverages 3D smooth convexes as primitives for
modeling geometrically-meaningful radiance fields from multi-view images.
Smooth convex shapes offer greater flexibility than Gaussians, allowing for a
better representation of 3D scenes with hard edges and dense volumes using
fewer primitives. Powered by our efficient CUDA-based rasterizer, 3DCS achieves
superior performance over 3DGS on benchmarks such as Mip-NeRF360, Tanks and
Temples, and Deep Blending. Specifically, our method attains an improvement of
up to 0.81 in PSNR and 0.026 in LPIPS compared to 3DGS while maintaining high
rendering speeds and reducing the number of required primitives. Our results
highlight the potential of 3D Convex Splatting to become the new standard for
high-quality scene reconstruction and novel view synthesis. Project page:
convexsplatting.github.io.
VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video
  Comprehension with Video-Text Duet Interaction Format
Authors: name: Yueqian Wang, name: Xiaojun Meng, name: Yuxuan Wang, name: Jianxin Liang, name: Jiansheng Wei, name: Huishuai Zhang, name: Dongyan Zhao
Published: 2024-11-27 02:15:34

Recent researches on video large language models (VideoLLM) predominantly
focus on model architectures and training datasets, leaving the interaction
format between the user and the model under-explored. In existing works, users
often interact with VideoLLMs by using the entire video and a query as input,
after which the model generates a response. This interaction format constrains
the application of VideoLLMs in scenarios such as live-streaming comprehension
where videos do not end and responses are required in a real-time manner, and
also results in unsatisfactory performance on time-sensitive tasks that
requires localizing video segments. In this paper, we focus on a video-text
duet interaction format. This interaction format is characterized by the
continuous playback of the video, and both the user and the model can insert
their text messages at any position during the video playback. When a text
message ends, the video continues to play, akin to the alternative of two
performers in a duet. We construct MMDuetIT, a video-text training dataset
designed to adapt VideoLLMs to video-text duet interaction format. We also
introduce the Multi-Answer Grounded Video Question Answering (MAGQA) task to
benchmark the real-time response ability of VideoLLMs. Trained on MMDuetIT,
MMDuet demonstrates that adopting the video-text duet interaction format
enables the model to achieve significant improvements in various time-sensitive
tasks (76% CIDEr on YouCook2 dense video captioning, 90\% mAP on QVHighlights
highlight detection and 25% R@0.5 on Charades-STA temporal video grounding)
with minimal training efforts, and also enable VideoLLMs to reply in a
real-time manner as the video plays. Code, data and demo are available at:
https://github.com/yellow-binary-tree/MMDuet.
Adaptive Blind All-in-One Image Restoration
Authors: name: David Serrano-Lozano, name: Luis Herranz, name: Shaolin Su, name: Javier Vazquez-Corral
Published: 2024-11-27 14:58:08

Blind all-in-one image restoration models aim to recover a high-quality image
from an input degraded with unknown distortions. However, these models require
all the possible degradation types to be defined during the training stage
while showing limited generalization to unseen degradations, which limits their
practical application in complex cases. In this paper, we propose a simple but
effective adaptive blind all-in-one restoration (ABAIR) model, which can
address multiple degradations, generalizes well to unseen degradations, and
efficiently incorporate new degradations by training a small fraction of
parameters. First, we train our baseline model on a large dataset of natural
images with multiple synthetic degradations, augmented with a segmentation head
to estimate per-pixel degradation types, resulting in a powerful backbone able
to generalize to a wide range of degradations. Second, we adapt our baseline
model to varying image restoration tasks using independent low-rank adapters.
Third, we learn to adaptively combine adapters to versatile images via a
flexible and lightweight degradation estimator. Our model is both powerful in
handling specific distortions and flexible in adapting to complex tasks, it not
only outperforms the state-of-the-art by a large margin on five- and three-task
IR setups, but also shows improved generalization to unseen degradations and
also composite distortions.
DreamCache: Finetuning-Free Lightweight Personalized Image Generation
  via Feature Caching
Authors: name: Emanuele Aiello, name: Umberto Michieli, name: Diego Valsesia, name: Mete Ozay, name: Enrico Magli
Published: 2024-11-26 15:03:14

Personalized image generation requires text-to-image generative models that
capture the core features of a reference subject to allow for controlled
generation across different contexts. Existing methods face challenges due to
complex training requirements, high inference costs, limited flexibility, or a
combination of these issues. In this paper, we introduce DreamCache, a scalable
approach for efficient and high-quality personalized image generation. By
caching a small number of reference image features from a subset of layers and
a single timestep of the pretrained diffusion denoiser, DreamCache enables
dynamic modulation of the generated image features through lightweight, trained
conditioning adapters. DreamCache achieves state-of-the-art image and text
alignment, utilizing an order of magnitude fewer extra parameters, and is both
more computationally effective and versatile than existing models.
Make-It-Animatable: An Efficient Framework for Authoring Animation-Ready
  3D Characters
Authors: name: Zhiyang Guo, name: Jinxu Xiang, name: Kai Ma, name: Wengang Zhou, name: Houqiang Li, name: Ran Zhang
Published: 2024-11-27 10:18:06

3D characters are essential to modern creative industries, but making them
animatable often demands extensive manual work in tasks like rigging and
skinning. Existing automatic rigging tools face several limitations, including
the necessity for manual annotations, rigid skeleton topologies, and limited
generalization across diverse shapes and poses. An alternative approach is to
generate animatable avatars pre-bound to a rigged template mesh. However, this
method often lacks flexibility and is typically limited to realistic human
shapes. To address these issues, we present Make-It-Animatable, a novel
data-driven method to make any 3D humanoid model ready for character animation
in less than one second, regardless of its shapes and poses. Our unified
framework generates high-quality blend weights, bones, and pose
transformations. By incorporating a particle-based shape autoencoder, our
approach supports various 3D representations, including meshes and 3D Gaussian
splats. Additionally, we employ a coarse-to-fine representation and a
structure-aware modeling strategy to ensure both accuracy and robustness, even
for characters with non-standard skeleton structures. We conducted extensive
experiments to validate our framework's effectiveness. Compared to existing
methods, our approach demonstrates significant improvements in both quality and
speed.
Omegance: A Single Parameter for Various Granularities in
  Diffusion-Based Synthesis
Authors: name: Xinyu Hou, name: Zongsheng Yue, name: Xiaoming Li, name: Chen Change Loy
Published: 2024-11-26 08:23:16

In this work, we introduce a single parameter omega, to effectively
control granularity in diffusion-based synthesis. This parameter is
incorporated during the denoising steps of the diffusion model's reverse
process. Our approach does not require model retraining, architectural
modifications, or additional computational overhead during inference, yet
enables precise control over the level of details in the generated outputs.
Moreover, spatial masks or denoising schedules with varying omega values can
be applied to achieve region-specific or timestep-specific granularity control.
Prior knowledge of image composition from control signals or reference images
further facilitates the creation of precise omega masks for granularity
control on specific objects. To highlight the parameter's role in controlling
subtle detail variations, the technique is named Omegance, combining "omega"
and "nuance". Our method demonstrates impressive performance across various
image and video synthesis tasks and is adaptable to advanced diffusion models.
The code is available at https://github.com/itsmag11/Omegance.
ChatRex: Taming Multimodal LLM for Joint Perception and Understanding
Authors: name: Qing Jiang, name: Gen luo, name: Yuqin Yang, name: Yuda Xiong, name: Yihao Chen, name: Zhaoyang Zeng, name: Tianhe Ren, name: Lei Zhang
Published: 2024-11-27 14:11:10

Perception and understanding are two pillars of computer vision. While
multimodal large language models (MLLM) have demonstrated remarkable visual
understanding capabilities, they arguably lack accurate perception abilities,
e.g. the stage-of-the-art model Qwen2-VL only achieves a 43.9 recall rate on
the COCO dataset, limiting many tasks requiring the combination of perception
and understanding. In this work, we aim to bridge this perception gap from both
model designing and data development perspectives. We first introduce ChatRex,
an MLLM with a decoupled perception design. Instead of having the LLM directly
predict box coordinates, we feed the output boxes from a universal proposal
network into the LLM, allowing it to output the corresponding box indices to
represent its detection results, turning the regression task into a
retrieval-based task that LLM handles more proficiently. From the data
perspective, we build a fully automated data engine and construct the
Rexverse-2M dataset which possesses multiple granularities to support the joint
training of perception and understanding. After standard two-stage training,
ChatRex demonstrates strong perception capabilities while preserving multimodal
understanding performance. The combination of these two capabilities
simultaneously unlocks many attractive applications, demonstrating the
complementary roles of both perception and understanding in MLLM. Code is
available at https://github.com/IDEA-Research/ChatRex.