Orionnnn/hub-stats huggingface dataset:
HeadGAP: Few-shot 3D Head Avatar via Generalizable Gaussian Priors
Authors: name: Xiaozheng Zheng, name: Chao Wen, name: Zhaohu Li, name: Weiyi Zhang, name: Zhuo Su, name: Xu Chang, name: Yang Zhao, name: Zheng Lv, name: Xiaoyuan Zhang, name: Yongjie Zhang, name: Guidong Wang, name: Lan Xu
Published: 2024-08-12 09:19:38

In this paper, we present a novel 3D head avatar creation approach capable of
generalizing from few-shot in-the-wild data with high-fidelity and animatable
robustness. Given the underconstrained nature of this problem, incorporating
prior knowledge is essential. Therefore, we propose a framework comprising
prior learning and avatar creation phases. The prior learning phase leverages
3D head priors derived from a large-scale multi-view dynamic dataset, and the
avatar creation phase applies these priors for few-shot personalization. Our
approach effectively captures these priors by utilizing a Gaussian
Splatting-based auto-decoder network with part-based dynamic modeling. Our
method employs identity-shared encoding with personalized latent codes for
individual identities to learn the attributes of Gaussian primitives. During
the avatar creation phase, we achieve fast head avatar personalization by
leveraging inversion and fine-tuning strategies. Extensive experiments
demonstrate that our model effectively exploits head priors and successfully
generalizes them to few-shot personalization, achieving photo-realistic
rendering quality, multi-view consistency, and stable animation.
UniPortrait: A Unified Framework for Identity-Preserving Single- and
  Multi-Human Image Personalization
Authors: name: Junjie He, name: Yifeng Geng, name: Liefeng Bo
Published: 2024-08-12 06:27:29

This paper presents UniPortrait, an innovative human image personalization
framework that unifies single- and multi-ID customization with high face
fidelity, extensive facial editability, free-form input description, and
diverse layout generation. UniPortrait consists of only two plug-and-play
modules: an ID embedding module and an ID routing module. The ID embedding
module extracts versatile editable facial features with a decoupling strategy
for each ID and embeds them into the context space of diffusion models. The ID
routing module then combines and distributes these embeddings adaptively to
their respective regions within the synthesized image, achieving the
customization of single and multiple IDs. With a carefully designed two-stage
training scheme, UniPortrait achieves superior performance in both single- and
multi-ID customization. Quantitative and qualitative experiments demonstrate
the advantages of our method over existing approaches as well as its good
scalability, e.g., the universal compatibility with existing generative control
tools. The project page is at
https://aigcdesigngroup.github.io/UniPortrait-Page/ .
ControlNeXt: Powerful and Efficient Control for Image and Video
  Generation
Authors: name: Bohao Peng, name: Jian Wang, name: Yuechen Zhang, name: Wenbo Li, name: Ming-Chang Yang, name: Jiaya Jia
Published: 2024-08-12 11:41:18

Diffusion models have demonstrated remarkable and robust abilities in both
image and video generation. To achieve greater control over generated results,
researchers introduce additional architectures, such as ControlNet, Adapters
and ReferenceNet, to integrate conditioning controls. However, current
controllable generation methods often require substantial additional
computational resources, especially for video generation, and face challenges
in training or exhibit weak control. In this paper, we propose ControlNeXt: a
powerful and efficient method for controllable image and video generation. We
first design a more straightforward and efficient architecture, replacing heavy
additional branches with minimal additional cost compared to the base model.
Such a concise structure also allows our method to seamlessly integrate with
other LoRA weights, enabling style alteration without the need for additional
training. As for training, we reduce up to 90% of learnable parameters compared
to the alternatives. Furthermore, we propose another method called Cross
Normalization (CN) as a replacement for Zero-Convolution' to achieve fast and
stable training convergence. We have conducted various experiments with
different base models across images and videos, demonstrating the robustness of
our method.
The AI Scientist: Towards Fully Automated Open-Ended Scientific
  Discovery
Authors: name: Chris Lu, name: Cong Lu, name: Robert Tjarko Lange, name: Jakob Foerster, name: Jeff Clune, name: David Ha
Published: 2024-08-12 16:58:11

One of the grand challenges of artificial general intelligence is developing
agents capable of conducting scientific research and discovering new knowledge.
While frontier models have already been used as aids to human scientists, e.g.
for brainstorming ideas, writing code, or prediction tasks, they still conduct
only a small part of the scientific process. This paper presents the first
comprehensive framework for fully automatic scientific discovery, enabling
frontier large language models to perform research independently and
communicate their findings. We introduce The AI Scientist, which generates
novel research ideas, writes code, executes experiments, visualizes results,
describes its findings by writing a full scientific paper, and then runs a
simulated review process for evaluation. In principle, this process can be
repeated to iteratively develop ideas in an open-ended fashion, acting like the
human scientific community. We demonstrate its versatility by applying it to
three distinct subfields of machine learning: diffusion modeling,
transformer-based language modeling, and learning dynamics. Each idea is
implemented and developed into a full paper at a cost of less than $15 per
paper. To evaluate the generated papers, we design and validate an automated
reviewer, which we show achieves near-human performance in evaluating paper
scores. The AI Scientist can produce papers that exceed the acceptance
threshold at a top machine learning conference as judged by our automated
reviewer. This approach signifies the beginning of a new era in scientific
discovery in machine learning: bringing the transformative benefits of AI
agents to the entire research process of AI itself, and taking us closer to a
world where endless affordable creativity and innovation can be unleashed on
the world's most challenging problems. Our code is open-sourced at
https://github.com/SakanaAI/AI-Scientist
BRAT: Bonus oRthogonAl Token for Architecture Agnostic Textual Inversion
Authors: name: James Baker
Published: 2024-08-08 23:04:26

Textual Inversion remains a popular method for personalizing diffusion
models, in order to teach models new subjects and styles. We note that textual
inversion has been underexplored using alternatives to the UNet, and experiment
with textual inversion with a vision transformer. We also seek to optimize
textual inversion using a strategy that does not require explicit use of the
UNet and its idiosyncratic layers, so we add bonus tokens and enforce
orthogonality. We find the use of the bonus token improves adherence to the
source images and the use of the vision transformer improves adherence to the
prompt. Code is available at https://github.com/jamesBaker361/tex_inv_plus.
Generating novel experimental hypotheses from language models: A case
  study on cross-dative generalization
Authors: name: Kanishka Misra, name: Najoung Kim
Published: 2024-08-09 14:17:36

Neural network language models (LMs) have been shown to successfully capture
complex linguistic knowledge. However, their utility for understanding language
acquisition is still debated. We contribute to this debate by presenting a case
study where we use LMs as simulated learners to derive novel experimental
hypotheses to be tested with humans. We apply this paradigm to study
cross-dative generalization (CDG): productive generalization of novel verbs
across dative constructions (she pilked me the ball/she pilked the ball to me)
-- acquisition of which is known to involve a large space of contextual
features -- using LMs trained on child-directed speech. We specifically ask:
"what properties of the training exposure facilitate a novel verb's
generalization to the (unmodeled) alternate construction?" To answer this, we
systematically vary the exposure context in which a novel dative verb occurs in
terms of the properties of the theme and recipient, and then analyze the LMs'
usage of the novel verb in the unmodeled dative construction. We find LMs to
replicate known patterns of children's CDG, as a precondition to exploring
novel hypotheses. Subsequent simulations reveal a nuanced role of the features
of the novel verbs' exposure context on the LMs' CDG. We find CDG to be
facilitated when the first postverbal argument of the exposure context is
pronominal, definite, short, and conforms to the prototypical animacy
expectations of the exposure dative. These patterns are characteristic of
harmonic alignment in datives, where the argument with features ranking higher
on the discourse prominence scale tends to precede the other. This gives rise
to a novel hypothesis that CDG is facilitated insofar as the features of the
exposure context -- in particular, its first postverbal argument -- are
harmonically aligned. We conclude by proposing future experiments that can test
this hypothesis in children.
MooER: LLM-based Speech Recognition and Translation Models from Moore
  Threads
Authors: name: Junhao Xu, name: Zhenlin Liang, name: Yi Liu, name: Yichao Hu, name: Jian Li, name: Yajun Zheng, name: Meng Cai, name: Hua Wang
Published: 2024-08-09 14:43:56

In this paper, we present MooER, a LLM-based large-scale automatic speech
recognition (ASR) / automatic speech translation (AST) model of Moore Threads.
A 5000h pseudo labeled dataset containing open source and self collected speech
data is used for training. We achieve performance comparable to other open
source models trained with up to hundreds of thousands of hours of labeled
speech data. Meanwhile, experiments conducted on Covost2 Zh2en testset suggest
that our model outperforms other open source Speech LLMs. A BLEU score of 25.2
can be obtained. The main contributions of this paper are summarized as
follows. First, this paper presents a training strategy for encoders and LLMs
on speech related tasks (including ASR and AST) using a small size of pseudo
labeled data without any extra manual annotation and selection. Second, we
release our ASR and AST models and plan to open-source our training code and
strategy in the near future. Moreover, a model trained on 8wh scale training
data is planned to be released later on.
mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal
  Large Language Models
Authors: name: Jiabo Ye, name: Haiyang Xu, name: Haowei Liu, name: Anwen Hu, name: Ming Yan, name: Qi Qian, name: Ji Zhang, name: Fei Huang, name: Jingren Zhou
Published: 2024-08-09 03:25:42

Multi-modal Large Language Models (MLLMs) have demonstrated remarkable
capabilities in executing instructions for a variety of single-image tasks.
Despite this progress, significant challenges remain in modeling long image
sequences. In this work, we introduce the versatile multi-modal large language
model, mPLUG-Owl3, which enhances the capability for long image-sequence
understanding in scenarios that incorporate retrieved image-text knowledge,
interleaved image-text, and lengthy videos. Specifically, we propose novel
hyper attention blocks to efficiently integrate vision and language into a
common language-guided semantic space, thereby facilitating the processing of
extended multi-image scenarios. Extensive experimental results suggest that
mPLUG-Owl3 achieves state-of-the-art performance among models with a similar
size on single-image, multi-image, and video benchmarks. Moreover, we propose a
challenging long visual sequence evaluation named Distractor Resistance to
assess the ability of models to maintain focus amidst distractions. Finally,
with the proposed architecture, mPLUG-Owl3 demonstrates outstanding performance
on ultra-long visual sequence inputs. We hope that mPLUG-Owl3 can contribute to
the development of more efficient and powerful multimodal large language
models.
VITA: Towards Open-Source Interactive Omni Multimodal LLM
Authors: name: Chaoyou Fu, name: Haojia Lin, name: Zuwei Long, name: Yunhang Shen, name: Meng Zhao, name: Yifan Zhang, name: Xiong Wang, name: Di Yin, name: Long Ma, name: Xiawu Zheng, name: Ran He, name: Rongrong Ji, name: Yunsheng Wu, name: Caifeng Shan, name: Xing Sun
Published: 2024-08-09 17:59:49

The remarkable multimodal capabilities and interactive experience of GPT-4o
underscore their necessity in practical applications, yet open-source models
rarely excel in both areas. In this paper, we introduce VITA, the first-ever
open-source Multimodal Large Language Model (MLLM) adept at simultaneous
processing and analysis of Video, Image, Text, and Audio modalities, and
meanwhile has an advanced multimodal interactive experience. Starting from
Mixtral 8x7B as a language foundation, we expand its Chinese vocabulary
followed by bilingual instruction tuning. We further endow the language model
with visual and audio capabilities through two-stage multi-task learning of
multimodal alignment and instruction tuning. VITA demonstrates robust
foundational capabilities of multilingual, vision, and audio understanding, as
evidenced by its strong performance across a range of both unimodal and
multimodal benchmarks. Beyond foundational capabilities, we have made
considerable progress in enhancing the natural multimodal human-computer
interaction experience. To the best of our knowledge, we are the first to
exploit non-awakening interaction and audio interrupt in MLLM. VITA is the
first step for the open-source community to explore the seamless integration of
multimodal understanding and interaction. While there is still lots of work to
be done on VITA to get close to close-source counterparts, we hope that its
role as a pioneer can serve as a cornerstone for subsequent research. Project
Page: https://vita-home.github.io.
Kalman-Inspired Feature Propagation for Video Face Super-Resolution
Authors: name: Ruicheng Feng, name: Chongyi Li, name: Chen Change Loy
Published: 2024-08-09 17:57:12

Despite the promising progress of face image super-resolution, video face
super-resolution remains relatively under-explored. Existing approaches either
adapt general video super-resolution networks to face datasets or apply
established face image super-resolution models independently on individual
video frames. These paradigms encounter challenges either in reconstructing
facial details or maintaining temporal consistency. To address these issues, we
introduce a novel framework called Kalman-inspired Feature Propagation (KEEP),
designed to maintain a stable face prior over time. The Kalman filtering
principles offer our method a recurrent ability to use the information from
previously restored frames to guide and regulate the restoration process of the
current frame. Extensive experiments demonstrate the effectiveness of our
method in capturing facial details consistently across video frames. Code and
video demo are available at https://jnjaby.github.io/projects/KEEP.
Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2
Authors: name: Tom Lieberum, name: Senthooran Rajamanoharan, name: Arthur Conmy, name: Lewis Smith, name: Nicolas Sonnerat, name: Vikrant Varma, name: János Kramár, name: Anca Dragan, name: Rohin Shah, name: Neel Nanda
Published: 2024-08-09 16:06:42

Sparse autoencoders (SAEs) are an unsupervised method for learning a sparse
decomposition of a neural network's latent representations into seemingly
interpretable features. Despite recent excitement about their potential,
research applications outside of industry are limited by the high cost of
training a comprehensive suite of SAEs. In this work, we introduce Gemma Scope,
an open suite of JumpReLU SAEs trained on all layers and sub-layers of Gemma 2
2B and 9B and select layers of Gemma 2 27B base models. We primarily train SAEs
on the Gemma 2 pre-trained models, but additionally release SAEs trained on
instruction-tuned Gemma 2 9B for comparison. We evaluate the quality of each
SAE on standard metrics and release these results. We hope that by releasing
these SAE weights, we can help make more ambitious safety and interpretability
research easier for the community. Weights and a tutorial can be found at
https://huggingface.co/google/gemma-scope and an interactive demo can be found
at https://www.neuronpedia.org/gemma-scope
MulliVC: Multi-lingual Voice Conversion With Cycle Consistency
Authors: name: Jiawei Huang, name: Chen Zhang, name: Yi Ren, name: Ziyue Jiang, name: Zhenhui Ye, name: Jinglin Liu, name: Jinzheng He, name: Xiang Yin, name: Zhou Zhao
Published: 2024-08-08 18:12:51

Voice conversion aims to modify the source speaker's voice to resemble the
target speaker while preserving the original speech content. Despite notable
advancements in voice conversion these days, multi-lingual voice conversion
(including both monolingual and cross-lingual scenarios) has yet to be
extensively studied. It faces two main challenges: 1) the considerable
variability in prosody and articulation habits across languages; and 2) the
rarity of paired multi-lingual datasets from the same speaker. In this paper,
we propose MulliVC, a novel voice conversion system that only converts timbre
and keeps original content and source language prosody without multi-lingual
paired data. Specifically, each training step of MulliVC contains three
substeps: In step one the model is trained with monolingual speech data; then,
steps two and three take inspiration from back translation, construct a
cyclical process to disentangle the timbre and other information (content,
prosody, and other language-related information) in the absence of
multi-lingual data from the same speaker. Both objective and subjective results
indicate that MulliVC significantly surpasses other methods in both monolingual
and cross-lingual contexts, demonstrating the system's efficacy and the
viability of the three-step approach with cycle consistency. Audio samples can
be found on our demo page (mullivc.github.io).
UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond
  Scaling
Authors: name: Haider Al-Tahan, name: Quentin Garrido, name: Randall Balestriero, name: Diane Bouchacourt, name: Caner Hazirbas, name: Mark Ibrahim
Published: 2024-08-09 01:41:05

Significant research efforts have been made to scale and improve
vision-language model (VLM) training approaches. Yet, with an ever-growing
number of benchmarks, researchers are tasked with the heavy burden of
implementing each protocol, bearing a non-trivial computational cost, and
making sense of how all these benchmarks translate into meaningful axes of
progress. To facilitate a systematic evaluation of VLM progress, we introduce
UniBench: a unified implementation of 50+ VLM benchmarks spanning a
comprehensive range of carefully categorized capabilities from object
recognition to spatial awareness, counting, and much more. We showcase the
utility of UniBench for measuring progress by evaluating nearly 60 publicly
available vision-language models, trained on scales of up to 12.8B samples. We
find that while scaling training data or model size can boost many
vision-language model capabilities, scaling offers little benefit for reasoning
or relations. Surprisingly, we also discover today's best VLMs struggle on
simple digit recognition and counting tasks, e.g. MNIST, which much simpler
networks can solve. Where scale falls short, we find that more precise
interventions, such as data quality or tailored-learning objectives offer more
promise. For practitioners, we also offer guidance on selecting a suitable VLM
for a given application. Finally, we release an easy-to-run UniBench code-base
with the full set of 50+ benchmarks and comparisons across 59 models as well as
a distilled, representative set of benchmarks that runs in 5 minutes on a
single GPU.
ToolSandbox: A Stateful, Conversational, Interactive Evaluation
  Benchmark for LLM Tool Use Capabilities
Authors: name: Jiarui Lu, name: Thomas Holleis, name: Yizhe Zhang, name: Bernhard Aumayer, name: Feng Nan, name: Felix Bai, name: Shuang Ma, name: Shen Ma, name: Mengyu Li, name: Guoli Yin, name: Zirui Wang, name: Ruoming Pang
Published: 2024-08-08 05:45:42

Recent large language models (LLMs) advancements sparked a growing research
interest in tool assisted LLMs solving real-world challenges, which calls for
comprehensive evaluation of tool-use capabilities. While previous works focused
on either evaluating over stateless web services (RESTful API), based on a
single turn user prompt, or an off-policy dialog trajectory, ToolSandbox
includes stateful tool execution, implicit state dependencies between tools, a
built-in user simulator supporting on-policy conversational evaluation and a
dynamic evaluation strategy for intermediate and final milestones over an
arbitrary trajectory. We show that open source and proprietary models have a
significant performance gap, and complex tasks like State Dependency,
Canonicalization and Insufficient Information defined in ToolSandbox are
challenging even the most capable SOTA LLMs, providing brand-new insights into
tool-use LLM capabilities. ToolSandbox evaluation framework is released at
https://github.com/apple/ToolSandbox
VGGHeads: A Large-Scale Synthetic Dataset for 3D Human Heads
Authors: name: Orest Kupyn, name: Eugene Khvedchenia, name: Christian Rupprecht
Published: 2024-07-25 17:58:17

Human head detection, keypoint estimation, and 3D head model fitting are
important tasks with many applications. However, traditional real-world
datasets often suffer from bias, privacy, and ethical concerns, and they have
been recorded in laboratory environments, which makes it difficult for trained
models to generalize. Here, we introduce VGGHeads -- a large scale synthetic
dataset generated with diffusion models for human head detection and 3D mesh
estimation. Our dataset comprises over 1 million high-resolution images, each
annotated with detailed 3D head meshes, facial landmarks, and bounding boxes.
Using this dataset we introduce a new model architecture capable of
simultaneous heads detection and head meshes reconstruction from a single image
in a single step. Through extensive experimental evaluations, we demonstrate
that models trained on our synthetic data achieve strong performance on real
images. Furthermore, the versatility of our dataset makes it applicable across
a broad spectrum of tasks, offering a general and comprehensive representation
of human heads. Additionally, we provide detailed information about the
synthetic data generation pipeline, enabling it to be re-used for other tasks
and domains.
Learning Task Decomposition to Assist Humans in Competitive Programming
Authors: name: Jiaxin Wen, name: Ruiqi Zhong, name: Pei Ke, name: Zhihong Shao, name: Hongning Wang, name: Minlie Huang
Published: 2024-06-07 03:27:51

When using language models (LMs) to solve complex problems, humans might
struggle to understand the LM-generated solutions and repair the flawed ones.
To assist humans in repairing them, we propose to automatically decompose
complex solutions into multiple simpler pieces that correspond to specific
subtasks. We introduce a novel objective for learning task decomposition,
termed assistive value (AssistV), which measures the feasibility and speed for
humans to repair the decomposed solution. We collect a dataset of human repair
experiences on different decomposed solutions. Utilizing the collected data as
in-context examples, we then learn to critique, refine, and rank decomposed
solutions to improve AssistV. We validate our method under competitive
programming problems: under 177 hours of human study, our method enables
non-experts to solve 33.3\% more problems, speeds them up by 3.3x, and empowers
them to match unassisted experts.