Orionnnn/hub-stats huggingface dataset:
SambaLingo: Teaching Large Language Models New Languages
Authors: name: Zoltan Csaki, name: Bo Li, name: Jonathan Li, name: Qiantong Xu, name: Pian Pawakapan, name: Leon Zhang, name: Yun Du, name: Hengyu Zhao, name: Changran Hu, name: Urmish Thakker
Published: 2024-04-08 19:48:36

Despite the widespread availability of LLMs, there remains a substantial gap
in their capabilities and availability across diverse languages. One approach
to address these issues has been to take an existing pre-trained LLM and
continue to train it on new languages. While prior works have experimented with
language adaptation, many questions around best practices and methodology have
not been covered. In this paper, we present a comprehensive investigation into
the adaptation of LLMs to new languages. Our study covers the key components in
this process, including vocabulary extension, direct preference optimization
and the data scarcity problem for human alignment in low-resource languages. We
scale these experiments across 9 languages and 2 parameter scales (7B and 70B).
We compare our models against Llama 2, Aya-101, XGLM, BLOOM and existing
language experts, outperforming all prior published baselines. Additionally,
all evaluation code and checkpoints are made public to facilitate future
research.
LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders
Authors: name: Parishad BehnamGhader, name: Vaibhav Adlakha, name: Marius Mosbach, name: Dzmitry Bahdanau, name: Nicolas Chapados, name: Siva Reddy
Published: 2024-04-09 02:51:05

Large decoder-only language models (LLMs) are the state-of-the-art models on
most of today's NLP tasks and benchmarks. Yet, the community is only slowly
adopting these models for text embedding tasks, which require rich
contextualized representations. In this work, we introduce LLM2Vec, a simple
unsupervised approach that can transform any decoder-only LLM into a strong
text encoder. LLM2Vec consists of three simple steps: 1) enabling bidirectional
attention, 2) masked next token prediction, and 3) unsupervised contrastive
learning. We demonstrate the effectiveness of LLM2Vec by applying it to 3
popular LLMs ranging from 1.3B to 7B parameters and evaluate the transformed
models on English word- and sequence-level tasks. We outperform encoder-only
models by a large margin on word-level tasks and reach a new unsupervised
state-of-the-art performance on the Massive Text Embeddings Benchmark (MTEB).
Moreover, when combining LLM2Vec with supervised contrastive learning, we
achieve state-of-the-art performance on MTEB among models that train only on
publicly available data. Our strong empirical results and extensive analysis
demonstrate that LLMs can be effectively transformed into universal text
encoders in a parameter-efficient manner without the need for expensive
adaptation or synthetic GPT-4 generated data.
InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model
  Handling Resolutions from 336 Pixels to 4K HD
Authors: name: Xiaoyi Dong, name: Pan Zhang, name: Yuhang Zang, name: Yuhang Cao, name: Bin Wang, name: Linke Ouyang, name: Songyang Zhang, name: Haodong Duan, name: Wenwei Zhang, name: Yining Li, name: Hang Yan, name: Yang Gao, name: Zhe Chen, name: Xinyue Zhang, name: Wei Li, name: Jingwen Li, name: Wenhai Wang, name: Kai Chen, name: Conghui He, name: Xingcheng Zhang, name: Jifeng Dai, name: Yu Qiao, name: Dahua Lin, name: Jiaqi Wang
Published: 2024-04-09 17:59:32

The Large Vision-Language Model (LVLM) field has seen significant
advancements, yet its progression has been hindered by challenges in
comprehending fine-grained visual content due to limited resolution. Recent
efforts have aimed to enhance the high-resolution understanding capabilities of
LVLMs, yet they remain capped at approximately 1500 x 1500 pixels and
constrained to a relatively narrow resolution range. This paper represents
InternLM-XComposer2-4KHD, a groundbreaking exploration into elevating LVLM
resolution capabilities up to 4K HD (3840 x 1600) and beyond. Concurrently,
considering the ultra-high resolution may not be necessary in all scenarios, it
supports a wide range of diverse resolutions from 336 pixels to 4K standard,
significantly broadening its scope of applicability. Specifically, this
research advances the patch division paradigm by introducing a novel extension:
dynamic resolution with automatic patch configuration. It maintains the
training image aspect ratios while automatically varying patch counts and
configuring layouts based on a pre-trained Vision Transformer (ViT) (336 x
336), leading to dynamic training resolution from 336 pixels to 4K standard.
Our research demonstrates that scaling training resolution up to 4K HD leads to
consistent performance enhancements without hitting the ceiling of potential
improvements. InternLM-XComposer2-4KHD shows superb capability that matches or
even surpasses GPT-4V and Gemini Pro in 10 of the 16 benchmarks. The
InternLM-XComposer2-4KHD model series with 7B parameters are publicly available
at https://github.com/InternLM/InternLM-XComposer.
MuPT: A Generative Symbolic Music Pretrained Transformer
Authors: name: Xingwei Qu, name: Yuelin Bai, name: Yinghao Ma, name: Ziya Zhou, name: Ka Man Lo, name: Jiaheng Liu, name: Ruibin Yuan, name: Lejun Min, name: Xueling Liu, name: Tianyu Zhang, name: Xinrun Du, name: Shuyue Guo, name: Yiming Liang, name: Yizhi Li, name: Shangda Wu, name: Junting Zhou, name: Tianyu Zheng, name: Ziyang Ma, name: Fengze Han, name: Wei Xue, name: Gus Xia, name: Emmanouil Benetos, name: Xiang Yue, name: Chenghua Lin, name: Xu Tan, name: Stephen W. Huang, name: Wenhu Chen, name: Jie Fu, name: Ge Zhang
Published: 2024-04-09 15:35:52

In this paper, we explore the application of Large Language Models (LLMs) to
the pre-training of music. While the prevalent use of MIDI in music modeling is
well-established, our findings suggest that LLMs are inherently more compatible
with ABC Notation, which aligns more closely with their design and strengths,
thereby enhancing the model's performance in musical composition. To address
the challenges associated with misaligned measures from different tracks during
generation, we propose the development of a Synchronized
Multi-Track ABC Notation (SMT-ABC Notation),
which aims to preserve coherence across multiple musical tracks. Our
contributions include a series of models capable of handling up to 8192 tokens,
covering 90\% of the symbolic music data in our training set. Furthermore, we
explore the implications of the Symbolic Music
Scaling Law (SMS Law) on model performance. The results
indicate a promising direction for future research in music generation,
offering extensive resources for community-led research through our open-source
contributions.
Magic-Boost: Boost 3D Generation with Mutli-View Conditioned Diffusion
Authors: name: Fan Yang, name: Jianfeng Zhang, name: Yichun Shi, name: Bowen Chen, name: Chenxu Zhang, name: Huichao Zhang, name: Xiaofeng Yang, name: Jiashi Feng, name: Guosheng Lin
Published: 2024-04-09 16:20:03

Benefiting from the rapid development of 2D diffusion models, 3D content
creation has made significant progress recently. One promising solution
involves the fine-tuning of pre-trained 2D diffusion models to harness their
capacity for producing multi-view images, which are then lifted into accurate
3D models via methods like fast-NeRFs or large reconstruction models. However,
as inconsistency still exists and limited generated resolution, the generation
results of such methods still lack intricate textures and complex geometries.
To solve this problem, we propose Magic-Boost, a multi-view conditioned
diffusion model that significantly refines coarse generative results through a
brief period of SDS optimization (sim15min). Compared to the previous text
or single image based diffusion models, Magic-Boost exhibits a robust
capability to generate images with high consistency from pseudo synthesized
multi-view images. It provides precise SDS guidance that well aligns with the
identity of the input images, enriching the local detail in both geometry and
texture of the initial generative results. Extensive experiments show
Magic-Boost greatly enhances the coarse inputs and generates high-quality 3D
assets with rich geometric and textural details. (Project Page:
https://magic-research.github.io/magic-boost/)
SpatialTracker: Tracking Any 2D Pixels in 3D Space
Authors: name: Yuxi Xiao, name: Qianqian Wang, name: Shangzhan Zhang, name: Nan Xue, name: Sida Peng, name: Yujun Shen, name: Xiaowei Zhou
Published: 2024-04-05 17:59:25

Recovering dense and long-range pixel motion in videos is a challenging
problem. Part of the difficulty arises from the 3D-to-2D projection process,
leading to occlusions and discontinuities in the 2D motion domain. While 2D
motion can be intricate, we posit that the underlying 3D motion can often be
simple and low-dimensional. In this work, we propose to estimate point
trajectories in 3D space to mitigate the issues caused by image projection. Our
method, named SpatialTracker, lifts 2D pixels to 3D using monocular depth
estimators, represents the 3D content of each frame efficiently using a
triplane representation, and performs iterative updates using a transformer to
estimate 3D trajectories. Tracking in 3D allows us to leverage
as-rigid-as-possible (ARAP) constraints while simultaneously learning a
rigidity embedding that clusters pixels into different rigid parts. Extensive
evaluation shows that our approach achieves state-of-the-art tracking
performance both qualitatively and quantitatively, particularly in challenging
scenarios such as out-of-plane rotation.
PhysAvatar: Learning the Physics of Dressed 3D Avatars from Visual
  Observations
Authors: name: Yang Zheng, name: Qingqing Zhao, name: Guandao Yang, name: Wang Yifan, name: Donglai Xiang, name: Florian Dubost, name: Dmitry Lagun, name: Thabo Beeler, name: Federico Tombari, name: Leonidas Guibas, name: Gordon Wetzstein
Published: 2024-04-05 21:44:57

Modeling and rendering photorealistic avatars is of crucial importance in
many applications. Existing methods that build a 3D avatar from visual
observations, however, struggle to reconstruct clothed humans. We introduce
PhysAvatar, a novel framework that combines inverse rendering with inverse
physics to automatically estimate the shape and appearance of a human from
multi-view video data along with the physical parameters of the fabric of their
clothes. For this purpose, we adopt a mesh-aligned 4D Gaussian technique for
spatio-temporal mesh tracking as well as a physically based inverse renderer to
estimate the intrinsic material properties. PhysAvatar integrates a physics
simulator to estimate the physical parameters of the garments using
gradient-based optimization in a principled manner. These novel capabilities
enable PhysAvatar to create high-quality novel-view renderings of avatars
dressed in loose-fitting clothes under motions and lighting conditions not seen
in the training data. This marks a significant advancement towards modeling
photorealistic digital humans using physically based inverse rendering with
physics in the loop. Our project website is at:
https://qingqing-zhao.github.io/PhysAvatar
MoMA: Multimodal LLM Adapter for Fast Personalized Image Generation
Authors: name: Kunpeng Song, name: Yizhe Zhu, name: Bingchen Liu, name: Qing Yan, name: Ahmed Elgammal, name: Xiao Yang
Published: 2024-04-08 16:55:49

In this paper, we present MoMA: an open-vocabulary, training-free
personalized image model that boasts flexible zero-shot capabilities. As
foundational text-to-image models rapidly evolve, the demand for robust
image-to-image translation grows. Addressing this need, MoMA specializes in
subject-driven personalized image generation. Utilizing an open-source,
Multimodal Large Language Model (MLLM), we train MoMA to serve a dual role as
both a feature extractor and a generator. This approach effectively synergizes
reference image and text prompt information to produce valuable image features,
facilitating an image diffusion model. To better leverage the generated
features, we further introduce a novel self-attention shortcut method that
efficiently transfers image features to an image diffusion model, improving the
resemblance of the target object in generated images. Remarkably, as a
tuning-free plug-and-play module, our model requires only a single reference
image and outperforms existing methods in generating images with high detail
fidelity, enhanced identity-preservation and prompt faithfulness. Our work is
open-source, thereby providing universal access to these advancements.
YaART: Yet Another ART Rendering Technology
Authors: name: Sergey Kastryulin, name: Artem Konev, name: Alexander Shishenya, name: Eugene Lyapustin, name: Artem Khurshudov, name: Alexander Tselousov, name: Nikita Vinokurov, name: Denis Kuznedelev, name: Alexander Markovich, name: Grigoriy Livshits, name: Alexey Kirillov, name: Anastasiia Tabisheva, name: Liubov Chubarova, name: Marina Kaminskaia, name: Alexander Ustyuzhanin, name: Artemii Shvetsov, name: Daniil Shlenskii, name: Valerii Startsev, name: Dmitrii Kornilov, name: Mikhail Romanov, name: Artem Babenko, name: Sergei Ovcharenko, name: Valentin Khrulkov
Published: 2024-04-08 16:51:19

In the rapidly progressing field of generative models, the development of
efficient and high-fidelity text-to-image diffusion systems represents a
significant frontier. This study introduces YaART, a novel production-grade
text-to-image cascaded diffusion model aligned to human preferences using
Reinforcement Learning from Human Feedback (RLHF). During the development of
YaART, we especially focus on the choices of the model and training dataset
sizes, the aspects that were not systematically investigated for text-to-image
cascaded diffusion models before. In particular, we comprehensively analyze how
these choices affect both the efficiency of the training process and the
quality of the generated images, which are highly important in practice.
Furthermore, we demonstrate that models trained on smaller datasets of
higher-quality images can successfully compete with those trained on larger
datasets, establishing a more efficient scenario of diffusion models training.
From the quality perspective, YaART is consistently preferred by users over
many existing state-of-the-art models.
ByteEdit: Boost, Comply and Accelerate Generative Image Editing
Authors: name: Yuxi Ren, name: Jie Wu, name: Yanzuo Lu, name: Huafeng Kuang, name: Xin Xia, name: Xionghui Wang, name: Qianqian Wang, name: Yixing Zhu, name: Pan Xie, name: Shiyin Wang, name: Xuefeng Xiao, name: Yitong Wang, name: Min Zheng, name: Lean Fu
Published: 2024-04-07 08:07:14

Recent advancements in diffusion-based generative image editing have sparked
a profound revolution, reshaping the landscape of image outpainting and
inpainting tasks. Despite these strides, the field grapples with inherent
challenges, including: i) inferior quality; ii) poor consistency; iii)
insufficient instrcution adherence; iv) suboptimal generation efficiency. To
address these obstacles, we present ByteEdit, an innovative feedback learning
framework meticulously designed to Boost, Comply, and Accelerate Generative
Image Editing tasks. ByteEdit seamlessly integrates image reward models
dedicated to enhancing aesthetics and image-text alignment, while also
introducing a dense, pixel-level reward model tailored to foster coherence in
the output. Furthermore, we propose a pioneering adversarial and progressive
feedback learning strategy to expedite the model's inference speed. Through
extensive large-scale user evaluations, we demonstrate that ByteEdit surpasses
leading generative image editing products, including Adobe, Canva, and MeiTu,
in both generation quality and consistency. ByteEdit-Outpainting exhibits a
remarkable enhancement of 388% and 135% in quality and consistency,
respectively, when compared to the baseline model. Experiments also verfied
that our acceleration models maintains excellent performance results in terms
of quality and consistency.
BeyondScene: Higher-Resolution Human-Centric Scene Generation With
  Pretrained Diffusion
Authors: name: Gwanghyun Kim, name: Hayeon Kim, name: Hoigi Seo, name: Dong Un Kang, name: Se Young Chun
Published: 2024-04-06 07:53:49

Generating higher-resolution human-centric scenes with details and controls
remains a challenge for existing text-to-image diffusion models. This challenge
stems from limited training image size, text encoder capacity (limited tokens),
and the inherent difficulty of generating complex scenes involving multiple
humans. While current methods attempted to address training size limit only,
they often yielded human-centric scenes with severe artifacts. We propose
BeyondScene, a novel framework that overcomes prior limitations, generating
exquisite higher-resolution (over 8K) human-centric scenes with exceptional
text-image correspondence and naturalness using existing pretrained diffusion
models. BeyondScene employs a staged and hierarchical approach to initially
generate a detailed base image focusing on crucial elements in instance
creation for multiple humans and detailed descriptions beyond token limit of
diffusion model, and then to seamlessly convert the base image to a
higher-resolution output, exceeding training image size and incorporating
details aware of text and instances via our novel instance-aware hierarchical
enlargement process that consists of our proposed high-frequency injected
forward diffusion and adaptive joint diffusion. BeyondScene surpasses existing
methods in terms of correspondence with detailed text descriptions and
naturalness, paving the way for advanced applications in higher-resolution
human-centric scene creation beyond the capacity of pretrained diffusion models
without costly retraining. Project page:
https://janeyeon.github.io/beyond-scene.
Aligning Diffusion Models by Optimizing Human Utility
Authors: name: Shufan Li, name: Konstantinos Kallidromitis, name: Akash Gokul, name: Yusuke Kato, name: Kazuki Kozuka
Published: 2024-04-06 01:23:23

We present Diffusion-KTO, a novel approach for aligning text-to-image
diffusion models by formulating the alignment objective as the maximization of
expected human utility. Since this objective applies to each generation
independently, Diffusion-KTO does not require collecting costly pairwise
preference data nor training a complex reward model. Instead, our objective
requires simple per-image binary feedback signals, e.g. likes or dislikes,
which are abundantly available. After fine-tuning using Diffusion-KTO,
text-to-image diffusion models exhibit superior performance compared to
existing techniques, including supervised fine-tuning and Diffusion-DPO, both
in terms of human judgment and automatic evaluation metrics such as PickScore
and ImageReward. Overall, Diffusion-KTO unlocks the potential of leveraging
readily available per-image binary signals and broadens the applicability of
aligning text-to-image diffusion models with human preferences.
DATENeRF: Depth-Aware Text-based Editing of NeRFs
Authors: name: Sara Rojas, name: Julien Philip, name: Kai Zhang, name: Sai Bi, name: Fujun Luan, name: Bernard Ghanem, name: Kalyan Sunkavall
Published: 2024-04-06 06:48:16

Recent advancements in diffusion models have shown remarkable proficiency in
editing 2D images based on text prompts. However, extending these techniques to
edit scenes in Neural Radiance Fields (NeRF) is complex, as editing individual
2D frames can result in inconsistencies across multiple views. Our crucial
insight is that a NeRF scene's geometry can serve as a bridge to integrate
these 2D edits. Utilizing this geometry, we employ a depth-conditioned
ControlNet to enhance the coherence of each 2D image modification. Moreover, we
introduce an inpainting approach that leverages the depth information of NeRF
scenes to distribute 2D edits across different images, ensuring robustness
against errors and resampling challenges. Our results reveal that this
methodology achieves more consistent, lifelike, and detailed edits than
existing leading methods for text-driven NeRF scene editing.
SwapAnything: Enabling Arbitrary Object Swapping in Personalized Visual
  Editing
Authors: name: Jing Gu, name: Yilin Wang, name: Nanxuan Zhao, name: Wei Xiong, name: Qing Liu, name: Zhifei Zhang, name: He Zhang, name: Jianming Zhang, name: HyunJoon Jung, name: Xin Eric Wang
Published: 2024-04-08 17:52:29

Effective editing of personal content holds a pivotal role in enabling
individuals to express their creativity, weaving captivating narratives within
their visual stories, and elevate the overall quality and impact of their
visual content. Therefore, in this work, we introduce SwapAnything, a novel
framework that can swap any objects in an image with personalized concepts
given by the reference, while keeping the context unchanged. Compared with
existing methods for personalized subject swapping, SwapAnything has three
unique advantages: (1) precise control of arbitrary objects and parts rather
than the main subject, (2) more faithful preservation of context pixels, (3)
better adaptation of the personalized concept to the image. First, we propose
targeted variable swapping to apply region control over latent feature maps and
swap masked variables for faithful context preservation and initial semantic
concept swapping. Then, we introduce appearance adaptation, to seamlessly adapt
the semantic concept into the original image in terms of target location,
shape, style, and content during the image generation process. Extensive
results on both human and automatic evaluation demonstrate significant
improvements of our approach over baseline methods on personalized swapping.
Furthermore, SwapAnything shows its precise and faithful swapping abilities
across single object, multiple objects, partial object, and cross-domain
swapping tasks. SwapAnything also achieves great performance on text-based
swapping and tasks beyond swapping such as object insertion.
Koala: Key frame-conditioned long video-LLM
Authors: name: Reuben Tan, name: Ximeng Sun, name: Ping Hu, name: Jui-hsien Wang, name: Hanieh Deilamsalehy, name: Bryan A. Plummer, name: Bryan Russell, name: Kate Saenko
Published: 2024-04-05 18:33:04

Long video question answering is a challenging task that involves recognizing
short-term activities and reasoning about their fine-grained relationships.
State-of-the-art video Large Language Models (vLLMs) hold promise as a viable
solution due to their demonstrated emergent capabilities on new tasks. However,
despite being trained on millions of short seconds-long videos, vLLMs are
unable to understand minutes-long videos and accurately answer questions about
them. To address this limitation, we propose a lightweight and self-supervised
approach, Key frame-conditioned long video-LLM (Koala), that introduces
learnable spatiotemporal queries to adapt pretrained vLLMs for generalizing to
longer videos. Our approach introduces two new tokenizers that condition on
visual tokens computed from sparse video key frames for understanding short and
long video moments. We train our proposed approach on HowTo100M and demonstrate
its effectiveness on zero-shot long video understanding benchmarks, where it
outperforms state-of-the-art large models by 3 - 6% in absolute accuracy across
all tasks. Surprisingly, we also empirically show that our approach not only
helps a pretrained vLLM to understand long videos but also improves its
accuracy on short-term action recognition.