Orionnnn/hub-stats huggingface dataset:
SeaLLMs -- Large Language Models for Southeast Asia
Authors: name: Xuan-Phi Nguyen, name: Wenxuan Zhang, name: Xin Li, name: Mahani Aljunied, name: Qingyu Tan, name: Liying Cheng, name: Guanzheng Chen, name: Yue Deng, name: Sen Yang, name: Chaoqun Liu, name: Hang Zhang, name: Lidong Bing
Published: 2023-12-01 17:17:56

Despite the remarkable achievements of large language models (LLMs) in
various tasks, there remains a linguistic bias that favors high-resource
languages, such as English, often at the expense of low-resource and regional
languages. To address this imbalance, we introduce SeaLLMs, an innovative
series of language models that specifically focuses on Southeast Asian (SEA)
languages. SeaLLMs are built upon the Llama-2 model and further advanced
through continued pre-training with an extended vocabulary, specialized
instruction and alignment tuning to better capture the intricacies of regional
languages. This allows them to respect and reflect local cultural norms,
customs, stylistic preferences, and legal considerations. Our comprehensive
evaluation demonstrates that SeaLLM-13b models exhibit superior performance
across a wide spectrum of linguistic tasks and assistant-style
instruction-following capabilities relative to comparable open-source models.
Moreover, they outperform ChatGPT-3.5 in non-Latin languages, such as Thai,
Khmer, Lao, and Burmese, by large margins while remaining lightweight and
cost-effective to operate.
GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphs
Authors: name: Gege Gao, name: Weiyang Liu, name: Anpei Chen, name: Andreas Geiger, name: Bernhard Schölkopf
Published: 2023-11-30 18:59:58

As pretrained text-to-image diffusion models become increasingly powerful,
recent efforts have been made to distill knowledge from these text-to-image
pretrained models for optimizing a text-guided 3D model. Most of the existing
methods generate a holistic 3D model from a plain text input. This can be
problematic when the text describes a complex scene with multiple objects,
because the vectorized text embeddings are inherently unable to capture a
complex description with multiple entities and relationships. Holistic 3D
modeling of the entire scene further prevents accurate grounding of text
entities and concepts. To address this limitation, we propose GraphDreamer, a
novel framework to generate compositional 3D scenes from scene graphs, where
objects are represented as nodes and their interactions as edges. By exploiting
node and edge information in scene graphs, our method makes better use of the
pretrained text-to-image diffusion model and is able to fully disentangle
different objects without image-level supervision. To facilitate modeling of
object-wise relationships, we use signed distance fields as representation and
impose a constraint to avoid inter-penetration of objects. To avoid manual
scene graph creation, we design a text prompt for ChatGPT to generate scene
graphs based on text inputs. We conduct both qualitative and quantitative
experiments to validate the effectiveness of GraphDreamer in generating
high-fidelity compositional 3D scenes with disentangled object entities.
Using Human Feedback to Fine-tune Diffusion Models without Any Reward
  Model
Authors: name: Kai Yang, name: Jian Tao, name: Jiafei Lyu, name: Chunjiang Ge, name: Jiaxin Chen, name: Qimai Li, name: Weihan Shen, name: Xiaolong Zhu, name: Xiu Li
Published: 2023-11-22 08:42:46

Using reinforcement learning with human feedback (RLHF) has shown significant
promise in fine-tuning diffusion models. Previous methods start by training a
reward model that aligns with human preferences, then leverage RL techniques to
fine-tune the underlying models. However, crafting an efficient reward model
demands extensive datasets, optimal architecture, and manual hyperparameter
tuning, making the process both time and cost-intensive. The direct preference
optimization (DPO) method, effective in fine-tuning large language models,
eliminates the necessity for a reward model. However, the extensive GPU memory
requirement of the diffusion model's denoising process hinders the direct
application of the DPO method. To address this issue, we introduce the Direct
Preference for Denoising Diffusion Policy Optimization (D3PO) method to
directly fine-tune diffusion models. The theoretical analysis demonstrates that
although D3PO omits training a reward model, it effectively functions as the
optimal reward model trained using human feedback data to guide the learning
process. This approach requires no training of a reward model, proving to be
more direct, cost-effective, and minimizing computational overhead. In
experiments, our method uses the relative scale of objectives as a proxy for
human preference, delivering comparable results to methods using ground-truth
rewards. Moreover, D3PO demonstrates the ability to reduce image distortion
rates and generate safer images, overcoming challenges lacking robust reward
models.
ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs
Authors: name: Viraj Shah, name: Nataniel Ruiz, name: Forrester Cole, name: Erika Lu, name: Svetlana Lazebnik, name: Yuanzhen Li, name: Varun Jampani
Published: 2023-11-22 18:59:36

Methods for finetuning generative models for concept-driven personalization
generally achieve strong results for subject-driven or style-driven generation.
Recently, low-rank adaptations (LoRA) have been proposed as a
parameter-efficient way of achieving concept-driven personalization. While
recent work explores the combination of separate LoRAs to achieve joint
generation of learned styles and subjects, existing techniques do not reliably
address the problem; they often compromise either subject fidelity or style
fidelity. We propose ZipLoRA, a method to cheaply and effectively merge
independently trained style and subject LoRAs in order to achieve generation of
any user-provided subject in any user-provided style. Experiments on a wide
range of subject and style combinations show that ZipLoRA can generate
compelling results with meaningful improvements over baselines in subject and
style fidelity while preserving the ability to recontextualize. Project page:
https://ziplora.github.io
FusionFrames: Efficient Architectural Aspects for Text-to-Video
  Generation Pipeline
Authors: name: Vladimir Arkhipkin, name: Zein Shaheen, name: Viacheslav Vasilev, name: Elizaveta Dakhova, name: Andrey Kuznetsov, name: Denis Dimitrov
Published: 2023-11-22 00:26:15

Multimedia generation approaches occupy a prominent place in artificial
intelligence research. Text-to-image models achieved high-quality results over
the last few years. However, video synthesis methods recently started to
develop. This paper presents a new two-stage latent diffusion text-to-video
generation architecture based on the text-to-image diffusion model. The first
stage concerns keyframes synthesis to figure the storyline of a video, while
the second one is devoted to interpolation frames generation to make movements
of the scene and objects smooth. We compare several temporal conditioning
approaches for keyframes generation. The results show the advantage of using
separate temporal blocks over temporal layers in terms of metrics reflecting
video generation quality aspects and human preference. The design of our
interpolation model significantly reduces computational costs compared to other
masked frame interpolation approaches. Furthermore, we evaluate different
configurations of MoVQ-based video decoding scheme to improve consistency and
achieve higher PSNR, SSIM, MSE, and LPIPS scores. Finally, we compare our
pipeline with existing solutions and achieve top-2 scores overall and top-1
among open-source solutions: CLIPSIM = 0.2976 and FVD = 433.054. Project page:
https://ai-forever.github.io/kandinsky-video/
LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes
Authors: name: Jaeyoung Chung, name: Suyoung Lee, name: Hyeongjin Nam, name: Jaerin Lee, name: Kyoung Mu Lee
Published: 2023-11-22 13:27:34

With the widespread usage of VR devices and contents, demands for 3D scene
generation techniques become more popular. Existing 3D scene generation models,
however, limit the target scene to specific domain, primarily due to their
training strategies using 3D scan dataset that is far from the real-world. To
address such limitation, we propose LucidDreamer, a domain-free scene
generation pipeline by fully leveraging the power of existing large-scale
diffusion-based generative model. Our LucidDreamer has two alternate steps:
Dreaming and Alignment. First, to generate multi-view consistent images from
inputs, we set the point cloud as a geometrical guideline for each image
generation. Specifically, we project a portion of point cloud to the desired
view and provide the projection as a guidance for inpainting using the
generative model. The inpainted images are lifted to 3D space with estimated
depth maps, composing a new points. Second, to aggregate the new points into
the 3D scene, we propose an aligning algorithm which harmoniously integrates
the portions of newly generated 3D scenes. The finally obtained 3D scene serves
as initial points for optimizing Gaussian splats. LucidDreamer produces
Gaussian splats that are highly-detailed compared to the previous 3D scene
generation methods, with no constraint on domain of the target scene.
Diffusion360: Seamless 360 Degree Panoramic Image Generation based on
  Diffusion Models
Authors: name: Mengyang Feng, name: Jinlin Liu, name: Miaomiao Cui, name: Xuansong Xie
Published: 2023-11-22 04:06:39

This is a technical report on the 360-degree panoramic image generation task
based on diffusion models. Unlike ordinary 2D images, 360-degree panoramic
images capture the entire 360^circtimes 180^circ field of view. So the
rightmost and the leftmost sides of the 360 panoramic image should be
continued, which is the main challenge in this field. However, the current
diffusion pipeline is not appropriate for generating such a seamless 360-degree
panoramic image. To this end, we propose a circular blending strategy on both
the denoising and VAE decoding stages to maintain the geometry continuity.
Based on this, we present two models for Text-to-360-panoramas and
Single-Image-to-360-panoramas tasks. The code has been released as an
open-source project at
https://github.com/ArcherFMY/SD-T2I-360PanoImage{https://github.com/ArcherFMY/SD-T2I-360PanoImage}
and
https://www.modelscope.cn/models/damo/cv_diffusion_text-to-360panorama-image_generation/summary{ModelScope}
Visual In-Context Prompting
Authors: name: Feng Li, name: Qing Jiang, name: Hao Zhang, name: Tianhe Ren, name: Shilong Liu, name: Xueyan Zou, name: Huaizhe Xu, name: Hongyang Li, name: Chunyuan Li, name: Jianwei Yang, name: Lei Zhang, name: Jianfeng Gao
Published: 2023-11-22 18:59:48

In-context prompting in large language models (LLMs) has become a prevalent
approach to improve zero-shot capabilities, but this idea is less explored in
the vision domain. Existing visual prompting methods focus on referring
segmentation to segment the most relevant object, falling short of addressing
many generic vision tasks like open-set segmentation and detection. In this
paper, we introduce a universal visual in-context prompting framework for both
tasks. In particular, we build on top of an encoder-decoder architecture, and
develop a versatile prompt encoder to support a variety of prompts like
strokes, boxes, and points. We further enhance it to take an arbitrary number
of reference image segments as the context. Our extensive explorations show
that the proposed visual in-context prompting elicits extraordinary referring
and generic segmentation capabilities to refer and detect, yielding competitive
performance to close-set in-domain datasets and showing promising results on
many open-set segmentation datasets. By joint training on COCO and SA-1B, our
model achieves 57.7 PQ on COCO and 23.2 PQ on ADE20K. Code will be
available at https://github.com/UX-Decoder/DINOv.
Diffusion Model Alignment Using Direct Preference Optimization
Authors: name: Bram Wallace, name: Meihua Dang, name: Rafael Rafailov, name: Linqi Zhou, name: Aaron Lou, name: Senthil Purushwalkam, name: Stefano Ermon, name: Caiming Xiong, name: Shafiq Joty, name: Nikhil Naik
Published: 2023-11-21 15:24:05

Large language models (LLMs) are fine-tuned using human comparison data with
Reinforcement Learning from Human Feedback (RLHF) methods to make them better
aligned with users' preferences. In contrast to LLMs, human preference learning
has not been widely explored in text-to-image diffusion models; the best
existing approach is to fine-tune a pretrained model using carefully curated
high quality images and captions to improve visual appeal and text alignment.
We propose Diffusion-DPO, a method to align diffusion models to human
preferences by directly optimizing on human comparison data. Diffusion-DPO is
adapted from the recently developed Direct Preference Optimization (DPO), a
simpler alternative to RLHF which directly optimizes a policy that best
satisfies human preferences under a classification objective. We re-formulate
DPO to account for a diffusion model notion of likelihood, utilizing the
evidence lower bound to derive a differentiable objective. Using the Pick-a-Pic
dataset of 851K crowdsourced pairwise preferences, we fine-tune the base model
of the state-of-the-art Stable Diffusion XL (SDXL)-1.0 model with
Diffusion-DPO. Our fine-tuned base model significantly outperforms both base
SDXL-1.0 and the larger SDXL-1.0 model consisting of an additional refinement
model in human evaluation, improving visual appeal and prompt alignment. We
also develop a variant that uses AI feedback and has comparable performance to
training on human preferences, opening the door for scaling of diffusion model
alignment methods.
PG-Video-LLaVA: Pixel Grounding Large Video-Language Models
Authors: name: Shehan Munasinghe, name: Rusiru Thushara, name: Muhammad Maaz, name: Hanoona Abdul Rasheed, name: Salman Khan, name: Mubarak Shah, name: Fahad Khan
Published: 2023-11-22 14:48:30

Extending image-based Large Multimodal Models (LMM) to videos is challenging
due to the inherent complexity of video data. The recent approaches extending
image-based LMM to videos either lack the grounding capabilities (e.g.,
VideoChat, Video-ChatGPT, Video-LLaMA) or do not utilize the audio-signals for
better video understanding (e.g., Video-ChatGPT). Addressing these gaps, we
propose Video-LLaVA, the first LMM with pixel-level grounding capability,
integrating audio cues by transcribing them into text to enrich video-context
understanding. Our framework uses an off-the-shelf tracker and a novel
grounding module, enabling it to spatially and temporally localize objects in
videos following user instructions. We evaluate Video-LLaVA using video-based
generative and question-answering benchmarks and introduce new benchmarks
specifically designed to measure prompt-based object grounding performance in
videos. Further, we propose the use of Vicuna over GPT-3.5, as utilized in
Video-ChatGPT, for video-based conversation benchmarking, ensuring
reproducibility of results which is a concern with the proprietary nature of
GPT-3.5. Our framework builds on SoTA image-based LLaVA model and extends its
advantages to the video domain, delivering promising gains on video-based
conversation and grounding tasks. Project Page:
https://github.com/mbzuai-oryx/Video-LLaVA
GAIA: a benchmark for General AI Assistants
Authors: name: Grégoire Mialon, name: Clémentine Fourrier, name: Craig Swift, name: Thomas Wolf, name: Yann LeCun, name: Thomas Scialom
Published: 2023-11-21 20:34:47

We introduce GAIA, a benchmark for General AI Assistants that, if solved,
would represent a milestone in AI research. GAIA proposes real-world questions
that require a set of fundamental abilities such as reasoning, multi-modality
handling, web browsing, and generally tool-use proficiency. GAIA questions are
conceptually simple for humans yet challenging for most advanced AIs: we show
that human respondents obtain 92\% vs. 15\% for GPT-4 equipped with plugins.
This notable performance disparity contrasts with the recent trend of LLMs
outperforming humans on tasks requiring professional skills in e.g. law or
chemistry. GAIA's philosophy departs from the current trend in AI benchmarks
suggesting to target tasks that are ever more difficult for humans. We posit
that the advent of Artificial General Intelligence (AGI) hinges on a system's
capability to exhibit similar robustness as the average human does on such
questions. Using GAIA's methodology, we devise 466 questions and their answer.
We release our questions while retaining answers to 300 of them to power a
leader-board available at https://huggingface.co/gaia-benchmark.
PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape
  Prediction
Authors: name: Peng Wang, name: Hao Tan, name: Sai Bi, name: Yinghao Xu, name: Fujun Luan, name: Kalyan Sunkavalli, name: Wenping Wang, name: Zexiang Xu, name: Kai Zhang
Published: 2023-11-20 18:57:55

We propose a Pose-Free Large Reconstruction Model (PF-LRM) for reconstructing
a 3D object from a few unposed images even with little visual overlap, while
simultaneously estimating the relative camera poses in ~1.3 seconds on a single
A100 GPU. PF-LRM is a highly scalable method utilizing the self-attention
blocks to exchange information between 3D object tokens and 2D image tokens; we
predict a coarse point cloud for each view, and then use a differentiable
Perspective-n-Point (PnP) solver to obtain camera poses. When trained on a huge
amount of multi-view posed data of ~1M objects, PF-LRM shows strong
cross-dataset generalization ability, and outperforms baseline methods by a
large margin in terms of pose prediction accuracy and 3D reconstruction quality
on various unseen evaluation datasets. We also demonstrate our model's
applicability in downstream text/image-to-3D task with fast feed-forward
inference. Our project website is at: https://totoro97.github.io/pf-lrm .
SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh
  Reconstruction and High-Quality Mesh Rendering
Authors: name: Antoine Guédon, name: Vincent Lepetit
Published: 2023-11-21 18:38:03

We propose a method to allow precise and extremely fast mesh extraction from
3D Gaussian Splatting. Gaussian Splatting has recently become very popular as
it yields realistic rendering while being significantly faster to train than
NeRFs. It is however challenging to extract a mesh from the millions of tiny 3D
gaussians as these gaussians tend to be unorganized after optimization and no
method has been proposed so far. Our first key contribution is a regularization
term that encourages the gaussians to align well with the surface of the scene.
We then introduce a method that exploits this alignment to extract a mesh from
the Gaussians using Poisson reconstruction, which is fast, scalable, and
preserves details, in contrast to the Marching Cubes algorithm usually applied
to extract meshes from Neural SDFs. Finally, we introduce an optional
refinement strategy that binds gaussians to the surface of the mesh, and
jointly optimizes these Gaussians and the mesh through Gaussian splatting
rendering. This enables easy editing, sculpting, rigging, animating,
compositing and relighting of the Gaussians using traditional softwares by
manipulating the mesh instead of the gaussians themselves. Retrieving such an
editable mesh for realistic rendering is done within minutes with our method,
compared to hours with the state-of-the-art methods on neural SDFs, while
providing a better rendering quality.
HierSpeech++: Bridging the Gap between Semantic and Acoustic
  Representation of Speech by Hierarchical Variational Inference for Zero-shot
  Speech Synthesis
Authors: name: Sang-Hoon Lee, name: Ha-Yeong Choi, name: Seung-Bin Kim, name: Seong-Whan Lee
Published: 2023-11-21 09:07:11

Large language models (LLM)-based speech synthesis has been widely adopted in
zero-shot speech synthesis. However, they require a large-scale data and
possess the same limitations as previous autoregressive speech models,
including slow inference speed and lack of robustness. This paper proposes
HierSpeech++, a fast and strong zero-shot speech synthesizer for text-to-speech
(TTS) and voice conversion (VC). We verified that hierarchical speech synthesis
frameworks could significantly improve the robustness and expressiveness of the
synthetic speech. Furthermore, we significantly improve the naturalness and
speaker similarity of synthetic speech even in zero-shot speech synthesis
scenarios. For text-to-speech, we adopt the text-to-vec framework, which
generates a self-supervised speech representation and an F0 representation
based on text representations and prosody prompts. Then, HierSpeech++ generates
speech from the generated vector, F0, and voice prompt. We further introduce a
high-efficient speech super-resolution framework from 16 kHz to 48 kHz. The
experimental results demonstrated that the hierarchical variational autoencoder
could be a strong zero-shot speech synthesizer given that it outperforms
LLM-based and diffusion-based models. Moreover, we achieved the first
human-level quality zero-shot speech synthesis. Audio samples and source code
are available at https://github.com/sh-lee-prml/HierSpeechpp.
PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics
Authors: name: Tianyi Xie, name: Zeshun Zong, name: Yuxin Qiu, name: Xuan Li, name: Yutao Feng, name: Yin Yang, name: Chenfanfu Jiang
Published: 2023-11-20 21:34:52

We introduce PhysGaussian, a new method that seamlessly integrates physically
grounded Newtonian dynamics within 3D Gaussians to achieve high-quality novel
motion synthesis. Employing a custom Material Point Method (MPM), our approach
enriches 3D Gaussian kernels with physically meaningful kinematic deformation
and mechanical stress attributes, all evolved in line with continuum mechanics
principles. A defining characteristic of our method is the seamless integration
between physical simulation and visual rendering: both components utilize the
same 3D Gaussian kernels as their discrete representations. This negates the
necessity for triangle/tetrahedron meshing, marching cubes, "cage meshes," or
any other geometry embedding, highlighting the principle of "what you see is
what you simulate (WS^2)." Our method demonstrates exceptional versatility
across a wide variety of materials--including elastic entities, metals,
non-Newtonian fluids, and granular materials--showcasing its strong
capabilities in creating diverse visual content with novel viewpoints and
movements. Our project page is at: https://xpandora.github.io/PhysGaussian/