Orionnnn/hub-stats huggingface dataset:
LARP: Language-Agent Role Play for Open-World Games
Authors: name: Ming Yan, name: Ruihao Li, name: Hao Zhang, name: Hao Wang, name: Zhilan Yang, name: Ji Yan
Published: 2023-12-24 10:08:59

Language agents have shown impressive problem-solving skills within defined
settings and brief timelines. Yet, with the ever-evolving complexities of
open-world simulations, there's a pressing need for agents that can flexibly
adapt to complex environments and consistently maintain a long-term memory to
ensure coherent actions. To bridge the gap between language agents and
open-world games, we introduce Language Agent for Role-Playing (LARP), which
includes a cognitive architecture that encompasses memory processing and a
decision-making assistant, an environment interaction module with a
feedback-driven learnable action space, and a postprocessing method that
promotes the alignment of various personalities. The LARP framework refines
interactions between users and agents, predefined with unique backgrounds and
personalities, ultimately enhancing the gaming experience in open-world
contexts. Furthermore, it highlights the diverse uses of language models in a
range of areas such as entertainment, education, and various simulation
scenarios. The project page is released at https://miao-ai-lab.github.io/LARP/.
PanGu-π: Enhancing Language Model Architectures via Nonlinearity
  Compensation
Authors: name: Yunhe Wang, name: Hanting Chen, name: Yehui Tang, name: Tianyu Guo, name: Kai Han, name: Ying Nie, name: Xutao Wang, name: Hailin Hu, name: Zheyuan Bai, name: Yun Wang, name: Fangcheng Liu, name: Zhicheng Liu, name: Jianyuan Guo, name: Sinan Zeng, name: Yinchen Zhang, name: Qinghua Xu, name: Qun Liu, name: Jun Yao, name: Chao Xu, name: Dacheng Tao
Published: 2023-12-27 11:49:24

The recent trend of large language models (LLMs) is to increase the scale of
both model size (\aka the number of parameters) and dataset to achieve better
generative ability, which is definitely proved by a lot of work such as the
famous GPT and Llama. However, large models often involve massive computational
costs, and practical applications cannot afford such high prices. However, the
method of constructing a strong model architecture for LLMs is rarely
discussed. We first analyze the state-of-the-art language model architectures
and observe the feature collapse problem. Based on the theoretical analysis, we
propose that the nonlinearity is also very important for language models, which
is usually studied in convolutional neural networks for vision tasks. The
series informed activation function is then introduced with tiny calculations
that can be ignored, and an augmented shortcut is further used to enhance the
model nonlinearity. We then demonstrate that the proposed approach is
significantly effective for enhancing the model nonlinearity through carefully
designed ablations; thus, we present a new efficient model architecture for
establishing modern, namely, PanGu-pi. Experiments are then conducted using
the same dataset and training strategy to compare PanGu-pi with
state-of-the-art LLMs. The results show that PanGu-pi-7B can achieve a
comparable performance to that of benchmarks with about 10\% inference
speed-up, and PanGu-pi-1B can achieve state-of-the-art performance in terms
of accuracy and efficiency. In addition, we have deployed PanGu-pi-7B in the
high-value domains of finance and law, developing an LLM named YunShan for
practical application. The results show that YunShan can surpass other models
with similar scales on benchmarks.
Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language
  Models
Authors: name: Yuqing Wang, name: Yun Zhao
Published: 2023-12-29 15:57:49

The burgeoning interest in Multimodal Large Language Models (MLLMs), such as
OpenAI's GPT-4V(ision), has significantly impacted both academic and industrial
realms. These models enhance Large Language Models (LLMs) with advanced visual
understanding capabilities, facilitating their application in a variety of
multimodal tasks. Recently, Google introduced Gemini, a cutting-edge MLLM
designed specifically for multimodal integration. Despite its advancements,
preliminary benchmarks indicate that Gemini lags behind GPT models in
commonsense reasoning tasks. However, this assessment, based on a limited
dataset (i.e., HellaSWAG), does not fully capture Gemini's authentic
commonsense reasoning potential. To address this gap, our study undertakes a
thorough evaluation of Gemini's performance in complex reasoning tasks that
necessitate the integration of commonsense knowledge across modalities. We
carry out a comprehensive analysis of 12 commonsense reasoning datasets,
ranging from general to domain-specific tasks. This includes 11 datasets
focused solely on language, as well as one that incorporates multimodal
elements. Our experiments across four LLMs and two MLLMs demonstrate Gemini's
competitive commonsense reasoning capabilities. Additionally, we identify
common challenges faced by current LLMs and MLLMs in addressing commonsense
problems, underscoring the need for further advancements in enhancing the
commonsense reasoning abilities of these models.
Compact Neural Graphics Primitives with Learned Hash Probing
Authors: name: Towaki Takikawa, name: Thomas Müller, name: Merlin Nimier-David, name: Alex Evans, name: Sanja Fidler, name: Alec Jacobson, name: Alexander Keller
Published: 2023-12-28 18:58:45

Neural graphics primitives are faster and achieve higher quality when their
neural networks are augmented by spatial data structures that hold trainable
features arranged in a grid. However, existing feature grids either come with a
large memory footprint (dense or factorized grids, trees, and hash tables) or
slow performance (index learning and vector quantization). In this paper, we
show that a hash table with learned probes has neither disadvantage, resulting
in a favorable combination of size and speed. Inference is faster than unprobed
hash tables at equal quality while training is only 1.2-2.6x slower,
significantly outperforming prior index learning approaches. We arrive at this
formulation by casting all feature grids into a common framework: they each
correspond to a lookup function that indexes into a table of feature vectors.
In this framework, the lookup functions of existing data structures can be
combined by simple arithmetic combinations of their indices, resulting in
Pareto optimal compression and speed.
Restoration by Generation with Constrained Priors
Authors: name: Zheng Ding, name: Xuaner Zhang, name: Zhuowen Tu, name: Zhihao Xia
Published: 2023-12-28 17:50:54

The inherent generative power of denoising diffusion models makes them
well-suited for image restoration tasks where the objective is to find the
optimal high-quality image within the generative space that closely resembles
the input image. We propose a method to adapt a pretrained diffusion model for
image restoration by simply adding noise to the input image to be restored and
then denoise. Our method is based on the observation that the space of a
generative model needs to be constrained. We impose this constraint by
finetuning the generative model with a set of anchor images that capture the
characteristics of the input image. With the constrained space, we can then
leverage the sampling strategy used for generation to do image restoration. We
evaluate against previous methods and show superior performances on multiple
real-world restoration datasets in preserving identity and image quality. We
also demonstrate an important and practical application on personalized
restoration, where we use a personal album as the anchor images to constrain
the generative space. This approach allows us to produce results that
accurately preserve high-frequency details, which previous works are unable to
do. Project webpage: https://gen2res.github.io.
SSR-Encoder: Encoding Selective Subject Representation for
  Subject-Driven Generation
Authors: name: Yuxuan Zhang, name: Jiaming Liu, name: Yiren Song, name: Rui Wang, name: Hao Tang, name: Jinpeng Yu, name: Huaxia Li, name: Xu Tang, name: Yao Hu, name: Han Pan, name: Zhongliang Jing
Published: 2023-12-26 14:39:11

Recent advancements in subject-driven image generation have led to zero-shot
generation, yet precise selection and focus on crucial subject representations
remain challenging. Addressing this, we introduce the SSR-Encoder, a novel
architecture designed for selectively capturing any subject from single or
multiple reference images. It responds to various query modalities including
text and masks, without necessitating test-time fine-tuning. The SSR-Encoder
combines a Token-to-Patch Aligner that aligns query inputs with image patches
and a Detail-Preserving Subject Encoder for extracting and preserving fine
features of the subjects, thereby generating subject embeddings. These
embeddings, used in conjunction with original text embeddings, condition the
generation process. Characterized by its model generalizability and efficiency,
the SSR-Encoder adapts to a range of custom models and control modules.
Enhanced by the Embedding Consistency Regularization Loss for improved
training, our extensive experiments demonstrate its effectiveness in versatile
and high-quality image generation, indicating its broad applicability. Project
page: https://ssr-encoder.github.io
Hyper-VolTran: Fast and Generalizable One-Shot Image to 3D Object
  Structure via HyperNetworks
Authors: name: Christian Simon, name: Sen He, name: Juan-Manuel Perez-Rua, name: Frost Xu, name: Amine Benhalloum, name: Tao Xiang
Published: 2023-12-24 08:42:37

Solving image-to-3D from a single view is an ill-posed problem, and current
neural reconstruction methods addressing it through diffusion models still rely
on scene-specific optimization, constraining their generalization capability.
To overcome the limitations of existing approaches regarding generalization and
consistency, we introduce a novel neural rendering technique. Our approach
employs the signed distance function as the surface representation and
incorporates generalizable priors through geometry-encoding volumes and
HyperNetworks. Specifically, our method builds neural encoding volumes from
generated multi-view inputs. We adjust the weights of the SDF network
conditioned on an input image at test-time to allow model adaptation to novel
scenes in a feed-forward manner via HyperNetworks. To mitigate artifacts
derived from the synthesized views, we propose the use of a volume transformer
module to improve the aggregation of image features instead of processing each
viewpoint separately. Through our proposed method, dubbed as Hyper-VolTran, we
avoid the bottleneck of scene-specific optimization and maintain consistency
across the images generated from multiple viewpoints. Our experiments show the
advantages of our proposed approach with consistent results and rapid
generation.
InsActor: Instruction-driven Physics-based Characters
Authors: name: Jiawei Ren, name: Mingyuan Zhang, name: Cunjun Yu, name: Xiao Ma, name: Liang Pan, name: Ziwei Liu
Published: 2023-12-28 17:10:31

Generating animation of physics-based characters with intuitive control has
long been a desirable task with numerous applications. However, generating
physically simulated animations that reflect high-level human instructions
remains a difficult problem due to the complexity of physical environments and
the richness of human language. In this paper, we present InsActor, a
principled generative framework that leverages recent advancements in
diffusion-based human motion models to produce instruction-driven animations of
physics-based characters. Our framework empowers InsActor to capture complex
relationships between high-level human instructions and character motions by
employing diffusion policies for flexibly conditioned motion planning. To
overcome invalid states and infeasible state transitions in planned motions,
InsActor discovers low-level skills and maps plans to latent skill sequences in
a compact latent space. Extensive experiments demonstrate that InsActor
achieves state-of-the-art results on various tasks, including
instruction-driven motion generation and instruction-driven waypoint heading.
Notably, the ability of InsActor to generate physically simulated animations
using high-level human instructions makes it a valuable tool, particularly in
executing long-horizon tasks with a rich set of instructions.
Unsupervised Universal Image Segmentation
Authors: name: Dantong Niu, name: Xudong Wang, name: Xinyang Han, name: Long Lian, name: Roei Herzig, name: Trevor Darrell
Published: 2023-12-28 18:59:04

Several unsupervised image segmentation approaches have been proposed which
eliminate the need for dense manually-annotated segmentation masks; current
models separately handle either semantic segmentation (e.g., STEGO) or
class-agnostic instance segmentation (e.g., CutLER), but not both (i.e.,
panoptic segmentation). We propose an Unsupervised Universal Segmentation model
(U2Seg) adept at performing various image segmentation tasks -- instance,
semantic and panoptic -- using a novel unified framework. U2Seg generates
pseudo semantic labels for these segmentation tasks via leveraging
self-supervised models followed by clustering; each cluster represents
different semantic and/or instance membership of pixels. We then self-train the
model on these pseudo semantic labels, yielding substantial performance gains
over specialized methods tailored to each task: a +2.6 AP^{box} boost
vs. CutLER in unsupervised instance segmentation on COCO and a +7.0 PixelAcc
increase (vs. STEGO) in unsupervised semantic segmentation on COCOStuff.
Moreover, our method sets up a new baseline for unsupervised panoptic
segmentation, which has not been previously explored. U2Seg is also a strong
pretrained model for few-shot segmentation, surpassing CutLER by +5.0
AP^{mask} when trained on a low-data regime, e.g., only 1% COCO
labels. We hope our simple yet effective method can inspire more research on
unsupervised universal image segmentation.
Spacetime Gaussian Feature Splatting for Real-Time Dynamic View
  Synthesis
Authors: name: Zhan Li, name: Zhang Chen, name: Zhong Li, name: Yi Xu
Published: 2023-12-28 04:14:55

Novel view synthesis of dynamic scenes has been an intriguing yet challenging
problem. Despite recent advancements, simultaneously achieving high-resolution
photorealistic results, real-time rendering, and compact storage remains a
formidable task. To address these challenges, we propose Spacetime Gaussian
Feature Splatting as a novel dynamic scene representation, composed of three
pivotal components. First, we formulate expressive Spacetime Gaussians by
enhancing 3D Gaussians with temporal opacity and parametric motion/rotation.
This enables Spacetime Gaussians to capture static, dynamic, as well as
transient content within a scene. Second, we introduce splatted feature
rendering, which replaces spherical harmonics with neural features. These
features facilitate the modeling of view- and time-dependent appearance while
maintaining small size. Third, we leverage the guidance of training error and
coarse depth to sample new Gaussians in areas that are challenging to converge
with existing pipelines. Experiments on several established real-world datasets
demonstrate that our method achieves state-of-the-art rendering quality and
speed, while retaining compact storage. At 8K resolution, our lite-version
model can render at 60 FPS on an Nvidia RTX 4090 GPU.
DreamGaussian4D: Generative 4D Gaussian Splatting
Authors: name: Jiawei Ren, name: Liang Pan, name: Jiaxiang Tang, name: Chi Zhang, name: Ang Cao, name: Gang Zeng, name: Ziwei Liu
Published: 2023-12-28 17:16:44

Remarkable progress has been made in 4D content generation recently. However,
existing methods suffer from long optimization time, lack of motion
controllability, and a low level of detail. In this paper, we introduce
DreamGaussian4D, an efficient 4D generation framework that builds on 4D
Gaussian Splatting representation. Our key insight is that the explicit
modeling of spatial transformations in Gaussian Splatting makes it more
suitable for the 4D generation setting compared with implicit representations.
DreamGaussian4D reduces the optimization time from several hours to just a few
minutes, allows flexible control of the generated 3D motion, and produces
animated meshes that can be efficiently rendered in 3D engines.
City-on-Web: Real-time Neural Rendering of Large-scale Scenes on the Web
Authors: name: Kaiwen Song, name: Juyong Zhang
Published: 2023-12-27 08:00:47

NeRF has significantly advanced 3D scene reconstruction, capturing intricate
details across various environments. Existing methods have successfully
leveraged radiance field baking to facilitate real-time rendering of small
scenes. However, when applied to large-scale scenes, these techniques encounter
significant challenges, struggling to provide a seamless real-time experience
due to limited resources in computation, memory, and bandwidth. In this paper,
we propose City-on-Web, which represents the whole scene by partitioning it
into manageable blocks, each with its own Level-of-Detail, ensuring high
fidelity, efficient memory management and fast rendering. Meanwhile, we
carefully design the training and inference process such that the final
rendering result on web is consistent with training. Thanks to our novel
representation and carefully designed training/inference process, we are the
first to achieve real-time rendering of large-scale scenes in
resource-constrained environments. Extensive experimental results demonstrate
that our method facilitates real-time rendering of large-scale scenes on a web
platform, achieving 32FPS at 1080P resolution with an RTX 3060 GPU, while
simultaneously achieving a quality that closely rivals that of state-of-the-art
methods. Project page: https://ustc3dv.github.io/City-on-Web/
DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision
Authors: name: Lu Ling, name: Yichen Sheng, name: Zhi Tu, name: Wentian Zhao, name: Cheng Xin, name: Kun Wan, name: Lantao Yu, name: Qianyu Guo, name: Zixun Yu, name: Yawen Lu, name: Xuanmao Li, name: Xingpeng Sun, name: Rohan Ashok, name: Aniruddha Mukherjee, name: Hao Kang, name: Xiangrui Kong, name: Gang Hua, name: Tianti Zhang, name: Bedrich Benes, name: Aniket Bera
Published: 2023-12-26 01:12:12

We have witnessed significant progress in deep learning-based 3D vision,
ranging from neural radiance field (NeRF) based 3D representation learning to
applications in novel view synthesis (NVS). However, existing scene-level
datasets for deep learning-based 3D vision, limited to either synthetic
environments or a narrow selection of real-world scenes, are quite
insufficient. This insufficiency not only hinders a comprehensive benchmark of
existing methods but also caps what could be explored in deep learning-based 3D
analysis. To address this critical gap, we present DL3DV-10K, a large-scale
scene dataset, featuring 51.2 million frames from 10,510 videos captured from
65 types of point-of-interest (POI) locations, covering both bounded and
unbounded scenes, with different levels of reflection, transparency, and
lighting. We conducted a comprehensive benchmark of recent NVS methods on
DL3DV-10K, which revealed valuable insights for future research in NVS. In
addition, we have obtained encouraging results in a pilot study to learn
generalizable NeRF from DL3DV-10K, which manifests the necessity of a
large-scale scene-level dataset to forge a path toward a foundation model for
learning 3D representation. Our DL3DV-10K dataset, benchmark results, and
models will be publicly accessible at https://dl3dv-10k.github.io/DL3DV-10K/.
DiffusionGAN3D: Boosting Text-guided 3D Generation and Domain Adaption
  by Combining 3D GANs and Diffusion Priors
Authors: name: Biwen Lei, name: Kai Yu, name: Mengyang Feng, name: Miaomiao Cui, name: Xuansong Xie
Published: 2023-12-28 05:46:26

Text-guided domain adaption and generation of 3D-aware portraits find many
applications in various fields. However, due to the lack of training data and
the challenges in handling the high variety of geometry and appearance, the
existing methods for these tasks suffer from issues like inflexibility,
instability, and low fidelity. In this paper, we propose a novel framework
DiffusionGAN3D, which boosts text-guided 3D domain adaption and generation by
combining 3D GANs and diffusion priors. Specifically, we integrate the
pre-trained 3D generative models (e.g., EG3D) and text-to-image diffusion
models. The former provides a strong foundation for stable and high-quality
avatar generation from text. And the diffusion models in turn offer powerful
priors and guide the 3D generator finetuning with informative direction to
achieve flexible and efficient text-guided domain adaption. To enhance the
diversity in domain adaption and the generation capability in text-to-avatar,
we introduce the relative distance loss and case-specific learnable triplane
respectively. Besides, we design a progressive texture refinement module to
improve the texture quality for both tasks above. Extensive experiments
demonstrate that the proposed framework achieves excellent results in both
domain adaption and text-to-avatar tasks, outperforming existing methods in
terms of generation quality and efficiency. The project homepage is at
https://younglbw.github.io/DiffusionGAN3D-homepage/.
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision,
  Language, Audio, and Action
Authors: name: Jiasen Lu, name: Christopher Clark, name: Sangho Lee, name: Zichen Zhang, name: Savya Khosla, name: Ryan Marten, name: Derek Hoiem, name: Aniruddha Kembhavi
Published: 2023-12-28 17:57:06

We present Unified-IO 2, the first autoregressive multimodal model that is
capable of understanding and generating image, text, audio, and action. To
unify different modalities, we tokenize inputs and outputs -- images, text,
audio, action, bounding boxes, etc., into a shared semantic space and then
process them with a single encoder-decoder transformer model. Since training
with such diverse modalities is challenging, we propose various architectural
improvements to stabilize model training. We train our model from scratch on a
large multimodal pre-training corpus from diverse sources with a multimodal
mixture of denoisers objective. To learn an expansive set of skills, such as
following multimodal instructions, we construct and finetune on an ensemble of
120 datasets with prompts and augmentations. With a single unified model,
Unified-IO 2 achieves state-of-the-art performance on the GRIT benchmark and
strong results in more than 35 benchmarks, including image generation and
understanding, natural language understanding, video and audio understanding,
and robotic manipulation. We release all our models to the research community.
Prompt Expansion for Adaptive Text-to-Image Generation
Authors: name: Siddhartha Datta, name: Alexander Ku, name: Deepak Ramachandran, name: Peter Anderson
Published: 2023-12-27 21:12:21

Text-to-image generation models are powerful but difficult to use. Users
craft specific prompts to get better images, though the images can be
repetitive. This paper proposes a Prompt Expansion framework that helps users
generate high-quality, diverse images with less effort. The Prompt Expansion
model takes a text query as input and outputs a set of expanded text prompts
that are optimized such that when passed to a text-to-image model, generates a
wider variety of appealing images. We conduct a human evaluation study that
shows that images generated through Prompt Expansion are more aesthetically
pleasing and diverse than those generated by baseline methods. Overall, this
paper presents a novel and effective approach to improving the text-to-image
generation experience.