Orionnnn/hub-stats huggingface dataset:
Large Language Models for Mathematicians
Authors: name: Simon Frieder, name: Julius Berner, name: Philipp Petersen, name: Thomas Lukasiewicz
Published: 2023-12-07 18:59:29

Large language models (LLMs) such as ChatGPT have received immense interest
for their general-purpose language understanding and, in particular, their
ability to generate high-quality text or computer code. For many professions,
LLMs represent an invaluable tool that can speed up and improve the quality of
work. In this note, we discuss to what extent they can aid professional
mathematicians. We first provide a mathematical description of the transformer
model used in all modern language models. Based on recent studies, we then
outline best practices and potential issues and report on the mathematical
abilities of language models. Finally, we shed light on the potential of LMMs
to change how mathematicians work.
DreamComposer: Controllable 3D Object Generation via Multi-View
  Conditions
Authors: name: Yunhan Yang, name: Yukun Huang, name: Xiaoyang Wu, name: Yuan-Chen Guo, name: Song-Hai Zhang, name: Hengshuang Zhao, name: Tong He, name: Xihui Liu
Published: 2023-12-06 16:55:53

Utilizing pre-trained 2D large-scale generative models, recent works are
capable of generating high-quality novel views from a single in-the-wild image.
However, due to the lack of information from multiple views, these works
encounter difficulties in generating controllable novel views. In this paper,
we present DreamComposer, a flexible and scalable framework that can enhance
existing view-aware diffusion models by injecting multi-view conditions.
Specifically, DreamComposer first uses a view-aware 3D lifting module to obtain
3D representations of an object from multiple views. Then, it renders the
latent features of the target view from 3D representations with the multi-view
feature fusion module. Finally the target view features extracted from
multi-view inputs are injected into a pre-trained diffusion model. Experiments
show that DreamComposer is compatible with state-of-the-art diffusion models
for zero-shot novel view synthesis, further enhancing them to generate
high-fidelity novel view images with multi-view conditions, ready for
controllable 3D object reconstruction and various other applications.
Schrodinger Bridges Beat Diffusion Models on Text-to-Speech Synthesis
Authors: name: Zehua Chen, name: Guande He, name: Kaiwen Zheng, name: Xu Tan, name: Jun Zhu
Published: 2023-12-06 13:31:55

In text-to-speech (TTS) synthesis, diffusion models have achieved promising
generation quality. However, because of the pre-defined data-to-noise diffusion
process, their prior distribution is restricted to a noisy representation,
which provides little information of the generation target. In this work, we
present a novel TTS system, Bridge-TTS, making the first attempt to substitute
the noisy Gaussian prior in established diffusion-based TTS methods with a
clean and deterministic one, which provides strong structural information of
the target. Specifically, we leverage the latent representation obtained from
text input as our prior, and build a fully tractable Schrodinger bridge between
it and the ground-truth mel-spectrogram, leading to a data-to-data process.
Moreover, the tractability and flexibility of our formulation allow us to
empirically study the design spaces such as noise schedules, as well as to
develop stochastic and deterministic samplers. Experimental results on the
LJ-Speech dataset illustrate the effectiveness of our method in terms of both
synthesis quality and sampling efficiency, significantly outperforming our
diffusion counterpart Grad-TTS in 50-step/1000-step synthesis and strong fast
TTS models in few-step scenarios. Project page: https://bridge-tts.github.io/
Gaussian Head Avatar: Ultra High-fidelity Head Avatar via Dynamic
  Gaussians
Authors: name: Yuelang Xu, name: Benwang Chen, name: Zhe Li, name: Hongwen Zhang, name: Lizhen Wang, name: Zerong Zheng, name: Yebin Liu
Published: 2023-12-05 11:01:44

Creating high-fidelity 3D head avatars has always been a research hotspot,
but there remains a great challenge under lightweight sparse view setups. In
this paper, we propose Gaussian Head Avatar represented by controllable 3D
Gaussians for high-fidelity head avatar modeling. We optimize the neutral 3D
Gaussians and a fully learned MLP-based deformation field to capture complex
expressions. The two parts benefit each other, thereby our method can model
fine-grained dynamic details while ensuring expression accuracy. Furthermore,
we devise a well-designed geometry-guided initialization strategy based on
implicit SDF and Deep Marching Tetrahedra for the stability and convergence of
the training procedure. Experiments show our approach outperforms other
state-of-the-art sparse-view methods, achieving ultra high-fidelity rendering
quality at 2K resolution even under exaggerated expressions.
HiFi4G: High-Fidelity Human Performance Rendering via Compact Gaussian
  Splatting
Authors: name: Yuheng Jiang, name: Zhehao Shen, name: Penghao Wang, name: Zhuo Su, name: Yu Hong, name: Yingliang Zhang, name: Jingyi Yu, name: Lan Xu
Published: 2023-12-06 12:36:53

We have recently seen tremendous progress in photo-real human modeling and
rendering. Yet, efficiently rendering realistic human performance and
integrating it into the rasterization pipeline remains challenging. In this
paper, we present HiFi4G, an explicit and compact Gaussian-based approach for
high-fidelity human performance rendering from dense footage. Our core
intuition is to marry the 3D Gaussian representation with non-rigid tracking,
achieving a compact and compression-friendly representation. We first propose a
dual-graph mechanism to obtain motion priors, with a coarse deformation graph
for effective initialization and a fine-grained Gaussian graph to enforce
subsequent constraints. Then, we utilize a 4D Gaussian optimization scheme with
adaptive spatial-temporal regularizers to effectively balance the non-rigid
prior and Gaussian updating. We also present a companion compression scheme
with residual compensation for immersive experiences on various platforms. It
achieves a substantial compression rate of approximately 25 times, with less
than 2MB of storage per frame. Extensive experiments demonstrate the
effectiveness of our approach, which significantly outperforms existing
approaches in terms of optimization speed, rendering quality, and storage
overhead.
HybridNeRF: Efficient Neural Rendering via Adaptive Volumetric Surfaces
Authors: name: Haithem Turki, name: Vasu Agrawal, name: Samuel Rota Bulò, name: Lorenzo Porzi, name: Peter Kontschieder, name: Deva Ramanan, name: Michael Zollhöfer, name: Christian Richardt
Published: 2023-12-05 22:04:49

Neural radiance fields provide state-of-the-art view synthesis quality but
tend to be slow to render. One reason is that they make use of volume
rendering, thus requiring many samples (and model queries) per ray at render
time. Although this representation is flexible and easy to optimize, most
real-world objects can be modeled more efficiently with surfaces instead of
volumes, requiring far fewer samples per ray. This observation has spurred
considerable progress in surface representations such as signed distance
functions, but these may struggle to model semi-opaque and thin structures. We
propose a method, HybridNeRF, that leverages the strengths of both
representations by rendering most objects as surfaces while modeling the
(typically) small fraction of challenging regions volumetrically. We evaluate
HybridNeRF against the challenging Eyeful Tower dataset along with other
commonly used view synthesis datasets. When comparing to state-of-the-art
baselines, including recent rasterization-based approaches, we improve error
rates by 15-30% while achieving real-time framerates (at least 36 FPS) for
virtual-reality resolutions (2Kx2K).
MotionCtrl: A Unified and Flexible Motion Controller for Video
  Generation
Authors: name: Zhouxia Wang, name: Ziyang Yuan, name: Xintao Wang, name: Tianshui Chen, name: Menghan Xia, name: Ping Luo, name: Ying Shan
Published: 2023-12-06 17:49:57

Motions in a video primarily consist of camera motion, induced by camera
movement, and object motion, resulting from object movement. Accurate control
of both camera and object motion is essential for video generation. However,
existing works either mainly focus on one type of motion or do not clearly
distinguish between the two, limiting their control capabilities and diversity.
Therefore, this paper presents MotionCtrl, a unified and flexible motion
controller for video generation designed to effectively and independently
control camera and object motion. The architecture and training strategy of
MotionCtrl are carefully devised, taking into account the inherent properties
of camera motion, object motion, and imperfect training data. Compared to
previous methods, MotionCtrl offers three main advantages: 1) It effectively
and independently controls camera motion and object motion, enabling more
fine-grained motion control and facilitating flexible and diverse combinations
of both types of motion. 2) Its motion conditions are determined by camera
poses and trajectories, which are appearance-free and minimally impact the
appearance or shape of objects in generated videos. 3) It is a relatively
generalizable model that can adapt to a wide array of camera poses and
trajectories once trained. Extensive qualitative and quantitative experiments
have been conducted to demonstrate the superiority of MotionCtrl over existing
methods.
MagicStick: Controllable Video Editing via Control Handle
  Transformations
Authors: name: Yue Ma, name: Xiaodong Cun, name: Yingqing He, name: Chenyang Qi, name: Xintao Wang, name: Ying Shan, name: Xiu Li, name: Qifeng Chen
Published: 2023-12-05 17:58:06

Text-based video editing has recently attracted considerable interest in
changing the style or replacing the objects with a similar structure. Beyond
this, we demonstrate that properties such as shape, size, location, motion,
etc., can also be edited in videos. Our key insight is that the keyframe
transformations of the specific internal feature (e.g., edge maps of objects or
human pose), can easily propagate to other frames to provide generation
guidance. We thus propose MagicStick, a controllable video editing method that
edits the video properties by utilizing the transformation on the extracted
internal control signals. In detail, to keep the appearance, we inflate both
the pretrained image diffusion model and ControlNet to the temporal dimension
and train low-rank adaptions (LORA) layers to fit the specific scenes. Then, in
editing, we perform an inversion and editing framework. Differently, finetuned
ControlNet is introduced in both inversion and generation for attention
guidance with the proposed attention remix between the spatial attention maps
of inversion and editing. Yet succinct, our method is the first method to show
the ability of video property editing from the pre-trained text-to-image model.
We present experiments on numerous examples within our unified framework. We
also compare with shape-aware text-based editing and handcrafted motion video
generation, demonstrating our superior temporal consistency and editing
capability than previous works. The code and models will be made publicly
available.
Self-conditioned Image Generation via Generating Representations
Authors: name: Tianhong Li, name: Dina Katabi, name: Kaiming He
Published: 2023-12-06 18:59:31

This paper presents Representation-Conditioned image
Generation (RCG), a simple yet effective image generation framework
which sets a new benchmark in class-unconditional image generation. RCG does
not condition on any human annotations. Instead, it conditions on a
self-supervised representation distribution which is mapped from the image
distribution using a pre-trained encoder. During generation, RCG samples from
such representation distribution using a representation diffusion model (RDM),
and employs a pixel generator to craft image pixels conditioned on the sampled
representation. Such a design provides substantial guidance during the
generative process, resulting in high-quality image generation. Tested on
ImageNet 256times256, RCG achieves a Frechet Inception Distance (FID) of
3.31 and an Inception Score (IS) of 253.4. These results not only significantly
improve the state-of-the-art of class-unconditional image generation but also
rival the current leading methods in class-conditional image generation,
bridging the long-standing performance gap between these two tasks. Code is
available at https://github.com/LTH14/rcg.
Context Diffusion: In-Context Aware Image Generation
Authors: name: Ivona Najdenkoska, name: Animesh Sinha, name: Abhimanyu Dubey, name: Dhruv Mahajan, name: Vignesh Ramanathan, name: Filip Radenovic
Published: 2023-12-06 16:19:51

We propose Context Diffusion, a diffusion-based framework that enables image
generation models to learn from visual examples presented in context. Recent
work tackles such in-context learning for image generation, where a query image
is provided alongside context examples and text prompts. However, the quality
and fidelity of the generated images deteriorate when the prompt is not
present, demonstrating that these models are unable to truly learn from the
visual context. To address this, we propose a novel framework that separates
the encoding of the visual context and preserving the structure of the query
images. This results in the ability to learn from the visual context and text
prompts, but also from either one of them. Furthermore, we enable our model to
handle few-shot settings, to effectively address diverse in-context learning
scenarios. Our experiments and user study demonstrate that Context Diffusion
excels in both in-domain and out-of-domain tasks, resulting in an overall
enhancement in image quality and fidelity compared to counterpart models.
Kandinsky 3.0 Technical Report
Authors: name: Vladimir Arkhipkin, name: Andrei Filatov, name: Viacheslav Vasilev, name: Anastasia Maltseva, name: Said Azizov, name: Igor Pavlov, name: Julia Agafonova, name: Andrey Kuznetsov, name: Denis Dimitrov
Published: 2023-12-06 14:13:38

We present Kandinsky 3.0, a large-scale text-to-image generation model based
on latent diffusion, continuing the series of text-to-image Kandinsky models
and reflecting our progress to achieve higher quality and realism of image
generation. Compared to previous versions of Kandinsky 2.x, Kandinsky 3.0
leverages a two times larger U-Net backbone, a ten times larger text encoder
and removes diffusion mapping. We describe the architecture of the model, the
data collection procedure, the training technique, and the production system of
user interaction. We focus on the key components that, as we have identified as
a result of a large number of experiments, had the most significant impact on
improving the quality of our model compared to the others. By our side-by-side
comparisons, Kandinsky becomes better in text understanding and works better on
specific domains. Project page: https://ai-forever.github.io/Kandinsky-3
Cache Me if You Can: Accelerating Diffusion Models through Block Caching
Authors: name: Felix Wimbauer, name: Bichen Wu, name: Edgar Schoenfeld, name: Xiaoliang Dai, name: Ji Hou, name: Zijian He, name: Artsiom Sanakoyeu, name: Peizhao Zhang, name: Sam Tsai, name: Jonas Kohler, name: Christian Rupprecht, name: Daniel Cremers, name: Peter Vajda, name: Jialiang Wang
Published: 2023-12-06 00:51:38

Diffusion models have recently revolutionized the field of image synthesis
due to their ability to generate photorealistic images. However, one of the
major drawbacks of diffusion models is that the image generation process is
costly. A large image-to-image network has to be applied many times to
iteratively refine an image from random noise. While many recent works propose
techniques to reduce the number of required steps, they generally treat the
underlying denoising network as a black box. In this work, we investigate the
behavior of the layers within the network and find that 1) the layers' output
changes smoothly over time, 2) the layers show distinct patterns of change, and
3) the change from step to step is often very small. We hypothesize that many
layer computations in the denoising network are redundant. Leveraging this, we
introduce block caching, in which we reuse outputs from layer blocks of
previous steps to speed up inference. Furthermore, we propose a technique to
automatically determine caching schedules based on each block's changes over
timesteps. In our experiments, we show through FID, human evaluation and
qualitative analysis that Block Caching allows to generate images with higher
visual quality at the same computational cost. We demonstrate this for
different state-of-the-art models (LDM and EMU) and solvers (DDIM and DPM).
LooseControl: Lifting ControlNet for Generalized Depth Conditioning
Authors: name: Shariq Farooq Bhat, name: Niloy J. Mitra, name: Peter Wonka
Published: 2023-12-05 19:00:20

We present LooseControl to allow generalized depth conditioning for
diffusion-based image generation. ControlNet, the SOTA for depth-conditioned
image generation, produces remarkable results but relies on having access to
detailed depth maps for guidance. Creating such exact depth maps, in many
scenarios, is challenging. This paper introduces a generalized version of depth
conditioning that enables many new content-creation workflows. Specifically, we
allow (C1) scene boundary control for loosely specifying scenes with only
boundary conditions, and (C2) 3D box control for specifying layout locations of
the target objects rather than the exact shape and appearance of the objects.
Using LooseControl, along with text guidance, users can create complex
environments (e.g., rooms, street views, etc.) by specifying only scene
boundaries and locations of primary objects. Further, we provide two editing
mechanisms to refine the results: (E1) 3D box editing enables the user to
refine images by changing, adding, or removing boxes while freezing the style
of the image. This yields minimal changes apart from changes induced by the
edited boxes. (E2) Attribute editing proposes possible editing directions to
change one particular aspect of the scene, such as the overall object density
or a particular object. Extensive tests and comparisons with baselines
demonstrate the generality of our method. We believe that LooseControl can
become an important design tool for easily creating complex environments and be
extended to other forms of guidance channels. Code and more information are
available at https://shariqfarooq123.github.io/loose-control/ .
Relightable Gaussian Codec Avatars
Authors: name: Shunsuke Saito, name: Gabriel Schwartz, name: Tomas Simon, name: Junxuan Li, name: Giljoo Nam
Published: 2023-12-06 18:59:58

The fidelity of relighting is bounded by both geometry and appearance
representations. For geometry, both mesh and volumetric approaches have
difficulty modeling intricate structures like 3D hair geometry. For appearance,
existing relighting models are limited in fidelity and often too slow to render
in real-time with high-resolution continuous environments. In this work, we
present Relightable Gaussian Codec Avatars, a method to build high-fidelity
relightable head avatars that can be animated to generate novel expressions.
Our geometry model based on 3D Gaussians can capture 3D-consistent
sub-millimeter details such as hair strands and pores on dynamic face
sequences. To support diverse materials of human heads such as the eyes, skin,
and hair in a unified manner, we present a novel relightable appearance model
based on learnable radiance transfer. Together with global illumination-aware
spherical harmonics for the diffuse components, we achieve real-time relighting
with spatially all-frequency reflections using spherical Gaussians. This
appearance model can be efficiently relit under both point light and continuous
illumination. We further improve the fidelity of eye reflections and enable
explicit gaze control by introducing relightable explicit eye models. Our
method outperforms existing approaches without compromising real-time
performance. We also demonstrate real-time relighting of avatars on a tethered
consumer VR headset, showcasing the efficiency and fidelity of our avatars.
OneLLM: One Framework to Align All Modalities with Language
Authors: name: Jiaming Han, name: Kaixiong Gong, name: Yiyuan Zhang, name: Jiaqi Wang, name: Kaipeng Zhang, name: Dahua Lin, name: Yu Qiao, name: Peng Gao, name: Xiangyu Yue
Published: 2023-12-06 18:59:19

Multimodal large language models (MLLMs) have gained significant attention
due to their strong multimodal understanding capability. However, existing
works rely heavily on modality-specific encoders, which usually differ in
architecture and are limited to common modalities. In this paper, we present
OneLLM, an MLLM that aligns eight modalities to language using a unified
framework. We achieve this through a unified multimodal encoder and a
progressive multimodal alignment pipeline. In detail, we first train an image
projection module to connect a vision encoder with LLM. Then, we build a
universal projection module (UPM) by mixing multiple image projection modules
and dynamic routing. Finally, we progressively align more modalities to LLM
with the UPM. To fully leverage the potential of OneLLM in following
instructions, we also curated a comprehensive multimodal instruction dataset,
including 2M items from image, audio, video, point cloud, depth/normal map, IMU
and fMRI brain activity. OneLLM is evaluated on 25 diverse benchmarks,
encompassing tasks such as multimodal captioning, question answering and
reasoning, where it delivers excellent performance. Code, data, model and
online demo are available at https://github.com/csuhan/OneLLM
Generative agent-based modeling with actions grounded in physical,
  social, or digital space using Concordia
Authors: name: Alexander Sasha Vezhnevets, name: John P. Agapiou, name: Avia Aharon, name: Ron Ziv, name: Jayd Matyas, name: Edgar A. Duéñez-Guzmán, name: William A. Cunningham, name: Simon Osindero, name: Danny Karmon, name: Joel Z. Leibo
Published: 2023-12-06 18:33:50

Agent-based modeling has been around for decades, and applied widely across
the social and natural sciences. The scope of this research method is now
poised to grow dramatically as it absorbs the new affordances provided by Large
Language Models (LLM)s. Generative Agent-Based Models (GABM) are not just
classic Agent-Based Models (ABM)s where the agents talk to one another. Rather,
GABMs are constructed using an LLM to apply common sense to situations, act
"reasonably", recall common semantic knowledge, produce API calls to control
digital technologies like apps, and communicate both within the simulation and
to researchers viewing it from the outside. Here we present Concordia, a
library to facilitate constructing and working with GABMs. Concordia makes it
easy to construct language-mediated simulations of physically- or
digitally-grounded environments. Concordia agents produce their behavior using
a flexible component system which mediates between two fundamental operations:
LLM calls and associative memory retrieval. A special agent called the Game
Master (GM), which was inspired by tabletop role-playing games, is responsible
for simulating the environment where the agents interact. Agents take actions
by describing what they want to do in natural language. The GM then translates
their actions into appropriate implementations. In a simulated physical world,
the GM checks the physical plausibility of agent actions and describes their
effects. In digital environments simulating technologies such as apps and
services, the GM may handle API calls to integrate with external tools such as
general AI assistants (e.g., Bard, ChatGPT), and digital apps (e.g., Calendar,
Email, Search, etc.). Concordia was designed to support a wide array of
applications both in scientific research and for evaluating performance of real
digital services by simulating users and/or generating synthetic data.