Orionnnn/hub-stats huggingface dataset:
Mirror-NeRF: Learning Neural Radiance Fields for Mirrors with
  Whitted-Style Ray Tracing
Authors: name: Junyi Zeng, name: Chong Bao, name: Rui Chen, name: Zilong Dong, name: Guofeng Zhang, name: Hujun Bao, name: Zhaopeng Cui
Published: 2023-08-07 03:48:07

Recently, Neural Radiance Fields (NeRF) has exhibited significant success in
novel view synthesis, surface reconstruction, etc. However, since no physical
reflection is considered in its rendering pipeline, NeRF mistakes the
reflection in the mirror as a separate virtual scene, leading to the inaccurate
reconstruction of the mirror and multi-view inconsistent reflections in the
mirror. In this paper, we present a novel neural rendering framework, named
Mirror-NeRF, which is able to learn accurate geometry and reflection of the
mirror and support various scene manipulation applications with mirrors, such
as adding new objects or mirrors into the scene and synthesizing the
reflections of these new objects in mirrors, controlling mirror roughness, etc.
To achieve this goal, we propose a unified radiance field by introducing the
reflection probability and tracing rays following the light transport model of
Whitted Ray Tracing, and also develop several techniques to facilitate the
learning process. Experiments and comparisons on both synthetic and real
datasets demonstrate the superiority of our method. The code and supplementary
material are available on the project webpage:
https://zju3dv.github.io/Mirror-NeRF/.
SynJax: Structured Probability Distributions for JAX
Authors: name: Miloš Stanojević, name: Laurent Sartran
Published: 2023-08-07 04:20:38

The development of deep learning software libraries enabled significant
progress in the field by allowing users to focus on modeling, while letting the
library to take care of the tedious and time-consuming task of optimizing
execution for modern hardware accelerators. However, this has benefited only
particular types of deep learning models, such as Transformers, whose
primitives map easily to the vectorized computation. The models that explicitly
account for structured objects, such as trees and segmentations, did not
benefit equally because they require custom algorithms that are difficult to
implement in a vectorized form.
  SynJax directly addresses this problem by providing an efficient vectorized
implementation of inference algorithms for structured distributions covering
alignment, tagging, segmentation, constituency trees and spanning trees. With
SynJax we can build large-scale differentiable models that explicitly model
structure in the data. The code is available at
https://github.com/deepmind/synjax.
FLIQS: One-Shot Mixed-Precision Floating-Point and Integer Quantization
  Search
Authors: name: Jordan Dotzel, name: Gang Wu, name: Andrew Li, name: Muhammad Umar, name: Yun Ni, name: Mohamed S. Abdelfattah, name: Zhiru Zhang, name: Liqun Cheng, name: Martin G. Dixon, name: Norman P. Jouppi, name: Quoc V. Le, name: Sheng Li
Published: 2023-08-07 04:17:19

Quantization has become a mainstream compression technique for reducing model
size, computational requirements, and energy consumption for modern deep neural
networks (DNNs). With the improved numerical support in recent hardware,
including multiple variants of integer and floating point, mixed-precision
quantization has become necessary to achieve high-quality results with low
model cost. Prior mixed-precision quantization methods have performed a
post-training quantization search, which compromises on accuracy, or a
differentiable quantization search, which leads to high memory usage from
branching. Therefore, we propose the first one-shot mixed-precision
quantization search that eliminates the need for retraining in both integer and
low-precision floating point models. We evaluate our floating-point and integer
quantization search (FLIQS) on multiple convolutional networks and vision
transformer models to discover Pareto-optimal models. Our approach discovers
models that improve upon uniform precision, manual mixed-precision, and recent
integer quantization search methods. With the proposed integer quantization
search, we increase the accuracy of ResNet-18 on ImageNet by 1.31% points and
ResNet-50 by 0.90% points with equivalent model cost over previous methods.
Additionally, for the first time, we explore a novel mixed-precision
floating-point search and improve MobileNetV2 by up to 0.98% points compared to
prior state-of-the-art FP8 models. Finally, we extend FLIQS to simultaneously
search a joint quantization and neural architecture space and improve the
ImageNet accuracy by 2.69% points with similar model cost on a MobileNetV2
search space.
Seeing through the Brain: Image Reconstruction of Visual Perception from
  Human Brain Signals
Authors: name: Yu-Ting Lan, name: Kan Ren, name: Yansen Wang, name: Wei-Long Zheng, name: Dongsheng Li, name: Bao-Liang Lu, name: Lili Qiu
Published: 2023-07-27 12:54:16

Seeing is believing, however, the underlying mechanism of how human visual
perceptions are intertwined with our cognitions is still a mystery. Thanks to
the recent advances in both neuroscience and artificial intelligence, we have
been able to record the visually evoked brain activities and mimic the visual
perception ability through computational approaches. In this paper, we pay
attention to visual stimuli reconstruction by reconstructing the observed
images based on portably accessible brain signals, i.e., electroencephalography
(EEG) data. Since EEG signals are dynamic in the time-series format and are
notorious to be noisy, processing and extracting useful information requires
more dedicated efforts; In this paper, we propose a comprehensive pipeline,
named NeuroImagen, for reconstructing visual stimuli images from EEG signals.
Specifically, we incorporate a novel multi-level perceptual information
decoding to draw multi-grained outputs from the given EEG data. A latent
diffusion model will then leverage the extracted information to reconstruct the
high-resolution visual stimuli images. The experimental results have
illustrated the effectiveness of image reconstruction and superior quantitative
performance of our proposed method.
AvatarVerse: High-quality & Stable 3D Avatar Creation from Text and Pose
Authors: name: Huichao Zhang, name: Bowen Chen, name: Hao Yang, name: Liao Qu, name: Xu Wang, name: Li Chen, name: Chao Long, name: Feida Zhu, name: Kang Du, name: Min Zheng
Published: 2023-08-07 14:09:46

Creating expressive, diverse and high-quality 3D avatars from highly
customized text descriptions and pose guidance is a challenging task, due to
the intricacy of modeling and texturing in 3D that ensure details and various
styles (realistic, fictional, etc). We present AvatarVerse, a stable pipeline
for generating expressive high-quality 3D avatars from nothing but text
descriptions and pose guidance. In specific, we introduce a 2D diffusion model
conditioned on DensePose signal to establish 3D pose control of avatars through
2D images, which enhances view consistency from partially observed scenarios.
It addresses the infamous Janus Problem and significantly stablizes the
generation process. Moreover, we propose a progressive high-resolution 3D
synthesis strategy, which obtains substantial improvement over the quality of
the created 3D avatars. To this end, the proposed AvatarVerse pipeline achieves
zero-shot 3D modeling of 3D avatars that are not only more expressive, but also
in higher quality and fidelity than previous works. Rigorous qualitative
evaluations and user studies showcase AvatarVerse's superiority in synthesizing
high-fidelity 3D avatars, leading to a new standard in high-quality and stable
3D avatar creation. Our project page is: https://avatarverse3d.github.io
From Discrete Tokens to High-Fidelity Audio Using Multi-Band Diffusion
Authors: name: Robin San Roman, name: Yossi Adi, name: Antoine Deleforge, name: Romain Serizel, name: Gabriel Synnaeve, name: Alexandre Défossez
Published: 2023-08-02 22:14:29

Deep generative models can generate high-fidelity audio conditioned on
various types of representations (e.g., mel-spectrograms, Mel-frequency
Cepstral Coefficients (MFCC)). Recently, such models have been used to
synthesize audio waveforms conditioned on highly compressed representations.
Although such methods produce impressive results, they are prone to generate
audible artifacts when the conditioning is flawed or imperfect. An alternative
modeling approach is to use diffusion models. However, these have mainly been
used as speech vocoders (i.e., conditioned on mel-spectrograms) or generating
relatively low sampling rate signals. In this work, we propose a high-fidelity
multi-band diffusion-based framework that generates any type of audio modality
(e.g., speech, music, environmental sounds) from low-bitrate discrete
representations. At equal bit rate, the proposed approach outperforms
state-of-the-art generative techniques in terms of perceptual quality. Training
and, evaluation code, along with audio samples, are available on the
facebookresearch/audiocraft Github page.
ConceptLab: Creative Generation using Diffusion Prior Constraints
Authors: name: Elad Richardson, name: Kfir Goldberg, name: Yuval Alaluf, name: Daniel Cohen-Or
Published: 2023-08-03 17:04:41

Recent text-to-image generative models have enabled us to transform our words
into vibrant, captivating imagery. The surge of personalization techniques that
has followed has also allowed us to imagine unique concepts in new scenes.
However, an intriguing question remains: How can we generate a new, imaginary
concept that has never been seen before? In this paper, we present the task of
creative text-to-image generation, where we seek to generate new members of a
broad category (e.g., generating a pet that differs from all existing pets). We
leverage the under-studied Diffusion Prior models and show that the creative
generation problem can be formulated as an optimization process over the output
space of the diffusion prior, resulting in a set of "prior constraints". To
keep our generated concept from converging into existing members, we
incorporate a question-answering model that adaptively adds new constraints to
the optimization problem, encouraging the model to discover increasingly more
unique creations. Finally, we show that our prior constraints can also serve as
a strong mixing mechanism allowing us to create hybrids between generated
concepts, introducing even more flexibility into the creative process.
AgentBench: Evaluating LLMs as Agents
Authors: name: Xiao Liu, name: Hao Yu, name: Hanchen Zhang, name: Yifan Xu, name: Xuanyu Lei, name: Hanyu Lai, name: Yu Gu, name: Hangliang Ding, name: Kaiwen Men, name: Kejuan Yang, name: Shudan Zhang, name: Xiang Deng, name: Aohan Zeng, name: Zhengxiao Du, name: Chenhui Zhang, name: Sheng Shen, name: Tianjun Zhang, name: Yu Su, name: Huan Sun, name: Minlie Huang, name: Yuxiao Dong, name: Jie Tang
Published: 2023-08-07 16:08:11

Large Language Models (LLMs) are becoming increasingly smart and autonomous,
targeting real-world pragmatic missions beyond traditional NLP tasks. As a
result, there has been an urgent need to evaluate LLMs as agents on challenging
tasks in interactive environments. We present AgentBench, a multi-dimensional
evolving benchmark that currently consists of 8 distinct environments to assess
LLM-as-Agent's reasoning and decision-making abilities in a multi-turn
open-ended generation setting. Our extensive test over 25 LLMs (including APIs
and open-sourced models) shows that, while top commercial LLMs present a strong
ability of acting as agents in complex environments, there is a significant
disparity in performance between them and open-sourced competitors. It also
serves as a component of an ongoing project with wider coverage and deeper
consideration towards systematic LLM evaluation. Datasets, environments, and an
integrated evaluation package for AgentBench are released at
https://github.com/THUDM/AgentBench
TPTU: Task Planning and Tool Usage of Large Language Model-based AI
  Agents
Authors: name: Jingqing Ruan, name: Yihong Chen, name: Bin Zhang, name: Zhiwei Xu, name: Tianpeng Bao, name: Guoqing Du, name: Shiwei Shi, name: Hangyu Mao, name: Xingyu Zeng, name: Rui Zhao
Published: 2023-08-07 09:22:03

With recent advancements in natural language processing, Large Language
Models (LLMs) have emerged as powerful tools for various real-world
applications. Despite their prowess, the intrinsic generative abilities of LLMs
may prove insufficient for handling complex tasks which necessitate a
combination of task planning and the usage of external tools. In this paper, we
first propose a structured framework tailored for LLM-based AI Agents and
discuss the crucial capabilities necessary for tackling intricate problems.
Within this framework, we design two distinct types of agents (i.e., one-step
agent and sequential agent) to execute the inference process. Subsequently, we
instantiate the framework using various LLMs and evaluate their Task Planning
and Tool Usage (TPTU) abilities on typical tasks. By highlighting key findings
and challenges, our goal is to provide a helpful resource for researchers and
practitioners to leverage the power of LLMs in their AI applications. Our study
emphasizes the substantial potential of these models, while also identifying
areas that need more investigation and improvement.
RecycleGPT: An Autoregressive Language Model with Recyclable Module
Authors: name: Yufan Jiang, name: Qiaozhi He, name: Xiaomin Zhuang, name: Zhihua Wu, name: Kunpeng Wang, name: Wenlai Zhao, name: Guangwen Yang
Published: 2023-08-07 09:14:33

Existing large language models have to run K times to generate a sequence of
K tokens. In this paper, we present RecycleGPT, a generative language model
with fast decoding speed by recycling pre-generated model states without
running the whole model in multiple steps. Our approach relies on the
observation that adjacent tokens in a sequence usually have strong correlations
and the next token in a sequence can be reasonably guessed or inferred based on
the preceding ones. Through theoretical evaluations and practical tests on
downstream text generation tasks, we demonstrate the effectiveness of our
approach in lowering inference latency, achieving up to 1.4x speedup while
preserving high performance.
Studying Large Language Model Generalization with Influence Functions
Authors: name: Roger Grosse, name: Juhan Bae, name: Cem Anil, name: Nelson Elhage, name: Alex Tamkin, name: Amirhossein Tajdini, name: Benoit Steiner, name: Dustin Li, name: Esin Durmus, name: Ethan Perez, name: Evan Hubinger, name: Kamilė Lukošiūtė, name: Karina Nguyen, name: Nicholas Joseph, name: Sam McCandlish, name: Jared Kaplan, name: Samuel R. Bowman
Published: 2023-08-07 04:47:42

When trying to gain better visibility into a machine learning model in order
to understand and mitigate the associated risks, a potentially valuable source
of evidence is: which training examples most contribute to a given behavior?
Influence functions aim to answer a counterfactual: how would the model's
parameters (and hence its outputs) change if a given sequence were added to the
training set? While influence functions have produced insights for small
models, they are difficult to scale to large language models (LLMs) due to the
difficulty of computing an inverse-Hessian-vector product (IHVP). We use the
Eigenvalue-corrected Kronecker-Factored Approximate Curvature (EK-FAC)
approximation to scale influence functions up to LLMs with up to 52 billion
parameters. In our experiments, EK-FAC achieves similar accuracy to traditional
influence function estimators despite the IHVP computation being orders of
magnitude faster. We investigate two algorithmic techniques to reduce the cost
of computing gradients of candidate training sequences: TF-IDF filtering and
query batching. We use influence functions to investigate the generalization
patterns of LLMs, including the sparsity of the influence patterns, increasing
abstraction with scale, math and programming abilities, cross-lingual
generalization, and role-playing behavior. Despite many apparently
sophisticated forms of generalization, we identify a surprising limitation:
influences decay to near-zero when the order of key phrases is flipped.
Overall, influence functions give us a powerful new tool for studying the
generalization properties of LLMs.
Tiny LVLM-eHub: Early Multimodal Experiments with Bard
Authors: name: Wenqi Shao, name: Yutao Hu, name: Peng Gao, name: Meng Lei, name: Kaipeng Zhang, name: Fanqing Meng, name: Peng Xu, name: Siyuan Huang, name: Hongsheng Li, name: Yu Qiao, name: Ping Luo
Published: 2023-08-07 17:17:05

Recent advancements in Large Vision-Language Models (LVLMs) have demonstrated
significant progress in tackling complex multimodal tasks. Among these
cutting-edge developments, Google's Bard stands out for its remarkable
multimodal capabilities, promoting comprehensive comprehension and reasoning
across various domains. This work presents an early and holistic evaluation of
LVLMs' multimodal abilities, with a particular focus on Bard, by proposing a
lightweight variant of LVLM-eHub, named Tiny LVLM-eHub. In comparison to the
vanilla version, Tiny LVLM-eHub possesses several appealing properties.
Firstly, it provides a systematic assessment of six categories of multimodal
capabilities, including visual perception, visual knowledge acquisition, visual
reasoning, visual commonsense, object hallucination, and embodied intelligence,
through quantitative evaluation of 42 standard text-related visual
benchmarks. Secondly, it conducts an in-depth analysis of LVLMs' predictions
using the ChatGPT Ensemble Evaluation (CEE), which leads to a robust and
accurate evaluation and exhibits improved alignment with human evaluation
compared to the word matching approach. Thirdly, it comprises a mere 2.1K
image-text pairs, facilitating ease of use for practitioners to evaluate their
own offline LVLMs. Through extensive experimental analysis, this study
demonstrates that Bard outperforms previous LVLMs in most multimodal
capabilities except object hallucination, to which Bard is still susceptible.
Tiny LVLM-eHub serves as a baseline evaluation for various LVLMs and encourages
innovative strategies aimed at advancing multimodal techniques. Our project is
publicly available at https://github.com/OpenGVLab/Multi-Modality-Arena.
AlphaStar Unplugged: Large-Scale Offline Reinforcement Learning
Authors: name: Michaël Mathieu, name: Sherjil Ozair, name: Srivatsan Srinivasan, name: Caglar Gulcehre, name: Shangtong Zhang, name: Ray Jiang, name: Tom Le Paine, name: Richard Powell, name: Konrad Żołna, name: Julian Schrittwieser, name: David Choi, name: Petko Georgiev, name: Daniel Toyama, name: Aja Huang, name: Roman Ring, name: Igor Babuschkin, name: Timo Ewalds, name: Mahyar Bordbar, name: Sarah Henderson, name: Sergio Gómez Colmenarejo, name: Aäron van den Oord, name: Wojciech Marian Czarnecki, name: Nando de Freitas, name: Oriol Vinyals
Published: 2023-08-07 12:21:37

StarCraft II is one of the most challenging simulated reinforcement learning
environments; it is partially observable, stochastic, multi-agent, and
mastering StarCraft II requires strategic planning over long time horizons with
real-time low-level execution. It also has an active professional competitive
scene. StarCraft II is uniquely suited for advancing offline RL algorithms,
both because of its challenging nature and because Blizzard has released a
massive dataset of millions of StarCraft II games played by human players. This
paper leverages that and establishes a benchmark, called AlphaStar Unplugged,
introducing unprecedented challenges for offline reinforcement learning. We
define a dataset (a subset of Blizzard's release), tools standardizing an API
for machine learning methods, and an evaluation protocol. We also present
baseline agents, including behavior cloning, offline variants of actor-critic
and MuZero. We improve the state of the art of agents using only offline data,
and we achieve 90% win rate against previously published AlphaStar behavior
cloning agent.
UniversalNER: Targeted Distillation from Large Language Models for Open
  Named Entity Recognition
Authors: name: Wenxuan Zhou, name: Sheng Zhang, name: Yu Gu, name: Muhao Chen, name: Hoifung Poon
Published: 2023-08-07 03:39:52

Large language models (LLMs) have demonstrated remarkable generalizability,
such as understanding arbitrary entities and relations. Instruction tuning has
proven effective for distilling LLMs into more cost-efficient models such as
Alpaca and Vicuna. Yet such student models still trail the original LLMs by
large margins in downstream applications. In this paper, we explore targeted
distillation with mission-focused instruction tuning to train student models
that can excel in a broad application class such as open information
extraction. Using named entity recognition (NER) for case study, we show how
ChatGPT can be distilled into much smaller UniversalNER models for open NER.
For evaluation, we assemble the largest NER benchmark to date, comprising 43
datasets across 9 diverse domains such as biomedicine, programming, social
media, law, finance. Without using any direct supervision, UniversalNER attains
remarkable NER accuracy across tens of thousands of entity types, outperforming
general instruction-tuned models such as Alpaca and Vicuna by over 30 absolute
F1 points in average. With a tiny fraction of parameters, UniversalNER not only
acquires ChatGPT's capability in recognizing arbitrary entity types, but also
outperforms its NER accuracy by 7-9 absolute F1 points in average. Remarkably,
UniversalNER even outperforms by a large margin state-of-the-art multi-task
instruction-tuned systems such as InstructUIE, which uses supervised NER
examples. We also conduct thorough ablation studies to assess the impact of
various components in our distillation approach. We will release the
distillation recipe, data, and UniversalNER models to facilitate future
research on targeted distillation.
Pre-Trained Large Language Models for Industrial Control
Authors: name: Lei Song, name: Chuheng Zhang, name: Li Zhao, name: Jiang Bian
Published: 2023-08-06 06:01:18

For industrial control, developing high-performance controllers with few
samples and low technical debt is appealing. Foundation models, possessing rich
prior knowledge obtained from pre-training with Internet-scale corpus, have the
potential to be a good controller with proper prompts. In this paper, we take
HVAC (Heating, Ventilation, and Air Conditioning) building control as an
example to examine the ability of GPT-4 (one of the first-tier foundation
models) as the controller. To control HVAC, we wrap the task as a language game
by providing text including a short description for the task, several selected
demonstrations, and the current observation to GPT-4 on each step and execute
the actions responded by GPT-4. We conduct series of experiments to answer the
following questions: 1)~How well can GPT-4 control HVAC? 2)~How well can GPT-4
generalize to different scenarios for HVAC control? 3) How different parts of
the text context affect the performance? In general, we found GPT-4 achieves
the performance comparable to RL methods with few samples and low technical
debt, indicating the potential of directly applying foundation models to
industrial control tasks.