Orionnnn/hub-stats huggingface dataset:
JaxMARL: Multi-Agent RL Environments in JAX
Authors: name: Alexander Rutherford, name: Benjamin Ellis, name: Matteo Gallici, name: Jonathan Cook, name: Andrei Lupu, name: Gardar Ingvarsson, name: Timon Willi, name: Akbir Khan, name: Christian Schroeder de Witt, name: Alexandra Souly, name: Saptarashmi Bandyopadhyay, name: Mikayel Samvelyan, name: Minqi Jiang, name: Robert Tjarko Lange, name: Shimon Whiteson, name: Bruno Lacerda, name: Nick Hawes, name: Tim Rocktaschel, name: Chris Lu, name: Jakob Nicolaus Foerster
Published: 2023-11-16 18:58:43

Benchmarks play an important role in the development of machine learning
algorithms. For example, research in reinforcement learning (RL) has been
heavily influenced by available environments and benchmarks. However, RL
environments are traditionally run on the CPU, limiting their scalability with
typical academic compute. Recent advancements in JAX have enabled the wider use
of hardware acceleration to overcome these computational hurdles, enabling
massively parallel RL training pipelines and environments. This is particularly
useful for multi-agent reinforcement learning (MARL) research. First of all,
multiple agents must be considered at each environment step, adding
computational burden, and secondly, the sample complexity is increased due to
non-stationarity, decentralised partial observability, or other MARL
challenges. In this paper, we present JaxMARL, the first open-source code base
that combines ease-of-use with GPU enabled efficiency, and supports a large
number of commonly used MARL environments as well as popular baseline
algorithms. When considering wall clock time, our experiments show that per-run
our JAX-based training pipeline is up to 12500x faster than existing
approaches. This enables efficient and thorough evaluations, with the potential
to alleviate the evaluation crisis of the field. We also introduce and
benchmark SMAX, a vectorised, simplified version of the popular StarCraft
Multi-Agent Challenge, which removes the need to run the StarCraft II game
engine. This not only enables GPU acceleration, but also provides a more
flexible MARL environment, unlocking the potential for self-play,
meta-learning, and other future applications in MARL. We provide code at
https://github.com/flairox/jaxmarl.
Open-Sourcing Highly Capable Foundation Models: An evaluation of risks,
  benefits, and alternative methods for pursuing open-source objectives
Authors: name: Elizabeth Seger, name: Noemi Dreksler, name: Richard Moulange, name: Emily Dardaman, name: Jonas Schuett, name: K. Wei, name: Christoph Winter, name: Mackenzie Arnold, name: Seán Ó hÉigeartaigh, name: Anton Korinek, name: Markus Anderljung, name: Ben Bucknall, name: Alan Chan, name: Eoghan Stafford, name: Leonie Koessler, name: Aviv Ovadya, name: Ben Garfinkel, name: Emma Bluemke, name: Michael Aird, name: Patrick Levermore, name: Julian Hazell, name: Abhishek Gupta
Published: 2023-09-29 17:03:45

Recent decisions by leading AI labs to either open-source their models or to
restrict access to their models has sparked debate about whether, and how,
increasingly capable AI models should be shared. Open-sourcing in AI typically
refers to making model architecture and weights freely and publicly accessible
for anyone to modify, study, build on, and use. This offers advantages such as
enabling external oversight, accelerating progress, and decentralizing control
over AI development and use. However, it also presents a growing potential for
misuse and unintended consequences. This paper offers an examination of the
risks and benefits of open-sourcing highly capable foundation models. While
open-sourcing has historically provided substantial net benefits for most
software and AI development processes, we argue that for some highly capable
foundation models likely to be developed in the near future, open-sourcing may
pose sufficiently extreme risks to outweigh the benefits. In such a case,
highly capable foundation models should not be open-sourced, at least not
initially. Alternative strategies, including non-open-source model sharing
options, are explored. The paper concludes with recommendations for developers,
standard-setting bodies, and governments for establishing safe and responsible
model sharing practices and preserving open-source benefits where safe.
Adaptive Shells for Efficient Neural Radiance Field Rendering
Authors: name: Zian Wang, name: Tianchang Shen, name: Merlin Nimier-David, name: Nicholas Sharp, name: Jun Gao, name: Alexander Keller, name: Sanja Fidler, name: Thomas Müller, name: Zan Gojcic
Published: 2023-11-16 18:58:55

Neural radiance fields achieve unprecedented quality for novel view
synthesis, but their volumetric formulation remains expensive, requiring a huge
number of samples to render high-resolution images. Volumetric encodings are
essential to represent fuzzy geometry such as foliage and hair, and they are
well-suited for stochastic optimization. Yet, many scenes ultimately consist
largely of solid surfaces which can be accurately rendered by a single sample
per pixel. Based on this insight, we propose a neural radiance formulation that
smoothly transitions between volumetric- and surface-based rendering, greatly
accelerating rendering speed and even improving visual fidelity. Our method
constructs an explicit mesh envelope which spatially bounds a neural volumetric
representation. In solid regions, the envelope nearly converges to a surface
and can often be rendered with a single sample. To this end, we generalize the
NeuS formulation with a learned spatially-varying kernel size which encodes the
spread of the density, fitting a wide kernel to volume-like regions and a tight
kernel to surface-like regions. We then extract an explicit mesh of a narrow
band around the surface, with width determined by the kernel size, and
fine-tune the radiance field within this band. At inference time, we cast rays
against the mesh and evaluate the radiance field only within the enclosed
region, greatly reducing the number of samples required. Experiments show that
our approach enables efficient rendering at very high fidelity. We also
demonstrate that the extracted envelope enables downstream applications such as
animation and simulation.
The Chosen One: Consistent Characters in Text-to-Image Diffusion Models
Authors: name: Omri Avrahami, name: Amir Hertz, name: Yael Vinker, name: Moab Arar, name: Shlomi Fruchter, name: Ohad Fried, name: Daniel Cohen-Or, name: Dani Lischinski
Published: 2023-11-16 18:59:51

Recent advances in text-to-image generation models have unlocked vast
potential for visual creativity. However, these models struggle with generation
of consistent characters, a crucial aspect for numerous real-world applications
such as story visualization, game development asset design, advertising, and
more. Current methods typically rely on multiple pre-existing images of the
target character or involve labor-intensive manual processes. In this work, we
propose a fully automated solution for consistent character generation, with
the sole input being a text prompt. We introduce an iterative procedure that,
at each stage, identifies a coherent set of images sharing a similar identity
and extracts a more consistent identity from this set. Our quantitative
analysis demonstrates that our method strikes a better balance between prompt
alignment and identity consistency compared to the baseline methods, and these
findings are reinforced by a user study. To conclude, we showcase several
practical applications of our approach. Project page is available at
https://omriavrahami.com/the-chosen-one
UFOGen: You Forward Once Large Scale Text-to-Image Generation via
  Diffusion GANs
Authors: name: Yanwu Xu, name: Yang Zhao, name: Zhisheng Xiao, name: Tingbo Hou
Published: 2023-11-14 23:07:50

Text-to-image diffusion models have demonstrated remarkable capabilities in
transforming textual prompts into coherent images, yet the computational cost
of their inference remains a persistent challenge. To address this issue, we
present UFOGen, a novel generative model designed for ultra-fast, one-step
text-to-image synthesis. In contrast to conventional approaches that focus on
improving samplers or employing distillation techniques for diffusion models,
UFOGen adopts a hybrid methodology, integrating diffusion models with a GAN
objective. Leveraging a newly introduced diffusion-GAN objective and
initialization with pre-trained diffusion models, UFOGen excels in efficiently
generating high-quality images conditioned on textual descriptions in a single
step. Beyond traditional text-to-image generation, UFOGen showcases versatility
in applications. Notably, UFOGen stands among the pioneering models enabling
one-step text-to-image generation and diverse downstream tasks, presenting a
significant advancement in the landscape of efficient generative models.
\blfootnote{*Work done as a student researcher of Google, dagger indicates
equal contribution.
Tied-Lora: Enhacing parameter efficiency of LoRA with weight tying
Authors: name: Adithya Renduchintala, name: Tugrul Konuk, name: Oleksii Kuchaiev
Published: 2023-11-16 05:29:39

We propose Tied-LoRA, a simple paradigm utilizes weight tying and selective
training to further increase parameter efficiency of the Low-rank adaptation
(LoRA) method. Our investigations include all feasible combinations parameter
training/freezing in conjunction with weight tying to identify the optimal
balance between performance and the number of trainable parameters. Through
experiments covering a variety of tasks and two base language models, we
provide analysis revealing trade-offs between efficiency and performance. Our
experiments uncovered a particular Tied-LoRA configuration that stands out by
demonstrating comparable performance across several tasks while employing only
13~\% percent of parameters utilized by the standard LoRA method.
Contrastive Chain-of-Thought Prompting
Authors: name: Yew Ken Chia, name: Guizhen Chen, name: Luu Anh Tuan, name: Soujanya Poria, name: Lidong Bing
Published: 2023-11-15 18:54:01

Despite the success of chain of thought in enhancing language model
reasoning, the underlying process remains less well understood. Although
logically sound reasoning appears inherently crucial for chain of thought,
prior studies surprisingly reveal minimal impact when using invalid
demonstrations instead. Furthermore, the conventional chain of thought does not
inform language models on what mistakes to avoid, which potentially leads to
more errors. Hence, inspired by how humans can learn from both positive and
negative examples, we propose contrastive chain of thought to enhance language
model reasoning. Compared to the conventional chain of thought, our approach
provides both valid and invalid reasoning demonstrations, to guide the model to
reason step-by-step while reducing reasoning mistakes. To improve
generalization, we introduce an automatic method to construct contrastive
demonstrations. Our experiments on reasoning benchmarks demonstrate that
contrastive chain of thought can serve as a general enhancement of
chain-of-thought prompting.
ML-Bench: Large Language Models Leverage Open-source Libraries for
  Machine Learning Tasks
Authors: name: Yuliang Liu, name: Xiangru Tang, name: Zefan Cai, name: Junjie Lu, name: Yichi Zhang, name: Yanjun Shao, name: Zexuan Deng, name: Helan Hu, name: Zengxian Yang, name: Kaikai An, name: Ruijun Huang, name: Shuzheng Si, name: Sheng Chen, name: Haozhe Zhao, name: Zhengliang Li, name: Liang Chen, name: Yiming Zong, name: Yan Wang, name: Tianyu Liu, name: Zhiwei Jiang, name: Baobao Chang, name: Yujia Qin, name: Wangchunshu Zhou, name: Yilun Zhao, name: Arman Cohan, name: Mark Gerstein
Published: 2023-11-16 12:03:21

Large language models have shown promising performance in code generation
benchmarks. However, a considerable divide exists between these benchmark
achievements and their practical applicability, primarily attributed to
real-world programming's reliance on pre-existing libraries. Instead of
evaluating LLMs to code from scratch, this work aims to propose a new
evaluation setup where LLMs use open-source libraries to finish machine
learning tasks. Therefore, we propose ML-Bench, an expansive benchmark
developed to assess the effectiveness of LLMs in leveraging existing functions
in open-source libraries. Consisting of 10044 samples spanning 130 tasks over
14 notable machine learning GitHub repositories. In this setting, given a
specific machine learning task instruction and the accompanying README in a
codebase, an LLM is tasked to generate code to accomplish the task. This
necessitates the comprehension of long and language-code interleaved documents,
as well as the understanding of complex cross-file code structures, introducing
new challenges. Notably, while GPT-4 exhibits remarkable improvement over other
LLMs, it manages to accomplish only 39.73\% of the tasks, leaving a huge space
for improvement. We address these challenges by proposing ML-Agent, designed to
effectively navigate the codebase, locate documentation, retrieve code, and
generate executable code. Empirical results demonstrate that ML-Agent, built
upon GPT-4, results in further improvements. Code, data, and models are
available at https://ml-bench.github.io/.
EDMSound: Spectrogram Based Diffusion Models for Efficient and
  High-Quality Audio Synthesis
Authors: name: Ge Zhu, name: Yutong Wen, name: Marc-André Carbonneau, name: Zhiyao Duan
Published: 2023-11-15 03:27:35

Audio diffusion models can synthesize a wide variety of sounds. Existing
models often operate on the latent domain with cascaded phase recovery modules
to reconstruct waveform. This poses challenges when generating high-fidelity
audio. In this paper, we propose EDMSound, a diffusion-based generative model
in spectrogram domain under the framework of elucidated diffusion models (EDM).
Combining with efficient deterministic sampler, we achieved similar Fr\'echet
audio distance (FAD) score as top-ranked baseline with only 10 steps and
reached state-of-the-art performance with 50 steps on the DCASE2023 foley sound
generation benchmark. We also revealed a potential concern regarding diffusion
based audio generation models that they tend to generate samples with high
perceptual similarity to the data from training data. Project page:
https://agentcooper2002.github.io/EDMSound/
Drivable 3D Gaussian Avatars
Authors: name: Wojciech Zielonka, name: Timur Bagautdinov, name: Shunsuke Saito, name: Michael Zollhöfer, name: Justus Thies, name: Javier Romero
Published: 2023-11-14 22:54:29

We present Drivable 3D Gaussian Avatars (D3GA), the first 3D controllable
model for human bodies rendered with Gaussian splats. Current photorealistic
drivable avatars require either accurate 3D registrations during training,
dense input images during testing, or both. The ones based on neural radiance
fields also tend to be prohibitively slow for telepresence applications. This
work uses the recently presented 3D Gaussian Splatting (3DGS) technique to
render realistic humans at real-time framerates, using dense calibrated
multi-view videos as input. To deform those primitives, we depart from the
commonly used point deformation method of linear blend skinning (LBS) and use a
classic volumetric deformation method: cage deformations. Given their smaller
size, we drive these deformations with joint angles and keypoints, which are
more suitable for communication applications. Our experiments on nine subjects
with varied body shapes, clothes, and motions obtain higher-quality results
than state-of-the-art methods when using the same training and test data.
Single-Image 3D Human Digitization with Shape-Guided Diffusion
Authors: name: Badour AlBahar, name: Shunsuke Saito, name: Hung-Yu Tseng, name: Changil Kim, name: Johannes Kopf, name: Jia-Bin Huang
Published: 2023-11-15 18:59:56

We present an approach to generate a 360-degree view of a person with a
consistent, high-resolution appearance from a single input image. NeRF and its
variants typically require videos or images from different viewpoints. Most
existing approaches taking monocular input either rely on ground-truth 3D scans
for supervision or lack 3D consistency. While recent 3D generative models show
promise of 3D consistent human digitization, these approaches do not generalize
well to diverse clothing appearances, and the results lack photorealism. Unlike
existing work, we utilize high-capacity 2D diffusion models pretrained for
general image synthesis tasks as an appearance prior of clothed humans. To
achieve better 3D consistency while retaining the input identity, we
progressively synthesize multiple views of the human in the input image by
inpainting missing regions with shape-guided diffusion conditioned on
silhouette and surface normal. We then fuse these synthesized multi-view images
via inverse rendering to obtain a fully textured high-resolution 3D mesh of the
given person. Experiments show that our approach outperforms prior methods and
achieves photorealistic 360-degree synthesis of a wide range of clothed humans
with complex textures from a single image.
DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction
  Model
Authors: name: Yinghao Xu, name: Hao Tan, name: Fujun Luan, name: Sai Bi, name: Peng Wang, name: Jiahao Li, name: Zifan Shi, name: Kalyan Sunkavalli, name: Gordon Wetzstein, name: Zexiang Xu, name: Kai Zhang
Published: 2023-11-15 18:58:41

We propose DMV3D, a novel 3D generation approach that uses a
transformer-based 3D large reconstruction model to denoise multi-view
diffusion. Our reconstruction model incorporates a triplane NeRF representation
and can denoise noisy multi-view images via NeRF reconstruction and rendering,
achieving single-stage 3D generation in sim30s on single A100 GPU. We train
DMV3D on large-scale multi-view image datasets of highly diverse
objects using only image reconstruction losses, without accessing 3D assets. We
demonstrate state-of-the-art results for the single-image reconstruction
problem where probabilistic modeling of unseen object parts is required for
generating diverse reconstructions with sharp textures. We also show
high-quality text-to-3D generation results outperforming previous 3D diffusion
models. Our project website is at: https://justimyhxu.github.io/projects/dmv3d/ .
Fusion-Eval: Integrating Evaluators with LLMs
Authors: name: Lei Shu, name: Nevan Wichers, name: Liangchen Luo, name: Yun Zhu, name: Yinxiao Liu, name: Jindong Chen, name: Lei Meng
Published: 2023-11-15 18:46:56

Evaluating Large Language Models (LLMs) is a complex task, especially
considering the intricacies of natural language understanding and the
expectations for high-level reasoning. Traditional evaluations typically lean
on human-based, model-based, or automatic-metrics-based paradigms, each with
its own advantages and shortcomings. We introduce "Fusion-Eval", a system that
employs LLMs not solely for direct evaluations, but to skillfully integrate
insights from diverse evaluators. This gives Fusion-Eval flexibility, enabling
it to work effectively across diverse tasks and make optimal use of multiple
references. In testing on the SummEval dataset, Fusion-Eval achieved a Spearman
correlation of 0.96, outperforming other evaluators. The success of Fusion-Eval
underscores the potential of LLMs to produce evaluations that closely align
human perspectives, setting a new standard in the field of LLM evaluation.
PEARL: Personalizing Large Language Model Writing Assistants with
  Generation-Calibrated Retrievers
Authors: name: Sheshera Mysore, name: Zhuoran Lu, name: Mengting Wan, name: Longqi Yang, name: Steve Menezes, name: Tina Baghaee, name: Emmanuel Barajas Gonzalez, name: Jennifer Neville, name: Tara Safavi
Published: 2023-11-15 18:19:58

Powerful large language models have facilitated the development of writing
assistants that promise to significantly improve the quality and efficiency of
composition and communication. However, a barrier to effective assistance is
the lack of personalization in LLM outputs to the author's communication style
and specialized knowledge. In this paper, we address this challenge by
proposing PEARL, a retrieval-augmented LLM writing assistant personalized with
a generation-calibrated retriever. Our retriever is trained to select historic
user-authored documents for prompt augmentation, such that they are likely to
best personalize LLM generations for a user request. We propose two key
novelties for training our retriever: 1) A training data selection method that
identifies user requests likely to benefit from personalization and documents
that provide that benefit; and 2) A scale-calibrating KL-divergence objective
that ensures that our retriever closely tracks the benefit of a document for
personalized generation. We demonstrate the effectiveness of PEARL in
generating personalized workplace social media posts and Reddit comments.
Finally, we showcase the potential of a generation-calibrated retriever to
double as a performance predictor and further improve low-quality generations
via LLM chaining.
SiRA: Sparse Mixture of Low Rank Adaptation
Authors: name: Yun Zhu, name: Nevan Wichers, name: Chu-Cheng Lin, name: Xinyi Wang, name: Tianlong Chen, name: Lei Shu, name: Han Lu, name: Canoee Liu, name: Liangchen Luo, name: Jindong Chen, name: Lei Meng
Published: 2023-11-15 18:15:37

Parameter Efficient Tuning has been an prominent approach to adapt the Large
Language Model to downstream tasks. Most previous works considers adding the
dense trainable parameters, where all parameters are used to adapt certain
task. We found this less effective empirically using the example of LoRA that
introducing more trainable parameters does not help. Motivated by this we
investigate the importance of leveraging "sparse" computation and propose SiRA:
sparse mixture of low rank adaption. SiRA leverages the Sparse Mixture of
Expert(SMoE) to boost the performance of LoRA. Specifically it enforces the top
k experts routing with a capacity limit restricting the maximum number of
tokens each expert can process. We propose a novel and simple expert dropout on
top of gating network to reduce the over-fitting issue. Through extensive
experiments, we verify SiRA performs better than LoRA and other mixture of
expert approaches across different single tasks and multitask settings.
Llamas Know What GPTs Don't Show: Surrogate Models for Confidence
  Estimation
Authors: name: Vaishnavi Shrivastava, name: Percy Liang, name: Ananya Kumar
Published: 2023-11-15 11:27:44

To maintain user trust, large language models (LLMs) should signal low
confidence on examples where they are incorrect, instead of misleading the
user. The standard approach of estimating confidence is to use the softmax
probabilities of these models, but as of November 2023, state-of-the-art LLMs
such as GPT-4 and Claude-v1.3 do not provide access to these probabilities. We
first study eliciting confidence linguistically -- asking an LLM for its
confidence in its answer -- which performs reasonably (80.5% AUC on GPT-4
averaged across 12 question-answering datasets -- 7% above a random baseline)
but leaves room for improvement. We then explore using a surrogate confidence
model -- using a model where we do have probabilities to evaluate the original
model's confidence in a given question. Surprisingly, even though these
probabilities come from a different and often weaker model, this method leads
to higher AUC than linguistic confidences on 9 out of 12 datasets. Our best
method composing linguistic confidences and surrogate model probabilities gives
state-of-the-art confidence estimates on all 12 datasets (84.6% average AUC on
GPT-4).