Orionnnn/hub-stats huggingface dataset:
Flexible Isosurface Extraction for Gradient-Based Mesh Optimization
Authors: name: Tianchang Shen, name: Jacob Munkberg, name: Jon Hasselgren, name: Kangxue Yin, name: Zian Wang, name: Wenzheng Chen, name: Zan Gojcic, name: Sanja Fidler, name: Nicholas Sharp, name: Jun Gao
Published: 2023-08-10 06:40:19

This work considers gradient-based mesh optimization, where we iteratively
optimize for a 3D surface mesh by representing it as the isosurface of a scalar
field, an increasingly common paradigm in applications including
photogrammetry, generative modeling, and inverse physics. Existing
implementations adapt classic isosurface extraction algorithms like Marching
Cubes or Dual Contouring; these techniques were designed to extract meshes from
fixed, known fields, and in the optimization setting they lack the degrees of
freedom to represent high-quality feature-preserving meshes, or suffer from
numerical instabilities. We introduce FlexiCubes, an isosurface representation
specifically designed for optimizing an unknown mesh with respect to geometric,
visual, or even physical objectives. Our main insight is to introduce
additional carefully-chosen parameters into the representation, which allow
local flexible adjustments to the extracted mesh geometry and connectivity.
These parameters are updated along with the underlying scalar field via
automatic differentiation when optimizing for a downstream task. We base our
extraction scheme on Dual Marching Cubes for improved topological properties,
and present extensions to optionally generate tetrahedral and
hierarchically-adaptive meshes. Extensive experiments validate FlexiCubes on
both synthetic benchmarks and real-world applications, showing that it offers
significant improvements in mesh quality and geometric fidelity.
OpenProteinSet: Training data for structural biology at scale
Authors: name: Gustaf Ahdritz, name: Nazim Bouatta, name: Sachin Kadyan, name: Lukas Jarosch, name: Daniel Berenberg, name: Ian Fisk, name: Andrew M. Watkins, name: Stephen Ra, name: Richard Bonneau, name: Mohammed AlQuraishi
Published: 2023-08-10 04:01:04

Multiple sequence alignments (MSAs) of proteins encode rich biological
information and have been workhorses in bioinformatic methods for tasks like
protein design and protein structure prediction for decades. Recent
breakthroughs like AlphaFold2 that use transformers to attend directly over
large quantities of raw MSAs have reaffirmed their importance. Generation of
MSAs is highly computationally intensive, however, and no datasets comparable
to those used to train AlphaFold2 have been made available to the research
community, hindering progress in machine learning for proteins. To remedy this
problem, we introduce OpenProteinSet, an open-source corpus of more than 16
million MSAs, associated structural homologs from the Protein Data Bank, and
AlphaFold2 protein structure predictions. We have previously demonstrated the
utility of OpenProteinSet by successfully retraining AlphaFold2 on it. We
expect OpenProteinSet to be broadly useful as training and validation data for
1) diverse tasks focused on protein structure, function, and design and 2)
large-scale multimodal machine learning research.
PDE-Refiner: Achieving Accurate Long Rollouts with Neural PDE Solvers
Authors: name: Phillip Lippe, name: Bastiaan S. Veeling, name: Paris Perdikaris, name: Richard E. Turner, name: Johannes Brandstetter
Published: 2023-08-10 17:53:05

Time-dependent partial differential equations (PDEs) are ubiquitous in
science and engineering. Recently, mostly due to the high computational cost of
traditional solution techniques, deep neural network based surrogates have
gained increased interest. The practical utility of such neural PDE solvers
relies on their ability to provide accurate, stable predictions over long time
horizons, which is a notoriously hard problem. In this work, we present a
large-scale analysis of common temporal rollout strategies, identifying the
neglect of non-dominant spatial frequency information, often associated with
high frequencies in PDE solutions, as the primary pitfall limiting stable,
accurate rollout performance. Based on these insights, we draw inspiration from
recent advances in diffusion models to introduce PDE-Refiner; a novel model
class that enables more accurate modeling of all frequency components via a
multistep refinement process. We validate PDE-Refiner on challenging benchmarks
of complex fluid dynamics, demonstrating stable and accurate rollouts that
consistently outperform state-of-the-art models, including neural, numerical,
and hybrid neural-numerical architectures. We further demonstrate that
PDE-Refiner greatly enhances data efficiency, since the denoising objective
implicitly induces a novel form of spectral data augmentation. Finally,
PDE-Refiner's connection to diffusion models enables an accurate and efficient
assessment of the model's predictive uncertainty, allowing us to estimate when
the surrogate becomes inaccurate.
AudioLDM 2: Learning Holistic Audio Generation with Self-supervised
  Pretraining
Authors: name: Haohe Liu, name: Qiao Tian, name: Yi Yuan, name: Xubo Liu, name: Xinhao Mei, name: Qiuqiang Kong, name: Yuping Wang, name: Wenwu Wang, name: Yuxuan Wang, name: Mark D. Plumbley
Published: 2023-08-10 17:55:13

Although audio generation shares commonalities across different types of
audio, such as speech, music, and sound effects, designing models for each type
requires careful consideration of specific objectives and biases that can
significantly differ from those of other types. To bring us closer to a unified
perspective of audio generation, this paper proposes a framework that utilizes
the same learning method for speech, music, and sound effect generation. Our
framework introduces a general representation of audio, called language of
audio (LOA). Any audio can be translated into LOA based on AudioMAE, a
self-supervised pre-trained representation learning model. In the generation
process, we translate any modalities into LOA by using a GPT-2 model, and we
perform self-supervised audio generation learning with a latent diffusion model
conditioned on LOA. The proposed framework naturally brings advantages such as
in-context learning abilities and reusable self-supervised pretrained AudioMAE
and latent diffusion models. Experiments on the major benchmarks of
text-to-audio, text-to-music, and text-to-speech demonstrate new
state-of-the-art or competitive performance to previous approaches. Our demo
and code are available at https://audioldm.github.io/audioldm2.
Alexa, play with robot: Introducing the First Alexa Prize SimBot
  Challenge on Embodied AI
Authors: name: Hangjie Shi, name: Leslie Ball, name: Govind Thattai, name: Desheng Zhang, name: Lucy Hu, name: Qiaozi Gao, name: Suhaila Shakiah, name: Xiaofeng Gao, name: Aishwarya Padmakumar, name: Bofei Yang, name: Cadence Chung, name: Dinakar Guthy, name: Gaurav Sukhatme, name: Karthika Arumugam, name: Matthew Wen, name: Osman Ipek, name: Patrick Lange, name: Rohan Khanna, name: Shreyas Pansare, name: Vasu Sharma, name: Chao Zhang, name: Cris Flagg, name: Daniel Pressel, name: Lavina Vaz, name: Luke Dai, name: Prasoon Goyal, name: Sattvik Sahai, name: Shaohua Liu, name: Yao Lu, name: Anna Gottardi, name: Shui Hu, name: Yang Liu, name: Dilek Hakkani-Tur, name: Kate Bland, name: Heather Rocker, name: James Jeun, name: Yadunandana Rao, name: Michael Johnston, name: Akshaya Iyengar, name: Arindam Mandal, name: Prem Natarajan, name: Reza Ghanadan
Published: 2023-08-09 20:56:56

The Alexa Prize program has empowered numerous university students to
explore, experiment, and showcase their talents in building conversational
agents through challenges like the SocialBot Grand Challenge and the TaskBot
Challenge. As conversational agents increasingly appear in multimodal and
embodied contexts, it is important to explore the affordances of conversational
interaction augmented with computer vision and physical embodiment. This paper
describes the SimBot Challenge, a new challenge in which university teams
compete to build robot assistants that complete tasks in a simulated physical
environment. This paper provides an overview of the SimBot Challenge, which
included both online and offline challenge phases. We describe the
infrastructure and support provided to the teams including Alexa Arena, the
simulated environment, and the ML toolkit provided to teams to accelerate their
building of vision and language models. We summarize the approaches the
participating teams took to overcome research challenges and extract key
lessons learned. Finally, we provide analysis of the performance of the
competing SimBots during the competition.
Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language
  Models' Alignment
Authors: name: Yang Liu, name: Yuanshun Yao, name: Jean-Francois Ton, name: Xiaoying Zhang, name: Ruocheng Guo Hao Cheng, name: Yegor Klochkov, name: Muhammad Faaiz Taufiq, name: Hang Li
Published: 2023-08-10 06:43:44

Ensuring alignment, which refers to making models behave in accordance with
human intentions [1,2], has become a critical task before deploying large
language models (LLMs) in real-world applications. For instance, OpenAI devoted
six months to iteratively aligning GPT-4 before its release [3]. However, a
major challenge faced by practitioners is the lack of clear guidance on
evaluating whether LLM outputs align with social norms, values, and
regulations. This obstacle hinders systematic iteration and deployment of LLMs.
To address this issue, this paper presents a comprehensive survey of key
dimensions that are crucial to consider when assessing LLM trustworthiness. The
survey covers seven major categories of LLM trustworthiness: reliability,
safety, fairness, resistance to misuse, explainability and reasoning, adherence
to social norms, and robustness. Each major category is further divided into
several sub-categories, resulting in a total of 29 sub-categories.
Additionally, a subset of 8 sub-categories is selected for further
investigation, where corresponding measurement studies are designed and
conducted on several widely-used LLMs. The measurement results indicate that,
in general, more aligned models tend to perform better in terms of overall
trustworthiness. However, the effectiveness of alignment varies across the
different trustworthiness categories considered. This highlights the importance
of conducting more fine-grained analyses, testing, and making continuous
improvements on LLM alignment. By shedding light on these key dimensions of LLM
trustworthiness, this paper aims to provide valuable insights and guidance to
practitioners in the field. Understanding and addressing these concerns will be
crucial in achieving reliable and ethically sound deployment of LLMs in various
applications.
FocalFormer3D : Focusing on Hard Instance for 3D Object Detection
Authors: name: Yilun Chen, name: Zhiding Yu, name: Yukang Chen, name: Shiyi Lan, name: Animashree Anandkumar, name: Jiaya Jia, name: Jose Alvarez
Published: 2023-08-08 20:06:12

False negatives (FN) in 3D object detection, {\em e.g.}, missing predictions
of pedestrians, vehicles, or other obstacles, can lead to potentially dangerous
situations in autonomous driving. While being fatal, this issue is understudied
in many current 3D detection methods. In this work, we propose Hard Instance
Probing (HIP), a general pipeline that identifies FN in a multi-stage
manner and guides the models to focus on excavating difficult instances. For 3D
object detection, we instantiate this method as FocalFormer3D, a simple yet
effective detector that excels at excavating difficult objects and improving
prediction recall. FocalFormer3D features a multi-stage query generation to
discover hard objects and a box-level transformer decoder to efficiently
distinguish objects from massive object candidates. Experimental results on the
nuScenes and Waymo datasets validate the superior performance of FocalFormer3D.
The advantage leads to strong performance on both detection and tracking, in
both LiDAR and multi-modal settings. Notably, FocalFormer3D achieves a 70.5 mAP
and 73.9 NDS on nuScenes detection benchmark, while the nuScenes tracking
benchmark shows 72.1 AMOTA, both ranking 1st place on the nuScenes LiDAR
leaderboard. Our code is available at
https://github.com/NVlabs/FocalFormer3D.
JEN-1: Text-Guided Universal Music Generation with Omnidirectional
  Diffusion Models
Authors: name: Peike Li, name: Boyu Chen, name: Yao Yao, name: Yikai Wang, name: Allen Wang, name: Alex Wang
Published: 2023-08-09 06:27:24

Music generation has attracted growing interest with the advancement of deep
generative models. However, generating music conditioned on textual
descriptions, known as text-to-music, remains challenging due to the complexity
of musical structures and high sampling rate requirements. Despite the task's
significance, prevailing generative models exhibit limitations in music
quality, computational efficiency, and generalization. This paper introduces
JEN-1, a universal high-fidelity model for text-to-music generation. JEN-1 is a
diffusion model incorporating both autoregressive and non-autoregressive
training. Through in-context learning, JEN-1 performs various generation tasks
including text-guided music generation, music inpainting, and continuation.
Evaluations demonstrate JEN-1's superior performance over state-of-the-art
methods in text-music alignment and music quality while maintaining
computational efficiency. Our demos are available at
http://futureverse.com/research/jen/demos/jen1
Accelerating LLM Inference with Staged Speculative Decoding
Authors: name: Benjamin Spector, name: Chris Re
Published: 2023-08-08 23:29:55

Recent advances with large language models (LLM) illustrate their diverse
capabilities. We propose a novel algorithm, staged speculative decoding, to
accelerate LLM inference in small-batch, on-device scenarios. We address the
low arithmetic intensity of small-batch inference by improving upon previous
work in speculative decoding. First, we restructure the speculative batch as a
tree, which reduces generation costs and increases the expected tokens per
batch. Second, we add a second stage of speculative decoding. Taken together,
we reduce single-batch decoding latency by 3.16x with a 762M parameter GPT-2-L
model while perfectly preserving output quality.
Shepherd: A Critic for Language Model Generation
Authors: name: Tianlu Wang, name: Ping Yu, name: Xiaoqing Ellen Tan, name: Sean O'Brien, name: Ramakanth Pasunuru, name: Jane Dwivedi-Yu, name: Olga Golovneva, name: Luke Zettlemoyer, name: Maryam Fazel-Zarandi, name: Asli Celikyilmaz
Published: 2023-08-08 21:23:23

As large language models improve, there is increasing interest in techniques
that leverage these models' capabilities to refine their own outputs. In this
work, we introduce Shepherd, a language model specifically tuned to critique
responses and suggest refinements, extending beyond the capabilities of an
untuned model to identify diverse errors and provide suggestions to remedy
them. At the core of our approach is a high quality feedback dataset, which we
curate from community feedback and human annotations. Even though Shepherd is
small (7B parameters), its critiques are either equivalent or preferred to
those from established models including ChatGPT. Using GPT-4 for evaluation,
Shepherd reaches an average win-rate of 53-87% compared to competitive
alternatives. In human evaluation, Shepherd strictly outperforms other models
and on average closely ties with ChatGPT.
3D Gaussian Splatting for Real-Time Radiance Field Rendering
Authors: name: Bernhard Kerbl, name: Georgios Kopanas, name: Thomas Leimkühler, name: George Drettakis
Published: 2023-08-08 06:37:06

Radiance Field methods have recently revolutionized novel-view synthesis of
scenes captured with multiple photos or videos. However, achieving high visual
quality still requires neural networks that are costly to train and render,
while recent faster methods inevitably trade off speed for quality. For
unbounded and complete scenes (rather than isolated objects) and 1080p
resolution rendering, no current method can achieve real-time display rates. We
introduce three key elements that allow us to achieve state-of-the-art visual
quality while maintaining competitive training times and importantly allow
high-quality real-time (>= 30 fps) novel-view synthesis at 1080p resolution.
First, starting from sparse points produced during camera calibration, we
represent the scene with 3D Gaussians that preserve desirable properties of
continuous volumetric radiance fields for scene optimization while avoiding
unnecessary computation in empty space; Second, we perform interleaved
optimization/density control of the 3D Gaussians, notably optimizing
anisotropic covariance to achieve an accurate representation of the scene;
Third, we develop a fast visibility-aware rendering algorithm that supports
anisotropic splatting and both accelerates training and allows realtime
rendering. We demonstrate state-of-the-art visual quality and real-time
rendering on several established datasets.
FLIRT: Feedback Loop In-context Red Teaming
Authors: name: Ninareh Mehrabi, name: Palash Goyal, name: Christophe Dupuy, name: Qian Hu, name: Shalini Ghosh, name: Richard Zemel, name: Kai-Wei Chang, name: Aram Galstyan, name: Rahul Gupta
Published: 2023-08-08 14:03:08

Warning: this paper contains content that may be inappropriate or offensive.
  As generative models become available for public use in various applications,
testing and analyzing vulnerabilities of these models has become a priority.
Here we propose an automatic red teaming framework that evaluates a given model
and exposes its vulnerabilities against unsafe and inappropriate content
generation. Our framework uses in-context learning in a feedback loop to red
team models and trigger them into unsafe content generation. We propose
different in-context attack strategies to automatically learn effective and
diverse adversarial prompts for text-to-image models. Our experiments
demonstrate that compared to baseline approaches, our proposed strategy is
significantly more effective in exposing vulnerabilities in Stable Diffusion
(SD) model, even when the latter is enhanced with safety features. Furthermore,
we demonstrate that the proposed framework is effective for red teaming
text-to-text models, resulting in significantly higher toxic response
generation rate compared to previously reported numbers.
SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore
Authors: name: Sewon Min, name: Suchin Gururangan, name: Eric Wallace, name: Hannaneh Hajishirzi, name: Noah A. Smith, name: Luke Zettlemoyer
Published: 2023-08-08 17:58:15

The legality of training language models (LMs) on copyrighted or otherwise
restricted data is under intense debate. However, as we show, model performance
significantly degrades if trained only on low-risk text (e.g., out-of-copyright
books or government documents), due to its limited size and domain coverage. We
present SILO, a new language model that manages this risk-performance tradeoff
during inference. SILO is built by (1) training a parametric LM on Open License
Corpus (OLC), a new corpus we curate with 228B tokens of public domain and
permissively licensed text and (2) augmenting it with a more general and easily
modifiable nonparametric datastore (e.g., containing copyrighted books or news)
that is only queried during inference. The datastore allows use of high-risk
data without training on it, supports sentence-level data attribution, and
enables data producers to opt out from the model by removing content from the
store. These capabilities can foster compliance with data-use regulations such
as the fair use doctrine in the United States and the GDPR in the European
Union. Our experiments show that the parametric LM struggles on domains not
covered by OLC. However, access to the datastore greatly improves out of domain
performance, closing 90% of the performance gap with an LM trained on the Pile,
a more diverse corpus with mostly high-risk text. We also analyze which
nonparametric approach works best, where the remaining errors lie, and how
performance scales with datastore size. Our results suggest that it is possible
to build high quality language models while mitigating their legal risk.
Simple synthetic data reduces sycophancy in large language models
Authors: name: Jerry Wei, name: Da Huang, name: Yifeng Lu, name: Denny Zhou, name: Quoc V. Le
Published: 2023-08-07 23:48:36

Sycophancy is an undesirable behavior where models tailor their responses to
follow a human user's view even when that view is not objectively correct
(e.g., adapting liberal views once a user reveals that they are liberal). In
this paper, we study the prevalence of sycophancy in language models and
propose a simple synthetic-data intervention to reduce this behavior.
  First, on a set of three sycophancy tasks (Perez et al., 2022) where models
are asked for an opinion on statements with no correct answers (e.g.,
politics), we observe that both model scaling and instruction tuning
significantly increase sycophancy for PaLM models up to 540B parameters.
Second, we extend sycophancy evaluations to simple addition statements that are
objectively incorrect, finding that despite knowing that these statements are
wrong, language models will still agree with them if the user does as well.
  To reduce sycophancy, we present a straightforward synthetic-data
intervention that takes public NLP tasks and encourages models to be robust to
user opinions on these tasks. Adding these data in a lightweight finetuning
step can significantly reduce sycophantic behavior on held-out prompts. Code
for generating synthetic data for intervention can be found at
https://github.com/google/sycophancy-intervention.
ReCLIP: Refine Contrastive Language Image Pre-Training with Source Free
  Domain Adaptation
Authors: name: Hu. Xuefeng, name: Zhang. Ke, name: Xia. Lu, name: Chen. Albert, name: Luo. Jiajia, name: Sun. Yuyin, name: Wang. Ken, name: Qiao. Nan, name: Zeng. Xiao, name: Sun. Min, name: Kuo. Cheng-Hao, name: Nevatia. Ram
Published: 2023-08-04 18:11:40

Large-scale Pre-Training Vision-Language Model such as CLIP has demonstrated
outstanding performance in zero-shot classification, e.g. achieving 76.3% top-1
accuracy on ImageNet without seeing any example, which leads to potential
benefits to many tasks that have no labeled data. However, while applying CLIP
to a downstream target domain, the presence of visual and text domain gaps and
cross-modality misalignment can greatly impact the model performance. To
address such challenges, we propose ReCLIP, the first source-free domain
adaptation method for vision-language models, which does not require any source
data or target labeled data. ReCLIP first learns a projection space to mitigate
the misaligned visual-text embeddings and learns pseudo labels, and then
deploys cross-modality self-training with the pseudo labels, to update visual
and text encoders, refine labels and reduce domain gaps and misalignments
iteratively. With extensive experiments, we demonstrate ReCLIP reduces the
average error rate of CLIP from 30.17% to 25.06% on 22 image classification
benchmarks.
3D Motion Magnification: Visualizing Subtle Motions with Time Varying
  Radiance Fields
Authors: name: Brandon Y. Feng, name: Hadi Alzayer, name: Michael Rubinstein, name: William T. Freeman, name: Jia-Bin Huang
Published: 2023-08-07 17:59:59

Motion magnification helps us visualize subtle, imperceptible motion.
However, prior methods only work for 2D videos captured with a fixed camera. We
present a 3D motion magnification method that can magnify subtle motions from
scenes captured by a moving camera, while supporting novel view rendering. We
represent the scene with time-varying radiance fields and leverage the Eulerian
principle for motion magnification to extract and amplify the variation of the
embedding of a fixed point over time. We study and validate our proposed
principle for 3D motion magnification using both implicit and tri-plane-based
radiance fields as our underlying 3D scene representation. We evaluate the
effectiveness of our method on both synthetic and real-world scenes captured
under various camera setups.