Orionnnn/hub-stats huggingface dataset:
Less is More: Focus Attention for Efficient DETR
Authors: name: Dehua Zheng, name: Wenhui Dong, name: Hailin Hu, name: Xinghao Chen, name: Yunhe Wang
Published: 2023-07-24 08:39:11

DETR-like models have significantly boosted the performance of detectors and
even outperformed classical convolutional models. However, all tokens are
treated equally without discrimination brings a redundant computational burden
in the traditional encoder structure. The recent sparsification strategies
exploit a subset of informative tokens to reduce attention complexity
maintaining performance through the sparse encoder. But these methods tend to
rely on unreliable model statistics. Moreover, simply reducing the token
population hinders the detection performance to a large extent, limiting the
application of these sparse models. We propose Focus-DETR, which focuses
attention on more informative tokens for a better trade-off between computation
efficiency and model accuracy. Specifically, we reconstruct the encoder with
dual attention, which includes a token scoring mechanism that considers both
localization and category semantic information of the objects from multi-scale
feature maps. We efficiently abandon the background queries and enhance the
semantic interaction of the fine-grained object queries based on the scores.
Compared with the state-of-the-art sparse DETR-like detectors under the same
setting, our Focus-DETR gets comparable complexity while achieving 50.4AP
(+2.2) on COCO. The code is available at
https://github.com/huawei-noah/noah-research/tree/master/Focus-DETR and
https://gitee.com/mindspore/models/tree/master/research/cv/Focus-DETR.
Interpolating between Images with Diffusion Models
Authors: name: Clinton J. Wang, name: Polina Golland
Published: 2023-07-24 07:03:22

One little-explored frontier of image generation and editing is the task of
interpolating between two input images, a feature missing from all currently
deployed image generation pipelines. We argue that such a feature can expand
the creative applications of such models, and propose a method for zero-shot
interpolation using latent diffusion models. We apply interpolation in the
latent space at a sequence of decreasing noise levels, then perform denoising
conditioned on interpolated text embeddings derived from textual inversion and
(optionally) subject poses. For greater consistency, or to specify additional
criteria, we can generate several candidates and use CLIP to select the highest
quality image. We obtain convincing interpolations across diverse subject
poses, image styles, and image content, and show that standard quantitative
metrics such as FID are insufficient to measure the quality of an
interpolation. Code and data are available at
https://clintonjwang.github.io/interpolation.
Prompting Large Language Models with Speech Recognition Abilities
Authors: name: Yassir Fathullah, name: Chunyang Wu, name: Egor Lakomkin, name: Junteng Jia, name: Yuan Shangguan, name: Ke Li, name: Jinxi Guo, name: Wenhan Xiong, name: Jay Mahadeokar, name: Ozlem Kalinli, name: Christian Fuegen, name: Mike Seltzer
Published: 2023-07-21 08:39:15

Large language models have proven themselves highly flexible, able to solve a
wide range of generative tasks, such as abstractive summarization and
open-ended question answering. In this paper we extend the capabilities of LLMs
by directly attaching a small audio encoder allowing it to perform speech
recognition. By directly prepending a sequence of audial embeddings to the text
token embeddings, the LLM can be converted to an automatic speech recognition
(ASR) system, and be used in the exact same manner as its textual counterpart.
Experiments on Multilingual LibriSpeech (MLS) show that incorporating a
conformer encoder into the open sourced LLaMA-7B allows it to outperform
monolingual baselines by 18% and perform multilingual speech recognition
despite LLaMA being trained overwhelmingly on English text. Furthermore, we
perform ablation studies to investigate whether the LLM can be completely
frozen during training to maintain its original capabilities, scaling up the
audio encoder, and increasing the audio encoder striding to generate fewer
embeddings. The results from these studies show that multilingual ASR is
possible even when the LLM is frozen or when strides of almost 1 second are
used in the audio encoder opening up the possibility for LLMs to operate on
long-form audio.
Evaluating the Ripple Effects of Knowledge Editing in Language Models
Authors: name: Roi Cohen, name: Eden Biran, name: Ori Yoran, name: Amir Globerson, name: Mor Geva
Published: 2023-07-24 17:52:46

Modern language models capture a large body of factual knowledge. However,
some facts can be incorrectly induced or become obsolete over time, resulting
in factually incorrect generations. This has led to the development of various
editing methods that allow updating facts encoded by the model. Evaluation of
these methods has primarily focused on testing whether an individual fact has
been successfully injected, and if similar predictions for other subjects have
not changed. Here we argue that such evaluation is limited, since injecting one
fact (e.g. ``Jack Depp is the son of Johnny Depp'') introduces a ``ripple
effect'' in the form of additional facts that the model needs to update
(e.g.``Jack Depp is the sibling of Lily-Rose Depp''). To address this issue, we
propose a novel set of evaluation criteria that consider the implications of an
edit on related facts. Using these criteria, we then construct , a
diagnostic benchmark of 5K factual edits, capturing a variety of types of
ripple effects. We evaluate prominent editing methods on , showing
that current methods fail to introduce consistent changes in the model's
knowledge. In addition, we find that a simple in-context editing baseline
obtains the best scores on our benchmark, suggesting a promising research
direction for model editing.
3D-LLM: Injecting the 3D World into Large Language Models
Authors: name: Yining Hong, name: Haoyu Zhen, name: Peihao Chen, name: Shuhong Zheng, name: Yilun Du, name: Zhenfang Chen, name: Chuang Gan
Published: 2023-07-24 17:59:02

Large language models (LLMs) and Vision-Language Models (VLMs) have been
proven to excel at multiple tasks, such as commonsense reasoning. Powerful as
these models can be, they are not grounded in the 3D physical world, which
involves richer concepts such as spatial relationships, affordances, physics,
layout, and so on. In this work, we propose to inject the 3D world into large
language models and introduce a whole new family of 3D-LLMs. Specifically,
3D-LLMs can take 3D point clouds and their features as input and perform a
diverse set of 3D-related tasks, including captioning, dense captioning, 3D
question answering, task decomposition, 3D grounding, 3D-assisted dialog,
navigation, and so on. Using three types of prompting mechanisms that we
design, we are able to collect over 300k 3D-language data covering these tasks.
To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that
obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as
our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism,
3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show
that our model outperforms state-of-the-art baselines by a large margin (e.g.,
the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore,
experiments on our held-in datasets for 3D captioning, task composition, and
3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative
examples also show that our model could perform more tasks beyond the scope of
existing LLMs and VLMs. Project Page: : https://vis-www.cs.umass.edu/3dllm/.
RLCD: Reinforcement Learning from Contrast Distillation for Language
  Model Alignment
Authors: name: Kevin Yang, name: Dan Klein, name: Asli Celikyilmaz, name: Nanyun Peng, name: Yuandong Tian
Published: 2023-07-24 17:23:22

We propose Reinforcement Learning from Contrast Distillation (RLCD), a method
for aligning language models to follow natural language principles without
using human feedback. RLCD trains a preference model using simulated preference
pairs that contain both a high-quality and low-quality example, generated using
contrasting positive and negative prompts. The preference model is then used to
improve a base unaligned language model via reinforcement learning.
Empirically, RLCD outperforms RLAIF (Bai et al., 2022b) and context
distillation (Huang et al., 2022) baselines across three diverse alignment
tasks--harmlessness, helpfulness, and story outline generation--and on both 7B
and 30B model scales for preference data simulation.
A Real-World WebAgent with Planning, Long Context Understanding, and
  Program Synthesis
Authors: name: Izzeddin Gur, name: Hiroki Furuta, name: Austin Huang, name: Mustafa Safdari, name: Yutaka Matsuo, name: Douglas Eck, name: Aleksandra Faust
Published: 2023-07-24 14:56:30

Pre-trained large language models (LLMs) have recently achieved better
generalization and sample efficiency in autonomous web navigation. However, the
performance on real-world websites has still suffered from (1) open domainness,
(2) limited context length, and (3) lack of inductive bias on HTML. We
introduce WebAgent, an LLM-driven agent that can complete the tasks on real
websites following natural language instructions. WebAgent plans ahead by
decomposing instructions into canonical sub-instructions, summarizes long HTML
documents into task-relevant snippets, and acts on websites via generated
Python programs from those. We design WebAgent with Flan-U-PaLM, for grounded
code generation, and HTML-T5, new pre-trained LLMs for long HTML documents
using local and global attention mechanisms and a mixture of long-span
denoising objectives, for planning and summarization. We empirically
demonstrate that our recipe improves the success on a real website by over 50%,
and that HTML-T5 is the best model to solve HTML-based tasks; achieving 14.9%
higher success rate than prior SoTA on the MiniWoB web navigation benchmark and
better accuracy on offline task planning evaluation.
Optimized Network Architectures for Large Language Model Training with
  Billions of Parameters
Authors: name: Weiyang Wang, name: Manya Ghobadi, name: Kayvon Shakeri, name: Ying Zhang, name: Naader Hasani
Published: 2023-07-22 21:18:41

This paper challenges the well-established paradigm for building any-to-any
networks for training Large Language Models (LLMs). We show that LLMs exhibit a
unique communication pattern where only small groups of GPUs require
high-bandwidth any-to-any communication within them, to achieve near-optimal
training performance. Across these groups of GPUs, the communication is
insignificant, sparse, and homogeneous. We propose a new network architecture
that closely resembles the communication requirement of LLMs. Our architecture
partitions the cluster into sets of GPUs interconnected with non-blocking
any-to-any high-bandwidth interconnects that we call HB domains. Across the HB
domains, the network only connects GPUs with communication demands. We call
this network a "rail-only" connection, and show that our proposed architecture
reduces the network cost by up to 75% compared to the state-of-the-art
any-to-any Clos networks without compromising the performance of LLM training.
Question Decomposition Improves the Faithfulness of Model-Generated
  Reasoning
Authors: name: Ansh Radhakrishnan, name: Karina Nguyen, name: Anna Chen, name: Carol Chen, name: Carson Denison, name: Danny Hernandez, name: Esin Durmus, name: Evan Hubinger, name: Jackson Kernion, name: Kamilė Lukošiūtė, name: Newton Cheng, name: Nicholas Joseph, name: Nicholas Schiefer, name: Oliver Rausch, name: Sam McCandlish, name: Sheer El Showk, name: Tamera Lanham, name: Tim Maxwell, name: Venkatesa Chandrasekaran, name: Zac Hatfield-Dodds, name: Jared Kaplan, name: Jan Brauner, name: Samuel R. Bowman, name: Ethan Perez
Published: 2023-07-17 00:54:10

As large language models (LLMs) perform more difficult tasks, it becomes
harder to verify the correctness and safety of their behavior. One approach to
help with this issue is to prompt LLMs to externalize their reasoning, e.g., by
having them generate step-by-step reasoning as they answer a question
(Chain-of-Thought; CoT). The reasoning may enable us to check the process that
models use to perform tasks. However, this approach relies on the stated
reasoning faithfully reflecting the model's actual reasoning, which is not
always the case. To improve over the faithfulness of CoT reasoning, we have
models generate reasoning by decomposing questions into subquestions.
Decomposition-based methods achieve strong performance on question-answering
tasks, sometimes approaching that of CoT while improving the faithfulness of
the model's stated reasoning on several recently-proposed metrics. By forcing
the model to answer simpler subquestions in separate contexts, we greatly
increase the faithfulness of model-generated reasoning over CoT, while still
achieving some of the performance gains of CoT. Our results show it is possible
to improve the faithfulness of model-generated reasoning; continued
improvements may lead to reasoning that enables us to verify the correctness
and safety of LLM behavior.
PUMA: Secure Inference of LLaMA-7B in Five Minutes
Authors: name: Ye Dong, name: Wen-jie Lu, name: Yancheng Zheng, name: Haoqi Wu, name: Derun Zhao, name: Jin Tan, name: Zhicong Huang, name: Cheng Hong, name: Tao Wei, name: Wenguang Cheng
Published: 2023-07-24 05:43:19

With ChatGPT as a representative, tons of companies have began to provide
services based on large Transformers models. However, using such a service
inevitably leak users' prompts to the model provider. Previous studies have
studied secure inference for Transformer models using secure multiparty
computation (MPC), where model parameters and clients' prompts are kept secret.
Despite this, these frameworks are still limited in terms of model performance,
efficiency, and deployment. To address these limitations, we propose framework
PUMA to enable fast and secure Transformer model inference. Our framework
designs high quality approximations for expensive functions, such as GeLU and
Softmax, which significantly reduce the cost of secure inference while
preserving the model performance. Additionally, we design secure Embedding and
LayerNorm procedures that faithfully implement the desired functionality
without undermining the Transformer architecture. PUMA is about 2x faster than
the state-of-the-art MPC framework MPCFORMER(ICLR 2023) and has similar
accuracy as plaintext models without fine-tuning (which the previous works
failed to achieve).
  One more thing, PUMA can evaluate LLaMA-7B in around 5 minutes to generate 1
token. To our best knowledge, this is the first time that a model with such a
parameter size is able to be evaluated under MPC. PUMA has been open-sourced in
the Github repository of SecretFlow-SPU.
CopyRNeRF: Protecting the CopyRight of Neural Radiance Fields
Authors: name: Ziyuan Luo, name: Qing Guo, name: Ka Chun Cheung, name: Simon See, name: Renjie Wan
Published: 2023-07-21 12:14:33

Neural Radiance Fields (NeRF) have the potential to be a major representation
of media. Since training a NeRF has never been an easy task, the protection of
its model copyright should be a priority. In this paper, by analyzing the pros
and cons of possible copyright protection solutions, we propose to protect the
copyright of NeRF models by replacing the original color representation in NeRF
with a watermarked color representation. Then, a distortion-resistant rendering
scheme is designed to guarantee robust message extraction in 2D renderings of
NeRF. Our proposed method can directly protect the copyright of NeRF models
while maintaining high rendering quality and bit accuracy when compared among
optional solutions.
FaceCLIPNeRF: Text-driven 3D Face Manipulation using Deformable Neural
  Radiance Fields
Authors: name: Sungwon Hwang, name: Junha Hyung, name: Daejin Kim, name: Min-Jung Kim, name: Jaegul Choo
Published: 2023-07-21 08:22:14

As recent advances in Neural Radiance Fields (NeRF) have enabled
high-fidelity 3D face reconstruction and novel view synthesis, its manipulation
also became an essential task in 3D vision. However, existing manipulation
methods require extensive human labor, such as a user-provided semantic mask
and manual attribute search unsuitable for non-expert users. Instead, our
approach is designed to require a single text to manipulate a face
reconstructed with NeRF. To do so, we first train a scene manipulator, a latent
code-conditional deformable NeRF, over a dynamic scene to control a face
deformation using the latent code. However, representing a scene deformation
with a single latent code is unfavorable for compositing local deformations
observed in different instances. As so, our proposed Position-conditional
Anchor Compositor (PAC) learns to represent a manipulated scene with spatially
varying latent codes. Their renderings with the scene manipulator are then
optimized to yield high cosine similarity to a target text in CLIP embedding
space for text-driven manipulation. To the best of our knowledge, our approach
is the first to address the text-driven manipulation of a face reconstructed
with NeRF. Extensive results, comparisons, and ablation studies demonstrate the
effectiveness of our approach.
Subject-Diffusion:Open Domain Personalized Text-to-Image Generation
  without Test-time Fine-tuning
Authors: name: Jian Ma, name: Junhao Liang, name: Chen Chen, name: Haonan Lu
Published: 2023-07-21 08:09:47

Recent progress in personalized image generation using diffusion models has
been significant. However, development in the area of open-domain and
non-fine-tuning personalized image generation is proceeding rather slowly. In
this paper, we propose Subject-Diffusion, a novel open-domain personalized
image generation model that, in addition to not requiring test-time
fine-tuning, also only requires a single reference image to support
personalized generation of single- or multi-subject in any domain. Firstly, we
construct an automatic data labeling tool and use the LAION-Aesthetics dataset
to construct a large-scale dataset consisting of 76M images and their
corresponding subject detection bounding boxes, segmentation masks and text
descriptions. Secondly, we design a new unified framework that combines text
and image semantics by incorporating coarse location and fine-grained reference
image control to maximize subject fidelity and generalization. Furthermore, we
also adopt an attention control mechanism to support multi-subject generation.
Extensive qualitative and quantitative results demonstrate that our method
outperforms other SOTA frameworks in single, multiple, and human customized
image generation. Please refer to our
https://oppo-mente-lab.github.io/subject_diffusion/{project page}
Diffusion Sampling with Momentum for Mitigating Divergence Artifacts
Authors: name: Suttisak Wizadwongsa, name: Worameth Chinchuthakun, name: Pramook Khungurn, name: Amit Raj, name: Supasorn Suwajanakorn
Published: 2023-07-20 14:37:30

Despite the remarkable success of diffusion models in image generation, slow
sampling remains a persistent issue. To accelerate the sampling process, prior
studies have reformulated diffusion sampling as an ODE/SDE and introduced
higher-order numerical methods. However, these methods often produce divergence
artifacts, especially with a low number of sampling steps, which limits the
achievable acceleration. In this paper, we investigate the potential causes of
these artifacts and suggest that the small stability regions of these methods
could be the principal cause. To address this issue, we propose two novel
techniques. The first technique involves the incorporation of Heavy Ball (HB)
momentum, a well-known technique for improving optimization, into existing
diffusion numerical methods to expand their stability regions. We also prove
that the resulting methods have first-order convergence. The second technique,
called Generalized Heavy Ball (GHVB), constructs a new high-order method that
offers a variable trade-off between accuracy and artifact suppression.
Experimental results show that our techniques are highly effective in reducing
artifacts and improving image quality, surpassing state-of-the-art diffusion
solvers on both pixel-based and latent-based diffusion models for low-step
sampling. Our research provides novel insights into the design of numerical
methods for future diffusion work.
FLASK: Fine-grained Language Model Evaluation based on Alignment Skill
  Sets
Authors: name: Seonghyeon Ye, name: Doyoung Kim, name: Sungdong Kim, name: Hyeonbin Hwang, name: Seungone Kim, name: Yongrae Jo, name: James Thorne, name: Juho Kim, name: Minjoon Seo
Published: 2023-07-20 14:56:35

Evaluation of Large Language Models (LLMs) is challenging because aligning to
human values requires the composition of multiple skills and the required set
of skills varies depending on the instruction. Recent studies have evaluated
the performance of LLMs in two ways, (1) automatic evaluation on several
independent benchmarks and (2) human or machined-based evaluation giving an
overall score to the response. However, both settings are coarse-grained
evaluations, not considering the nature of user instructions that require
instance-wise skill composition, which limits the interpretation of the true
capabilities of LLMs. In this paper, we introduce FLASK (Fine-grained Language
Model Evaluation based on Alignment SKill Sets), a fine-grained evaluation
protocol that can be used for both model-based and human-based evaluation which
decomposes coarse-level scoring to an instance-wise skill set-level.
Specifically, we define 12 fine-grained skills needed for LLMs to follow
open-ended user instructions and construct an evaluation set by allocating a
set of skills for each instance. Additionally, by annotating the target domains
and difficulty level for each instance, FLASK provides a holistic view with a
comprehensive analysis of a model's performance depending on skill, domain, and
difficulty. Through using FLASK, we compare multiple open-sourced and
proprietary LLMs and observe highly-correlated findings between model-based and
human-based evaluations. FLASK enables developers to more accurately measure
the model performance and how it can be improved by analyzing factors that make
LLMs proficient in particular skills. For practitioners, FLASK can be used to
recommend suitable models for particular situations through comprehensive
comparison among various LLMs. We release the evaluation data and code
implementation at https://github.com/kaistAI/FLASK.