Orionnnn/hub-stats huggingface dataset:
SVNR: Spatially-variant Noise Removal with Denoising Diffusion
Authors: name: Naama Pearl, name: Yaron Brodsky, name: Dana Berman, name: Assaf Zomet, name: Alex Rav Acha, name: Daniel Cohen-Or, name: Dani Lischinski
Published: 2023-06-28 09:32:00

Denoising diffusion models have recently shown impressive results in
generative tasks. By learning powerful priors from huge collections of training
images, such models are able to gradually modify complete noise to a clean
natural image via a sequence of small denoising steps, seemingly making them
well-suited for single image denoising. However, effectively applying denoising
diffusion models to removal of realistic noise is more challenging than it may
seem, since their formulation is based on additive white Gaussian noise, unlike
noise in real-world images. In this work, we present SVNR, a novel formulation
of denoising diffusion that assumes a more realistic, spatially-variant noise
model. SVNR enables using the noisy input image as the starting point for the
denoising diffusion process, in addition to conditioning the process on it. To
this end, we adapt the diffusion process to allow each pixel to have its own
time embedding, and propose training and inference schemes that support
spatially-varying time maps. Our formulation also accounts for the correlation
that exists between the condition image and the samples along the modified
diffusion process. In our experiments we demonstrate the advantages of our
approach over a strong diffusion model baseline, as well as over a
state-of-the-art single image denoising method.
Towards Language Models That Can See: Computer Vision Through the LENS
  of Natural Language
Authors: name: William Berrios, name: Gautam Mittal, name: Tristan Thrush, name: Douwe Kiela, name: Amanpreet Singh
Published: 2023-06-28 17:57:10

We propose LENS, a modular approach for tackling computer vision problems by
leveraging the power of large language models (LLMs). Our system uses a
language model to reason over outputs from a set of independent and highly
descriptive vision modules that provide exhaustive information about an image.
We evaluate the approach on pure computer vision settings such as zero- and
few-shot object recognition, as well as on vision and language problems. LENS
can be applied to any off-the-shelf LLM and we find that the LLMs with LENS
perform highly competitively with much bigger and much more sophisticated
systems, without any multimodal training whatsoever. We open-source our code at
https://github.com/ContextualAI/lens and provide an interactive demo.
Towards Measuring the Representation of Subjective Global Opinions in
  Language Models
Authors: name: Esin Durmus, name: Karina Nyugen, name: Thomas I. Liao, name: Nicholas Schiefer, name: Amanda Askell, name: Anton Bakhtin, name: Carol Chen, name: Zac Hatfield-Dodds, name: Danny Hernandez, name: Nicholas Joseph, name: Liane Lovitt, name: Sam McCandlish, name: Orowa Sikder, name: Alex Tamkin, name: Janel Thamkul, name: Jared Kaplan, name: Jack Clark, name: Deep Ganguli
Published: 2023-06-28 17:31:53

Large language models (LLMs) may not equitably represent diverse global
perspectives on societal issues. In this paper, we develop a quantitative
framework to evaluate whose opinions model-generated responses are more similar
to. We first build a dataset, GlobalOpinionQA, comprised of questions and
answers from cross-national surveys designed to capture diverse opinions on
global issues across different countries. Next, we define a metric that
quantifies the similarity between LLM-generated survey responses and human
responses, conditioned on country. With our framework, we run three experiments
on an LLM trained to be helpful, honest, and harmless with Constitutional AI.
By default, LLM responses tend to be more similar to the opinions of certain
populations, such as those from the USA, and some European and South American
countries, highlighting the potential for biases. When we prompt the model to
consider a particular country's perspective, responses shift to be more similar
to the opinions of the prompted populations, but can reflect harmful cultural
stereotypes. When we translate GlobalOpinionQA questions to a target language,
the model's responses do not necessarily become the most similar to the
opinions of speakers of those languages. We release our dataset for others to
use and build on. Our data is at
https://huggingface.co/datasets/Anthropic/llm_global_opinions. We also provide
an interactive visualization at https://llmglobalvalues.anthropic.com.
HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide
  Resolution
Authors: name: Eric Nguyen, name: Michael Poli, name: Marjan Faizi, name: Armin Thomas, name: Callum Birch-Sykes, name: Michael Wornow, name: Aman Patel, name: Clayton Rabideau, name: Stefano Massaroli, name: Yoshua Bengio, name: Stefano Ermon, name: Stephen A. Baccus, name: Chris Ré
Published: 2023-06-27 20:46:34

Genomic (DNA) sequences encode an enormous amount of information for gene
regulation and protein synthesis. Similar to natural language models,
researchers have proposed foundation models in genomics to learn generalizable
features from unlabeled genome data that can then be fine-tuned for downstream
tasks such as identifying regulatory elements. Due to the quadratic scaling of
attention, previous Transformer-based genomic models have used 512 to 4k tokens
as context (<0.001% of the human genome), significantly limiting the modeling
of long-range interactions in DNA. In addition, these methods rely on
tokenizers to aggregate meaningful DNA units, losing single nucleotide
resolution where subtle genetic variations can completely alter protein
function via single nucleotide polymorphisms (SNPs). Recently, Hyena, a large
language model based on implicit convolutions was shown to match attention in
quality while allowing longer context lengths and lower time complexity.
Leveraging Hyenas new long-range capabilities, we present HyenaDNA, a genomic
foundation model pretrained on the human reference genome with context lengths
of up to 1 million tokens at the single nucleotide-level, an up to 500x
increase over previous dense attention-based models. HyenaDNA scales
sub-quadratically in sequence length (training up to 160x faster than
Transformer), uses single nucleotide tokens, and has full global context at
each layer. We explore what longer context enables - including the first use of
in-context learning in genomics for simple adaptation to novel tasks without
updating pretrained model weights. On fine-tuned benchmarks from the Nucleotide
Transformer, HyenaDNA reaches state-of-the-art (SotA) on 12 of 17 datasets
using a model with orders of magnitude less parameters and pretraining data. On
the GenomicBenchmarks, HyenaDNA surpasses SotA on all 8 datasets on average by
+9 accuracy points.
CLIPA-v2: Scaling CLIP Training with 81.1% Zero-shot ImageNet Accuracy
  within a \10,000 Budget; An Extra 4,000 Unlocks 81.8% Accuracy
Authors: name: Xianhang Li, name: Zeyu Wang, name: Cihang Xie
Published: 2023-06-27 17:51:06

The recent work CLIPA presents an inverse scaling law for CLIP training --
whereby the larger the image/text encoders used, the shorter the sequence
length of image/text tokens that can be applied in training. This finding
enables us to train high-performance CLIP models with significantly reduced
computations. Building upon this work, we hereby present CLIPA-v2 with two key
contributions. Technically, we find this inverse scaling law is also applicable
in the finetuning stage, enabling further reduction in computational needs.
Empirically, we explore CLIPA at scale, extending the experiments up to the
H/14 model with ~13B image-text pairs seen during training.
  Our results are exciting -- by only allocating a budget of \10,000, our CLIP
model achieves an impressive zero-shot ImageNet accuracy of 81.1%, surpassing
the prior best CLIP model (from OpenCLIP, 80.1%) by 1.0% and meanwhile reducing
the computational cost by ~39X. Moreover, with an additional investment of
4,000, we can further elevate the zero-shot ImageNet accuracy to 81.8%. Our
code and models are available at https://github.com/UCSC-VLAA/CLIPA.
3D-Speaker: A Large-Scale Multi-Device, Multi-Distance, and
  Multi-Dialect Corpus for Speech Representation Disentanglement
Authors: name: Siqi Zheng, name: Luyao Cheng, name: Yafeng Chen, name: Hui Wang, name: Qian Chen
Published: 2023-06-27 10:09:43

Disentangling uncorrelated information in speech utterances is a crucial
research topic within speech community. Different speech-related tasks focus on
extracting distinct speech representations while minimizing the affects of
other uncorrelated information. We present a large-scale speech corpus to
facilitate the research of speech representation disentanglement. 3D-Speaker
contains over 10,000 speakers, each of whom are simultaneously recorded by
multiple Devices, locating at different Distances, and some speakers are
speaking multiple Dialects. The controlled combinations of multi-dimensional
audio data yield a matrix of a diverse blend of speech representation
entanglement, thereby motivating intriguing methods to untangle them. The
multi-domain nature of 3D-Speaker also makes it a suitable resource to evaluate
large universal speech models and experiment methods of out-of-domain learning
and self-supervised learning. https://3dspeaker.github.io/
MIMIC: Masked Image Modeling with Image Correspondences
Authors: name: Kalyani Marathe, name: Mahtab Bigverdi, name: Nishat Khan, name: Tuhin Kundu, name: Aniruddha Kembhavi, name: Linda G. Shapiro, name: Ranjay Krishna
Published: 2023-06-27 00:40:12

Many pixelwise dense prediction tasks-depth estimation and semantic
segmentation in computer vision today rely on pretrained image representations.
Therefore, curating effective pretraining datasets is vital. Unfortunately, the
effective pretraining datasets are those with multi-view scenes and have only
been curated using annotated 3D meshes, point clouds, and camera parameters
from simulated environments. We propose a dataset-curation mechanism that does
not require any annotations. We mine two datasets: MIMIC-1M with 1.3M and
MIMIC-3M with 3.1M multi-view image pairs from open-sourced video datasets and
from synthetic 3D environments. We train multiple self-supervised models with
different masked image modeling objectives to showcase the following findings:
Representations trained on MIMIC-3M outperform those mined using annotations on
multiple downstream tasks, including depth estimation, semantic segmentation,
surface normals, and pose estimation. They also outperform representations that
are frozen and when downstream training data is limited to few-shot. Larger
dataset (MIMIC-3M) significantly improves performance, which is promising since
our curation method can arbitrarily scale to produce even larger datasets.
MIMIC code, dataset, and pretrained models are open-sourced at
https://github.com/RAIVNLab/MIMIC.
Length Generalization in Arithmetic Transformers
Authors: name: Samy Jelassi, name: Stéphane d'Ascoli, name: Carles Domingo-Enrich, name: Yuhuai Wu, name: Yuanzhi Li, name: François Charton
Published: 2023-06-27 11:53:25

We examine how transformers cope with two challenges: learning basic integer
arithmetic, and generalizing to longer sequences than seen during training. We
find that relative position embeddings enable length generalization for simple
tasks, such as addition: models trained on 5-digit numbers can perform
15-digit sums. However, this method fails for multiplication, and we propose
train set priming: adding a few (10 to 50) long sequences to the training
set. We show that priming allows models trained on 5-digit times 3-digit
multiplications to generalize to 35times 3 examples. We also show that
models can be primed for different generalization lengths, and that the priming
sample size scales as the logarithm of the training set size. Finally, we
discuss potential applications of priming beyond arithmetic.
PoseDiffusion: Solving Pose Estimation via Diffusion-aided Bundle
  Adjustment
Authors: name: Jianyuan Wang, name: Christian Rupprecht, name: David Novotny
Published: 2023-06-27 17:59:07

Camera pose estimation is a long-standing computer vision problem that to
date often relies on classical methods, such as handcrafted keypoint matching,
RANSAC and bundle adjustment. In this paper, we propose to formulate the
Structure from Motion (SfM) problem inside a probabilistic diffusion framework,
modelling the conditional distribution of camera poses given input images. This
novel view of an old problem has several advantages. (i) The nature of the
diffusion framework mirrors the iterative procedure of bundle adjustment. (ii)
The formulation allows a seamless integration of geometric constraints from
epipolar geometry. (iii) It excels in typically difficult scenarios such as
sparse views with wide baselines. (iv) The method can predict intrinsics and
extrinsics for an arbitrary amount of images. We demonstrate that our method
PoseDiffusion significantly improves over the classic SfM pipelines and the
learned approaches on two real-world datasets. Finally, it is observed that our
method can generalize across datasets without further training. Project page:
https://posediffusion.github.io/
Extending Context Window of Large Language Models via Positional
  Interpolation
Authors: name: Shouyuan Chen, name: Sherman Wong, name: Liangjian Chen, name: Yuandong Tian
Published: 2023-06-27 16:26:26

We present Position Interpolation (PI) that extends the context window sizes
of RoPE-based pretrained LLMs such as LLaMA models to up to 32768 with minimal
fine-tuning (within 1000 steps), while demonstrating strong empirical results
on various tasks that require long context, including passkey retrieval,
language modeling, and long document summarization from LLaMA 7B to 65B.
Meanwhile, the extended model by Position Interpolation preserve quality
relatively well on tasks within its original context window. To achieve this
goal, Position Interpolation linearly down-scales the input position indices to
match the original context window size, rather than extrapolating beyond the
trained context length which may lead to catastrophically high attention scores
that completely ruin the self-attention mechanism. Our theoretical study shows
that the upper bound of interpolation is at least sim 600 times smaller
than that of extrapolation, further demonstrating its stability. Models
extended via Position Interpolation retain its original architecture and can
reuse most pre-existing optimization and infrastructure.
Are aligned neural networks adversarially aligned?
Authors: name: Nicholas Carlini, name: Milad Nasr, name: Christopher A. Choquette-Choo, name: Matthew Jagielski, name: Irena Gao, name: Anas Awadalla, name: Pang Wei Koh, name: Daphne Ippolito, name: Katherine Lee, name: Florian Tramer, name: Ludwig Schmidt
Published: 2023-06-26 17:18:44

Large language models are now tuned to align with the goals of their
creators, namely to be "helpful and harmless." These models should respond
helpfully to user questions, but refuse to answer requests that could cause
harm. However, adversarial users can construct inputs which circumvent attempts
at alignment. In this work, we study to what extent these models remain
aligned, even when interacting with an adversarial user who constructs
worst-case inputs (adversarial examples). These inputs are designed to cause
the model to emit harmful content that would otherwise be prohibited. We show
that existing NLP-based optimization attacks are insufficiently powerful to
reliably attack aligned text models: even when current NLP-based attacks fail,
we can find adversarial inputs with brute force. As a result, the failure of
current attacks should not be seen as proof that aligned text models remain
aligned under adversarial inputs.
  However the recent trend in large-scale ML models is multimodal models that
allow users to provide images that influence the text that is generated. We
show these models can be easily attacked, i.e., induced to perform arbitrary
un-aligned behavior through adversarial perturbation of the input image. We
conjecture that improved NLP attacks may demonstrate this same level of
adversarial control over text-only models.
Understanding In-Context Learning via Supportive Pretraining Data
Authors: name: Xiaochuang Han, name: Daniel Simig, name: Todor Mihaylov, name: Yulia Tsvetkov, name: Asli Celikyilmaz, name: Tianlu Wang
Published: 2023-06-26 22:14:04

In-context learning (ICL) improves language models' performance on a variety
of NLP tasks by simply demonstrating a handful of examples at inference time.
It is not well understood why ICL ability emerges, as the model has never been
specifically trained on such demonstrations. Unlike prior work that explores
implicit mechanisms behind ICL, we study ICL via investigating the pretraining
data. Specifically, we first adapt an iterative, gradient-based approach to
find a small subset of pretraining data that supports ICL. We observe that a
continued pretraining on this small subset significantly improves the model's
ICL ability, by up to 18%. We then compare the supportive subset constrastively
with random subsets of pretraining data and discover: (1) The supportive
pretraining data to ICL do not have a higher domain relevance to downstream
tasks. (2) The supportive pretraining data have a higher mass of rarely
occurring, long-tail tokens. (3) The supportive pretraining data are
challenging examples where the information gain from long-range context is
below average, indicating learning to incorporate difficult long-range context
encourages ICL. Our work takes a first step towards understanding ICL via
analyzing instance-level pretraining data. Our insights have a potential to
enhance the ICL ability of language models by actively guiding the construction
of pretraining data in the future.
LeanDojo: Theorem Proving with Retrieval-Augmented Language Models
Authors: name: Kaiyu Yang, name: Aidan M. Swope, name: Alex Gu, name: Rahul Chalamala, name: Peiyang Song, name: Shixing Yu, name: Saad Godil, name: Ryan Prenger, name: Anima Anandkumar
Published: 2023-06-27 17:05:32

Large language models (LLMs) have shown promise in proving formal theorems
using proof assistants such as Lean. However, existing methods are difficult to
reproduce or build on, due to private code, data, and large compute
requirements. This has created substantial barriers to research on machine
learning methods for theorem proving. This paper removes these barriers by
introducing LeanDojo: an open-source Lean playground consisting of toolkits,
data, models, and benchmarks. LeanDojo extracts data from Lean and enables
interaction with the proof environment programmatically. It contains
fine-grained annotations of premises in proofs, providing valuable data for
premise selection: a key bottleneck in theorem proving. Using this data, we
develop ReProver (Retrieval-Augmented Prover): the first LLM-based prover that
is augmented with retrieval for selecting premises from a vast math library. It
is inexpensive and needs only one GPU week of training. Our retriever leverages
LeanDojo's program analysis capability to identify accessible premises and hard
negative examples, which makes retrieval much more effective. Furthermore, we
construct a new benchmark consisting of 96,962 theorems and proofs extracted
from Lean's math library. It features challenging data split requiring the
prover to generalize to theorems relying on novel premises that are never used
in training. We use this benchmark for training and evaluation, and
experimental results demonstrate the effectiveness of ReProver over
non-retrieval baselines and GPT-4. We thus provide the first set of open-source
LLM-based theorem provers without any proprietary datasets and release it under
a permissive MIT license to facilitate further research.
Aligning Large Multi-Modal Model with Robust Instruction Tuning
Authors: name: Fuxiao Liu, name: Kevin Lin, name: Linjie Li, name: Jianfeng Wang, name: Yaser Yacoob, name: Lijuan Wang
Published: 2023-06-26 10:26:33

Despite the promising progress in multi-modal tasks, current large
multi-modal models (LMM) are prone to hallucinating inconsistent descriptions
with respect to the associated image and human instructions. This paper
addresses this issue by introducing the first large and diverse visual
instruction tuning dataset, named Large-scale Robust Visual (LRV)-Instruction.
Our dataset consists of 120k visual instructions generated by GPT4, covering 16
vision-and-language tasks with open-ended instructions and answers. Unlike
existing studies that primarily focus on positive instruction samples, we
design LRV-Instruction to include both positive and negative instructions for
more robust visual instruction tuning. Our negative instructions are designed
at two semantic levels: (i) Nonexistent Element Manipulation and (ii) Existent
Element Manipulation. To efficiently measure the hallucination generated by
LMMs, we propose GPT4-Assisted Visual Instruction Evaluation (GAVIE), a novel
approach to evaluate visual instruction tuning without the need for
human-annotated groundtruth answers and can adapt to diverse instruction
formats. We conduct comprehensive experiments to investigate the hallucination
of LMMs. Our results demonstrate that existing LMMs exhibit significant
hallucination when presented with our negative instructions, particularly with
Existent Element Manipulation instructions. Moreover, by finetuning MiniGPT4 on
LRV-Instruction, we successfully mitigate hallucination while improving
performance on public datasets using less training data compared to
state-of-the-art methods. Additionally, we observed that a balanced ratio of
positive and negative instances in the training data leads to a more robust
model. Our project link is available at https://fuxiaoliu.github.io/LRV/.
RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse
  Tools
Authors: name: Haochen Shi, name: Huazhe Xu, name: Samuel Clarke, name: Yunzhu Li, name: Jiajun Wu
Published: 2023-06-26 06:30:29

Humans excel in complex long-horizon soft body manipulation tasks via
flexible tool use: bread baking requires a knife to slice the dough and a
rolling pin to flatten it. Often regarded as a hallmark of human cognition,
tool use in autonomous robots remains limited due to challenges in
understanding tool-object interactions. Here we develop an intelligent robotic
system, RoboCook, which perceives, models, and manipulates elasto-plastic
objects with various tools. RoboCook uses point cloud scene representations,
models tool-object interactions with Graph Neural Networks (GNNs), and combines
tool classification with self-supervised policy learning to devise manipulation
plans. We demonstrate that from just 20 minutes of real-world interaction data
per tool, a general-purpose robot arm can learn complex long-horizon soft
object manipulation tasks, such as making dumplings and alphabet letter
cookies. Extensive evaluations show that RoboCook substantially outperforms
state-of-the-art approaches, exhibits robustness against severe external
disturbances, and demonstrates adaptability to different materials.