Orionnnn/hub-stats huggingface dataset:
MultiLoRA: Democratizing LoRA for Better Multi-Task Learning
Authors: name: Yiming Wang, name: Yu Lin, name: Xiaodong Zeng, name: Guannan Zhang
Published: 2023-11-20 02:59:18

LoRA achieves remarkable resource efficiency and comparable performance when
adapting LLMs for specific tasks. Since ChatGPT demonstrated superior
performance on various tasks, there has been a growing desire to adapt one
model for all tasks. However, the explicit low-rank of LoRA limits the
adaptation performance in complex multi-task scenarios. LoRA is dominated by a
small number of top singular vectors while fine-tuning decomposes into a set of
less important unitary transforms. In this paper, we propose MultiLoRA for
better multi-task adaptation by reducing the dominance of top singular vectors
observed in LoRA. MultiLoRA scales LoRA modules horizontally and change
parameter initialization of adaptation matrices to reduce parameter dependency,
thus yields more balanced unitary subspaces. We unprecedentedly construct
specialized training data by mixing datasets of instruction follow, natural
language understanding, world knowledge, to cover semantically and
syntactically different samples. With only 2.5% of additional parameters,
MultiLoRA outperforms single LoRA counterparts and fine-tuning on multiple
benchmarks and model scales. Further investigation into weight update matrices
of MultiLoRA exhibits reduced dependency on top singular vectors and more
democratic unitary transform contributions.
GPQA: A Graduate-Level Google-Proof Q&A Benchmark
Authors: name: David Rein, name: Betty Li Hou, name: Asa Cooper Stickland, name: Jackson Petty, name: Richard Yuanzhe Pang, name: Julien Dirani, name: Julian Michael, name: Samuel R. Bowman
Published: 2023-11-20 18:57:34

We present GPQA, a challenging dataset of 448 multiple-choice questions
written by domain experts in biology, physics, and chemistry. We ensure that
the questions are high-quality and extremely difficult: experts who have or are
pursuing PhDs in the corresponding domains reach 65% accuracy (74% when
discounting clear mistakes the experts identified in retrospect), while highly
skilled non-expert validators only reach 34% accuracy, despite spending on
average over 30 minutes with unrestricted access to the web (i.e., the
questions are "Google-proof"). The questions are also difficult for
state-of-the-art AI systems, with our strongest GPT-4 based baseline achieving
39% accuracy. If we are to use future AI systems to help us answer very hard
questions, for example, when developing new scientific knowledge, we need to
develop scalable oversight methods that enable humans to supervise their
outputs, which may be difficult even if the supervisors are themselves skilled
and knowledgeable. The difficulty of GPQA both for skilled non-experts and
frontier AI systems should enable realistic scalable oversight experiments,
which we hope can help devise ways for human experts to reliably get truthful
information from AI systems that surpass human capabilities.
GPT-4V(ision) for Robotics: Multimodal Task Planning from Human
  Demonstration
Authors: name: Naoki Wake, name: Atsushi Kanehira, name: Kazuhiro Sasabuchi, name: Jun Takamatsu, name: Katsushi Ikeuchi
Published: 2023-11-20 18:54:39

We introduce a pipeline that enhances a general-purpose Vision Language
Model, GPT-4V(ision), by integrating observations of human actions to
facilitate robotic manipulation. This system analyzes videos of humans
performing tasks and creates executable robot programs that incorporate
affordance insights. The computation starts by analyzing the videos with GPT-4V
to convert environmental and action details into text, followed by a
GPT-4-empowered task planner. In the following analyses, vision systems
reanalyze the video with the task plan. Object names are grounded using an
open-vocabulary object detector, while focus on the hand-object relation helps
to detect the moment of grasping and releasing. This spatiotemporal grounding
allows the vision systems to further gather affordance data (e.g., grasp type,
way points, and body postures). Experiments across various scenarios
demonstrate this method's efficacy in achieving real robots' operations from
human demonstrations in a zero-shot manner. The prompts of GPT-4V/GPT-4 are
available at this project page:
https://microsoft.github.io/GPT4Vision-Robot-Manipulation-Prompts/
ToolTalk: Evaluating Tool-Usage in a Conversational Setting
Authors: name: Nicholas Farn, name: Richard Shin
Published: 2023-11-15 23:50:31

Large language models (LLMs) have displayed massive improvements in reason-
ing and decision-making skills and can hold natural conversations with users.
Many recent works seek to augment LLM-based assistants with external tools so
they can access private or up-to-date information and carry out actions on
behalf of users. To better measure the performance of these assistants, this
paper introduces ToolTalk, a benchmark consisting of complex user intents re-
quiring multi-step tool usage specified through dialogue. ToolTalk contains 28
tools grouped into 7 plugins, and includes a complete simulated implementa-
tion of each tool, allowing for fully automated evaluation of assistants that
rely on execution feedback. ToolTalk also emphasizes tools that externally
affect the world rather than only tools for referencing or searching
information. We evaluate GPT-3.5 and GPT-4 on ToolTalk resulting in success
rates of 26% and 50% respectively. Our analysis of the errors reveals three
major categories and suggests some future directions for improvement. We
release ToolTalk at https://github.com/microsoft/ToolTalk.
I&S-ViT: An Inclusive & Stable Method for Pushing the Limit of
  Post-Training ViTs Quantization
Authors: name: Yunshan Zhong, name: Jiawei Hu, name: Mingbao Lin, name: Mengzhao Chen, name: Rongrong Ji
Published: 2023-11-16 13:07:47

Albeit the scalable performance of vision transformers (ViTs), the dense
computational costs (training & inference) undermine their position in
industrial applications. Post-training quantization (PTQ), tuning ViTs with a
tiny dataset and running in a low-bit format, well addresses the cost issue but
unluckily bears more performance drops in lower-bit cases. In this paper, we
introduce I&S-ViT, a novel method that regulates the PTQ of ViTs in an
inclusive and stable fashion. I&S-ViT first identifies two issues in the PTQ of
ViTs: (1) Quantization inefficiency in the prevalent log2 quantizer for
post-Softmax activations; (2) Rugged and magnified loss landscape in
coarse-grained quantization granularity for post-LayerNorm activations. Then,
I&S-ViT addresses these issues by introducing: (1) A novel shift-uniform-log2
quantizer (SULQ) that incorporates a shift mechanism followed by uniform
quantization to achieve both an inclusive domain representation and accurate
distribution approximation; (2) A three-stage smooth optimization strategy
(SOS) that amalgamates the strengths of channel-wise and layer-wise
quantization to enable stable learning. Comprehensive evaluations across
diverse vision tasks validate I&S-ViT' superiority over existing PTQ of ViTs
methods, particularly in low-bit scenarios. For instance, I&S-ViT elevates the
performance of 3-bit ViT-B by an impressive 50.68%.
SelfEval: Leveraging the discriminative nature of generative models for
  evaluation
Authors: name: Sai Saketh Rambhatla, name: Ishan Misra
Published: 2023-11-17 18:58:16

In this work, we show that text-to-image generative models can be 'inverted'
to assess their own text-image understanding capabilities in a completely
automated manner.
  Our method, called SelfEval, uses the generative model to compute the
likelihood of real images given text prompts, making the generative model
directly applicable to discriminative tasks.
  Using SelfEval, we repurpose standard datasets created for evaluating
multimodal text-image discriminative models to evaluate generative models in a
fine-grained manner: assessing their performance on attribute binding, color
recognition, counting, shape recognition, spatial understanding.
  To the best of our knowledge SelfEval is the first automated metric to show a
high degree of agreement for measuring text-faithfulness with the gold-standard
human evaluations across multiple models and benchmarks.
  Moreover, SelfEval enables us to evaluate generative models on challenging
tasks such as Winoground image-score where they demonstrate competitive
performance to discriminative models.
  We also show severe drawbacks of standard automated metrics such as
CLIP-score to measure text faithfulness on benchmarks such as DrawBench, and
how SelfEval sidesteps these issues.
  We hope SelfEval enables easy and reliable automated evaluation for diffusion
models.
Distilling and Retrieving Generalizable Knowledge for Robot Manipulation
  via Language Corrections
Authors: name: Lihan Zha, name: Yuchen Cui, name: Li-Heng Lin, name: Minae Kwon, name: Montserrat Gonzalez Arenas, name: Andy Zeng, name: Fei Xia, name: Dorsa Sadigh
Published: 2023-11-17 18:00:20

Today's robot policies exhibit subpar performance when faced with the
challenge of generalizing to novel environments. Human corrective feedback is a
crucial form of guidance to enable such generalization. However, adapting to
and learning from online human corrections is a non-trivial endeavor: not only
do robots need to remember human feedback over time to retrieve the right
information in new settings and reduce the intervention rate, but also they
would need to be able to respond to feedback that can be arbitrary corrections
about high-level human preferences to low-level adjustments to skill
parameters. In this work, we present Distillation and Retrieval of Online
Corrections (DROC), a large language model (LLM)-based system that can respond
to arbitrary forms of language feedback, distill generalizable knowledge from
corrections, and retrieve relevant past experiences based on textual and visual
similarity for improving performance in novel settings. DROC is able to respond
to a sequence of online language corrections that address failures in both
high-level task plans and low-level skill primitives. We demonstrate that DROC
effectively distills the relevant information from the sequence of online
corrections in a knowledge base and retrieves that knowledge in settings with
new task or object instances. DROC outperforms other techniques that directly
generate robot code via LLMs by using only half of the total number of
corrections needed in the first round and requires little to no corrections
after two iterations. We show further results, videos, prompts and code on
https://sites.google.com/stanford.edu/droc .
Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as
  an Alternative to Attention Layers in Transformers
Authors: name: Vukasin Bozic, name: Danilo Dordervic, name: Daniele Coppola, name: Joseph Thommes
Published: 2023-11-17 16:58:52

This work presents an analysis of the effectiveness of using standard shallow
feed-forward networks to mimic the behavior of the attention mechanism in the
original Transformer model, a state-of-the-art architecture for
sequence-to-sequence tasks. We substitute key elements of the attention
mechanism in the Transformer with simple feed-forward networks, trained using
the original components via knowledge distillation. Our experiments, conducted
on the IWSLT2017 dataset, reveal the capacity of these "attentionless
Transformers" to rival the performance of the original architecture. Through
rigorous ablation studies, and experimenting with various replacement network
types and sizes, we offer insights that support the viability of our approach.
This not only sheds light on the adaptability of shallow feed-forward networks
in emulating attention mechanisms but also underscores their potential to
streamline complex architectures for sequence-to-sequence tasks.
MetaDreamer: Efficient Text-to-3D Creation With Disentangling Geometry
  and Texture
Authors: name: Lincong Feng, name: Muyu Wang, name: Maoyu Wang, name: Kuo Xu, name: Xiaoli Liu
Published: 2023-11-16 11:35:10

Generative models for 3D object synthesis have seen significant advancements
with the incorporation of prior knowledge distilled from 2D diffusion models.
Nevertheless, challenges persist in the form of multi-view geometric
inconsistencies and slow generation speeds within the existing 3D synthesis
frameworks. This can be attributed to two factors: firstly, the deficiency of
abundant geometric a priori knowledge in optimization, and secondly, the
entanglement issue between geometry and texture in conventional 3D generation
methods.In response, we introduce MetaDreammer, a two-stage optimization
approach that leverages rich 2D and 3D prior knowledge. In the first stage, our
emphasis is on optimizing the geometric representation to ensure multi-view
consistency and accuracy of 3D objects. In the second stage, we concentrate on
fine-tuning the geometry and optimizing the texture, thereby achieving a more
refined 3D object. Through leveraging 2D and 3D prior knowledge in two stages,
respectively, we effectively mitigate the interdependence between geometry and
texture. MetaDreamer establishes clear optimization objectives for each stage,
resulting in significant time savings in the 3D generation process. Ultimately,
MetaDreamer can generate high-quality 3D objects based on textual prompts
within 20 minutes, and to the best of our knowledge, it is the most efficient
text-to-3D generation method. Furthermore, we introduce image control into the
process, enhancing the controllability of 3D generation. Extensive empirical
evidence confirms that our method is not only highly efficient but also
achieves a quality level that is at the forefront of current state-of-the-art
3D generation techniques.
Emu Video: Factorizing Text-to-Video Generation by Explicit Image
  Conditioning
Authors: name: Rohit Girdhar, name: Mannat Singh, name: Andrew Brown, name: Quentin Duval, name: Samaneh Azadi, name: Sai Saketh Rambhatla, name: Akbar Shah, name: Xi Yin, name: Devi Parikh, name: Ishan Misra
Published: 2023-11-17 18:59:04

We present Emu Video, a text-to-video generation model that factorizes the
generation into two steps: first generating an image conditioned on the text,
and then generating a video conditioned on the text and the generated image. We
identify critical design decisions--adjusted noise schedules for diffusion, and
multi-stage training--that enable us to directly generate high quality and high
resolution videos, without requiring a deep cascade of models as in prior work.
In human evaluations, our generated videos are strongly preferred in quality
compared to all prior work--81% vs. Google's Imagen Video, 90% vs. Nvidia's
PYOCO, and 96% vs. Meta's Make-A-Video. Our model outperforms commercial
solutions such as RunwayML's Gen2 and Pika Labs. Finally, our factorizing
approach naturally lends itself to animating images based on a user's text
prompt, where our generations are preferred 96% over prior work.
Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2
Authors: name: Hamish Ivison, name: Yizhong Wang, name: Valentina Pyatkin, name: Nathan Lambert, name: Matthew Peters, name: Pradeep Dasigi, name: Joel Jang, name: David Wadden, name: Noah A. Smith, name: Iz Beltagy, name: Hannaneh Hajishirzi
Published: 2023-11-17 18:45:45

Since the release of T\"ULU [Wang et al., 2023b], open resources for
instruction tuning have developed quickly, from better base models to new
finetuning techniques. We test and incorporate a number of these advances into
T\"ULU, resulting in T\"ULU 2, a suite of improved T\"ULU models for advancing
the understanding and best practices of adapting pretrained language models to
downstream tasks and user preferences. Concretely, we release: (1)
T\"ULU-V2-mix, an improved collection of high-quality instruction datasets; (2)
T\"ULU 2, LLAMA-2 models finetuned on the V2 mixture; (3) T\"ULU 2+DPO, T\"ULU
2 models trained with direct preference optimization (DPO), including the
largest DPO-trained model to date (T\"ULU 2+DPO 70B); (4) CODE T\"ULU 2, CODE
LLAMA models finetuned on our V2 mix that outperform CODE LLAMA and its
instruction-tuned variant, CODE LLAMA-Instruct. Our evaluation from multiple
perspectives shows that the T\"ULU 2 suite achieves state-of-the-art
performance among open models and matches or exceeds the performance of
GPT-3.5-turbo-0301 on several benchmarks. We release all the checkpoints, data,
training and evaluation code to facilitate future open efforts on adapting
large language models.
Testing Language Model Agents Safely in the Wild
Authors: name: Silen Naihin, name: David Atkinson, name: Marc Green, name: Merwane Hamadi, name: Craig Swift, name: Douglas Schonholtz, name: Adam Tauman Kalai, name: David Bau
Published: 2023-11-17 14:06:05

A prerequisite for safe autonomy-in-the-wild is safe testing-in-the-wild. Yet
real-world autonomous tests face several unique safety challenges, both due to
the possibility of causing harm during a test, as well as the risk of
encountering new unsafe agent behavior through interactions with real-world and
potentially malicious actors. We propose a framework for conducting safe
autonomous agent tests on the open internet: agent actions are audited by a
context-sensitive monitor that enforces a stringent safety boundary to stop an
unsafe test, with suspect behavior ranked and logged to be examined by humans.
We a design a basic safety monitor that is flexible enough to monitor existing
LLM agents, and, using an adversarial simulated agent, we measure its ability
to identify and stop unsafe situations. Then we apply the safety monitor on a
battery of real-world tests of AutoGPT, and we identify several limitations and
challenges that will face the creation of safe in-the-wild tests as autonomous
agents grow more capable.
VideoCon: Robust Video-Language Alignment via Contrast Captions
Authors: name: Hritik Bansal, name: Yonatan Bitton, name: Idan Szpektor, name: Kai-Wei Chang, name: Aditya Grover
Published: 2023-11-15 19:51:57

Despite being (pre)trained on a massive amount of data, state-of-the-art
video-language alignment models are not robust to semantically-plausible
contrastive changes in the video captions. Our work addresses this by
identifying a broad spectrum of contrast misalignments, such as replacing
entities, actions, and flipping event order, which alignment models should be
robust against. To this end, we introduce the VideoCon, a video-language
alignment dataset constructed by a large language model that generates
plausible contrast video captions and explanations for differences between
original and contrast video captions. Then, a generative video-language model
is finetuned with VideoCon to assess video-language entailment and generate
explanations. Our VideoCon-based alignment model significantly outperforms
current models. It exhibits a 12-point increase in AUC for the video-language
alignment task on human-generated contrast captions. Finally, our model sets
new state of the art zero-shot performance in temporally-extensive
video-language tasks such as text-to-video retrieval (SSv2-Temporal) and video
question answering (ATP-Hard). Moreover, our model shows superior performance
on novel videos and human-crafted captions and explanations. Our code and data
are available at https://github.com/Hritikbansal/videocon.
Video-LLaVA: Learning United Visual Representation by Alignment Before
  Projection
Authors: name: Bin Lin, name: Bin Zhu, name: Yang Ye, name: Munan Ning, name: Peng Jin, name: Li Yuan
Published: 2023-11-16 10:59:44

The Large Vision-Language Model (LVLM) has enhanced the performance of
various downstream tasks in visual-language understanding. Most existing
approaches encode images and videos into separate feature spaces, which are
then fed as inputs to large language models. However, due to the lack of
unified tokenization for images and videos, namely misalignment before
projection, it becomes challenging for a Large Language Model (LLM) to learn
multi-modal interactions from several poor projection layers. In this work, we
unify visual representation into the language feature space to advance the
foundational LLM towards a unified LVLM. As a result, we establish a simple but
robust LVLM baseline, Video-LLaVA, which learns from a mixed dataset of images
and videos, mutually enhancing each other. Video-LLaVA achieves superior
performances on a broad range of 9 image benchmarks across 5 image
question-answering datasets and 4 image benchmark toolkits. Additionally, our
Video-LLaVA also outperforms Video-ChatGPT by 5.8%, 9.9%, 18.6%, and 10.1% on
MSRVTT, MSVD, TGIF, and ActivityNet, respectively. Notably, extensive
experiments demonstrate that Video-LLaVA mutually benefits images and videos
within a unified visual representation, outperforming models designed
specifically for images or videos.
UnifiedVisionGPT: Streamlining Vision-Oriented AI through Generalized
  Multimodal Framework
Authors: name: Chris Kelly, name: Luhui Hu, name: Cindy Yang, name: Yu Tian, name: Deshun Yang, name: Bang Yang, name: Zaoshan Huang, name: Zihao Li, name: Yuexian Zou
Published: 2023-11-16 13:01:25

In the current landscape of artificial intelligence, foundation models serve
as the bedrock for advancements in both language and vision domains. OpenAI
GPT-4 has emerged as the pinnacle in large language models (LLMs), while the
computer vision (CV) domain boasts a plethora of state-of-the-art (SOTA) models
such as Meta's SAM and DINO, and YOLOS. However, the financial and
computational burdens of training new models from scratch remain a significant
barrier to progress. In response to this challenge, we introduce
UnifiedVisionGPT, a novel framework designed to consolidate and automate the
integration of SOTA vision models, thereby facilitating the development of
vision-oriented AI. UnifiedVisionGPT distinguishes itself through four key
features: (1) provides a versatile multimodal framework adaptable to a wide
range of applications, building upon the strengths of multimodal foundation
models; (2) seamlessly integrates various SOTA vision models to create a
comprehensive multimodal platform, capitalizing on the best components of each
model; (3) prioritizes vision-oriented AI, ensuring a more rapid progression in
the CV domain compared to the current trajectory of LLMs; and (4) introduces
automation in the selection of SOTA vision models, generating optimal results
based on diverse multimodal inputs such as text prompts and images. This paper
outlines the architecture and capabilities of UnifiedVisionGPT, demonstrating
its potential to revolutionize the field of computer vision through enhanced
efficiency, versatility, generalization, and performance. Our implementation,
along with the unified multimodal framework and comprehensive dataset, is made
publicly available at https://github.com/LHBuilder/SA-Segment-Anything.