Orionnnn/hub-stats huggingface dataset:
Teaching Language Models to Self-Improve through Interactive
  Demonstrations
Authors: name: Xiao Yu, name: Baolin Peng, name: Michel Galley, name: Jianfeng Gao, name: Zhou Yu
Published: 2023-10-20 14:11:04

The self-improving ability of large language models (LLMs), enabled by
prompting them to analyze and revise their own outputs, has garnered
significant interest in recent research. However, this ability has been shown
to be absent and difficult to learn for smaller models, thus widening the
performance gap between state-of-the-art LLMs and more cost-effective and
faster ones. To reduce this gap, we introduce TriPosT, a training algorithm
that endows smaller models with such self-improvement ability, and show that
our approach can improve a LLaMA-7b's performance on math and reasoning tasks
by up to 7.13%. In contrast to prior work, we achieve this by using the smaller
model to interact with LLMs to collect feedback and improvements on its own
generations. We then replay this experience to train the small model. Our
experiments on four math and reasoning datasets show that the interactive
experience of learning from and correcting its own mistakes is crucial for
small models to improve their performance.
Tuna: Instruction Tuning using Feedback from Large Language Models
Authors: name: Haoran Li, name: Yiran Liu, name: Xingxing Zhang, name: Wei Lu, name: Furu Wei
Published: 2023-10-20 09:55:06

Instruction tuning of open-source large language models (LLMs) like LLaMA,
using direct outputs from more powerful LLMs such as Instruct-GPT and GPT-4,
has proven to be a cost-effective way to align model behaviors with human
preferences. However, the instruction-tuned model has only seen one response
per instruction, lacking the knowledge of potentially better responses. In this
paper, we propose finetuning an instruction-tuned LLM using our novel
probabilistic ranking and contextual ranking approaches to
increase the likelihood of generating better responses. Probabilistic ranking
enables the instruction-tuned model to inherit the relative rankings of
high-quality and low-quality responses from the teacher LLM. On the other hand,
learning with contextual ranking allows the model to refine its own response
distribution using the contextual understanding ability of stronger LLMs.
Furthermore, we apply probabilistic ranking and contextual ranking sequentially
to the instruction-tuned LLM. The resulting model, which we call Tuna,
consistently improves the performance on Super Natural Instructions (119 test
tasks), LMentry (25 test tasks), Vicuna QA, and can even obtain better results
than several strong reinforcement learning baselines. Our code and data are
available at  https://github.com/microsoft/LMOps.
Democratizing Reasoning Ability: Tailored Learning from Large Language
  Model
Authors: name: Zhaoyang Wang, name: Shaohan Huang, name: Yuxuan Liu, name: Jiahai Wang, name: Minghui Song, name: Zihan Zhang, name: Haizhen Huang, name: Furu Wei, name: Weiwei Deng, name: Feng Sun, name: Qi Zhang
Published: 2023-10-20 07:50:10

Large language models (LLMs) exhibit impressive emergent abilities in natural
language processing, but their democratization is hindered due to huge
computation requirements and closed-source nature. Recent research on advancing
open-source smaller LMs by distilling knowledge from black-box LLMs has
obtained promising results in the instruction-following ability. However, the
reasoning ability which is more challenging to foster, is relatively rarely
explored. In this paper, we propose a tailored learning approach to distill
such reasoning ability to smaller LMs to facilitate the democratization of the
exclusive reasoning ability. In contrast to merely employing LLM as a data
annotator, we exploit the potential of LLM as a reasoning teacher by building
an interactive multi-round learning paradigm. This paradigm enables the student
to expose its deficiencies to the black-box teacher who then can provide
customized training data in return. Further, to exploit the reasoning potential
of the smaller LM, we propose self-reflection learning to motivate the student
to learn from self-made mistakes. The learning from self-reflection and LLM are
all tailored to the student's learning status, thanks to the seamless
integration with the multi-round learning paradigm. Comprehensive experiments
and analysis on mathematical and commonsense reasoning tasks demonstrate the
effectiveness of our method. The code will be available at
https://github.com/Raibows/Learn-to-Reason.
SALMONN: Towards Generic Hearing Abilities for Large Language Models
Authors: name: Changli Tang, name: Wenyi Yu, name: Guangzhi Sun, name: Xianzhao Chen, name: Tian Tan, name: Wei Li, name: Lu Lu, name: Zejun Ma, name: Chao Zhang
Published: 2023-10-20 05:41:57

Hearing is arguably an essential ability of artificial intelligence (AI)
agents in the physical world, which refers to the perception and understanding
of general auditory information consisting of at least three types of sounds:
speech, audio events, and music. In this paper, we propose SALMONN, a speech
audio language music open neural network, built by integrating a pre-trained
text-based large language model (LLM) with speech and audio encoders into a
single multimodal model. SALMONN enables the LLM to directly process and
understand general audio inputs and achieve competitive performances on a
number of speech and audio tasks used in training, such as automatic speech
recognition and translation, auditory-information-based question answering,
emotion recognition, speaker verification, and music and audio captioning
etc. SALMONN also has a diverse set of emergent abilities unseen in
the training, which includes but is not limited to speech translation to
untrained languages, speech-based slot filling, spoken-query-based question
answering, audio-based storytelling, and speech audio co-reasoning
etc. The presence of the cross-modal emergent abilities is studied,
and a novel few-shot activation tuning approach is proposed to activate such
abilities of SALMONN. To our knowledge, SALMONN is the first model of its type
and can be regarded as a step towards AI with generic hearing abilities. An
interactive demo of SALMONN is available at
\url{https://github.com/bytedance/SALMONN}, and the training code and
model checkpoints will be released upon acceptance.
ToolChain*: Efficient Action Space Navigation in Large Language Models
  with A* Search
Authors: name: Yuchen Zhuang, name: Xiang Chen, name: Tong Yu, name: Saayan Mitra, name: Victor Bursztyn, name: Ryan A. Rossi, name: Somdeb Sarkhel, name: Chao Zhang
Published: 2023-10-20 02:24:35

Large language models (LLMs) have demonstrated powerful decision-making and
planning capabilities in solving complicated real-world problems. LLM-based
autonomous agents can interact with diverse tools (e.g., functional APIs) and
generate solution plans that execute a series of API function calls in a
step-by-step manner. The multitude of candidate API function calls
significantly expands the action space, amplifying the critical need for
efficient action space navigation. However, existing methods either struggle
with unidirectional exploration in expansive action spaces, trapped into a
locally optimal solution, or suffer from exhaustively traversing all potential
actions, causing inefficient navigation. To address these issues, we propose
ToolChain*, an efficient tree search-based planning algorithm for LLM-based
agents. It formulates the entire action space as a decision tree, where each
node represents a possible API function call involved in a solution plan. By
incorporating the A* search algorithm with task-specific cost function design,
it efficiently prunes high-cost branches that may involve incorrect actions,
identifying the most low-cost valid path as the solution. Extensive experiments
on multiple tool-use and reasoning tasks demonstrate that ToolChain*
efficiently balances exploration and exploitation within an expansive action
space. It outperforms state-of-the-art baselines on planning and reasoning
tasks by 3.1% and 3.5% on average while requiring 7.35x and 2.31x less time,
respectively.
Auto-Instruct: Automatic Instruction Generation and Ranking for
  Black-Box Language Models
Authors: name: Zhihan Zhang, name: Shuohang Wang, name: Wenhao Yu, name: Yichong Xu, name: Dan Iter, name: Qingkai Zeng, name: Yang Liu, name: Chenguang Zhu, name: Meng Jiang
Published: 2023-10-19 19:52:55

Large language models (LLMs) can perform a wide range of tasks by following
natural language instructions, without the necessity of task-specific
fine-tuning. Unfortunately, the performance of LLMs is greatly influenced by
the quality of these instructions, and manually writing effective instructions
for each task is a laborious and subjective process. In this paper, we
introduce Auto-Instruct, a novel method to automatically improve the quality of
instructions provided to LLMs. Our method leverages the inherent generative
ability of LLMs to produce diverse candidate instructions for a given task, and
then ranks them using a scoring model trained on a variety of 575 existing NLP
tasks. In experiments on 118 out-of-domain tasks, Auto-Instruct surpasses both
human-written instructions and existing baselines of LLM-generated
instructions. Furthermore, our method exhibits notable generalizability even
with other LLMs that are not incorporated into its training process.
Creative Robot Tool Use with Large Language Models
Authors: name: Mengdi Xu, name: Peide Huang, name: Wenhao Yu, name: Shiqi Liu, name: Xilun Zhang, name: Yaru Niu, name: Tingnan Zhang, name: Fei Xia, name: Jie Tan, name: Ding Zhao
Published: 2023-10-19 18:02:15

Tool use is a hallmark of advanced intelligence, exemplified in both animal
behavior and robotic capabilities. This paper investigates the feasibility of
imbuing robots with the ability to creatively use tools in tasks that involve
implicit physical constraints and long-term planning. Leveraging Large Language
Models (LLMs), we develop RoboTool, a system that accepts natural language
instructions and outputs executable code for controlling robots in both
simulated and real-world environments. RoboTool incorporates four pivotal
components: (i) an "Analyzer" that interprets natural language to discern key
task-related concepts, (ii) a "Planner" that generates comprehensive strategies
based on the language input and key concepts, (iii) a "Calculator" that
computes parameters for each skill, and (iv) a "Coder" that translates these
plans into executable Python code. Our results show that RoboTool can not only
comprehend explicit or implicit physical constraints and environmental factors
but also demonstrate creative tool use. Unlike traditional Task and Motion
Planning (TAMP) methods that rely on explicit optimization, our LLM-based
system offers a more flexible, efficient, and user-friendly solution for
complex robotics tasks. Through extensive experiments, we validate that
RoboTool is proficient in handling tasks that would otherwise be infeasible
without the creative use of tools, thereby expanding the capabilities of
robotic systems. Demos are available on our project page:
https://creative-robotool.github.io/.
H2O Open Ecosystem for State-of-the-art Large Language Models
Authors: name: Arno Candel, name: Jon McKinney, name: Philipp Singer, name: Pascal Pfeiffer, name: Maximilian Jeblick, name: Chun Ming Lee, name: Marcos V. Conde
Published: 2023-10-17 09:40:58

Large Language Models (LLMs) represent a revolution in AI. However, they also
pose many significant risks, such as the presence of biased, private,
copyrighted or harmful text. For this reason we need open, transparent and safe
solutions. We introduce a complete open-source ecosystem for developing and
testing LLMs. The goal of this project is to boost open alternatives to
closed-source approaches. We release h2oGPT, a family of fine-tuned LLMs from 7
to 70 Billion parameters. We also introduce H2O LLM Studio, a framework and
no-code GUI designed for efficient fine-tuning, evaluation, and deployment of
LLMs using the most recent state-of-the-art techniques. Our code and models are
licensed under fully permissive Apache 2.0 licenses. We believe open-source
language models help to boost AI development and make it more accessible and
trustworthy. The demo is available at: https://gpt.h2o.ai/
Vision-Language Models are Zero-Shot Reward Models for Reinforcement
  Learning
Authors: name: Juan Rocamonde, name: Victoriano Montesinos, name: Elvis Nava, name: Ethan Perez, name: David Lindner
Published: 2023-10-19 17:17:06

Reinforcement learning (RL) requires either manually specifying a reward
function, which is often infeasible, or learning a reward model from a large
amount of human feedback, which is often very expensive. We study a more
sample-efficient alternative: using pretrained vision-language models (VLMs) as
zero-shot reward models (RMs) to specify tasks via natural language. We propose
a natural and general approach to using VLMs as reward models, which we call
VLM-RMs. We use VLM-RMs based on CLIP to train a MuJoCo humanoid to learn
complex tasks without a manually specified reward function, such as kneeling,
doing the splits, and sitting in a lotus position. For each of these tasks, we
only provide a single sentence text prompt describing the desired task with
minimal prompt engineering. We provide videos of the trained agents at:
https://sites.google.com/view/vlm-rm. We can improve performance by providing a
second ``baseline'' prompt and projecting out parts of the CLIP embedding space
irrelevant to distinguish between goal and baseline. Further, we find a strong
scaling effect for VLM-RMs: larger VLMs trained with more compute and data are
better reward models. The failure modes of VLM-RMs we encountered are all
related to known capability limitations of current VLMs, such as limited
spatial reasoning ability or visually unrealistic environments that are far
off-distribution for the VLM. We find that VLM-RMs are remarkably robust as
long as the VLM is large enough. This suggests that future VLMs will become
more and more useful reward models for a wide range of RL applications.
Enhancing High-Resolution 3D Generation through Pixel-wise Gradient
  Clipping
Authors: name: Zijie Pan, name: Jiachen Lu, name: Xiatian Zhu, name: Li Zhang
Published: 2023-10-19 05:15:17

High-resolution 3D object generation remains a challenging task primarily due
to the limited availability of comprehensive annotated training data. Recent
advancements have aimed to overcome this constraint by harnessing image
generative models, pretrained on extensive curated web datasets, using
knowledge transfer techniques like Score Distillation Sampling (SDS).
Efficiently addressing the requirements of high-resolution rendering often
necessitates the adoption of latent representation-based models, such as the
Latent Diffusion Model (LDM). In this framework, a significant challenge
arises: To compute gradients for individual image pixels, it is necessary to
backpropagate gradients from the designated latent space through the frozen
components of the image model, such as the VAE encoder used within LDM.
However, this gradient propagation pathway has never been optimized, remaining
uncontrolled during training. We find that the unregulated gradients adversely
affect the 3D model's capacity in acquiring texture-related information from
the image generative model, leading to poor quality appearance synthesis. To
address this overarching challenge, we propose an innovative operation termed
Pixel-wise Gradient Clipping (PGC) designed for seamless integration into
existing 3D generative models, thereby enhancing their synthesis quality.
Specifically, we control the magnitude of stochastic gradients by clipping the
pixel-wise gradients efficiently, while preserving crucial texture-related
gradient directions. Despite this simplicity and minimal extra cost, extensive
experiments demonstrate the efficacy of our PGC in enhancing the performance of
existing 3D generative models for high-resolution object rendering.
An Image is Worth Multiple Words: Learning Object Level Concepts using
  Multi-Concept Prompt Learning
Authors: name: Chen Jin, name: Ryutaro Tanno, name: Amrutha Saseendran, name: Tom Diethe, name: Philip Teare
Published: 2023-10-18 19:18:19

Textural Inversion, a prompt learning method, learns a singular embedding for
a new "word" to represent image style and appearance, allowing it to be
integrated into natural language sentences to generate novel synthesised
images. However, identifying and integrating multiple object-level concepts
within one scene poses significant challenges even when embeddings for
individual concepts are attainable. This is further confirmed by our empirical
tests. To address this challenge, we introduce a framework for Multi-Concept
Prompt Learning (MCPL), where multiple new "words" are simultaneously learned
from a single sentence-image pair. To enhance the accuracy of word-concept
correlation, we propose three regularisation techniques: Attention Masking
(AttnMask) to concentrate learning on relevant areas; Prompts Contrastive Loss
(PromptCL) to separate the embeddings of different concepts; and Bind adjective
(Bind adj.) to associate new "words" with known words. We evaluate via image
generation, editing, and attention visualisation with diverse images. Extensive
quantitative comparisons demonstrate that our method can learn more
semantically disentangled concepts with enhanced word-concept correlation.
Additionally, we introduce a novel dataset and evaluation protocol tailored for
this new task of learning object-level concepts.
AutoMix: Automatically Mixing Language Models
Authors: name: Aman Madaan, name: Pranjal Aggarwal, name: Ankit Anand, name: Srividya Pranavi Potharaju, name: Swaroop Mishra, name: Pei Zhou, name: Aditya Gupta, name: Dheeraj Rajagopal, name: Karthik Kappaganthu, name: Yiming Yang, name: Shyam Upadhyay, name: Mausam, name: Manaal Faruqui
Published: 2023-10-19 17:57:39

Large language models (LLMs) are now available in various sizes and
configurations from cloud API providers. While this diversity offers a broad
spectrum of choices, effectively leveraging the options to optimize
computational cost and performance remains challenging. In this work, we
present AutoMix, an approach that strategically routes queries to larger LMs,
based on the approximate correctness of outputs from a smaller LM. Central to
AutoMix is a few-shot self-verification mechanism, which estimates the
reliability of its own outputs without requiring training. Given that
verifications can be noisy, we employ a meta verifier in AutoMix to refine the
accuracy of these assessments. Our experiments using LLAMA2-13/70B, on five
context-grounded reasoning datasets demonstrate that AutoMix surpasses
established baselines, improving the incremental benefit per cost by up to 89%.
Our code and data are available at https://github.com/automix-llm/automix.
An Emulator for Fine-Tuning Large Language Models using Small Language
  Models
Authors: name: Eric Mitchell, name: Rafael Rafailov, name: Archit Sharma, name: Chelsea Finn, name: Christopher D. Manning
Published: 2023-10-19 17:57:16

Widely used language models (LMs) are typically built by scaling up a
two-stage training pipeline: a pre-training stage that uses a very large,
diverse dataset of text and a fine-tuning (sometimes, 'alignment') stage that
uses targeted examples or other specifications of desired behaviors. While it
has been hypothesized that knowledge and skills come from pre-training, and
fine-tuning mostly filters this knowledge and skillset, this intuition has not
been extensively tested. To aid in doing so, we introduce a novel technique for
decoupling the knowledge and skills gained in these two stages, enabling a
direct answer to the question, "What would happen if we combined the knowledge
learned by a large model during pre-training with the knowledge learned by a
small model during fine-tuning (or vice versa)?" Using an RL-based framework
derived from recent developments in learning from human preferences, we
introduce emulated fine-tuning (EFT), a principled and practical method for
sampling from a distribution that approximates (or 'emulates') the result of
pre-training and fine-tuning at different scales. Our experiments with EFT show
that scaling up fine-tuning tends to improve helpfulness, while scaling up
pre-training tends to improve factuality. Beyond decoupling scale, we show that
EFT enables test-time adjustment of competing behavioral traits like
helpfulness and harmlessness without additional training. Finally, a special
case of emulated fine-tuning, which we call LM up-scaling, avoids
resource-intensive fine-tuning of large pre-trained models by ensembling them
with small fine-tuned models, essentially emulating the result of fine-tuning
the large pre-trained model. Up-scaling consistently improves helpfulness and
factuality of instruction-following models in the Llama, Llama-2, and Falcon
families, without additional hyperparameters or training.
3D-GPT: Procedural 3D Modeling with Large Language Models
Authors: name: Chunyi Sun, name: Junlin Han, name: Weijian Deng, name: Xinlong Wang, name: Zishan Qin, name: Stephen Gould
Published: 2023-10-19 17:41:48

In the pursuit of efficient automated content creation, procedural
generation, leveraging modifiable parameters and rule-based systems, emerges as
a promising approach. Nonetheless, it could be a demanding endeavor, given its
intricate nature necessitating a deep understanding of rules, algorithms, and
parameters. To reduce workload, we introduce 3D-GPT, a framework utilizing
large language models~(LLMs) for instruction-driven 3D modeling. 3D-GPT
positions LLMs as proficient problem solvers, dissecting the procedural 3D
modeling tasks into accessible segments and appointing the apt agent for each
task. 3D-GPT integrates three core agents: the task dispatch agent, the
conceptualization agent, and the modeling agent. They collaboratively achieve
two objectives. First, it enhances concise initial scene descriptions, evolving
them into detailed forms while dynamically adapting the text based on
subsequent instructions. Second, it integrates procedural generation,
extracting parameter values from enriched text to effortlessly interface with
3D software for asset creation. Our empirical investigations confirm that
3D-GPT not only interprets and executes instructions, delivering reliable
results but also collaborates effectively with human designers. Furthermore, it
seamlessly integrates with Blender, unlocking expanded manipulation
possibilities. Our work highlights the potential of LLMs in 3D modeling,
offering a basic framework for future advancements in scene generation and
animation.