Orionnnn/hub-stats huggingface dataset:
Safe RLHF: Safe Reinforcement Learning from Human Feedback
Authors: name: Josef Dai, name: Xuehai Pan, name: Ruiyang Sun, name: Jiaming Ji, name: Xinbo Xu, name: Mickel Liu, name: Yizhou Wang, name: Yaodong Yang
Published: 2023-10-19 14:22:03

With the development of large language models (LLMs), striking a balance
between the performance and safety of AI systems has never been more critical.
However, the inherent tension between the objectives of helpfulness and
harmlessness presents a significant challenge during LLM training. To address
this issue, we propose Safe Reinforcement Learning from Human Feedback (Safe
RLHF), a novel algorithm for human value alignment. Safe RLHF explicitly
decouples human preferences regarding helpfulness and harmlessness, effectively
avoiding the crowdworkers' confusion about the tension and allowing us to train
separate reward and cost models. We formalize the safety concern of LLMs as an
optimization task of maximizing the reward function while satisfying specified
cost constraints. Leveraging the Lagrangian method to solve this constrained
problem, Safe RLHF dynamically adjusts the balance between the two objectives
during fine-tuning. Through a three-round fine-tuning using Safe RLHF, we
demonstrate a superior ability to mitigate harmful responses while enhancing
model performance compared to existing value-aligned algorithms.
Experimentally, we fine-tuned the Alpaca-7B using Safe RLHF and aligned it with
collected human preferences, significantly improving its helpfulness and
harmlessness according to human evaluations.
Loop Copilot: Conducting AI Ensembles for Music Generation and Iterative
  Editing
Authors: name: Yixiao Zhang, name: Akira Maezawa, name: Gus Xia, name: Kazuhiko Yamamoto, name: Simon Dixon
Published: 2023-10-19 01:20:12

Creating music is iterative, requiring varied methods at each stage. However,
existing AI music systems fall short in orchestrating multiple subsystems for
diverse needs. To address this gap, we introduce Loop Copilot, a novel system
that enables users to generate and iteratively refine music through an
interactive, multi-round dialogue interface. The system uses a large language
model to interpret user intentions and select appropriate AI models for task
execution. Each backend model is specialized for a specific task, and their
outputs are aggregated to meet the user's requirements. To ensure musical
coherence, essential attributes are maintained in a centralized table. We
evaluate the effectiveness of the proposed system through semi-structured
interviews and questionnaires, highlighting its utility not only in
facilitating music creation but also its potential for broader applications.
Eureka: Human-Level Reward Design via Coding Large Language Models
Authors: name: Yecheng Jason Ma, name: William Liang, name: Guanzhi Wang, name: De-An Huang, name: Osbert Bastani, name: Dinesh Jayaraman, name: Yuke Zhu, name: Linxi Fan, name: Anima Anandkumar
Published: 2023-10-19 17:31:01

Large Language Models (LLMs) have excelled as high-level semantic planners
for sequential decision-making tasks. However, harnessing them to learn complex
low-level manipulation tasks, such as dexterous pen spinning, remains an open
problem. We bridge this fundamental gap and present Eureka, a human-level
reward design algorithm powered by LLMs. Eureka exploits the remarkable
zero-shot generation, code-writing, and in-context improvement capabilities of
state-of-the-art LLMs, such as GPT-4, to perform evolutionary optimization over
reward code. The resulting rewards can then be used to acquire complex skills
via reinforcement learning. Without any task-specific prompting or pre-defined
reward templates, Eureka generates reward functions that outperform expert
human-engineered rewards. In a diverse suite of 29 open-source RL environments
that include 10 distinct robot morphologies, Eureka outperforms human experts
on 83% of the tasks, leading to an average normalized improvement of 52%. The
generality of Eureka also enables a new gradient-free in-context learning
approach to reinforcement learning from human feedback (RLHF), readily
incorporating human inputs to improve the quality and the safety of the
generated rewards without model updating. Finally, using Eureka rewards in a
curriculum learning setting, we demonstrate for the first time, a simulated
Shadow Hand capable of performing pen spinning tricks, adeptly manipulating a
pen in circles at rapid speed.
AgentTuning: Enabling Generalized Agent Abilities for LLMs
Authors: name: Aohan Zeng, name: Mingdao Liu, name: Rui Lu, name: Bowen Wang, name: Xiao Liu, name: Yuxiao Dong, name: Jie Tang
Published: 2023-10-19 15:19:53

Open large language models (LLMs) with great performance in various tasks
have significantly advanced the development of LLMs. However, they are far
inferior to commercial models such as ChatGPT and GPT-4 when acting as agents
to tackle complex tasks in the real world. These agent tasks employ LLMs as the
central controller responsible for planning, memorization, and tool
utilization, necessitating both fine-grained prompting methods and robust LLMs
to achieve satisfactory performance. Though many prompting methods have been
proposed to complete particular agent tasks, there is lack of research focusing
on improving the agent capabilities of LLMs themselves without compromising
their general abilities. In this work, we present AgentTuning, a simple and
general method to enhance the agent abilities of LLMs while maintaining their
general LLM capabilities. We construct AgentInstruct, a lightweight
instruction-tuning dataset containing high-quality interaction trajectories. We
employ a hybrid instruction-tuning strategy by combining AgentInstruct with
open-source instructions from general domains. AgentTuning is used to
instruction-tune the Llama 2 series, resulting in AgentLM. Our evaluations show
that AgentTuning enables LLMs' agent capabilities without compromising general
abilities. The AgentLM-70B is comparable to GPT-3.5-turbo on unseen agent
tasks, demonstrating generalized agent capabilities. We open source the
AgentInstruct and AgentLM-7B, 13B, and 70B models at
https://github.com/THUDM/AgentTuning , serving open and powerful alternatives
to commercial LLMs for agent tasks.
Progressive3D: Progressively Local Editing for Text-to-3D Content
  Creation with Complex Semantic Prompts
Authors: name: Xinhua Cheng, name: Tianyu Yang, name: Jianan Wang, name: Yu Li, name: Lei Zhang, name: Jian Zhang, name: Li Yuan
Published: 2023-10-18 08:23:14

Recent text-to-3D generation methods achieve impressive 3D content creation
capacity thanks to the advances in image diffusion models and optimizing
strategies. However, current methods struggle to generate correct 3D content
for a complex prompt in semantics, i.e., a prompt describing multiple
interacted objects binding with different attributes. In this work, we propose
a general framework named Progressive3D, which decomposes the entire generation
into a series of locally progressive editing steps to create precise 3D content
for complex prompts, and we constrain the content change to only occur in
regions determined by user-defined region prompts in each editing step.
Furthermore, we propose an overlapped semantic component suppression technique
to encourage the optimization process to focus more on the semantic differences
between prompts. Extensive experiments demonstrate that the proposed
Progressive3D framework generates precise 3D content for prompts with complex
semantics and is general for various text-to-3D methods driven by different 3D
representations.
MusicAgent: An AI Agent for Music Understanding and Generation with
  Large Language Models
Authors: name: Dingyao Yu, name: Kaitao Song, name: Peiling Lu, name: Tianyu He, name: Xu Tan, name: Wei Ye, name: Shikun Zhang, name: Jiang Bian
Published: 2023-10-18 13:31:10

AI-empowered music processing is a diverse field that encompasses dozens of
tasks, ranging from generation tasks (e.g., timbre synthesis) to comprehension
tasks (e.g., music classification). For developers and amateurs, it is very
difficult to grasp all of these task to satisfy their requirements in music
processing, especially considering the huge differences in the representations
of music data and the model applicability across platforms among various tasks.
Consequently, it is necessary to build a system to organize and integrate these
tasks, and thus help practitioners to automatically analyze their demand and
call suitable tools as solutions to fulfill their requirements. Inspired by the
recent success of large language models (LLMs) in task automation, we develop a
system, named MusicAgent, which integrates numerous music-related tools and an
autonomous workflow to address user requirements. More specifically, we build
1) toolset that collects tools from diverse sources, including Hugging Face,
GitHub, and Web API, etc. 2) an autonomous workflow empowered by LLMs (e.g.,
ChatGPT) to organize these tools and automatically decompose user requests into
multiple sub-tasks and invoke corresponding music tools. The primary goal of
this system is to free users from the intricacies of AI-music tools, enabling
them to concentrate on the creative aspect. By granting users the freedom to
effortlessly combine tools, the system offers a seamless and enriching music
experience.
Self-RAG: Learning to Retrieve, Generate, and Critique through
  Self-Reflection
Authors: name: Akari Asai, name: Zeqiu Wu, name: Yizhong Wang, name: Avirup Sil, name: Hannaneh Hajishirzi
Published: 2023-10-17 18:18:32

Despite their remarkable capabilities, large language models (LLMs) often
produce responses containing factual inaccuracies due to their sole reliance on
the parametric knowledge they encapsulate. Retrieval-Augmented Generation
(RAG), an ad hoc approach that augments LMs with retrieval of relevant
knowledge, decreases such issues. However, indiscriminately retrieving and
incorporating a fixed number of retrieved passages, regardless of whether
retrieval is necessary, or passages are relevant, diminishes LM versatility or
can lead to unhelpful response generation. We introduce a new framework called
Self-Reflective Retrieval-Augmented Generation (Self-RAG) that enhances an LM's
quality and factuality through retrieval and self-reflection. Our framework
trains a single arbitrary LM that adaptively retrieves passages on-demand, and
generates and reflects on retrieved passages and its own generations using
special tokens, called reflection tokens. Generating reflection tokens makes
the LM controllable during the inference phase, enabling it to tailor its
behavior to diverse task requirements. Experiments show that Self-RAG (7B and
13B parameters) significantly outperforms state-of-the-art LLMs and
retrieval-augmented models on a diverse set of tasks. Specifically, Self-RAG
outperforms ChatGPT and retrieval-augmented Llama2-chat on Open-domain QA,
reasoning and fact verification tasks, and it shows significant gains in
improving factuality and citation accuracy for long-form generations relative
to these models.
VeRA: Vector-based Random Matrix Adaptation
Authors: name: Dawid Jan Kopiczko, name: Tijmen Blankevoort, name: Yuki Markus Asano
Published: 2023-10-17 17:59:46

Low-rank adapation (LoRA) is a popular method that reduces the number of
trainable parameters when finetuning large language models, but still faces
acute storage challenges when scaling to even larger models or deploying
numerous per-user or per-task adapted models. In this work, we present
Vector-based Random Matrix Adaptation (VeRA), which reduces the number of
trainable parameters by 10x compared to LoRA, yet maintains the same
performance. It achieves this by using a single pair of low-rank matrices
shared across all layers and learning small scaling vectors instead. We
demonstrate its effectiveness on the GLUE and E2E benchmarks, and show its
application in instruction-following with just 1.4M parameters using the Llama2
7B model.
4K4D: Real-Time 4D View Synthesis at 4K Resolution
Authors: name: Zhen Xu, name: Sida Peng, name: Haotong Lin, name: Guangzhao He, name: Jiaming Sun, name: Yujun Shen, name: Hujun Bao, name: Xiaowei Zhou
Published: 2023-10-17 17:57:38

This paper targets high-fidelity and real-time view synthesis of dynamic 3D
scenes at 4K resolution. Recently, some methods on dynamic view synthesis have
shown impressive rendering quality. However, their speed is still limited when
rendering high-resolution images. To overcome this problem, we propose 4K4D, a
4D point cloud representation that supports hardware rasterization and enables
unprecedented rendering speed. Our representation is built on a 4D feature grid
so that the points are naturally regularized and can be robustly optimized. In
addition, we design a novel hybrid appearance model that significantly boosts
the rendering quality while preserving efficiency. Moreover, we develop a
differentiable depth peeling algorithm to effectively learn the proposed model
from RGB videos. Experiments show that our representation can be rendered at
over 400 FPS on the DNA-Rendering dataset at 1080p resolution and 80 FPS on the
ENeRF-Outdoor dataset at 4K resolution using an RTX 4090 GPU, which is 30x
faster than previous methods and achieves the state-of-the-art rendering
quality. We will release the code for reproducibility.
LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation
Authors: name: Ruiqi Wu, name: Liangyu Chen, name: Tong Yang, name: Chunle Guo, name: Chongyi Li, name: Xiangyu Zhang
Published: 2023-10-16 19:03:19

With the impressive progress in diffusion-based text-to-image generation, extending such powerful generative ability to text-to-video raises enormous attention. Existing methods either require large-scale text-video pairs and a large number of training resources or learn motions that are precisely aligned with template videos. It is non-trivial to balance a trade-off between the degree of generation freedom and the resource costs for video generation. In our study, we present a few-shot-based tuning framework, LAMP, which enables text-to-image diffusion model Learn A specific Motion Pattern with 8~16 videos on a single GPU. Specifically, we design a first-frame-conditioned pipeline that uses an off-the-shelf text-to-image model for content generation so that our tuned video diffusion model mainly focuses on motion learning. The well-developed text-to-image techniques can provide visually pleasing and diverse content as generation conditions, which highly improves video quality and generation freedom. To capture the features of temporal dimension, we expand the pretrained 2D convolution layers of the T2I model to our novel temporal-spatial motion learning layers and modify the attention blocks to the temporal level. Additionally, we develop an effective inference trick, shared-noise sampling, which can improve the stability of videos with computational costs. Our method can also be flexibly applied to other tasks, e.g. real-world image animation and video editing. Extensive experiments demonstrate that LAMP can effectively learn the motion pattern on limited data and generate high-quality videos. The code and models are available at https://rq-wu.github.io/projects/LAMP.
BitNet: Scaling 1-bit Transformers for Large Language Models
Authors: name: Hongyu Wang, name: Shuming Ma, name: Li Dong, name: Shaohan Huang, name: Huaijie Wang, name: Lingxiao Ma, name: Fan Yang, name: Ruiping Wang, name: Yi Wu, name: Furu Wei
Published: 2023-10-17 17:59:15

The increasing size of large language models has posed challenges for
deployment and raised concerns about environmental impact due to high energy
consumption. In this work, we introduce BitNet, a scalable and stable 1-bit
Transformer architecture designed for large language models. Specifically, we
introduce BitLinear as a drop-in replacement of the nn.Linear layer in order to
train 1-bit weights from scratch. Experimental results on language modeling
show that BitNet achieves competitive performance while substantially reducing
memory footprint and energy consumption, compared to state-of-the-art 8-bit
quantization methods and FP16 Transformer baselines. Furthermore, BitNet
exhibits a scaling law akin to full-precision Transformers, suggesting its
potential for effective scaling to even larger language models while
maintaining efficiency and performance benefits.
EvalCrafter: Benchmarking and Evaluating Large Video Generation Models
Authors: name: Yaofang Liu, name: Xiaodong Cun, name: Xuebo Liu, name: Xintao Wang, name: Yong Zhang, name: Haoxin Chen, name: Yang Liu, name: Tieyong Zeng, name: Raymond Chan, name: Ying Shan
Published: 2023-10-17 17:50:46

The vision and language generative models have been overgrown in recent
years. For video generation, various open-sourced models and public-available
services are released for generating high-visual quality videos. However, these
methods often use a few academic metrics, for example, FVD or IS, to evaluate
the performance. We argue that it is hard to judge the large conditional
generative models from the simple metrics since these models are often trained
on very large datasets with multi-aspect abilities. Thus, we propose a new
framework and pipeline to exhaustively evaluate the performance of the
generated videos. To achieve this, we first conduct a new prompt list for
text-to-video generation by analyzing the real-world prompt list with the help
of the large language model. Then, we evaluate the state-of-the-art video
generative models on our carefully designed benchmarks, in terms of visual
qualities, content qualities, motion qualities, and text-caption alignment with
around 18 objective metrics. To obtain the final leaderboard of the models, we
also fit a series of coefficients to align the objective metrics to the users'
opinions. Based on the proposed opinion alignment method, our final score shows
a higher correlation than simply averaging the metrics, showing the
effectiveness of the proposed evaluation method.
CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code
  Completion
Authors: name: Yangruibo Ding, name: Zijian Wang, name: Wasi Uddin Ahmad, name: Hantian Ding, name: Ming Tan, name: Nihal Jain, name: Murali Krishna Ramanathan, name: Ramesh Nallapati, name: Parminder Bhatia, name: Dan Roth, name: Bing Xiang
Published: 2023-10-17 13:18:01

Code completion models have made significant progress in recent years, yet
current popular evaluation datasets, such as HumanEval and MBPP, predominantly
focus on code completion tasks within a single file. This over-simplified
setting falls short of representing the real-world software development
scenario where repositories span multiple files with numerous cross-file
dependencies, and accessing and understanding cross-file context is often
required to complete the code correctly.
  To fill in this gap, we propose CrossCodeEval, a diverse and multilingual
code completion benchmark that necessitates an in-depth cross-file contextual
understanding to complete the code accurately. CrossCodeEval is built on a
diverse set of real-world, open-sourced, permissively-licensed repositories in
four popular programming languages: Python, Java, TypeScript, and C#. To create
examples that strictly require cross-file context for accurate completion, we
propose a straightforward yet efficient static-analysis-based approach to
pinpoint the use of cross-file context within the current file.
  Extensive experiments on state-of-the-art code language models like CodeGen
and StarCoder demonstrate that CrossCodeEval is extremely challenging when the
relevant cross-file context is absent, and we see clear improvements when
adding these context into the prompt. However, despite such improvements, the
pinnacle of performance remains notably unattained even with the
highest-performing model, indicating that CrossCodeEval is also capable of
assessing model's capability in leveraging extensive context to make better
code completion. Finally, we benchmarked various methods in retrieving
cross-file context, and show that CrossCodeEval can also be used to measure the
capability of code retrievers.
TEQ: Trainable Equivalent Transformation for Quantization of LLMs
Authors: name: Wenhua Cheng, name: Yiyang Cai, name: Kaokao Lv, name: Haihao Shen
Published: 2023-10-17 02:42:34

As large language models (LLMs) become more prevalent, there is a growing
need for new and improved quantization methods that can meet the
computationalast layer demands of these modern architectures while maintaining
the accuracy. In this paper, we present TEQ, a trainable equivalent
transformation that preserves the FP32 precision of the model output while
taking advantage of low-precision quantization, especially 3 and 4 bits
weight-only quantization. The training process is lightweight, requiring only
1K steps and fewer than 0.1 percent of the original model's trainable
parameters. Furthermore, the transformation does not add any computational
overhead during inference. Our results are on-par with the state-of-the-art
(SOTA) methods on typical LLMs. Our approach can be combined with other methods
to achieve even better performance. The code is available at
https://github.com/intel/neural-compressor.
Approximating Two-Layer Feedforward Networks for Efficient Transformers
Authors: name: Róbert Csordás, name: Kazuki Irie, name: Jürgen Schmidhuber
Published: 2023-10-16 21:23:16

How to reduce compute and memory requirements of neural networks (NNs)
without sacrificing performance? Many recent works use sparse Mixtures of
Experts (MoEs) to build resource-efficient large language models (LMs). Here we
introduce several novel perspectives on MoEs, presenting a general framework
that unifies various methods to approximate two-layer NNs (e.g., feedforward
blocks of Transformers), including product-key memories (PKMs). Leveraging
insights from this framework, we propose methods to improve both MoEs and PKMs.
Unlike prior work that compares MoEs with dense baselines under the
compute-equal condition, our evaluation condition is parameter-equal, which is
crucial to properly evaluate LMs. We show that our MoEs are competitive with
the dense Transformer-XL on both the WikiText-103 and enwiki8 datasets at two
different scales, while being much more resource efficient. This demonstrates
that MoEs are relevant not only to extremely large LMs but also to any-scale
resource-efficient LMs. Our code is public.
Context-Aware Meta-Learning
Authors: name: Christopher Fifty, name: Dennis Duan, name: Ronald G. Junkins, name: Ehsan Amid, name: Jure Leskovec, name: Christopher Ré, name: Sebastian Thrun
Published: 2023-10-17 03:35:27

Large Language Models like ChatGPT demonstrate a remarkable capacity to learn
new concepts during inference without any fine-tuning. However, visual models
trained to detect new objects during inference have been unable to replicate
this ability, and instead either perform poorly or require meta-training and/or
fine-tuning on similar objects. In this work, we propose a meta-learning
algorithm that emulates Large Language Models by learning new visual concepts
during inference without fine-tuning. Our approach leverages a frozen
pre-trained feature extractor, and analogous to in-context learning, recasts
meta-learning as sequence modeling over datapoints with known labels and a test
datapoint with an unknown label. On 8 out of 11 meta-learning benchmarks, our
approach -- without meta-training or fine-tuning -- exceeds or matches the
state-of-the-art algorithm, P>M>F, which is meta-trained on these benchmarks.