Orionnnn/hub-stats huggingface dataset:
D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning
Authors: name: Rafael Rafailov, name: Kyle Hatch, name: Anikait Singh, name: Laura Smith, name: Aviral Kumar, name: Ilya Kostrikov, name: Philippe Hansen-Estruch, name: Victor Kolev, name: Philip Ball, name: Jiajun Wu, name: Chelsea Finn, name: Sergey Levine
Published: 2024-08-15 22:27:00

Offline reinforcement learning algorithms hold the promise of enabling
data-driven RL methods that do not require costly or dangerous real-world
exploration and benefit from large pre-collected datasets. This in turn can
facilitate real-world applications, as well as a more standardized approach to
RL research. Furthermore, offline RL methods can provide effective
initializations for online finetuning to overcome challenges with exploration.
However, evaluating progress on offline RL algorithms requires effective and
challenging benchmarks that capture properties of real-world tasks, provide a
range of task difficulties, and cover a range of challenges both in terms of
the parameters of the domain (e.g., length of the horizon, sparsity of rewards)
and the parameters of the data (e.g., narrow demonstration data or broad
exploratory data). While considerable progress in offline RL in recent years
has been enabled by simpler benchmark tasks, the most widely used datasets are
increasingly saturating in performance and may fail to reflect properties of
realistic tasks. We propose a new benchmark for offline RL that focuses on
realistic simulations of robotic manipulation and locomotion environments,
based on models of real-world robotic systems, and comprising a variety of data
sources, including scripted data, play-style data collected by human
teleoperators, and other data sources. Our proposed benchmark covers
state-based and image-based domains, and supports both offline RL and online
fine-tuning evaluation, with some of the tasks specifically designed to require
both pre-training and fine-tuning. We hope that our proposed benchmark will
facilitate further progress on both offline RL and fine-tuning algorithms.
Website with code, examples, tasks, and data is available at
https://sites.google.com/view/d5rl/
Automated Design of Agentic Systems
Authors: name: Shengran Hu, name: Cong Lu, name: Jeff Clune
Published: 2024-08-15 21:59:23

Researchers are investing substantial effort in developing powerful
general-purpose agents, wherein Foundation Models are used as modules within
agentic systems (e.g. Chain-of-Thought, Self-Reflection, Toolformer). However,
the history of machine learning teaches us that hand-designed solutions are
eventually replaced by learned solutions. We formulate a new research area,
Automated Design of Agentic Systems (ADAS), which aims to automatically create
powerful agentic system designs, including inventing novel building blocks
and/or combining them in new ways. We further demonstrate that there is an
unexplored yet promising approach within ADAS where agents can be defined in
code and new agents can be automatically discovered by a meta agent programming
ever better ones in code. Given that programming languages are Turing Complete,
this approach theoretically enables the learning of any possible agentic
system: including novel prompts, tool use, control flows, and combinations
thereof. We present a simple yet effective algorithm named Meta Agent Search to
demonstrate this idea, where a meta agent iteratively programs interesting new
agents based on an ever-growing archive of previous discoveries. Through
extensive experiments across multiple domains including coding, science, and
math, we show that our algorithm can progressively invent agents with novel
designs that greatly outperform state-of-the-art hand-designed agents.
Importantly, we consistently observe the surprising result that agents invented
by Meta Agent Search maintain superior performance even when transferred across
domains and models, demonstrating their robustness and generality. Provided we
develop it safely, our work illustrates the potential of an exciting new
research direction toward automatically designing ever-more powerful agentic
systems to benefit humanity.
TurboEdit: Instant text-based image editing
Authors: name: Zongze Wu, name: Nicholas Kolkin, name: Jonathan Brandt, name: Richard Zhang, name: Eli Shechtman
Published: 2024-08-14 18:02:24

We address the challenges of precise image inversion and disentangled image
editing in the context of few-step diffusion models. We introduce an encoder
based iterative inversion technique. The inversion network is conditioned on
the input image and the reconstructed image from the previous step, allowing
for correction of the next reconstruction towards the input image. We
demonstrate that disentangled controls can be easily achieved in the few-step
diffusion model by conditioning on an (automatically generated) detailed text
prompt. To manipulate the inverted image, we freeze the noise maps and modify
one attribute in the text prompt (either manually or via instruction based
editing driven by an LLM), resulting in the generation of a new image similar
to the input image with only one attribute changed. It can further control the
editing strength and accept instructive text prompt. Our approach facilitates
realistic text-guided image edits in real-time, requiring only 8 number of
functional evaluations (NFEs) in inversion (one-time cost) and 4 NFEs per edit.
Our method is not only fast, but also significantly outperforms
state-of-the-art multi-step diffusion editing techniques.
JPEG-LM: LLMs as Image Generators with Canonical Codec Representations
Authors: name: Xiaochuang Han, name: Marjan Ghazvininejad, name: Pang Wei Koh, name: Yulia Tsvetkov
Published: 2024-08-15 23:57:02

Recent work in image and video generation has been adopting the
autoregressive LLM architecture due to its generality and potentially easy
integration into multi-modal systems. The crux of applying autoregressive
training in language generation to visual generation is discretization --
representing continuous data like images and videos as discrete tokens. Common
methods of discretizing images and videos include modeling raw pixel values,
which are prohibitively lengthy, or vector quantization, which requires
convoluted pre-hoc training. In this work, we propose to directly model images
and videos as compressed files saved on computers via canonical codecs (e.g.,
JPEG, AVC/H.264). Using the default Llama architecture without any
vision-specific modifications, we pretrain JPEG-LM from scratch to generate
images (and AVC-LM to generate videos as a proof of concept), by directly
outputting compressed file bytes in JPEG and AVC formats. Evaluation of image
generation shows that this simple and straightforward approach is more
effective than pixel-based modeling and sophisticated vector quantization
baselines (on which our method yields a 31% reduction in FID). Our analysis
shows that JPEG-LM has an especial advantage over vector quantization models in
generating long-tail visual elements. Overall, we show that using canonical
codec representations can help lower the barriers between language generation
and visual generation, facilitating future research on multi-modal
language/image/video LLMs.
The ShareLM Collection and Plugin: Contributing Human-Model Chats for
  the Benefit of the Community
Authors: name: Shachar Don-Yehiya, name: Leshem Choshen, name: Omri Abend
Published: 2024-08-15 17:46:54

Human-model conversations provide a window into users' real-world scenarios,
behavior, and needs, and thus are a valuable resource for model development and
research. While for-profit companies collect user data through the APIs of
their models, using it internally to improve their own models, the open source
and research community lags behind.
  We introduce the ShareLM collection, a unified set of human conversations
with large language models, and its accompanying plugin, a Web extension for
voluntarily contributing user-model conversations. Where few platforms share
their chats, the ShareLM plugin adds this functionality, thus, allowing users
to share conversations from most platforms. The plugin allows the user to rate
their conversations, both at the conversation and the response levels, and
delete conversations they prefer to keep private before they ever leave the
user's local storage. We release the plugin conversations as part of the
ShareLM collection, and call for more community effort in the field of open
human-model data.
  The code, plugin, and data are available.
FuseChat: Knowledge Fusion of Chat Models
Authors: name: Fanqi Wan, name: Longguang Zhong, name: Ziyi Yang, name: Ruijun Chen, name: Xiaojun Quan
Published: 2024-08-15 07:37:24

While training large language models (LLMs) from scratch can indeed lead to
models with distinct capabilities and strengths, it incurs substantial costs
and may lead to redundancy in competencies. Knowledge fusion aims to integrate
existing LLMs of diverse architectures and capabilities into a more potent LLM
through lightweight continual training, thereby reducing the need for costly
LLM development. In this work, we propose a new framework for the knowledge
fusion of chat LLMs through two main stages, resulting in FuseChat. Firstly, we
conduct pairwise knowledge fusion on source chat LLMs of varying structures and
scales to create multiple target LLMs with identical structure and size via
lightweight fine-tuning. During this process, a statistics-based token
alignment approach is introduced as the cornerstone for fusing LLMs with
different structures. Secondly, we merge these target LLMs within the parameter
space, where we propose a novel method for determining the merging coefficients
based on the magnitude of parameter updates before and after fine-tuning. We
implement and validate FuseChat using six prominent chat LLMs with diverse
architectures and scales, including OpenChat-3.5-7B, Starling-LM-7B-alpha,
NH2-SOLAR-10.7B, InternLM2-Chat-20B, Mixtral-8x7B-Instruct, and
Qwen-1.5-Chat-72B. Experimental results on two instruction-following
benchmarks, AlpacaEval 2.0 and MT-Bench, demonstrate the superiority of
FuseChat-7B over baselines of various sizes. Our model is even comparable to
the larger Mixtral-8x7B-Instruct and approaches GPT-3.5-Turbo-1106 on MT-Bench.
Our code, model weights, and data are public at
https://github.com/fanqiwan/FuseAI.
I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative
  Self-Enhancement Paradigm
Authors: name: Yiming Liang, name: Ge Zhang, name: Xingwei Qu, name: Tianyu Zheng, name: Jiawei Guo, name: Xinrun Du, name: Zhenzhu Yang, name: Jiaheng Liu, name: Chenghua Lin, name: Lei Ma, name: Wenhao Huang, name: Jiajun Zhang
Published: 2024-08-15 10:44:38

Large Language Models (LLMs) have achieved significant advancements, however,
the common learning paradigm treats LLMs as passive information repositories,
neglecting their potential for active learning and alignment. Some approaches
train LLMs using their own generated synthetic data, exploring the possibility
of active alignment. However, there is still a huge gap between these one-time
alignment methods and the continuous automatic alignment of humans. In this
paper, we introduce I-SHEEP, an Iterative
Self-EnHancEmEnt Paradigm.This
human-like paradigm enables LLMs to continuously self-align from
scratch with nothing. Compared to the one-time alignment method Dromedary
sun2023principledriven, which refers to the first iteration in this
paper, I-SHEEP can significantly enhance capacities on both Qwen and Llama
models. I-SHEEP achieves a maximum relative improvement of 78.2\% in the Alpaca
Eval, 24.0\% in the MT Bench, and an absolute increase of 8.88\% in the IFEval
accuracy over subsequent iterations in Qwen-1.5 72B model. Additionally,
I-SHEEP surpasses the base model in various standard benchmark generation
tasks, achieving an average improvement of 24.77\% in code generation tasks,
12.04\% in TrivialQA, and 20.29\% in SQuAD. We also provide new insights based
on the experiment results. Our codes, datasets, and models are available at
https://anonymous.4open.science/r/I-SHEEP.
BAM! Just Like That: Simple and Efficient Parameter Upcycling for
  Mixture of Experts
Authors: name: Qizhen Zhang, name: Nikolas Gritsch, name: Dwaraknath Gnaneshwar, name: Simon Guo, name: David Cairuz, name: Bharat Venkitesh, name: Jakob Foerster, name: Phil Blunsom, name: Sebastian Ruder, name: Ahmet Ustun, name: Acyr Locatelli
Published: 2024-08-15 17:19:12

The Mixture of Experts (MoE) framework has become a popular architecture for
large language models due to its superior performance over dense models.
However, training MoEs from scratch in a large-scale regime is prohibitively
expensive. Existing methods mitigate this by pre-training multiple dense expert
models independently and using them to initialize an MoE. This is done by using
experts' feed-forward network (FFN) to initialize the MoE's experts while
merging other parameters. However, this method limits the reuse of dense model
parameters to only the FFN layers, thereby constraining the advantages when
"upcycling" these models into MoEs. We propose BAM (Branch-Attend-Mix), a
simple yet effective method that addresses this shortcoming. BAM makes full use
of specialized dense models by not only using their FFN to initialize the MoE
layers but also leveraging experts' attention parameters fully by initializing
them into a soft-variant of Mixture of Attention (MoA) layers. We explore two
methods for upcycling attention parameters: 1) initializing separate attention
experts from dense models including all attention parameters for the best model
performance; and 2) sharing key and value parameters across all experts to
facilitate for better inference efficiency. To further improve efficiency, we
adopt a parallel attention transformer architecture to MoEs, which allows the
attention experts and FFN experts to be computed concurrently. Our experiments
on seed models ranging from 590 million to 2 billion parameters demonstrate
that BAM surpasses baselines in both perplexity and downstream task
performance, within the same computational and data constraints.
Training Language Models on the Knowledge Graph: Insights on
  Hallucinations and Their Detectability
Authors: name: Jiri Hron, name: Laura Culp, name: Gamaleldin Elsayed, name: Rosanne Liu, name: Ben Adlam, name: Maxwell Bileschi, name: Bernd Bohnet, name: JD Co-Reyes, name: Noah Fiedel, name: C. Daniel Freeman, name: Izzeddin Gur, name: Kathleen Kenealy, name: Jaehoon Lee, name: Peter J. Liu, name: Gaurav Mishra, name: Igor Mordatch, name: Azade Nova, name: Roman Novak, name: Aaron Parisi, name: Jeffrey Pennington, name: Alex Rizkowsky, name: Isabelle Simpson, name: Hanie Sedghi, name: Jascha Sohl-dickstein, name: Kevin Swersky, name: Sharad Vikram, name: Tris Warkentin, name: Lechao Xiao, name: Kelvin Xu, name: Jasper Snoek, name: Simon Kornblith
Published: 2024-08-14 23:34:28

While many capabilities of language models (LMs) improve with increased
training budget, the influence of scale on hallucinations is not yet fully
understood. Hallucinations come in many forms, and there is no universally
accepted definition. We thus focus on studying only those hallucinations where
a correct answer appears verbatim in the training set. To fully control the
training data content, we construct a knowledge graph (KG)-based dataset, and
use it to train a set of increasingly large LMs. We find that for a fixed
dataset, larger and longer-trained LMs hallucinate less. However, hallucinating
on leq5% of the training data requires an order of magnitude larger model,
and thus an order of magnitude more compute, than Hoffmann et al. (2022)
reported was optimal. Given this costliness, we study how hallucination
detectors depend on scale. While we see detector size improves performance on
fixed LM's outputs, we find an inverse relationship between the scale of the LM
and the detectability of its hallucinations.
Towards flexible perception with visual memory
Authors: name: Robert Geirhos, name: Priyank Jaini, name: Austin Stone, name: Sourabh Medapati, name: Xi Yi, name: George Toderici, name: Abhijit Ogale, name: Jonathon Shlens
Published: 2024-08-15 14:19:13

Training a neural network is a monolithic endeavor, akin to carving knowledge
into stone: once the process is completed, editing the knowledge in a network
is nearly impossible, since all information is distributed across the network's
weights. We here explore a simple, compelling alternative by marrying the
representational power of deep neural networks with the flexibility of a
database. Decomposing the task of image classification into image similarity
(from a pre-trained embedding) and search (via fast nearest neighbor retrieval
from a knowledge database), we build a simple and flexible visual memory that
has the following key capabilities: (1.) The ability to flexibly add data
across scales: from individual samples all the way to entire classes and
billion-scale data; (2.) The ability to remove data through unlearning and
memory pruning; (3.) An interpretable decision-mechanism on which we can
intervene to control its behavior. Taken together, these capabilities
comprehensively demonstrate the benefits of an explicit visual memory. We hope
that it might contribute to a conversation on how knowledge should be
represented in deep vision models -- beyond carving it in ``stone'' weights.
Accelerating High-Fidelity Waveform Generation via Adversarial Flow
  Matching Optimization
Authors: name: Sang-Hoon Lee, name: Ha-Yeong Choi, name: Seong-Whan Lee
Published: 2024-08-15 08:34:00

This paper introduces PeriodWave-Turbo, a high-fidelity and high-efficient
waveform generation model via adversarial flow matching optimization. Recently,
conditional flow matching (CFM) generative models have been successfully
adopted for waveform generation tasks, leveraging a single vector field
estimation objective for training. Although these models can generate
high-fidelity waveform signals, they require significantly more ODE steps
compared to GAN-based models, which only need a single generation step.
Additionally, the generated samples often lack high-frequency information due
to noisy vector field estimation, which fails to ensure high-frequency
reproduction. To address this limitation, we enhance pre-trained CFM-based
generative models by incorporating a fixed-step generator modification. We
utilized reconstruction losses and adversarial feedback to accelerate
high-fidelity waveform generation. Through adversarial flow matching
optimization, it only requires 1,000 steps of fine-tuning to achieve
state-of-the-art performance across various objective metrics. Moreover, we
significantly reduce inference speed from 16 steps to 2 or 4 steps.
Additionally, by scaling up the backbone of PeriodWave from 29M to 70M
parameters for improved generalization, PeriodWave-Turbo achieves unprecedented
performance, with a perceptual evaluation of speech quality (PESQ) score of
4.454 on the LibriTTS dataset. Audio samples, source code and checkpoints will
be available at https://github.com/sh-lee-prml/PeriodWave.
Heavy Labels Out! Dataset Distillation with Label Space Lightening
Authors: name: Ruonan Yu, name: Songhua Liu, name: Zigeng Chen, name: Jingwen Ye, name: Xinchao Wang
Published: 2024-08-15 15:08:58

Dataset distillation or condensation aims to condense a large-scale training
dataset into a much smaller synthetic one such that the training performance of
distilled and original sets on neural networks are similar. Although the number
of training samples can be reduced substantially, current state-of-the-art
methods heavily rely on enormous soft labels to achieve satisfactory
performance. As a result, the required storage can be comparable even to
original datasets, especially for large-scale ones. To solve this problem,
instead of storing these heavy labels, we propose a novel label-lightening
framework termed HeLlO aiming at effective image-to-label projectors, with
which synthetic labels can be directly generated online from synthetic images.
Specifically, to construct such projectors, we leverage prior knowledge in
open-source foundation models, e.g., CLIP, and introduce a LoRA-like
fine-tuning strategy to mitigate the gap between pre-trained and target
distributions, so that original models for soft-label generation can be
distilled into a group of low-rank matrices. Moreover, an effective image
optimization method is proposed to further mitigate the potential error between
the original and distilled label generators. Extensive experiments demonstrate
that with only about 0.003% of the original storage required for a complete set
of soft labels, we achieve comparable performance to current state-of-the-art
dataset distillation methods on large-scale datasets. Our code will be
available.
MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and
  3D Editing
Authors: name: Chenjie Cao, name: Chaohui Yu, name: Yanwei Fu, name: Fan Wang, name: Xiangyang Xue
Published: 2024-08-15 07:57:28

Novel View Synthesis (NVS) and 3D generation have recently achieved prominent
improvements. However, these works mainly focus on confined categories or
synthetic 3D assets, which are discouraged from generalizing to challenging
in-the-wild scenes and fail to be employed with 2D synthesis directly.
Moreover, these methods heavily depended on camera poses, limiting their
real-world applications. To overcome these issues, we propose MVInpainter,
re-formulating the 3D editing as a multi-view 2D inpainting task. Specifically,
MVInpainter partially inpaints multi-view images with the reference guidance
rather than intractably generating an entirely novel view from scratch, which
largely simplifies the difficulty of in-the-wild NVS and leverages unmasked
clues instead of explicit pose conditions. To ensure cross-view consistency,
MVInpainter is enhanced by video priors from motion components and appearance
guidance from concatenated reference key&value attention. Furthermore,
MVInpainter incorporates slot attention to aggregate high-level optical flow
features from unmasked regions to control the camera movement with pose-free
training and inference. Sufficient scene-level experiments on both
object-centric and forward-facing datasets verify the effectiveness of
MVInpainter, including diverse tasks, such as multi-view object removal,
synthesis, insertion, and replacement. The project page is
https://ewrfcas.github.io/MVInpainter/.
DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for
  Reinforcement Learning and Monte-Carlo Tree Search
Authors: name: Huajian Xin, name: Z. Z. Ren, name: Junxiao Song, name: Zhihong Shao, name: Wanjia Zhao, name: Haocheng Wang, name: Bo Liu, name: Liyue Zhang, name: Xuan Lu, name: Qiushi Du, name: Wenjun Gao, name: Qihao Zhu, name: Dejian Yang, name: Zhibin Gou, name: Z. F. Wu, name: Fuli Luo, name: Chong Ruan
Published: 2024-08-15 13:40:03

We introduce DeepSeek-Prover-V1.5, an open-source language model designed for
theorem proving in Lean 4, which enhances DeepSeek-Prover-V1 by optimizing both
training and inference processes. Pre-trained on DeepSeekMath-Base with
specialization in formal mathematical languages, the model undergoes supervised
fine-tuning using an enhanced formal theorem proving dataset derived from
DeepSeek-Prover-V1. Further refinement is achieved through reinforcement
learning from proof assistant feedback (RLPAF). Beyond the single-pass
whole-proof generation approach of DeepSeek-Prover-V1, we propose RMaxTS, a
variant of Monte-Carlo tree search that employs an intrinsic-reward-driven
exploration strategy to generate diverse proof paths. DeepSeek-Prover-V1.5
demonstrates significant improvements over DeepSeek-Prover-V1, achieving new
state-of-the-art results on the test set of the high school level miniF2F
benchmark (63.5%) and the undergraduate level ProofNet benchmark (25.3%).