Orionnnn/hub-stats huggingface dataset:
Self-Play Preference Optimization for Language Model Alignment
Authors: name: Yue Wu, name: Zhiqing Sun, name: Huizhuo Yuan, name: Kaixuan Ji, name: Yiming Yang, name: Quanquan Gu
Published: 2024-05-01 17:59:20

Traditional reinforcement learning from human feedback (RLHF) approaches
relying on parametric models like the Bradley-Terry model fall short in
capturing the intransitivity and irrationality in human preferences. Recent
advancements suggest that directly working with preference probabilities can
yield a more accurate reflection of human preferences, enabling more flexible
and accurate language model alignment. In this paper, we propose a
self-play-based method for language model alignment, which treats the problem
as a constant-sum two-player game aimed at identifying the Nash equilibrium
policy. Our approach, dubbed Self-Play Preference Optimization (SPPO),
approximates the Nash equilibrium through iterative policy updates and enjoys
theoretical convergence guarantee. Our method can effectively increase the
log-likelihood of the chosen response and decrease that of the rejected
response, which cannot be trivially achieved by symmetric pairwise loss such as
Direct Preference Optimization (DPO) and Identity Preference Optimization
(IPO). In our experiments, using only 60k prompts (without responses) from the
UltraFeedback dataset and without any prompt augmentation, by leveraging a
pre-trained preference model PairRM with only 0.4B parameters, SPPO can obtain
a model from fine-tuning Mistral-7B-Instruct-v0.2 that achieves the
state-of-the-art length-controlled win-rate of 28.53% against GPT-4-Turbo on
AlpacaEval 2.0. It also outperforms the (iterative) DPO and IPO on MT-Bench and
the Open LLM Leaderboard. Notably, the strong performance of SPPO is achieved
without additional external supervision (e.g., responses, preferences, etc.)
from GPT-4 or other stronger language models.
Clover: Regressive Lightweight Speculative Decoding with Sequential
  Knowledge
Authors: name: Bin Xiao, name: Chunan Shi, name: Xiaonan Nie, name: Fan Yang, name: Xiangwei Deng, name: Lei Su, name: Weipeng Chen, name: Bin Cui
Published: 2024-05-01 00:46:22

Large language models (LLMs) suffer from low efficiency as the mismatch
between the requirement of auto-regressive decoding and the design of most
contemporary GPUs. Specifically, billions to trillions of parameters must be
loaded to the GPU cache through its limited memory bandwidth for computation,
but only a small batch of tokens is actually computed. Consequently, the GPU
spends most of its time on memory transfer instead of computation. Recently,
parallel decoding, a type of speculative decoding algorithms, is becoming more
popular and has demonstrated impressive efficiency improvement in generation.
It introduces extra decoding heads to large models, enabling them to predict
multiple subsequent tokens simultaneously and verify these candidate
continuations in a single decoding step. However, this approach deviates from
the training objective of next token prediction used during pre-training,
resulting in a low hit rate for candidate tokens. In this paper, we propose a
new speculative decoding algorithm, Clover, which integrates sequential
knowledge into the parallel decoding process. This enhancement improves the hit
rate of speculators and thus boosts the overall efficiency. Clover transmits
the sequential knowledge from pre-speculated tokens via the Regressive
Connection, then employs an Attention Decoder to integrate these speculated
tokens. Additionally, Clover incorporates an Augmenting Block that modifies the
hidden states to better align with the purpose of speculative generation rather
than next token prediction. The experiment results demonstrate that Clover
outperforms the baseline by up to 91% on Baichuan-Small and 146% on
Baichuan-Large, respectively, and exceeds the performance of the previously
top-performing method, Medusa, by up to 37% on Baichuan-Small and 57% on
Baichuan-Large, respectively.
SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General
  Sound
Authors: name: Haohe Liu, name: Xuenan Xu, name: Yi Yuan, name: Mengyue Wu, name: Wenwu Wang, name: Mark D. Plumbley
Published: 2024-04-30 22:51:36

Large language models (LLMs) have significantly advanced audio processing
through audio codecs that convert audio into discrete tokens, enabling the
application of language modelling techniques to audio data. However,
traditional codecs often operate at high bitrates or within narrow domains such
as speech and lack the semantic clues required for efficient language
modelling. Addressing these challenges, we introduce SemantiCodec, a novel
codec designed to compress audio into fewer than a hundred tokens per second
across diverse audio types, including speech, general audio, and music, without
compromising quality. SemantiCodec features a dual-encoder architecture: a
semantic encoder using a self-supervised AudioMAE, discretized using k-means
clustering on extensive audio data, and an acoustic encoder to capture the
remaining details. The semantic and acoustic encoder outputs are used to
reconstruct audio via a diffusion-model-based decoder. SemantiCodec is
presented in three variants with token rates of 25, 50, and 100 per second,
supporting a range of ultra-low bit rates between 0.31 kbps and 1.43 kbps.
Experimental results demonstrate that SemantiCodec significantly outperforms
the state-of-the-art Descript codec on reconstruction quality. Our results also
suggest that SemantiCodec contains significantly richer semantic information
than all evaluated audio codecs, even at significantly lower bitrates. Our code
and demos are available at https://haoheliu.github.io/SemantiCodec/.
Is Bigger Edit Batch Size Always Better? -- An Empirical Study on Model
  Editing with Llama-3
Authors: name: Junsang Yoon, name: Akshat Gupta, name: Gopala Anumanchipalli
Published: 2024-05-01 17:50:37

This study presents a targeted model editing analysis focused on the latest
large language model, Llama-3. We explore the efficacy of popular model editing
techniques - ROME, MEMIT, and EMMET, which are designed for precise layer
interventions. We identify the most effective layers for targeted edits through
an evaluation that encompasses up to 4096 edits across three distinct
strategies: sequential editing, batch editing, and a hybrid approach we call as
sequential-batch editing. Our findings indicate that increasing edit
batch-sizes may degrade model performance more significantly than using smaller
edit batches sequentially for equal number of edits. With this, we argue that
sequential model editing is an important component for scaling model editing
methods and future research should focus on methods that combine both batched
and sequential editing. This observation suggests a potential limitation in
current model editing methods which push towards bigger edit batch sizes, and
we hope it paves way for future investigations into optimizing batch sizes and
model editing performance.
A Careful Examination of Large Language Model Performance on Grade
  School Arithmetic
Authors: name: Hugh Zhang, name: Jeff Da, name: Dean Lee, name: Vaughn Robinson, name: Catherine Wu, name: Will Song, name: Tiffany Zhao, name: Pranav Raja, name: Dylan Slack, name: Qin Lyu, name: Sean Hendryx, name: Russell Kaplan, name: Michele, name: Lunati, name: Summer Yue
Published: 2024-05-01 05:52:05

Large language models (LLMs) have achieved impressive success on many
benchmarks for mathematical reasoning. However, there is growing concern that
some of this performance actually reflects dataset contamination, where data
closely resembling benchmark questions leaks into the training data, instead of
true reasoning ability. To investigate this claim rigorously, we commission
Grade School Math 1000 (GSM1k). GSM1k is designed to mirror the style and
complexity of the established GSM8k benchmark, the gold standard for measuring
elementary mathematical reasoning. We ensure that the two benchmarks are
comparable across important metrics such as human solve rates, number of steps
in solution, answer magnitude, and more. When evaluating leading open- and
closed-source LLMs on GSM1k, we observe accuracy drops of up to 13%, with
several families of models (e.g., Phi and Mistral) showing evidence of
systematic overfitting across almost all model sizes. At the same time, many
models, especially those on the frontier, (e.g., Gemini/GPT/Claude) show
minimal signs of overfitting. Further analysis suggests a positive relationship
(Spearman's r^2=0.32) between a model's probability of generating an example
from GSM8k and its performance gap between GSM8k and GSM1k, suggesting that
many models may have partially memorized GSM8k.
Spectrally Pruned Gaussian Fields with Neural Compensation
Authors: name: Runyi Yang, name: Zhenxin Zhu, name: Zhou Jiang, name: Baijun Ye, name: Xiaoxue Chen, name: Yifei Zhang, name: Yuantao Chen, name: Jian Zhao, name: Hao Zhao
Published: 2024-05-01 17:59:45

Recently, 3D Gaussian Splatting, as a novel 3D representation, has garnered
attention for its fast rendering speed and high rendering quality. However,
this comes with high memory consumption, e.g., a well-trained Gaussian field
may utilize three million Gaussian primitives and over 700 MB of memory. We
credit this high memory footprint to the lack of consideration for the
relationship between primitives. In this paper, we propose a memory-efficient
Gaussian field named SUNDAE with spectral pruning and neural compensation. On
one hand, we construct a graph on the set of Gaussian primitives to model their
relationship and design a spectral down-sampling module to prune out primitives
while preserving desired signals. On the other hand, to compensate for the
quality loss of pruning Gaussians, we exploit a lightweight neural network head
to mix splatted features, which effectively compensates for quality losses
while capturing the relationship between primitives in its weights. We
demonstrate the performance of SUNDAE with extensive results. For example,
SUNDAE can achieve 26.80 PSNR at 145 FPS using 104 MB memory while the vanilla
Gaussian splatting algorithm achieves 25.60 PSNR at 160 FPS using 523 MB
memory, on the Mip-NeRF360 dataset. Codes are publicly available at
https://runyiyang.github.io/projects/SUNDAE/.
Lightplane: Highly-Scalable Components for Neural 3D Fields
Authors: name: Ang Cao, name: Justin Johnson, name: Andrea Vedaldi, name: David Novotny
Published: 2024-04-30 17:59:51

Contemporary 3D research, particularly in reconstruction and generation,
heavily relies on 2D images for inputs or supervision. However, current designs
for these 2D-3D mapping are memory-intensive, posing a significant bottleneck
for existing methods and hindering new applications. In response, we propose a
pair of highly scalable components for 3D neural fields: Lightplane Render and
Splatter, which significantly reduce memory usage in 2D-3D mapping. These
innovations enable the processing of vastly more and higher resolution images
with small memory and computational costs. We demonstrate their utility in
various applications, from benefiting single-scene optimization with
image-level losses to realizing a versatile pipeline for dramatically scaling
3D reconstruction and generation. Code:
https://github.com/facebookresearch/lightplane.
MotionLCM: Real-time Controllable Motion Generation via Latent
  Consistency Model
Authors: name: Wenxun Dai, name: Ling-Hao Chen, name: Jingbo Wang, name: Jinpeng Liu, name: Bo Dai, name: Yansong Tang
Published: 2024-04-30 17:59:47

This work introduces MotionLCM, extending controllable motion generation to a
real-time level. Existing methods for spatial control in text-conditioned
motion generation suffer from significant runtime inefficiency. To address this
issue, we first propose the motion latent consistency model (MotionLCM) for
motion generation, building upon the latent diffusion model (MLD). By employing
one-step (or few-step) inference, we further improve the runtime efficiency of
the motion latent diffusion model for motion generation. To ensure effective
controllability, we incorporate a motion ControlNet within the latent space of
MotionLCM and enable explicit control signals (e.g., pelvis trajectory) in the
vanilla motion space to control the generation process directly, similar to
controlling other latent-free diffusion models for motion generation. By
employing these techniques, our approach can generate human motions with text
and control signals in real-time. Experimental results demonstrate the
remarkable generation and controlling capabilities of MotionLCM while
maintaining real-time runtime efficiency.
KAN: Kolmogorov-Arnold Networks
Authors: name: Ziming Liu, name: Yixuan Wang, name: Sachin Vaidya, name: Fabian Ruehle, name: James Halverson, name: Marin Soljačić, name: Thomas Y. Hou, name: Max Tegmark
Published: 2024-04-30 17:58:29

Inspired by the Kolmogorov-Arnold representation theorem, we propose
Kolmogorov-Arnold Networks (KANs) as promising alternatives to Multi-Layer
Perceptrons (MLPs). While MLPs have fixed activation functions on nodes
("neurons"), KANs have learnable activation functions on edges ("weights").
KANs have no linear weights at all -- every weight parameter is replaced by a
univariate function parametrized as a spline. We show that this seemingly
simple change makes KANs outperform MLPs in terms of accuracy and
interpretability. For accuracy, much smaller KANs can achieve comparable or
better accuracy than much larger MLPs in data fitting and PDE solving.
Theoretically and empirically, KANs possess faster neural scaling laws than
MLPs. For interpretability, KANs can be intuitively visualized and can easily
interact with human users. Through two examples in mathematics and physics,
KANs are shown to be useful collaborators helping scientists (re)discover
mathematical and physical laws. In summary, KANs are promising alternatives for
MLPs, opening opportunities for further improving today's deep learning models
which rely heavily on MLPs.
Invisible Stitch: Generating Smooth 3D Scenes with Depth Inpainting
Authors: name: Paul Engstler, name: Andrea Vedaldi, name: Iro Laina, name: Christian Rupprecht
Published: 2024-04-30 17:59:40

3D scene generation has quickly become a challenging new research direction,
fueled by consistent improvements of 2D generative diffusion models. Most prior
work in this area generates scenes by iteratively stitching newly generated
frames with existing geometry. These works often depend on pre-trained
monocular depth estimators to lift the generated images into 3D, fusing them
with the existing scene representation. These approaches are then often
evaluated via a text metric, measuring the similarity between the generated
images and a given text prompt. In this work, we make two fundamental
contributions to the field of 3D scene generation. First, we note that lifting
images to 3D with a monocular depth estimation model is suboptimal as it
ignores the geometry of the existing scene. We thus introduce a novel depth
completion model, trained via teacher distillation and self-training to learn
the 3D fusion process, resulting in improved geometric coherence of the scene.
Second, we introduce a new benchmarking scheme for scene generation methods
that is based on ground truth geometry, and thus measures the quality of the
structure of the scene.
MicroDreamer: Zero-shot 3D Generation in sim20 Seconds by Score-based
  Iterative Reconstruction
Authors: name: Luxi Chen, name: Zhengyi Wang, name: Chongxuan Li, name: Tingting Gao, name: Hang Su, name: Jun Zhu
Published: 2024-04-30 12:56:14

Optimization-based approaches, such as score distillation sampling (SDS),
show promise in zero-shot 3D generation but suffer from low efficiency,
primarily due to the high number of function evaluations (NFEs) required for
each sample. In this paper, we introduce score-based iterative reconstruction
(SIR), an efficient and general algorithm for 3D generation with a multi-view
score-based diffusion model. Given the images produced by the diffusion model,
SIR reduces NFEs by repeatedly optimizing 3D parameters, unlike the single
optimization in SDS, mimicking the 3D reconstruction process. With other
improvements including optimization in the pixel space, we present an efficient
approach called MicroDreamer that generally applies to various 3D
representations and 3D generation tasks. In particular, retaining a comparable
performance, MicroDreamer is 5-20 times faster than SDS in generating neural
radiance field and takes about 20 seconds to generate meshes from 3D Gaussian
splitting on a single A100 GPU, halving the time of the fastest zero-shot
baseline, DreamGaussian. Our code is available at
https://github.com/ML-GSAI/MicroDreamer.
Visual Fact Checker: Enabling High-Fidelity Detailed Caption Generation
Authors: name: Yunhao Ge, name: Xiaohui Zeng, name: Jacob Samuel Huffman, name: Tsung-Yi Lin, name: Ming-Yu Liu, name: Yin Cui
Published: 2024-04-30 17:55:27

Existing automatic captioning methods for visual content face challenges such
as lack of detail, content hallucination, and poor instruction following. In
this work, we propose VisualFactChecker (VFC), a flexible training-free
pipeline that generates high-fidelity and detailed captions for both 2D images
and 3D objects. VFC consists of three steps: 1) proposal, where image-to-text
captioning models propose multiple initial captions; 2) verification, where a
large language model (LLM) utilizes tools such as object detection and VQA
models to fact-check proposed captions; 3) captioning, where an LLM generates
the final caption by summarizing caption proposals and the fact check
verification results. In this step, VFC can flexibly generate captions in
various styles following complex instructions. We conduct comprehensive
captioning evaluations using four metrics: 1) CLIP-Score for image-text
similarity; 2) CLIP-Image-Score for measuring the image-image similarity
between the original and the reconstructed image generated by a text-to-image
model using the caption. 3) human study on Amazon Mechanical Turk; 4) GPT-4V
for fine-grained evaluation. Evaluation results show that VFC outperforms
state-of-the-art open-sourced captioning methods for 2D images on the COCO
dataset and 3D assets on the Objaverse dataset. Our study demonstrates that by
combining open-source models into a pipeline, we can attain captioning
capability comparable to proprietary models such as GPT-4V, despite being over
10x smaller in model size.
Extending Llama-3's Context Ten-Fold Overnight
Authors: name: Peitian Zhang, name: Ninglu Shao, name: Zheng Liu, name: Shitao Xiao, name: Hongjin Qian, name: Qiwei Ye, name: Zhicheng Dou
Published: 2024-04-30 13:25:20

We extend the context length of Llama-3-8B-Instruct from 8K to 80K via QLoRA
fine-tuning. The entire training cycle is super efficient, which takes 8 hours
on one 8xA800 (80G) GPU machine. The resulted model exhibits superior
performances across a broad range of evaluation tasks, such as NIHS, topic
retrieval, and long-context language understanding; meanwhile, it also well
preserves the original capability over short contexts. The dramatic context
extension is mainly attributed to merely 3.5K synthetic training samples
generated by GPT-4 , which indicates the LLMs' inherent (yet largely
underestimated) potential to extend its original context length. In fact, the
context length could be extended far beyond 80K with more computation
resources. Therefore, the team will publicly release the entire resources
(including data, model, data generation pipeline, training code) so as to
facilitate the future research from the community:
https://github.com/FlagOpen/FlagEmbedding.
Octopus v4: Graph of language models
Authors: name: Wei Chen, name: Zhiyuan Li
Published: 2024-04-30 06:55:45

Language models have been effective in a wide range of applications, yet the
most sophisticated models are often proprietary. For example, GPT-4 by OpenAI
and various models by Anthropic are expensive and consume substantial energy.
In contrast, the open-source community has produced competitive models, like
Llama3. Furthermore, niche-specific smaller language models, such as those
tailored for legal, medical or financial tasks, have outperformed their
proprietary counterparts. This paper introduces a novel approach that employs
functional tokens to integrate multiple open-source models,
each optimized for particular tasks. Our newly developed Octopus v4 model
leverages functional tokens to intelligently direct user queries to
the most appropriate vertical model and reformat the query to achieve the best
performance. Octopus v4, an evolution of the Octopus v1, v2, and v3 models,
excels in selection and parameter understanding and reformatting. Additionally,
we explore the use of graph as a versatile data structure that effectively
coordinates multiple open-source models by harnessing the capabilities of the
Octopus model and functional tokens. Use our open-sourced GitHub
(https://www.nexa4ai.com/) to try Octopus v4 models
(https://huggingface.co/NexaAIDev/Octopus-v4), and contrite to a larger
graph of language models. By activating models less than 10B parameters, we
achieved SOTA MMLU score of 74.8 among the same level models.
Iterative Reasoning Preference Optimization
Authors: name: Richard Yuanzhe Pang, name: Weizhe Yuan, name: Kyunghyun Cho, name: He He, name: Sainbayar Sukhbaatar, name: Jason Weston
Published: 2024-04-30 17:28:05

Iterative preference optimization methods have recently been shown to perform
well for general instruction tuning tasks, but typically make little
improvement on reasoning tasks (Yuan et al., 2024, Chen et al., 2024). In this
work we develop an iterative approach that optimizes the preference between
competing generated Chain-of-Thought (CoT) candidates by optimizing for winning
vs. losing reasoning steps that lead to the correct answer. We train using a
modified DPO loss (Rafailov et al., 2023) with an additional negative
log-likelihood term, which we find to be crucial. We show reasoning improves
across repeated iterations of this scheme. While only relying on examples in
the training set, our approach results in increasing accuracy for
Llama-2-70B-Chat from 55.6% to 81.6% on GSM8K (and 88.7% with majority voting
out of 32 samples), from 12.5% to 20.8% on MATH, and from 77.8% to 86.7% on
ARC-Challenge, which outperforms other Llama-2-based models not relying on
additionally sourced datasets.