Orionnnn/hub-stats huggingface dataset:
LLM-jp: A Cross-organizational Project for the Research and Development
  of Fully Open Japanese LLMs
Authors: name: LLM-jp, name: Akiko Aizawa, name: Eiji Aramaki, name: Bowen Chen, name: Fei Cheng, name: Hiroyuki Deguchi, name: Rintaro Enomoto, name: Kazuki Fujii, name: Kensuke Fukumoto, name: Takuya Fukushima, name: Namgi Han, name: Yuto Harada, name: Chikara Hashimoto, name: Tatsuya Hiraoka, name: Shohei Hisada, name: Sosuke Hosokawa, name: Lu Jie, name: Keisuke Kamata, name: Teruhito Kanazawa, name: Hiroki Kanezashi, name: Hiroshi Kataoka, name: Satoru Katsumata, name: Daisuke Kawahara, name: Seiya Kawano, name: Atsushi Keyaki, name: Keisuke Kiryu, name: Hirokazu Kiyomaru, name: Takashi Kodama, name: Takahiro Kubo, name: Yohei Kuga, name: Ryoma Kumon, name: Shuhei Kurita, name: Sadao Kurohashi, name: Conglong Li, name: Taiki Maekawa, name: Hiroshi Matsuda, name: Yusuke Miyao, name: Kentaro Mizuki, name: Sakae Mizuki, name: Yugo Murawaki, name: Ryo Nakamura, name: Taishi Nakamura, name: Kouta Nakayama, name: Tomoka Nakazato, name: Takuro Niitsuma, name: Jiro Nishitoba, name: Yusuke Oda, name: Hayato Ogawa, name: Takumi Okamoto, name: Naoaki Okazaki, name: Yohei Oseki, name: Shintaro Ozaki, name: Koki Ryu, name: Rafal Rzepka, name: Keisuke Sakaguchi, name: Shota Sasaki, name: Satoshi Sekine, name: Kohei Suda, name: Saku Sugawara, name: Issa Sugiura, name: Hiroaki Sugiyama, name: Hisami Suzuki, name: Jun Suzuki, name: Toyotaro Suzumura, name: Kensuke Tachibana, name: Yu Takagi, name: Kyosuke Takami, name: Koichi Takeda, name: Masashi Takeshita, name: Masahiro Tanaka, name: Kenjiro Taura, name: Arseny Tolmachev, name: Nobuhiro Ueda, name: Zhen Wan, name: Shuntaro Yada, name: Sakiko Yahata, name: Yuya Yamamoto, name: Yusuke Yamauchi, name: Hitomi Yanaka, name: Rio Yokota, name: Koichiro Yoshino
Published: 2024-07-04 14:33:03

This paper introduces LLM-jp, a cross-organizational project for the research
and development of Japanese large language models (LLMs). LLM-jp aims to
develop open-source and strong Japanese LLMs, and as of this writing, more than
1,500 participants from academia and industry are working together for this
purpose. This paper presents the background of the establishment of LLM-jp,
summaries of its activities, and technical reports on the LLMs developed by
LLM-jp. For the latest activities, visit https://llm-jp.nii.ac.jp/en/.
On scalable oversight with weak LLMs judging strong LLMs
Authors: name: Zachary Kenton, name: Noah Y. Siegel, name: János Kramár, name: Jonah Brown-Cohen, name: Samuel Albanie, name: Jannis Bulian, name: Rishabh Agarwal, name: David Lindner, name: Yunhao Tang, name: Noah D. Goodman, name: Rohin Shah
Published: 2024-07-05 16:29:15

Scalable oversight protocols aim to enable humans to accurately supervise
superhuman AI. In this paper we study debate, where two AI's compete to
convince a judge; consultancy, where a single AI tries to convince a judge that
asks questions; and compare to a baseline of direct question-answering, where
the judge just answers outright without the AI. We use large language models
(LLMs) as both AI agents and as stand-ins for human judges, taking the judge
models to be weaker than agent models. We benchmark on a diverse range of
asymmetries between judges and agents, extending previous work on a single
extractive QA task with information asymmetry, to also include mathematics,
coding, logic and multimodal reasoning asymmetries. We find that debate
outperforms consultancy across all tasks when the consultant is randomly
assigned to argue for the correct/incorrect answer. Comparing debate to direct
question answering, the results depend on the type of task: in extractive QA
tasks with information asymmetry debate outperforms direct question answering,
but in other tasks without information asymmetry the results are mixed.
Previous work assigned debaters/consultants an answer to argue for. When we
allow them to instead choose which answer to argue for, we find judges are less
frequently convinced by the wrong answer in debate than in consultancy.
Further, we find that stronger debater models increase judge accuracy, though
more modestly than in previous studies.
Learning to (Learn at Test Time): RNNs with Expressive Hidden States
Authors: name: Yu Sun, name: Xinhao Li, name: Karan Dalal, name: Jiarui Xu, name: Arjun Vikram, name: Genghan Zhang, name: Yann Dubois, name: Xinlei Chen, name: Xiaolong Wang, name: Sanmi Koyejo, name: Tatsunori Hashimoto, name: Carlos Guestrin
Published: 2024-07-05 16:23:20

Self-attention performs well in long context but has quadratic complexity.
Existing RNN layers have linear complexity, but their performance in long
context is limited by the expressive power of their hidden state. We propose a
new class of sequence modeling layers with linear complexity and an expressive
hidden state. The key idea is to make the hidden state a machine learning model
itself, and the update rule a step of self-supervised learning. Since the
hidden state is updated by training even on test sequences, our layers are
called Test-Time Training (TTT) layers. We consider two instantiations:
TTT-Linear and TTT-MLP, whose hidden state is a linear model and a two-layer
MLP respectively. We evaluate our instantiations at the scale of 125M to 1.3B
parameters, comparing with a strong Transformer and Mamba, a modern RNN. Both
TTT-Linear and TTT-MLP match or exceed the baselines. Similar to Transformer,
they can keep reducing perplexity by conditioning on more tokens, while Mamba
cannot after 16k context. With preliminary systems optimization, TTT-Linear is
already faster than Transformer at 8k context and matches Mamba in wall-clock
time. TTT-MLP still faces challenges in memory I/O, but shows larger potential
in long context, pointing to a promising direction for future research.
Planetarium: A Rigorous Benchmark for Translating Text to Structured
  Planning Languages
Authors: name: Max Zuo, name: Francisco Piedrahita Velez, name: Xiaochen Li, name: Michael L. Littman, name: Stephen H. Bach
Published: 2024-07-03 17:59:53

Many recent works have explored using language models for planning problems.
One line of research focuses on translating natural language descriptions of
planning tasks into structured planning languages, such as the planning domain
definition language (PDDL). While this approach is promising, accurately
measuring the quality of generated PDDL code continues to pose significant
challenges. First, generated PDDL code is typically evaluated using planning
validators that check whether the problem can be solved with a planner. This
method is insufficient because a language model might generate valid PDDL code
that does not align with the natural language description of the task. Second,
existing evaluation sets often have natural language descriptions of the
planning task that closely resemble the ground truth PDDL, reducing the
challenge of the task. To bridge this gap, we introduce \benchmarkName, a
benchmark designed to evaluate language models' ability to generate PDDL code
from natural language descriptions of planning tasks. We begin by creating a
PDDL equivalence algorithm that rigorously evaluates the correctness of PDDL
code generated by language models by flexibly comparing it against a ground
truth PDDL. Then, we present a dataset of 132,037 text-to-PDDL pairs across
13 different tasks, with varying levels of difficulty. Finally, we evaluate
several API-access and open-weight language models that reveal this task's
complexity. For example, 87.6% of the PDDL problem descriptions generated by
GPT-4o are syntactically parseable, 82.2% are valid, solve-able problems,
but only 35.1% are semantically correct, highlighting the need for a more
rigorous benchmark for this problem.
Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion
Authors: name: Boyuan Chen, name: Diego Marti Monso, name: Yilun Du, name: Max Simchowitz, name: Russ Tedrake, name: Vincent Sitzmann
Published: 2024-07-01 15:43:25

This paper presents Diffusion Forcing, a new training paradigm where a
diffusion model is trained to denoise a set of tokens with independent
per-token noise levels. We apply Diffusion Forcing to sequence generative
modeling by training a causal next-token prediction model to generate one or
several future tokens without fully diffusing past ones. Our approach is shown
to combine the strengths of next-token prediction models, such as
variable-length generation, with the strengths of full-sequence diffusion
models, such as the ability to guide sampling to desirable trajectories. Our
method offers a range of additional capabilities, such as (1) rolling-out
sequences of continuous tokens, such as video, with lengths past the training
horizon, where baselines diverge and (2) new sampling and guiding schemes that
uniquely profit from Diffusion Forcing's variable-horizon and causal
architecture, and which lead to marked performance gains in decision-making and
planning tasks. In addition to its empirical success, our method is proven to
optimize a variational lower bound on the likelihoods of all subsequences of
tokens drawn from the true joint distribution. Project website:
https://boyuan.space/diffusion-forcing/
Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for
  Sparse Architectural Large Language Models
Authors: name: Zihan Wang, name: Deli Chen, name: Damai Dai, name: Runxin Xu, name: Zhuoshu Li, name: Y. Wu
Published: 2024-07-02 03:11:13

Parameter-efficient fine-tuning (PEFT) is crucial for customizing Large
Language Models (LLMs) with constrained resources. Although there have been
various PEFT methods for dense-architecture LLMs, PEFT for sparse-architecture
LLMs is still underexplored. In this work, we study the PEFT method for LLMs
with the Mixture-of-Experts (MoE) architecture and the contents of this work
are mainly threefold: (1) We investigate the dispersion degree of the activated
experts in customized tasks, and found that the routing distribution for a
specific task tends to be highly concentrated, while the distribution of
activated experts varies significantly across different tasks. (2) We propose
Expert-Specialized Fine-Tuning, or ESFT, which tunes the experts most relevant
to downstream tasks while freezing the other experts and modules; experimental
results demonstrate that our method not only improves the tuning efficiency,
but also matches or even surpasses the performance of full-parameter
fine-tuning. (3) We further analyze the impact of the MoE architecture on
expert-specialized fine-tuning. We find that MoE models with finer-grained
experts are more advantageous in selecting the combination of experts that are
most relevant to downstream tasks, thereby enhancing both the training
efficiency and effectiveness.
Eliminating Position Bias of Language Models: A Mechanistic Approach
Authors: name: Ziqi Wang, name: Hanlin Zhang, name: Xiner Li, name: Kuan-Hao Huang, name: Chi Han, name: Shuiwang Ji, name: Sham M. Kakade, name: Hao Peng, name: Heng Ji
Published: 2024-07-01 09:06:57

Position bias has proven to be a prevalent issue of modern language models
(LMs), where the models prioritize content based on its position within the
given context. This bias often leads to unexpected model failures and hurts
performance, robustness, and reliability across various applications. Our
mechanistic analysis attributes the position bias to two components employed in
nearly all state-of-the-art LMs: causal attention and relative positional
encodings. Specifically, we find that causal attention generally causes models
to favor distant content, while relative positional encodings like RoPE prefer
nearby ones based on the analysis of retrieval-augmented question answering
(QA). Further, our empirical study on object detection reveals that position
bias is also present in vision-language models (VLMs).
  Based on the above analyses, we propose to ELIMINATE position bias caused by
different input segment orders (e.g., options in LM-as-a-judge, retrieved
documents in QA) in a TRAINING-FREE ZERO-SHOT manner. Our method changes the
causal attention to bidirectional attention between segments and utilizes model
attention values to decide the relative orders of segments instead of using the
order provided in input prompts, therefore enabling Position-INvariant
inferencE (PINE) at the segment level. By eliminating position bias, models
achieve better performance and reliability in downstream tasks where position
bias widely exists, such as LM-as-a-judge and retrieval-augmented QA.
  Notably, PINE is especially useful when adapting LMs for evaluating reasoning
pairs: it consistently provides 8 to 10 percentage points performance gains in
most cases, and makes Llama-3-70B-Instruct perform even better than
GPT-4-0125-preview on the RewardBench reasoning subset.
A False Sense of Safety: Unsafe Information Leakage in 'Safe' AI
  Responses
Authors: name: David Glukhov, name: Ziwen Han, name: Ilia Shumailov, name: Vardan Papyan, name: Nicolas Papernot
Published: 2024-07-02 16:19:25

Large Language Models (LLMs) are vulnerable to
jailbreaksx2013methods to elicit harmful or generally impermissible
outputs. Safety measures are developed and assessed on their effectiveness at
defending against jailbreak attacks, indicating a belief that safety is
equivalent to robustness. We assert that current defense mechanisms, such as
output filters and alignment fine-tuning, are, and will remain, fundamentally
insufficient for ensuring model safety. These defenses fail to address risks
arising from dual-intent queries and the ability to composite innocuous outputs
to achieve harmful goals. To address this critical gap, we introduce an
information-theoretic threat model called inferential adversaries who exploit
impermissible information leakage from model outputs to achieve malicious
goals. We distinguish these from commonly studied security adversaries who only
seek to force victim models to generate specific impermissible outputs. We
demonstrate the feasibility of automating inferential adversaries through
question decomposition and response aggregation. To provide safety guarantees,
we define an information censorship criterion for censorship mechanisms,
bounding the leakage of impermissible information. We propose a defense
mechanism which ensures this bound and reveal an intrinsic safety-utility
trade-off. Our work provides the first theoretically grounded understanding of
the requirements for releasing safe LLMs and the utility costs involved.
TabReD: A Benchmark of Tabular Machine Learning in-the-Wild
Authors: name: Ivan Rubachev, name: Nikolay Kartashev, name: Yury Gorishniy, name: Artem Babenko
Published: 2024-06-27 17:55:31

Benchmarks that closely reflect downstream application scenarios are
essential for the streamlined adoption of new research in tabular machine
learning (ML). In this work, we examine existing tabular benchmarks and find
two common characteristics of industry-grade tabular data that are
underrepresented in the datasets available to the academic community. First,
tabular data often changes over time in real-world deployment scenarios. This
impacts model performance and requires time-based train and test splits for
correct model evaluation. Yet, existing academic tabular datasets often lack
timestamp metadata to enable such evaluation. Second, a considerable portion of
datasets in production settings stem from extensive data acquisition and
feature engineering pipelines. For each specific dataset, this can have a
different impact on the absolute and relative number of predictive,
uninformative, and correlated features, which in turn can affect model
selection. To fill the aforementioned gaps in academic benchmarks, we introduce
TabReD -- a collection of eight industry-grade tabular datasets covering a wide
range of domains from finance to food delivery services. We assess a large
number of tabular ML models in the feature-rich, temporally-evolving data
setting facilitated by TabReD. We demonstrate that evaluation on time-based
data splits leads to different methods ranking, compared to evaluation on
random splits more common in academic benchmarks. Furthermore, on the TabReD
datasets, MLP-like architectures and GBDT show the best results, while more
sophisticated DL models are yet to prove their effectiveness.
Investigating Decoder-only Large Language Models for Speech-to-text
  Translation
Authors: name: Chao-Wei Huang, name: Hui Lu, name: Hongyu Gong, name: Hirofumi Inaguma, name: Ilia Kulikov, name: Ruslan Mavlyutov, name: Sravya Popuri
Published: 2024-07-03 14:42:49

Large language models (LLMs), known for their exceptional reasoning
capabilities, generalizability, and fluency across diverse domains, present a
promising avenue for enhancing speech-related tasks. In this paper, we focus on
integrating decoder-only LLMs to the task of speech-to-text translation (S2TT).
We propose a decoder-only architecture that enables the LLM to directly consume
the encoded speech representation and generate the text translation.
Additionally, we investigate the effects of different parameter-efficient
fine-tuning techniques and task formulation. Our model achieves
state-of-the-art performance on CoVoST 2 and FLEURS among models trained
without proprietary data. We also conduct analyses to validate the design
choices of our proposed model and bring insights to the integration of LLMs to
S2TT.
DisCo-Diff: Enhancing Continuous Diffusion Models with Discrete Latents
Authors: name: Yilun Xu, name: Gabriele Corso, name: Tommi Jaakkola, name: Arash Vahdat, name: Karsten Kreis
Published: 2024-07-03 17:42:46

Diffusion models (DMs) have revolutionized generative learning. They utilize
a diffusion process to encode data into a simple Gaussian distribution.
However, encoding a complex, potentially multimodal data distribution into a
single continuous Gaussian distribution arguably represents an unnecessarily
challenging learning problem. We propose Discrete-Continuous Latent Variable
Diffusion Models (DisCo-Diff) to simplify this task by introducing
complementary discrete latent variables. We augment DMs with learnable discrete
latents, inferred with an encoder, and train DM and encoder end-to-end.
DisCo-Diff does not rely on pre-trained networks, making the framework
universally applicable. The discrete latents significantly simplify learning
the DM's complex noise-to-data mapping by reducing the curvature of the DM's
generative ODE. An additional autoregressive transformer models the
distribution of the discrete latents, a simple step because DisCo-Diff requires
only few discrete variables with small codebooks. We validate DisCo-Diff on toy
data, several image synthesis tasks as well as molecular docking, and find that
introducing discrete latents consistently improves model performance. For
example, DisCo-Diff achieves state-of-the-art FID scores on class-conditioned
ImageNet-64/128 datasets with ODE sampler.
PicoAudio: Enabling Precise Timestamp and Frequency Controllability of
  Audio Events in Text-to-audio Generation
Authors: name: Zeyu Xie, name: Xuenan Xu, name: Zhizheng Wu, name: Mengyue Wu
Published: 2024-07-03 07:33:14

Recently, audio generation tasks have attracted considerable research
interests. Precise temporal controllability is essential to integrate audio
generation with real applications. In this work, we propose a temporal
controlled audio generation framework, PicoAudio. PicoAudio integrates temporal
information to guide audio generation through tailored model design. It
leverages data crawling, segmentation, filtering, and simulation of
fine-grained temporally-aligned audio-text data. Both subjective and objective
evaluations demonstrate that PicoAudio dramantically surpasses current
state-of-the-art generation models in terms of timestamp and occurrence
frequency controllability. The generated samples are available on the demo
website https://PicoAudio.github.io.
No Training, No Problem: Rethinking Classifier-Free Guidance for
  Diffusion Models
Authors: name: Seyedmorteza Sadat, name: Manuel Kansy, name: Otmar Hilliges, name: Romann M. Weber
Published: 2024-07-02 22:04:00

Classifier-free guidance (CFG) has become the standard method for enhancing
the quality of conditional diffusion models. However, employing CFG requires
either training an unconditional model alongside the main diffusion model or
modifying the training procedure by periodically inserting a null condition.
There is also no clear extension of CFG to unconditional models. In this paper,
we revisit the core principles of CFG and introduce a new method, independent
condition guidance (ICG), which provides the benefits of CFG without the need
for any special training procedures. Our approach streamlines the training
process of conditional diffusion models and can also be applied during
inference on any pre-trained conditional model. Additionally, by leveraging the
time-step information encoded in all diffusion networks, we propose an
extension of CFG, called time-step guidance (TSG), which can be applied to any
diffusion model, including unconditional ones. Our guidance techniques are easy
to implement and have the same sampling cost as CFG. Through extensive
experiments, we demonstrate that ICG matches the performance of standard CFG
across various conditional diffusion models. Moreover, we show that TSG
improves generation quality in a manner similar to CFG, without relying on any
conditional information.
InternLM-XComposer-2.5: A Versatile Large Vision Language Model
  Supporting Long-Contextual Input and Output
Authors: name: Pan Zhang, name: Xiaoyi Dong, name: Yuhang Zang, name: Yuhang Cao, name: Rui Qian, name: Lin Chen, name: Qipeng Guo, name: Haodong Duan, name: Bin Wang, name: Linke Ouyang, name: Songyang Zhang, name: Wenwei Zhang, name: Yining Li, name: Yang Gao, name: Peng Sun, name: Xinyue Zhang, name: Wei Li, name: Jingwen Li, name: Wenhai Wang, name: Hang Yan, name: Conghui He, name: Xingcheng Zhang, name: Kai Chen, name: Jifeng Dai, name: Yu Qiao, name: Dahua Lin, name: Jiaqi Wang
Published: 2024-07-03 17:59:21

We present InternLM-XComposer-2.5 (IXC-2.5), a versatile large-vision
language model that supports long-contextual input and output. IXC-2.5 excels
in various text-image comprehension and composition applications, achieving
GPT-4V level capabilities with merely 7B LLM backend. Trained with 24K
interleaved image-text contexts, it can seamlessly extend to 96K long contexts
via RoPE extrapolation. This long-context capability allows IXC-2.5 to excel in
tasks requiring extensive input and output contexts. Compared to its previous
2.0 version, InternLM-XComposer-2.5 features three major upgrades in
vision-language comprehension: (1) Ultra-High Resolution Understanding, (2)
Fine-Grained Video Understanding, and (3) Multi-Turn Multi-Image Dialogue. In
addition to comprehension, IXC-2.5 extends to two compelling applications using
extra LoRA parameters for text-image composition: (1) Crafting Webpages and (2)
Composing High-Quality Text-Image Articles. IXC-2.5 has been evaluated on 28
benchmarks, outperforming existing open-source state-of-the-art models on 16
benchmarks. It also surpasses or competes closely with GPT-4V and Gemini Pro on
16 key tasks. The InternLM-XComposer-2.5 is publicly available at
https://github.com/InternLM/InternLM-XComposer.