Orionnnn/hub-stats huggingface dataset:
A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts
Authors: name: Kuang-Huei Lee, name: Xinyun Chen, name: Hiroki Furuta, name: John Canny, name: Ian Fischer
Published: 2024-02-15 05:40:21

Current Large Language Models (LLMs) are not only limited to some maximum
context length, but also are not able to robustly consume long inputs. To
address these limitations, we propose ReadAgent, an LLM agent system that
increases effective context length up to 20x in our experiments. Inspired by
how humans interactively read long documents, we implement ReadAgent as a
simple prompting system that uses the advanced language capabilities of LLMs to
(1) decide what content to store together in a memory episode, (2) compress
those memory episodes into short episodic memories called gist memories, and
(3) take actions to look up passages in the original text if ReadAgent needs to
remind itself of relevant details to complete a task. We evaluate ReadAgent
against baselines using retrieval methods, using the original long contexts,
and using the gist memories. These evaluations are performed on three
long-document reading comprehension tasks: QuALITY, NarrativeQA, and QMSum.
ReadAgent outperforms the baselines on all three tasks while extending the
effective context window by 3-20x.
Rolling Diffusion Models
Authors: name: David Ruhe, name: Jonathan Heek, name: Tim Salimans, name: Emiel Hoogeboom
Published: 2024-02-12 08:16:10

Diffusion models have recently been increasingly applied to temporal data
such as video, fluid mechanics simulations, or climate data. These methods
generally treat subsequent frames equally regarding the amount of noise in the
diffusion process. This paper explores Rolling Diffusion: a new approach that
uses a sliding window denoising process. It ensures that the diffusion process
progressively corrupts through time by assigning more noise to frames that
appear later in a sequence, reflecting greater uncertainty about the future as
the generation process unfolds. Empirically, we show that when the temporal
dynamics are complex, Rolling Diffusion is superior to standard diffusion. In
particular, this result is demonstrated in a video prediction task using the
Kinetics-600 video dataset and in a chaotic fluid dynamics forecasting
experiment.
OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset
Authors: name: Shubham Toshniwal, name: Ivan Moshkov, name: Sean Narenthiran, name: Daria Gitman, name: Fei Jia, name: Igor Gitman
Published: 2024-02-15 18:26:11

Recent work has shown the immense potential of synthetically generated
datasets for training large language models (LLMs), especially for acquiring
targeted skills. Current large-scale math instruction tuning datasets such as
MetaMathQA (Yu et al., 2024) and MAmmoTH (Yue et al., 2024) are constructed
using outputs from closed-source LLMs with commercially restrictive licenses. A
key reason limiting the use of open-source LLMs in these data generation
pipelines has been the wide gap between the mathematical skills of the best
closed-source LLMs, such as GPT-4, and the best open-source LLMs. Building on
the recent progress in open-source LLMs, our proposed prompting novelty, and
some brute-force scaling, we construct OpenMathInstruct-1, a math instruction
tuning dataset with 1.8M problem-solution pairs. The dataset is constructed by
synthesizing code-interpreter solutions for GSM8K and MATH, two popular math
reasoning benchmarks, using the recently released and permissively licensed
Mixtral model. Our best model, OpenMath-CodeLlama-70B, trained on a subset of
OpenMathInstruct-1, achieves a score of 84.6% on GSM8K and 50.7% on MATH, which
is competitive with the best gpt-distilled models. We release our code, models,
and the OpenMathInstruct-1 dataset under a commercially permissive license.
Data Engineering for Scaling Language Models to 128K Context
Authors: name: Yao Fu, name: Rameswar Panda, name: Xinyao Niu, name: Xiang Yue, name: Hannaneh Hajishirzi, name: Yoon Kim, name: Hao Peng
Published: 2024-02-15 18:19:16

We study the continual pretraining recipe for scaling language models'
context lengths to 128K, with a focus on data engineering. We hypothesize that
long context modeling, in particular the ability to utilize information
at arbitrary input locations, is a capability that is mostly already acquired
through large-scale pretraining, and that this capability can be readily
extended to contexts substantially longer than seen during training~(e.g., 4K
to 128K) through lightweight continual pretraining on appropriate data mixture.
We investigate the quantity and quality of the data for
continual pretraining: (1) for quantity, we show that 500 million to 5 billion
tokens are enough to enable the model to retrieve information anywhere within
the 128K context; (2) for quality, our results equally emphasize domain
balance and length upsampling. Concretely, we find that naively
upsampling longer data on certain domains like books, a common practice of
existing work, gives suboptimal performance, and that a balanced domain mixture
is important. We demonstrate that continual pretraining of the full model on
1B-5B tokens of such data is an effective and affordable strategy for scaling
the context length of language models to 128K. Our recipe outperforms strong
open-source long-context models and closes the gap to frontier models like
GPT-4 128K.
Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation
Authors: name: Huizhuo Yuan, name: Zixiang Chen, name: Kaixuan Ji, name: Quanquan Gu
Published: 2024-02-15 18:59:18

Fine-tuning Diffusion Models remains an underexplored frontier in generative
artificial intelligence (GenAI), especially when compared with the remarkable
progress made in fine-tuning Large Language Models (LLMs). While cutting-edge
diffusion models such as Stable Diffusion (SD) and SDXL rely on supervised
fine-tuning, their performance inevitably plateaus after seeing a certain
volume of data. Recently, reinforcement learning (RL) has been employed to
fine-tune diffusion models with human preference data, but it requires at least
two images ("winner" and "loser" images) for each text prompt. In this paper,
we introduce an innovative technique called self-play fine-tuning for diffusion
models (SPIN-Diffusion), where the diffusion model engages in competition with
its earlier versions, facilitating an iterative self-improvement process. Our
approach offers an alternative to conventional supervised fine-tuning and RL
strategies, significantly improving both model performance and alignment. Our
experiments on the Pick-a-Pic dataset reveal that SPIN-Diffusion outperforms
the existing supervised fine-tuning method in aspects of human preference
alignment and visual appeal right from its first iteration. By the second
iteration, it exceeds the performance of RLHF-based methods across all metrics,
achieving these results with less data.
Computing Power and the Governance of Artificial Intelligence
Authors: name: Girish Sastry, name: Lennart Heim, name: Haydn Belfield, name: Markus Anderljung, name: Miles Brundage, name: Julian Hazell, name: Cullen O'Keefe, name: Gillian K. Hadfield, name: Richard Ngo, name: Konstantin Pilz, name: George Gor, name: Emma Bluemke, name: Sarah Shoker, name: Janet Egan, name: Robert F. Trager, name: Shahar Avin, name: Adrian Weller, name: Yoshua Bengio, name: Diane Coyle
Published: 2024-02-13 21:10:21

Computing power, or "compute," is crucial for the development and deployment
of artificial intelligence (AI) capabilities. As a result, governments and
companies have started to leverage compute as a means to govern AI. For
example, governments are investing in domestic compute capacity, controlling
the flow of compute to competing countries, and subsidizing compute access to
certain sectors. However, these efforts only scratch the surface of how compute
can be used to govern AI development and deployment. Relative to other key
inputs to AI (data and algorithms), AI-relevant compute is a particularly
effective point of intervention: it is detectable, excludable, and
quantifiable, and is produced via an extremely concentrated supply chain. These
characteristics, alongside the singular importance of compute for cutting-edge
AI models, suggest that governing compute can contribute to achieving common
policy objectives, such as ensuring the safety and beneficial use of AI. More
precisely, policymakers could use compute to facilitate regulatory visibility
of AI, allocate resources to promote beneficial outcomes, and enforce
restrictions against irresponsible or malicious AI development and usage.
However, while compute-based policies and technologies have the potential to
assist in these areas, there is significant variation in their readiness for
implementation. Some ideas are currently being piloted, while others are
hindered by the need for fundamental research. Furthermore, naive or poorly
scoped approaches to compute governance carry significant risks in areas like
privacy, economic impacts, and centralization of power. We end by suggesting
guardrails to minimize these risks from compute governance.
Towards Next-Level Post-Training Quantization of Hyper-Scale
  Transformers
Authors: name: Junhan Kim, name: Kyungphil Park, name: Chungman Lee, name: Ho-young Kim, name: Joonyoung Kim, name: Yongkweon Jeon
Published: 2024-02-14 05:58:43

With the increasing complexity of generative AI models, post-training
quantization (PTQ) has emerged as a promising solution for deploying
hyper-scale models on edge devices such as mobile devices and TVs. Existing PTQ
schemes, however, consume considerable time and resources, which could be a
bottleneck in real situations where frequent model updates and multiple
hyper-parameter tunings are required. As a cost-effective alternative, one-shot
PTQ schemes have been proposed. Still, the performance is somewhat limited
because they cannot consider the inter-layer dependency within the attention
module, which is a very important feature of Transformers. In this paper, we
thus propose a novel PTQ algorithm that balances accuracy and efficiency. The
key idea of the proposed algorithm called aespa is to perform quantization
layer-wise for efficiency while considering cross-layer dependency to preserve
the attention score. Through extensive experiments on various language models
and complexity analysis, we demonstrate that aespa is accurate and efficient in
quantizing Transformer models.
PRDP: Proximal Reward Difference Prediction for Large-Scale Reward
  Finetuning of Diffusion Models
Authors: name: Fei Deng, name: Qifei Wang, name: Wei Wei, name: Matthias Grundmann, name: Tingbo Hou
Published: 2024-02-13 18:58:16

Reward finetuning has emerged as a promising approach to aligning foundation
models with downstream objectives. Remarkable success has been achieved in the
language domain by using reinforcement learning (RL) to maximize rewards that
reflect human preference. However, in the vision domain, existing RL-based
reward finetuning methods are limited by their instability in large-scale
training, rendering them incapable of generalizing to complex, unseen prompts.
In this paper, we propose Proximal Reward Difference Prediction (PRDP),
enabling stable black-box reward finetuning for diffusion models for the first
time on large-scale prompt datasets with over 100K prompts. Our key innovation
is the Reward Difference Prediction (RDP) objective that has the same optimal
solution as the RL objective while enjoying better training stability.
Specifically, the RDP objective is a supervised regression objective that tasks
the diffusion model with predicting the reward difference of generated image
pairs from their denoising trajectories. We theoretically prove that the
diffusion model that obtains perfect reward difference prediction is exactly
the maximizer of the RL objective. We further develop an online algorithm with
proximal updates to stably optimize the RDP objective. In experiments, we
demonstrate that PRDP can match the reward maximization ability of
well-established RL-based methods in small-scale training. Furthermore, through
large-scale training on text prompts from the Human Preference Dataset v2 and
the Pick-a-Pic v1 dataset, PRDP achieves superior generation quality on a
diverse set of complex, unseen prompts whereas RL-based methods completely
fail.
MPIrigen: MPI Code Generation through Domain-Specific Language Models
Authors: name: Nadav Schneider, name: Niranjan Hasabnis, name: Vy A. Vo, name: Tal Kadosh, name: Neva Krien, name: Mihai Capotă, name: Abdul Wasay, name: Guy Tamir, name: Ted Willke, name: Nesreen Ahmed, name: Yuval Pinter, name: Timothy Mattson, name: Gal Oren
Published: 2024-02-14 12:24:21

The imperative need to scale computation across numerous nodes highlights the
significance of efficient parallel computing, particularly in the realm of
Message Passing Interface (MPI) integration. The challenging parallel
programming task of generating MPI-based parallel programs has remained
unexplored. This study first investigates the performance of state-of-the-art
language models in generating MPI-based parallel programs. Findings reveal that
widely used models such as GPT-3.5 and PolyCoder (specialized multi-lingual
code models) exhibit notable performance degradation, when generating MPI-based
programs compared to general-purpose programs. In contrast, domain-specific
models such as MonoCoder, which are pretrained on MPI-related programming
languages of C and C++, outperform larger models. Subsequently, we introduce a
dedicated downstream task of MPI-based program generation by fine-tuning
MonoCoder on HPCorpusMPI. We call the resulting model as MPIrigen. We propose
an innovative preprocessing for completion only after observing the whole code,
thus enabling better completion with a wider context. Comparative analysis
against GPT-3.5 zero-shot performance, using a novel HPC-oriented evaluation
method, demonstrates that MPIrigen excels in generating accurate MPI functions
up to 0.8 accuracy in location and function predictions, and with more than 0.9
accuracy for argument predictions. The success of this tailored solution
underscores the importance of domain-specific fine-tuning in optimizing
language models for parallel computing code generation, paving the way for a
new generation of automatic parallelization tools. The sources of this work are
available at our GitHub MPIrigen repository:
https://github.com/Scientific-Computing-Lab-NRCN/MPI-rigen
Transformers Can Achieve Length Generalization But Not Robustly
Authors: name: Yongchao Zhou, name: Uri Alon, name: Xinyun Chen, name: Xuezhi Wang, name: Rishabh Agarwal, name: Denny Zhou
Published: 2024-02-14 18:18:29

Length generalization, defined as the ability to extrapolate from shorter
training sequences to longer test ones, is a significant challenge for language
models. This issue persists even with large-scale Transformers handling
relatively straightforward tasks. In this paper, we test the Transformer's
ability of length generalization using the task of addition of two integers. We
show that the success of length generalization is intricately linked to the
data format and the type of position encoding. Using the right combination of
data format and position encodings, we show for the first time that standard
Transformers can extrapolate to a sequence length that is 2.5x the input
length. Nevertheless, unlike in-distribution generalization, length
generalization remains fragile, significantly influenced by factors like random
weight initialization and training data order, leading to large variances
across different random seeds.
Premise Order Matters in Reasoning with Large Language Models
Authors: name: Xinyun Chen, name: Ryan A. Chi, name: Xuezhi Wang, name: Denny Zhou
Published: 2024-02-14 04:50:18

Large language models (LLMs) have accomplished remarkable reasoning
performance in various domains. However, in the domain of reasoning tasks, we
discover a frailty: LLMs are surprisingly brittle to the ordering of the
premises, despite the fact that such ordering does not alter the underlying
task. In particular, we observe that LLMs achieve the best performance when the
premise order aligns with the context required in intermediate reasoning steps.
For example, in deductive reasoning tasks, presenting the premises in the same
order as the ground truth proof in the prompt (as opposed to random ordering)
drastically increases the model's accuracy. We first examine the effect of
premise ordering on deductive reasoning on a variety of LLMs, and our
evaluation shows that permuting the premise order can cause a performance drop
of over 30%. In addition, we release the benchmark R-GSM, based on GSM8K, to
examine the ordering effect for mathematical problem-solving, and we again
observe a significant drop in accuracy, relative to the original GSM8K
benchmark.
GhostWriter: Augmenting Collaborative Human-AI Writing Experiences
  Through Personalization and Agency
Authors: name: Catherine Yeh, name: Gonzalo Ramos, name: Rachel Ng, name: Andy Huntington, name: Richard Banks
Published: 2024-02-13 23:48:59

Large language models (LLMs) are becoming more prevalent and have found a
ubiquitous use in providing different forms of writing assistance. However,
LLM-powered writing systems can frustrate users due to their limited
personalization and control, which can be exacerbated when users lack
experience with prompt engineering. We see design as one way to address these
challenges and introduce GhostWriter, an AI-enhanced writing design probe where
users can exercise enhanced agency and personalization. GhostWriter leverages
LLMs to learn the user's intended writing style implicitly as they write, while
allowing explicit teaching moments through manual style edits and annotations.
We study 18 participants who use GhostWriter on two different writing tasks,
observing that it helps users craft personalized text generations and empowers
them by providing multiple ways to control the system's writing style. From
this study, we present insights regarding people's relationship with
AI-assisted writing and offer design recommendations for future work.
L3GO: Language Agents with Chain-of-3D-Thoughts for Generating
  Unconventional Objects
Authors: name: Yutaro Yamada, name: Khyathi Chandu, name: Yuchen Lin, name: Jack Hessel, name: Ilker Yildirim, name: Yejin Choi
Published: 2024-02-14 09:51:05

Diffusion-based image generation models such as DALL-E 3 and Stable
Diffusion-XL demonstrate remarkable capabilities in generating images with
realistic and unique compositions. Yet, these models are not robust in
precisely reasoning about physical and spatial configurations of objects,
especially when instructed with unconventional, thereby out-of-distribution
descriptions, such as "a chair with five legs". In this paper, we propose a
language agent with chain-of-3D-thoughts (L3GO), an inference-time approach
that can reason about part-based 3D mesh generation of unconventional objects
that current data-driven diffusion models struggle with. More concretely, we
use large language models as agents to compose a desired object via
trial-and-error within the 3D simulation environment. To facilitate our
investigation, we develop a new benchmark, Unconventionally Feasible Objects
(UFO), as well as SimpleBlenv, a wrapper environment built on top of Blender
where language agents can build and compose atomic building blocks via API
calls. Human and automatic GPT-4V evaluations show that our approach surpasses
the standard GPT-4 and other language agents (e.g., ReAct and Reflexion) for 3D
mesh generation on ShapeNet. Moreover, when tested on our UFO benchmark, our
approach outperforms other state-of-the-art text-to-2D image and text-to-3D
models based on human evaluation.
Magic-Me: Identity-Specific Video Customized Diffusion
Authors: name: Ze Ma, name: Daquan Zhou, name: Chun-Hsiao Yeh, name: Xue-She Wang, name: Xiuyu Li, name: Huanrui Yang, name: Zhen Dong, name: Kurt Keutzer, name: Jiashi Feng
Published: 2024-02-14 18:13:51

Creating content for a specific identity (ID) has shown significant interest
in the field of generative models. In the field of text-to-image generation
(T2I), subject-driven content generation has achieved great progress with the
ID in the images controllable. However, extending it to video generation is not
well explored. In this work, we propose a simple yet effective subject identity
controllable video generation framework, termed Video Custom Diffusion (VCD).
With a specified subject ID defined by a few images, VCD reinforces the
identity information extraction and injects frame-wise correlation at the
initialization stage for stable video outputs with identity preserved to a
large extent. To achieve this, we propose three novel components that are
essential for high-quality ID preservation: 1) an ID module trained with the
cropped identity by prompt-to-segmentation to disentangle the ID information
and the background noise for more accurate ID token learning; 2) a
text-to-video (T2V) VCD module with 3D Gaussian Noise Prior for better
inter-frame consistency and 3) video-to-video (V2V) Face VCD and Tiled VCD
modules to deblur the face and upscale the video for higher resolution.
  Despite its simplicity, we conducted extensive experiments to verify that VCD
is able to generate stable and high-quality videos with better ID over the
selected strong baselines. Besides, due to the transferability of the ID
module, VCD is also working well with finetuned text-to-image models available
publically, further improving its usability. The codes are available at
https://github.com/Zhen-Dong/Magic-Me.
ChatCell: Facilitating Single-Cell Analysis with Natural Language
Authors: name: Yin Fang, name: Kangwei Liu, name: Ningyu Zhang, name: Xinle Deng, name: Penghui Yang, name: Zhuo Chen, name: Xiangru Tang, name: Mark Gerstein, name: Xiaohui Fan, name: Huajun Chen
Published: 2024-02-13 09:06:14

As Large Language Models (LLMs) rapidly evolve, their influence in science is
becoming increasingly prominent. The emerging capabilities of LLMs in task
generalization and free-form dialogue can significantly advance fields like
chemistry and biology. However, the field of single-cell biology, which forms
the foundational building blocks of living organisms, still faces several
challenges. High knowledge barriers and limited scalability in current methods
restrict the full exploitation of LLMs in mastering single-cell data, impeding
direct accessibility and rapid iteration. To this end, we introduce ChatCell,
which signifies a paradigm shift by facilitating single-cell analysis with
natural language. Leveraging vocabulary adaptation and unified sequence
generation, ChatCell has acquired profound expertise in single-cell biology and
the capability to accommodate a diverse range of analysis tasks. Extensive
experiments further demonstrate ChatCell's robust performance and potential to
deepen single-cell insights, paving the way for more accessible and intuitive
exploration in this pivotal field. Our project homepage is available at
https://zjunlp.github.io/project/ChatCell.