Collections

24

TRAMS: Training-free Memory Selection for Long-range Language Modeling

Paper • 2310.15494 • Published Oct 24, 2023 • 1
A Long Way to Go: Investigating Length Correlations in RLHF

Paper • 2310.03716 • Published Oct 5, 2023 • 9
YaRN: Efficient Context Window Extension of Large Language Models

Paper • 2309.00071 • Published Aug 31, 2023 • 59
Giraffe: Adventures in Expanding Context Lengths in LLMs

Paper • 2308.10882 • Published Aug 21, 2023 • 1

-

Sequence Parallelism: Long Sequence Training from System Perspective

Paper • 2105.13120 • Published May 26, 2021 • 5
Ring Attention with Blockwise Transformers for Near-Infinite Context

Paper • 2310.01889 • Published Oct 3, 2023 • 8
Striped Attention: Faster Ring Attention for Causal Transformers

Paper • 2311.09431 • Published Nov 15, 2023 • 4
DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models

Paper • 2309.14509 • Published Sep 25, 2023 • 16

LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning

Paper • 2401.01325 • Published Jan 2 • 25

TRAMS: Training-free Memory Selection for Long-range Language Modeling

A Long Way to Go: Investigating Length Correlations in RLHF

YaRN: Efficient Context Window Extension of Large Language Models

Giraffe: Adventures in Expanding Context Lengths in LLMs

Sequence Parallelism: Long Sequence Training from System Perspective

Ring Attention with Blockwise Transformers for Near-Infinite Context

Striped Attention: Faster Ring Attention for Causal Transformers

DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models

LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens

Data Engineering for Scaling Language Models to 128K Context

LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaboration

The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Survey

Chain-of-Thought Reasoning Without Prompting

How to Train Data-Efficient LLMs

BitDelta: Your Fine-Tune May Only Be Worth One Bit

A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts

E^2-LLM: Efficient and Extreme Length Extension of Large Language Models

LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning

Extending LLMs' Context Window with 100 Samples

LongAlign: A Recipe for Long Context Alignment of Large Language Models

Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models

Simple linear attention language models balance the recall-throughput tradeoff

ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition

Linear Transformers are Versatile In-Context Learners

Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads

APAR: LLMs Can Do Auto-Parallel Auto-Regressive Decoding

Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache

MambaByte: Token-free Selective State Space Model

Self-Rewarding Language Models

ReFT: Reasoning with Reinforced Fine-Tuning

Tuning Language Models by Proxy

TrustLLM: Trustworthiness in Large Language Models

Soaring from 4K to 400K: Extending LLM's Context with Activation Beacon

MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers

YaRN: Efficient Context Window Extension of Large Language Models

Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache

LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning