Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length
Paper
•
2404.08801
•
Published
•
62
Context Length
-
-
Ring Attention with Blockwise Transformers for Near-Infinite Context
Paper • 2310.01889 • Published • 8 -
World Model on Million-Length Video And Language With RingAttention
Paper • 2402.08268 • Published • 33 -
Scaling Transformer to 1M tokens and beyond with RMT
Paper • 2304.11062 • Published • 2 -
Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention
Paper • 2404.07143 • Published • 92 -
LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
Paper • 2402.13753 • Published • 104 -
Make Your LLM Fully Utilize the Context
Paper • 2404.16811 • Published • 51 -
Long-context LLMs Struggle with Long In-context Learning
Paper • 2404.02060 • Published • 33 -
RULER: What's the Real Context Size of Your Long-Context Language Models?
Paper • 2404.06654 • Published • 32 -
Soaring from 4K to 400K: Extending LLM's Context with Activation Beacon
Paper • 2401.03462 • Published • 25 -
Flexibly Scaling Large Language Models Contexts Through Extensible Tokenization
Paper • 2401.07793 • Published • 3 -
KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
Paper • 2401.18079 • Published • 5 -
LongNet: Scaling Transformers to 1,000,000,000 Tokens
Paper • 2307.02486 • Published • 79 -
YaRN: Efficient Context Window Extension of Large Language Models
Paper • 2309.00071 • Published • 59 -
Compressed Context Memory For Online Language Model Interaction
Paper • 2312.03414 • Published -
Extending LLMs' Context Window with 100 Samples
Paper • 2401.07004 • Published • 14