Gmc2 (Maozhou)

upvoted a paper 5 days ago

A Unified Sequence Parallelism Approach for Long Context Generative AI

Paper • 2405.07719 • Published 9 days ago • 1

upvoted a paper 6 days ago

Beyond Scaling Laws: Understanding Transformer Performance with Associative Memory

Paper • 2405.08707 • Published 8 days ago • 24

upvoted an article 7 days ago

Article

Welcome Mixtral - a SOTA Mixture of Experts on Hugging Face

Dec 11, 2023

• 5

upvoted an article 9 days ago

Article

Mixture of Experts Explained

Dec 11, 2023

• 70

upvoted 2 papers 11 days ago

Generating Long Sequences with Sparse Transformers

Paper • 1904.10509 • Published Apr 23, 2019 • 1

Longformer: The Long-Document Transformer

Paper • 2004.05150 • Published Apr 10, 2020 • 3

upvoted a paper 13 days ago

vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention

Paper • 2405.04437 • Published 15 days ago • 3

upvoted a paper 15 days ago

Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing

Paper • 2306.12929 • Published Jun 22, 2023 • 11

upvoted a paper 25 days ago

Make Your LLM Fully Utilize the Context

Paper • 2404.16811 • Published 27 days ago • 51

upvoted a paper 27 days ago

OpenELM: An Efficient Language Model Family with Open-source Training and Inference Framework

Paper • 2404.14619 • Published 29 days ago • 120

upvoted a collection about 1 month ago

Meta Llama 3

Collection

This collection hosts the transformers and original repos of the Meta Llama 3 and Llama Guard 2 releases • 5 items • Updated Apr 18 • 533

upvoted 2 papers about 1 month ago

Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length

Paper • 2404.08801 • Published Apr 12 • 62

Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

Paper • 2404.07143 • Published Apr 10 • 92

upvoted an article about 1 month ago

Article

Llama 2 is here - get it on Hugging Face

Jul 18, 2023

• 15

upvoted a paper about 1 month ago

MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies

Paper • 2404.06395 • Published Apr 9 • 18

upvoted an article about 1 month ago

Article

The Technology Behind BLOOM Training

Jul 14, 2022

• 3

upvoted 7 papers about 2 months ago

Octopus v2: On-device language model for super agent

Paper • 2404.01744 • Published Apr 2 • 53

MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens

Paper • 2404.03413 • Published Apr 4 • 21

Long-context LLMs Struggle with Long In-context Learning

Paper • 2404.02060 • Published Apr 2 • 33

Mixture-of-Depths: Dynamically allocating compute in transformer-based language models

Paper • 2404.02258 • Published Apr 2 • 101

Rethinking Memory and Communication Cost for Efficient Large Language Model Training

Paper • 2310.06003 • Published Oct 9, 2023 • 2

Optimized Network Architectures for Large Language Model Training with Billions of Parameters

Paper • 2307.12169 • Published Jul 22, 2023 • 9

Mixtral of Experts

Paper • 2401.04088 • Published Jan 8 • 152

upvoted 10 papers 2 months ago

Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference

Paper • 2403.14520 • Published Mar 21 • 31

Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey

Paper • 2311.12351 • Published Nov 21, 2023 • 3

Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache

Paper • 2401.02669 • Published Jan 5 • 10

Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models

Paper • 2402.02244 • Published Feb 3 • 1

BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences

Paper • 2403.09347 • Published Mar 14 • 20

LightSeq: Sequence Level Parallelism for Distributed Training of Long Context Transformers

Paper • 2310.03294 • Published Oct 5, 2023 • 2

upvoted a collection 2 months ago

Long context

Collection

72 items • Updated 4 days ago • 21

upvoted 8 papers 2 months ago

DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models

Paper • 2309.14509 • Published Sep 25, 2023 • 16

Striped Attention: Faster Ring Attention for Causal Transformers

Paper • 2311.09431 • Published Nov 15, 2023 • 4

Ring Attention with Blockwise Transformers for Near-Infinite Context

Paper • 2310.01889 • Published Oct 3, 2023 • 8

Sequence Parallelism: Long Sequence Training from System Perspective

Paper • 2105.13120 • Published May 26, 2021 • 5

Algorithmic progress in language models

Paper • 2403.05812 • Published Mar 9 • 17

DeepSeek-VL: Towards Real-World Vision-Language Understanding

Paper • 2403.05525 • Published Mar 8 • 38

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

Paper • 2403.05530 • Published Mar 8 • 50

Yi: Open Foundation Models by 01.AI

Paper • 2403.04652 • Published Mar 7 • 59

upvoted a paper 3 months ago

MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs

Paper • 2402.15627 • Published Feb 23 • 31

upvoted 3 papers 4 months ago

Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research

Paper • 2402.00159 • Published Jan 31 • 55

Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads

Paper • 2401.10774 • Published Jan 19 • 50

MoE-LLaVA: Mixture of Experts for Large Vision-Language Models

Paper • 2401.15947 • Published Jan 29 • 46

Maozhou

AI & ML interests

Organizations

Gmc2's activity

Welcome Mixtral - a SOTA Mixture of Experts on Hugging Face

Mixture of Experts Explained

Llama 2 is here - get it on Hugging Face

The Technology Behind BLOOM Training