GitHub repository: DarcStar-Solutions-Tech/dilated-attention-pytorch
Primary language: Python
Description: (Unofficial) Implementation of dilated attention from "LongNet: Scaling Transformers to 1,000,000,000 Tokens" (https://arxiv.org/abs/2307.02486)