Add x-transformers library by lucidrains
Author: asigalov61
Pull request: no
Labels: name: Feature request

### Feature request

Hey guys!

I would like to sincerely ask you to consider adding x-transformers library to Hugging Face transformers.
https://github.com/lucidrains/x-transformers

This library is very popular and versatile and its being used in many projects!

I am actually very surprised that this library was overlooked.

### Motivation

I am a heavy Hugging Face PRO user with many models and implementations that use x-transformers library so it would be very handy and cool to be able to have it integrated in Hugging Face so that the users can easily use mine and other peoples' models which use this library.

### Your contribution

I can help with whatever is needed to help integrate this library.

Comments:
I don't understand what's the request exactly here ? you mean like adding the library as a dependency even though nothing in transformers depend on it ?

@IlyasMoutawwakil No. I wanted to suggest adding it so that the models made with x-transformers could be uploaded and used from_pre_trained method.

There is even a menu of the libraries for that in the models cards:

![Image](https://github.com/user-attachments/assets/7233d664-94c7-490e-9474-781b9fe43636)

I realize now that maybe my request belongs to hugging face hub, not transformers, since the models are on the hub?

The documentation is a bit confusing about where to ask for something like that and its not clear from the docs where such request should be posted.

Thank you.

Alex.

@IlyasMoutawwakil I've asked Microsoft Copilot and it explained to me that what I need is to have an x-transformers classes for AutoModel/AutoConfig added in transformers in order to use x-transformers models without having to write a separate codebase and use remote code when pushing/pulling and using the model from the HF hub.

```
from transformers import PretrainedConfig

class XTransformersConfig(PretrainedConfig):
    model_type = "x_transformers"
    def __init__(self, dim=512, depth=6, heads=8, xattn=True, vocab_size=50_000, **kwargs):
        super().__init__(**kwargs)
        self.dim = dim
        self.depth = depth
        self.heads = heads
        self.xattn = xattn
        self.vocab_size = vocab_size
```

```
import torch.nn as nn
from transformers import PreTrainedModel
from x_transformers import XTransformer
from transformers.modeling_outputs import CausalLMOutput
from .configuration_x_transformers import XTransformersConfig

class XTransformersForCausalLM(PreTrainedModel):
    config_class = XTransformersConfig
    base_model_prefix = "transformer"

    def __init__(self, config):
        super().__init__(config)
        self.transformer = XTransformer(
            dim=config.dim, depth=config.depth,
            heads=config.heads, xattn=config.xattn,
            vocab_size=config.vocab_size
        )
        self.lm_head = nn.Linear(config.dim, config.vocab_size, bias=False)
        self.post_init()

    def get_input_embeddings(self): return self.transformer.token_emb
    def set_input_embeddings(self, new_emb): self.transformer.token_emb = new_emb

    def forward(self, input_ids, attention_mask=None, labels=None, **kwargs):
        hidden = self.transformer(input_ids, attn_mask=attention_mask)
        logits = self.lm_head(hidden)
        loss = None
        if labels is not None:
            shift_logits = logits[..., :-1, :].contiguous()
            shift_labels = labels[..., 1:].contiguous()
            loss = nn.CrossEntropyLoss()(
                shift_logits.view(-1, self.config.vocab_size),
                shift_labels.view(-1),
            )
        return CausalLMOutput(loss=loss, logits=logits)
```


Hi @asigalov61 could you open an issue on the x-transformers library and ping me there, I can help you in making x-transformers an integrated library on the hub with push and load capabilities.

See also here for more info: https://huggingface.co/docs/huggingface_hub/guides/integrations