File size: 2,432 Bytes
c479b7e 4222993 c479b7e 4222993 c479b7e 4222993 c479b7e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 |
---
tags:
- antibody language model
- antibody
- protein language model
base_model: Rostlab/prot_t5_xl_uniref50
license: mit
---
# IgT5 unpaired model
Model pretrained on protein and antibody sequences using a masked language modeling (MLM) objective. It was introduced in the paper [Large scale paired antibody language models](https://arxiv.org/abs/2403.17889).
The model is finetuned from ProtT5 using unpaired antibody sequences from the [Observed Antibody Space](https://opig.stats.ox.ac.uk/webapps/oas/).
# Use
The encoder part of the model and tokeniser can be loaded using the `transformers` library
```python
from transformers import T5EncoderModel, T5Tokenizer
tokeniser = T5Tokenizer.from_pretrained("Exscientia/IgT5_unpaired", do_lower_case=False)
model = T5EncoderModel.from_pretrained("Exscientia/IgT5_unpaired")
```
The tokeniser is used to prepare batch inputs
```python
# single chain sequences
sequences = [
"EVVMTQSPASLSVSPGERATLSCRARASLGISTDLAWYQQRPGQAPRLLIYGASTRATGIPARFSGSGSGTEFTLTISSLQSEDSAVYYCQQYSNWPLTFGGGTKVEIK",
"ALTQPASVSGSPGQSITISCTGTSSDVGGYNYVSWYQQHPGKAPKLMIYDVSKRPSGVSNRFSGSKSGNTASLTISGLQSEDEADYYCNSLTSISTWVFGGGTKLTVL"
]
# The tokeniser expects input of the form ["E V V M...", "A L T Q..."]
sequences = [' '.join(sequence) for sequence in sequences]
tokens = tokeniser.batch_encode_plus(
sequences,
add_special_tokens=True,
pad_to_max_length=True,
return_tensors="pt",
return_special_tokens_mask=True
)
```
Note that the tokeniser adds a `</s>` token at the end of each sequence and pads using the `<pad>` token. For example a batch containing sequences `E V V M`, `A L` will be tokenised to `E V V M </s>` and `A L </s> <pad> <pad>`.
Sequence embeddings are generated by feeding tokens through the model
```python
output = model(
input_ids=tokens['input_ids'],
attention_mask=tokens['attention_mask']
)
residue_embeddings = output.last_hidden_state
```
To obtain a sequence representation, the residue tokens can be averaged over like so
```python
import torch
# mask special tokens before summing over embeddings
residue_embeddings[tokens["special_tokens_mask"] == 1] = 0
sequence_embeddings_sum = residue_embeddings.sum(1)
# average embedding by dividing sum by sequence lengths
sequence_lengths = torch.sum(tokens["special_tokens_mask"] == 0, dim=1)
sequence_embeddings = sequence_embeddings_sum / sequence_lengths.unsqueeze(1)
```
|