|
--- |
|
license: apache-2.0 |
|
--- |
|
 |
|
|
|
# Multiformer |
|
|
|
Multiformer is a multi-task vision transformer architecture designed to provide strong perception capabilities with a nimble and lightweight architecture. |
|
|
|
- [Publication](https://natecibik.medium.com/multiformer-51b81df826b7) |
|
- [Training Report](https://api.wandb.ai/links/indezera/fynqkt8r) |
|
- [GitHub](https://github.com/FoamoftheSea/shift-experiments) |
|
|
|
This model uses a custom branch of the transformers library, which can be installed easily using the instructions below. |
|
|
|
For training and evaluation, a custom MultitaskTrainer class is used that can handle complex nested losses and successfully log them to wandb. |
|
|
|
A [training/eval](https://github.com/FoamoftheSea/shift-experiments/blob/main/scripts/model_train_eval/train_multiformer.py) and [inference](https://github.com/FoamoftheSea/shift-experiments/blob/main/scripts/inference/multiformer_inference.py) script are both available in the project repository. |
|
|
|
## Setup Instructions |
|
|
|
1. Open a terminal and navigate to your root folder, then run |
|
```shell |
|
git clone https://github.com/FoamoftheSea/shift-experiments.git |
|
``` |
|
2. Follow the setup instructions for your operating system found in the [README](https://github.com/FoamoftheSea/shift-experiments/blob/main/README.md) |
|
|
|
## Quick Load Multiformer-M1 |
|
|
|
You should now be able to run the following code to load a Multiformer-M1 with pretrained weights: |
|
|
|
```python |
|
from transformers import AutoModel |
|
|
|
multiformer = AutoModel.from_pretrained("FoamoftheSea/multiformer-m1) |
|
``` |
