README.md

---
license: apache-2.0
library_name: timm
pipeline_tag: image-classification
---
# RedRocket Joint Tagger Project

## JTP-1.1: PILOT2

**NEW 2024-07-02**: This model is an incremental improvement over PILOT.  It features a gated model head and some changes were made to training hyperparameters.  Inference code will need some changes to account for the architecture changes.  Both models will be hosted on HF spaces until more testing between the two can be done.

## JTP-1: PILOT

<!-- Provide a quick summary of what the model is/does. -->

This model is a multi-label classifier model designed and trained by RedRocket for use on furry images, using E621 tags.

PILOT is the first model of this series.  It is trained on **over 9000** tags -- tags were selected with the criteria of being e621 tags with more than 500 occurrences, that are not artist or character tags.

## Model Details

### Model Description

<!-- Provide a longer summary of what this model is. -->

- **Developed by:** RedRocket
- **Compute power provided by:** Minotoro and Frosting.ai (thank you)
- **Model type:** Multi-label classifier
- **License:** Apache 2.0
- **Finetuned from model:** [ViT-SO400M-14-SigLIP](https://huggingface.co/timm/ViT-SO400M-14-SigLIP)

### Model Sources

<!-- Provide the basic links for the model. -->

- **Repository:** Here!
- **Demo (PILOT):** https://huggingface.co/spaces/RedRocket/JointTaggerProject-Inference
- **Demo (PILOT2):** https://huggingface.co/spaces/RedRocket/JointTaggerProject-Inference-Beta

## Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

### Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

Use it to tag furry images.

### Downstream Use

<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->

Use it to train a text-to-image model on synthetic tags.  It might just be good enough for that by now.  Should wait for more extensive evaluation though to be safe.  I would suggest *supplementing* tags rather than replacing them.

### Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

Use it to tag non-furry images.  It might not work terribly well but it might also work surprisingly well!  Great entertainment value either way.

## Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

This model may contain biases.  Tags that are poorly tagged in the original data may be weakly predicted by the classifier, for instance.  Tags that are very commonly present alongside other tags may be hallucinated.

The model has been known to show biases towards English defaultness, specifically outputting the tag `english_text` on text that does not belong to any specific language, for example Arabic numerals and onomatopoeia.

### Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Check at least some portion of your outputs manually, preferably a diverse sample, to verify the correctness of its outputs, and apply a different threshold if it seems necessary.

## How to Get Started with the Model

Use the included code to launch a Gradio demo for playing with the model.  We recommend a threshold of 0.2 for starting out.  Validation stats during training showed a Bookmaker's Informedness of 0.725 at this value (this means that the model is that much better at guessing tags than random guessing).  Manual evaluation seems to suggest that a large portion of the gap between that value and 1 is likely to be due to false negatives from the dataset.

## Training Details

### Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

The model was trained on a roughly 4 million image subset of e621.  No dataset filtering was applied.

Loss weighting was informed by a Bayesian prior model trained on a set of tags from non-deleted post tag strings from an e621 database dump.

### Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

Images go in, logits come out.  You can't explain that.

Loss objective is `F.binary_cross_entropy_with_logits(output, target, torch.maximum(target, 1.0 - prior_output))`.

#### Preprocessing [optional]

Image preprocessing should be done in the following order:
1. Resize image to **longest side** 384.
2. torchvision.transforms.ToTensor()
3. Composite the alpha channel, if present, with 50% gray.
4. Normalize to mean 0.5 and std 0.5 (changing the range from (0, 1) to (-1, 1))
5. Pad image to 384x384 (torchvision.transforms.CenterCrop((384,384)) will do this)


#### Training Hyperparameters

- **Training regime:** Model was trained for 4 epochs on a batch size of 512, using Schedule Free Adam

#### Speeds, Sizes, Times [optional]

<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->

[More Information Needed]

## Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

### Testing Data, Factors & Metrics

#### Testing Data

<!-- This should link to a Dataset Card if possible. -->

A validation set of approximately 128,000 images was reserved for validation testing.

#### Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

Bookmaker's Informedness at thresholds of 0.2, 0.3, and 0.5, as well as loss on the validation set were monitored throughout training.  Training was terminated at 5 epochs, and the checkpoint with the lowest validation loss (end of 4th epoch) was taken.

### Results

It seems to tag furry images fairly well.