Can't reproduce results from simple examples on huggingface with popular transformer models?
Author: DGaffney
Pull request: no
Labels: name: bug

### System Info

Transformers 4.46.2, Python 3.11.11

### Who can help?

@amyeroberts, @qubvel

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

If I use the Inference API widget on the model card tab of this page, for the following case, (i.e, download this image, then enter as text cat,not_cat), I get a score of 0.53,0.47 cat/not_cat:
![image](https://github.com/user-attachments/assets/3872f990-9ca2-4668-8d35-d98b4eeb7d6f)


If I run what I would assume would be the identical code for computing the result, I get a wildly different score. What gives?

from PIL import Image
import requests

from transformers import CLIPProcessor, CLIPModel

model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(text=["cat", "not_cat"], images=image, return_tensors="pt", padding=True)

outputs = model(**inputs)
logits_per_image = outputs.logits_per_image # this is the image-text similarity score
logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities
Yields: tensor([[0.2322, 0.7678]], grad_fn=<SoftmaxBackward0>)

This is totally unworkable if this is the case? I have to be missing something, right? I can't be this off-base. This is on Transformers 4.46.2, but seems to happen with latest as well?

### Expected behavior

I would expect the results to be even somewhat close to what the inference API endpoint produces? What do I do to get them to be closer? What is wrong here?

Comments:
Yes this is due to the fact that the inference API uses the pipeline API behind the scenes, which by default uses the prefix "A photo of {class}" as seen [here](https://github.com/huggingface/transformers/blob/e5fd865ebae062b7cf03a81b8c6affeb39f30bec/src/transformers/pipelines/zero_shot_image_classification.py#L136).

This was also noted in https://github.com/huggingface/transformers/issues/30951 for which a disclaimer was added in the docs of SigLIP + the task page, but we could add a similar disclaimer in the docs of the pipeline.

Awesome ok this it vital info thank you for this. Adding it upfront maybe helpful for other folks I imagine. Also, what would a “canonical” negative class look like, if you had to advise on one style? “*not* a photo of a cat”?DevinOn Jan 4, 2025, at 10:41 AM, NielsRogge ***@***.***> wrote:﻿
Yes this is due to the fact that the inference API uses the pipeline API behind the scenes, which by default uses the prefix "A photo of {class}" as seen here.
This was also noted in #30951 for which a disclaimer was added in the docs, but we could add a similar disclaimer in the docs of the pipeline.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>