How to get Image embedding using Fuyu

#37
by oaishi - opened

Hi, Thanks so much for sharing the pretrained model. The results look amazing. I wanted to use Fuyu to get the embedding of chart images and compare them to each other. Anyone has any tips how to achieve it?

Hi, I do not think that is good idea. You have no image embeddings or encoder to begin with, I guess your best bet is just put in the image without the text prompt and try to compare the features in some layer below the head. As far as I know this release is also not the model finetuned for inference on the chart images.

Thanks, @ludeksvoboda ! I appreciate your feedback!

Sign up or log in to comment