getting different embedding values for the same image when trying to generate embeddings from two different sagemaker instances of the same model.

#26
by rnekkanti - opened

Hi All,

I have this model deployed in multiple endpoints. Each endpoint is built has its own model.tar.gz file , but they all are built using the same base repo. When I try to generate the image embeddings for these endpoints, I would expect same embedding values generated when I pass the same image to these endpoints.
But instead I am seeing a difference in the values generated.

Anybody else face this issue?

Hi Rajendra,

I had a similar problem with embedding models when using different GPU instances and with batch sizes different from the ones used in the train.
Here are some references:

Hope this helps you.

Best regards,
Milutin

Sign up or log in to comment