Torchserve docker image, cant load model saved on m1 mac. Works locally

maverick_coop: <p>my handler is:</p>
<pre><code class="lang-auto">
class CustomHandler(BaseHandler):

    def __init__(self):
        self._context = None
        self.initialized = False
        self.model = None
        self.device = None

    def initialize(self, context):
        self.device = 'cpu'
        self.manifest = context.manifest
        properties = context.system_properties
        serialized_file = self.manifest["model"]["serializedFile"]
        model_dir = properties.get("model_dir")
        model_pt_path = os.path.join(model_dir, serialized_file)
        
        model_file = self.manifest["model"].get("modelFile", "")
        module = importlib.import_module(model_file.split(".")[0])
        model_class_definitions = list_classes_from_module(module)
        model_class = model_class_definitions[0]

        self.model = model_class()
        self.model.load_state_dict(torch.load(model_pt_path, map_location=torch.device('cpu')))
        self.model = self.model.to(torch.device('cpu'))
        self.model.eval()
        self.initialized = True

    def preprocess(self, data):
        preprocessed_data = []
        for row in data:
            img = row['image'].decode()
            decoded_img = self.decode_img(img)
            pill_img = self.convert(decoded_img)
            transformed_img = self.transform(28, 28)(pill_img)
            preprocessed_data.append(transformed_img)
        return torch.stack(preprocessed_data)

    def postprocess(self, data):
        return data.argmax(dim=1).tolist()

    def decode_img(self, image_file):
        return base64.b64decode(image_file)
    
    def convert(self, image_data):
        return Image.open(io.BytesIO(image_data)).convert('L')
    
    def transform(self, height, width):
        return transforms.Compose([    
            transforms.Resize((height, width)),
            transforms.ToTensor(),
            transforms.Normalize(mean=[0.5],
                                 std=[0.5])
        ])
</code></pre>
<p>and i get an error:</p>
<pre><code class="lang-auto">ensor = torch.from_numpy(data).to(dtype=dtype, device=device)
2022-10-25T05:12:08,616 [INFO ] W-9001-shape_classifier_1.0-stdout MODEL_LOG - RuntimeError: Expected one of cpu, cuda, xpu, mkldnn, opengl, opencl, ideep, hip, ve, ort, mlc, xla, lazy, vulkan, meta, hpu device type at start of device string: mps
</code></pre>

maverick_coop: <p>Dockerfile</p>
<pre><code class="lang-auto">FROM pytorch/torchserve:0.6.0-cpu

USER root
RUN apt-get update
RUN apt-get install -y libgl1-mesa-glx
RUN apt-get install -y libglib2.0-0

# COPY ./requirements.txt /home/model-server/requirements.txt
COPY ./models/best_model/best_model.pth /home/model-server/best_model.pth
COPY ./ ./
USER model-server

# RUN pip install --no-cache-dir --upgrade -r requirements.txt
RUN pip install Pillow==9.0.0

RUN torch-model-archiver \ 
    --model-name shape_classifier \
    --version 1.0 \
    --model-file src/model/classifier.py \
    --serialized-file models/best_model/best_model.pth \
    --export-path model-store \
    --handler src/handler/model_handler.py \
    --force

CMD ["torchserve", "--start", "--model-store", "model-store", "--models", "all"]

</code></pre>

marksaroufim: <p>We haven’t tested out Torchserve on M1, mostly because noone on the dev team currently has an M1. I should be getting one around December and we can revisit this sometime then</p>

lsmith77: <p>Any progress here or hints how to get TorchServe running on Apple Silicon chips?</p>

marksaroufim: <p>Answered here <a href="https://github.com/pytorch/serve/issues/1678" class="inline-onebox">Torchserve support mac M1 chip · Issue #1678 · pytorch/serve · GitHub</a></p>