Repository: apple/coremltools

Issue #775: Output specifications support for Pytorch converter
Created: 2020-07-09T09:39:42Z
🌱 Describe your Feature Request
As the documentation explicitly point it out: "it is good practice to think about the interface to the model. This includes names and types of inputs and outputs".
When converting a model from PyTorch, one can just not explicitly set output type, name and size. Those specifications are also very useful when we need to scale / preprocess inputs and outputs.
Use case
In my case I need my model to output an imageType instead of a multiArray.  I'd like to provide specifications like this:
mlmodel = coremltools.convert(model,
                    inputs=[ct.ImageType(name="input", shape=(1, 3, 1024, 1024), scale=1/255)],
                    outputs=[ct.ImageType(name="output", shape=(1, 3, 1024, 1024), scale=255)])

but  get the following error : ValueError: outputs must not be specified for PyTorch
Alternative
The alternative to get the output specifications right is to manually change them through the mlmodel specs :

Specify multiArray shapes

spec = mlmodel.get_spec()
spec.description.output[0].type.multiArrayType.shape.append(3) 
spec.description.output[0].type.multiArrayType.shape.append(1024)
spec.description.output[0].type.multiArrayType.shape.append(1024)


Set output type to imageType

from coremltools.proto import FeatureTypes_pb2 as ft 

for output in spec.description.output: 
    
    # Change output type
    output.type.imageType.colorSpace = ft.ImageFeatureType.ColorSpace.Value('RGB') 
    channels, height, width = tuple(output.type.multiArrayType.shape)
    
    # Set image shape
    output.type.imageType.width = width 
    output.type.imageType.height = height


Convert back the modified specs into a new mlmodel

mlmodel_modif = coremltools.models.MLModel(spec)
But when calling predict on this new model i got the following error:
RuntimeError: {
    NSLocalizedDescription = "Batch or sequence image output is unsupported for image output 1223";
}

System environment :

coremltools version  4.0b1
Pytorch 1.5
MacOs 10.15.4
XCode 11.5
Python 3.7

Duplicate issue #934: CoreML Model spec - change output type to dictionary [Double : String]
Created: 2020-09-19T13:59:15Z
❓Question
Hello everybody,
For the past week I have been struggling to run inference on a classifier I built using Google's AutoML Vision tool.
At first I thought everything would go smoothly because Google allows to export a CoreML version of the final model. I assumed I would only need to use Apple's CoreML library to make it work. When I export the model Google provides a .mlmodel file and a dict.txt file with the classification labels. For the current model I have 100 labels.
This is my Swift code to run inference on the model.
private lazy var classificationRequest: VNCoreMLRequest = {
        do {

            let classificationModel = try VNCoreMLModel(for: NewGenusModel().model)

            let request = VNCoreMLRequest(model: classificationModel, completionHandler: { [weak self] request, error in
                self?.processClassifications(for: request, error: error)
            })

            request.imageCropAndScaleOption = .scaleFit
            return request
        }
        catch {
            fatalError("Error! Can't use Model.")
        }
    }()

    func classifyImage(receivedImage: UIImage) {

        let orientation = CGImagePropertyOrientation(rawValue: UInt32(receivedImage.imageOrientation.rawValue))

        if let image = CIImage(image: receivedImage) {
            DispatchQueue.global(qos: .userInitiated).async {

                let handler = VNImageRequestHandler(ciImage: image, orientation: orientation!)
                do {
                    try handler.perform([self.classificationRequest])
                }
                catch {
                    fatalError("Error classifying image!")
                }
            }
        }
    }

The problem started when I tried to pass a UIImage to run inference on the model. The input type of the original model was MultiArray (Float32 1 x 224 x 224 x 3). Using Coremltools library I was able to convert the input type to Image (Color 224 x 224) using Python.
This worked and here is my code:
import coremltools
import coremltools.proto.FeatureTypes_pb2 as ft

spec = coremltools.utils.load_spec("model.mlmodel")

input = spec.description.input[0]
input.type.imageType.colorSpace = ft.ImageFeatureType.RGB
input.type.imageType.height = 224
input.type.imageType.width = 224

coremltools.utils.save_spec(spec, "newModel.mlmodel")

My problem now is with the output type. I want to be able to access the confidence of the classification as well as the result label of the classification. Again using coremltools I was able to to access the output description and I got this.
name: "scores"
type {
  multiArrayType {
    dataType: FLOAT32
  }
}

Using the Netron app I found out that all CoreML models have a different specification from a few months ago.
format: Core ML v4
type: Neural Network
input: image_0 - float32[1,224,224,3]
output: float32

Before the specifications were:
format: Core ML v1
type: Neural Network Classifier
input: image_0 - image<RGB,224x224>
output: (scores_0 - map<string,float64>, classLabel - string)
This is what I need. This means something changed on the Google AutoML Vision side. All models were generated similarly and apparently with the same setting.

How can I convert the output specification of my model using Coremltools?
Thank you for your help!

Closing comment (2022-03-24T19:14:46Z): I believe this is the same issue as #775.
I'm going to close this issue as a duplicate. If this is not the same as #775, please explain how it's different and I'm happy to reopen this issue.