| To convert a ControlNet model: | |
| conda activate coreml_stable_diffusion | |
| python -m python_coreml_stable_diffusion.torch2coreml --convert-controlnet lllyasviel/control_v11p_sd15_softedge --model-version "runwayml/stable-diffusion-v1-5" --bundle-resources-for-swift-cli --attention-implementation SPLIT_EINSUM -o "./SoftEdge" | |
| python -m python_coreml_stable_diffusion.torch2coreml --convert-controlnet lllyasviel/control_v11p_sd15_softedge --model-version "runwayml/stable-diffusion-v1-5" --bundle-resources-for-swift-cli --attention-implementation ORIGINAL --latent-w 64 --latent-h 64 --compute-unit CPU_AND_GPU -o "./SoftEdge" | |
| Adjust --attention-implementation, --compute-unit, --latent-h, --latent-w, and -o to fit the type of model you want to end up with. These need to match the main model you plan to use them with. You can't mix sizes, but you can sometimes mix attention implementations - see the last note below abot this. | |
| The --convert-controlnet argument needs to point to the ControlNet model's specific repo. There is an index of these repos at: https://huggingface.co/lllyasviel. The name for the argument needs to be in the form of: lllyasviel/control_Version_Info_name | |
| The final converted model will be the .mlmodelc file in the Model_Name/Resources/controlnet folder. | |
| Move it to your ControlNet store folder, rename it as desired, and discard everything else. | |
| NOTES AND COMPLICATIONS | |
| The first time you do a conversion, you may need to log in to Hugging face using their command line log in tool and a token. They have instructions over there if you've never done it before. You need to be logged in there because the command will pull about 4 GB of checkpoint files needed to do model conversions. (That is what the argument --model-version "runwayml/stable-diffusion-v1-5" does. This will only happen once. The files will be saved in a hidden .cache folder in you home (User) directory. If you've been playing with model conversionss before, it is possible that you already have these files cached. | |
| Converted ControlNets that are SPLIT_EINSUM run extremely slowly using CPU and NE. They run at normal speed using CPU and GPU. But you can also just use an ORIGINAL ControlNet with a SPLIT_EINSUM main model, using CPU and GPU, and get normal speed as well. So there is probably no point in making a SPLIT_EINSUM ControlNet model. |