VioletXF/manganinja-coreml

MangaNinja Core ML

Downloadable Core ML components for reference-based manga coloring, usable by any app. Converted from Ali-ViLab's MangaNinja using Apple's coremltools. This is an unofficial, experimental conversion, not a model trained by VioletXF.

All seven components are converted and pass Apple's Core ML compiler. Image quality, numerical parity, device memory and inference speed have not been validated. This port uses reference-only conditioning; interactive point guidance is omitted. Do not interpret compilation as evidence of output quality.

The model assets total 4,941,942,005 bytes (4.94 GB / 4.60 GiB). Weights are FP16; public tensor inputs and outputs are Float32. Models target iOS 18+ / corresponding Core ML support on Apple silicon. A consumer app may require a newer OS. Installation compiles packages locally and needs additional temporary disk space; runtime memory is separate from download size.

Component	Download bytes
lineart.mlpackage	9,137,771
vae_encoder.mlpackage	68,470,412
vae_decoder.mlpackage	99,164,266
clip.mlpackage	608,239,478
reference.mlpackage	1,714,063,009
control.mlpackage	722,926,619
denoiser.mlpackage	1,719,891,909

manifest.json lists every required package file plus contract.json, with byte counts and SHA-256 digests. Resolve this repository's commit SHA before downloading and fetch all files from that immutable revision. Do not mix packages or contracts from different releases. No app bundle, credentials, comic images or reference images are included.

Download

hf download VioletXF/manganinja-coreml --local-dir ./manganinja-coreml

Compile each .mlpackage with MLModel.compileModel(at:), then load the resulting .mlmodelc. There is no single image-to-image model: the host must execute the components in the order below. Core ML deployment tools may generate a larger installed representation. Autocomic is one consumer; these packages do not depend on Autocomic identifiers or its server.

Tensor contract and inference

All image tensors use RGB NCHW batch 1. Resize the target and reference to 512×512 for this fixed-resolution contract. A host can combine generated chroma with the original page to preserve its high-resolution ink and aspect ratio.

1. CLIP: normalize a 224×224 RGB reference with clipMean and clipStd from contract.json; input image [1,3,224,224], output embedding [1,1,768]. Optionally rank candidate colored references against the target's CLIP embedding using cosine similarity. Feed the selected reference embedding to the subsequent networks.
2. VAE encoder: input reference image [1,3,512,512] in [-1,1] and standard Gaussian noise [1,4,64,64]; output latent [1,4,64,64] is already scaled by 0.18215.
3. Reference UNet: inputs latent and embedding; emits bank_0 … bank_15. Shapes and corresponding upstream module names are recorded in contract.json. Timestep zero is embedded in this graph. Retain these features and release the reference network before denoising.
4. Line-art extractor: input target image [1,3,512,512] in [0,1]; output lineart [1,3,512,512] is already inverted, thresholded and normalized to [-1,1]. Do not VAE-encode this conditioning image.
5. Start target latent [1,4,64,64] from standard Gaussian noise. For each of the 20 contract timesteps, execute two sequential branches: unconditional and reference-conditioned. ControlNet inputs are latent, timestep [1], embedding, and lineart. It emits residual_0 … residual_12 (12 down blocks plus mid block).
6. Denoiser: pass those same latent/timestep/embedding inputs, all 13 residuals, all 16 reference banks, and conditioned [1] (0 for unconditional, 1 for reference). contract.unconditional contains the first empty-prompt token's 768 values for the unconditional embedding. The graph masks reference attention when conditioned=0. Output is noise [1,4,64,64].
7. Combine branches as u + 4 * (r - u). This is the upstream two-guidance average with identical reference/point branches when point guidance is omitted, rather than a claim of equivalence to interactive PointNet inference.
8. Deterministic DDIM (eta=0): for timestep t, a = alphas[t]; p = alphas[t-50] or finalAlpha when t-50<0. Compute x0 = (latent - sqrt(1-a)*noise)/sqrt(a) then latent = sqrt(p)*x0 + sqrt(1-p)*noise. Timesteps are 951, 901, …, 1. No additional latent scaling or scheduler clipping is required.
9. VAE decoder: input final latent; output image [1,3,512,512] in [0,1]. The graph already divides by 0.18215 and performs output normalization/clamping.

Keep only the networks needed for the current stage. Cache CLIP embeddings/reference banks if appropriate; avoid loading all seven networks together. Diffusion remains expensive on a phone or tablet and this release does not establish a safe minimum device memory budget.

Reproduce the conversion

Pinned source: ali-vilab/MangaNinjia@6363c81aaedab0a435d18cba9209a7e842881ad7. Checkpoint: Johanan0528/MangaNinjia@4e6237c1d22415272bf98426616fe478cd3202a0 (note the spelling MangaNinjia). All supporting repository revisions, source byte counts and hashes are in conversion/sources.json; exact Python dependencies are in conversion/requirements.txt.

python3.11 -m venv .venv
. .venv/bin/activate
pip install -r conversion/requirements.txt
git clone https://github.com/ali-vilab/MangaNinjia upstream
git -C upstream checkout 6363c81aaedab0a435d18cba9209a7e842881ad7
for component in lineart vae_encoder vae_decoder clip reference control denoiser; do
  python conversion/convert.py "$component" --upstream upstream --work work
done
python conversion/package_release.py --models work/coreml --destination release

Conversion uses torch.jit tracing (which executes the network) with check_trace disabled, then Core ML MLProgram FP16 conversion. The script migrates legacy VAE attention keys and validates/removes deterministic CLIP positional buffers before strict weight loading. Explicit attention-bank tensors replace Python hooks. No quantization beyond FP16 is applied. Export component-by-component to reduce working disk and RAM; optional --release-source removes only that component's task-owned downloaded checkpoint after successful export. Running it after VAE encoding causes decoding to download the shared checkpoint again, so normally use it only after VAE decoding.

Licensing

Conversion does not change upstream permissions. Do not assume this collection has a single unrestricted commercial license.

• MangaNinja source declares CC BY-NC 4.0. Its official Hugging Face checkpoint card labels Apache 2.0; this discrepancy remains unresolved. Preserve attribution to the original authors and review the upstream terms before use or redistribution.
• Stable Diffusion v1.5 VAE/configuration: CreativeML OpenRAIL-M. Source: stable-diffusion-v1-5/stable-diffusion-v1-5.
• CLIP ViT-L/14: OpenAI MIT notice, sourced from openai/CLIP.
• Line-art extractor: Caroline Chan MIT notice, retained by MangaNinja; weights from lllyasviel/Annotators.
• ControlNet architecture/configuration from lllyasviel/control_v11p_sd15_lineart, with MangaNinja-trained ControlNet weights.

Model behavior can invent or change colors. This release is for experimentation; it does not promise faithful character colors or cross-page consistency.