chrisvoncsefalvay/bronchoscopy-diffusion-teacher-sd1.5-controlnet-depth-normals

Bronchoscopy Diffusion Teacher SD1.5 ControlNet Depth + Depth-Derived Pseudo-Normals

This repository contains a bronchoscopy appearance teacher for SynAIR-G-style airway rendering. It packages two SD1.5 ControlNet branches:

• depth-controlnet: fine-tuned from lllyasviel/control_v11f1p_sd15_depth on masked BREA-depth bronchoscopy controls.
• normal-controlnet: fine-tuned from lllyasviel/control_v11p_sd15_normalbae on masked pseudo-normal controls derived from the BREA depth gradients.

The source dataset does not contain native normal maps. The second branch uses pseudo-normal RGB images computed from local gradients in the inferred depth map, masked to the field of view.

The frozen base model is stable-diffusion-v1-5/stable-diffusion-v1-5. The training data is chrisvoncsefalvay/single-bronchoscopy-depth, a unified bronchoscopy dataset built from BI2K, BM-BronchoLC, and UAAL with masks, compact clinical descriptors, and inferred depth.

Intended Use

Use this model as a teacher for generating bronchoscopy-like RGB appearance from airway renderings. The expected inference setup is SD1.5 with MultiControlNet:

• compact clinical prompt;
• masked depth condition;
• masked depth-derived pseudo-normal condition, or rendered normals converted to the same convention for downstream mesh work;
• depth scale 0.45;
• normal scale 0.35;
• img2img strength 0.20;
• guidance scale 4.0;
• 16 denoising steps.

The model is intended for research, simulation, and synthetic-data generation. It is not a diagnostic model and should not be used for clinical decision-making.

Training Data

Source dataset: chrisvoncsefalvay/single-bronchoscopy-depth

Preparation summary:

• rows prepared: 12066;
• training split: 11518;
• validation split: 548;
• image size: 512x512;
• target masking: dataset FOV mask applied before training;
• depth control: masked, percentile-normalised BREA inverse-depth control; darker values indicate closer tissue and brighter values indicate farther/open lumen regions;
• normal control: pseudo-normal RGB derived from depth gradients, because the dataset does not provide native normals.

Selected Checkpoints

• depth branch: step 1000, validation loss 0.06977873452706262, copied into depth-controlnet;
• pseudo-normal branch: step 1000, validation loss 0.07001201336970553, copied into normal-controlnet.

Later training checkpoints were still completed as part of the scheduled full run, but these uploaded branches were selected by validation MSE on the prepared validation split.

Limitations

• Depth is inferred by BREA-Depth, not calibrated scanner geometry; use the uploaded depth-control polarity as dark-close/bright-far-open.
• Normals are derived from depth gradients, not measured or rendered surface normals.
• The source datasets are open bronchoscopy corpora with heterogeneous devices and annotations.
• Outputs can still contain colour noise, specular artefacts, or hallucinated pathology-like texture.
• This model should be validated on held-out rendered paths before being used as a mesh-texture teacher.

Loading Sketch

from diffusers import ControlNetModel, MultiControlNetModel, StableDiffusionControlNetImg2ImgPipeline
import torch

repo = "chrisvoncsefalvay/bronchoscopy-diffusion-teacher-sd1.5-controlnet-depth-normals"
depth = ControlNetModel.from_pretrained(repo, subfolder="depth-controlnet", torch_dtype=torch.float16)
normal = ControlNetModel.from_pretrained(repo, subfolder="normal-controlnet", torch_dtype=torch.float16)
controlnet = MultiControlNetModel([depth, normal])
pipe = StableDiffusionControlNetImg2ImgPipeline.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5",
    controlnet=controlnet,
    torch_dtype=torch.float16,
    safety_checker=None,
    requires_safety_checker=False,
).to("cuda")

See teacher_config.json for the exact default inference settings.