BlockDetail/PartialSketchControlNet

Model Card Partial Sketch ControlNet

Prior sketch-to-image ControlNet models are trained on pairs of high-fidelity images and their edge maps. In this configuration, a region of an edge map with no edges corresponds to a region of an original image with no detail. As a result, ControlNet-generated images typically contain objects only in regions where control sketches contain strokes. We seek a solution that does not treat empty space in a sketch as intended whitespace, but as a signal that a user has yet to specify the region's contents. We look to the system to produce images that contain suggestions for how the region could plausibly be filled. To obtain this behavior, our approach is to produce training data that pairs of high-fidelity images with corresponding partial sketches at various levels of completion.

• Repository: [https://github.com/BlockDetail/Block-and-Detail]
• Demo: [https://huggingface.co/spaces/BlockDetail/Block-and-Detail]

Uses

This model can be loaded directly with diffusers.ControlNetModel.from_pretrained.