Manga-Text-Segmentation-Test-1

Segmentation of text in manga.

Model

The model is available in HuggingFace. The model has a UNet 3+ decoder with a DINOv3 ViT-L/16 semantic backbone, an EfficientNetV2-M local-detail backbone, and a Fidelity-Aware Projection Module. The DINOv3 encoder uses LoRA adaptation and the decoder/local-detail path is fully trainable. The model is implemented in Keras 3 and produces one semantic text mask for the page.

Example

Input: Output: ^{(source: manga109, © Tadashi Satō)}

Dataset

The supervised checkpoint was trained on 450 Manga109 pages with pixel-level anime2 annotations. Black and pink annotation pixels are treated as the same text foreground class. For the original Manga109 images, please refer to Manga 109 website. The original Manga109 mask-dataset reference is available at zenodo.

Citations:

@InProceedings{10.1007/978-3-030-67070-2_38,
author="Del Gobbo, Juli{\'a}n
and Matuk Herrera, Rosana",
editor="Bartoli, Adrien
and Fusiello, Andrea",
title="Unconstrained Text Detection in Manga: A New Dataset and Baseline",
booktitle="Computer Vision -- ECCV 2020 Workshops",
year="2020",
publisher="Springer International Publishing",
address="Cham",
pages="629--646",
abstract="The detection and recognition of unconstrained text is an open problem in research. Text in comic books has unusual styles that raise many challenges for text detection. This work aims to binarize text in a comic genre with highly sophisticated text styles: Japanese manga. To overcome the lack of a manga dataset with text annotations at a pixel level, we create our own. To improve the evaluation and search of an optimal model, in addition to standard metrics in binarization, we implement other special metrics. Using these resources, we designed and evaluated a deep network model, outperforming current methods for text binarization in manga in most metrics.",
isbn="978-3-030-67070-2"
}

@dataset{segmentation_manga_dataset,
  author       = {julian del gobbo and
                  Rosana Matuk Herrera},
  title        = {{Mask Dataset for: Unconstrained Text Detection in 
                   Manga: a New Dataset and Baseline}},
  month        = feb,
  year         = 2021,
  publisher    = {Zenodo},
  version      = {1.0},
  doi          = {10.5281/zenodo.4511796},
  url          = {https://doi.org/10.5281/zenodo.4511796}
}