Package: doc-ufcn
Doc-UFCNThis Python 3 library contains a public implementation of Doc-UFCN, a fully convolutional network presented in the paperMultiple Document Datasets Pre-training Improves Text Line Detection With Deep Neural Networks. This library has been developed by the original authors fromTeklia.The model is designed to run various Document Layout Analysis (DLA) tasks like the text line detection or page segmentation.This library can be used to train a model, fine-tune a model or directly apply a trained Doc-UFCN model to document images.Getting startedTo use Doc-UFCN in your own scripts, install it using pip:pipinstalldoc-ufcnModel inferenceWith only a few lines of code, the trained model is loaded, applied to an image and the detected objects along with some visualizations are obtained.UsageTo apply Doc-UFCN to an image, one need to first add a few imports (optionally, set the logging config to make logs appear on stdout) and to load an image. Note that the image should be in RGB.importcv2importloggingimportsysfromdoc_ufcn.mainimportDocUFCNlogging.basicConfig(format="[%(levelname)s]%(message)s",stream=sys.stdout,level=logging.INFO)image=cv2.cvtColor(cv2.imread(IMAGE_PATH),cv2.COLOR_BGR2RGB)Then one can initialize and load a trained model with the parameters used during training. The number of classes should include the background that must have been put as the first channel during training. By default, the model is loaded in evaluation mode. To load it in training mode, usemode="train".nb_of_classes=2mean=[0,0,0]std=[1,1,1]input_size=768model_path="trained_model.pth"model=DocUFCN(nb_of_classes,input_size,'cpu')model.load(model_path,mean,std,mode="eval")To run the inference on a GPU, one can replacecpuby the name of the GPU. In the end, one can run the prediction:detected_polygons=model.predict(image)OutputWhen running inference on an image, the detected objects are returned as in the following example. The objects belonging to a class (except for the background class) are returned as a list containing the confidence score and the polygon coordinates of each object.{1:[{'confidence':0.99,'polygon':[(490,140),(490,1596),(2866,1598),(2870,140)]}...],...}In addition, one can directly retrieve the raw probabilities output by the model usingmodel.predict(image, raw_output=True). A tensor of size(nb_of_classes, height, width)is then returned along with the polygons and can be used for further processing.Lastly, two visualizations can be returned by the model:A mask of the detected objects, usingmask_output=Trueparameter;An overlap of the detected objects on the input image, usingoverlap_output=Trueparameter.By default, only the detected polygons are returned, to return the four outputs, one can use:detected_polygons,probabilities,mask,overlap=model.predict(image,raw_output=True,mask_output=True,overlap_output=True)ModelsWe provide various open-source models, stored onHuggingFaceand every model prefixed bydoc-ufcn-is supported. For example, to download ourgeneric page detection modeland load it, one can use:fromdoc_ufcnimportmodelsfromdoc_ufcn.mainimportDocUFCNmodel_path,parameters=models.download_model('generic-page')model=DocUFCN(len(parameters['classes']),parameters['input_size'],'cpu')model.load(model_path,parameters['mean'],parameters['std'])By default, the most recent version of the model will be downloaded. One can also use a specific version using the following line:model_path,parameters=models.download_model('generic-page',version="main")TrainingThe Doc-UFCN tool is split into three parts:The code to train the model on given datasets;The code to predict the segmentation of images according to the trained model;The code to evaluate the model based on the predictions.A csv configuration file allows to run a batch of experiments at once and also to train, predict or evaluate on combined datasets by only specifying the paths to the datasets folders.Preparing the environmentFirst of all, one needs an environment to run the three experiments presented before. Create a new environment and install the needed packages:pipinstalldoc-ufcn[training]Preparing the dataTo train and test the model, all the images and their annotations of a dataset should be in a folder following this hierarchy:.
├── dataset_name
│   ├── test
│   │   ├── images
│   │   └── labels_json
│   ├── train
│   │   ├── images
│   │   ├── labels
│   │   └── labels_json
│   └── val
│       ├── images
│       ├── labels
│       └── labels_json
├── ...The labels should be generated directly at the network input size (img_size) to avoid resizing (that can cause mergings of regions).
In addition, the evaluation is run over json files containing the original polygons coordinates that should be in thelabels_jsonfolders.Preparing the configuration filesexperiments_config.jsonDifferent files must be updated according to the task one want to run. Since we can run multiple experiments at once, the first configuration fileexperiments_config.jsonallows to specify the common parameters to use for all the experiments:ParameterDescriptionDefault valueclasses_namesList with the names of the classes /must be in the same orderas the colors defined inclasses_colorsfield["background", "text_line"]classes_colorsList with the color codes of the classes[[0, 0, 0]], [0, 0, 255]]img_sizeNetwork input size /must be the sameas the one used during the label generation768no_of_epochsNumber of epochs to train the models200batch_sizeSize of batchs to use during trainingNoneno_of_paramsMaximum number of parameters supported by the CPU/GPUNonebin_sizeSize between two groups of images.20min_ccThreshold to use when removing of small connected components50save_imageList with the sets ["train", "val", "test"] for which we want to save the predicted masks["val", "test"]use_ampWhether to use Automatic Mixed Precision during trainingfalseThe background classmustalways be defined at the first position in theclasses_namesandclasses_colorsfields.Automatic Mixed Precision allows to speed up the training while using less memory (possibility to increase the batch size). Either the batch size or the number of parameters should be defined.experiments.csvIn the root directory, one has to create anexperiments.csvfile (seeexample_experiments.csv). It contains the experiments names as well as the paths to the datasets and parameters used to continue a training or to fine-tune a model.ParameterDescriptionDefault value / exampleexperiment_nameName of the experimentstepsList of steps to run ["normalization_params", "train", "prediction", "evaluation"]normalization_params;train;prediction;evaluationtrainPaths to the training datasetspath_to_dataset1;path_to_dataset_2valPaths to the validation datasetspath_to_dataset1;path_to_dataset_2testPaths to the evaluation datasetspath_to_dataset1;path_to_dataset_2restore_modelName of a saved model to resume or fine-tune a trainingsame_classesWhether the classes of the current experiment are the same as those of the model to resumeTruelossWhether to use an initial loss (initial) or the best (best) saved loss of the restored modelinitialNote: All the steps are dependent, e.g to run the"prediction"step, oneneedsthe results of the"normalization_params"and"train"steps.ExampleTheexample_experiments.csvfile shows an example on how to build the experiments csv file.experiment_namestepstrainvaltestrestore_modelsame_classeslossexp1normalization_params;train;prediction;evaluation~/data/DLA/dataset1;~/data/dataset2~/data/DLA/dataset1;~/data/dataset2~/data/DLA/dataset3;~/data/dataset2exp1prediction;evaluation~/data/DLA/dataset4exp1train;prediction;evaluation~/data/DLA/dataset4~/data/DLA/dataset4~/data/DLA/dataset4last_model.pthinitialexp2normalization_params;train;prediction;evaluation~/data/DLA/dataset5~/data/DLA/dataset5~/data/DLA/dataset5The first line will start a standard training on two datasets (dataset1 and dataset2) and will be tested also on two datasets (dataset2 and dataset3).The second line will use the model trained during the first experiment (same experiment_name) and only test it on another dataset (dataset4).The third line will also use the first trained model (same experiment_name) but will fine-tune it on dataset4.restore_modelindicates which model to fine-tune andlossindicates that the loss should be initialized (datasets used to fine-tune are not the same as the one used for first training).The last line will run a standard new training on dataset5.Start an experimentTo start the experiment:$./run_dla_experiment.sh-cexperiments.csvThere's a way to be notified in slack when training has finished (successfully or not):Create a webhook herehttps://my.slack.com/services/new/incoming-webhook/;Save the webhook key into~/.notify-slack-cfg(looks like:T02TKKSAX/B246MJ6HX/WXt2BWPfNhSKxdoFNFblczW9)Make sure that the notifier is working:pythontools/notify-slack.py"WARN: notifier works"The slack notification is used by default;To start the experiment without this slack notification run:$./run_dla_experiment.sh-cexperiments.csv-sfalseFollow a trainingTensorboardOne can see the training progress using Tensorboard. In a new terminal:$tensorboard--logdir./runs/experiment_nameThe model and the useful file for visualization are stored in./runs/experiment_name.MLflowMLflow logging is also available in Doc-UFCN. Information about the instance and the experiment need to be specified in the configuration fileexperiments_config.json, under the keymlflow.#experiments_config.json{..."mlflow":{"experiment_id":...,"run_name":null,"tracking_uri":...,"s3_endpoint_url":...,"aws_access_key_id":...,"aws_secret_access_key":...}}experiment_id: ID of the MLflow experiment where the run will be recorded,run_name: Optional name of the created run,tracking_uri: URL towards the MLflow instance, seeMLFLOW_TRACKING_URI,s3_endpoint_url: URL towards the MLflow instance's storage, seeMLFLOW_S3_ENDPOINT_URL,aws_access_key_idandaws_secret_access_key:AWS credentialsif the storage is Amazon S3 or Amazon S3-compatible. Only needed when logging artifacts.Result of an experimentThe logs of an experiment are saved inDLA_train.logfile.Once a model has been trained, it can be found in./runs/experiment_name/model.pth.The predictions are in./runs/experiment_name/predictions.The evaluation results are in./runs/experiment_name/results.Resume a trainingTo resume a training, just by adding epochs for example, one just has to:Remove the"normalization_params"step;Indicate the name of the model to resume inrestore_modelparameter;Set the value of thelossat"best".Model fine-tuningSeveral models are available onHuggingFacefor inference but they can also be used as pre-trained weights. It is then possible to fine-tune them on a new task and/or new data.For this, it is necessary to follow the same data preparation steps as above, as well as the specification of the configuration files. To fine-tune a model, one needs to:Remove the"normalization_params"step;Indicate the name of the model to resume inrestore_modelparameter. If the name contains a.pthextension, the corresponding local file will be retrieved. Otherwise, to retrieve a model from HuggingFace, one just needs to put the name of the model starting withdoc-ufcn-;Set the value of thelossat"initial";Set thesame_classesparameter toFalseif the classes used for training the new model are different from the classes used during the pre-training.Once these parameters have been updated, the training can be started and followed as described above.HuggingFace appYou may easily create and deploy a web application to demo the prediction of your Doc-UFCN models. This application can be easily integrated in aHuggingFace space: an example of aHuggingFace Spacegenerated using this code is availablehere.To use HuggingFace app, it is necessary to install gradio using pip:pipinstallgradioThe code was last tested withgradio==3.18.0.An example is available in thehuggingface/folder. You need to create a YAML configuration file with the following parameters:ParameterDescriptionDefault valuetitleTitle of the app (supports Markdown)RequireddescriptionDescription of the app (supports Markdown)RequiredexamplesPaths towards the image examplesRequiredmodelsList of modelsRequiredIn models parameter, for each model, fill in the following parameters:ParameterDescriptionDefault valuemodel_nameName of the modelRequiredtitleTitle of the model (supports Markdown)RequireddescriptionDescription of the model (supports Markdown)Requiredclasses_colorsList with the colors of the classesRequiredA ready-to-use configuration file is available inhuggingface/config.yaml. You may update it to your needs.Once the configuration file is ready, just run the following command:$python3hugging_face/app.py--confighugging_face/config.yamlIf the--publicargument is specified, the script will generate a public, shareable link that you can send to anyone. More information about public links on gradio is available in thegradio documentation.To generate the shareable link, use the following command:$python3hugging_face/app.py--confighugging_face/config.yaml--publicThe model's predictions will be shown on the application in JSON format:A dictionary mapping an object idx (starting from 1) to a dictionary describing the detected object:polygonkey : list, the coordinates of the points of the polygon,confidencekey : float, confidence of the model,channelkey : str, the name of the predicted class.[
  {
    "polygon": [[],..,[]]
    ,
    "confidence": 0.91
    ,
    "channel": "text_line"
  },
  ..
]Cite us!If you want to cite us in one of your works, please use the following citation.@inproceedings{boillet2020,
    author ={Boillet, Mélodie and Kermorvant, Christopher and Paquet, Thierry},
    title ={{Multiple Document Datasets Pre-training Improves Text Line Detection With
              Deep Neural Networks}},
    booktitle ={2020 25th International Conference on Pattern Recognition (ICPR)},
    year ={2021},
    month = Jan,
    pages ={2134-2141},
    doi ={10.1109/ICPR48806.2021.9412447}}LicenseThis library is under the 3-Clause BSD License.ContactIf you have any questions or suggestions, please contactMélodie Boillet.