Package: pydeepimagej
PydeepImageJPython code to export trained models into theBioImage Model Zooformat and read them in Fiji & ImageJ using the deepImageJ plugin.Creates a configuration class in Python with all the information about the trained model needed for its correct use in Fiji & ImageJ.Includes the metadata of an example image.Includes all expected results and needed pre / post-processing routines.Creates basic cover images for the model card in the BioImage Model Zoo.Creates de the version 0.3.2 of theBioImage Model Zoo specification file:model.yamlSeedeepImageJ webpagefor more information about how to use the model in Fiji & ImageJ.Requirements & InstallationPyDeepImageJ requires Python 3 to run.TensorFlow: It runs using the local installation of TensorFlow, i.e. the one corresponding to the trained model. However, deepImageJ is only compatible with TensorFlow versions <= 2.2.1.To install pydeepImageJ either clone this repository or use PyPi viapip:$pipinstallpydeepimagejor$gitclonehttps://github.com/deepimagej/pydeepimagej.git
$cdpydeepimagej
$pipinstall.Reference:Gómez-de-Mariscal, E., García-López-de-Haro, C., Ouyang, W., Donati, L., Lundberg, L., Unser, M., Muñoz-Barrutia, A. and Sage, D., "DeepImageJ: A user-friendly environment to run deep learning models in ImageJ", Nat Methods 18, 1192–1195 (2021).https://doi.org/10.1038/s41592-021-01262-9Read the paper online with this link:rdcu.be/cyG3KBioengineering and Aerospace Engineering Department, Universidad Carlos III de Madrid, SpainScience for Life Laboratory, KTH – Royal Institute of Technology, Stockholm, SwedenBiomedical Imaging Group, Ecole polytechnique federale de Lausanne (EPFL), SwitzerlandCorresponding authors:mamunozb@ing.uc3m.es,daniel.sage@epfl.chCopyright © 2019. Universidad Carlos III, Madrid; Spain and EPFL, Lausanne, Switzerland.How to cite@article{gomez2021deepimagej,title={DeepImageJ: A user-friendly environment to run deep learning models in ImageJ},author={G{\'o}mez-de-Mariscal, Estibaliz and Garc{\'i}a-L{\'o}pez-de-Haro, Carlos and Ouyang, Wei and Donati, Laur{\`e}ne and Lundberg, Emma and Unser, Michael and Mu{\~{n}}oz-Barrutia, Arrate and Sage, Daniel},journal={Nature Methods},year={2021},volume={18},number={10},pages={1192-1195},URL={https://doi.org/10.1038/s41592-021-01262-9},doi={10.1038/s41592-021-01262-9}}LicenseBSD 2-Clause LicenseExample of how to use itTry a Jupyter notebook in Google Colaboratory:Otherwise, follow the next steps:Letmodelbe a Keras or TensorFlow trained model. Initialize the configuration class with the trained modelmodelfrompydeepimagej.yamlimportBioImageModelZooConfig# MinimumSize needs to be given as it cannot be always estimated. See Additional commands for hints.dij_config=BioImageModelZooConfig(model,MinimumSize)Update model informationdij_config.Name='My trained model v0.1'dij_config.Description='Brief description of the task to perform by the trained model'dij_config.Authors.Names=['First author','Second Author','Third Author who create the configuration specifications']dij_config.Authors.Affiliations=['First author affiliation','Second author affiliation','Third author affiliation']dij_config.References=['Gómez-de-Mariscal, E., García-López-de-Haro, C. et al., bioRxiv 2019','Second citation']dij_config.DOI=['https://doi.org/10.1101/799270','second citation doi']dij_config.GitHub='https://github.com/deepimagej/pydeepimagej'dij_config.License='BSD-3'dij_config.Documentation='https://useful_documentation.pdf'dij_config.Tags=['deepimagej','segmentation','Fiji','microscopy']dij_config.CoverImage=['./input.png','./output.png']dij_config.Framework='TensorFlow'# Parent model in the BioImage Model Zoo whose trained weights were used as pretrained weights.dij_config.Parent="https://bioimage.io/#/?id=deepimagej%2FUNet2DPancreaticSegmentation"1. Pre & post-processing specification.1.1. Specify the pre&post-processing steps following the BioImage Model Zoo specifications.If the pre-processing or the post-processing can be defined using the implementations defined at
, then it is also possible to specify them with some code:dij_config.add_bioimageio_spec('pre-processing','scale_range',mode='per_sample',axes='xyzc',min_percentile=0,max_percentile=100)dij_config.add_bioimageio_spec('post-processing','binarize',threshold=threshold)TheBioImageModelZooConfigclass will include as many steps as times the previous functions are called. For example:# Make sure that there's no pre-processing specified.dij_config.BioImage_Preprocessing=Nonedij_config.add_bioimageio_spec('pre-processing','scale_range',mode='per_sample',axes='xyzc',min_percentile=min_percentile,max_percentile=max_percentile)dij_config.add_bioimageio_spec('pre-processing','scale_linear',gain=255,offset=0,axes='xy')dij_config.BioImage_Preprocessing:
[{'scale_range': {'kwargs': {'axes': 'xyzc',
  'max_percentile': 100,
  'min_percentile': 0,
  'mode': 'per_sample'}}},
 {'scale_range': {'kwargs': {'axes': 'xy', 'gain': 255, 'offset': 0}}}]The same applies for the post-processing:dij_config.BioImage_Postprocessing=Nonedij_config.add_bioimageio_spec('post-processing','scale_range',mode='per_sample',axes='xyzc',min_percentile=0,max_percentile=100)dij_config.add_bioimageio_spec('post-processing','scale_linear',gain=255,offset=0,axes='xy')dij_config.add_bioimageio_spec('post-processing','binarize',threshold=threshold)dij_config.BioImage_Postprocessing:
[{'scale_range': {'kwargs': {'axes': 'xyzc',
  'max_percentile': 100,
  'min_percentile': 0,
  'mode': 'per_sample'}}},
 {'scale_range': {'kwargs': {'axes': 'xy', 'gain': 255, 'offset': 0}}},
 {'binarize': {'kwargs': {'threshold': 0.5}}}]1.2. Prepare an ImageJ pre/post-processing macro.You may need to preprocess the input image before the inference. Some ImageJ macro routines can be downloaded fromhereand included in the model specifications. Note that ImageJ macros are text files so it is easy to modify them inside a Python script (see an example). To add any ImageJ macro code we need to runadd_preprocessing(local_path_to_the_macro_file, 'name_to_store_the_macro_in_the_bundled_model'):path_preprocessing="PercentileNormalization.ijm"# Download the macro fileurllib.request.urlretrieve("https://raw.githubusercontent.com/deepimagej/imagej-macros/master/PercentileNormalization.ijm",path_preprocessing)# Include it in the configuration classdij_config.add_preprocessing(path_preprocessing,"preprocessing")The same holds for the postprocessing.path_postprocessing="8bitBinarize.ijm"urllib.request.urlretrieve("https://raw.githubusercontent.com/deepimagej/imagej-macros/master/8bitBinarize.ijm",path_postprocessing)# Include the info about the postprocessingdij_config.add_postprocessing(path_postprocessing,"postprocessing")DeepImageJ accepts two pre/post-processing routines. The images will be processed in the order in which we include them withadd_postprocessing. Thus, in this example, the output of the model is first binarized with'8bitBinarize.ijm'and then, processed with'another_macro.ijm':path_second_postprocessing='./folder/another_macro.ijm'dij_config.add_postprocessing(path_second_postprocessing,'postprocessing_2')2. Add information about the example image.Lettest_imgbe an example image to test the model inference andtest_predictionbe the resulting image after the post-processing. It is possible to export the trained model with these two, so an end user can see an example.PixelSizeshould be a list of values according totest_imgdimensions and given in microns (µm).PixelSize=[0.64,0.64,1]# Pixel size of a 3D volume with axis yxzdij_config.add_test_info(test_img,test_prediction,PixelSize)2.1. Create some covers for the model card in the BioImage Model Zoo.Lettest_imgandtest_maskbe the input and output example images, and./input.pngand./output.pngthe names we want to use to store them within bundled model.dij_configstretches the intensity range of the given images to the [0, 255] range so the images can be exported as 8-bits images and visualized properly on the website.dij_config.create_covers([test_img,test_mask])dij_config.Covers=['./input.png','./output.png']3. Store weights using specific formats.The weights of a trained model can be stored either as a TensorFlow SavedModel bundle (saved_model.pb+variables/) or as a Keras HDF5 model (model.h5). Letmodelbe a trained model in TensorFlow. With pydeepimagej, the weights information can be included as follows:dij_config.add_weights_formats(model,'KerasHDF5',authors=['Authors','who','trained it'])dij_config.add_weights_formats(model,'TensorFlow',parent="keras_hdf5",authors=['Authors who','converted the model','into this new format'])which in themodel.yamlappear as :weights:keras_hdf5:source:./keras_model.h5sha256:9f7512eb28de4c6c4182f976dd8e53e9da7f342e14b2528ef897a970fb26875dauthors:-Authors-who-trained ittensorflow_saved_model_bundle:source:./tensorflow_saved_model_bundle.zipsha256:2c552aa561c3c3c9063f42b78dda380e2b85a8ad04e434604af5cbb50eaaa54dparent:keras_hdf5authors:-Authors who-converted the model-into this new format4. EXPORT THE MODELdeepimagej_model_path='./my_trained_model_deepimagej'dij_config.export_model(deepimagej_model_path)When exporting the model, a new folder with a deepImageJ 2.1.0 bundled model is created. The folder is also provided as a zip file, so it can be easily transferable.Additional commandsChange one line in an ImageJ macro# Download the macro filepath_postprocessing="8bitBinarize.ijm"urllib.request.urlretrieve("https://raw.githubusercontent.com/deepimagej/imagej-macros/master/8bitBinarize.ijm",path_postprocessing)# Modify the threshold in the macro to the chosen thresholdijmacro=open(path_postprocessing,"r")list_of_lines=ijmacro.readlines()# Line 21 is the one corresponding to the optimal thresholdlist_of_lines[21]="optimalThreshold ={}\n".format(128)ijmacro.close()ijmacro=open(path_postprocessing,"w")ijmacro.writelines(list_of_lines)ijmacro.close()Estimation of the step size for the shape of the input image.If the model is an encoder-decoder with skip connections, and the input shape of your trained model is not fixed (i.e.[None, None, 1]), the input shape still needs to fit some requirements. You can caluculate it knowing the number of poolings in the encoder path of the network:importnumpyasnppooling_steps=0forkeras_layerinmodel.layers:ifkeras_layer.name.startswith('max')or"pool"inkeras_layer.name:pooling_steps+=1MinimumSize=np.str(2**(pooling_steps))Exceptionspydeepimagej is meant to connect Python with DeepImageJ so images can be processed in the Fiji & ImageJ ecosystem. Hence, images (tensors) are expected to have at least 3 dimensions: height, width and channels. For this reason, models with input shapes of less than 4 dimensions (model.input_shape = [batch, height, width, channels]are not considered. For example, if you have the following situation:model=tf.keras.Sequential([tf.keras.layers.Flatten(input_shape=(28,28)),tf.keras.layers.Dense(128,activation='relu'),tf.keras.layers.Dense(10)])please, modify it tomodel=tf.keras.Sequential([tf.keras.layers.Flatten(input_shape=(28,28,1)),tf.keras.layers.Dense(128,activation='relu'),tf.keras.layers.Dense(10)])Code references used in this package:This code uses similar functions to the ones inStarDistpackage for the calculation of a pixel's receptive field in a network. Citations:Uwe Schmidt, Martin Weigert, Coleman Broaddus, and Gene Myers.
Cell Detection with Star-convex Polygons.
International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), Granada, Spain, September 2018.
DOI:10.1007/978-3-030-00934-2_30Martin Weigert, Uwe Schmidt, Robert Haase, Ko Sugawara, and Gene Myers.
Star-convex Polyhedra for 3D Object Detection and Segmentation in Microscopy.
The IEEE Winter Conference on Applications of Computer Vision (WACV), Snowmass Village, Colorado, March 2020
DOI:10.1109/WACV45572.2020.9093435TODO listAddapt pydeepimagej to PyTorch models so it can export trained models into TorchScript format.Consider multiple inputs and outputs.