Package: xu2net
U2-Net: U Square NetThis is the official repo for our paperU2-Net(U square net)published in Pattern Recognition 2020:U2-Net: Going Deeper with Nested U-Structure for Salient Object DetectionXuebin Qin,Zichen Zhang,Chenyang Huang,Masood Dehghan,Osmar R. ZaianeandMartin JagersandContact: xuebin[at]ualberta[dot]caUpdates !!!(2021-Aug-24)We played a bit more about fusing the orignal image and the generated portraits to composite different styles. You can(1) Download this repo bygit clone https://github.com/NathanUA/U-2-Net.git(2) Download the u2net_portrait.pth fromGoogleDriveorBaidu Pan(提取码：chgd)model and put it into the directory:./saved_models/u2net_portrait/,(3) run the code by commandpython u2net_portrait_composite.py -s 20 -a 0.5,where-sindicates the sigma of gaussian function for blurring the orignal image and-adenotes the alpha weights of the orignal image when fusing them.(2021-July-16)A newbackground removal webappdeveloped by Изатоп Василий.(2021-May-26)ThankDang Quoc Quyfor hisArt Transfer APPbuilt upon U2-Net.(2021-May-5)ThankAK391for sharing hisGradio Web Demo of U2-Net.(2021-Apr-29)ThanksJonathan Benavides Vallejofor releasing his AppLensOCR: Extract Text & Image, which uses U2-Net for extracting the image foreground.(2021-Apr-18)ThanksAndrea Scuderifor releasing his AppClipping Camera, which is an U2-Net driven realtime camera app and "is able to detect relevant object from the scene and clip them to apply fancy filters".(2021-Mar-17)Dennis Bappertre-trained the U2-Net model forhuman portrait matting. The results look very promising and he also provided the details of the training process and data generation(and augmentation) strategy, which are inspiring.(2021-Mar-11)Dr. Tim developed avideo version rembgfor removing video backgrounds using U2-Net. The awesome demo results can be found onYouTube.(2021-Mar-02)We found some other interesting applications of our U2-Net includingMOJO CUT,Real-Time Background Removal on Iphone,Video Background Removal,Another Online Portrait Generation Demo on AWS,AI Scissor.(2021-Feb-15)We just released an online demohttp://profu.aifor the portrait generation. Please feel free to give it a try and provide any suggestions or comments.(2021-Feb-06)Recently, some people asked the problem of using U2-Net for human segmentation, so we trained another example model for human segemntation based onSupervisely Person Dataset.(1) To run the human segmentation model, please first downlowd theu2net_human_seg.pthmodel weights into./saved_models/u2net_human_seg/.(2) Prepare the to-be-segmented images into the corresponding directory, e.g../test_data/test_human_images/.(3) Run the inference by command:python u2net_human_seg_test.pyand the results will be output into the corresponding dirctory, e.g../test_data/u2net_test_human_images_results/Notes: Due to the labeling accuracy of the Supervisely Person Dataset, the human segmentation model (u2net_human_seg.pth) here won't give you hair-level accuracy. But it should be more robust than u2net trained with DUTS-TR dataset on general human segmentation task. It can be used for human portrait segmentation, human body segmentation, etc.(2020-Dec-28)Some interesting applications and useful tools based on U2-Net:(1)Xiaolong Liudeveloped several very interesting applications based on U2-Net includingHuman Portrait Drawing(As far as I know, Xiaolong is the first one who uses U2-Net for portrait generation),image mattingandso on.(2)Vladimir Seregindeveloped an interesting tool,NN based lineart, for comparing the portrait results of U2-Net and that of another popular model,ArtLine, developed byVijish Madhavan.(3)Daniel Gatisbuilt a python tool,Rembg, for image backgrounds removal based on U2-Net. I think this tool will greatly facilitate the application of U2-Net in different fields.(2020-Nov-21)Recently, we found an interesting application of U2-Net forhuman portrait drawing. Therefore, we trained another model for this task based on theAPDrawingGAN dataset.Usage for portrait generationClone this repo to localgit clone https://github.com/NathanUA/U-2-Net.gitDownload the u2net_portrait.pth fromGoogleDriveorBaidu Pan(提取码：chgd)model and put it into the directory:./saved_models/u2net_portrait/.Run on the testing set.(1) Download the train and test set fromAPDrawingGAN. These images and their ground truth are stitched side-by-side (512x1024). You need to split each of these images into two 512x512 images and put them into./test_data/test_portrait_images/portrait_im/. You can also download the split testing set onGoogleDrive.(2) Running the inference with commandpython u2net_portrait_test.pywill ouptut the results into./test_data/test_portrait_images/portrait_results.Run on your own dataset.(1) Prepare your images and put them into./test_data/test_portrait_images/your_portrait_im/.To obtain enough details of the protrait, human head region in the input image should be close to or larger than 512x512. The head background should be relatively clear.(2) Run the prediction by commandpython u2net_portrait_demo.pywill outputs the results to./test_data/test_portrait_images/your_portrait_results/.(3) The difference betweenpython u2net_portrait_demo.pyandpython u2net_portrait_test.pyis that we added a simpleface detectionstep before the portrait generation inu2net_portrait_demo.py.  Because the testing set of APDrawingGAN are normalized and cropped to 512x512 for including only heads of humans, while our own dataset may varies with different resolutions and contents. Therefore, the codepython u2net_portrait_demo.pywill detect the biggest face from the given image and then crop, pad and resize the ROI to 512x512 for feeding to the network. The following figure shows how to take your own photos for generating high quality portraits.(2020-Sep-13)Our U2-Net based model is the6thinMICCAI 2020 Thyroid Nodule Segmentation Challenge.(2020-May-18)The official paper of ourU2-Net (U square net)(PDF in elsevier(free until July 5 2020),PDF in arxiv) is now available. If you are not able to access that, please feel free to drop me an email.(2020-May-16)We fixed the upsampling issue of the network. Now, the model should be able to handlearbitrary input size. (Tips: This modification is to facilitate the retraining of U2-Net on your own datasets. When using our pre-trained model on SOD datasets, please keep the input size as 320x320 to guarantee the performance.)(2020-May-16)We highly appreciateCyril Diagnefor building this fantastic AR project:AR Copy and Pasteusing ourU2-Net(Qinet al, PR 2020) andBASNet(Qinet al, CVPR 2019). Thedemo videoin twitter has achieved over5Mviews, which is phenomenal and shows us more application possibilities of SOD.U2-Net Results (176.3 MB)Our previous work:BASNet (CVPR 2019)Required librariesPython 3.6numpy 1.15.2scikit-image 0.14.0python-opencv
PIL 5.2.0PyTorch 0.4.0torchvision 0.2.1globUsage for salient object detectionClone this repogit clone https://github.com/NathanUA/U-2-Net.gitDownload the pre-trained model u2net.pth (176.3 MB) fromGoogleDriveorBaidu Pan 提取码: pf9kor u2netp.pth (4.7 MB) fromGoogleDriveorBaidu Pan 提取码: 8xsiand put it into the dirctory './saved_models/u2net/' and './saved_models/u2netp/'Cd to the directory 'U-2-Net', run the train or inference process by command:python u2net_train.pyorpython u2net_test.pyrespectively. The 'model_name' in both files can be changed to 'u2net' or 'u2netp' for using different models.We also provide the predicted saliency maps (u2net results,u2netp results) for datasets SOD, ECSSD, DUT-OMRON, PASCAL-S, HKU-IS and DUTS-TE.U2-Net ArchitectureQuantitative ComparisonQualitative ComparisonCitation@InProceedings{Qin_2020_PR,
title = {U2-Net: Going Deeper with Nested U-Structure for Salient Object Detection},
author = {Qin, Xuebin and Zhang, Zichen and Huang, Chenyang and Dehghan, Masood and Zaiane, Osmar and Jagersand, Martin},
journal = {Pattern Recognition},
volume = {106},
pages = {107404},
year = {2020}
}