Package: pix2pixhd
pix2pixHDProject|Youtube|PaperPytorch implementation of our method for high-resolution (e.g. 2048x1024) photorealistic image-to-image translation. It can be used for turning semantic label maps into photo-realistic images or synthesizing portraits from face label maps.High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANsTing-Chun Wang1,Ming-Yu Liu1,Jun-Yan Zhu2, Andrew Tao1,Jan Kautz1,Bryan Catanzaro11NVIDIA Corporation,2UC BerkeleyIn CVPR 2018.Image-to-image translation at 2k/1k resolutionOur label-to-streetview results- Interactive editing results- Additional streetview resultsLabel-to-face and interactive editing resultsOur editing interfacePrerequisitesLinux or macOSPython 2 or 3[optionally:] NVIDIA GPU (11G memory or larger) + CUDA cuDNNGetting StartedInstallationCreate avirtual environment, and activate itClone this repo:git clone https://github.com/NVIDIA/pix2pixHD
  pip install pix2pixHDTestingA few example Cityscapes test images are included in thedatasetsfolder.Please download the pre-trained Cityscapes model fromhere(google drive link), and put it under./checkpoints/label2city_1024p/Test the model (bash ./scripts/test_1024p.sh):#!./scripts/test_1024p.shpix2pixhd-test--namelabel2city_1024p--netGlocal--ngf32--resize_or_cropnoneThe test results will be saved to a html file here:./results/label2city_1024p/test_latest/index.html.More example scripts can be found in thescriptsdirectory.For other options, cf.pix2pixhd-test --helpDatasetWe use the Cityscapes dataset. To train a model on the full dataset, please download it from theofficial website(registration required).
After downloading, please put it under thedatasetsfolder in the same way the example images are provided.TrainingTrain a model at 1024 x 512 resolution (bash ./scripts/train_512p.sh):#!./scripts/train_512p.shpix2pixhd-train--namelabel2city_512pTo view training results, please checkout intermediate results in./checkpoints/label2city_512p/web/index.html.
If you have tensorflow installed, you can see tensorboard logs in./checkpoints/label2city_512p/logsby adding--tf_logto the training scripts.For other options, cf.pix2pixhd-train --helpMulti-GPU trainingTrain a model using multiple GPUs (bash ./scripts/train_512p_multigpu.sh):#!./scripts/train_512p_multigpu.shpix2pixhd-train--namelabel2city_512p--batchSize8--gpu_ids0,1,2,3,4,5,6,7Note: this is not tested and we trained our model using single GPU only. Please use at your own discretion.Training with Automatic Mixed Precision (AMP) for faster speedTo train with mixed precision support, please first install apex from:https://github.com/NVIDIA/apexYou can then train the model by adding--fp16. For example,#!./scripts/train_512p_fp16.shpython-mtorch.distributed.launchpix2pixhd/train.py--namelabel2city_512p--fp16In our test case, it trains about 80% faster with AMP on a Volta machine.Training at full resolutionTo train the images at full resolution (2048 x 1024) requires a GPU with 24G memory (bash ./scripts/train_1024p_24G.sh), or 16G memory if using mixed precision (AMP).If only GPUs with 12G memory are available, please use the 12G script (bash ./scripts/train_1024p_12G.sh), which will crop the images during training. Performance is not guaranteed using this script.Training with your own datasetIf you want to train with your own dataset, please generate label maps which are one-channel whose pixel values correspond to the object labels (i.e. 0,1,...,N-1, where N is the number of labels). This is because we need to generate one-hot vectors from the label maps. Please also specity--label_nc Nduring both training and testing.If your input is not a label map, please just specify--label_nc 0which will directly use the RGB colors as input. The folders should then be namedtrain_A,train_Binstead oftrain_label,train_img, where the goal is to translate images from A to B.If you don't have instance maps or don't want to use them, please specify--no_instance.The default setting for preprocessing isscale_width, which will scale the width of all training images toopt.loadSize(1024) while keeping the aspect ratio. If you want a different setting, please change it by using the--resize_or_cropoption. For example,scale_width_and_cropfirst resizes the image to have widthopt.loadSizeand then does random cropping of size(opt.fineSize, opt.fineSize).cropskips the resizing step and only performs random cropping. If you don't want any preprocessing, please specifynone, which will do nothing other than making sure the image is divisible by 32.More Training/Test DetailsFlags: seeoptions/train_options.pyandoptions/base_options.pyfor all the training flags; seeoptions/test_options.pyandoptions/base_options.pyfor all the test flags.Instance map: we take in both label maps and instance maps as input. If you don't want to use instance maps, please specify the flag--no_instance.CitationIf you find this useful for your research, please use the following.@inproceedings{wang2018pix2pixHD,title={High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs},author={Ting-Chun Wang and Ming-Yu Liu and Jun-Yan Zhu and Andrew Tao and Jan Kautz and Bryan Catanzaro},booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},year={2018}}AcknowledgmentsThis code borrows heavily frompytorch-CycleGAN-and-pix2pix.