A computer-implemented method for automatically performing cross-domain based medical image analysis on a medical image of a patient, comprising: receiving a medical image of a patient from a first domain; inputting the medical image of the patient to a first encoder of a machine learning based cross-domain deep image-to-image network, the cross-domain deep image to image network trained with an adversarial network to perform a target medical image analysis task for the first domain and a second domain, the cross-domain deep image-to-image network including the first encoder for the first domain, a second encoder for the second domain, and a decoder; wherein the first encoder, the second encoder and the decoder are deep neural networks; and automatically generating an output image that provides a result of the target medical image analysis task from the input medical image using the cross-domain deep image-to-image network by the first encoder converting the input medical image from the first domain to a feature map representing a high-level representation of the input medical image from the first domain and the decoder generating the output image from the feature map generated by the first encoder, wherein the first encoder is trained to convert the input medical image from the first domain to the feature map, the second encoder is trained to convert a second input medical image from the second domain to a feature map, and the decoder is trained to generate output images from the feature maps generated by the first encoder and the feature maps generated by the second encoder, wherein the first encoder, the second encoder, and the decoder are trained together with a discriminator as the adversarial network to optimize a minimax objective function based on the discriminator receiving as input feature maps generated by the first encoder from training input images from the first domain and feature maps generated by the second encoder from training input images from the second domain and classifying each feature map as being from the first domain or the second domain, and wherein the first domain and the second domain are different medical imaging modalities or different imaging protocols. receiving a medical image of a patient from a first domain; inputting the medical image of the patient to a first encoder of a machine learning based cross-domain deep image-to-image network, the cross-domain deep image to image network trained with an adversarial network to perform a target medical image analysis task for the first domain and a second domain, the cross-domain deep image-to-image network including the first encoder for the first domain, a second encoder for the second domain, and a decoder; wherein the first encoder, the second encoder and the decoder are deep neural networks; and automatically generating an output image that provides a result of the target medical image analysis task from the input medical image using the cross-domain deep image-to-image network by the first encoder converting the input medical image from the first domain to a feature map representing a high-level representation of the input medical image from the first domain and the decoder generating the output image from the feature map generated by the first encoder, wherein the first encoder is trained to convert the input medical image from the first domain to the feature map, the second encoder is trained to convert a second input medical image from the second domain to a feature map, and the decoder is trained to generate output images from the feature maps generated by the first encoder and the feature maps generated by the second encoder, wherein the first encoder, the second encoder, and the decoder are trained together with a discriminator as the adversarial network to optimize a minimax objective function based on the discriminator receiving as input feature maps generated by the first encoder from training input images from the first domain and feature maps generated by the second encoder from training input images from the second domain and classifying each feature map as being from the first domain or the second domain, and wherein the first domain and the second domain are different medical imaging modalities or different imaging protocols.