Difference In the output image when using traced model(.pt) with C++ OpenCV

username: Apurv25
cooked: <p>Hello,<br>
I have retrained the model based on <a href="https://github.com/VITA-Group/EnlightenGAN" rel="noopener nofollow ugc">EnlightenGAN</a>. Further I have traced the model in order to execute it in a C++ application using libTorch v1.6. However, I am getting slightly different results as compared to the python(executing the traced model) version.</p>
<p>The model  requires the input RGB tensor the attention map Image tensor as input. The attention map is basically to inform the model about the image region which requires contrast enhancement.</p>
<p>Below is the code to get inference the output from PT model in python.</p>
<pre><code class="lang-auto">
def getTransform():
    transform_list = []
    transform_list += [transforms.ToTensor(),
                       transforms.Normalize((0.5, 0.5, 0.5),
                                            (0.5, 0.5, 0.5))]
    return transforms.Compose(transform_list)

def convertToCV(tensor):
    
    tensor = torch.squeeze(tensor)
    tensor = tensor.cpu().float().detach()
    tensor = torch.unsqueeze(tensor, 0)
    tensor = tensor.permute(1, 2, 0)
    tensor = ((tensor +1)/2.0) * 255.0
    tensor = tensor.numpy()
    return tensor

def proprocess(image):
    
    transform = getTransform()
    trgbImage = transform(image)
    r,g,b = trgbImage[0]+1, trgbImage[1]+1, trgbImage[2]+1
    tattentionImage = 1. - ((0.299*r+0.587*g+0.114*b)/2.)
    tattentionImage = torch.unsqueeze(tattentionImage, 0)
    trgbImage = torch.unsqueeze(trgbImage, 0)
    tattentionImage = torch.unsqueeze(tattentionImage, 0)

    return trgbImage, tattentionImage

def run(inputPath, OutputPath):

    modelToLoad = torch.jit.load("./EGAN.pt")
    print("OK")
    count =0 
    for filename in os.listdir(inputPath):
        
        print("Processing Image : ", filename)
        inputImage = cv2.imread(os.path.join(inputPath,filename))
        
        rgbImage, attentionImage = proprocess(inputImage)
        
        fake, real = modelToLoad.forward(rgbImage,attentionImage )
        
        fake_B = convertToCV(fake)
        fake_B1 = cv2.cvtColor(fake_B, cv2.COLOR_RGB2BGR)
        cv2.imwrite(OutputPath + "pic1.png" , fake_B )
</code></pre>
<p>The C++ version for the inference code is below</p>
<pre><code class="lang-auto">#define A 0.299
#define B 0.5870
#define C 0.114

cv::Mat torchTensortoCVMat(torch::Tensor&amp; tensor)
{
	tensor = tensor.squeeze(0);
	tensor = tensor.to(torch::kCPU).to(torch::kFloat32).detach();
	tensor = tensor.permute({ 1, 2, 0 }).contiguous();
	tensor = tensor.add(1).div(2.0).mul(255.0);
	tensor = tensor.to(torch::kU8);

	int64_t height = tensor.size(0);
	int64_t width  = tensor.size(1);
	cv::Mat mat    = cv::Mat(cv::Size(width, height), CV_8UC3, tensor.data_ptr&lt;uchar&gt;());
	return mat;
}

std::vector&lt;torch::jit::IValue&gt; CV2Tensor(const cv::Mat&amp; cv_Image)
{
	torch::Tensor tInputImage = (torch::from_blob(cv_Image.data, { cv_Image.rows, cv_Image.cols, cv_Image.channels() }, torch::kByte));
	tInputImage = tInputImage.to(torch::kFloat).div(255);
	tInputImage = tInputImage.sub(0.5).div(0.5).permute({ 2, 0, 1 });

	torch::Tensor red   = tInputImage[0] + 1 ;
	torch::Tensor green = tInputImage[1] + 1 ;
	torch::Tensor blue  = tInputImage[2] + 1;

	red   = red.mul(A);
	green = green.mul(B);
	blue  = blue.mul(C);

	torch::Tensor channelSum = red.add(green).add(blue);
	channelSum = channelSum.div(2.);
	torch::Tensor tGrayImage = 1. - channelSum;

	tGrayImage.unsqueeze_(0);
	tGrayImage.unsqueeze_(0);
	tInputImage.unsqueeze_(0);

	std::vector&lt;torch::jit::IValue&gt; input;
	input.push_back(tInputImage);
	input.push_back(tGrayImage);

	return input;
}

void enhanceImage(const std::string&amp; Img, torch::jit::script::Module&amp; network,const std::string&amp; outputPath, std::string&amp; fileName)
{
	cv::Mat rgbImage;
	cv::Mat inputImage = cv::imread(Img);
	std::vector&lt;torch::jit::IValue&gt; input = CV2Tensor(inputImage);

	try
	{
		auto outputs = network.forward(input).toTuple();
		torch::Tensor resultFake = outputs-&gt;elements()[0].toTensor();
	
		cv::Mat output1 = torchTensortoCVMat(resultFake);
		cv::imshow("out1.png", output1);
		cv::waitKey(0);
	}
	catch (std::exception&amp; e)
	{
		std::cout &lt;&lt; e.what() &lt;&lt; std::endl;
	}
}	
</code></pre>
<p>I have also checked the tensor output at all the steps, and they are same. However, after the conversion the output image has color flowing out from brighter regions of input image as show below.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/5/25fbc3ddd3c2aa354c15f8b1c5bc07083a6e2d3e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/25fbc3ddd3c2aa354c15f8b1c5bc07083a6e2d3e" title=""><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/25fbc3ddd3c2aa354c15f8b1c5bc07083a6e2d3e_2_333x250.jpeg" alt="" data-base62-sha1="5q14nAEPVvnGw8vYmHFz8SXgEQm" width="333" height="250" role="presentation" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/25fbc3ddd3c2aa354c15f8b1c5bc07083a6e2d3e_2_333x250.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/25fbc3ddd3c2aa354c15f8b1c5bc07083a6e2d3e_2_499x375.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/25fbc3ddd3c2aa354c15f8b1c5bc07083a6e2d3e_2_666x500.jpeg 2x" data-dominant-color="40362D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename"></span><span class="informations">1440×1080 89.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/0/b0b1bbb6fb9562a229b4a18e1261996617314dfb.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b0b1bbb6fb9562a229b4a18e1261996617314dfb" title="out1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b0b1bbb6fb9562a229b4a18e1261996617314dfb_2_333x250.jpeg" alt="out1" data-base62-sha1="pd6Pfh0Bg8uq4Fv8b00dTsvSQtB" width="333" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b0b1bbb6fb9562a229b4a18e1261996617314dfb_2_333x250.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b0b1bbb6fb9562a229b4a18e1261996617314dfb_2_499x375.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b0b1bbb6fb9562a229b4a18e1261996617314dfb_2_666x500.jpeg 2x" data-dominant-color="3F362D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">out1</span><span class="informations">1440×1080 92.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code>           Python Version                         C++ Version
</code></pre>
<p>I have tried many attempt but I am totally puzzeled as to how I should solve the problem. Any help is most welcome.</p>
<p>Thanks.<br>
PS : Let me know if more info is required.</p>

username: tom
cooked: <p>So I’m not sure I understand the code in detail, but are you doing the BGR2RGB for the decoding? PIL/ToTensor gives RGB and opencv has BGR by default.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Apurv25
cooked: <p>Hi, no I do not perform BGR2RGB for decoding. The reason is that I do not use PIL image in Python/C++ version. I am using the Opencv Mat and performing the Tensor operation.</p>
<p>PS: Should I post more details (such as includes) in the code?</p>
<p>Best Regards<br>
Apurv</p>

username: tom
cooked: <p>Ah, right, the torchvision transform in your snippet had me confused. How do you check c++ and Python produce the same output? I would probably stuck the c++ code into an extension to check each step.</p>

username: Apurv25
cooked: <p>Hi,<br>
I found the problem in the above implementation. In the CPP version, I was not clamping the values after doing denormalisation. I have put a clamping function and now it is working as expected.<br>
The edit part if anyone stumbles on the same problem is below:</p>
<pre><code class="lang-auto">tensor = tensor.mul(0.5).add(0.5).mul(255.0); -- &gt; tensor = tensor.mul(0.5).add(0.5).mul(255.0).clamp(0, 255);
</code></pre>
<p>Without clamping it as causing overflow in brighter regions of the image.</p>
<p>Is there a better way of comparing Tensor output from Python and C++?<br>
Currently I was checking the correctness like a caveman <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=10" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"><br>
Checking via printing it out.</p>
<p>Best regards<br>
Apurv</p>

username: tom
cooked: <aside class="quote no-group" data-username="Apurv25" data-post="5" data-topic="132246">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apurv25/48/59985_2.png" class="avatar"> Apurv25:</div>
<blockquote>
<p>Is there a better way of comparing Tensor output from Python and C++?</p>
</blockquote>
</aside>
<p>Check out the cpp extension tutorial for calling your cpp code from Python. I find that quite convenient.<br>
I doalot of printing, though, too.</p>