Drastically different inference results on different machines?

username: CCL
cooked: <p>I trained a segmentation model in Pytorch and tested it to give an F1 score of  <code>0.93</code>  on my local computer (Windows, conda, CUDA 10.2, Pytorch 1.2). However, the F1 score dropped to  <code>0.3</code>  when testing on a Linux server (conda, CUDA 9.0, Pytorch 1.1). I double checked that both sets of code, label files, test sets were the same, and there is no “explicit” random sampling in my code (even if so, the effect shouldn’t be so drastic).</p>
<p>I tried the following to solve my problem:<br>
I suspected that it might have something to do with Pytorch versions, so installed Pytorch 1.1 on my Windows machine to match the server’s, but got the same  <code>0.93</code>  score. I then thought it had to do with CUDA itself, so eliminated the GPU factor altogether and did inference on cpu on server, I still got  <code>0.3</code>  on server.</p>
<p>What could possibly be causing this huge discrepancy?</p>

username: simaiden
cooked: <p>Are you are setting the model to evaluation using <code>model.eval()</code> ?</p>

username: CCL
cooked: <p>Yes, and both codes are the same</p>

username: ptrblck
cooked: <p>Could you check the output of a single input and post the max absolute difference between the predictions?<br>
If this difference is already high, the next step would be to bisect the model and check each layer’s output.</p>

username: CCL
cooked: <p>Do you mean the pixelwise max absolute difference of a predicted mask? I’m doing binary segmentation, and treat predictions as positive if <code>len(mask_pred[mask_pred &gt; 0.5]) &gt; 0</code>.</p>

username: ptrblck
cooked: <p>Yes, I meant the output of the model.<br>
Your code would yield a positive prediction, if at least one pixel belongs to class1, is that correct?</p>

username: CCL
cooked: <p>I checked both the maximum (across all samples) difference before and after sigmoid on the 2 machines and they were the same!</p>
<p>Before: 37.93775939941406<br>
After: 0.9999369382858276</p>
<p>The maximums &amp; ranges for individual samples do differ though!</p>

username: ptrblck
cooked: <p>Do the provided values represent the difference between both models?<br>
I’m not sure I understand the last post correctly.<br>
What was same and was differs now?</p>

username: CCL
cooked: <p>Oh I meant that 37.9 is the max difference on both machines, before sigmoid. 0.99 is the max difference on both machines after sigmoid. Although the overall max difference is the same, I noticed that the difference of individual samples are different across machines.</p>

username: ptrblck
cooked: <p>Ah OK, thanks for the clarification.<br>
In that case, I would recommend to check each layer’s output and narrow down, where this difference is created.</p>

username: CCL
cooked: <p>Thanks I’ll do that and post an update! Do you think it’s a Pytorch version issue? Maybe the code for some operations changed moving from 1.1.0 to 1.2.0?</p>

username: ptrblck
cooked: <p>You mentioned you’ve installed the same PyTorch versions and use the CPU now?<br>
Make sure to use the same versions before staring to debug.</p>
<p>Also, I would recommend to use the lastest stable version (<code>1.4</code>) as well as Python3, as we’ve had a weird issue with Python2.7 recently (<a href="https://discuss.pytorch.org/t/is-weight-norm-different-in-different-python-versions/71108">link</a>).</p>

username: CCL
cooked: <p>Yes I forgot about that, I did test out different versions. It’s just that the Linux server uses CUDA 9.0 and 1.1.0 is the highest supported version if I’m not mistaken. I’m using python 3.7 for both (3.7.3 linux, 3.7.6 windows).</p>

username: ptrblck
cooked: <p>If you are using the binaries, they will ship with their own CUDA and cudnn libs.<br>
You would only have to provide an NVIDIA driver.</p>