Pytorch Training Network with CuDNN Inference Network

username: avyz
cooked: <p>Hi,</p>
<p>I am training LeNet for handwritten digit classification on google colab using Pytorch with MNIST data set and saving weights and biases of the network in text files for each layer. Then loading these network parameters to the locally created LeNet (for inference) built using CuDNN library.<br>
INPUT-&gt;CONV1-&gt;POOL1-&gt;CONV2-&gt;POOL2-&gt;FC1-&gt;RELU-&gt;FC2-&gt;SMAX</p>
<p>Pytorch seems to return weights in NCHW format, hence using NCHW format for all the tensors in CuDNN library. The flattened weights and biases are read into buffers from the text files.</p>
<p>For the same test dataset, on cloud I am getting a classification error of ~5%, whereas using the same weights and bias on a locally created network, I am getting an error of ~10%. Can anyone please suggest the reason behind this difference in the error? Is there any detail between Pytorch and CuDNN that I am missing?</p>
<p><strong>Here is a little Pytorch snippet for creating the network-</strong></p>
<blockquote>
<p>class LeNet(nn.Module):<br>
# network structure<br>
def <strong>init</strong>(self):<br>
super(LeNet, self).<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(1, 20, 5)<br>
self.conv2 = nn.Conv2d(20, 50, 5)<br>
self.fc1   = nn.Linear(50x4x4, 500)<br>
self.fc2   = nn.Linear(500, 10)</p>
<pre><code>def forward(self, x):
    x = F.max_pool2d(self.conv1(x), (2, 2))
    x = F.max_pool2d(self.conv2(x), (2, 2))
    x = x.view(-1, self.num_flat_features(x))
    x = F.relu(self.fc1(x))
    x = F.softmax(self.fc2(x))
    return x

def num_flat_features(self, x):
    size = x.size()[1:]
    return np.prod(size)
</code></pre>
</blockquote>
<p><strong>Here is a little Pytorch snippet for saving weights and biases-</strong></p>
<blockquote>
<p>for name, param in net_gpu.named_parameters():<br>
v1 = name.partition(‘.’)[0]<br>
v2 = name.partition(‘.’)[2]<br>
arr = np.array(param.data.cpu())<br>
f = open(v1 + “_” + v2 + “.txt”, “w”)<br>
for val in arr:<br>
np.savetxt(f, val.flat)<br>
f.close()</p>
</blockquote>
<p><strong>Here is a CuDNN sippet for creating inference network-</strong></p>
<blockquote>
<p>// Conv1 layer<br>
cudnnConvolutionForward(cudnnHandle, &amp;alpha, dataTensor,<br>
data, conv1filterDesc, pconv1, conv1Desc,<br>
conv1algo, workspace, m_workspaceSize, &amp;beta,<br>
conv1Tensor, conv1);<br>
cudnnAddTensor(cudnnHandle, &amp;alpha, conv1BiasTensor,<br>
pconv1bias, &amp;alpha, conv1Tensor, conv1);</p>
<pre><code>  // Pool1 layer
  cudnnPoolingForward(cudnnHandle, poolDesc, &amp;alpha, conv1Tensor,
  	conv1, &amp;beta, pool1Tensor, pool1);

  // Conv2 layer
  cudnnConvolutionForward(cudnnHandle, &amp;alpha, pool1Tensor,
  	pool1, conv2filterDesc, pconv2, conv2Desc,
  	conv2algo, workspace, m_workspaceSize, &amp;beta,
  	conv2Tensor, conv2);
  cudnnAddTensor(cudnnHandle, &amp;alpha, conv2BiasTensor,
  	pconv2bias, &amp;alpha, conv2Tensor, conv2);

  // Pool2 layer
  cudnnPoolingForward(cudnnHandle, poolDesc, &amp;alpha, conv2Tensor,
  	conv2, &amp;beta, pool2Tensor, pool2);

  // FC1 layer
  // Forward propagate neurons using weights (fc1 = pfc1'*pool2)
  cublasSgemm(cublasHandle, CUBLAS_OP_T, CUBLAS_OP_N,
  	ref_fc1.outputs, m_batchSize, ref_fc1.inputs,
  	&amp;alpha,
  	pfc1, ref_fc1.inputs,
  	pool2, ref_fc1.inputs,
  	&amp;beta,
  	fc1, ref_fc1.outputs);
  // Add bias using GEMM's "beta" (fc1 += pfc1bias*1_vec')
  cublasSgemm(cublasHandle, CUBLAS_OP_N, CUBLAS_OP_N,
  	ref_fc1.outputs, m_batchSize, 1,
  	&amp;alpha,
  	pfc1bias, ref_fc1.outputs,
  	onevec, 1,
  	&amp;alpha,
  	fc1, ref_fc1.outputs);

  // ReLU activation
  cudnnActivationForward(cudnnHandle, fc1Activation, &amp;alpha,
  	fc1Tensor, fc1, &amp;beta, fc1Tensor, fc1relu);

  // FC2 layer
  // Forward propagate neurons using weights (fc2 = pfc2'*fc1relu)
  cublasSgemm(cublasHandle, CUBLAS_OP_T, CUBLAS_OP_N,
  	ref_fc2.outputs, m_batchSize, ref_fc2.inputs,
  	&amp;alpha,
  	pfc2, ref_fc2.inputs,
  	fc1relu, ref_fc2.inputs,
  	&amp;beta,
  	fc2, ref_fc2.outputs);
  // Add bias using GEMM's "beta" (fc2 += pfc2bias*1_vec')
  cublasSgemm(cublasHandle, CUBLAS_OP_N, CUBLAS_OP_N,
  	ref_fc2.outputs, m_batchSize, 1,
  	&amp;alpha,
  	pfc2bias, ref_fc2.outputs,
  	onevec, 1,
  	&amp;alpha,
  	fc2, ref_fc2.outputs);

  // Softmax loss
  cudnnSoftmaxForward(cudnnHandle, CUDNN_SOFTMAX_ACCURATE, CUDNN_SOFTMAX_MODE_CHANNEL,
  	&amp;alpha, fc2Tensor, fc2, &amp;beta, fc2Tensor, result);
</code></pre>
</blockquote>
<p>Any suggestion on this will be highly appreciated. Thanks!</p>

username: ptrblck
cooked: <p>You could use the <a href="https://docs.nvidia.com/deeplearning/cudnn/developer-guide/index.html#api-logging">cudnn API logging</a> on both workloads and compare the logs to isolate potential differences between both runs.</p>

username: avyz
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I can get logs for CuDNN on the local machine by using environment variables. To get the log on google colab for Pytorch code, I was using-</p>
<blockquote>
<p>import os<br>
os.environ[‘CUDNN_LOGINFO_DBG’] = ‘1’<br>
os.environ[‘CUDNN_LOGDEST_DBG’] = ‘log.txt’</p>
</blockquote>
<p>But, it doesn’t seem to return any log in .txt file. I also tried to print log on console by using stdout, but it doesn’t work as well. Please suggest how to get log on google colab in Pytorch?</p>

username: ptrblck
cooked: <p>Unfortunately, I don’t know how cudnn logging can be enabled in Jupyter notebooks or Google Colab and also don’t know if it’s a limitation of the former of latter.<br>
One common issue users are seeing when trying to set env variables inside a notebook is the order:<br>
e.g. setting <code>CUDA_VISIBLE_DEVICES</code> or <code>CUDA_LAUNCH_BLOCKING</code> after creating the CUDA context wouldn’t have any effect. Maybe you are facing a similar issue and could move the <code>os.environ</code> calls to the very beginning of the notebook.</p>

username: avyz
cooked: <p>Moving <code>os.environ</code> calls to the very beginning helped save the log to .txt file in google colab. That was helpful!</p>
<p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a></p>