How should time differ using fp16 amp ? and should memory consumption decrease?

Bika: <p>I have a rtx 2080ti and i am training this model.</p>
<blockquote>
<p>import time<br>
start = time.time()<br>
epochs = 40</p>
<p>for epoch in range(epochs):<br>
train_accuracy = 0<br>
count = 0<br>
accumulated_loss = 0<br>
for train_in,train_label in tqdm(batch(X_train,Y_train)):<br>
<span class="hashtag-raw">#print</span>(train_in)<br>
torch.cuda.synchronize()<br>
output = model(train_in.to(device))<br>
<span class="hashtag-raw">#print</span>(output.shape)<br>
output = output.squeeze(1)<br>
<span class="hashtag-raw">#print</span>(output.shape)<br>
train_label = train_label.squeeze(-1)<br>
loss = criterion(output,train_label.long().to(device))<br>
<span class="hashtag-raw">#with</span> amp.scale_loss(loss, optimizer) as scaled_loss:<br>
<span class="hashtag-raw">#scaled_loss</span>.backward()<br>
loss.backward()<br>
optimizer.step()<br>
count += 1<br>
train_accuracy += accuracy(output,train_label)<br>
accumulated_loss += loss#scaled_loss<br>
optimizer.zero_grad()</p>
<pre><code>print('Training Epoch {} with Accuracy :{} Loss :{}'.format(epoch+1,train_accuracy/count,accumulated_loss/count))
with torch.no_grad():
    test_acc = 0
    test_count = 0
    for X_t,Y_t in tqdm(batch(X_test,Y_test)):
        test_output = model(X_t.to(device))
        test_output = test_output.squeeze(1)

        Y_t = Y_t.squeeze(-1)
        test_accuracy = accuracy(test_output,Y_t)
        test_acc += test_accuracy
        test_count += 1
        
print('Testing Epoch {} with Accuracy :{}'.format(epoch+1,test_acc/test_count))  
</code></pre>
<p>end = time.time()<br>
print(“time equal {}”.format(end-start))</p>
</blockquote>
<p>the running time for training without fp16 is 24 and with fp16 is nearly 32. This message appears to me when i check the setup of “apex” library:</p>
<blockquote>
<p>Compiling cuda extensions with<br>
nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2017 NVIDIA Corporation<br>
Built on Fri_Nov__3_21:07:56_CDT_2017<br>
Cuda compilation tools, release 9.1, V9.1.85<br>
from /usr/bin</p>
<p>Traceback (most recent call last):<br>
File “setup.py”, line 100, in <br>
check_cuda_torch_binary_vs_bare_metal(torch.utils.cpp_extension.CUDA_HOME)<br>
File “setup.py”, line 77, in check_cuda_torch_binary_vs_bare_metal<br>
"<a href="https://github.com/NVIDIA/apex/pull/323#discussion_r287021798" class="inline-onebox" rel="noopener nofollow ugc">Hard error on mismatch between torch.version.cuda and + the Cuda toolkit version being used to compile Apex by mcarilli · Pull Request #323 · NVIDIA/apex · GitHub</a>.  "<br>
RuntimeError: Cuda extensions are being compiled with a version of Cuda that does not match the &gt;version used to compile Pytorch binaries.  Pytorch binaries were compiled with Cuda 10.1.<br>
In some cases, a minor-version mismatch will not cause later errors:  &gt;<a href="https://github.com/NVIDIA/apex/pull/323#discussion_r287021798" class="inline-onebox" rel="noopener nofollow ugc">Hard error on mismatch between torch.version.cuda and + the Cuda toolkit version being used to compile Apex by mcarilli · Pull Request #323 · NVIDIA/apex · GitHub</a>.  You can try commenting out this &gt;check (at your own risk).</p>
</blockquote>
<p>I am using pytorch 1.4 , cuda 10.1. I know there is difference in cuda version , but is that the only problem ? And if there is a version difference how can the model train ?<br>
Anybody help i have been in this for a very long time.<br>
I am now download nvidia-toolkit cuda 10.1 and try to set it up maybe it will fix this.<br>
Any clarification ?</p>

ptrblck: <p>You could disable the check for the minor CUDA version as suggested or the better solution would be to install the matching CUDA verison on your system.<br>
Once this is done, you could rebuild apex with the CUDA extensions, which should give you performance benefits.</p>

Bika: <p>My nvidia driver version is 435 and pytorch cuda is 10.1. I cant find a nvidia toolkit that support both versions. Any solution ?<br>
I was using cuda toolkit 9 when i compiled the results, can it work properly ?<br>
I am new in this CUDA world</p>

ptrblck: <p>CUDA10.1 will work with this driver as seen in <a href="https://docs.nvidia.com/deploy/cuda-compatibility/index.html#binary-compatibility__table-toolkit-driver">this table</a>.</p>

Bika: <p>If i downloaded this cuda version , i should get the speedup you talked about in your Github Post <a href="https://github.com/ptrblck/apex/blob/apex_tutorial/tutorials/apex_mixed_precision_intro.ipynb" rel="nofollow noopener">https://github.com/ptrblck/apex/blob/apex_tutorial/tutorials/apex_mixed_precision_intro.ipynb</a> ?<br>
And also how the library is training while its not fully installed/working ? wheter O1,O2 … it still train</p>

ptrblck: <p>If you didn’t build <code>apex</code> with the CUDA extensions, the methods will “work” but might be slow.</p>

Bika: <blockquote>
<p>-cpp_ext --cuda_ext<br>
torch.<strong>version</strong>  =  1.4.0<br>
setup.py:43: UserWarning: Option --pyprof not specified. Not installing PyProf dependencies!<br>
warnings.warn(“Option --pyprof not specified. Not installing PyProf dependencies!”)</p>
<p>Compiling cuda extensions with<br>
nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2019 NVIDIA Corporation<br>
Built on Fri_Feb__8_19:08:17_PST_2019<br>
Cuda compilation tools, release 10.1, V10.1.105<br>
from /usr/local/cuda/bin</p>
<p>running install<br>
running bdist_egg<br>
running egg_info<br>
creating apex.egg-info<br>
writing apex.egg-info/PKG-INFO<br>
writing dependency_links to apex.egg-info/dependency_links.txt<br>
writing top-level names to apex.egg-info/top_level.txt<br>
writing manifest file ‘apex.egg-info/SOURCES.txt’<br>
reading manifest file ‘apex.egg-info/SOURCES.txt’<br>
writing manifest file ‘apex.egg-info/SOURCES.txt’<br>
installing library code to build/bdist.linux-x86_64/egg<br>
running install_lib<br>
running build_py<br>
creating build<br>
creating build/lib.linux-x86_64-3.7<br>
creating build/lib.linux-x86_64-3.7/apex<br>
copying apex/<strong>init</strong>.py → build/lib.linux-x86_64-3.7/apex<br>
creating build/lib.linux-x86_64-3.7/apex/parallel<br>
copying apex/parallel/multiproc.py → build/lib.linux-x86_64-3.7/apex/parallel<br>
copying apex/parallel/<strong>init</strong>.py → build/lib.linux-x86_64-3.7/apex/parallel<br>
copying apex/parallel/LARC.py → build/lib.linux-x86_64-3.7/apex/parallel<br>
copying apex/parallel/distributed.py → build/lib.linux-x86_64-3.7/apex/parallel<br>
copying apex/parallel/sync_batchnorm_kernel.py → build/lib.linux-x86_64-3.7/apex/parallel<br>
copying apex/parallel/optimized_sync_batchnorm_kernel.py → build/lib.linux-x86_64-3.7/apex/parallel<br>
copying apex/parallel/optimized_sync_batchnorm.py → build/lib.linux-x86_64-3.7/apex/parallel<br>
copying apex/parallel/sync_batchnorm.py → build/lib.linux-x86_64-3.7/apex/parallel<br>
creating build/lib.linux-x86_64-3.7/apex/contrib<br>
copying apex/contrib/<strong>init</strong>.py → build/lib.linux-x86_64-3.7/apex/contrib<br>
creating build/lib.linux-x86_64-3.7/apex/optimizers<br>
copying apex/optimizers/fused_novograd.py → build/lib.linux-x86_64-3.7/apex/optimizers<br>
copying apex/optimizers/<strong>init</strong>.py → build/lib.linux-x86_64-3.7/apex/optimizers<br>
copying apex/optimizers/fused_adam.py → build/lib.linux-x86_64-3.7/apex/optimizers<br>
copying apex/optimizers/fused_lamb.py → build/lib.linux-x86_64-3.7/apex/optimizers<br>
copying apex/optimizers/fused_sgd.py → build/lib.linux-x86_64-3.7/apex/optimizers<br>
creating build/lib.linux-x86_64-3.7/apex/amp<br>
copying build/lib.linux-x86_64-3.7/apex/optimizers/fused_sgd.py → build/bdist.linux-x86_64/egg/apex/optimizers<br>
creating build/bdist.linux-x86_64/egg/apex/amp<br>
copying build/lib.linux-x86_64-3.7/apex/amp/scaler.py → build/bdist.linux-x86_64/egg/apex/amp<br>
copying build/lib.linux-x86_64-3.7/apex/amp/compat.py → build/bdist.linux-x86_64/egg/apex/amp<br>
copying build/lib.linux-x86_64-3.7/apex/amp/<strong>init</strong>.py → build/bdist.linux-x86_64/egg/apex/amp<br>
copying build/lib.linux-x86_64-3.7/apex/amp/<strong>version</strong>.py → build/bdist.linux-x86_64/egg/apex/amp<br>
copying build/lib.linux-x86_64-3.7/apex/amp/_process_optimizer.py → build/bdist.linux-x86_64/egg/apex/amp<br>
copying build/lib.linux-x86_64-3.7/apex/amp/handle.py → build/bdist.linux-x86_64/egg/apex/amp<br>
copying build/lib.linux-x86_64-3.7/apex/amp/amp.py → build/bdist.linux-x86_64/egg/apex/amp<br>
copying build/lib.linux-x86_64-3.7/apex/amp/_amp_state.py → build/bdist.linux-x86_64/egg/apex/amp<br>
copying build/lib.linux-x86_64-3.7/apex/amp/_initialize.py → build/bdist.linux-x86_64/egg/apex/amp<br>
creating build/bdist.linux-x86_64/egg/apex/amp/lists<br>
copying build/lib.linux-x86_64-3.7/apex/amp/lists/tensor_overrides.py → build/bdist.linux-x86_64/egg/apex/amp/lists<br>
copying build/lib.linux-x86_64-3.7/apex/amp/lists/<strong>init</strong>.py → build/bdist.linux-x86_64/egg/apex/amp/lists<br>
copying build/lib.linux-x86_64-3.7/apex/amp/lists/torch_overrides.py → build/bdist.linux-x86_64/egg/apex/amp/lists<br>
copying build/lib.linux-x86_64-3.7/apex/amp/lists/functional_overrides.py → build/bdist.linux-x86_64/egg/apex/amp/lists<br>
copying build/lib.linux-x86_64-3.7/apex/amp/rnn_compat.py → build/bdist.linux-x86_64/egg/apex/amp<br>
copying build/lib.linux-x86_64-3.7/apex/amp/opt.py → build/bdist.linux-x86_64/egg/apex/amp<br>
copying build/lib.linux-x86_64-3.7/apex/amp/frontend.py → build/bdist.linux-x86_64/egg/apex/amp<br>
copying build/lib.linux-x86_64-3.7/apex/amp/wrap.py → build/bdist.linux-x86_64/egg/apex/amp<br>
copying build/lib.linux-x86_64-3.7/apex/amp/utils.py → build/bdist.linux-x86_64/egg/apex/amp<br>
creating build/bdist.linux-x86_64/egg/apex/multi_tensor_apply<br>
copying build/lib.linux-x86_64-3.7/apex/multi_tensor_apply/<strong>init</strong>.py → build/bdist.linux-x86_64/egg/apex/multi_tensor_apply<br>
copying build/lib.linux-x86_64-3.7/apex/multi_tensor_apply/multi_tensor_apply.py → build/bdist.linux-x86_64/egg/apex/multi_tensor_apply<br>
creating build/bdist.linux-x86_64/egg/apex/pyprof<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/<strong>init</strong>.py → build/bdist.linux-x86_64/egg/apex/pyprof<br>
creating build/bdist.linux-x86_64/egg/apex/pyprof/prof<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/prof/utility.py → build/bdist.linux-x86_64/egg/apex/pyprof/prof<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/prof/recurrentCell.py → build/bdist.linux-x86_64/egg/apex/pyprof/prof<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/prof/normalization.py → build/bdist.linux-x86_64/egg/apex/pyprof/prof<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/prof/data.py → build/bdist.linux-x86_64/egg/apex/pyprof/prof<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/prof/<strong>main</strong>.py → build/bdist.linux-x86_64/egg/apex/pyprof/prof<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/prof/<strong>init</strong>.py → build/bdist.linux-x86_64/egg/apex/pyprof/prof<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/prof/pointwise.py → build/bdist.linux-x86_64/egg/apex/pyprof/prof<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/prof/linear.py → build/bdist.linux-x86_64/egg/apex/pyprof/prof<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/prof/base.py → build/bdist.linux-x86_64/egg/apex/pyprof/prof<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/prof/usage.py → build/bdist.linux-x86_64/egg/apex/pyprof/prof<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/prof/convert.py → build/bdist.linux-x86_64/egg/apex/pyprof/prof<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/prof/randomSample.py → build/bdist.linux-x86_64/egg/apex/pyprof/prof<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/prof/activation.py → build/bdist.linux-x86_64/egg/apex/pyprof/prof<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/prof/output.py → build/bdist.linux-x86_64/egg/apex/pyprof/prof<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/prof/optim.py → build/bdist.linux-x86_64/egg/apex/pyprof/prof<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/prof/misc.py → build/bdist.linux-x86_64/egg/apex/pyprof/prof<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/prof/reduction.py → build/bdist.linux-x86_64/egg/apex/pyprof/prof<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/prof/pooling.py → build/bdist.linux-x86_64/egg/apex/pyprof/prof<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/prof/embedding.py → build/bdist.linux-x86_64/egg/apex/pyprof/prof<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/prof/index_slice_join_mutate.py → build/bdist.linux-x86_64/egg/apex/pyprof/prof<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/prof/prof.py → build/bdist.linux-x86_64/egg/apex/pyprof/prof<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/prof/blas.py → build/bdist.linux-x86_64/egg/apex/pyprof/prof<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/prof/dropout.py → build/bdist.linux-x86_64/egg/apex/pyprof/prof<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/prof/softmax.py → build/bdist.linux-x86_64/egg/apex/pyprof/prof<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/prof/loss.py → build/bdist.linux-x86_64/egg/apex/pyprof/prof<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/prof/conv.py → build/bdist.linux-x86_64/egg/apex/pyprof/prof<br>
creating build/bdist.linux-x86_64/egg/apex/pyprof/nvtx<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/nvtx/<strong>init</strong>.py → build/bdist.linux-x86_64/egg/apex/pyprof/nvtx<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/nvtx/nvmarker.py → build/bdist.linux-x86_64/egg/apex/pyprof/nvtx<br>
creating build/bdist.linux-x86_64/egg/apex/pyprof/parse<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/parse/db.py → build/bdist.linux-x86_64/egg/apex/pyprof/parse<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/parse/<strong>main</strong>.py → build/bdist.linux-x86_64/egg/apex/pyprof/parse<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/parse/<strong>init</strong>.py → build/bdist.linux-x86_64/egg/apex/pyprof/parse<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/parse/parse.py → build/bdist.linux-x86_64/egg/apex/pyprof/parse<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/parse/kernel.py → build/bdist.linux-x86_64/egg/apex/pyprof/parse<br>
copying build/lib.linux-x86_64-3.7/apex/pyprof/parse/nvvp.py → build/bdist.linux-x86_64/egg/apex/pyprof/parse<br>
creating build/bdist.linux-x86_64/egg/apex/RNN<br>
copying build/lib.linux-x86_64-3.7/apex/RNN/<strong>init</strong>.py → build/bdist.linux-x86_64/egg/apex/RNN<br>
copying build/lib.linux-x86_64-3.7/apex/RNN/RNNBackend.py → build/bdist.linux-x86_64/egg/apex/RNN<br>
copying build/lib.linux-x86_64-3.7/apex/RNN/cells.py → build/bdist.linux-x86_64/egg/apex/RNN<br>
copying build/lib.linux-x86_64-3.7/apex/RNN/models.py → build/bdist.linux-x86_64/egg/apex/RNN<br>
creating build/bdist.linux-x86_64/egg/apex/normalization<br>
copying build/lib.linux-x86_64-3.7/apex/normalization/<strong>init</strong>.py → build/bdist.linux-x86_64/egg/apex/normalization<br>
copying build/lib.linux-x86_64-3.7/apex/normalization/fused_layer_norm.py → build/bdist.linux-x86_64/egg/apex/normalization<br>
creating build/bdist.linux-x86_64/egg/apex/reparameterization<br>
copying build/lib.linux-x86_64-3.7/apex/reparameterization/weight_norm.py → build/bdist.linux-x86_64/egg/apex/reparameterization<br>
copying build/lib.linux-x86_64-3.7/apex/reparameterization/<strong>init</strong>.py → build/bdist.linux-x86_64/egg/apex/reparameterization<br>
copying build/lib.linux-x86_64-3.7/apex/reparameterization/reparameterization.py → build/bdist.linux-x86_64/egg/apex/reparameterization<br>
creating build/bdist.linux-x86_64/egg/apex/fp16_utils<br>
copying build/lib.linux-x86_64-3.7/apex/fp16_utils/<strong>init</strong>.py → build/bdist.linux-x86_64/egg/apex/fp16_utils<br>
copying build/lib.linux-x86_64-3.7/apex/fp16_utils/loss_scaler.py → build/bdist.linux-x86_64/egg/apex/fp16_utils<br>
copying build/lib.linux-x86_64-3.7/apex/fp16_utils/fp16util.py → build/bdist.linux-x86_64/egg/apex/fp16_utils<br>
copying build/lib.linux-x86_64-3.7/apex/fp16_utils/fp16_optimizer.py → build/bdist.linux-x86_64/egg/apex/fp16_utils<br>
copying build/lib.linux-x86_64-3.7/apex_C.cpython-37m-x86_64-linux-gnu.so → build/bdist.linux-x86_64/egg<br>
copying build/lib.linux-x86_64-3.7/syncbn.cpython-37m-x86_64-linux-gnu.so → build/bdist.linux-x86_64/egg<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/<strong>init</strong>.py to <strong>init</strong>.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/parallel/multiproc.py to multiproc.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/parallel/<strong>init</strong>.py to <strong>init</strong>.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/parallel/LARC.py to LARC.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/parallel/distributed.py to distributed.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/parallel/sync_batchnorm_kernel.py to sync_batchnorm_kernel.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/parallel/optimized_sync_batchnorm_kernel.py to optimized_sync_batchnorm_kernel.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/parallel/optimized_sync_batchnorm.py to optimized_sync_batchnorm.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/parallel/sync_batchnorm.py to sync_batchnorm.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/contrib/<strong>init</strong>.py to <strong>init</strong>.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/contrib/optimizers/<strong>init</strong>.py to <strong>init</strong>.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/contrib/optimizers/fused_adam.py to fused_adam.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/contrib/optimizers/fused_sgd.py to fused_sgd.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/contrib/optimizers/fp16_optimizer.py to fp16_optimizer.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/contrib/xentropy/<strong>init</strong>.py to <strong>init</strong>.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/contrib/xentropy/softmax_xentropy.py to softmax_xentropy.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/contrib/groupbn/<strong>init</strong>.py to <strong>init</strong>.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/contrib/groupbn/batch_norm.py to batch_norm.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/optimizers/fused_novograd.py to fused_novograd.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/optimizers/<strong>init</strong>.py to <strong>init</strong>.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/optimizers/fused_adam.py to fused_adam.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/optimizers/fused_lamb.py to fused_lamb.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/optimizers/fused_sgd.py to fused_sgd.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/amp/scaler.py to scaler.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/amp/compat.py to compat.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/amp/<strong>init</strong>.py to <strong>init</strong>.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/amp/<strong>version</strong>.py to <strong>version</strong>.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/amp/_process_optimizer.py to _process_optimizer.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/amp/handle.py to handle.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/amp/amp.py to amp.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/amp/_amp_state.py to _amp_state.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/amp/_initialize.py to _initialize.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/amp/lists/tensor_overrides.py to tensor_overrides.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/amp/lists/<strong>init</strong>.py to <strong>init</strong>.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/amp/lists/torch_overrides.py to torch_overrides.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/amp/lists/functional_overrides.py to functional_overrides.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/amp/rnn_compat.py to rnn_compat.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/amp/opt.py to opt.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/amp/frontend.py to frontend.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/amp/wrap.py to wrap.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/amp/utils.py to utils.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/multi_tensor_apply/<strong>init</strong>.py to <strong>init</strong>.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/multi_tensor_apply/multi_tensor_apply.py to multi_tensor_apply.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/<strong>init</strong>.py to <strong>init</strong>.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/prof/utility.py to utility.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/prof/recurrentCell.py to recurrentCell.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/prof/normalization.py to normalization.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/prof/data.py to data.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/prof/<strong>main</strong>.py to <strong>main</strong>.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/prof/<strong>init</strong>.py to <strong>init</strong>.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/prof/pointwise.py to pointwise.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/prof/linear.py to linear.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/prof/base.py to base.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/prof/usage.py to usage.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/prof/convert.py to convert.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/prof/randomSample.py to randomSample.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/prof/activation.py to activation.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/prof/output.py to output.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/prof/optim.py to optim.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/prof/misc.py to misc.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/prof/reduction.py to reduction.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/prof/pooling.py to pooling.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/prof/embedding.py to embedding.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/prof/index_slice_join_mutate.py to index_slice_join_mutate.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/prof/prof.py to prof.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/prof/blas.py to blas.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/prof/dropout.py to dropout.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/prof/softmax.py to softmax.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/prof/loss.py to loss.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/prof/conv.py to conv.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/nvtx/<strong>init</strong>.py to <strong>init</strong>.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/nvtx/nvmarker.py to nvmarker.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/parse/db.py to db.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/parse/<strong>main</strong>.py to <strong>main</strong>.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/parse/<strong>init</strong>.py to <strong>init</strong>.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/parse/parse.py to parse.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/parse/kernel.py to kernel.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/pyprof/parse/nvvp.py to nvvp.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/RNN/<strong>init</strong>.py to <strong>init</strong>.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/RNN/RNNBackend.py to RNNBackend.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/RNN/cells.py to cells.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/RNN/models.py to models.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/normalization/<strong>init</strong>.py to <strong>init</strong>.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/normalization/fused_layer_norm.py to fused_layer_norm.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/reparameterization/weight_norm.py to weight_norm.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/reparameterization/<strong>init</strong>.py to <strong>init</strong>.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/reparameterization/reparameterization.py to reparameterization.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/fp16_utils/<strong>init</strong>.py to <strong>init</strong>.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/fp16_utils/loss_scaler.py to loss_scaler.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/fp16_utils/fp16util.py to fp16util.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/apex/fp16_utils/fp16_optimizer.py to fp16_optimizer.cpython-37.pyc<br>
creating stub loader for apex_C.cpython-37m-x86_64-linux-gnu.so<br>
creating stub loader for amp_C.cpython-37m-x86_64-linux-gnu.so<br>
creating stub loader for syncbn.cpython-37m-x86_64-linux-gnu.so<br>
creating stub loader for fused_layer_norm_cuda.cpython-37m-x86_64-linux-gnu.so<br>
byte-compiling build/bdist.linux-x86_64/egg/apex_C.py to apex_C.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/amp_C.py to amp_C.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/syncbn.py to syncbn.cpython-37.pyc<br>
byte-compiling build/bdist.linux-x86_64/egg/fused_layer_norm_cuda.py to fused_layer_norm_cuda.cpython-37.pyc<br>
creating build/bdist.linux-x86_64/egg/EGG-INFO<br>
copying apex.egg-info/PKG-INFO → build/bdist.linux-x86_64/egg/EGG-INFO<br>
copying apex.egg-info/SOURCES.txt → build/bdist.linux-x86_64/egg/EGG-INFO<br>
copying apex.egg-info/dependency_links.txt → build/bdist.linux-x86_64/egg/EGG-INFO<br>
copying apex.egg-info/top_level.txt → build/bdist.linux-x86_64/egg/EGG-INFO<br>
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt<br>
zip_safe flag not set; analyzing archive contents…<br>
<strong>pycache</strong>.amp_C.cpython-37: module references <strong>file</strong><br>
<strong>pycache</strong>.apex_C.cpython-37: module references <strong>file</strong><br>
<strong>pycache</strong>.fused_layer_norm_cuda.cpython-37: module references <strong>file</strong><br>
<strong>pycache</strong>.syncbn.cpython-37: module references <strong>file</strong><br>
apex.pyprof.nvtx.<strong>pycache</strong>.nvmarker.cpython-37: module references <strong>file</strong><br>
apex.pyprof.nvtx.<strong>pycache</strong>.nvmarker.cpython-37: module references <strong>path</strong><br>
creating dist<br>
creating ‘dist/apex-0.1-py3.7-linux-x86_64.egg’ and adding ‘build/bdist.linux-x86_64/egg’ to it<br>
removing ‘build/bdist.linux-x86_64/egg’ (and everything under it)<br>
Processing apex-0.1-py3.7-linux-x86_64.egg<br>
creating /home/ahmed/anaconda3/lib/python3.7/site-packages/apex-0.1-py3.7-linux-x86_64.egg<br>
Extracting apex-0.1-py3.7-linux-x86_64.egg to /home/ahmed/anaconda3/lib/python3.7/site-packages<br>
Adding apex 0.1 to easy-install.pth file</p>
<p>Installed /home/ahmed/anaconda3/lib/python3.7/site-packages/apex-0.1-py3.7-linux-x86_64.egg<br>
Processing dependencies for apex==0.1<br>
Finished processing dependencies for apex==0.1</p>
</blockquote>
<p>Does this mean its working ? Should i restart my model to check for speedups ? Thank you much sir for your patience. Really appreciated</p>

Bika: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> no speedup. I restarted the notebook and using O1 level. This is maybe the model isnot large ? i am using batch size of 512</p>

ptrblck: <p>Might be. What model are you using at the moment?</p>
<p>Also, it seems you are storing the computation graph in this line of code:</p>
<pre><code class="lang-python">accumulated_loss += loss#scaled_loss
</code></pre>
<p>If you don’t need to call <code>backward</code> on <code>accumulated_loss</code>, make sure to use <code>accumulated_loss += loss.detach()</code>.</p>

Bika: <p>I am using this model.</p>
<blockquote>
<p>class CNN_TC(nn.Module):<br>
def <strong>init</strong>(self,googleNews_vectors,Filters):<br>
super().<strong>init</strong>()<br>
self.embedding = nn.Embedding.from_pretrained(googleNews_vectors,freeze=True)<br>
self.embedding.weight.requires_grad = False<br>
self.conv1 = nn.Conv1d(1,1,kernel_size=(Filters[0] * 300),stride=(300))<br>
self.conv2 = nn.Conv1d(1,1,kernel_size=(Filters[1] * 300),stride=(300))<br>
self.conv3 = nn.Conv1d(1,1,kernel_size=(Filters[2] * 300),stride=(300))</p>
<pre><code>    self.max_conv1 = nn.MaxPool1d(kernel_size=(15))
    self.max_conv2 = nn.MaxPool1d(kernel_size=(11))
    self.max_conv3 = nn.MaxPool1d(kernel_size=(9))
    
    self.FFN1 = nn.Linear(9,100)
    self.FFN2 = nn.Linear(100,2)
    
def forward(self,inputs):
    X = self.embedding(inputs)
    X = X.view(X.shape[0],1,-1).contiguous()

    max_feature1 = self.max_conv1(f.relu(self.conv1(X)))
    max_feature2 = self.max_conv2(f.relu(self.conv2(X)))
    max_feature3 = self.max_conv3(f.relu(self.conv3(X)))
    features = torch.cat((max_feature1,max_feature2,max_feature3), -1).squeeze(0)
                         
    output = self.FFN2(f.relu(self.FFN1(features)))
    return output
</code></pre>
</blockquote>

ptrblck: <p>Could you post the shape of your input so that we could profile it, please?</p>

Bika: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> the shape of the input is [512,46] , you can remove the embedding layer and make it [512,46,300] if you dont want to download the google neg 300</p>

ptrblck: <p><code>[512, 46, 300]</code> doesn’t work, as the first conv expects the input to have a single input channel.<br>
If I use a single input channel, the activation is too small after a certain layer:</p>
<pre><code class="lang-python">Given input size: (1x1x1). Calculated output size: (1x1x0). Output size is too small
</code></pre>
<p>Using an input of <code>x = torch.randint(0, 100, (512, 46))</code> and initializing the embedding via <code>nn.Embedding(100, 300)</code>, yields:</p>
<pre><code class="lang-python">size mismatch, m1: [512 x 12], m2: [9 x 100] at /opt/conda/conda-bld/pytorch_1581149526594/work/aten/src/TH/generic/THTensorMath.cpp:41
</code></pre>
<p>Could you post an executable code snippet, please?</p>

Bika: <p>The second line in the forward function transforms the input to [512,1, 46*300],</p>
<aside class="quote no-group" data-username="Bika" data-post="10" data-topic="68981">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bika/48/20326_2.png" class="avatar"> Bika:</div>
<blockquote>
<p>X = X.view(X.shape[0],1,-1).contiguous()</p>
</blockquote>
</aside>

ptrblck: <p>After fixing an issue with <code>squeeze</code> (changed it to <code>.squeeze(1)</code>) I get 742.706us for a single forward pass. Since the time is really low due to the small workload, you won’t see any speedup unfortunately.</p>
<p>If your workload is a bit higher (e.g. resnet50), you should see a speedup of approx. 2x.</p>

Bika: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thank you , i will try it then. Should i use O1, or O3 for the opt_lvl ? I mean does fp16 would train faster or not always the case ?</p>

ptrblck: <p>You should use <code>opt_level='O1'</code> as this is the safest mixed-precision mode.<br>
For cudnn&gt;=7.3 you should see a speedup for convolutions, while matrix multiplications require a shape of multiples of 8 to be executed on the TensorCores (<a href="https://github.com/NVIDIA/apex/issues/221">post with some details</a>).</p>

Bika: <p>My tensor cores wont work for anything not multiple of 8 ? I thought that they are triggered on their own. And for the cudnn is there is a certain command to trigger it to use the tensor cores or just multiple of 8 ?<br>
To check if i understand this correctly , if i make the embeddings 256 instead of 300 , that means that all the conv layers are 3x256,4x256 … which is a multiple of 8 , is this correct ? And does this contribute in the speedup ?</p>

ptrblck: <p>Convolutions do not have the shape requirements starting with cudnn7.3, while matrix multiplications still have it.<br>
Embedding layers are a lookup table so the dense embedded output will be used in the next layers, no?</p>

Bika: <p>I mean that when i change the embedding size to 256 , the first conv1d layer matrix operation would multiply [256*3] with the 3 words from the 46 length sequence with embedding 256 , so the whole operation is a multiple of 8 , or i am still not getting it right ? or what do you recommend to change to benefit from the tensor cores speed up</p>