Torchvision's inception_v3 takes much longer to load than other models

daveboat: <p>If I try to load resnet50 after downloading the pretrained weights, with the code</p>
<pre><code class="lang-auto">start = time.time()
resnet = torchvision.models.resnet50(pretrained=True)
print('resnet50 loaded in {} seconds'.format(time.time()-start))
</code></pre>
<p>I get</p>
<pre><code class="lang-auto">resnet50 loaded in 0.41980624198913574 seconds
</code></pre>
<p>and similar times for vgg, densenet, etc. However, for inception_v3, I get</p>
<pre><code class="lang-auto">inception_v3 loaded in 184.83905911445618 seconds
</code></pre>
<p>Is there a reason inception takes so much longer to load?</p>

ptrblck: <p>We had recently the same issue <a href="https://discuss.pytorch.org/t/loading-inceptionv3-taking-a-lot-of-time/68067">here</a>.<br>
Do you see the same loading time when you use <code>pretrained=False</code>?</p>

daveboat: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="68756">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>We had recently the same issue <a href="https://discuss.pytorch.org/t/loading-inceptionv3-taking-a-lot-of-time/68067">here </a>.<br>
Do you see the same loading time when you use <code>pretrained=False</code> ?</p>
</blockquote>
</aside>
<p>Yes, the times are identical with and without pretrained. I’m using the latest PyPI versions of torch (1.4.0) and torchvision (0.5.0), and it happens on multiple machines.</p>

daveboat: <p>I did a tiny bit of digging, and the slowdown is 99% due to a single line in Inception3’s initialization:</p>
<pre><code class="lang-auto">values = torch.as_tensor(X.rvs(m.weight.numel()), dtype=m.weight.dtype)
</code></pre>
<p>Specifically, getting the truncated normal distribution samples via <code>X.rvs(m.weight.numel())</code>. Could it be that a change to scipy.stats slowed this down? Also, is it possible to perform the same truncated norm with torch’s built-in tensor operations?</p>

ptrblck: <p>Do you also see the long operating time when you call this single line of code in isolation?</p>

daveboat: <p>If I change the weight initialization to something like</p>
<pre><code class="lang-auto">        for m in self.modules():
            if isinstance(m, nn.Conv2d) or isinstance(m, nn.Linear):
                import scipy.stats as stats
                stddev = m.stddev if hasattr(m, 'stddev') else 0.1
                X = stats.truncnorm(-2, 2, scale=stddev)
                #values = torch.as_tensor(X.rvs(m.weight.numel()), dtype=m.weight.dtype)
                #values = values.view(m.weight.size())
                foo = X.rvs(m.weight.numel())
                values = torch.zeros_like(m.weight.size())
                with torch.no_grad():
                    m.weight.copy_(values)
            elif isinstance(m, nn.BatchNorm2d):
                nn.init.constant_(m.weight, 1)
                nn.init.constant_(m.bias, 0)
</code></pre>
<p>Then I still see the slowdown, which disappears if I remove the X.rvs() call. A single call to X.rvs() doesn’t take a particularly long time, but the loop iterates over ~300 layers.</p>

ptrblck: <p>Thanks for this information!<br>
Could you post the <code>scipy</code> version so that I can reproduce it?<br>
Also, thanks for narrowing down this issue. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

daveboat: <p>I’m using the latest PyPI version of scipy, 1.4.1.</p>

ptrblck: <p>It might be related to <a href="https://github.com/scipy/scipy/issues/11299">this scipy.stats issue</a>.<br>
CC <a class="mention" href="/u/fmassa">@fmassa</a></p>

daveboat: <p>It definitely seems like that scipy.stats issue is the same issue we’re seeing here. In the meantime, I’ve just removed the weight initialization, since I always use the pretrained weights anyways.</p>

vbvg2008: <p>I am also experiencing the same issue in a tensorflow docker container. <a class="mention" href="/u/ptrblck">@ptrblck</a> you can reproduce this issue by the following 3 steps:</p>
<ol>
<li>docker pull tensorflow/tensorflow:2.1.0-gpu-py3</li>
<li>create a docker container of the image, do <code>pip install torch torchvision</code> inside container.</li>
<li>model = torchvision.models.inception_v3()</li>
</ol>
<p>I think tensorflow is irrelevant for this issue, but the version of other packages in the container will be useful for you to find the root cause.</p>

ptrblck: <p>The root cause should be isolated by <a class="mention" href="/u/daveboat">@daveboat</a> and linked to in my last post pointing to a performance regression in <code>rvs</code> calls in <code>scipy</code>.</p>

jitesh: <p>Uninstalling scipy version 1.4 and installing 1.3.3 fixed the issue for me. The loading time is much faster now. Just had to do <code>pip install --upgrade scipy==1.3.3</code></p>

Can_Nguyen: <p><a class="mention" href="/u/jitesh">@jitesh</a> Thanks! That works.</p>

jitesh: <p>No problem. Glad I could help!</p>

tommyjiang: <aside class="quote no-group" data-username="jitesh" data-post="13" data-topic="68756">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jitesh/48/11907_2.png" class="avatar"> jitesh:</div>
<blockquote>
<p>pip install --upgrade scipy==1.3.3</p>
</blockquote>
</aside>
<p>I use <code>scipy=1.2.0</code> to solve this.</p>

jitesh: <p>Yeah, I guess that works as well</p>

Kamil_Adamczewski: <p>Thanks <a class="mention" href="/u/jitesh">@jitesh</a> <a class="mention" href="/u/ptrblck">@ptrblck</a> for engaging in this discussion. What a relief.</p>