the different train speed between pip install torch==1.2 and build torch1.2 from source

wuyujiji: <aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/46245" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/46245" target="_blank" rel="noopener nofollow ugc">the different train speed between pip install torch==1.2 and build torch1.2 from source</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-10-13" data-time="12:57:58" data-timezone="UTC">12:57PM - 13 Oct 20 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2020-10-13" data-time="18:43:38" data-timezone="UTC">06:43PM - 13 Oct 20 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/wuyujiji" target="_blank" rel="noopener nofollow ugc">
          <img alt="wuyujiji" src="https://avatars2.githubusercontent.com/u/25579435?v=4" class="onebox-avatar-inline" width="20" height="20">
          wuyujiji
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">❓ Questions and Help
Hi, recently I train a model with pytorch1.2. I found a phenomenon that train speed of each iter...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: build</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">shadow review</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


ptrblck: <p>Which libraries are you using for the custom build and which are used in the <code>1.2</code> binaries?<br>
Also, why are you comparing the speed of such an old PyTorch version?</p>

wuyujiji: <p>the libraries are listed in <a href="https://github.com/pytorch/pytorch/issues/46245" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/46245</a>, the reason of comparing speed is that I want to reproduce the usr’s training speed, his pytorch 1.2 environment is build by pip install , but my only build from source since our internal platform’s limitation.</p>

ptrblck: <p>If you made sure the binary and your local build are equal you could use profiling tools such as NSIGHT or use the built-in profiler in PyTorch.<br>
Also, note that your profiling should synchronize the device before starting and stopping the timer, but I assume you are already familiar with profiling PyTorch ops.</p>

wuyujiji: <p>I have already profiling and saving to timeline.json, the most cost time of each train step is IndexPutBackward op (0.8s vs. 0.2s)</p>