iteration second is too long 

plant3032: <p>Hi i’m training the CNN model on my new, second local computer(i.e, i have two computer which have same invironment of software and hardware), but the iteration time is much, much longer than first computer (at least three times longer than first computer)</p>
<p>and i found that interval(?) of using CPU and GPU(CUDA) is much longer in this second computer, and i don’t know why this happened. (and…does this situation have anythig to do with that Torch’s DataLoader’s num_workers or pin_memory?, or is this problem is because of the learning rate? i changed learning rate : 1e-4, batch size : 32 to learning rate : 1.25e-5, batch size : 4 in second computer, and nothing is changed without this parameter)</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0a6f2dc7a9af740f01c3144f8224b4829e1361f7.png" alt="image" data-base62-sha1="1uiYoZLTcBuQgABqnSu9bX5m0uP" width="547" height="263"><br>
(in here, Once the GPU has been used, it has not been used for a long time since )<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/4/b46b393113840b4897868ab692e00cb9155480b6.png" alt="problem 2" data-base62-sha1="pK3FwAaIS8bwxLB97ThLztL0VlI" width="539" height="257"><br>
(like this…) <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<p>I mean, at my first computer, the usage on the CPU remained almost the same, and the usage graph on the GPU came out quite frequently(like below picture), like attached photo of below (as attached above, the usage graph appears once and does not appear at all for a long time)<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/2/6264dea0ec9fd878747746bc1eea2cb882bcefc5.png" alt="situation 1" data-base62-sha1="e2qU4caBtGUZYCVv9kIRkE2Tyyp" width="550" height="257"></p>
<p>does anyone have the solution of this situation? or is this situation is common situation?<br>
(additionally, total number of train image is 2600, validation image is 399)</p>
<p>i’m sorry for these stupid question…and Thanks for all your response <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>You might need to profile the code to narrow down where the bottleneck in your new system might be. Without seeing a profile and a timeline one can only guess, which might not be too helpful.</p>

plant3032: <p>Thanks a lot for your advise <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face.png?v=12" title=":smiling_face:" class="emoji" alt=":smiling_face:" loading="lazy" width="20" height="20"> , and i want to ask again which part you need for additional information (I’m not sure which part I should attach because my code is quite separated by function) <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=12" title=":sob:" class="emoji" alt=":sob:" loading="lazy" width="20" height="20"></p>
<p>(plus, second per iteration is increase almost exponentially, like 15s/it(at 10epoch) → 25s/it(at 30epoch) → 636.10s/it(at 201 epoch)… <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face_with_tear.png?v=12" title=":smiling_face_with_tear:" class="emoji" alt=":smiling_face_with_tear:" loading="lazy" width="20" height="20">)</p>

ptrblck: <p>I wasn’t asking for the code as the issue seems to be system-dependent. Instead you should profile both systems using the native PyTorch profiler or e.g. Nsight Systems to check where the difference comes from.</p>

plant3032: <p>Thank you so much for your advice. I’ll check it out step by step in the way you advised me <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face.png?v=12" title=":smiling_face:" class="emoji" alt=":smiling_face:" loading="lazy" width="20" height="20">  <img src="https://discuss.pytorch.org/images/emoji/apple/face_holding_back_tears.png?v=12" title=":face_holding_back_tears:" class="emoji" alt=":face_holding_back_tears:" loading="lazy" width="20" height="20"></p>