FSDP: can not use conv2d?

Ziyu_Huang: <p>Hi! I meet this and meet a bug called:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/9/39b5a4f6e8fa68b3e6476a9af9161a41815bed54.png" data-download-href="https://discuss.pytorch.org/uploads/default/39b5a4f6e8fa68b3e6476a9af9161a41815bed54" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/9/39b5a4f6e8fa68b3e6476a9af9161a41815bed54_2_690x498.png" alt="image" data-base62-sha1="8ewo08lrDgnSXiLu1Mh1uHNJNGs" width="690" height="498" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/9/39b5a4f6e8fa68b3e6476a9af9161a41815bed54_2_690x498.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/9/39b5a4f6e8fa68b3e6476a9af9161a41815bed54_2_1035x747.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/9/39b5a4f6e8fa68b3e6476a9af9161a41815bed54.png 2x" data-dominant-color="101111"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1249×902 110 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1b4a9eac06e4aa357664eceec332a58979bfba2e.png" data-download-href="https://discuss.pytorch.org/uploads/default/1b4a9eac06e4aa357664eceec332a58979bfba2e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b4a9eac06e4aa357664eceec332a58979bfba2e_2_662x500.png" alt="image" data-base62-sha1="3TqLmtPW8It7MxcmtuA04FePxqS" width="662" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b4a9eac06e4aa357664eceec332a58979bfba2e_2_662x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b4a9eac06e4aa357664eceec332a58979bfba2e_2_993x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/b/1b4a9eac06e4aa357664eceec332a58979bfba2e.png 2x" data-dominant-color="141415"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1196×902 118 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Could anyone kindly help me? Thank you!!!</p>

ptrblck: <p>It seems <code>PatchEmbeddingSTD</code> is causing the error as apparently its <code>.weight</code> attribute is missing while a module expects it. Could you check where this module is coming from and how its <code>self.weight</code> attribute is initialized and used?</p>

Ziyu_Huang: <p>I have printed all the parameters out:<br>
For FSDP, it shows only(!) this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/5/5540d4803e2089efca2ff2fdf4e7958a9b02ddf4.png" data-download-href="https://discuss.pytorch.org/uploads/default/5540d4803e2089efca2ff2fdf4e7958a9b02ddf4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/5540d4803e2089efca2ff2fdf4e7958a9b02ddf4_2_690x324.png" alt="image" data-base62-sha1="cabtVSfW164X3MYCE3F7HPi1paI" width="690" height="324" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/5540d4803e2089efca2ff2fdf4e7958a9b02ddf4_2_690x324.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/5540d4803e2089efca2ff2fdf4e7958a9b02ddf4_2_1035x486.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/5540d4803e2089efca2ff2fdf4e7958a9b02ddf4_2_1380x648.png 2x" data-dominant-color="0E0E0F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1920×902 130 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
For model without FSDP, it shows this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/1/3122866a074871996773bf26c0b2126ad2e5fa4a.png" data-download-href="https://discuss.pytorch.org/uploads/default/3122866a074871996773bf26c0b2126ad2e5fa4a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/3122866a074871996773bf26c0b2126ad2e5fa4a_2_690x324.png" alt="image" data-base62-sha1="70FmFwILbwtguEwnjp0FgSgzgXo" width="690" height="324" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/3122866a074871996773bf26c0b2126ad2e5fa4a_2_690x324.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/3122866a074871996773bf26c0b2126ad2e5fa4a_2_1035x486.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/3122866a074871996773bf26c0b2126ad2e5fa4a_2_1380x648.png 2x" data-dominant-color="0D0D0E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1920×902 107 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/1/611837554e8ae58036732119c9e1364f316512f5.png" data-download-href="https://discuss.pytorch.org/uploads/default/611837554e8ae58036732119c9e1364f316512f5" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/611837554e8ae58036732119c9e1364f316512f5_2_690x324.png" alt="image" data-base62-sha1="dQWcf8po5DCFHRqisPyQFjcOq69" width="690" height="324" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/611837554e8ae58036732119c9e1364f316512f5_2_690x324.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/611837554e8ae58036732119c9e1364f316512f5_2_1035x486.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/611837554e8ae58036732119c9e1364f316512f5_2_1380x648.png 2x" data-dominant-color="0D0D0E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1920×902 49 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
And much more…</p>
<p>I am thinking maybe FSDP itself has some problem? Maybe it does not support embedding?</p>
<p>Thank you!!!</p>

pbelevich: <p><a class="mention" href="/u/yanli_zhao">@Yanli_Zhao</a> could you please take a look?</p>