Errors while fine-tuning a pretrained model

username: Najeh_Nafti
cooked: <p>I got the below error when I tried to pretrain a model from github on my own dataset:</p>
<pre><code class="lang-auto">File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1224, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for Generator:
	size mismatch for linear.weight: copying a param with shape torch.Size([16384, 128]) from checkpoint, the shape in current model is torch.Size([24576, 17]).
size mismatch for linear.bias: copying a param with shape torch.Size([16384]) from checkpoint, the shape in current model is torch.Size([24576]).
	size mismatch for linear.u0: copying a param with shape torch.Size([1, 16384]) from checkpoint, the shape in current model is torch.Size([1, 24576]).
	size mismatch for blocks.0.0.conv1.weight: copying a param with shape torch.Size([1024, 1024, 3, 3]) from checkpoint, the shape in current model is torch.Size([1536, 1536, 3, 3]).
	size mismatch for blocks.0.0.conv1.bias: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for blocks.0.0.conv1.u0: copying a param with shape torch.Size([1, 1024]) from checkpoint, the shape in current model is torch.Size([1, 1536]).
	size mismatch for blocks.0.0.conv2.weight: copying a param with shape torch.Size([1024, 1024, 3, 3]) from checkpoint, the shape in current model is torch.Size([1536, 1536, 3, 3]).
	size mismatch for blocks.0.0.conv2.bias: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for blocks.0.0.conv2.u0: copying a param with shape torch.Size([1, 1024]) from checkpoint, the shape in current model is torch.Size([1, 1536]).
	size mismatch for blocks.0.0.conv_sc.weight: copying a param with shape torch.Size([1024, 1024, 1, 1]) from checkpoint, the shape in current model is torch.Size([1536, 1536, 1, 1]).
	size mismatch for blocks.0.0.conv_sc.bias: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for blocks.0.0.conv_sc.u0: copying a param with shape torch.Size([1, 1024]) from checkpoint, the shape in current model is torch.Size([1, 1536]).
	size mismatch for blocks.0.0.bn1.stored_mean: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for blocks.0.0.bn1.stored_var: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for blocks.0.0.bn2.stored_mean: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for blocks.0.0.bn2.stored_var: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for blocks.1.0.conv1.weight: copying a param with shape torch.Size([512, 1024, 3, 3]) from checkpoint, the shape in current model is torch.Size([768, 1536, 3, 3]).
	size mismatch for blocks.1.0.conv1.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for blocks.1.0.conv1.u0: copying a param with shape torch.Size([1, 512]) from checkpoint, the shape in current model is torch.Size([1, 768]).
	size mismatch for blocks.1.0.conv2.weight: copying a param with shape torch.Size([512, 512, 3, 3]) from checkpoint, the shape in current model is torch.Size([768, 768, 3, 3]).
	size mismatch for blocks.1.0.conv2.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for blocks.1.0.conv2.u0: copying a param with shape torch.Size([1, 512]) from checkpoint, the shape in current model is torch.Size([1, 768]).
	size mismatch for blocks.1.0.conv_sc.weight: copying a param with shape torch.Size([512, 1024, 1, 1]) from checkpoint, the shape in current model is torch.Size([768, 1536, 1, 1]).
	size mismatch for blocks.1.0.conv_sc.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for blocks.1.0.conv_sc.u0: copying a param with shape torch.Size([1, 512]) from checkpoint, the shape in current model is torch.Size([1, 768]).
	size mismatch for blocks.1.0.bn1.stored_mean: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for blocks.1.0.bn1.stored_var: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for blocks.1.0.bn2.stored_mean: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for blocks.1.0.bn2.stored_var: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for blocks.2.0.conv1.weight: copying a param with shape torch.Size([512, 512, 3, 3]) from checkpoint, the shape in current model is torch.Size([768, 768, 3, 3]).
	size mismatch for blocks.2.0.conv1.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for blocks.2.0.conv1.u0: copying a param with shape torch.Size([1, 512]) from checkpoint, the shape in current model is torch.Size([1, 768]).
	size mismatch for blocks.2.0.conv2.weight: copying a param with shape torch.Size([512, 512, 3, 3]) from checkpoint, the shape in current model is torch.Size([768, 768, 3, 3]).
	size mismatch for blocks.2.0.conv2.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for blocks.2.0.conv2.u0: copying a param with shape torch.Size([1, 512]) from checkpoint, the shape in current model is torch.Size([1, 768]).
	size mismatch for blocks.2.0.conv_sc.weight: copying a param with shape torch.Size([512, 512, 1, 1]) from checkpoint, the shape in current model is torch.Size([768, 768, 1, 1]).
	size mismatch for blocks.2.0.conv_sc.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for blocks.2.0.conv_sc.u0: copying a param with shape torch.Size([1, 512]) from checkpoint, the shape in current model is torch.Size([1, 768]).
	size mismatch for blocks.2.0.bn1.stored_mean: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for blocks.2.0.bn1.stored_var: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for blocks.2.0.bn2.stored_mean: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for blocks.2.0.bn2.stored_var: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for blocks.3.0.conv1.weight: copying a param with shape torch.Size([256, 512, 3, 3]) from checkpoint, the shape in current model is torch.Size([384, 768, 3, 3]).
	size mismatch for blocks.3.0.conv1.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([384]).
	size mismatch for blocks.3.0.conv1.u0: copying a param with shape torch.Size([1, 256]) from checkpoint, the shape in current model is torch.Size([1, 384]).
	size mismatch for blocks.3.0.conv2.weight: copying a param with shape torch.Size([256, 256, 3, 3]) from checkpoint, the shape in current model is torch.Size([384, 384, 3, 3]).
	size mismatch for blocks.3.0.conv2.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([384]).
	size mismatch for blocks.3.0.conv2.u0: copying a param with shape torch.Size([1, 256]) from checkpoint, the shape in current model is torch.Size([1, 384]).
	size mismatch for blocks.3.0.conv_sc.weight: copying a param with shape torch.Size([256, 512, 1, 1]) from checkpoint, the shape in current model is torch.Size([384, 768, 1, 1]).
	size mismatch for blocks.3.0.conv_sc.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([384]).
	size mismatch for blocks.3.0.conv_sc.u0: copying a param with shape torch.Size([1, 256]) from checkpoint, the shape in current model is torch.Size([1, 384]).
	size mismatch for blocks.3.0.bn1.stored_mean: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for blocks.3.0.bn1.stored_var: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for blocks.3.0.bn2.stored_mean: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([384]).
	size mismatch for blocks.3.0.bn2.stored_var: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([384]).
	size mismatch for blocks.4.0.conv1.weight: copying a param with shape torch.Size([128, 256, 3, 3]) from checkpoint, the shape in current model is torch.Size([192, 384, 3, 3]).
	size mismatch for blocks.4.0.conv1.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([192]).
	size mismatch for blocks.4.0.conv1.u0: copying a param with shape torch.Size([1, 128]) from checkpoint, the shape in current model is torch.Size([1, 192]).
	size mismatch for blocks.4.0.conv2.weight: copying a param with shape torch.Size([128, 128, 3, 3]) from checkpoint, the shape in current model is torch.Size([192, 192, 3, 3]).
	size mismatch for blocks.4.0.conv2.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([192]).
	size mismatch for blocks.4.0.conv2.u0: copying a param with shape torch.Size([1, 128]) from checkpoint, the shape in current model is torch.Size([1, 192]).
	size mismatch for blocks.4.0.conv_sc.weight: copying a param with shape torch.Size([128, 256, 1, 1]) from checkpoint, the shape in current model is torch.Size([192, 384, 1, 1]).
	size mismatch for blocks.4.0.conv_sc.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([192]).
	size mismatch for blocks.4.0.conv_sc.u0: copying a param with shape torch.Size([1, 128]) from checkpoint, the shape in current model is torch.Size([1, 192]).
	size mismatch for blocks.4.0.bn1.stored_mean: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([384]).
	size mismatch for blocks.4.0.bn1.stored_var: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([384]).
	size mismatch for blocks.4.0.bn2.stored_mean: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([192]).
	size mismatch for blocks.4.0.bn2.stored_var: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([192]).
	size mismatch for blocks.5.0.conv1.weight: copying a param with shape torch.Size([64, 128, 3, 3]) from checkpoint, the shape in current model is torch.Size([96, 192, 3, 3]).
	size mismatch for blocks.5.0.conv1.bias: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([96]).
	size mismatch for blocks.5.0.conv1.u0: copying a param with shape torch.Size([1, 64]) from checkpoint, the shape in current model is torch.Size([1, 96]).
	size mismatch for blocks.5.0.conv2.weight: copying a param with shape torch.Size([64, 64, 3, 3]) from checkpoint, the shape in current model is torch.Size([96, 96, 3, 3]).
	size mismatch for blocks.5.0.conv2.bias: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([96]).
	size mismatch for blocks.5.0.conv2.u0: copying a param with shape torch.Size([1, 64]) from checkpoint, the shape in current model is torch.Size([1, 96]).
	size mismatch for blocks.5.0.conv_sc.weight: copying a param with shape torch.Size([64, 128, 1, 1]) from checkpoint, the shape in current model is torch.Size([96, 192, 1, 1]).
	size mismatch for blocks.5.0.conv_sc.bias: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([96]).
	size mismatch for blocks.5.0.conv_sc.u0: copying a param with shape torch.Size([1, 64]) from checkpoint, the shape in current model is torch.Size([1, 96]).
	size mismatch for blocks.5.0.bn1.stored_mean: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([192]).
	size mismatch for blocks.5.0.bn1.stored_var: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([192]).
	size mismatch for blocks.5.0.bn2.stored_mean: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([96]).
	size mismatch for blocks.5.0.bn2.stored_var: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([96]).
	size mismatch for output_layer.0.gain: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([96]).
	size mismatch for output_layer.0.bias: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([96]).
	size mismatch for output_layer.0.stored_mean: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([96]).
	size mismatch for output_layer.0.stored_var: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([96]).
	size mismatch for output_layer.2.weight: copying a param with shape torch.Size([3, 64, 3, 3]) from checkpoint, the shape in current model is torch.Size([3, 96, 3, 3]).

</code></pre>

username: Dwight_Foster
cooked: <p>Can you show your code. And are you loading the correct model because this error is telling you that you aren’t.</p>

username: Shima_Shahfar
cooked: <p>Is the dimensions of the network you are creating the same as the one for the network you are using?</p>

username: Najeh_Nafti
cooked: <p>I’m fine tuning the unetgan model which is a combination between bigGAN and Unet:<br>
<a href="https://github.com/boschresearch/unetgan" rel="noopener nofollow ugc">https://github.com/boschresearch/unetgan</a></p>
<p>and I am loading the correct model</p>

username: Najeh_Nafti
cooked: <p>yes the data have the same dimension</p>

username: Shima_Shahfar
cooked: <p>That’s odd because the error shows the dimension of the network you’ve created is different from the one you are loading. And if you didn’t change the architecture or hard-coded any thing related to the network, you should at least be able to load the weights.<br>
Would you please try to use their own <code>load_weights</code> function and see if you can load the weights? You can find it in  <code> utils.py</code>.</p>

username: Najeh_Nafti
cooked: <p>I think i don’t have the same dimension, how can i verify that?</p>

username: Najeh_Nafti
cooked: <p>No I can not load the weights</p>

username: Shima_Shahfar
cooked: <p>Can you please share your code for the part you are loading the weights and initiating the model?</p>

username: Najeh_Nafti
cooked: <p>this is the used load_weight function:</p>
<pre><code class="lang-auto">def load_weights(G, D, state_dict, weights_root, experiment_name,config, epoch_id = '',
                 name_suffix=None, G_ema=None, strict=True, load_optim=True):
  #"/dlc/Employees/esc2rng/biggan_abst/56932_1708_c_ctrl/weights"
  #root = '/'.join([weights_root, experiment_name])
  #root = "/dlc/Employees/esc2rng/biggan_abst/56932_1708_c_ctrl/weights/BigGAN_coco_seed0_Gch64_Dch64_bs64_Glr5.0e-05_Dlr2.0e-04_Gnlrelu_Dnlrelu_Ginitortho_Dinitortho_Gattn64_Dattn64/"
  root = config["resume_from"]
  if name_suffix:
    print('Loading %s weights from %s...' % (name_suffix, root))
  else:
    print('Loading weights from %s...' % root)
  print("epoch id : ", epoch_id)
  if G is not None:
    G.load_state_dict(
      torch.load('%s/%s.pth' % (root, join_strings('_', ['G', epoch_id,  name_suffix]))),
      strict=strict)
    if load_optim:
      s = torch.load('%s/%s.pth' % (root, join_strings('_', ['G_optim', epoch_id,  name_suffix])))
      print("&gt;&gt;" , len(s))
      #print(s)
      G.optim.load_state_dict(
        torch.load('%s/%s.pth' % (root, join_strings('_', ['G_optim', epoch_id,  name_suffix]))))
  if D is not None:
    D.load_state_dict(
      torch.load('%s/%s.pth' % (root, join_strings('_', ['D', epoch_id,  name_suffix]))),
      strict=strict)
    if load_optim:
      D.optim.load_state_dict(
        torch.load('%s/%s.pth' % (root, join_strings('_', ['D_optim', epoch_id,  name_suffix]))))
  # Load state dict
  for item in state_dict:
    D = torch.load('%s/%s.pth' % (root, join_strings('_', ['state_dict', epoch_id,  name_suffix])))
    if item in D:
      state_dict[item] = D[item]
    else:
      print(item, " not in state_dict, creating it ...")
      state_dict[item] = []

  if G_ema is not None:
    G_ema.load_state_dict(
      torch.load('%s/%s.pth' % (root, join_strings('_', ['G_ema', epoch_id, name_suffix]))),
      strict=strict)

</code></pre>

username: Shima_Shahfar
cooked: <p>Yes, I saw this in the Github repo you shared. I meant how do you create your model and call this function when you are trying to load the weights.</p>

username: Najeh_Nafti
cooked: <p>After define the dataloader in train.py file I used this code</p>
<pre><code class="lang-auto">!python train.py \
--dataset mydata   \
--which_best FID \
--batch_size 16 --num_G_accumulations 1 --num_D_accumulations 1 \
--num_D_steps 1 --G_lr 5e-5 --D_lr 2e-4 --D_B2 0.999 --G_B2 0.999 \
--G_attn 0 --D_attn 0 \
--SN_eps 1e-6 --BN_eps 1e-5 --adam_eps 1e-6 \
--G_ortho 0.0 \
--seed 99 \
--G_init ortho --D_init ortho \
--G_eval_mode \
--G_ch 64 --D_ch 64 \
--hier --dim_z 128 \
--ema --use_ema --ema_start 21000 \
--accumulate_stats --num_standing_accumulations 100  \
--test_every 10000 --save_every 10000 --num_best_copies 2 --num_save_copies 1 --seed 0 \
--sample_every 4000   \
--id mydata_unet_bce_noatt_cutmix_consist --gpus "0,1"  \
--unconditional --warmup_epochs 20 \
--unet_mixup --consistency_loss_and_augmentation \
--base_root /content/drive/MyDrive/unetgan/results \
--data_folder /content/drive/MyDrive/mydata \
--resume_from /content/drive/MyDrive/unetgan/pretrained_model  --resume --epoch_id ep_82
</code></pre>

username: Shima_Shahfar
cooked: <p>First of all, you don’t have to pass all the parameters when you are using the default value.<br>
I hope you solved it by now but I suggest try loading the pre-trained weights for a dataset they trained on and not your own dataset and see if it works. Then if it was working fine, it’s probably because of the first issue I mentioned. Usually, the dimensions of feature maps will be defined using the size of your input to prevent hard-coding for different input sizes. But I think the output of your dataset have a different dimensions from the one used in pre-training. You can check it by creating an object from your dataset class and just printing the shape of a sample.</p>

username: Najeh_Nafti
cooked: <p>It works for one of the datasets, so could i change the content of folders of the worked dataset by my own data and fine-tune the model using the pretrained weights?</p>

username: Shima_Shahfar
cooked: <p>No, most probably it won’t work that way. Just make the output of your dataset to have the same shape and formatting as the output in that dataset.</p>