The best method for module cloning with parameter sharing?

supakjk: Then, what would be the best approach for cloning with parameter sharing? I mean, weights and grads will be automatically shared if we just forward the module with different inputs and dealing with the outputs. However, if there are other non-module variables that I want to share, I am not sure what would be the most elegant way. Thanks!

apaszke: I’d say it really depends on the use case. I can’t think of any general recipe right now.

f10w: <SCODE>def _get_clones(module, N):
    return ModuleList([copy.deepcopy(module) for i in range(N)])
<ECODE> I hope somebody could some up with a solution. Thank you very much in advance for your help!

ptrblck: 

f10w: Sorry my statement was confusing (but still technically correct I think). Actually the biases are not used in the convolution layers and thus they all point to the same object. To detect shared parameters, I use the following function from fairseq: <SCODE>def _catalog_shared_params(module, memo=None, prefix=""):
    """Taken from https://github.com/pytorch/fairseq/blob/main/fairseq/trainer.py
    """
    if memo is None:
        first_call = True
        memo = {}
    else:
        first_call = False
    for name, param in module._parameters.items():
        param_prefix = prefix + ("." if prefix else "") + name
        if param not in memo:
            memo[param] = []
        memo[param].append(param_prefix)
    for name, m in module._modules.items():
        if m is None:
            continue
        submodule_prefix = prefix + ("." if prefix else "") + name
        _catalog_shared_params(m, memo, submodule_prefix)
    if first_call:
        return [x for x in memo.values() if len(x) > 1]
<ECODE> <SCODE>model = torchvision.models.__dict__['resnet50']()
shared_params = _catalog_shared_params(model)
print(f'shared_params:\n{shared_params}')
<ECODE> Output: <SCODE>shared_params:
[['conv1.bias', 'layer1.0.conv1.bias', 'layer1.0.conv2.bias', 'layer1.0.conv3.bias', 'layer1.0.downsample.0.bias', 'layer1.1.conv1.bias', 'layer1.1.conv2.bias', 'layer1.1.conv3.bias', 'layer1.2.conv1.bias', 'layer1.2.conv2.bias', 'layer1.2.conv3.bias', 'layer2.0.conv1.bias', 'layer2.0.conv2.bias', 'layer2.0.conv3.bias', 'layer2.0.downsample.0.bias', 'layer2.1.conv1.bias', 'layer2.1.conv2.bias', 'layer2.1.conv3.bias', 'layer2.2.conv1.bias', 'layer2.2.conv2.bias', 'layer2.2.conv3.bias', 'layer2.3.conv1.bias', 'layer2.3.conv2.bias', 'layer2.3.conv3.bias', 'layer3.0.conv1.bias', 'layer3.0.conv2.bias', 'layer3.0.conv3.bias', 'layer3.0.downsample.0.bias', 'layer3.1.conv1.bias', 'layer3.1.conv2.bias', 'layer3.1.conv3.bias', 'layer3.2.conv1.bias', 'layer3.2.conv2.bias', 'layer3.2.conv3.bias', 'layer3.3.conv1.bias', 'layer3.3.conv2.bias', 'layer3.3.conv3.bias', 'layer3.4.conv1.bias', 'layer3.4.conv2.bias', 'layer3.4.conv3.bias', 'layer3.5.conv1.bias', 'layer3.5.conv2.bias', 'layer3.5.conv3.bias', 'layer4.0.conv1.bias', 'layer4.0.conv2.bias', 'layer4.0.conv3.bias', 'layer4.0.downsample.0.bias', 'layer4.1.conv1.bias', 'layer4.1.conv2.bias', 'layer4.1.conv3.bias', 'layer4.2.conv1.bias', 'layer4.2.conv2.bias', 'layer4.2.conv3.bias']]
<ECODE>

f10w: I’ve just checked again and it appears that after cloning the new biases are also shared, which is expected and thus the above is actually not a good example of use cases. Let me come back to this question in a future when I have another example. Thanks!