Repository: dmlc/gluon-nlp

Issue:
Conflict between weight tied and weight sharing
## Description
This bug is the same as https://github.com/apache/incubator-mxnet/issues/17184 It is an error introduced in gluonnlp models instead of mxnet side, so I move the issue here.

We experience some weight initialization error when we use weight sharing and weight tied simultaneously. We share weights between `model` and `model_eval`. The code is shown below:

```python
model = nlp.model.train.AWDRNN(args.model, len(vocab), args.emsize, args.nhid, args.nlayers,
                               args.tied, args.dropout, args.weight_dropout,
                               args.dropout_h, args.dropout_i, args.dropout_e)
model_eval = nlp.model.AWDRNN(args.model, len(vocab), args.emsize, args.nhid, args.nlayers,
                              args.tied, args.dropout, args.weight_dropout,
                              args.dropout_h, args.dropout_i, args.dropout_e,
                              params=model.collect_params())

model.initialize(mx.init.Xavier(), ctx=context)

model.hybridize(static_alloc=True)

print(model)

def check_initialized(net):
    params = net.collect_params()
    for param in params:
        try:
            params[param].list_ctx()
        except RuntimeError:
            return False
    return True

print(check_initialized(model))
print(check_initialized(model_eval))
```

### Log Message
If `args.tied` is set `True`, we get the following log message:
```python
True
False
```
If we turn off `args.tied`, the initialization works correctly.

### To Reproduce
(If you developed your own code, please provide a short script that reproduces the error. For existing examples, please provide link.)
The file can be found in (https://github.com/dmlc/gluon-nlp/blob/v0.8.x/scripts/language_model/word_language_model.py). To reproduce the above message, you may need to replace line 153 onward  with the above code snippet. Run the following command:

```
python -m pdb word_language_model.py --tied --dropout_e=0
```
You will encounter the above error.

## What have you tried to solve it?

The parameter that not initialized properly is the parameter `awdrnn0_hybridsequential0_embedding0_bias`. It is the weight used in the decoder of AWDRNN.  After some investigation, we found it is the tied weights introducing this error:

```python
    if self._tie_weights:
         output.add(nn.Dense(self._vocab_size, flatten=False,
                             params=self.embedding[0].params))
```

I print some debug information which may be helpful here:
```
(Pdb) model_eval.decoder[0]._params['awdrnn0_hybridsequential0_embedding0_bias'].list_ctx()
*** RuntimeError: Parameter 'awdrnn0_hybridsequential0_embedding0_bias' has not been initialized
```

## Environment

We recommend using our script for collecting the diagnositc information. Run the following command and paste the outputs below:
```
curl --retry 10 -s https://raw.githubusercontent.com/dmlc/gluon-nlp/master/tools/diagnose.py | python

# paste outputs here
```



Discussion:

Resolving patch:
diff --git a/src/gluonnlp/model/train/language_model.py b/src/gluonnlp/model/train/language_model.py
--- a/src/gluonnlp/model/train/language_model.py
+++ b/src/gluonnlp/model/train/language_model.py
@@ -18,7 +18,7 @@
 __all__ = ['AWDRNN', 'StandardRNN', 'BigRNN']
 
 from mxnet import init, nd, autograd
-from mxnet.gluon import nn, Block, HybridBlock, contrib, rnn
+from mxnet.gluon import nn, Block, HybridBlock, contrib, rnn, ParameterDict
 from mxnet import sym
 
 from ..utils import _get_rnn_layer, apply_weight_drop
@@ -71,6 +71,9 @@ def __init__(self, mode, vocab_size, embed_size=400, hidden_size=1150, num_layer
         self._drop_e = drop_e
         self._weight_drop = weight_drop
         self._tie_weights = tie_weights
+        self._shared_params = None
+        if 'params' in kwargs:
+            self._shared_params = kwargs['params']
 
         with self.name_scope():
             self.embedding = self._get_embedding()
@@ -103,8 +106,19 @@ def _get_decoder(self):
         output = nn.HybridSequential()
         with output.name_scope():
             if self._tie_weights:
-                output.add(nn.Dense(self._vocab_size, flatten=False,
-                                    params=self.embedding[0].params))
+                if self._shared_params is not None:
+                    # self.embedding[0].params do not contain the bias, it
+                    # may leave the decoder bias uninitialized. We resolve this
+                    # issue by creating a new ParameterDict and stuffing
+                    # every shared params into the ParameterDict.
+                    shared_params = self.embedding[0].params
+                    shared_params = ParameterDict(shared_params.prefix)
+                    shared_params.update(self._shared_params)
+                    output.add(nn.Dense(self._vocab_size, flatten=False,
+                                        params=shared_params))
+                else:
+                    output.add(nn.Dense(self._vocab_size, flatten=False,
+                                        params=self.embedding[0].params))
             else:
                 output.add(nn.Dense(self._vocab_size, flatten=False))
         return output


Test patch: