Repository: apple/coremltools

Issue:
PyTorch to CoreML LSTM model conversion v4 fails with ValueError: Layer with name "_lstm_h0_reshaped_expanded" has already been added. Please use a unique name.

Relevance:-----------------------------------------------------------

While there other open issues in the coremltools v4 TOT repo regarding PyTorch trained LSTMs, such as #755 (multi layer LSTM) and #776 (handling of LSTM initials h0/c0), I was looking for workarounds on how to build multi-layer bidirectional LSTMs. Now I ran into a blocking bug with error:

> Layer with name "_lstm_h0_reshaped_expanded" has already been added. Please use a unique name.

In other words, another blocker on the conversion of PyTorch multilayer LSTMs (potential GRU/RNN as well).

On the good side, a little digging in the conversion code leads to potential fix as well (see code and potential solution below) but I need your please to help to assess whether this approach is best, whether the unidirectional LSTM and GRU also need fixes, etc.

Reproducible:-----------------------------------------------------------

Yes

Testcase:-----------------------------------------------------------

Attached. Run e.g. as 
python3 -O ../testLstmTwoLayer.py 
[testLstmTwoLayer.txt](https://github.com/apple/coremltools/files/4967763/testLstmTwoLayer.txt)

Setup:-----------------------------------------------------------

Torch version : 1.5.1
CoreML tools version : TOT  705244e2be26c3fb7881fd7a731d25a55f5e4765 July 23

Log:-----------------------------------------------------------

Torch version : 1.5.1
CoreML tools version : 4.0b1
small_model(
  (lstm1): LSTM(28, 16, batch_first=True, bidirectional=True)
  (lstm2): LSTM(32, 8, batch_first=True, bidirectional=True)
  (fc1): Linear(in_features=16, out_features=11, bias=True)
)
Converting Frontend ==> MIL Ops:  98%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍     | 46/47 [00:00<00:00, 218.23 ops/s]
Running MIL optimization passes: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 13/13 [00:00<00:00, 17.78 passes/s]
Translating MIL ==> MLModel Ops:  69%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████                                                                                 | 22/32 [00:00<00:00, 14936.01 ops/s]
Traceback (most recent call last):
  File "../testLstmTwoLayer.py", line 74, in <module>
    inputs=[ ct.TensorType(name="input1", shape=dummy_input.shape) ],
  File "~/Library/Python/3.7/lib/python/site-packages/coremltools/converters/_converters_entry.py", line 299, in convert
    **kwargs
  File "~/Library/Python/3.7/lib/python/site-packages/coremltools/converters/mil/converter.py", line 122, in _convert
    out = backend_converter(prog, **kwargs)
  File "~/Library/Python/3.7/lib/python/site-packages/coremltools/converters/mil/converter.py", line 72, in __call__
    return load(*args, **kwargs)
  File "~/Library/Python/3.7/lib/python/site-packages/coremltools/converters/mil/backend/nn/load.py", line 239, in load
    prog.functions["main"].outputs,
  File "~/Library/Python/3.7/lib/python/site-packages/coremltools/converters/mil/backend/nn/op_mapping.py", line 50, in convert_ops
    mapper(const_context, builder, op)
  File "~/Library/Python/3.7/lib/python/site-packages/coremltools/converters/mil/backend/nn/op_mapping.py", line 1399, in lstm
    _expand_dim(builder, initial_h + postfix, initial_h, [2, 3, 4])
  File "~/Library/Python/3.7/lib/python/site-packages/coremltools/converters/mil/backend/nn/op_mapping.py", line 208, in _expand_dim
    name=node_name, input_name=input_name, output_name=node_name, axes=axes
  File "~/Library/Python/3.7/lib/python/site-packages/coremltools/models/neural_network/builder.py", line 7056, in add_expand_dims
    spec_layer = self._add_generic_layer(name, [input_name], [output_name])
  File "~/Library/Python/3.7/lib/python/site-packages/coremltools/models/neural_network/builder.py", line 1029, in _add_generic_layer
    % name
ValueError: Layer with name "_lstm_h0_reshaped_expanded" has already been added. Please use a unique name.

Interpretation and potential fix: -----------------------------------------------------------

This is observed with git coremltools, TOT 705244e2be26c3fb7881fd7a731d25a55f5e4765 July 23

The error
> Layer with name "_lstm_h0_reshaped_expanded" has already been added. Please use a unique name.

seems to indicate that the naming scheme in the converter is using a global name scheme.

While the testcase runs fine with ONE layer (set flag twoLayerModelFlag to False) when you run with TWO layers (twoLayerModelFlag=True) ) (or more presumably) the error occurs.

Originally, you have on TOT the use of fixed postfixes like "_expanded" (for h0) and "_expanded2" (for c0) which leads to the issue it seems. When you have 2 layers or more, it seems the same variable name is generated multiple times.

The line 1399 in coremltools/converters/mil/backend/nn/op_mapping.py
points to the code below (The line numbers might be a little off due to my logging lines).

Original code:

def lstm(const_context, builder, op):
...

# Roughly Line 1375 et al
#
    elif direction == "bidirectional":
        # Expand initial_h and initial_c
        _expand_dim(builder, initial_h + "_expanded", initial_h, [2, 3, 4]) # <------------------
         initial_h += "_expanded"
        
        # initial_h may have the same name as initial_c (e.g., same Var)
        _expand_dim(builder, initial_c + "_expanded2", initial_c, [2, 3, 4])  # <------------------
        initial_c += "_expanded2"
		
Therefore, I tried a quick randomization fix along the lines below to generate UNIQUE names every time you handle a new BLSTM layer.

Potential fix:

def lstm(const_context, builder, op):
...

# Roughly Line 1375 et al
#
    elif direction == "bidirectional":
        # Expand initial_h and initial_c
        if 1:
            # New
            import random
            mymax = 10000000 # Better, use sys.maxint
            n = random.randrange(1,mymax)
            postfix = "_expanded_" + str(n)
            
            n2 = random.randrange(1,mymax)
            postfix2 = "_expanded2_" + str(n2)

            mynodename = initial_h + "_expanded"
            print ("nodename " + str(mynodename))
            print ("postfix" + str(postfix))
            print ("postfix2" + str(postfix2))
        else:
			#ORG
            postfix  = "_expanded"
            postfix2 = "_expanded2"
            
        _expand_dim(builder, initial_h + postfix, initial_h, [2, 3, 4])  # <------------------
        initial_h += postfix
        
        # initial_h may have the same name as initial_c (e.g., same Var)
        _expand_dim(builder, initial_c + postfix2, initial_c, [2, 3, 4])  # <------------------
        initial_c += postfix2
		
While the conversion now completes, I have not checked whether "everything" is fine. which is why I ask for your help for a proper fix please. Note we might need a fix for other uniLSTM, GRU or RNN in the same file as well.

As the other issues  #755 (multi layer LSTM) and #776 (handling of initials h0/c0) are still blocking us, it would be an immense help if we could fix this new issue quickly such that I can properly stack multiple BLSTMs. Thanks.

Here is the output WITH the fix above:

python3 -O ../testLstmTwoLayer.py 
WARNING:root:Keras version 2.4.3 detected. Last version known to be fully compatible of Keras is 2.2.4 .
Torch version : 1.5.1
CoreML tools version : 4.0b1
small_model(
  (lstm1): LSTM(28, 16, batch_first=True, bidirectional=True)
  (lstm2): LSTM(32, 8, batch_first=True, bidirectional=True)
  (fc1): Linear(in_features=16, out_features=11, bias=True)
)
Converting Frontend ==> MIL Ops:  98%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍     | 46/47 [00:00<00:00, 213.31 ops/s]
Running MIL optimization passes: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 13/13 [00:00<00:00, 17.83 passes/s]
Translating MIL ==> MLModel Ops:   0%|                                                                                                                                                                                                                                                                               | 0/32 [00:00<?, ? ops/s]
nodename _lstm_h0_reshaped_expanded
postfix_expanded_6035530			# <------------------ randomized
postfix2_expanded2_4582868			# <------------------ randomized
nodename _lstm_h0_reshaped_expanded
postfix_expanded_3955283
postfix2_expanded2_1781011
Translating MIL ==> MLModel Ops: 100%



Discussion:

Resolving patch:
diff --git a/coremltools/converters/mil/backend/nn/op_mapping.py b/coremltools/converters/mil/backend/nn/op_mapping.py
--- a/coremltools/converters/mil/backend/nn/op_mapping.py
+++ b/coremltools/converters/mil/backend/nn/op_mapping.py
@@ -1375,12 +1375,18 @@ def lstm(const_context, builder, op):
         _squeeze(builder, op.outputs[2].name, output_names[2], axes=[0, 3, 4])
 
     elif direction == "bidirectional":
-        # Expand initial_h and initial_c
-        _expand_dim(builder, initial_h + "_expanded", initial_h, [2, 3, 4])
-        initial_h += "_expanded"
+        # Issue #810
+        num_layer = len(builder.layers)
+        initial_h_expand = initial_h + "_expanded" + "_" + str(num_layer)
+        if not (initial_h_expand in set(builder.layers)):
+            _expand_dim(builder, initial_h_expand, initial_h, [2, 3, 4])
+        initial_h = initial_h_expand
+        
         # initial_h may have the same name as initial_c (e.g., same Var)
-        _expand_dim(builder, initial_c + "_expanded2", initial_c, [2, 3, 4])
-        initial_c += "_expanded2"
+        initial_c_expand = initial_c + "_expanded2" + "_" + str(num_layer)
+        if not (initial_c_expand in set(builder.layers)):
+            _expand_dim(builder, initial_c_expand, initial_c, [2, 3, 4])
+        initial_c = initial_c_expand
 
         initial_h_f = initial_h + "_forward"
         initial_h_r = initial_h + "_reverse"


Test patch:
diff --git a/coremltools/converters/mil/frontend/torch/test/test_torch_ops.py b/coremltools/converters/mil/frontend/torch/test/test_torch_ops.py
--- a/coremltools/converters/mil/frontend/torch/test/test_torch_ops.py
+++ b/coremltools/converters/mil/frontend/torch/test/test_torch_ops.py
@@ -491,6 +491,72 @@ def test_lstm_xexception(
                 backend= backend,
             )
 
+# Workaround for GitHub Issue #824
+# i.e. the return h_n/c_n for a converted BLSTM are mangled. 
+# Therefore, just look at output 'y' (for now) which is correct.
+class StripCellAndHidden(nn.Module):
+    def __init__(self,flagReturnTuple_):
+        super(StripCellAndHidden, self).__init__()
+        self.flagReturnTuple = flagReturnTuple_
+
+    def forward(self,x):
+        # Pass tuple, not tensor, to avoid issue in coremltools/converters/mil/frontend/torch/test/testing_utils.py on "if not expected_results:"
+        # Pass tensor when we need input for LSTM #2 as part of nn.Sequential()
+        return tuple(x[0]) if self.flagReturnTuple else x[0] 
+        
+# Check GitHub Issue #810, assume num_layers == 2 and bidirectional == True
+class TestStackedBLSTM:
+    @pytest.mark.parametrize(
+        "input_size, hidden_size, num_layers, bias, batch_first, dropout, bidirectional, backend",
+        itertools.product([7], [5], [2], [True, False], [True, False], [0.3], [True], backends),
+    )
+    def test_lstm(
+        self,
+        input_size,
+        hidden_size,
+        num_layers,
+        bias,
+        batch_first,
+        dropout,
+        bidirectional,
+        backend,
+    ):
+        model = nn.Sequential(
+            nn.LSTM(
+                input_size=input_size,
+                hidden_size=hidden_size,
+                num_layers=1,
+                bias=bias,
+                batch_first=batch_first,
+                dropout=dropout,
+                bidirectional=True),
+            StripCellAndHidden(False),
+            nn.LSTM(
+                input_size=2*hidden_size,
+                hidden_size=hidden_size,
+                num_layers=1,
+                bias=bias,
+                batch_first=batch_first,
+                dropout=dropout,
+                bidirectional=True),
+            StripCellAndHidden(True)
+        )
+
+        SEQUENCE_LENGTH = 3
+        BATCH_SIZE = 2
+
+        num_directions = int(bidirectional) + 1
+
+        # (seq_len, batch, input_size)
+        if batch_first:
+            _input = torch.rand(BATCH_SIZE, SEQUENCE_LENGTH, input_size)
+        else:
+            _input = torch.randn(SEQUENCE_LENGTH, BATCH_SIZE, input_size)
+
+        # Do not use h_0/c_0 input and do not check h_n/c_n output, GitHub Issue #824     
+        expected_results = model(_input)
+
+        run_compare_torch(_input, model, expected_results, input_as_shape=False, backend=backend)
 
 class TestConcat:
     # This tests an edge case where the list of tensors to concatenate only