Repository: ludwig-ai/ludwig

Issue:
Shape mismatch when introducing multiple levels of dependencies
**Describe the bug**

When introducing multiple levels of dependencies, the shape of the _concatenated hidden states_ does not match the _input size for the dense layer of the output feature_.

In my case, the text output feature `qty_frac` depends on text output feature `summary`, and numerical output feature `qty` in turn depends on `qty_frac`.

I get the following error when running `ludwig train`:
```python-traceback
RuntimeError: mat1 and mat2 shapes cannot be multiplied (6x768 and 512x1)
```



**To Reproduce**

Minimal, reproducible example using bash and docker as only dependencies:
```bash
#!/usr/bin/env bash
FEATURE_LIST=$(
    docker run -i mikefarah/yq -o json -I 0 e '.' - <<EOF
- name: document
  type: text
- name: summary
  type: text
- name: qty_frac
  type: text
- name: qty
  type: numerical
EOF
)

mkdir /tmp/ludwig-debug
docker run \
    -it \
    -v /tmp/ludwig-debug/:/workdir \
    ludwigai/ludwig:nightly \
    synthesize_dataset \
    --features $FEATURE_LIST \
    --dataset_size 10 \
    --output_path /workdir/synthetic_data.csv

cat <<EOF >/tmp/ludwig-debug/config.yml
input_features:
  - name: document
    type: text
    level: word
output_features:
  - name: summary
    type: text
    level: word
    decoder: generator
  - name: qty_frac
    type: text
    level: word
    decoder: generator
    dependencies:
      - summary
  - name: qty
    type: numerical
    dependencies:
      - qty_frac
EOF

docker run \
    -it \
    -v /tmp/ludwig-debug/:/workdir \
    ludwigai/ludwig:nightly \
    train \
    --dataset /workdir/synthetic_data.csv \
    --config_file /workdir/config.yml \
    --output_directory /workdir/results
```

**Expected behavior**

Training starts without error.

**Screenshots**

Excerpt from the traceback:
```python-traceback
  File "/usr/local/lib/python3.7/site-packages/ludwig/features/numerical_feature.py", line 269, in logits
    return self.decoder_obj(hidden)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/ludwig/decoders/generic_decoders.py", line 58, in forward
    return self.dense(inputs)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/ludwig/utils/torch_utils.py", line 212, in forward
    output = torch.squeeze(self.dense(input), dim=-1)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 103, in forward
    return F.linear(input, self.weight, self.bias)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/functional.py", line 1848, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (6x768 and 512x1)
```

**Environment:**

See reproducible example, run in environment with:
- bash: `GNU bash, version 5.0.17(1)-release (x86_64-pc-linux-gnu)`
- docker: `Docker version 20.10.11+azure-3, build dea9396e184290f638ea873c76db7c80efd5a1d2`

The `ludwigai/ludwig:nightly` Docker image was built from main at 89d18365c41c4ded68edd2095349ce4a6caf5d18.



Discussion:
For reference, here's the [related conversation on Slack](https://ludwig-ai.slack.com/archives/C01PN6M2RSM/p1642806615331569).
After debugging, I found out that the `input_size` field of `TextOutputFeature` "qty_frac" is being overwritten after being correctly computed. 

#### Defined multiple inheritance

`SequenceOutputFeature` inherits from `OutputFeature`:
https://github.com/ludwig-ai/ludwig/blob/65b3b1a7cdfc7149a2a8559eef1edbe4d38a3080/ludwig/features/sequence_feature.py#L215

`TextOutputFeature` in turn inherits from `SequenceOutputFeature`:
https://github.com/ludwig-ai/ludwig/blob/65b3b1a7cdfc7149a2a8559eef1edbe4d38a3080/ludwig/features/text_feature.py#L311

#### Modifications to `input_size`

1. In the constructor of `OutputFeature`, the input size is computed correctly based on dependencies:
https://github.com/ludwig-ai/ludwig/blob/65b3b1a7cdfc7149a2a8559eef1edbe4d38a3080/ludwig/features/base_feature.py#L219

2. In the constructor of `SequenceOutputFeature`, it gets overwritten with the default. The computed dependency information is lost.
https://github.com/ludwig-ai/ludwig/blob/65b3b1a7cdfc7149a2a8559eef1edbe4d38a3080/ludwig/features/sequence_feature.py#L230-L232

Resolving patch:
diff --git a/ludwig/features/base_feature.py b/ludwig/features/base_feature.py
--- a/ludwig/features/base_feature.py
+++ b/ludwig/features/base_feature.py
@@ -217,6 +217,7 @@ def __init__(self, feature: Dict[str, Any], other_output_features: Dict[str, "Ou
         logger.debug("  FCStack")
 
         self.input_size = get_input_size_with_dependencies(self.input_size, self.dependencies, other_output_features)
+        feature["input_size"] = self.input_size  # needed for future overrides
 
         self.fc_stack = FCStack(
             first_layer_input_size=self.input_size,
diff --git a/ludwig/features/numerical_feature.py b/ludwig/features/numerical_feature.py
--- a/ludwig/features/numerical_feature.py
+++ b/ludwig/features/numerical_feature.py
@@ -343,8 +343,6 @@ class NumericalOutputFeature(NumericalFeatureMixin, OutputFeature):
     def __init__(self, feature, output_features: Dict[str, OutputFeature]):
         super().__init__(feature, output_features)
         self.overwrite_defaults(feature)
-
-        feature["input_size"] = self.input_shape[-1]
         self.decoder_obj = self.initialize_decoder(feature)
         self._setup_loss()
         self._setup_metrics()
diff --git a/ludwig/features/set_feature.py b/ludwig/features/set_feature.py
--- a/ludwig/features/set_feature.py
+++ b/ludwig/features/set_feature.py
@@ -178,15 +178,13 @@ def output_shape(self) -> torch.Size:
 
 class SetOutputFeature(SetFeatureMixin, OutputFeature):
     decoder = "classifier"
-    num_classes = 0
     loss = {TYPE: SIGMOID_CROSS_ENTROPY}
     metric_functions = {LOSS: None, JACCARD: None}
     default_validation_metric = JACCARD
+    num_classes = 0
+    threshold = 0.5
 
     def __init__(self, feature, output_features: Dict[str, OutputFeature]):
-        self.num_classes = 0
-        self.threshold = 0.5
-
         super().__init__(feature, output_features)
         self.overwrite_defaults(feature)
         self.decoder_obj = self.initialize_decoder(feature)


Test patch:
diff --git a/tests/integration_tests/test_experiment.py b/tests/integration_tests/test_experiment.py
--- a/tests/integration_tests/test_experiment.py
+++ b/tests/integration_tests/test_experiment.py
@@ -43,6 +43,7 @@
     ENCODERS,
     generate_data,
     generate_output_features_with_dependencies,
+    generate_output_features_with_dependencies_complex,
     h3_feature,
     HF_ENCODERS,
     HF_ENCODERS_SHORT,
@@ -238,6 +239,7 @@ def test_experiment_multilabel_with_class_weights(csv_filename):
         generate_output_features_with_dependencies("sequence_feature", ["category_feature", "numerical_feature"]),
         # output features with dependencies
         generate_output_features_with_dependencies("category_feature", ["sequence_feature"]),
+        generate_output_features_with_dependencies_complex(),
     ],
 )
 def test_experiment_multiple_seq_seq(csv_filename, output_features):
diff --git a/tests/integration_tests/utils.py b/tests/integration_tests/utils.py
--- a/tests/integration_tests/utils.py
+++ b/tests/integration_tests/utils.py
@@ -365,6 +365,23 @@ def generate_output_features_with_dependencies(main_feature, dependencies):
     return output_features
 
 
+def generate_output_features_with_dependencies_complex():
+    """Generates multiple output features specifications with dependencies."""
+
+    tf = text_feature(vocab_size=4, max_len=5, decoder="generator")
+    sf = sequence_feature(vocab_size=4, max_len=5, decoder="generator", dependencies=[tf["name"]])
+    nf = numerical_feature(dependencies=[tf["name"]])
+    vf = vector_feature(dependencies=[sf["name"], nf["name"]])
+    set_f = set_feature(vocab_size=4, dependencies=[tf["name"], vf["name"]])
+    cf = category_feature(vocab_size=4, dependencies=[sf["name"], nf["name"], set_f["name"]])
+
+    # The correct order ids[tf, sf, nf, vf, set_f, cf]
+    # # shuffling it to test the robustness of the topological sort
+    output_features = [nf, tf, set_f, vf, cf, sf, nf]
+
+    return output_features
+
+
 def _subproc_wrapper(fn, queue, *args, **kwargs):
     fn = cloudpickle.loads(fn)
     try: