Repository: rwth-i6/returnn

Issue:
Specified out dim tags potentially needs duplicated logic
This is a follow-up to the introduction of out dim tag options for all layers (#597).

Take `ConvLayer` as an example, and the `out_spatial_dims` option.

How would you work with this in practice? Consider both the case where the user writes the config and net dict by hand, but also more importantly the case of returnn-common.

Some aspects:

* Consider padding="same", stride=1, i.e. the output spatial dim should be the same as the input spatial dim.
Now when the user provides some custom out spatial dim, what should happen? Should the `ConvLayer` automatically use `declare_same_as`? Already in `get_out_data_from_opts` or only in `__init__`? I'm a bit afraid that this could cause very nasty and hard to find bugs, or it might make debugging for the user difficult if some dim tags got merged which should not have been merged. The user might not expect such side effect.

* Consider a static spatial dimension, and stride > 1. Should the user calculate by hand the resulting dimension? Maybe actually this is not a bad idea, although sometimes it could also be annoying when you don't care about it exactly. But consider returnn-common. The out spatial dim tag would usually be automatically created. (See: https://github.com/rwth-i6/returnn_common/issues/17#issuecomment-1004026430) But this means we need to calculate the output dimension beforehand, and duplicate the same logic  `calc_out_dim` on returnn-common side (or maybe just call `ConvLayer.calc_out_dim`). In any case, this makes the logic on returnn-common side quite a bit more complicated, and also un-intuitive. This is bad. It should not be complicated and un-intuitive.

The same issue is for many layers, also e.g. `MergeDimsLayer` etc. See all the layers mentioned in #597.

Some potential solutions:

* We could allow `Dim.dimension == NotSpecified` or so (or introduce yet another flag to `Dim`), to mark the passed out spatial dim as not-yet-specified. This explicitly tells `get_out_data_from_opts` that it can fill in the details. E.g. call `declare_same_as` or define `dimension = ...`. This would not be so confusing because it is explicitly written down that there should be modifying side effects on this dim. One issue with this is that we would again need to go through all layers in #597 and extend the logic in all `get_out_data_from_opts`.

* On returnn-common side, we could not pass the out spatial dims at all. Due to https://github.com/rwth-i6/returnn_common/issues/47, we can just take whatever the layer creates in `get_out_data_from_opts`. We might set `auto_generated = False` then. This would be a very simple solution.

On returnn-common side, this is reported here: https://github.com/rwth-i6/returnn_common/issues/116



Discussion:
> * We could allow `Dim.dimension == NotSpecified` or so (or introduce yet another flag to `Dim`), to mark the passed out spatial dim as not-yet-specified. This explicitly tells `get_out_data_from_opts` that it can fill in the details. E.g. call `declare_same_as` or define `dimension = ...`. This would not be so confusing because it is explicitly written down that there should be modifying side effects on this dim. One issue with this is that we would again need to go through all layers in [Specify dim tags for layers that create new axes #597](https://github.com/rwth-i6/returnn/issues/597) and extend the logic in all `get_out_data_from_opts`.

We already have such logic for dim tags with dynamic sizes. E.g. when you declare some dim tag in your config (where it has `dyn_size_ext=None` by default), and it is used as `out_spatial_dim` by some layer, then it's `dyn_size_ext` is set implicitly via `declare_same_as`. So here we don't even use such special `NotSpecified` value.
I think the assumption so far was that whenever `dimension is None`, `dyn_size_ext` should be completed.
I think this is fine to also have for static dims, but then we should also change it such that exactly one of `Dim.dimension` and `Dim.dyn_size_ext` is always `None`, and the other never (either `NotSpecified`, or having an actual value).
Now it would also be nicer to actually sub-class `Dim` into some `StaticDim` and `DynamicDim`, where `StaticDim` only has one attribute `dimension` and not `dyn_size_ext` and vice-versa. 
> * Consider padding="same", stride=1, i.e. the output spatial dim should be the same as the input spatial dim.
>   Now when the user provides some custom out spatial dim, what should happen? Should the `ConvLayer` automatically use `declare_same_as`? Already in `get_out_data_from_opts` or only in `__init__`? I'm a bit afraid that this could cause very nasty and hard to find bugs, or it might make debugging for the user difficult if some dim tags got merged which should not have been merged. The user might not expect such side effect.

I think RETURNN should never declare two dims equal that are both user defined, I think that is confusing and very implicit.
In the case that `out_spatial_dim` is not declared explicitly in such case, then I'm not sure what I'd like. Probably a more restrictive behavior, i.e. behavior that does not declare them as equal, is better in many cases but perhaps inconvenient.
> * On returnn-common side, we could not pass the out spatial dims at all. Due to [Make shape and dims available? returnn_common#47](https://github.com/rwth-i6/returnn_common/issues/47), we can just take whatever the layer creates in `get_out_data_from_opts`. We might set `auto_generated = False` then. This would be a very simple solution.

I think I would like it more if there would always be an `out_spatial_dim` arg, otherwise you would need to write something like this:
```
conv = Conv1d(...)
out, temp_out_spatial_dim = conv(...)
temp_out_spatial_dim.declare_same_as(my_actual_out_spatial_dim)
```
I dislike here that I need this temporary variable `temp_out_spatial_dim` which coincides with the actual dim I want.
> > * On returnn-common side, we could not pass the out spatial dims at all. Due to [Make shape and dims available? returnn_common#47](https://github.com/rwth-i6/returnn_common/issues/47), we can just take whatever the layer creates in `get_out_data_from_opts`. We might set `auto_generated = False` then. This would be a very simple solution.
> 
> I think I would like it more if there would always be an `out_spatial_dim` arg, otherwise you would need to write something like this:
> 
> ```
> conv = Conv1d(...)
> out, temp_out_spatial_dim = conv(...)
> temp_out_spatial_dim.declare_same_as(my_actual_out_spatial_dim)
> ```
> 
> I dislike here that I need this temporary variable `temp_out_spatial_dim` which coincides with the actual dim I want.

Why so complicated? Why not just:

```
conv = Conv1d(...)
out, my_actual_out_spatial_dim = conv(...)
```

But also, you are basically referring to https://github.com/rwth-i6/returnn_common/issues/115 here.

However, this is not what I meant. I meant, when the user calls `conv(...)` without specifying `out_spatial_dim`, how should this be implemented. We can either create a new default out spatial dim and pass it to `out_spatial_dims` to RETURNN. Or we could not pass any `out_spatial_dims` and then via https://github.com/rwth-i6/returnn_common/issues/47 get the out spatial dims from the resulting `Data`. Although maybe this is ambiguous in some cases, so not sure if this is a good solution.

> > * We could allow `Dim.dimension == NotSpecified` or so (or introduce yet another flag to `Dim`), to mark the passed out spatial dim as not-yet-specified. This explicitly tells `get_out_data_from_opts` that it can fill in the details. E.g. call `declare_same_as` or define `dimension = ...`. This would not be so confusing because it is explicitly written down that there should be modifying side effects on this dim. One issue with this is that we would again need to go through all layers in [Specify dim tags for layers that create new axes #597](https://github.com/rwth-i6/returnn/issues/597) and extend the logic in all `get_out_data_from_opts`.
> 
> We already have such logic for dim tags with dynamic sizes. E.g. when you declare some dim tag in your config (where it has `dyn_size_ext=None` by default), and it is used as `out_spatial_dim` by some layer, then it's `dyn_size_ext` is set implicitly via `declare_same_as`. So here we don't even use such special `NotSpecified` value. I think the assumption so far was that whenever `dimension is None`, `dyn_size_ext` should be completed.

For dynamic dims, this is clear because when `dimension is None` (meaning dynamic dim) and `dyn_size_ext` is not set, it is clear that it is not yet set.

For static dims, this is not clear. Should `dimension is None` have two meanings now, i.e. meaning either dynamic dim or static dim but not defined yet?

I don't like that `dimension is None` can have these two meanings. That is why I suggested to have `dimension is NotSpecified` to clearly distinguish it.

`dimension is NotSpecified` could also mean it is dynamic, or static, but in any case it is not specified yet. This would simplify the code on returnn-common most because in the case the user does not provide `out_spatial_dim`, we can simply create `out_spatial_dim = nn.SpatialDim(dimension=NotSpecified, ...)` and it will work for all cases, static or dynamic.

> I think this is fine to also have for static dims, but then we should also change it such that exactly one of `Dim.dimension` and `Dim.dyn_size_ext` is always `None`, and the other never (either `NotSpecified`, or having an actual value).

I'm not sure I understand. How exactly?

> Now it would also be nicer to actually sub-class `Dim` into some `StaticDim` and `DynamicDim`, where `StaticDim` only has one attribute `dimension` and not `dyn_size_ext` and vice-versa.

Remember, we had such discussion about subclassing `Dim` already for `SpatialDim` and `FeatureDim` and decided that this is not a good idea, for various reasons. I think mostly to not break old configs. I think the same applies here, right?

> E.g. when you declare some dim tag in your config (where it has `dyn_size_ext=None` by default), and it is used as `out_spatial_dim` by some layer, then it's `dyn_size_ext` is set implicitly via `declare_same_as`.

Note that this logic is not always (or actually only rarely) via `get_out_data_from_opts`. E.g. `ConvLayer.get_out_data_from_opts` does not do this. But we need it in `get_out_data_from_opts` such that it works for returnn-common (https://github.com/rwth-i6/returnn_common/issues/47).

**Edit** For `ConvLayer` and `PoolLayer`, this would be correctly handled in `get_out_data_from_opts` via #971.

 
> I think RETURNN should never declare two dims equal that are both user defined, I think that is confusing and very implicit.

Isn't this in contradiction to what you said later:

> E.g. when you declare some dim tag in your config (where it has `dyn_size_ext=None` by default), and it is used as `out_spatial_dim` by some layer, then it's `dyn_size_ext` is set implicitly via `declare_same_as`.

?

> In the case that `out_spatial_dim` is not declared explicitly in such case, then I'm not sure what I'd like. Probably a more restrictive behavior, i.e. behavior that does not declare them as equal, is better in many cases but perhaps inconvenient.

In case of returnn-common, all dim tags are user defined (or rather, none will have `auto_generated=True`). It does not make a difference where exactly they are defined.

`auto_generated` so far also only has one meaning: that `description` is used for equality. I would not add too much behavior depending on this flag.

> > ```
> > conv = Conv1d(...)
> > out, temp_out_spatial_dim = conv(...)
> > temp_out_spatial_dim.declare_same_as(my_actual_out_spatial_dim)
> > ```
> >
> > I dislike here that I need this temporary variable `temp_out_spatial_dim` which coincides with the actual dim I want.
> 
> Why so complicated? Why not just:
> 
> ```
> conv = Conv1d(...)
> out, my_actual_out_spatial_dim = conv(...)
> ```
> 
> But also, you are basically referring to [rwth-i6/returnn_common#115](https://github.com/rwth-i6/returnn_common/issues/115) here.
> 
> However, this is not what I meant. I meant, when the user calls `conv(...)` without specifying `out_spatial_dim`, how should this be implemented. We can either create a new default out spatial dim and pass it to `out_spatial_dims` to RETURNN. Or we could not pass any `out_spatial_dims` and then via [rwth-i6/returnn_common#47](https://github.com/rwth-i6/returnn_common/issues/47) get the out spatial dims from the resulting `Data`. Although maybe this is ambiguous in some cases, so not sure if this is a good solution.

Yes, I am referring to that issue. I thought there (in [rwth-i6/returnn_common#115](https://github.com/rwth-i6/returnn_common/issues/115)) you are arguing that maybe it is not even needed to ever have such explicit `out_spatial_dim` arg, because you can just use the default one in any case.
With my example, I meant that `my_actual_out_spatial_dim` already existed before, e.g. because I want to share it between two layers. But in general, that's also how I write my configs currently, at the top of the config, I declare some dim tags that I will use later, and then pass them to all layers via `out_spatial_dim` to set them. So you would argue that the user would in most cases just use the dim tag returned by the `conv(..)` call directly, and not specify their own with their own description? Then, your code snippet would work (at least if I only need to set this dim tag two the output of one layer, and not to twice).
So, if returnn-common's `conv` didn't take an `out_spatial_dim` arg, then I would have to do something like this:
```
my_spatial_dim = SpatialDim(...)
y1, tmp_dim1 = conv(...)
y2, tmp_dim2 = conv(...)
tmp_dim1.declare_same_as(my_spatial_dim)
tmp_dim2.declare_same_as(my_spatial_dim)
```
Or:
```
y1, my_spatial_dim = conv(...)
y2, tmp_dim = conv(...)
tmp_dim.declare_same_as(my_spatial_dim)
```
Right?
> > > * We could allow `Dim.dimension == NotSpecified` or so (or introduce yet another flag to `Dim`), to mark the passed out spatial dim as not-yet-specified. This explicitly tells `get_out_data_from_opts` that it can fill in the details. E.g. call `declare_same_as` or define `dimension = ...`. This would not be so confusing because it is explicitly written down that there should be modifying side effects on this dim. One issue with this is that we would again need to go through all layers in [Specify dim tags for layers that create new axes #597](https://github.com/rwth-i6/returnn/issues/597) and extend the logic in all `get_out_data_from_opts`.
> > 
> > 
> > We already have such logic for dim tags with dynamic sizes. E.g. when you declare some dim tag in your config (where it has `dyn_size_ext=None` by default), and it is used as `out_spatial_dim` by some layer, then it's `dyn_size_ext` is set implicitly via `declare_same_as`. So here we don't even use such special `NotSpecified` value. I think the assumption so far was that whenever `dimension is None`, `dyn_size_ext` should be completed.
> 
> For dynamic dims, this is clear because when `dimension is None` (meaning dynamic dim) and `dyn_size_ext` is not set, it is clear that it is not yet set.
> 
> For static dims, this is not clear. Should `dimension is None` have two meanings now, i.e. meaning either dynamic dim or static dim but not defined yet?
> 
> I don't like that `dimension is None` can have these two meanings. That is why I suggested to have `dimension is NotSpecified` to clearly distinguish it.
> 
> `dimension is NotSpecified` could also mean it is dynamic, or static, but in any case it is not specified yet. This would simplify the code on returnn-common most because in the case the user does not provide `out_spatial_dim`, we can simply create `out_spatial_dim = nn.SpatialDim(dimension=NotSpecified, ...)` and it will work for all cases, static or dynamic.
> 
> > I think this is fine to also have for static dims, but then we should also change it such that exactly one of `Dim.dimension` and `Dim.dyn_size_ext` is always `None`, and the other never (either `NotSpecified`, or having an actual value).
> 
> I'm not sure I understand. How exactly?
> 
> > Now it would also be nicer to actually sub-class `Dim` into some `StaticDim` and `DynamicDim`, where `StaticDim` only has one attribute `dimension` and not `dyn_size_ext` and vice-versa.

Okay, perhaps `NotSpecified` is not a good name for this, rather `InitializedLater` or so and maybe it's clearer then what I meant. Sorry, calling this `NotSpecified` probably implies something else.
I would propose that the contract of `Dim` is that at each time:
 * `Dim.dimension` is `InitializedLater` or `None` or just some int
 * `Dim.dyn_size_ext` is `InitializedLater` or `None` or just some `Data`
 * exactly one of `Dim.dimension` and `Dim.dyn_size_ext` is `None` at any point. Perhaps we can make this more explicit by adding a `is_static` attribute.
With this, it's always clear whether this dim tag is static or not.

> Remember, we had such discussion about subclassing `Dim` already for `SpatialDim` and `FeatureDim` and decided that this is not a good idea, for various reasons. I think mostly to not break old configs. I think the same applies here, right?

Yeah I remember, but I thought that we mainly decided on this for simplicity but that it didn't matter too much. I'm not sure about backwards compatibility.
Now here I would argue for having two subclasses because then we avoid that `Dim.dimension` or `Dim.dyn_size_ext` can ever be `None`. They just wouldn't exist as attribute at all, so there is less confusion.
> > E.g. when you declare some dim tag in your config (where it has `dyn_size_ext=None` by default), and it is used as `out_spatial_dim` by some layer, then it's `dyn_size_ext` is set implicitly via `declare_same_as`.
> 
> Note that this logic is not always (or actually only rarely) via `get_out_data_from_opts`. E.g. `ConvLayer.get_out_data_from_opts` does not do this. But we need it in `get_out_data_from_opts` such that it works for returnn-common ([rwth-i6/returnn_common#47](https://github.com/rwth-i6/returnn_common/issues/47)).

Oh okay, I looked at `RangeFromLengthLayer` when writing this, which uses `declare_same_as` in `get_out_data_from_opts`.
This is inconsistent then.
> > > ```
> > > conv = Conv1d(...)
> > > out, temp_out_spatial_dim = conv(...)
> > > temp_out_spatial_dim.declare_same_as(my_actual_out_spatial_dim)
> > > ```
> > 
> > Why so complicated? Why not just:
> > ```
> > conv = Conv1d(...)
> > out, my_actual_out_spatial_dim = conv(...)
> > ```
> > 
> > But also, you are basically referring to [rwth-i6/returnn_common#115](https://github.com/rwth-i6/returnn_common/issues/115) here.
> > However, this is not what I meant. I meant, when the user calls `conv(...)` without specifying `out_spatial_dim`, how should this be implemented. We can either create a new default out spatial dim and pass it to `out_spatial_dims` to RETURNN. Or we could not pass any `out_spatial_dims` and then via [rwth-i6/returnn_common#47](https://github.com/rwth-i6/returnn_common/issues/47) get the out spatial dims from the resulting `Data`. Although maybe this is ambiguous in some cases, so not sure if this is a good solution.
> 
> Yes, I am referring to that issue. I thought there (in [rwth-i6/returnn_common#115](https://github.com/rwth-i6/returnn_common/issues/115)) you are arguing that maybe it is not even needed to ever have such explicit `out_spatial_dim` arg, because you can just use the default one in any case. With my example, I meant that `my_actual_out_spatial_dim` already existed before, e.g. because I want to share it between two layers.

What do you mean by share?

The out spatial dim is the same as the in spatial dim? Then you might only want to have this said explicitly for error checking / verification. This is a different use case. I'm not sure that I would mix this now here.

The out spatial dim will be the same as some other one where you e.g. do the same kind of conv or pool? Now here you explicitly want to mark them as same. But is this the most intuitive way? Also, in those cases for conv and pool, this is not needed, this will be automatic, due to the dim tag math logic.

> But in general, that's also how I write my configs currently, at the top of the config, I declare some dim tags that I will use later, and then pass them to all layers via `out_spatial_dim` to set them.

But you only do this because when writing the config and net dict by hand and not using returnn-common, you don't really have any other choice. When you look at what returnn-common creates, then this is actually exactly like this. It will first define all dim tags and then create the net dict.

> So you would argue that the user would in most cases just use the dim tag returned by the `conv(..)` call directly, and not specify their own with their own description?

I don't really know. I assume so.

If you just want to set a custom description, you can still do that with my code snippet, like:
```
conv = Conv1d(...)
out, my_actual_out_spatial_dim = conv(...)
my_actual_out_spatial_dim.description = "whatever"
```

> Then, your code snippet would work (at least if I only need to set this dim tag two the output of one layer, and not to twice). So, if returnn-common's `conv` didn't take an `out_spatial_dim` arg, then I would have to do something like this:
> 
> ```
> my_spatial_dim = SpatialDim(...)
> y1, tmp_dim1 = conv(...)
> y2, tmp_dim2 = conv(...)
> tmp_dim1.declare_same_as(my_spatial_dim)
> tmp_dim2.declare_same_as(my_spatial_dim)
> ```
>
> Or:
> 
> ```
> y1, my_spatial_dim = conv(...)
> y2, tmp_dim = conv(...)
> tmp_dim.declare_same_as(my_spatial_dim)
> ```
> 
> Right?

Why not like:
```python
y1, my_spatial_dim = conv(...)
y2, my_spatial_dim_ = conv(...)
assert my_spatial_dim == my_spatial_dim_
```


> > I think RETURNN should never declare two dims equal that are both user defined, I think that is confusing and very implicit.
> 
> Isn't this in contradiction to what you said later:
> 
> > E.g. when you declare some dim tag in your config (where it has `dyn_size_ext=None` by default), and it is used as `out_spatial_dim` by some layer, then it's `dyn_size_ext` is set implicitly via `declare_same_as`.
> 
> ?

I meant that I find this confusing if the dim tags are both user defined.
Most layers create some auto generated `out_spatial_dim` first, and then declare this as equal to the user-provided `out_spatial_dim` (this is how it is in RETURNN right now, in returnn-common it's maybe different, see below).
I don't love this, because it's also implicit, but I find this okay because the user never had a reference to the implicitly auto generated `out_spatial_dim`. So there is not that much confusion.

What I am talking about is:
```
a_dim = SpatialDim(..)
b_dim = SpatialDim(..)
x  # of shape [a_dim]
y = conv(x, padding=same, stride=1, out_spatial_dim=b_dim)  # of shape [b_dim]
# the shape of y and x should be different! I, the user, explicitly defined it to be different!
# e.g.:
c = x + y  # should require explicit broadcasting to [a_dim, b_dim]!
```

> > In the case that `out_spatial_dim` is not declared explicitly in such case, then I'm not sure what I'd like. Probably a more restrictive behavior, i.e. behavior that does not declare them as equal, is better in many cases but perhaps inconvenient.
> 
> In case of returnn-common, all dim tags are user defined (or rather, none will have `auto_generated=True`). It does not make a difference where exactly they are defined.
> 
> `auto_generated` so far also only has one meaning: that `description` is used for equality. I would not add too much behavior depending on this flag.

Oh okay, I didn't know that returnn-common doesn't use auto generated dims. So okay, above I meant with "auto generated" not the RETURNN flag, but the fact that the layer created the dim tag and not the input dim tag is used.
I think we are really talking about different use cases. Perhaps I don't really understand the main concept here, and perhaps my experience with dim tags is also quite specific and not so much how you would use it on a everyday basis.
So I'll just share a list of use cases where I like dim tags, and love that they are explicit:
* For self attention, I want to have a `key_dim` and `value_dim`. They usually have the same static dimension, but I don't want to mix them up, because they are conceptually incomparable. So I don't want `key_dim` to be the same as `value_dim`.
* The `key_dim` is created in the output of multiple different layers: I have some layer creating a list of keys and a list of queries, both using `key_dim` as feature dim. So, if in returnn-common or so both these layers would create different dims by default, then I would need to explicitly declare them as the same, right? (or is this not the case because this is the feature dim, and it works differently?)
* Similarly to this I had a spatial time axis, `key-time` or so, which I want to sort my tensor along. So the output of this operation still has exactly the same sequence lengths, but I want to give this a new dim tag (`sorted-key-time` or so). Here, it is super important for me that `key-time` and `sorted-key-time` are entirely different, and that RETURNN prevents me from combining a sorted and unsorted version of a tensor accidentally (via the explicit broadcasting). Errors like this would be difficult to catch because everything would still run without error but lead to wrong results. I had a few of these problems before and then having these different dim tags was really helpful.
* When I talked about `conv` in my examples, I thought thought about this conceptually like my `sort` layer, where when I as the user specify `out_spatial_dim` explicitly, I want to do this to be sure that I cannot accidentally combine it with some tensor where I didn't apply the convolution on before. So I wanted to get an entirely new, independent dim tag as the input.
But I never used convolutions actually. So, in the case of `conv`, this is maybe not what users want. I guess my use case (with the `sort`), is quite different from normal usage.
Also I wasn't aware that (if I understand correctly) `conv` currently always marks my input dim equal to the output dim if they share the same seq lengths.
> > > I think RETURNN should never declare two dims equal that are both user defined, I think that is confusing and very implicit.
> > 
> > Isn't this in contradiction to what you said later:
> > > E.g. when you declare some dim tag in your config (where it has `dyn_size_ext=None` by default), and it is used as `out_spatial_dim` by some layer, then it's `dyn_size_ext` is set implicitly via `declare_same_as`.
> > 
> > 
> > ?
> 
> I meant that I find this confusing if the dim tags are both user defined. Most layers create some auto generated `out_spatial_dim` first, and then declare this as equal to the user-provided `out_spatial_dim` (this is how it is in RETURNN right now, in returnn-common it's maybe different, see below). I don't love this, because it's also implicit, but I find this okay because the user never had a reference to the implicitly auto generated `out_spatial_dim`. So there is not that much confusion.

But consider the case that the out-spatial-dim is the same as in-spatial-dim (stride 1, padding same). Then the implicitly auto generated `out_spatial_dim` is the same as the in-spatial-dim. The in-spatial-dim might be user generated, and the out-spatial-dim might be another user generate dim, and then this side effect would make them equal.

> What I am talking about is:
> 
> ```
> a_dim = SpatialDim(..)
> b_dim = SpatialDim(..)
> x  # of shape [a_dim]
> y = conv(x, padding=same, stride=1, out_spatial_dim=b_dim)  # of shape [b_dim]
> # the shape of y and x should be different! I, the user, explicitly defined it to be different!
> # e.g.:
> c = x + y  # should require explicit broadcasting to [a_dim, b_dim]!
> ```

But now you are mixing things. The `ConvLayer` is not supposed to make a new different copy of a dim tag. We have `ReinterpretDataLayer` with `set_dim_tags` specifically for this use case. This is what we use for self-attention. I don't think that other layers should also do this.

Also, why would you ever want this here? I think the cases where you really want to introduce a new separate copy of a dim tag is sth like self attention. And it is good that we have one explicit layer for it (`ReinterpretDataLayer`), and not as a side effect from some other unrelated layer.

In your example, I'm actually not sure what I would expect to happen. Maybe `conv` internally would have some `assert in_spatial_dim == out_spatial_dim`. Or I would expect `out_spatial_dim.declare_same_as(in_spatial_dim)` here. You seem to disagree on the second point, and I also don't like it too much. Except when it is explicitly marked as not-specified / initialized-later, then it is ok. When it is not marked as such, I think I would disallow this. I would be confused if `a_dim != b_dim` in this example.

> I think we are really talking about different use cases. Perhaps I don't really understand the main concept here, and perhaps my experience with dim tags is also quite specific and not so much how you would use it on a everyday basis. So I'll just share a list of use cases where I like dim tags, and love that they are explicit:
> 
> * For self attention, I want to have a `key_dim` and `value_dim`. They usually have the same static dimension, but I don't want to mix them up, because they are conceptually incomparable. So I don't want `key_dim` to be the same as `value_dim`.
> * The `key_dim` is created in the output of multiple different layers: I have some layer creating a list of keys and a list of queries, both using `key_dim` as feature dim. So, if in returnn-common or so both these layers would create different dims by default, then I would need to explicitly declare them as the same, right? (or is this not the case because this is the feature dim, and it works differently?)

No, the feature dim (`out_dim` in most modules, like `Linear`, `Conv`) is (obviously) a required argument.

This issue here is only about any implicit new dim tags by a module or function, such as out spatial dims in `Conv`, `pool`, or also like `merge_dims`, etc.

> * Similarly to this I had a spatial time axis, `key-time` or so, which I want to sort my tensor along. So the output of this operation still has exactly the same sequence lengths, but I want to give this a new dim tag (`sorted-key-time` or so). Here, it is super important for me that `key-time` and `sorted-key-time` are entirely different, and that RETURNN prevents me from combining a sorted and unsorted version of a tensor accidentally (via the explicit broadcasting). Errors like this would be difficult to catch because everything would still run without error but lead to wrong results. I had a few of these problems before and then having these different dim tags was really helpful.

Yes, and for this, we have the `ReinterpretDataLayer` with `set_dim_tags` to make this very explicit. This is the same in returnn-common.

The implementation of self-attention is:
```python
self.key_dim_total = key_dim_total
self.key_dim_per_head = key_dim_total.div_left(num_heads)
self.value_dim_total = value_dim_total
self.value_dim_per_head = value_dim_total.div_left(num_heads)
self.num_heads = num_heads
self.qkv_dim_total = 2 * key_dim_total + value_dim_total
self.qkv_dim_per_head = 2 * self.key_dim_per_head + self.value_dim_per_head
self.qkv = nn.Linear(self.qkv_dim_total)

...

hist_dim = nn.SpatialDim(f"{nn.NameCtx.current_ctx().layer_abs_name_scope}:history")
qkv = self.qkv(source)
qkv = nn.split_dims(
  qkv, axis=self.qkv_dim_total, dims=(self.num_heads, self.qkv_dim_per_head), name="qkv_split_dims")
q, k, v = nn.split(
  qkv, axis=self.qkv_dim_per_head,
  out_dims=(self.key_dim_per_head, self.key_dim_per_head, self.value_dim_per_head),
  name="qkv_split")
if axis == nn.single_step_dim:
  assert causal is None or causal
  assert state
  new_state = nn.LayerState()
  k, _, new_state.k_accum = nn.cum_concat_step(k, state=state.k_accum, out_spatial_dim=hist_dim, name="k_accum")
  v, _, new_state.v_accum = nn.cum_concat_step(v, state=state.v_accum, out_spatial_dim=hist_dim, name="v_accum")
else:
  new_state = None
  assert not state
  if causal:
    raise NotImplementedError(
      "Causal attention on sequence level not implemented. "
      "We can easily extend CumConcatLayer on RETURNN side for this, to accept any axis argument. "
      "However, normally any causal attention should always be inside a loop and this should never be needed.")
  k, _ = nn.reinterpret_new_dim(k, in_dim=axis, out_dim=hist_dim, name="k_new_dim")
  v, _ = nn.reinterpret_new_dim(v, in_dim=axis, out_dim=hist_dim, name="v_new_dim")
att = dot_attention(q, k, v, key_dim=self.key_dim_per_head, axis=hist_dim, att_dropout=self.att_dropout)
output, _ = nn.merge_dims(
  att, axes=(self.num_heads, self.value_dim_per_head), out_dim=self.value_dim_total, name="output")
return output, new_state
```

> * When I talked about `conv` in my examples, I thought thought about this conceptually like my `sort` layer, where when I as the user specify `out_spatial_dim` explicitly, I want to do this to be sure that I cannot accidentally combine it with some tensor where I didn't apply the convolution on before. So I wanted to get an entirely new, independent dim tag as the input.
>   But I never used convolutions actually. So, in the case of `conv`, this is maybe not what users want. I guess my use case (with the `sort`), is quite different from normal usage.
>   Also I wasn't aware that (if I understand correctly) `conv` currently always marks my input dim equal to the output dim if they share the same seq lengths.

I still think, when the user really wants explicitly that after `conv`, or whatever other op, it is assigned a new separate dim tag, this should be explicit via a separate function. This is `nn.reinterpret_new_dim` in the code above.

When there is `conv` which would actually yield the same spatial dim, I would not expect it to be different at that point. Similar as I would not expect that `Linear` would give me a new spatial dim.


Okay, I see now. Sorry for my confusion. I wasn't aware about what e.g. `conv` would do, but okay, I agree now that `conv` with same padding, no stride should always require `in_spatial_dim == out_spatial_dim`. And as you said, I think I would just enforce this and not have it automatically declare them as same.
I think for returnn-common, it does indeed make sense that `conv` does not take a `out_spatial_dim` arg then (https://github.com/rwth-i6/returnn_common/issues/115) and instead just returns it.
For `conv` in particular I would even find it a little confusing if I had to specify it when it has to be the same as the input dim.

So okay, I understand now. Sorry that this is mixing up a lot of issues now, also on returnn-commons' side.

About this issue: Yes, I would find some explicit "`InitializedLater`/`InferredAutomatically`/etc." placeholder for both `Dim.dimension` and `Dim.dyn_size_ext` helpful. We already have this for `Dim.dyn_size_ext`, but it's very implicit there and I'd like it more if it was more explicit there too. And then we can do the same for `Dim.dimension`.
I still think sub-classing `Dim` into `DynamicDim` and `StaticDim` could clean this up, but I'm not entirely aware how much existing code that breaks.
Well, this is still a novel feature of RETURNN (`out_spatial_dims` and related options, #597) and I think not much people are really using it so far. Maybe no-one actually, not sure. returnn-common now is a first use case where it is consistently used all the time. But returnn-common is also still experimental and we need to figure out the best way. It's really not so clear what's best. I guess we will only really see after some experience.

You are raising the right concerns. It should not lead to unexpected behavior, and be explicit. And it should not be confusing. And it should also be simple and straightforward to use.

Btw, instead of `InitializedLater`/`InferredAutomatically`, I think `NotSpecified` is actually ok.

Btw, regarding implicit logic to fill in details in `out_spatial_dims`, what we have now for `ConvLayer.__init__`, `RangeLayer.get_out_data_from_opts`, `CumConcatLayer.__init__`, etc, this is also how `ReinterpretDataLayer.__init__` works for the `set_dims` option.

We need to differentiate the cases:
- where the out dim tag would become the same as some input dim tag (`declare_same_as`),
- where the out dim tag gets completed by setting `dyn_size_ext` or `dimension but this would be a new dim tag, not equal to any existing.

We probably should make it a bit more consistent. If possible, I think both cases should be covered in `get_out_data_from_opts` of a layer and not only later in `__init__`.

We also should not break existing code. So we might accept `dimension is None and dyn_size_ext is None` also as not being specified yet. All examples for self attention using `ReinterpretDataLayer` `set_dims` or `CumConcatLayer` are like this. Although maybe such code is still rare yet.

If we interpret `dimension is None and dyn_size_ext is None` as not being specified, maybe we don't actually need any new behavior? Currently `dimension is None and dyn_size_ext is None` also assumes that the dimension is dynamic then (because `dimension is None`). Earlier I argued that we should not allow for a different interpretation here but maybe we don't really have a choice now to not break old code, and maybe it's not such a big deal when checked together with `dyn_size_ext`.


Resolving patch:
diff --git a/returnn/tf/layers/basic.py b/returnn/tf/layers/basic.py
--- a/returnn/tf/layers/basic.py
+++ b/returnn/tf/layers/basic.py
@@ -1007,30 +1007,9 @@ def __init__(self, axis, slice_start=None, slice_end=None, slice_step=None, out_
     axis = self.input_data.get_axis_from_description(axis)
     dim_slice = slice(slice_start, slice_end, slice_step)
     slices = [slice(None, None)] * axis + [dim_slice]
-    input_dim_tag = self.input_data.dim_tags[axis]
-    output_dim_tag = self.output.dim_tags[axis]
-    if input_dim_tag.dyn_size_ext and output_dim_tag.dyn_size is None:
-      assert input_dim_tag.dyn_size_ext.placeholder is not None
-      if not output_dim_tag.dyn_size_ext:
-        output_dim_tag.dyn_size_ext = input_dim_tag.dyn_size_ext.copy_template(name="%s:slice-size" % self.name)
-      dyn_size = input_dim_tag.dyn_size_ext.placeholder
-      if slice_start:
-        assert slice_start > 0
-        dyn_size = tf.maximum(0, dyn_size - slice_start)
-      if slice_end is not None:
-        if slice_end >= 0:
-          dyn_size = tf.minimum(slice_end, dyn_size)
-        else:  # slice_end < 0
-          dyn_size = tf.maximum(0, dyn_size + slice_end)
-      if slice_step:
-        dyn_size = tf.cast(tf_compat.v1.ceil(tf.divide(dyn_size, slice_step)), tf.int32)
-      output_dim_tag.dyn_size_ext.placeholder = dyn_size
-      existing_tag = Dim.get_tag_from_size_tensor(dyn_size)
-      if existing_tag:
-        output_dim_tag.declare_same_as(existing_tag)
-      else:
-        output_dim_tag.set_tag_on_size_tensor(dyn_size)
-    self.output.placeholder = self.input_data.placeholder[slices]
+    y = self.input_data.placeholder[slices]
+    y.set_shape(self.output.batch_shape)  # can be necessary for slice_end>0
+    self.output.placeholder = y
 
   @classmethod
   def get_out_data_from_opts(
@@ -1051,13 +1030,22 @@ def get_out_data_from_opts(
     input_data = get_concat_sources_data_template(sources)
     axis = input_data.get_axis_from_description(axis)
     dim_tag = input_data.dim_tags[axis]
-    dim_slice = slice(slice_start, slice_end, slice_step)
-    new_dim = len(range(dim_tag.dimension)[dim_slice]) if dim_tag.dimension is not None else None
+    out_dim_ = dim_tag
+    if slice_start:
+      assert slice_start >= 0
+      out_dim_ = out_dim_.sub_left(slice_start)
+    if slice_end is not None:
+      if slice_end >= 0:
+        out_dim_ = Dim(
+          kind=dim_tag.kind, description="%s:slice-end" % name,
+          dimension=slice_end - (slice_start or 0), auto_generated=True)
+      else:  # slice_end < 0
+        out_dim_ = out_dim_ + slice_end
+    if slice_step and slice_step != 1:
+      out_dim_ = out_dim_.ceildiv_right(abs(slice_step))
     if out_dim:
-      assert out_dim.dimension == new_dim
-    else:
-      out_dim = Dim(kind=dim_tag.kind, description="%s:slice" % name, dimension=new_dim, auto_generated=True)
-    return input_data.copy_template_replace_dim_tag(axis=axis, new_dim_tag=out_dim, name="%s_output" % name)
+      out_dim_.declare_same_as(out_dim)
+    return input_data.copy_template_replace_dim_tag(axis=axis, new_dim_tag=out_dim_, name="%s_output" % name)
 
 
 class SliceNdLayer(_ConcatInputLayer):
@@ -1230,13 +1218,14 @@ def get_out_data_from_opts(cls, name, sources=(), start=None, size=None, axis="T
     else:
       # size might be None here in which case we set the dyn_size in __init__
       assert size is None or isinstance(size, int)
+      out_spatial_dim_ = Dim(
+        kind=Dim.Types.Spatial,
+        description="sliced-time:%s" % name,
+        dimension=size, auto_generated=True)
       if out_spatial_dim:
-        assert out_spatial_dim.dimension == size
+        out_spatial_dim_.declare_same_as(out_spatial_dim)
       else:
-        out_spatial_dim = Dim(
-          kind=Dim.Types.Spatial,
-          description="sliced-time:%s" % name,
-          dimension=size, auto_generated=True)
+        out_spatial_dim = out_spatial_dim_
     gather_positions_data = gather_positions_data.copy_add_dim_by_tag(
       out_spatial_dim, unbroadcast=True, axis=start_data.batch_ndim)
     position = InternalLayer(
@@ -2955,18 +2944,16 @@ def get_out_data_from_opts(cls, name, sources, n_out=NotSpecified, out_dim=None,
     input_data = get_concat_sources_data_template(sources)
     assert not input_data.sparse
     assert input_data.dim % 2 == 0
-    dim = input_data.dim // 2
+    out_dim_ = input_data.dim_tags[input_data.feature_dim_axis] // 2
     if out_dim:
-      assert out_dim.dimension == dim
-    else:
-      out_dim = FeatureDim("%s:gating" % name, dimension=dim, auto_generated=True)
+      out_dim_.declare_same_as(out_dim)
     if n_out is not NotSpecified:
-      assert n_out == dim
+      assert n_out == input_data.dim // 2
     return Data(
       name="%s_output" % name,
       dtype=input_data.dtype,
       dim_tags=[
-        out_dim if i == input_data.feature_dim_axis else d
+        out_dim_ if i == input_data.feature_dim_axis else d
         for (i, d) in enumerate(input_data.dim_tags)],
       sparse=False,
       time_dim_axis=input_data.time_dim_axis,
@@ -3005,6 +2992,7 @@ def __init__(self, window_size=None, window_dim=None, window_left=None, window_r
     :param int stride: return only each Nth window
     :param kwargs:
     """
+    out_spatial_dim  # noqa  # via get_out_data_from_opts
     super(WindowLayer, self).__init__(**kwargs)
     if not window_size:
       assert window_dim and window_dim.dimension
@@ -3030,25 +3018,6 @@ def __init__(self, window_size=None, window_dim=None, window_left=None, window_r
     else:
       axis = data.get_axis_from_description(axis)
       new_dim_axis = axis + 1  # new axis will be added right after
-      in_spatial_dim = data.dim_tags[axis]
-      out_spatial_dim_ = self.output.dim_tags[axis]
-      if out_spatial_dim:
-        assert out_spatial_dim_ == out_spatial_dim
-        if (padding.lower() == "same" or window_size == 1) and stride == 1:  # no change in spatial dim
-          assert in_spatial_dim == out_spatial_dim
-      if in_spatial_dim != out_spatial_dim_ and out_spatial_dim_.dimension is None:
-        if not out_spatial_dim_.dyn_size_ext:
-          out_spatial_dim_.dyn_size_ext = in_spatial_dim.dyn_size_ext.copy_template(name="%s:spatial-size" % self.name)
-        if out_spatial_dim_.dyn_size_ext.placeholder is None:
-          from ..util.basic import same_control_flow_ctx
-          from ..util.data import Dim
-          assert in_spatial_dim.dyn_size is not None
-          size = in_spatial_dim.dyn_size
-          with same_control_flow_ctx(size):
-            size = ConvLayer.calc_out_dim(
-              in_dim=size,
-              filter_size=window_size, stride=stride, dilation_rate=1, padding=padding)
-          out_spatial_dim_.dyn_size_ext.placeholder = size
 
       from returnn.tf.util.basic import windowed_nd
       self.output.placeholder = windowed_nd(
@@ -3087,27 +3056,19 @@ def get_out_data_from_opts(cls, name, network, sources, window_size=None, window
     else:
       axis = data.get_axis_from_description(axis)
       in_spatial_dim = data.dim_tags[axis]
-      if (padding.lower() == "same" or window_size == 1) and stride == 1:  # no change in spatial dim
-        out_spatial_dim = in_spatial_dim  # error check in __init__
-      else:  # new spatial dim
-        if not out_spatial_dim:
-          dim = None
-          if in_spatial_dim.dimension is not None:
-            dim = ConvLayer.calc_out_dim(
-              in_dim=in_spatial_dim.dimension,
-              filter_size=window_size, stride=stride, dilation_rate=1, padding=padding)
-          out_spatial_dim = Dim(
-            kind=Dim.Types.Spatial, description="%s:spatial" % name,
-            dimension=dim, derived_from_tag=in_spatial_dim, auto_generated=True,
-            batch=data.batch, control_flow_ctx=data.control_flow_ctx)
-      data = data.copy_template_replace_dim_tag(axis=axis, new_dim_tag=out_spatial_dim)
+      out_spatial_dim_ = ConvLayer.calc_out_dim(
+        in_dim=in_spatial_dim,
+        filter_size=window_size, stride=stride, dilation_rate=1, padding=padding)
+      assert isinstance(out_spatial_dim_, Dim)
+      if out_spatial_dim:
+        out_spatial_dim_.declare_same_as(out_spatial_dim)
+      data = data.copy_template_replace_dim_tag(axis=axis, new_dim_tag=out_spatial_dim_)
       new_dim_axis = axis + 1  # add new axis right after
+    window_dim_ = Dim(
+      kind=Dim.Types.Spatial, description="%s:window" % name, dimension=window_size, auto_generated=True)
     if window_dim:
-      assert window_dim.dimension == window_size
-    else:
-      window_dim = Dim(
-        kind=Dim.Types.Spatial, description="%s:window" % name, dimension=window_size, auto_generated=True)
-    return data.copy_add_dim_by_tag(axis=new_dim_axis, dim_tag=window_dim, unbroadcast=True)
+      window_dim_.declare_same_as(window_dim)
+    return data.copy_add_dim_by_tag(axis=new_dim_axis, dim_tag=window_dim_, unbroadcast=True)
 
   # noinspection PyMethodOverriding
   @classmethod
@@ -3231,25 +3192,6 @@ def __init__(self, axes, padding, out_dims=None, value=0, mode="constant", **kwa
       self.output.placeholder = tf_util.pad_replicate(self.input_data.placeholder, axes, padding)
     else:
       self.output.placeholder = tf.pad(self.input_data.placeholder, paddings=paddings, mode=mode, constant_values=value)
-    for a in axes:
-      p = sum(paddings[a])
-      in_tag = self.input_data.dim_tags[a]
-      out_tag = self.output.dim_tags[a]
-      a = self.input_data.get_batch_axis_excluding_batch(a)
-      if a is None:
-        continue
-      if in_tag.dyn_size is None:
-        continue
-      if p == 0:
-        continue
-      size = in_tag.dyn_size
-      with tf_util.same_control_flow_ctx(size):
-        size = tf_util.simplify_add(size, p)
-      size_tag = Dim.get_tag_from_size_tensor(size)
-      if not size_tag:
-        out_tag.set_tag_on_size_tensor(size, batch=in_tag.batch)
-      else:
-        out_tag.declare_same_as(size_tag)
 
   @classmethod
   def _transform_padding(cls, padding, axes):
@@ -3302,10 +3244,8 @@ def get_out_data_from_opts(cls, name, sources, axes, padding, out_dims=None, **k
       pad_left, pad_right = padding[i]
       out_dim = pad_left + dim_tags[a] + pad_right
       if out_dims:
-        assert out_dims[i].dimension == out_dim.dimension
-        dim_tags[a] = out_dims[i]
-      else:
-        dim_tags[a] = out_dim
+        out_dim.declare_same_as(out_dims[i])
+      dim_tags[a] = out_dim
     return data.copy_template_new_dim_tags(dim_tags, keep_special_axes=True)
 
 
@@ -3488,6 +3428,7 @@ def get_out_data_from_opts(cls, name, axes, keep_order=NotSpecified,
     :rtype: Data
     """
     from returnn.util import BehaviorVersion
+    from returnn.util.basic import prod
     if keep_order is NotSpecified:
       keep_order = True if BehaviorVersion.get() >= 6 else False
     assert not out_type, "currently ignored"
@@ -3497,21 +3438,18 @@ def get_out_data_from_opts(cls, name, axes, keep_order=NotSpecified,
     data = input_data.copy(name="%s_output" % name)
     if len(axes) <= 1:
       return data
-    import numpy
     res_dim = None
     if all([data.batch_shape[i] is not None for i in axes]):
-      res_dim = int(numpy.prod([data.batch_shape[i] for i in axes]))
+      res_dim = int(prod([data.batch_shape[i] for i in axes]))
     merge_dim_tags = [data.dim_tags[axis] for axis in axes]
     merge_target_axis = cls._get_target_axis(input_data=data, merge_axes=axes)
+    out_dim_ = prod(merge_dim_tags)
+    assert isinstance(out_dim_, Dim)
+    assert out_dim_.dimension == res_dim
     if out_dim:
-      assert out_dim.dimension == res_dim
-    else:
-      from numpy import prod
-      out_dim = prod(merge_dim_tags)
-      assert isinstance(out_dim, Dim)
-      assert out_dim.dimension == res_dim
+      out_dim_.declare_same_as(out_dim)
     new_dim_tags = [d for (i, d) in enumerate(data.dim_tags) if i not in axes]
-    new_dim_tags.insert(merge_target_axis, out_dim)
+    new_dim_tags.insert(merge_target_axis, out_dim_)
 
     data_opts = data.get_kwargs(include_special_axes=False)
     data_opts["dim_tags"] = new_dim_tags
@@ -4412,12 +4350,13 @@ def get_out_data_from_opts(cls, name, sources, axis, repetitions, out_dim=None,
     original_axis = data.get_axis_from_description(axis, allow_int=False)
     tag = data.dim_tags[original_axis]
     data = data.copy_move_axis(original_axis, data.get_batch_axis(0))
-    if not out_dim:
-      if isinstance(repetitions, int):
-        out_dim = tag * repetitions
-      else:
-        out_dim = Dim(description="repeated:%s" % name, kind=tag.kind, derived_from_tag=tag, auto_generated=True)
-    return data.copy_template_replace_dim_tag(axis=data.get_batch_axis(0), new_dim_tag=out_dim)
+    if isinstance(repetitions, int):
+      out_dim_ = tag * repetitions
+    else:
+      out_dim_ = Dim(description="repeated:%s" % name, kind=tag.kind, derived_from_tag=tag, auto_generated=True)
+    if out_dim:
+      out_dim_.declare_same_as(out_dim)
+    return data.copy_template_replace_dim_tag(axis=data.get_batch_axis(0), new_dim_tag=out_dim_)
 
 
 class TileLayer(_ConcatInputLayer):
@@ -4459,13 +4398,9 @@ def get_out_data_from_opts(cls, name, sources, multiples, out_dims=None, **kwarg
     dim_tags = list(data.dim_tags)
     for axis, multiple in multiples.items():
       axis_int = data.get_axis_from_description(axis, allow_int=False)
-      tag = dim_tags[axis_int]
-      dim = None if tag.dimension is None else (tag.dimension * multiple)
+      tag = multiple * dim_tags[axis_int]
       if out_dims and axis in out_dims:
-        tag = out_dims[axis]
-        assert tag.dimension == dim
-      else:
-        tag = multiple * tag
+        tag.declare_same_as(out_dims[axis])
       dim_tags[axis_int] = tag
     return data.copy_template_new_dim_tags(dim_tags, keep_special_axes=True)
 
@@ -5021,13 +4956,10 @@ def __init__(self, filter_size, padding, strides=1, dilation_rate=1, groups=1,
       self.output_before_activation = OutputWithActivation(y)
     y = self.output_before_activation.y
     self.output.placeholder = y
-    self.set_output_dim_tags(
-      self.output, num_batch_dims=num_batch_dims, in_spatial_dims=in_spatial_dims_, out_spatial_dims=out_spatial_dims,
-      filter_size=filter_size, strides=strides, dilation_rate=dilation_rate, padding=padding)
 
   @classmethod
   def set_output_dim_tags(cls, output, num_batch_dims, in_spatial_dims, out_spatial_dims,
-                          filter_size, strides, dilation_rate, padding, calc_dyn_size=True):
+                          filter_size, strides, dilation_rate, padding):
     """
     :param Data output:
     :param int num_batch_dims:
@@ -5037,7 +4969,6 @@ def set_output_dim_tags(cls, output, num_batch_dims, in_spatial_dims, out_spatia
     :param list[int]|tuple[int] strides:
     :param list[int]|tuple[int] dilation_rate:
     :param str padding:
-    :param bool calc_dyn_size:
     """
     if output.feature_dim_axis == num_batch_dims:
       out_spatial_dims_ = output.dim_tags[num_batch_dims + 1:]
@@ -5045,7 +4976,9 @@ def set_output_dim_tags(cls, output, num_batch_dims, in_spatial_dims, out_spatia
       assert output.feature_dim_axis == output.batch_ndim - 1
       out_spatial_dims_ = output.dim_tags[num_batch_dims:-1]
     if out_spatial_dims:
-      assert list(out_spatial_dims_) == list(out_spatial_dims)
+      assert len(out_spatial_dims_) == len(out_spatial_dims)
+      for i, (out_spatial_dim_, out_spatial_dim) in enumerate(zip(out_spatial_dims_, out_spatial_dims)):
+        out_spatial_dim_.declare_same_as(out_spatial_dim)
     assert len(out_spatial_dims_) == len(in_spatial_dims) == len(filter_size) == len(strides) == len(dilation_rate)
     for i, in_tag in enumerate(in_spatial_dims):
       out_tag = out_spatial_dims_[i]
@@ -5055,25 +4988,6 @@ def set_output_dim_tags(cls, output, num_batch_dims, in_spatial_dims, out_spatia
         dilation_rate=dilation_rate[i], padding=padding)
       assert isinstance(out_tag_calc, Dim)
       out_tag_calc.declare_same_as(out_tag)
-      if in_tag.dimension is not None:
-        size = in_tag.dimension
-        size = cls.calc_out_dim(
-          in_dim=size,
-          filter_size=filter_size[i], stride=strides[i],
-          dilation_rate=dilation_rate[i], padding=padding)
-        assert out_tag.dimension == size
-      elif in_tag.dimension is None and in_tag.dyn_size is not None and calc_dyn_size:
-        size = in_tag.dyn_size
-        with tf_util.same_control_flow_ctx(size):
-          size = cls.calc_out_dim(
-            in_dim=size,
-            filter_size=filter_size[i], stride=strides[i],
-            dilation_rate=dilation_rate[i], padding=padding)
-        size_tag = Dim.get_tag_from_size_tensor(size)
-        if not size_tag:
-          out_tag.set_tag_on_size_tensor(size, batch=in_tag.batch)
-        else:
-          out_tag.declare_same_as(size_tag)
 
   @classmethod
   def _check_defined_in_spatial_dims(cls, cond):
@@ -5184,7 +5098,7 @@ def transform_input(cls, input_data, network, in_dim=None, in_spatial_dims=None,
   @classmethod
   def calc_out_dim(cls, in_dim, filter_size, stride, padding, dilation_rate=1):
     """
-    :param int|tf.Tensor|Dim|T in_dim: dimension in some axis
+    :param T|int|tf.Tensor|Dim in_dim: dimension in some axis
     :param int filter_size: e.g. 2, for the corresponding axis
     :param int stride: e.g. 1, for the corresponding axis
     :param int dilation_rate: e.g. 1
@@ -5212,7 +5126,7 @@ def ceildiv(a, b):
       if isinstance(in_dim, Dim):
         filter_left_dilated = (filter_size - 1) * dilation_rate // 2
         filter_right_dilated = (filter_size - 1) * dilation_rate - filter_left_dilated
-        valid_part = (-filter_left_dilated) + in_dim + (-filter_right_dilated)
+        valid_part = in_dim.sub_left(filter_left_dilated).sub_right(filter_right_dilated)
         return valid_part.ceildiv_right(stride)
       return tf_util.simplify_non_negative_seq_length(
         ceildiv(
@@ -5320,8 +5234,7 @@ def get_out_data_from_opts(
     if len(old_spatial_dim_tags) == len(filter_size):
       cls.set_output_dim_tags(
         out, num_batch_dims=num_batch_dims, in_spatial_dims=old_spatial_dim_tags, out_spatial_dims=out_spatial_dims,
-        filter_size=filter_size, strides=strides, dilation_rate=dilation_rate, padding=padding,
-        calc_dyn_size=False)
+        filter_size=filter_size, strides=strides, dilation_rate=dilation_rate, padding=padding)
     return out
 
   def get_dep_layers(self):
@@ -5445,9 +5358,6 @@ def __init__(self, mode, pool_size, padding="VALID", dilation_rate=1, strides=No
     if num_batch_dims > 1:
       y = tf.reshape(y, tf.concat([extended_batch_shape, tf.shape(y)[1:]], axis=0))
     self.output.placeholder = y
-    ConvLayer.set_output_dim_tags(
-      self.output, num_batch_dims=num_batch_dims, in_spatial_dims=in_spatial_dims_, out_spatial_dims=out_spatial_dims,
-      filter_size=pool_size, strides=strides, dilation_rate=dilation_rate, padding=padding)
 
   @classmethod
   def get_out_data_from_opts(cls, name, sources, network,
@@ -5683,26 +5593,6 @@ def __init__(self, filter_size, strides=None,
       self.output_before_activation = OutputWithActivation(y)
     y = self.output_before_activation.y
     self.output.placeholder = y
-    for idx, axis in enumerate(spatial_axes):
-      input_tag = input_data.dim_tags[axis]
-      output_tag = self.output.dim_tags[axis]
-      if input_tag == output_tag:
-        continue
-      assert not input_tag.is_batch_dim() and not output_tag.is_batch_dim()
-      if input_tag.dimension is None:
-        assert output_tag.dimension is None
-        assert input_tag.dyn_size is not None
-        size = input_tag.dyn_size
-        with tf_util.same_control_flow_ctx(size):
-          size = self.deconv_output_length(
-            size,
-            filter_size=filter_size[idx], stride=strides[idx],
-            padding=padding, output_padding=output_padding[idx])
-          r = remove_padding[idx]
-          if r:
-            assert isinstance(r, int)
-            size = tf_util.simplify_add(size, -r * 2)
-        output_tag.set_tag_on_size_tensor(size, batch=self.output.batch)
 
   @staticmethod
   def deconv_output_length(input_length,
@@ -5715,7 +5605,9 @@ def deconv_output_length(input_length,
     Determines output length of a transposed convolution given input length.
     Copied from conv_utils.deconv_output_length, adapted with simplification.
 
-    :param T|int|tf.Tensor input_length:
+    Also see :func:`ConvLayer.calc_out_dim`.
+
+    :param T|int|tf.Tensor|Dim input_length:
     :param int filter_size:
     :param str padding: one of `"same"`, `"valid"`, `"full"`.
     :param int|None output_padding: amount of padding along the output dimension.
@@ -5736,13 +5628,19 @@ def deconv_output_length(input_length,
     # Infer length if output padding is None, else compute the exact length
     if output_padding is None:
       if padding == 'valid':
-        length = tf_util.simplify_add(input_length, max(filter_size - stride, 0))
+        if isinstance(input_length, Dim):
+          length = input_length + max(filter_size - stride, 0)
+        else:
+          length = tf_util.simplify_add(input_length, max(filter_size - stride, 0))
       elif padding == 'full':
-        length = tf_util.simplify_add(input_length, -(stride + filter_size - 2))
+        if isinstance(input_length, Dim):
+          length = input_length - (stride + filter_size - 2)
+        else:
+          length = tf_util.simplify_add(input_length, -(stride + filter_size - 2))
       elif padding == 'same':
         length = input_length
       else:
-        length = None
+        raise Exception("invalid padding %r" % (padding,))
     else:  # output_padding
       if padding == 'same':
         pad = filter_size // 2
@@ -5751,8 +5649,11 @@ def deconv_output_length(input_length,
       elif padding == 'full':
         pad = filter_size - 1
       else:
-        pad = None
-      length = tf_util.simplify_add(input_length, -stride + filter_size - 2 * pad + output_padding)
+        raise Exception("invalid padding %r" % (padding,))
+      if isinstance(input_length, Dim):
+        length = input_length + (-stride + filter_size - 2 * pad + output_padding)
+      else:
+        length = tf_util.simplify_add(input_length, -stride + filter_size - 2 * pad + output_padding)
     return length
 
   @classmethod
@@ -5796,22 +5697,16 @@ def get_out_data_from_opts(cls, name, sources, network,
     dim_tags = list(data.dim_tags[:num_batch_dims])  # [B]
     if out_spatial_dims:
       assert len(out_spatial_dims) == len(filter_size)
-      # Be relaxed about incorrect input data. Throw errors later. This can also work during template construction.
-      dim_tags += out_spatial_dims
-    else:
-      for i in range(len(filter_size)):
-        old_tag = old_spatial_dim_tags[i] if i < len(old_spatial_dim_tags) else None
-        if old_tag and (filter_size[i] == strides[i] == 1 or (strides[i] == 1 and padding == "SAME")):
-          dim_tags.append(old_tag)  # identity in this axis
-          continue
-        new_dim = None
-        if old_tag and old_tag.dimension is not None:
-          new_dim = cls.deconv_output_length(
-            old_tag.dimension, filter_size=filter_size[i], stride=strides[i],
-            padding=padding, output_padding=output_padding[i]) - remove_padding[i] * 2
-        dim_tags.append(Dim(
-          kind=Dim.Types.Spatial, description="%s:conv:s%i" % (name, i), dimension=new_dim,
-          derived_from_tag=old_tag, undefined=not old_tag, auto_generated=True))
+    # Be relaxed about incorrect input data. Throw errors later. This can also work during template construction.
+    for i in range(len(filter_size)):
+      old_tag = old_spatial_dim_tags[i] if i < len(old_spatial_dim_tags) else None
+      new_tag = cls.deconv_output_length(
+        old_tag, filter_size=filter_size[i], stride=strides[i],
+        padding=padding, output_padding=output_padding[i])
+      new_tag = new_tag.sub_left(remove_padding[i]).sub_right(remove_padding[i])
+      if out_spatial_dims:
+        new_tag.declare_same_as(out_spatial_dims[i])
+      dim_tags.append(new_tag)
     if not out_dim:
       assert n_out
       out_dim = FeatureDim("%s:channel" % name, dimension=n_out, auto_generated=True)
@@ -6106,11 +6001,10 @@ def get_out_data_from_opts(cls, num_pieces, sources, name, out_dim=None, **kwarg
     assert out.have_feature_axis()
     assert not out.sparse
     assert out.dim % num_pieces == 0
-    dim = out.dim // num_pieces
-    if not out_dim:
-      out_dim = out.feature_dim_or_sparse_dim // num_pieces
-    assert out_dim.dimension == dim
-    return out.copy_template_replace_dim_tag(axis=out.feature_dim_axis, new_dim_tag=out_dim)
+    out_dim_ = out.feature_dim_or_sparse_dim // num_pieces
+    if out_dim:
+      out_dim_.declare_same_as(out_dim)
+    return out.copy_template_replace_dim_tag(axis=out.feature_dim_axis, new_dim_tag=out_dim_)
 
 
 class SqueezeLayer(_ConcatInputLayer):
@@ -6227,11 +6121,11 @@ def get_out_data_from_opts(cls, name, sources, axis=None, out_spatial_dim=None,
     if axis is None:
       axis = axis_
     out = common_source.copy_template(name="%s_output" % name)
-    if not out_spatial_dim:
-      out_spatial_dim = Dim(
-        kind=Dim.Types.Spatial, description="%s:stack" % name, dimension=len(sources), auto_generated=True)
-    assert out_spatial_dim.dimension == len(sources)
-    out = out.copy_add_dim_by_tag(axis=axis, dim_tag=out_spatial_dim, unbroadcast=True)
+    out_spatial_dim_ = Dim(
+      kind=Dim.Types.Spatial, description="%s:stack" % name, dimension=len(sources), auto_generated=True)
+    if out_spatial_dim:
+      out_spatial_dim_.declare_same_as(out_spatial_dim)
+    out = out.copy_add_dim_by_tag(axis=axis, dim_tag=out_spatial_dim_, unbroadcast=True)
     return out
 
 
@@ -6452,10 +6346,6 @@ def __init__(self, axis="T", out_dim=None, prefix=0.0, repeat=1, size_base=None,
       assert repeat >= 0
     self.repeat = repeat
     out_dim = self.output.dim_tags[axis_int]
-    if in_dim.dyn_size is not None and out_dim.dyn_size is None:
-      if not out_dim.dyn_size_ext:
-        out_dim.dyn_size_ext = in_dim.dyn_size_ext.copy()
-      out_dim.dyn_size_ext.placeholder = in_dim.dyn_size_ext.placeholder + repeat
     max_repeat = repeat if isinstance(repeat, int) else tf.maximum(tf.reduce_max(repeat), 0)
     shape = [((self.output.batch_shape[i] or tf.shape(input_data.placeholder)[i])
               if (i != axis_int)
@@ -6506,17 +6396,15 @@ def get_out_data_from_opts(cls, name, sources, axis="T", out_dim=None, size_base
     x = get_concat_sources_data_template(sources, name="%s_output" % name)
     axis_int = x.get_axis_from_description(axis, allow_int=False)
     in_dim = x.dim_tags[axis_int]
-    out_dim_int = None
-    if in_dim.dimension and isinstance(repeat, int):
-      out_dim_int = in_dim.dimension + repeat
     if size_base:
       assert not out_dim
-      out_dim = size_base.output.get_time_dim_tag()
-    if not out_dim:
-      out_dim = (
+      out_dim_ = size_base.output.get_time_dim_tag()
+    else:
+      out_dim_ = (
         repeat if isinstance(repeat, int) else SpatialDim("%s:repeat" % repeat.name, auto_generated=True)) + in_dim
-    assert out_dim.dimension == out_dim_int
-    x = x.copy_template_replace_dim_tag(axis=axis_int, new_dim_tag=out_dim)
+    if out_dim:
+      out_dim_.declare_same_as(out_dim)
+    x = x.copy_template_replace_dim_tag(axis=axis_int, new_dim_tag=out_dim_)
     if isinstance(repeat, LayerBase):
       x = x.copy_as_batch_spatial_major()
     return x
@@ -6571,8 +6459,6 @@ def __init__(self, axis="T", out_dim=None, postfix=0.0, repeat=1, **kwargs):
       seq_mask = tf.less(idx_range, size_ext.placeholder)
       assert seq_mask.get_shape().ndims == self.output.batch_ndim
       self.output.placeholder = tf_util.where_bc(seq_mask, x, c)
-      out_dim = self.output.dim_tags[axis_int]
-      out_dim.dyn_size = in_dim.dyn_size + repeat
 
   @classmethod
   def get_out_data_from_opts(cls, name, sources, axis="T", out_dim=None, postfix=0.0, repeat=1, **kwargs):
@@ -6588,13 +6474,10 @@ def get_out_data_from_opts(cls, name, sources, axis="T", out_dim=None, postfix=0
     x = get_concat_sources_data_template(sources, name="%s_output" % name)
     axis_int = x.get_axis_from_description(axis, allow_int=False)
     in_dim = x.dim_tags[axis_int]
-    out_dim_int = None
-    if in_dim.dimension:
-      out_dim_int = in_dim.dimension + repeat
-    if not out_dim:
-      out_dim = in_dim + repeat
-    assert out_dim.dimension == out_dim_int
-    x = x.copy_template_replace_dim_tag(axis=axis_int, new_dim_tag=out_dim)
+    out_dim_ = in_dim + repeat
+    if out_dim:
+      out_dim_.declare_same_as(out_dim)
+    x = x.copy_template_replace_dim_tag(axis=axis_int, new_dim_tag=out_dim_)
     return x
 
   @classmethod
@@ -7275,26 +7158,22 @@ def __init__(self, factor, axis, out_dim=None, kind="nn", fill_value=None, fill_
     :param Dim|None out_dim:
     :param str kind: "linear", "nn"/"nearest_neighbor", "cubic", "fill"
     :param None|int|float fill_value: if kind=="fill"
-    :param float fill_dropout: if set, will dropout in the same axis
+    :param float|None fill_dropout: if set, will dropout in the same axis
     """
     out_dim  # noqa  # via get_out_data_from_opts
     super(ResizeLayer, self).__init__(**kwargs)
     # self.output.shape and self.output.batch_dim_axis are already set here via self.get_out_data_from_opts().
     input_data = self.input_data.copy_as_batch_major()
-    axis = self.input_data.get_axis_from_description(axis)
+    axis = input_data.get_axis_from_description(axis)
     assert axis > 0, "batch-dim resize not supported"
     input_data = input_data.copy_move_axis(old_axis=axis, new_axis=1)
     axis = 1
-    self.output.placeholder = input_data.placeholder
-    out_dyn_size = input_data.dim_tags[axis].dyn_size
-    if out_dyn_size is not None:
-      out_dyn_size = out_dyn_size * factor
 
     # images expected shape: [batch, height, width, channels]
     remaining_axes = [i for i in range(self.output.batch_ndim) if i not in (0, axis)]
-    x = dimshuffle(self.output.placeholder, [0, axis, 'x'] + remaining_axes)  # [batch,height,width] + remaining_axes
+    x = dimshuffle(input_data.placeholder, [0, axis, 'x'] + remaining_axes)  # [batch,height,width] + remaining_axes
     from returnn.tf.util.basic import get_shape, optional_mul
-    shape = get_shape(self.output.placeholder)
+    shape = get_shape(input_data.placeholder)
     remaining_shape = [shape[i] for i in remaining_axes]
     remaining_dim = optional_mul(*remaining_shape) if remaining_axes else 1
     x = tf.reshape(x, [shape[0], shape[axis], 1, remaining_dim])  # [batch,height,width,channels]
@@ -7337,21 +7216,23 @@ def __init__(self, factor, axis, out_dim=None, kind="nn", fill_value=None, fill_
       mask = expand_dims_unbroadcast(mask, axis=0, dim=shape[0])  # (batch,new_size)
       x = tf.boolean_mask(x, mask)  # [batch*new_size_dropped] + remaining_shape
       x = tf.reshape(x, [shape[0], new_size_dropped] + remaining_shape)  # [batch, new_size_dropped] + remaining_shape
+      out_dyn_size = input_data.dim_tags[axis].dyn_size
       if out_dyn_size is not None:
+        out_dyn_size = out_dyn_size * factor
         orig_mask = tf.sequence_mask(
           out_dyn_size, maxlen=new_size, dtype=tf.bool)  # (batch,new_size)
         out_dyn_size = tf.reduce_sum(tf.cast(tf.logical_and(mask, orig_mask), tf.int32), axis=1)
-    if out_dyn_size is not None:
-      self.output.dim_tags[axis].dyn_size = out_dyn_size
+        self.output.dim_tags[axis].dyn_size = out_dyn_size
     self.output.placeholder = x
 
   @classmethod
-  def get_out_data_from_opts(cls, factor, axis, sources, name, out_dim=None, **kwargs):
+  def get_out_data_from_opts(cls, factor, axis, sources, name, fill_dropout=None, out_dim=None, **kwargs):
     """
     :param int factor:
     :param Dim|str axis:
     :param list[LayerBase] sources:
     :param str name:
+    :param float|None fill_dropout:
     :param Dim|None out_dim:
     :rtype: Data
     """
@@ -7362,12 +7243,13 @@ def get_out_data_from_opts(cls, factor, axis, sources, name, out_dim=None, **kwa
     axis = 1
     assert axis != out.batch_dim_axis, "batch-dim resize not supported"
     tag = out.dim_tags[axis]
-    dim = None if tag.dimension is None else (tag.dimension * factor)
-    if out_dim:
-      assert out_dim.dimension == dim
+    if fill_dropout:
+      out_dim_ = Dim(kind=tag.kind, description="%s_resize" % name, auto_generated=True)  # unknown dim
     else:
-      out_dim = Dim(kind=tag.kind, description="%s_resize" % name, dimension=dim, auto_generated=True)
-    return out.copy_template_replace_dim_tag(axis=axis, new_dim_tag=out_dim)
+      out_dim_ = tag * factor
+    if out_dim:
+      out_dim_.declare_same_as(out_dim)
+    return out.copy_template_replace_dim_tag(axis=axis, new_dim_tag=out_dim_)
 
 
 class CombineDimsLayer(MergeDimsLayer):
diff --git a/returnn/tf/util/data.py b/returnn/tf/util/data.py
--- a/returnn/tf/util/data.py
+++ b/returnn/tf/util/data.py
@@ -428,6 +428,9 @@ def dyn_size(self):
       If the dyn size can potentially be of a different shape, directly access dyn_size_ext.
     :rtype: tf.Tensor|None
     """
+    if self.dimension is None and (not self.dyn_size_ext or self.dyn_size_ext.placeholder is None):
+      # Try to complete.
+      self.complete_dyn_size()
     if self.dyn_size_ext:
       return self.dyn_size_ext.placeholder
     return None
@@ -611,14 +614,31 @@ def complete_dyn_size(self):
       if kind.endswith("_left"):
         kind = kind[:-len("_left")]
 
+      import numpy
+      from tensorflow.python.framework import tensor_util
+
+      def _is_negative(x__):
+        if isinstance(x__, numpy.ndarray):
+          return (x__ < 0).any()
+        if isinstance(x__, (int, float)):
+          return x__ < 0
+        assert isinstance(x__, tf.Tensor)
+        x__ = tensor_util.constant_value(x__)
+        if x__ is not None:
+          return _is_negative(x__)
+        return False
+
       def _bin_op(a, b):
         with tf_util.same_control_flow_ctx([a, b]):
           if kind == "add":
-            return tf_util.simplify_add(a, b)
+            use_relu = _is_negative(a) or _is_negative(b)  # for dynamic tensors, assume all positive
+            if use_relu:
+              return tf.convert_to_tensor(tf_util.simplify_non_negative_seq_length(tf_util.simplify_add(a, b)))
+            return tf.convert_to_tensor(tf_util.simplify_add(a, b))
           elif kind == "sub":
-            return tf_util.simplify_sub(a, b)
+            return tf.convert_to_tensor(tf_util.simplify_non_negative_seq_length(tf_util.simplify_sub(a, b)))
           elif kind == "mul":
-            return a * b
+            return tf.convert_to_tensor(tf_util.optional_mul(a, b))
           elif kind in ("floordiv", "truediv"):  # truediv assumes there is no remainder
             return a // b
           elif kind == "ceildiv":
@@ -641,7 +661,8 @@ def _bin_op(a, b):
             continue
           y.placeholder = _bin_op(y.placeholder, x.dimension)
           continue
-        x = x.get_for_batch_ctx(self.batch, self.control_flow_ctx)
+        if self.batch:
+          x = x.get_for_batch_ctx(self.batch, self.control_flow_ctx)
         x.complete_dyn_size()
         if not x.dyn_size_ext or x.dyn_size_ext.placeholder is None:
           return
@@ -899,6 +920,12 @@ def declare_same_as(self, other):
     elif other_same_base.dyn_size_ext is None or not other_same_base._validate_in_current_graph():
       other_same_base.dyn_size_ext = self.get_dyn_size_ext_for_batch_ctx(
         other_same_base.batch, other_same_base.control_flow_ctx)
+    if self.is_dim_known() and other.is_dim_known():
+      assert self.dimension == other.dimension
+    elif self.is_dim_known() and not other.is_dim_known():
+      other.dimension = self.dimension
+    elif not self.is_dim_known() and other.is_dim_known():
+      self.dimension = other.dimension
     if self._vocab and not other_same_base._vocab:
       other_same_base._vocab = self._vocab
     elif other_same_base._vocab and not self._vocab:


Test patch:
diff --git a/tests/test_TFNetworkLayer.py b/tests/test_TFNetworkLayer.py
--- a/tests/test_TFNetworkLayer.py
+++ b/tests/test_TFNetworkLayer.py
@@ -2564,6 +2564,22 @@ def test_MergeDimsLayer_modified_time_dim():
     session.run(out.placeholder, feed_dict=make_feed_dict(net.extern_data))
 
 
+def test_MergeDimsLayer_unspecified_out_dim():
+  # https://github.com/rwth-i6/returnn/issues/955
+  # https://github.com/rwth-i6/returnn_common/issues/117
+  config = Config({
+    "extern_data": {"data": {"shape": (None, 3, 5)}},
+  })
+  out_dim = SpatialDim("out")
+  with make_scope() as session:
+    net = TFNetwork(config=config)
+    net.construct_from_dict({
+      "output": {
+        "class": "merge_dims", "from": "data", "axes": ["dim:3", "dim:5"], "keep_order": True,
+        "out_dim": out_dim},
+    })
+
+
 def test_FlattenBatchLayer():
   from returnn.tf.util.data import BatchInfo
   n_batch, n_time, n_in = 3, 4, 2