Remove "controller" attribute if it is now longer needed.

In case "controller" was set to some number, we now remove it after it is no
longer needed to avoid locking side to non-existing controller type.

This fixes bug #21964.