Unfortunate. Another option is to do a full copy of the nested map back into a mutable one above when we encounter a nested type.