fix #35176: ottava layout in various special situations

Replaces PR https://github.com/musescore/MuseScore/pull/1364; see comments there.