Don't allow setting a new return position if there already is one and we haven't reached it yet