fix intermittent crash in rotate-on-detail-view crash fix

Was sometimes crashing due to trying to remove observer too late
after detecting the newly loaded adapter. (Race condition?)

Change-Id: Ib62d78a2b7d863606070bf6d72ec37b9791bf243