Fix "Switch to buffer with activity" skipping buffers with notifications

This caused two buffer switches in a row, in effect forgetting the notifications