Fix "Switch to buffer with activity" skipping buffers with notifications

This caused two buffer switches in a row, in effect forgetting the notifications
Merge pull request #170 from lorenzhs/fixactivityswitch

Fix "Switch to buffer with activity" skipping buffers with notifications