Don't prune the backlog if a buffer has unread highlights