Optimize calculating bonuses from buffs display

Only consider buffs that the player actually has. This considerably
improves the load time of the character pane. It goes from ~300ms to
~150ms for me.