fbserver: Better tracking of pressed key/buttons.

Adding O(1), removing O(n).