use a sorted set and skip timeouts that are in the future