Use atomic longs in the map and locking the whole class shouldn't be needed