If the defender doesn't have any building on the map, you are not subject to the raid dice limit and can collect more than one raid die.