Yes, you can drive with an empty carrier. However, it is generally best to remove it if you aren't using it. This will improve your fuel economy. It is also likely to increase the longevity of the box if it is stored away from the elements. Driving will be slightly easier, particularly in high winds.