The soil from the outside garden is denser and more decadent than potting soil. It doesn't hold enough water, nutrients, and air to support the potted plants.