Most gardeners do because many soils used for gardening are less than ideal.
Perhaps they tend to be coarse-textured and droughty. Maybe they are heavy or finer-textured and have a tendency to stay wet for too long and become cloddy when finally dry. Maybe recent construction has scraped away much of the surface soil or created severely compacted conditions. The only practical way to improve these less than ideal soils is to add organic residues and do it often. Most Wisconsin soils have between 1 and 5 percent stable organic matter in them.