Your horse won't build muscles without training stimulus. Setting a training stimulus means that you work up to or slightly over the performance limit. Of course, this does not mean that you let your horse gallop to the point of exhaustion! Rather, you work into the zone that is clearly strenuous.
Hobby riders tend to do this too little! Most people who ride their horses for pleasure tend to do too little rather than too much.