Free time gives children a chance to indulge their imaginations. Take little boys who play with plastic army men, for example. Regardless of any perceived violent overtones to that kind of play, children benefit from acting out their fantasies. Even if you take all the toy guns away from your children, their creative energy will seek an outlet. In the end, the key to any activity schedule is balance.
Children require a manageable mix of structured and unstructured fun.
If your children's schedules don't have enough downtime and you want them to unload an activity, always offer them a choice. Ask which activity they prefer to drop, and explain why you think it's a good idea. Also apologize for making them drop it. If you couch your mandate in a desire to spend more time with them, make good on your promise. A year later, he was back on the horse.