The best flooring for a compact home gym depends on the type of equipment you're purchasing.
For basic home gyms with limited equipment (perhaps a basic bench or a couple of sets of lightweight dumbbells), you may simply need foam mats or a yoga mat to help protect your floors from scuffs or scrapes and to make the exercises more comfortable on your hands or knees. But if you're purchasing a particularly heavy home gym unit or you'll be lifting heavier weights (dumbbells, kettlebells, or barbells, especially), you may want to invest in heavier-duty rubber flooring to help protect your home's floors from weight-related damage.