Although that works, limits are reached for space constrictions and efficient use of each bus. The best bet would be to have one card per bus and all the buses communicating with each other. InfiniBand is designed for high bandwidth, so the first likely targets would be midrange to high-end PC servers, those with 4 to 16 processors and more than 4 GB of memory. Some of those are still rare but, as technology progresses, they become more accessible and usable by the masses. The first thing you'll notice is that the switched architecture can take the peripherals out of the computer's main box.
Some products already follow that trend.
For example, the Dell PowerEdge 6000 series has mostly room for the processors and main memory, requiring peripherals and storage to go into secondary units. While breaking out the peripheral slots and storage into secondary units can introduce more potential failure points, the main processor node itself can be less complex. We don't need to support dual or triple peer PCI buses and hot-swappable disk systems in the main box. A smaller chance exists that maintenance on the drives or peripheral cards will cause trouble to the processor node's other internals. Without the drive system and any other moving parts within, the processor node needs fewer fans to keep it cool, consuming less power.