Only install libvirt if the driver is libvirt(!)

Ib85769e9a3470bede836ac0b87861ce76c0aedcf broke other drivers by force-installing
libvirt.  Only seen when we rebased the CI on the latest devstack-gate.

Change-Id: I99f5dbf8f0af8bdb2b76fb6834b20a96344dfa7d