Revert "bus: driverd: don't attempt to remove from empty list"

We need to find the real reason for the failure, this just
avoids the symptoms.

This reverts commit 215c7625dd510759290d45d6c46434e4338843ec.