Fix build if just one of libverbs or librdma is installed

Signed-off-by: Jens Axboe <axboe@kernel.dk>