Fix linkage without infiniband verbs

Change-Id: I628f976e7241c05bd855fd5e1ec7ba00ede6d065
Signed-off-by: Or Kehati <ork@mellanox.com>