Makefile: enable libdrm fo non xorg builds

Signed-off-by: Andreas Müller <schnitzeltony@googlemail.com>