Fix libmath and X11 libs not being linked