See if enabling hardening on Ubuntu is now fixed.

We remove the unsetting of the LDFLAGS so on Ubuntu it should now start
using the default hardening flags again.