config/optimize: clean ldflags

things like -Os -fexcess-precision=fast -flto -ffat-lto-objects  ...
have no job in ldflags