Add lto flags at link time as well.  Check for -ffat-lto-objects flat seperately.