Only calculate fitted values if requested