Are you certain that the orientation flag is always in the first IFD?  It was never clear to me what is in the first one and what might be in the EXIFIFD, but maybe I just didn't read the spec carefully enough.