UBSan: Do not downcast to SwFlyFrmFmt prematurely

Change-Id: Ifcc05a5b23630fffed3a2f2636909bd1e1dff1fe