Strange result.
`00 00 00 00 00 00 00 80` mantissa corresponds to real.nan mantissa. May be it means something.
Please, if you do not complicate, rewrite top `toUbyte` to run this algorithm in runtime: replace `if(__ctfe)` condition with `if(1)`.
After this, please run this function with real.max arg in runtime: `auto val = real.max; auto x = toUbyte(val);` and write x bytes.
If `x` value is correct than trouble in you compiler. What difference between current dmd master and you version?