Don't put all specialize opt in fast_compile_gpu!

This fix buildbot tests in FAST_COMPILE.
Fix test in FAST_COMPILE as now we register more opt in fast_compile.