config: Fix the host and build options for Canadian cross builds.

The ARM on 4.9.0 with clang fails and needs an extra option and
this is only valid building the build cross compiler. This change
fxies the host and build flags so they only apply where needed.