build: Don't work around cortex-a8 issues when targeting a9 only

Don't work around cortex-a8 issues when other compiler flags
show we want to target cortex-a9 only

Change-Id: If5a077243fc15298fbcbe2ffc857b5e961dd29da
Signed-off-by: Bernhard Rosenkraenzer <Bernhard.Rosenkranzer@linaro.org>