Makefile: Fix up the optimization flags

Some optimization flags can't be applied to some parts or else boot fails.

Signed-off-by: Zhao Wei Liew <zhaoweiliew@gmail.com>