Properly handle physical address if set in flush_dcache and invalidate_icache