interrupts: Only change memory context if it actually is different