[Refactor] Use non-const MemoryAccess base addresses