Fix remaining pointer issues and improve the Memory API