Fixed bug the remote memory handle need to be set for each time