Make sure not add pending in every releaseUnrollMemory call