Correctly restore konts when memoizing