Should we try an assert here that the allocated temp table bytes is back down to 0?