Fix double free bugs for sampler pointers