Cleanup resources properly on incoming shuffles