Fix shuffle handle clean issue