Properly remove task refs from hasmap