Remove handle when dispose is called for the first time