Don't end drag when parent removed