Only remove delegates on destroy when its the shared instance