Don't destroy the interface destination when bringing it down