Remove bridge and vlan interface unconditionally on removing vlan network

It's observed that the low layer interfaces could be down for some reason,
such as that the physical interface is in disconnected state. In this case,
we still need destroy the interfaces to avoid that they're left on the system
util reboot.

Signed-off-by: Mark Wu <wudxw@linux.vnet.ibm.com>