basemixer: Remove pads associated with a port when it is unhandled

Change-Id: I2b419850aabd28eb575493df0caadf159ad328b6