kmsaudiomixer: Remove sometimes src pads

Change-Id: I462d02b21aac8d64df7ebed5b5818f2066db44a4