kmsaudiomixer: Avoid invalid read in remove_adder

Change-Id: I885ce6ffa761bca6ae3a60cd070f0f5f4afe9802