kmsaudiomixer: Fix workaround added in 2a4d2001ca014f6c35756a21caf225ba33609a5d

Manage the case where an input pad is removed before the chain of
elements to manage the audio input is created.

Change-Id: I66d2ef32a81273ad103c3c679e498b8ad59d84f0