Only use non-silent samples for the model adaption