make sure all docs have non-empty tokens after transformation