At the moment, we assume that the user put the appropriate labels in the `labels` argument.
If `labels` is None, we try to infer it the best we can.

This could lead to some problem see #2029 and the discussion in #2094.