Copied over behavior for one class binary case from current code, the labels default to negative