Some tweaks to the label inference code