I would avoid doing any king of magic with the label inference.  What is the current behavior?