Instance-Dependent Positive and Unlabeled Learning With Labeling Bias Estimation

This paper studies instance-dependent <underline>P</underline>ositive and <underline>U</underline>nlabeled (PU) classification, where whether a positive example will be labeled (indicated by <inline-formula><tex-math notation="LaTeX">$s$</tex-math><alternatives><mml:math><mml:mi>s</mml:mi></mml:math><inline-graphic xlink:href="gong-ieq1-3061456.gif"/></alternatives></inline-formula>) is not only related to the class label <inline-formula><tex-math notation="LaTeX">$y$</tex-math><alternatives><mml:math><mml:mi>y</mml:mi></mml:math><inline-graphic xlink:href="gong-ieq2-3061456.gif"/></alternatives></inline-formula>, but also depends on the observation <inline-formula><tex-math notation="LaTeX">$\mathbf {x}$</tex-math><alternatives><mml:math><mml:mi mathvariant="bold">x</mml:mi></mml:math><inline-graphic xlink:href="gong-ieq3-3061456.gif"/></alternatives></inline-formula>. Therefore, the labeling probability on positive examples is not uniform as previous works assumed, but is biased to some simple or critical data points. To depict the above dependency relationship, a graphical model is built in this paper which further leads to a maximization problem on the induced likelihood function regarding <inline-formula><tex-math notation="LaTeX">$P(s,y|\mathbf {x})$</tex-math><alternatives><mml:math><mml:mrow><mml:mi>P</mml:mi><mml:mo>(</mml:mo><mml:mi>s</mml:mi><mml:mo>,</mml:mo><mml:mi>y</mml:mi><mml:mo>|</mml:mo><mml:mi mathvariant="bold">x</mml:mi><mml:mo>)</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="gong-ieq4-3061456.gif"/></alternatives></inline-formula>. By utilizing the well-known EM and Adam optimization techniques, the labeling probability of any positive example <inline-formula><tex-math notation="LaTeX">$P(s=1|y=1,\mathbf {x})$</tex-math><alternatives><mml:math><mml:mrow><mml:mi>P</mml:mi><mml:mo>(</mml:mo><mml:mi>s</mml:mi><mml:mo>=</mml:mo><mml:mn>1</mml:mn><mml:mo>|</mml:mo><mml:mi>y</mml:mi><mml:mo>=</mml:mo><mml:mn>1</mml:mn><mml:mo>,</mml:mo><mml:mi mathvariant="bold">x</mml:mi><mml:mo>)</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="gong-ieq5-3061456.gif"/></alternatives></inline-formula> as well as the classifier induced by <inline-formula><tex-math notation="LaTeX">$P(y|\mathbf {x})$</tex-math><alternatives><mml:math><mml:mrow><mml:mi>P</mml:mi><mml:mo>(</mml:mo><mml:mi>y</mml:mi><mml:mo>|</mml:mo><mml:mi mathvariant="bold">x</mml:mi><mml:mo>)</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="gong-ieq6-3061456.gif"/></alternatives></inline-formula> can be acquired. Theoretically, we prove that the critical solution always exists, and is locally unique for linear model if some sufficient conditions are met. Moreover, we upper bound the generalization error for both linear logistic and non-linear network instantiations of our algorithm, with the convergence rate of expected risk to empirical risk as <inline-formula><tex-math notation="LaTeX">$\mathcal {O}(1/\sqrt{k}+1/\sqrt{n-k}+1/\sqrt{n})$</tex-math><alternatives><mml:math><mml:mrow><mml:mi mathvariant="script">O</mml:mi><mml:mo>(</mml:mo><mml:mn>1</mml:mn><mml:mo>/</mml:mo><mml:msqrt><mml:mi>k</mml:mi></mml:msqrt><mml:mo>+</mml:mo><mml:mn>1</mml:mn><mml:mo>/</mml:mo><mml:msqrt><mml:mrow><mml:mi>n</mml:mi><mml:mo>-</mml:mo><mml:mi>k</mml:mi></mml:mrow></mml:msqrt><mml:mo>+</mml:mo><mml:mn>1</mml:mn><mml:mo>/</mml:mo><mml:msqrt><mml:mi>n</mml:mi></mml:msqrt><mml:mo>)</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="gong-ieq7-3061456.gif"/></alternatives></inline-formula> (<inline-formula><tex-math notation="LaTeX">$k$</tex-math><alternatives><mml:math><mml:mi>k</mml:mi></mml:math><inline-graphic xlink:href="gong-ieq8-3061456.gif"/></alternatives></inline-formula> and <inline-formula><tex-math notation="LaTeX">$n$</tex-math><alternatives><mml:math><mml:mi>n</mml:mi></mml:math><inline-graphic xlink:href="gong-ieq9-3061456.gif"/></alternatives></inline-formula> are the sizes of positive set and the entire training set, respectively). Empirically, we compare our method with state-of-the-art instance-independent and instance-dependent PU algorithms on a wide range of synthetic, benchmark and real-world datasets, and the experimental results firmly demonstrate the advantage of the proposed method over the existing PU approaches.