GenKL: An Iterative Framework for Resolving Label Ambiguity and Label Non-conformity in Web Images Via a New Generalized KL Divergence

Abstract Web image datasets curated online inherently contain ambiguous in-distribution instances and out-of-distribution instances, which we collectively call non-conforming (NC) instances. In many recent approaches for mitigating the negative effects of NC instances, the core implicit assumption is that the NC instances can be found via entropy maximization. For “entropy” to be well-defined, we are interpreting the output prediction vector of an instance as the parameter vector of a multinomial random variable, with respect to some trained model with a softmax output layer. Hence, entropy maximization is based on the idealized assumption that NC instances have predictions that are “almost” uniformly distributed. However, in real-world web image datasets, there are numerous NC instances whose predictions are far from being uniformly distributed. To tackle the limitation of entropy maximization, we propose $$(\alpha , \beta )$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:mo>(</mml:mo><mml:mi>α</mml:mi><mml:mo>,</mml:mo><mml:mi>β</mml:mi><mml:mo>)</mml:mo></mml:mrow></mml:math> -generalized KL divergence, $${\mathcal {D}}_{\text {KL}}^{\alpha , \beta }(p\Vert q)$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:msubsup><mml:mi>D</mml:mi><mml:mrow><mml:mtext>KL</mml:mtext></mml:mrow><mml:mrow><mml:mi>α</mml:mi><mml:mo>,</mml:mo><mml:mi>β</mml:mi></mml:mrow></mml:msubsup><mml:mrow><mml:mo>(</mml:mo><mml:mi>p</mml:mi><mml:mo>‖</mml:mo><mml:mi>q</mml:mi><mml:mo>)</mml:mo></mml:mrow></mml:mrow></mml:math> , which can be used to identify significantly more NC instances. Theoretical properties of $${\mathcal {D}}_{\text {KL}}^{\alpha , \beta }(p\Vert q)$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:msubsup><mml:mi>D</mml:mi><mml:mrow><mml:mtext>KL</mml:mtext></mml:mrow><mml:mrow><mml:mi>α</mml:mi><mml:mo>,</mml:mo><mml:mi>β</mml:mi></mml:mrow></mml:msubsup><mml:mrow><mml:mo>(</mml:mo><mml:mi>p</mml:mi><mml:mo>‖</mml:mo><mml:mi>q</mml:mi><mml:mo>)</mml:mo></mml:mrow></mml:mrow></mml:math> are proven, and we also show empirically that a simple use of $${\mathcal {D}}_{\text {KL}}^{\alpha , \beta }(p\Vert q)$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:msubsup><mml:mi>D</mml:mi><mml:mrow><mml:mtext>KL</mml:mtext></mml:mrow><mml:mrow><mml:mi>α</mml:mi><mml:mo>,</mml:mo><mml:mi>β</mml:mi></mml:mrow></mml:msubsup><mml:mrow><mml:mo>(</mml:mo><mml:mi>p</mml:mi><mml:mo>‖</mml:mo><mml:mi>q</mml:mi><mml:mo>)</mml:mo></mml:mrow></mml:mrow></mml:math> outperforms all baselines on the NC instance identification task. Building upon $$(\alpha ,\beta )$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:mo>(</mml:mo><mml:mi>α</mml:mi><mml:mo>,</mml:mo><mml:mi>β</mml:mi><mml:mo>)</mml:mo></mml:mrow></mml:math> -generalized KL divergence, we also introduce a new iterative training framework, GenKL , that identifies and relabels NC instances. When evaluated on three web image datasets, Clothing1M, Food101/Food101N, and mini WebVision 1.0, we achieved new state-of-the-art classification accuracies: $$81.34\%$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:mn>81.34</mml:mn><mml:mo>%</mml:mo></mml:mrow></mml:math> , $$85.73\%$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:mn>85.73</mml:mn><mml:mo>%</mml:mo></mml:mrow></mml:math> and $$78.99\%$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:mn>78.99</mml:mn><mml:mo>%</mml:mo></mml:mrow></mml:math> / $$92.54\%$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:mn>92.54</mml:mn><mml:mo>%</mml:mo></mml:mrow></mml:math> (top-1/top-5), respectively.