How isotropic kernels perform on simple invariants

Abstract We investigate how the training curve of isotropic kernel methods depends on the symmetry of the task to be learned, in several settings. (i) We consider a regression task, where the target function is a Gaussian random field that depends only on <?CDATA $d_\parallel$?> <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:msub> <mml:mi>d</mml:mi> <mml:mo>∥</mml:mo> </mml:msub> </mml:math> variables, fewer than the input dimension d . We compute the expected test error ϵ that follows <?CDATA $\epsilon\sim p^{-\beta}$?> <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:mi>ϵ</mml:mi> <mml:mo>∼</mml:mo> <mml:msup> <mml:mi>p</mml:mi> <mml:mrow> <mml:mo>−</mml:mo> <mml:mi>β</mml:mi> </mml:mrow> </mml:msup> </mml:math> where p is the size of the training set. We find that β ∼ 1/ d independently of <?CDATA $d_\parallel$?> <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:msub> <mml:mi>d</mml:mi> <mml:mo>∥</mml:mo> </mml:msub> </mml:math> , supporting previous findings that the presence of invariants does not resolve the curse of dimensionality for kernel regression. (ii) Next we consider support-vector binary classification and introduce the stripe model , where the data label depends on a single coordinate <?CDATA $y(\underline x) = y(x_1)$?> <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:mi>y</mml:mi> <mml:mo stretchy="false">(</mml:mo> <mml:munder> <mml:mi>x</mml:mi> <mml:mo accent="true" stretchy="true">_</mml:mo> </mml:munder> <mml:mo stretchy="false">)</mml:mo> <mml:mo>=</mml:mo> <mml:mi>y</mml:mi> <mml:mo stretchy="false">(</mml:mo> <mml:msub> <mml:mi>x</mml:mi> <mml:mn>1</mml:mn> </mml:msub> <mml:mo stretchy="false">)</mml:mo> </mml:math> , corresponding to parallel decision boundaries separating labels of different signs, and consider that there is no margin at these interfaces. We argue and confirm numerically that, for large bandwidth, <?CDATA $\beta = \frac{d-1+\xi}{3d-3+\xi}$?> <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:mi>β</mml:mi> <mml:mo>=</mml:mo> <mml:mfrac> <mml:mrow> <mml:mi>d</mml:mi> <mml:mo>−</mml:mo> <mml:mn>1</mml:mn> <mml:mo>+</mml:mo> <mml:mi>ξ</mml:mi> </mml:mrow> <mml:mrow> <mml:mn>3</mml:mn> <mml:mi>d</mml:mi> <mml:mo>−</mml:mo> <mml:mn>3</mml:mn> <mml:mo>+</mml:mo> <mml:mi>ξ</mml:mi> </mml:mrow> </mml:mfrac> </mml:math> , where ξ ∈ (0, 2) is the exponent characterizing the singularity of the kernel at the origin. This estimation improves classical bounds obtainable from Rademacher complexity. In this setting there is no curse of dimensionality since <?CDATA $\beta\rightarrow 1/3$?> <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:mi>β</mml:mi> <mml:mo stretchy="false">→</mml:mo> <mml:mn>1</mml:mn> <mml:mrow> <mml:mo>/</mml:mo> </mml:mrow> <mml:mn>3</mml:mn> </mml:math> as <?CDATA $d\rightarrow\infty$?> <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:mi>d</mml:mi> <mml:mo stretchy="false">→</mml:mo> <mml:mi mathvariant="normal">∞</mml:mi> </mml:math> . (iii) We confirm these findings for the spherical model , for which <?CDATA $y(\underline x) = y(||\underline x||)$?> <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:mi>y</mml:mi> <mml:mo stretchy="false">(</mml:mo> <mml:munder> <mml:mi>x</mml:mi> <mml:mo accent="true" stretchy="true">_</mml:mo> </mml:munder> <mml:mo stretchy="false">)</mml:mo> <mml:mo>=</mml:mo> <mml:mi>y</mml:mi> <mml:mo stretchy="false">(</mml:mo> <mml:mrow> <mml:mo>|</mml:mo> </mml:mrow> <mml:mrow> <mml:mo>|</mml:mo> </mml:mrow> <mml:munder> <mml:mi>x</mml:mi> <mml:mo accent="true" stretchy="true">_</mml:mo> </mml:munder> <mml:mrow> <mml:mo>|</mml:mo> </mml:mrow> <mml:mrow> <mml:mo>|</mml:mo> </mml:mrow> <mml:mo stretchy="false">)</mml:mo> </mml:math> . (iv) In the stripe model, we show that, if the data are compressed along their invariants by some factor λ (an operation believed to take place in deep networks), the test error is reduced by a factor <?CDATA $\lambda^{-\frac{2(d-1)}{3d-3+\xi}}$?> <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:msup> <mml:mi>λ</mml:mi> <mml:mrow> <mml:mo>−</mml:mo> <mml:mfrac> <mml:mrow> <mml:mn>2</mml:mn> <mml:mo stretchy="false">(</mml:mo> <mml:mi>d</mml:mi> <mml:mo>−</mml:mo> <mml:mn>1</mml:mn> <mml:mo stretchy="false">)</mml:mo> </mml:mrow> <mml:mrow> <mml:mn>3</mml:mn> <mml:mi>d</mml:mi> <mml:mo>−</mml:mo> <mml:mn>3</mml:mn> <mml:mo>+</mml:mo> <mml:mi>ξ</mml:mi> </mml:mrow> </mml:mfrac> </mml:mrow> </mml:msup> </mml:math> .