Least-Mean-Squares Coresets for Infinite Streams

Consider a stream of <inline-formula><tex-math notation="LaTeX">$d$</tex-math><alternatives><mml:math><mml:mi>d</mml:mi></mml:math><inline-graphic xlink:href="statman-ieq1-3180808.gif"/></alternatives></inline-formula>-dimensional rows (points in <inline-formula><tex-math notation="LaTeX">$\mathbb {R}^{d}$</tex-math><alternatives><mml:math><mml:msup><mml:mi mathvariant="double-struck">R</mml:mi><mml:mi>d</mml:mi></mml:msup></mml:math><inline-graphic xlink:href="statman-ieq2-3180808.gif"/></alternatives></inline-formula>) arriving sequentially. An <inline-formula><tex-math notation="LaTeX">$\epsilon$</tex-math><alternatives><mml:math><mml:mi>ε</mml:mi></mml:math><inline-graphic xlink:href="statman-ieq3-3180808.gif"/></alternatives></inline-formula>-coreset is a positively weighted subset that approximates their sum of squared distances to any linear subspace of <inline-formula><tex-math notation="LaTeX">$\mathbb {R}^{d}$</tex-math><alternatives><mml:math><mml:msup><mml:mi mathvariant="double-struck">R</mml:mi><mml:mi>d</mml:mi></mml:msup></mml:math><inline-graphic xlink:href="statman-ieq4-3180808.gif"/></alternatives></inline-formula>, up to a <inline-formula><tex-math notation="LaTeX">$1 \pm \epsilon$</tex-math><alternatives><mml:math><mml:mrow><mml:mn>1</mml:mn><mml:mo>±</mml:mo><mml:mi>ε</mml:mi></mml:mrow></mml:math><inline-graphic xlink:href="statman-ieq5-3180808.gif"/></alternatives></inline-formula> factor. Unlike other data summarizations, such a coreset: (1) can be used to minimize faster any optimization function that uses this sum, such as regularized or constrained regression, (2) preserves input sparsity; (3) easily interpretable; (4) avoids numerical errors; (5) applies to problems with constraints on the input, such as subspaces that are spanned by few input points. Our main result is the first algorithm that returns such an <inline-formula><tex-math notation="LaTeX">$\epsilon$</tex-math><alternatives><mml:math><mml:mi>ε</mml:mi></mml:math><inline-graphic xlink:href="statman-ieq6-3180808.gif"/></alternatives></inline-formula>-coreset using finite and constant memory during the streaming, i.e., independent of <inline-formula><tex-math notation="LaTeX">$n$</tex-math><alternatives><mml:math><mml:mi>n</mml:mi></mml:math><inline-graphic xlink:href="statman-ieq7-3180808.gif"/></alternatives></inline-formula>, the number of rows seen so far. The coreset consists of <inline-formula><tex-math notation="LaTeX">$O(d \log ^{2}\;d / \epsilon ^{2})$</tex-math><alternatives><mml:math><mml:mrow><mml:mi>O</mml:mi><mml:mo>(</mml:mo><mml:mi>d</mml:mi><mml:msup><mml:mo form="prefix">log</mml:mo><mml:mn>2</mml:mn></mml:msup><mml:mspace width="0.277778em"/><mml:mi>d</mml:mi><mml:mo>/</mml:mo><mml:msup><mml:mi>ε</mml:mi><mml:mn>2</mml:mn></mml:msup><mml:mo>)</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="statman-ieq8-3180808.gif"/></alternatives></inline-formula> weighted rows, which is nearly optimal according to existing lower bounds of <inline-formula><tex-math notation="LaTeX">$\Omega (d / \epsilon ^{2})$</tex-math><alternatives><mml:math><mml:mrow><mml:mi>Ω</mml:mi><mml:mo>(</mml:mo><mml:mi>d</mml:mi><mml:mo>/</mml:mo><mml:msup><mml:mi>ε</mml:mi><mml:mn>2</mml:mn></mml:msup><mml:mo>)</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="statman-ieq9-3180808.gif"/></alternatives></inline-formula>. We support our findings with experiments on the Wikipedia dataset benchmarked against state-of-the-art algorithms.